Skip to content

Docker Deployment#

The REF is designed to be run as a set of services that can be deployed to any cloud provider. Each of these services can be run as a separate docker container. This allows for easy scaling and deployment of each of these services.

An example docker-compose file is provided in the repository for the CMIP7 AFT deployment of the REF using the Celery executor.

This stack contains the following services:

  • climate-ref: The compute engine service that orchestrates the execution of the diagnostics.
  • redis: The message broker that manages the communication between the REF and the task queue.
  • postgres: The database that stores the results of the diagnostics and the metadata about the datasets.
  • climate-ref-pmp, climate-ref-esmvaltool and climate-ref-ilamb: The diagnostic providers that run the diagnostics.

One limitation of running the REF via docker is that all the containers must use the same paths to refer to datasets and the scratch space where outputs are written. In practise this is achieved by sharing a set of mounts for all containers.

The REF docker compose stack requires some bootstrapping to be done before it can be run. This bootstrapping migrates the database and builds the conda environments for PMP and ESMValTool.

bash scripts/initialise-docker.sh

An example docker-compose file is provided in the repository for the CMIP7 AFT deployment of the REF, using the Celery executor and Redis as the message broker. This can be run with:

docker-compose up

This will start the services and allow you to view the logs in the terminal. This can be run in the background with:

docker-compose up -d

The docker-compose file also mounts the sample data directory to the /ref/data directory in the container. These data can be ingested into the REF using the following commands (note that /ref/data/CMIP6 refers to a directory inside the docker container, not on your local filesystem).

docker-compose run --rm climate-ref datasets ingest --source-type cmip6 /ref/data/CMIP6
docker-compose run --rm climate-ref datasets ingest --source-type obs4mips /ref/data/obs4ref

After the data has been ingested, the REF can be run using the following command.

docker-compose run --rm climate-ref solve

This stack and its associated data volumes can be cleaned up with the following command.

docker-compose down -v --remove-orphans

This is remove the database, any results that have been generated by the REF and remove any orphaned containers. All data will be lost, so be careful with this command.