Testing diagnostic providers locally¶
Diagnostic providers can be run locally without requiring the rest of the REF infrastructure. This is useful for testing and debugging diagnostics.
Running a diagnostic locally requires that the target REF diagnostics package, e.g. climate_ref_example,
and its dependencies are installed in the current Python environment.
This guide will walk you through how to run a diagnostic provider locally.
In [2]:
Copied!
provider = climate_ref_example.provider
provider
provider = climate_ref_example.provider
provider
Out[2]:
DiagnosticProvider(name='Example', version='0.7.0')
We select a diagnostic which simply calculates the annual mean, global mean timeseries of a dataset.
The data requirements of this diagnostic filter out all variables except tas and rsut.
The group_by specification ensures that each execution has a unique combination of
source_id, variant_id, variable_id and experiment_id values.
In [3]:
Copied!
diagnostic = provider.get("global-mean-timeseries")
diagnostic = provider.get("global-mean-timeseries")
In [4]:
Copied!
prettyprinter.pprint(diagnostic.data_requirements[0])
prettyprinter.pprint(diagnostic.data_requirements[0])
climate_ref_core.diagnostics.DataRequirement(
source_type=climate_ref_core.datasets.SourceDatasetType.CMIP6,
filters=(
climate_ref_core.datasets.FacetFilter(
facets={'variable_id': ('tas', 'rsut')}
),
),
group_by=(
'source_id',
'variable_id',
'experiment_id',
'variant_label'
),
constraints=(
climate_ref_core.constraints.AddSupplementaryDataset(
supplementary_facets={'variable_id': 'areacella'},
matching_facets=('source_id', 'grid_label'),
optional_matching_facets=(
'table_id',
'experiment_id',
'member_id',
'version'
)
),
climate_ref_core.constraints.RequireContiguousTimerange(
group_by=('instance_id',)
)
)
)
In [5]:
Copied!
config = Config.default()
provider.configure(config)
db = Database.from_config(config)
config = Config.default()
provider.configure(config)
db = Database.from_config(config)
2025-11-06 11:44:48.244 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.256 | DEBUG | climate_ref_core.providers:configure:81 - Configuring provider example using ignore_datasets_file /home/docs/.cache/climate_ref/default_ignore_datasets.yaml
2025-11-06 11:44:48.265 | INFO | climate_ref.database:__init__:158 - Connecting to database at sqlite:////home/docs/.config/climate_ref/db/climate_ref.db
2025-11-06 11:44:48.272 | DEBUG | climate_ref.database:migrate:200 - Current database revision: 20cd136a5b04
2025-11-06 11:44:48.273 | INFO | climate_ref.database:_create_backup:86 - Creating backup of database at /home/docs/.config/climate_ref/db/backups/climate_ref_20251106_114448.db
2025-11-06 11:44:48.279 | DEBUG | env_py:<module>:21 - Running alembic env
2025-11-06 11:44:48.298 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: source_id
2025-11-06 11:44:48.299 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: reference_source_id
2025-11-06 11:44:48.300 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: experiment_id
2025-11-06 11:44:48.301 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: variable_id
2025-11-06 11:44:48.302 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: reference_variable_id
2025-11-06 11:44:48.302 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: grid_label
2025-11-06 11:44:48.303 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: member_id
2025-11-06 11:44:48.304 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: variant_label
2025-11-06 11:44:48.305 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: metric
2025-11-06 11:44:48.306 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: region
2025-11-06 11:44:48.306 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: season
2025-11-06 11:44:48.307 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: mode
2025-11-06 11:44:48.308 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: method
2025-11-06 11:44:48.308 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered MetricValue dimension: statistic
2025-11-06 11:44:48.309 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: source_id
2025-11-06 11:44:48.310 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: reference_source_id
2025-11-06 11:44:48.311 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: experiment_id
2025-11-06 11:44:48.311 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: variable_id
2025-11-06 11:44:48.312 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: reference_variable_id
2025-11-06 11:44:48.312 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: grid_label
2025-11-06 11:44:48.312 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: member_id
2025-11-06 11:44:48.313 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: variant_label
2025-11-06 11:44:48.314 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: metric
2025-11-06 11:44:48.314 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: region
2025-11-06 11:44:48.315 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: season
2025-11-06 11:44:48.315 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: mode
2025-11-06 11:44:48.316 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: method
2025-11-06 11:44:48.316 | DEBUG | climate_ref.models.mixins:register_cv_dimensions:117 - Registered ExecutionOutput dimension: statistic
Load the data catalog containing the CMIP6 datasets.
This contains the datasets that have been ingested into the REF database.
You could also use the find_local_datasets function to find local datasets on disk,
thereby bypassing the need for a database.
In [6]:
Copied!
# Load the data catalog containing the
data_catalog = get_dataset_adapter("cmip6").load_catalog(db)
data_catalog.head()
# Load the data catalog containing the
data_catalog = get_dataset_adapter("cmip6").load_catalog(db)
data_catalog.head()
2025-11-06 11:44:48.320 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
Out[6]:
| start_time | end_time | path | activity_id | branch_method | branch_time_in_child | branch_time_in_parent | experiment | experiment_id | frequency | ... | variable_id | variant_label | member_id | vertical_levels | version | standard_name | long_name | units | finalised | instance_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 57 | None | None | /home/docs/checkouts/readthedocs.org/user_buil... | CMIP | standard | 0.0 | 267840.0 | pre-industrial control | piControl | fx | ... | areacello | r1i1p1f1 | r1i1p1f1 | 1 | v20190709 | cell_area | Grid-Cell Area for Ocean Variables | m2 | True | CMIP6.CMIP.MOHC.HadGEM3-GC31-LL.piControl.r1i1... |
| 58 | None | None | /home/docs/checkouts/readthedocs.org/user_buil... | CMIP | standard | 0.0 | 267840.0 | pre-industrial control | piControl | fx | ... | areacella | r1i1p1f1 | r1i1p1f1 | 1 | v20190709 | cell_area | Grid-Cell Area for Atmospheric Grid Variables | m2 | True | CMIP6.CMIP.MOHC.HadGEM3-GC31-LL.piControl.r1i1... |
| 59 | 1870-01-16 12:00:00 | 1889-12-16 12:00:00 | /home/docs/checkouts/readthedocs.org/user_buil... | CMIP | standard | 0.0 | 146097.0 | pre-industrial control simulation with CO2 con... | esm-piControl | mon | ... | tas | r1i1p1f1 | r1i1p1f1 | 1 | v20190815 | air_temperature | Near-Surface Air Temperature | K | True | CMIP6.CMIP.MPI-M.MPI-ESM1-2-LR.esm-piControl.r... |
| 59 | 1910-01-16 12:00:00 | 1915-12-16 12:00:00 | /home/docs/checkouts/readthedocs.org/user_buil... | CMIP | standard | 0.0 | 146097.0 | pre-industrial control simulation with CO2 con... | esm-piControl | mon | ... | tas | r1i1p1f1 | r1i1p1f1 | 1 | v20190815 | air_temperature | Near-Surface Air Temperature | K | True | CMIP6.CMIP.MPI-M.MPI-ESM1-2-LR.esm-piControl.r... |
| 59 | 1890-01-16 12:00:00 | 1909-12-16 12:00:00 | /home/docs/checkouts/readthedocs.org/user_buil... | CMIP | standard | 0.0 | 146097.0 | pre-industrial control simulation with CO2 con... | esm-piControl | mon | ... | tas | r1i1p1f1 | r1i1p1f1 | 1 | v20190815 | air_temperature | Near-Surface Air Temperature | K | True | CMIP6.CMIP.MPI-M.MPI-ESM1-2-LR.esm-piControl.r... |
5 rows × 36 columns
Below, the unique combinations of the metadata values that apply to the groupby are shown:
In [7]:
Copied!
data_catalog[["source_id", "variant_label", "variable_id", "experiment_id"]].drop_duplicates()
data_catalog[["source_id", "variant_label", "variable_id", "experiment_id"]].drop_duplicates()
Out[7]:
| source_id | variant_label | variable_id | experiment_id | |
|---|---|---|---|---|
| 57 | HadGEM3-GC31-LL | r1i1p1f1 | areacello | piControl |
| 58 | HadGEM3-GC31-LL | r1i1p1f1 | areacella | piControl |
| 59 | MPI-ESM1-2-LR | r1i1p1f1 | tas | esm-piControl |
| 60 | MPI-ESM1-2-LR | r1i1p1f1 | areacella | esm-piControl |
| 61 | CESM2 | r1i1p1f1 | cli | historical |
| ... | ... | ... | ... | ... |
| 52 | ACCESS-ESM1-5 | r1i1p1f1 | rsut | piControl |
| 53 | ACCESS-ESM1-5 | r1i1p1f1 | tas | piControl |
| 54 | ACCESS-ESM1-5 | r1i1p1f1 | areacella | piControl |
| 55 | HadGEM3-GC31-LL | r1i1p1f3 | tas | historical |
| 56 | HadGEM3-GC31-LL | r1i1p1f3 | siconc | historical |
85 rows × 4 columns
Executions¶
An execution is a combination of a diagnostic, a provider, and a group of datasets.
The DiagnosticSolver is used to determine which executions are required given a set of requirements
and the currently available datasets.
This doesn't require the use of the REF database.
In [8]:
Copied!
executions_generator = solve_executions(
data_catalog={
SourceDatasetType.CMIP6: data_catalog,
},
diagnostic=diagnostic,
provider=provider,
)
# Convert from a generator to a list to inspect the complete set of executions
executions = list(executions_generator)
prettyprinter.pprint(executions)
executions_generator = solve_executions(
data_catalog={
SourceDatasetType.CMIP6: data_catalog,
},
diagnostic=diagnostic,
provider=provider,
)
# Convert from a generator to a list to inspect the complete set of executions
executions = list(executions_generator)
prettyprinter.pprint(executions)
2025-11-06 11:44:48.569 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.578 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.586 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.594 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.603 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.611 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.619 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.627 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.635 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.643 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.651 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.658 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.667 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.675 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
2025-11-06 11:44:48.683 | DEBUG | climate_ref.config:default:537 - Loading default configuration from /home/docs/.config/climate_ref/ref.toml
[
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
13 0101-01-16 12:00:00 0125-12-16 12:00:00
15 None None
path activity_id \
13 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
15 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
13 standard 0.0 0.0
15 standard 0.0 0.0
experiment experiment_id frequency ... variable_id \
13 abrupt quadrupling of CO2 abrupt-4xCO2 mon ... rsut
15 abrupt quadrupling of CO2 abrupt-4xCO2 fx ... areacella
variant_label member_id vertical_levels version \
13 r1i1p1f1 r1i1p1f1 1 v20191115
15 r1i1p1f1 r1i1p1f1 1 v20191115
standard_name \
13 toa_outgoing_shortwave_flux
15 cell_area
long_name units finalised \
13 TOA Outgoing Shortwave Radiation W m-2 True
15 Grid-Cell Area for Atmospheric Grid Variables m2 True
instance_id
13 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1...
15 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'abrupt-4xCO2'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'rsut'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
23 1996-01-16 12:00:00 2014-12-16 12:00:00
45 None None
path activity_id \
23 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
45 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
23 standard 0.0 21915.0
45 standard 0.0 21915.0
experiment experiment_id frequency ... \
23 all-forcing simulation of the recent past historical mon ...
45 all-forcing simulation of the recent past historical fx ...
variable_id variant_label member_id vertical_levels version \
23 rsut r1i1p1f1 r1i1p1f1 1 v20191115
45 areacella r1i1p1f1 r1i1p1f1 1 v20191115
standard_name \
23 toa_outgoing_shortwave_flux
45 cell_area
long_name units finalised \
23 TOA Outgoing Shortwave Radiation W m-2 True
45 Grid-Cell Area for Atmospheric Grid Variables m2 True
instance_id
23 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.historical.r1i1...
45 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.historical.r1i1...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'historical'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'rsut'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
52 0101-01-16 12:00:00 0125-12-16 12:00:00
54 None None
path activity_id \
52 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
54 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
52 standard 0.0 36524.0
54 standard 0.0 36524.0
experiment experiment_id frequency ... variable_id \
52 pre-industrial control piControl mon ... rsut
54 pre-industrial control piControl fx ... areacella
variant_label member_id vertical_levels version \
52 r1i1p1f1 r1i1p1f1 1 v20210316
54 r1i1p1f1 r1i1p1f1 1 v20210316
standard_name \
52 toa_outgoing_shortwave_flux
54 cell_area
long_name units finalised \
52 TOA Outgoing Shortwave Radiation W m-2 True
54 Grid-Cell Area for Atmospheric Grid Variables m2 True
instance_id
52 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.piControl.r1i1p...
54 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.piControl.r1i1p...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'piControl'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'rsut'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
84 2015-01-16 12:00:00 2025-12-16 12:00:00
87 None None
path activity_id \
84 /home/docs/checkouts/readthedocs.org/user_buil... ScenarioMIP
87 /home/docs/checkouts/readthedocs.org/user_buil... ScenarioMIP
branch_method branch_time_in_child branch_time_in_parent \
84 standard 60265.0 60265.0
87 standard 60265.0 60265.0
experiment experiment_id frequency ... variable_id \
84 update of RCP2.6 based on SSP1 ssp126 mon ... rsut
87 update of RCP2.6 based on SSP1 ssp126 fx ... areacella
variant_label member_id vertical_levels version \
84 r1i1p1f1 r1i1p1f1 1 v20210318
87 r1i1p1f1 r1i1p1f1 1 v20210318
standard_name \
84 toa_outgoing_shortwave_flux
87 cell_area
long_name units finalised \
84 TOA Outgoing Shortwave Radiation W m-2 True
87 Grid-Cell Area for Atmospheric Grid Variables m2 True
instance_id
84 CMIP6.ScenarioMIP.CSIRO.ACCESS-ESM1-5.ssp126.r...
87 CMIP6.ScenarioMIP.CSIRO.ACCESS-ESM1-5.ssp126.r...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'ssp126'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'rsut'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
9 0101-01-16 12:00:00 0250-12-16 12:00:00
10 None None
path activity_id \
9 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
10 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
9 standard 0.0 0.0
10 standard 0.0 0.0
experiment experiment_id frequency ... \
9 1 percent per year increase in CO2 1pctCO2 mon ...
10 1 percent per year increase in CO2 1pctCO2 fx ...
variable_id variant_label member_id vertical_levels version \
9 tas r1i1p1f1 r1i1p1f1 1 v20191115
10 areacella r1i1p1f1 r1i1p1f1 1 v20191115
standard_name long_name units \
9 air_temperature Near-Surface Air Temperature K
10 cell_area Grid-Cell Area for Atmospheric Grid Variables m2
finalised instance_id
9 True CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.1pctCO2.r1i1p1f...
10 True CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.1pctCO2.r1i1p1f...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', '1pctCO2'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'tas'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
14 0101-01-16 12:00:00 0125-12-16 12:00:00
15 None None
path activity_id \
14 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
15 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
14 standard 0.0 0.0
15 standard 0.0 0.0
experiment experiment_id frequency ... variable_id \
14 abrupt quadrupling of CO2 abrupt-4xCO2 mon ... tas
15 abrupt quadrupling of CO2 abrupt-4xCO2 fx ... areacella
variant_label member_id vertical_levels version standard_name \
14 r1i1p1f1 r1i1p1f1 1 v20191115 air_temperature
15 r1i1p1f1 r1i1p1f1 1 v20191115 cell_area
long_name units finalised \
14 Near-Surface Air Temperature K True
15 Grid-Cell Area for Atmospheric Grid Variables m2 True
instance_id
14 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1...
15 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'abrupt-4xCO2'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'tas'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
1 0168-01-16 12:00:00 0268-12-16 12:00:00
2 None None
path activity_id \
1 /home/docs/checkouts/readthedocs.org/user_buil... C4MIP CDRMIP
2 /home/docs/checkouts/readthedocs.org/user_buil... C4MIP CDRMIP
branch_method branch_time_in_child branch_time_in_parent \
1 standard 24471.0 24471.0
2 standard 24471.0 24471.0
experiment experiment_id \
1 zero emissions simulation branched from 1% run... esm-1pct-brch-1000PgC
2 zero emissions simulation branched from 1% run... esm-1pct-brch-1000PgC
frequency ... variable_id variant_label member_id vertical_levels \
1 mon ... tas r1i1p1f1 r1i1p1f1 1
2 fx ... areacella r1i1p1f1 r1i1p1f1 1
version standard_name long_name \
1 v20191206 air_temperature Near-Surface Air Temperature
2 v20191206 cell_area Grid-Cell Area for Atmospheric Grid Variables
units finalised instance_id
1 K True CMIP6.C4MIP CDRMIP.CSIRO.ACCESS-ESM1-5.esm-1pc...
2 m2 True CMIP6.C4MIP CDRMIP.CSIRO.ACCESS-ESM1-5.esm-1pc...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'esm-1pct-brch-1000PgC'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'tas'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
25 1850-01-16 12:00:00 2014-12-16 12:00:00
45 None None
path activity_id \
25 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
45 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
25 standard 0.0 21915.0
45 standard 0.0 21915.0
experiment experiment_id frequency ... \
25 all-forcing simulation of the recent past historical mon ...
45 all-forcing simulation of the recent past historical fx ...
variable_id variant_label member_id vertical_levels version \
25 tas r1i1p1f1 r1i1p1f1 1 v20191115
45 areacella r1i1p1f1 r1i1p1f1 1 v20191115
standard_name long_name units \
25 air_temperature Near-Surface Air Temperature K
45 cell_area Grid-Cell Area for Atmospheric Grid Variables m2
finalised instance_id
25 True CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.historical.r1i1...
45 True CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.historical.r1i1...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'historical'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'tas'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
53 0101-01-16 12:00:00 0180-12-16 12:00:00
54 None None
path activity_id \
53 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
54 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
53 standard 0.0 36524.0
54 standard 0.0 36524.0
experiment experiment_id frequency ... variable_id \
53 pre-industrial control piControl mon ... tas
54 pre-industrial control piControl fx ... areacella
variant_label member_id vertical_levels version standard_name \
53 r1i1p1f1 r1i1p1f1 1 v20210316 air_temperature
54 r1i1p1f1 r1i1p1f1 1 v20210316 cell_area
long_name units finalised \
53 Near-Surface Air Temperature K True
54 Grid-Cell Area for Atmospheric Grid Variables m2 True
instance_id
53 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.piControl.r1i1p...
54 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.piControl.r1i1p...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'piControl'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'tas'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
85 2015-01-16 12:00:00 2100-12-16 12:00:00
87 None None
path activity_id \
85 /home/docs/checkouts/readthedocs.org/user_buil... ScenarioMIP
87 /home/docs/checkouts/readthedocs.org/user_buil... ScenarioMIP
branch_method branch_time_in_child branch_time_in_parent \
85 standard 60265.0 60265.0
87 standard 60265.0 60265.0
experiment experiment_id frequency ... variable_id \
85 update of RCP2.6 based on SSP1 ssp126 mon ... tas
87 update of RCP2.6 based on SSP1 ssp126 fx ... areacella
variant_label member_id vertical_levels version standard_name \
85 r1i1p1f1 r1i1p1f1 1 v20210318 air_temperature
87 r1i1p1f1 r1i1p1f1 1 v20210318 cell_area
long_name units finalised \
85 Near-Surface Air Temperature K True
87 Grid-Cell Area for Atmospheric Grid Variables m2 True
instance_id
85 CMIP6.ScenarioMIP.CSIRO.ACCESS-ESM1-5.ssp126.r...
87 CMIP6.ScenarioMIP.CSIRO.ACCESS-ESM1-5.ssp126.r...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'ssp126'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'tas'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
68 1996-01-15 12:00:00 2014-12-15 12:00:00
75 None None
path activity_id \
68 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
75 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
68 standard 674885.0 219000.0
75 standard 674885.0 219000.0
experiment experiment_id frequency ... \
68 all-forcing simulation of the recent past historical mon ...
75 all-forcing simulation of the recent past historical fx ...
variable_id variant_label member_id vertical_levels version \
68 rsut r1i1p1f1 r1i1p1f1 1 v20190308
75 areacella r1i1p1f1 r1i1p1f1 1 v20190308
standard_name \
68 toa_outgoing_shortwave_flux
75 cell_area
long_name units finalised \
68 TOA Outgoing Shortwave Radiation W m-2 True
75 Grid-Cell Area for Atmospheric Grid Variables m2 True
instance_id
68 CMIP6.CMIP.NCAR.CESM2.historical.r1i1p1f1.Amon...
75 CMIP6.CMIP.NCAR.CESM2.historical.r1i1p1f1.fx.a...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'historical'), ('source_id', 'CESM2'), ('variable_id', 'rsut'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
5 1979-01-16 12:00:00 2014-12-16 12:00:00
8 None None
path activity_id \
5 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
8 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
5 Spin-up documentation 0.0 1223115.0
8 Spin-up documentation 0.0 1223115.0
experiment experiment_id frequency ... \
5 all-forcing simulation of the recent past historical mon ...
8 all-forcing simulation of the recent past historical fx ...
variable_id variant_label member_id vertical_levels version \
5 tas r1i1p1f1 r1i1p1f1 1 v20190429
8 areacella r1i1p1f1 r1i1p1f1 1 v20190429
standard_name long_name units \
5 air_temperature Near-Surface Air Temperature K
8 cell_area Grid-Cell Area for Atmospheric Grid Variables m2
finalised instance_id
5 True CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.A...
8 True CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.f...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'historical'), ('source_id', 'CanESM5'), ('variable_id', 'tas'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
55 1979-01-16 00:00:00 2014-12-16 00:00:00
58 None None
path activity_id \
55 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
58 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
55 standard 0.0 0.0
58 standard 0.0 267840.0
experiment experiment_id frequency ... \
55 all-forcing simulation of the recent past historical mon ...
58 pre-industrial control piControl fx ...
variable_id variant_label member_id vertical_levels version \
55 tas r1i1p1f3 r1i1p1f3 1 v20190624
58 areacella r1i1p1f1 r1i1p1f1 1 v20190709
standard_name long_name units \
55 air_temperature Near-Surface Air Temperature K
58 cell_area Grid-Cell Area for Atmospheric Grid Variables m2
finalised instance_id
55 True CMIP6.CMIP.MOHC.HadGEM3-GC31-LL.historical.r1i...
58 True CMIP6.CMIP.MOHC.HadGEM3-GC31-LL.piControl.r1i1...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'historical'), ('source_id', 'HadGEM3-GC31-LL'), ('variable_id', 'tas'), ('variant_label', 'r1i1p1f3')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
4 1850-01-16 12:00:00 1869-12-16 12:00:00
4 1890-01-16 12:00:00 1909-12-16 12:00:00
4 1910-01-16 12:00:00 1914-12-16 12:00:00
4 1870-01-16 12:00:00 1889-12-16 12:00:00
60 None None
path activity_id \
4 /home/docs/checkouts/readthedocs.org/user_buil... C4MIP CDRMIP
4 /home/docs/checkouts/readthedocs.org/user_buil... C4MIP CDRMIP
4 /home/docs/checkouts/readthedocs.org/user_buil... C4MIP CDRMIP
4 /home/docs/checkouts/readthedocs.org/user_buil... C4MIP CDRMIP
60 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
4 standard 0.0 0.0
4 standard 0.0 0.0
4 standard 0.0 0.0
4 standard 0.0 0.0
60 standard 0.0 146097.0
experiment experiment_id \
4 emissions driven 1% run esm-1pctCO2
4 emissions driven 1% run esm-1pctCO2
4 emissions driven 1% run esm-1pctCO2
4 emissions driven 1% run esm-1pctCO2
60 pre-industrial control simulation with CO2 con... esm-piControl
frequency ... variable_id variant_label member_id vertical_levels \
4 mon ... tas r1i1p1f1 r1i1p1f1 1
4 mon ... tas r1i1p1f1 r1i1p1f1 1
4 mon ... tas r1i1p1f1 r1i1p1f1 1
4 mon ... tas r1i1p1f1 r1i1p1f1 1
60 fx ... areacella r1i1p1f1 r1i1p1f1 1
version standard_name long_name \
4 v20190815 air_temperature Near-Surface Air Temperature
4 v20190815 air_temperature Near-Surface Air Temperature
4 v20190815 air_temperature Near-Surface Air Temperature
4 v20190815 air_temperature Near-Surface Air Temperature
60 v20190815 cell_area Grid-Cell Area for Atmospheric Grid Variables
units finalised instance_id
4 K True CMIP6.C4MIP CDRMIP.MPI-M.MPI-ESM1-2-LR.esm-1pc...
4 K True CMIP6.C4MIP CDRMIP.MPI-M.MPI-ESM1-2-LR.esm-1pc...
4 K True CMIP6.C4MIP CDRMIP.MPI-M.MPI-ESM1-2-LR.esm-1pc...
4 K True CMIP6.C4MIP CDRMIP.MPI-M.MPI-ESM1-2-LR.esm-1pc...
60 m2 True CMIP6.CMIP.MPI-M.MPI-ESM1-2-LR.esm-piControl.r...
[5 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'esm-1pctCO2'), ('source_id', 'MPI-ESM1-2-LR'), ('variable_id', 'tas'), ('variant_label', 'r1i1p1f1')))})
),
climate_ref.solver.DiagnosticExecution(
provider=DiagnosticProvider(name='Example', version='0.7.0'),
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
59 1870-01-16 12:00:00 1889-12-16 12:00:00
59 1910-01-16 12:00:00 1915-12-16 12:00:00
59 1890-01-16 12:00:00 1909-12-16 12:00:00
59 1850-01-16 12:00:00 1869-12-16 12:00:00
60 None None
path activity_id \
59 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
59 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
59 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
59 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
60 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
59 standard 0.0 146097.0
59 standard 0.0 146097.0
59 standard 0.0 146097.0
59 standard 0.0 146097.0
60 standard 0.0 146097.0
experiment experiment_id \
59 pre-industrial control simulation with CO2 con... esm-piControl
59 pre-industrial control simulation with CO2 con... esm-piControl
59 pre-industrial control simulation with CO2 con... esm-piControl
59 pre-industrial control simulation with CO2 con... esm-piControl
60 pre-industrial control simulation with CO2 con... esm-piControl
frequency ... variable_id variant_label member_id vertical_levels \
59 mon ... tas r1i1p1f1 r1i1p1f1 1
59 mon ... tas r1i1p1f1 r1i1p1f1 1
59 mon ... tas r1i1p1f1 r1i1p1f1 1
59 mon ... tas r1i1p1f1 r1i1p1f1 1
60 fx ... areacella r1i1p1f1 r1i1p1f1 1
version standard_name long_name \
59 v20190815 air_temperature Near-Surface Air Temperature
59 v20190815 air_temperature Near-Surface Air Temperature
59 v20190815 air_temperature Near-Surface Air Temperature
59 v20190815 air_temperature Near-Surface Air Temperature
60 v20190815 cell_area Grid-Cell Area for Atmospheric Grid Variables
units finalised instance_id
59 K True CMIP6.CMIP.MPI-M.MPI-ESM1-2-LR.esm-piControl.r...
59 K True CMIP6.CMIP.MPI-M.MPI-ESM1-2-LR.esm-piControl.r...
59 K True CMIP6.CMIP.MPI-M.MPI-ESM1-2-LR.esm-piControl.r...
59 K True CMIP6.CMIP.MPI-M.MPI-ESM1-2-LR.esm-piControl.r...
60 m2 True CMIP6.CMIP.MPI-M.MPI-ESM1-2-LR.esm-piControl.r...
[5 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'esm-piControl'), ('source_id', 'MPI-ESM1-2-LR'), ('variable_id', 'tas'), ('variant_label', 'r1i1p1f1')))})
)
]
We get multiple proposed executions.
In [9]:
Copied!
pd.concat(execution.datasets["cmip6"] for execution in executions)[
["experiment_id", "variable_id"]
].drop_duplicates()
pd.concat(execution.datasets["cmip6"] for execution in executions)[
["experiment_id", "variable_id"]
].drop_duplicates()
Out[9]:
| experiment_id | variable_id | |
|---|---|---|
| 13 | abrupt-4xCO2 | rsut |
| 15 | abrupt-4xCO2 | areacella |
| 23 | historical | rsut |
| 45 | historical | areacella |
| 52 | piControl | rsut |
| 54 | piControl | areacella |
| 84 | ssp126 | rsut |
| 87 | ssp126 | areacella |
| 9 | 1pctCO2 | tas |
| 10 | 1pctCO2 | areacella |
| 14 | abrupt-4xCO2 | tas |
| 1 | esm-1pct-brch-1000PgC | tas |
| 2 | esm-1pct-brch-1000PgC | areacella |
| 25 | historical | tas |
| 53 | piControl | tas |
| 85 | ssp126 | tas |
| 4 | esm-1pctCO2 | tas |
| 60 | esm-piControl | areacella |
| 59 | esm-piControl | tas |
Each execution contains a single unique dataset because of the groupby definition. The data catalog for the diagnostic execution may contain more than one row as a dataset may contain multiple files.
In [10]:
Copied!
executions[0].datasets["cmip6"].instance_id.unique().tolist()
executions[0].datasets["cmip6"].instance_id.unique().tolist()
Out[10]:
['CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1i1p1f1.Amon.rsut.gn.v20191115', 'CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1i1p1f1.fx.areacella.gn.v20191115']
In [11]:
Copied!
executions[0].datasets["cmip6"]
executions[0].datasets["cmip6"]
Out[11]:
| start_time | end_time | path | activity_id | branch_method | branch_time_in_child | branch_time_in_parent | experiment | experiment_id | frequency | ... | variable_id | variant_label | member_id | vertical_levels | version | standard_name | long_name | units | finalised | instance_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 13 | 0101-01-16 12:00:00 | 0125-12-16 12:00:00 | /home/docs/checkouts/readthedocs.org/user_buil... | CMIP | standard | 0.0 | 0.0 | abrupt quadrupling of CO2 | abrupt-4xCO2 | mon | ... | rsut | r1i1p1f1 | r1i1p1f1 | 1 | v20191115 | toa_outgoing_shortwave_flux | TOA Outgoing Shortwave Radiation | W m-2 | True | CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1... |
| 15 | None | None | /home/docs/checkouts/readthedocs.org/user_buil... | CMIP | standard | 0.0 | 0.0 | abrupt quadrupling of CO2 | abrupt-4xCO2 | fx | ... | areacella | r1i1p1f1 | r1i1p1f1 | 1 | v20191115 | cell_area | Grid-Cell Area for Atmospheric Grid Variables | m2 | True | CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1... |
2 rows × 36 columns
Diagnostic Definitions¶
Each diagnostic execution requires a ExecutionDefinition object.
This object contains the information about where data should be stored
and which datasets should be used for the diagnostic calculation.
In [12]:
Copied!
output_directory = Path("../out")
definition = executions[0].build_execution_definition(output_directory)
prettyprinter.pprint(definition)
output_directory = Path("../out")
definition = executions[0].build_execution_definition(output_directory)
prettyprinter.pprint(definition)
climate_ref_core.diagnostics.ExecutionDefinition(
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
key='cmip6_abrupt-4xCO2_ACCESS-ESM1-5_rsut_r1i1p1f1',
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
13 0101-01-16 12:00:00 0125-12-16 12:00:00
15 None None
path activity_id \
13 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
15 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
13 standard 0.0 0.0
15 standard 0.0 0.0
experiment experiment_id frequency ... variable_id \
13 abrupt quadrupling of CO2 abrupt-4xCO2 mon ... rsut
15 abrupt quadrupling of CO2 abrupt-4xCO2 fx ... areacella
variant_label member_id vertical_levels version \
13 r1i1p1f1 r1i1p1f1 1 v20191115
15 r1i1p1f1 r1i1p1f1 1 v20191115
standard_name \
13 toa_outgoing_shortwave_flux
15 cell_area
long_name units finalised \
13 TOA Outgoing Shortwave Radiation W m-2 True
15 Grid-Cell Area for Atmospheric Grid Variables m2 True
instance_id
13 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1...
15 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'abrupt-4xCO2'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'rsut'), ('variant_label', 'r1i1p1f1')))}),
output_directory=pathlib.PosixPath(
'/home/docs/checkouts/readthedocs.org/user_builds/climate-ref/'
'checkouts/out/example/global-mean-timeseries/'
'1b9348361d7eb1f5f676e5badf09227174a72ae4'
),
_root_directory=pathlib.PosixPath(
'/home/docs/checkouts/readthedocs.org/user_builds/climate-ref/'
'checkouts/out'
)
)
Running directly locally¶
A diagnostic can be run directly if you want to run a calculation synchronously without any additional infrastructure.
This will not perform and validation/verification of the output executions.
In [13]:
Copied!
definition.output_directory.mkdir(parents=True, exist_ok=True)
direct_result = diagnostic.run(definition=definition)
assert direct_result.successful
prettyprinter.pprint(direct_result)
definition.output_directory.mkdir(parents=True, exist_ok=True)
direct_result = diagnostic.run(definition=definition)
assert direct_result.successful
prettyprinter.pprint(direct_result)
climate_ref_core.diagnostics.ExecutionResult(
definition=climate_ref_core.diagnostics.ExecutionDefinition(
diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'),
key='cmip6_abrupt-4xCO2_ACCESS-ESM1-5_rsut_r1i1p1f1',
datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \
13 0101-01-16 12:00:00 0125-12-16 12:00:00
15 None None
path activity_id \
13 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
15 /home/docs/checkouts/readthedocs.org/user_buil... CMIP
branch_method branch_time_in_child branch_time_in_parent \
13 standard 0.0 0.0
15 standard 0.0 0.0
experiment experiment_id frequency ... variable_id \
13 abrupt quadrupling of CO2 abrupt-4xCO2 mon ... rsut
15 abrupt quadrupling of CO2 abrupt-4xCO2 fx ... areacella
variant_label member_id vertical_levels version \
13 r1i1p1f1 r1i1p1f1 1 v20191115
15 r1i1p1f1 r1i1p1f1 1 v20191115
standard_name \
13 toa_outgoing_shortwave_flux
15 cell_area
long_name units finalised \
13 TOA Outgoing Shortwave Radiation W m-2 True
15 Grid-Cell Area for Atmospheric Grid Variables m2 True
instance_id
13 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1...
15 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1...
[2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'abrupt-4xCO2'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'rsut'), ('variant_label', 'r1i1p1f1')))}),
output_directory=pathlib.PosixPath(
'/home/docs/checkouts/readthedocs.org/user_builds/climate-ref/'
'checkouts/out/example/global-mean-timeseries/'
'1b9348361d7eb1f5f676e5badf09227174a72ae4'
),
_root_directory=pathlib.PosixPath(
'/home/docs/checkouts/readthedocs.org/user_builds/climate-ref/'
'checkouts/out'
)
),
output_bundle_filename=pathlib.PosixPath('output.json'),
metric_bundle_filename=pathlib.PosixPath('diagnostic.json'),
successful=True,
series_filename=pathlib.PosixPath('series.json')
)
Diagnostic calculations¶
Diagnostic calculations are typically run using an Executor which provides an abstraction to enable diagnostics to be run in multiple different ways. These executors can run diagnostics locally, on a cluster, or on a remote service
The default executor is the [LocalExecutor][climate_ref.executor.LocalExecutor] This executor runs executions in parallel using a process pool. Another option is the [SynchronousExecutor][climate_ref.executor.SynchronousExecutor]. This executor runs the execution in the main process which can be useful for debugging purposes.
The executor can be specified in the configuration, or
using the REF_EXECUTOR environment variable which takes precedence.
The [LocalExecutor][climate_ref.executor.LocalExecutor] is the default executor,
if no other configuration is provided.
In [14]:
Copied!
# In this example a [SynchronousExecutor][climate_ref.executor.SynchronousExecutor] is used to run
# the diagnostic.
# This executor runs the execution in the main process which can be useful for debugging purposes.
# In this example a [SynchronousExecutor][climate_ref.executor.SynchronousExecutor] is used to run
# the diagnostic.
# This executor runs the execution in the main process which can be useful for debugging purposes.
In [15]:
Copied!
executor = SynchronousExecutor(config=config, database=db)
diagnostic = provider.get("global-mean-timeseries")
executor.run(definition)
executor = SynchronousExecutor(config=config, database=db)
diagnostic = provider.get("global-mean-timeseries")
executor.run(definition)
2025-11-06 11:44:49.684 | INFO | climate_ref_core.executor:execute_locally:37 - Executing 'example/global-mean-timeseries/cmip6_abrupt-4xCO2_ACCESS-ESM1-5_rsut_r1i1p1f1'
2025-11-06 11:44:49.685 | WARNING | climate_ref_core.executor:execute_locally:41 - Output directory /home/docs/checkouts/readthedocs.org/user_builds/climate-ref/checkouts/out/example/global-mean-timeseries/1b9348361d7eb1f5f676e5badf09227174a72ae4 already exists. Removing the existing directory.
2025-11-06 11:44:49.691 | INFO | climate_ref_core.logging:redirect_logs:199 - Running definition ExecutionDefinition( diagnostic=GlobalMeanTimeseries(name='Global Mean Timeseries'), key='cmip6_abrupt-4xCO2_ACCESS-ESM1-5_rsut_r1i1p1f1', datasets=ExecutionDatasetCollection({<SourceDatasetType.CMIP6: 'cmip6'>: DatasetCollection(datasets= start_time end_time \ 13 0101-01-16 12:00:00 0125-12-16 12:00:00 15 None None path activity_id \ 13 /home/docs/checkouts/readthedocs.org/user_buil... CMIP 15 /home/docs/checkouts/readthedocs.org/user_buil... CMIP branch_method branch_time_in_child branch_time_in_parent \ 13 standard 0.0 0.0 15 standard 0.0 0.0 experiment experiment_id frequency ... variable_id \ 13 abrupt quadrupling of CO2 abrupt-4xCO2 mon ... rsut 15 abrupt quadrupling of CO2 abrupt-4xCO2 fx ... areacella variant_label member_id vertical_levels version \ 13 r1i1p1f1 r1i1p1f1 1 v20191115 15 r1i1p1f1 r1i1p1f1 1 v20191115 standard_name \ 13 toa_outgoing_shortwave_flux 15 cell_area long_name units finalised \ 13 TOA Outgoing Shortwave Radiation W m-2 True 15 Grid-Cell Area for Atmospheric Grid Variables m2 True instance_id 13 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1... 15 CMIP6.CMIP.CSIRO.ACCESS-ESM1-5.abrupt-4xCO2.r1... [2 rows x 36 columns], slug_column='instance_id', selector=(('experiment_id', 'abrupt-4xCO2'), ('source_id', 'ACCESS-ESM1-5'), ('variable_id', 'rsut'), ('variant_label', 'r1i1p1f1')))}), output_directory=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/climate-ref/checkouts/out/example/global-mean-timeseries/1b9348361d7eb1f5f676e5badf09227174a72ae4'), _root_directory=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/climate-ref/checkouts/out') )
2025-11-06 11:44:49.970 | INFO | climate_ref_core.logging:redirect_logs:206 - Diagnostic execution complete. Results available in example/global-mean-timeseries/1b9348361d7eb1f5f676e5badf09227174a72ae4
In [16]:
Copied!
output_file = definition.to_output_path("output.json")
with open(output_file) as fh:
# Load the output bundle and pretty print
loaded_result = json.loads(fh.read())
print(json.dumps(loaded_result, indent=2))
output_file = definition.to_output_path("output.json")
with open(output_file) as fh:
# Load the output bundle and pretty print
loaded_result = json.loads(fh.read())
print(json.dumps(loaded_result, indent=2))
{
"index": "index.html",
"provenance": {
"environment": {},
"modeldata": [],
"obsdata": {},
"log": "cmec_output.log"
},
"data": {},
"plots": {},
"html": {},
"metrics": null,
"diagnostics": {}
}
In [ ]:
Copied!