From 0e867e401609631cc9648a65d6459be324377b68 Mon Sep 17 00:00:00 2001 From: Jared Lewis Date: Thu, 13 Aug 2026 00:34:41 +0000 Subject: [PATCH 1/4] docs: document the obs4MIPs reference data that must come from ESGF The download tutorial covered the obs4REF registry and the CMIP6 input data, but never said which reference datasets neither of those provides. Five obs4MIPs datasets are only available from ESGF: 20CR-V2 psl, C3S-GTO-ECV-9-0 toz, CERES-EBAF-4-2-1 rlut/rlutcs/rsut/rsutcs, ERA-5 psl/ta/tas/ua, and NOAA-NCEI-LAI-AVHRR-5-0 lai. Missing any of them costs you the diagnostics that need it, and because an unmet reference requirement plans no executions rather than failing, the loss is silent. Document them with the diagnostics that require each, and give the `--kind obs4mips` snippet that fetches just this data. scripts/fetch-esgf.py maintains its request list by hand, and its TODO to derive the list from the data requirements is still open, so add tests that check the two against each other: every required source_id is either in the obs4REF registry or requested here, every dataset a requirement can select from the committed ESGF catalog snapshot is fetched, and no request names a source_id nothing requires. Each was confirmed to fail when the corresponding drift is introduced. The requirement side cannot be checked by taking the cross product of a requirement's facets: ESGF intersects facets, so the PMP ENSO requirement naming four sources and eight variables would appear to need 32 combinations when only 12 exist. The tests resolve against the catalog instead, which is also why the snapshot has to be filtered to its ESGF rows -- it is built from an archive that holds the obs4REF collection too. Constraint: The committed catalog snapshot only covers datasets someone has already downloaded, so the pair-level test cannot see a brand new source; the source-level test covers that case Rejected: Generate the request list from the data requirements now | the facet intersection means a generated list needs per-source grouping logic, which is the open TODO, not a docs change Confidence: high Scope-risk: narrow Directive: When adding an obs4MIPs data requirement, add or extend an Obs4MIPsRequest grouped by source_id -- one request naming several sources and several variables returns nothing --- changelog/864.docs.md | 5 + docs/getting-started/02-download-datasets.md | 51 +++++ scripts/fetch-esgf.py | 8 +- tests/scripts/test_fetch_esgf_coverage.py | 195 +++++++++++++++++++ 4 files changed, 257 insertions(+), 2 deletions(-) create mode 100644 changelog/864.docs.md create mode 100644 tests/scripts/test_fetch_esgf_coverage.py diff --git a/changelog/864.docs.md b/changelog/864.docs.md new file mode 100644 index 000000000..c237e3173 --- /dev/null +++ b/changelog/864.docs.md @@ -0,0 +1,5 @@ +Documented the obs4MIPs reference datasets that must be fetched from ESGF because the obs4REF registry does not carry them +(`20CR-V2`, `C3S-GTO-ECV-9-0`, `CERES-EBAF-4-2-1`, `ERA-5` and `NOAA-NCEI-LAI-AVHRR-5-0`), +including which diagnostics require each one and a `scripts/fetch-esgf.py --kind obs4mips` snippet for fetching them. +`scripts/fetch-esgf.py` maintains its request list by hand, so it is now covered by tests +that check it against the providers' data requirements. diff --git a/docs/getting-started/02-download-datasets.md b/docs/getting-started/02-download-datasets.md index 531ab6144..e0999ca0c 100644 --- a/docs/getting-started/02-download-datasets.md +++ b/docs/getting-started/02-download-datasets.md @@ -54,6 +54,57 @@ In the future, these datasets will be available on ESGF and can be fetched direc ref datasets fetch-data --registry obs4ref --output-directory $REF_CONFIGURATION/datasets/obs4ref ``` +[](){#fetch-obs4mips-datasets} + +## 2. Fetching obs4MIPs datasets from ESGF + +The obs4REF registry does not cover every reference dataset the diagnostics need. +The following are already published to obs4MIPs on ESGF and must be fetched from there: + +| `source_id` | Variables | Required by | +| --- | --- | --- | +| `20CR-V2` | `psl` | `pmp/extratropical-modes-of-variability-{nam,nao,npo,pna,sam}` | +| `C3S-GTO-ECV-9-0` | `toz` | `esmvaltool/ozone-{annual-cycle,lat-time,nh-mar,sh-oct}` | +| `CERES-EBAF-4-2-1` | `rlut`, `rlutcs`, `rsut`, `rsutcs` | `esmvaltool/cloud-radiative-effects` | +| `ERA-5` | `psl`, `ta`, `tas`, `ua` | `esmvaltool/cloud-scatterplots-reference`, `esmvaltool/regional-historical-{annual-cycle,timeseries,trend}` | +| `NOAA-NCEI-LAI-AVHRR-5-0` | `lai` | `ilamb/lai-avh15c1` | + +A diagnostic whose reference data is missing simply plans no executions, +so an incomplete fetch shows up as a diagnostic that never runs rather than as an error. + +The same [./scripts/fetch-esgf.py](https://github.com/Climate-REF/climate-ref/blob/main/scripts/fetch-esgf.py) +script used for the CMIP6 input data can fetch these, +and `--kind obs4mips` restricts it to the reference data: + +```bash +python scripts/fetch-esgf.py --kind obs4mips +``` + +This is a much smaller download than the CMIP6 input data (a few GB). +Files land in the [intake-esgf `local_cache`](https://intake-esgf.readthedocs.io/en/latest/configure.html), +and are ingested with the `obs4mips` source type, the same as the obs4REF collection. + +/// admonition | Note + +The script also fetches `CERES-EBAF-4-2`, `GPCP-Monthly-3-2`, `HadISST-1-1` and `TropFlux-1-0`, +which the obs4REF registry ships as well. +These are the ESGF-published copies of datasets that were curated for the REF before publication, +so if you have already fetched the obs4REF registry you do not need them. + +Fetching both is safe. Where the two copies carry the same version they share an `instance_id` +and ingest as a single dataset. Where the published copy is newer +(for example `CERES-EBAF-4-2` `v20240513` on ESGF against `v20230209` in obs4REF) +both are ingested and the catalog uses the later version. +/// + +/// admonition | Why one request per `source_id`? + +An ESGF search intersects its facets, +so a single request naming every `source_id` and every `variable_id` would ask each source +for variables it does not have and return nothing. +The requests in `scripts/fetch-esgf.py` are therefore grouped by `source_id`. +/// + ### Future work The Climate-REF team is working on providing a more integrated way to fetch and manage these datasets from the Next Generation ESGF infrastructure that in the process of being deployed. diff --git a/scripts/fetch-esgf.py b/scripts/fetch-esgf.py index c665ec6a9..fb7d7dc56 100644 --- a/scripts/fetch-esgf.py +++ b/scripts/fetch-esgf.py @@ -129,7 +129,10 @@ def fetch(self, remove_ensembles: bool = True) -> PathDict: Request = CMIP6Request | Obs4MIPsRequest -# TODO use the data requirements from the diagnostics directly +# TODO use the data requirements from the diagnostics directly. +# Until then, `tests/scripts/test_fetch_esgf_coverage.py` checks the obs4MIPs requests below +# against those requirements, so a diagnostic cannot start asking for reference data that +# nothing here fetches. requests: list[Request] = [ CMIP6Request( id="esmvaltool-climate-at-global-warmings-levels", @@ -409,7 +412,8 @@ def fetch(self, remove_ensembles: bool = True) -> PathDict: variable_id="ts", ), ), - # The PMP psl modes of variability request source_id "20CR", which is the obs4REF spelling. + # The PMP psl modes of variability diagnostics require source_id "20CR-V2". + # The obs4REF registry carries an older "20CR", which does not satisfy them. Obs4MIPsRequest( id="pmp-modes-20cr-obs4mips", facets=dict( diff --git a/tests/scripts/test_fetch_esgf_coverage.py b/tests/scripts/test_fetch_esgf_coverage.py new file mode 100644 index 000000000..e4f106c7b --- /dev/null +++ b/tests/scripts/test_fetch_esgf_coverage.py @@ -0,0 +1,195 @@ +""" +Tests that `scripts/fetch-esgf.py` fetches the obs4MIPs reference data the diagnostics ask for. + +The requests in that script are maintained by hand, so they can drift from the providers' +data requirements without anything failing until a solve quietly plans no executions for a +diagnostic. These tests close that gap offline: they read the requirements from the installed +providers and check them against the request list and the committed ESGF catalog snapshot. + +No network access is performed; nothing is fetched. +""" + +import importlib.util +from pathlib import Path + +import pandas as pd +import pytest + +from climate_ref_core.diagnostics import DataRequirement +from climate_ref_core.source_types import SourceDatasetType + +REPO_ROOT = Path(__file__).parents[2] +SCRIPT = REPO_ROOT / "scripts" / "fetch-esgf.py" +OBS4MIPS_CATALOG = REPO_ROOT / "tests" / "test-data" / "esgf-catalog" / "obs4mips_catalog.parquet" +OBS4REF_REGISTRY = ( + REPO_ROOT + / "packages" + / "climate-ref" + / "src" + / "climate_ref" + / "dataset_registry" + / "obs4ref_reference.txt" +) + +# Number of path parts in an obs4REF registry key: +# obs4REF/{institution_id}/{source_id}/{frequency}/{variable_id}/{grid_label}/{version}/{filename} +_OBS4REF_KEY_PARTS = 8 +_KEY_SOURCE_ID = 2 +_KEY_VARIABLE_ID = 4 + + +@pytest.fixture(scope="module") +def script(): + spec = importlib.util.spec_from_file_location("fetch_esgf", SCRIPT) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@pytest.fixture(scope="module") +def obs4mips_requirements() -> list[tuple[str, DataRequirement]]: + """Every obs4MIPs-typed data requirement across the default providers, tagged with its slug.""" + climate_ref_esmvaltool = pytest.importorskip("climate_ref_esmvaltool") + climate_ref_ilamb = pytest.importorskip("climate_ref_ilamb") + climate_ref_pmp = pytest.importorskip("climate_ref_pmp") + + requirements = [] + for provider in ( + climate_ref_esmvaltool.provider, + climate_ref_pmp.provider, + climate_ref_ilamb.provider, + ): + for diagnostic in provider.diagnostics(): + # A diagnostic declares either a flat sequence of requirements or a sequence of + # alternative branches. Every branch is a way the diagnostic can run, so the data + # for all of them has to be fetchable. + for item in diagnostic.data_requirements: + branch = item if isinstance(item, (list, tuple)) else [item] + for requirement in branch: + if requirement.source_type is SourceDatasetType.obs4MIPs: + requirements.append((f"{provider.slug}/{diagnostic.slug}", requirement)) + assert requirements, "no obs4MIPs requirements found; the providers failed to load" + return requirements + + +@pytest.fixture(scope="module") +def obs4ref_source_ids() -> set[str]: + """The `source_id`s the obs4REF registry provides, which need no ESGF request.""" + source_ids = set() + for raw_line in OBS4REF_REGISTRY.read_text().splitlines(): + line = raw_line.strip() + if not line or line.startswith("#"): + continue + parts = line.split()[0].split("/") + if len(parts) == _OBS4REF_KEY_PARTS: + source_ids.add(parts[_KEY_SOURCE_ID]) + return source_ids + + +@pytest.fixture(scope="module") +def esgf_catalog() -> pd.DataFrame: + """ + The ESGF-published part of the committed obs4MIPs catalog snapshot. + + The snapshot is built from a local archive that also holds the obs4REF collection, + so rows whose path sits under `obs4REF/` are excluded: they are provided by the registry + rather than by a fetch from ESGF. + """ + catalog = pd.read_parquet(OBS4MIPS_CATALOG) + return catalog[~catalog["path"].str.contains("/obs4REF/")] + + +def _requested_pairs(script) -> dict[tuple[str, str], str]: + """Map every (source_id, variable_id) the script requests to the id of the request.""" + pairs = {} + for request in script.requests: + if not isinstance(request, script.Obs4MIPsRequest): + continue + source_ids = request.facets["source_id"] + variable_ids = request.facets["variable_id"] + source_ids = (source_ids,) if isinstance(source_ids, str) else source_ids + variable_ids = (variable_ids,) if isinstance(variable_ids, str) else variable_ids + for source_id in source_ids: + for variable_id in variable_ids: + pairs[(source_id, variable_id)] = request.id + return pairs + + +def _requirement_source_ids(requirement: DataRequirement) -> set[str]: + source_ids: set[str] = set() + for facet_filter in requirement.filters: + value = facet_filter.facets.get("source_id") + if value is None: + continue + source_ids.update((value,) if isinstance(value, str) else value) + return source_ids + + +def test_every_required_source_is_obtainable(script, obs4mips_requirements, obs4ref_source_ids): + """ + Every `source_id` a diagnostic asks for is either in the obs4REF registry or fetched from ESGF. + + This is the check that catches a new diagnostic naming a reference dataset that no one + can obtain, which would otherwise surface only as a diagnostic that never runs. + """ + requested = {source_id for source_id, _ in _requested_pairs(script)} + + unobtainable = { + source_id: slug + for slug, requirement in obs4mips_requirements + for source_id in _requirement_source_ids(requirement) + if source_id not in obs4ref_source_ids and source_id not in requested + } + + assert not unobtainable, ( + "These source_ids are required by a diagnostic but are in neither the obs4REF registry " + f"nor an Obs4MIPsRequest in {SCRIPT.name}: {unobtainable}" + ) + + +def test_every_selectable_dataset_is_fetched(script, obs4mips_requirements, esgf_catalog): + """ + Every ESGF dataset a requirement can actually select is covered by a request. + + Resolving against the catalog rather than taking the cross product of the requirement's + facets matters: ESGF intersects its facets, so a requirement naming four sources and eight + variables does not need all thirty-two combinations, only the ones that exist. + """ + requested = _requested_pairs(script) + + missing: dict[tuple[str, str], set[str]] = {} + for slug, requirement in obs4mips_requirements: + selected = requirement.apply_filters(esgf_catalog) + for source_id, variable_id in zip(selected["source_id"], selected["variable_id"]): + if (source_id, variable_id) not in requested: + missing.setdefault((source_id, variable_id), set()).add(slug) + + assert not missing, ( + f"These obs4MIPs datasets are selected by a diagnostic but no request in {SCRIPT.name} " + f"fetches them: { {pair: sorted(slugs) for pair, slugs in missing.items()} }" + ) + + +def test_no_request_is_stale(script, obs4mips_requirements, obs4ref_source_ids): + """ + Every requested `source_id` is still named by a requirement. + + A request left behind after a diagnostic stops using a dataset silently downloads data + that nothing will read. + """ + required = { + source_id + for _, requirement in obs4mips_requirements + for source_id in _requirement_source_ids(requirement) + } + + stale = { + source_id: request_id + for (source_id, _), request_id in _requested_pairs(script).items() + if source_id not in required + } + + assert not stale, ( + f"These requests in {SCRIPT.name} fetch a source_id that no diagnostic requires: {stale}" + ) From 041a2c53f301faa27dbde5957e3097535c642172 Mon Sep 17 00:00:00 2001 From: Jared Lewis Date: Thu, 13 Aug 2026 00:55:54 +0000 Subject: [PATCH 2/4] docs: warn that fetching both copies of a reference dataset double-covers it The note added in the previous commit said fetching both the obs4REF and the ESGF copy of a dataset was safe because same-version copies share an instance_id and ingest as one dataset. They do ingest as one dataset, but that dataset then holds both sets of files. GPCP-Monthly-3-2 pr v20231205 ends up with 42: obs4REF's single 1983-2023 file plus ESGF's 41 yearly files covering the same period. A diagnostic reading it sees every time step twice, and both pmp/enso_tel and pmp/enso_proc read it. Recommend the five ESGF-only datasets by request id instead of --kind obs4mips, which also pulls the four duplicated ones, and keep --kind obs4mips for deployments that do not use the obs4REF registry. This is guidance for the current behaviour, not a fix. The fix is for the catalog to prefer the published obs4MIPs copy over the obs4REF one at the same version, which needs the obs4REF datasets to be ingested as their own source type first. Constraint: obs4REF and ESGF copies at the same version are indistinguishable once ingested, because both carry activity_id obs4MIPs and merge on instance_id before anything can choose between them Confidence: high Scope-risk: narrow Not-tested: The downstream effect on a diagnostic execution is inferred from the duplicated file list, not observed in a completed run --- changelog/864.docs.md | 5 ++- docs/getting-started/02-download-datasets.md | 45 ++++++++++++++------ 2 files changed, 37 insertions(+), 13 deletions(-) diff --git a/changelog/864.docs.md b/changelog/864.docs.md index c237e3173..8e209f942 100644 --- a/changelog/864.docs.md +++ b/changelog/864.docs.md @@ -1,5 +1,8 @@ Documented the obs4MIPs reference datasets that must be fetched from ESGF because the obs4REF registry does not carry them (`20CR-V2`, `C3S-GTO-ECV-9-0`, `CERES-EBAF-4-2-1`, `ERA-5` and `NOAA-NCEI-LAI-AVHRR-5-0`), -including which diagnostics require each one and a `scripts/fetch-esgf.py --kind obs4mips` snippet for fetching them. +including which diagnostics require each one and the `scripts/fetch-esgf.py` commands that fetch them. +Warned against fetching the four datasets that both obs4REF and ESGF provide: +when the two copies share a version they ingest as a single dataset holding both sets of files, +which covers the record twice. `scripts/fetch-esgf.py` maintains its request list by hand, so it is now covered by tests that check it against the providers' data requirements. diff --git a/docs/getting-started/02-download-datasets.md b/docs/getting-started/02-download-datasets.md index e0999ca0c..06f16213d 100644 --- a/docs/getting-started/02-download-datasets.md +++ b/docs/getting-started/02-download-datasets.md @@ -73,28 +73,49 @@ A diagnostic whose reference data is missing simply plans no executions, so an incomplete fetch shows up as a diagnostic that never runs rather than as an error. The same [./scripts/fetch-esgf.py](https://github.com/Climate-REF/climate-ref/blob/main/scripts/fetch-esgf.py) -script used for the CMIP6 input data can fetch these, -and `--kind obs4mips` restricts it to the reference data: +script used for the CMIP6 input data fetches these. +If you fetched the obs4REF registry in the previous step, ask for exactly these five: ```bash -python scripts/fetch-esgf.py --kind obs4mips +python scripts/fetch-esgf.py --request-id pmp-modes-20cr-obs4mips +python scripts/fetch-esgf.py --request-id esmvaltool-ozone-obs4mips +python scripts/fetch-esgf.py --request-id esmvaltool-cloud-radiative-effects-obs4mips +python scripts/fetch-esgf.py --request-id esmvaltool-cloud-scatterplots-obs4mips +python scripts/fetch-esgf.py --request-id esmvaltool-historical-obs4mips +python scripts/fetch-esgf.py --request-id ilamb-lai-obs4mips ``` +(`ERA-5` is split across two requests, hence six commands for five datasets.) + +`--kind obs4mips` fetches all of the reference data in one go, +but it also re-fetches four datasets the obs4REF registry already provides, +so only use it if you are **not** using that registry — see the warning below. + This is a much smaller download than the CMIP6 input data (a few GB). Files land in the [intake-esgf `local_cache`](https://intake-esgf.readthedocs.io/en/latest/configure.html), and are ingested with the `obs4mips` source type, the same as the obs4REF collection. -/// admonition | Note +/// admonition | Do not fetch these twice + type: warning -The script also fetches `CERES-EBAF-4-2`, `GPCP-Monthly-3-2`, `HadISST-1-1` and `TropFlux-1-0`, -which the obs4REF registry ships as well. +The script also fetches `CERES-EBAF-4-2`, `GPCP-Monthly-3-2`, `HadISST-1-1` and `TropFlux-1-0`. These are the ESGF-published copies of datasets that were curated for the REF before publication, -so if you have already fetched the obs4REF registry you do not need them. - -Fetching both is safe. Where the two copies carry the same version they share an `instance_id` -and ingest as a single dataset. Where the published copy is newer -(for example `CERES-EBAF-4-2` `v20240513` on ESGF against `v20230209` in obs4REF) -both are ingested and the catalog uses the later version. +so the obs4REF registry ships them as well. + +**If you have already fetched the obs4REF registry, do not fetch these from ESGF as well.** +Where the two copies carry the same version they share an `instance_id` and ingest as a single +dataset holding *both* sets of files, which covers the record twice. `GPCP-Monthly-3-2` `pr` +`v20231205` is the clearest case: obs4REF ships one file spanning 1983-2023 and ESGF ships 41 +yearly files spanning the same period, giving one dataset of 42 files. A diagnostic reading it +sees every time step twice. + +The four requests that fetch them (`pmp-enso-ceres-obs4mips`, `pmp-enso-gpcp-obs4mips`, +`pmp-enso-tropflux-obs4mips` and `pmp-hadisst-obs4mips`) are for deployments that do not use +the obs4REF registry at all. + +Where the published copy carries a *newer* version there is no such problem +(for example `CERES-EBAF-4-2` `v20240513` on ESGF against `v20230209` in obs4REF): +the two ingest as separate datasets and the catalog uses the later version. /// /// admonition | Why one request per `source_id`? From 613dd9dd3ab06619a3d49ef8242efba60846eb05 Mon Sep 17 00:00:00 2001 From: Jared Lewis Date: Thu, 13 Aug 2026 15:18:33 +1000 Subject: [PATCH 3/4] chore: clean up docs --- changelog/864.docs.md | 5 --- docs/getting-started/02-download-datasets.md | 38 +++++++++----------- scripts/fetch-esgf.py | 8 ++--- tests/scripts/test_fetch_esgf_coverage.py | 27 ++++++-------- 4 files changed, 30 insertions(+), 48 deletions(-) diff --git a/changelog/864.docs.md b/changelog/864.docs.md index 8e209f942..7988508e6 100644 --- a/changelog/864.docs.md +++ b/changelog/864.docs.md @@ -1,8 +1,3 @@ Documented the obs4MIPs reference datasets that must be fetched from ESGF because the obs4REF registry does not carry them (`20CR-V2`, `C3S-GTO-ECV-9-0`, `CERES-EBAF-4-2-1`, `ERA-5` and `NOAA-NCEI-LAI-AVHRR-5-0`), including which diagnostics require each one and the `scripts/fetch-esgf.py` commands that fetch them. -Warned against fetching the four datasets that both obs4REF and ESGF provide: -when the two copies share a version they ingest as a single dataset holding both sets of files, -which covers the record twice. -`scripts/fetch-esgf.py` maintains its request list by hand, so it is now covered by tests -that check it against the providers' data requirements. diff --git a/docs/getting-started/02-download-datasets.md b/docs/getting-started/02-download-datasets.md index 06f16213d..0b186c1ba 100644 --- a/docs/getting-started/02-download-datasets.md +++ b/docs/getting-started/02-download-datasets.md @@ -1,16 +1,23 @@ # Download Required Datasets -This tutorial covers how to fetch all reference datasets needed to run Climate-REF diagnostics. You may see references to *fetch*, *download*, and *retrieve* all of which refer to the process of copying data from one computer system to another. [Ingesting](../concepts.md) these datasets is covered in the next tutorial. +This tutorial covers how to fetch all reference datasets needed to run Climate-REF diagnostics. +You may see references to *fetch*, *download*, and *retrieve* all of which refer to the process of copying data from one computer system to another. [Ingesting](../concepts.md) these datasets is covered in the next tutorial. These commands should be rerun after new releases of Climate-REF to ensure you have the latest datasets. ## Input datasets -The Climate-REF requires local input datasets from CMIP6/CMIP6plus to evaluate. Depending on where you are running the REF, a local archive of CMIP6 datasets may be available already, if not the target datasets can be fetched from [ESGF](https://esgf-node.ornl.gov/search) directly. We have provided a script in [./scripts/fetch-esgf.py](https://github.com/Climate-REF/climate-ref/blob/main/scripts/fetch-esgf.py) for fetching the datasets that can be evaluated by the REF. This involves a moderate volume of data, requireing more than 4TB of storage when assessing a single ensemble member per model. +The Climate-REF requires local input datasets from CMIP6/CMIP6plus/CMIP7 to evaluate. +Depending on where you are running the REF, a local archive of CMIP6 datasets may be available already, +if not the target datasets can be fetched from [ESGF](https://esgf-node.ornl.gov/search) directly. +We have provided a script in [./scripts/fetch-esgf.py](https://github.com/Climate-REF/climate-ref/blob/main/scripts/fetch-esgf.py) for fetching the datasets that can be evaluated by the REF. +This involves a moderate volume of data, requireing more than 4TB of storage when assessing a single ensemble member per model. -Note that not all of these datasets are required. The Climate-REF will determine which diagnostics can be evaluated according the datasets that are available. +Note that not all of these datasets are required. +The Climate-REF will determine which diagnostics can be evaluated according the datasets that are available. -The data used by the Climate-REF do not necessarily need to have been previously published to ESGF. As long as the datasets match the data requirements of the diagnostics and they conform with the CMIP6 era cmorisation process they can be evaluated via the REF. +The data used by the Climate-REF do not necessarily need to have been previously published to ESGF. +As long as the datasets match the data requirements of the diagnostics and they conform with the CMIP6 era cmorisation process they can be evaluated via the REF. If you are preparing data for a modelling centre, start with the [modelling centre onboarding guide](modelling-centres.md) @@ -91,7 +98,6 @@ python scripts/fetch-esgf.py --request-id ilamb-lai-obs4mips but it also re-fetches four datasets the obs4REF registry already provides, so only use it if you are **not** using that registry — see the warning below. -This is a much smaller download than the CMIP6 input data (a few GB). Files land in the [intake-esgf `local_cache`](https://intake-esgf.readthedocs.io/en/latest/configure.html), and are ingested with the `obs4mips` source type, the same as the obs4REF collection. @@ -104,26 +110,14 @@ so the obs4REF registry ships them as well. **If you have already fetched the obs4REF registry, do not fetch these from ESGF as well.** Where the two copies carry the same version they share an `instance_id` and ingest as a single -dataset holding *both* sets of files, which covers the record twice. `GPCP-Monthly-3-2` `pr` -`v20231205` is the clearest case: obs4REF ships one file spanning 1983-2023 and ESGF ships 41 -yearly files spanning the same period, giving one dataset of 42 files. A diagnostic reading it -sees every time step twice. +dataset holding *both* sets of files, which covers the record twice. +`GPCP-Monthly-3-2` `pr` `v20231205` is the clearest case: obs4REF ships one file spanning 1983-2023 and ESGF ships 41 yearly files spanning the same period, giving one dataset of 42 files. +A diagnostic reading it sees every time step twice. -The four requests that fetch them (`pmp-enso-ceres-obs4mips`, `pmp-enso-gpcp-obs4mips`, -`pmp-enso-tropflux-obs4mips` and `pmp-hadisst-obs4mips`) are for deployments that do not use -the obs4REF registry at all. - -Where the published copy carries a *newer* version there is no such problem -(for example `CERES-EBAF-4-2` `v20240513` on ESGF against `v20230209` in obs4REF): +Where the published copy carries a *newer* version there is no such problem: the two ingest as separate datasets and the catalog uses the later version. -/// - -/// admonition | Why one request per `source_id`? -An ESGF search intersects its facets, -so a single request naming every `source_id` and every `variable_id` would ask each source -for variables it does not have and return nothing. -The requests in `scripts/fetch-esgf.py` are therefore grouped by `source_id`. +This is temporary until we split the obs4REF ingest from the obs4MIPs /// ### Future work diff --git a/scripts/fetch-esgf.py b/scripts/fetch-esgf.py index fb7d7dc56..82bbaa92b 100644 --- a/scripts/fetch-esgf.py +++ b/scripts/fetch-esgf.py @@ -130,9 +130,9 @@ def fetch(self, remove_ensembles: bool = True) -> PathDict: Request = CMIP6Request | Obs4MIPsRequest # TODO use the data requirements from the diagnostics directly. -# Until then, `tests/scripts/test_fetch_esgf_coverage.py` checks the obs4MIPs requests below -# against those requirements, so a diagnostic cannot start asking for reference data that -# nothing here fetches. +# Until then, `tests/scripts/test_fetch_esgf_coverage.py` +# checks the obs4MIPs requests below against those requirements, +# so a diagnostic cannot start asking for reference data that nothing here fetches. requests: list[Request] = [ CMIP6Request( id="esmvaltool-climate-at-global-warmings-levels", @@ -412,8 +412,6 @@ def fetch(self, remove_ensembles: bool = True) -> PathDict: variable_id="ts", ), ), - # The PMP psl modes of variability diagnostics require source_id "20CR-V2". - # The obs4REF registry carries an older "20CR", which does not satisfy them. Obs4MIPsRequest( id="pmp-modes-20cr-obs4mips", facets=dict( diff --git a/tests/scripts/test_fetch_esgf_coverage.py b/tests/scripts/test_fetch_esgf_coverage.py index e4f106c7b..b7baf94d2 100644 --- a/tests/scripts/test_fetch_esgf_coverage.py +++ b/tests/scripts/test_fetch_esgf_coverage.py @@ -1,12 +1,10 @@ """ Tests that `scripts/fetch-esgf.py` fetches the obs4MIPs reference data the diagnostics ask for. -The requests in that script are maintained by hand, so they can drift from the providers' -data requirements without anything failing until a solve quietly plans no executions for a -diagnostic. These tests close that gap offline: they read the requirements from the installed -providers and check them against the request list and the committed ESGF catalog snapshot. +The requests in that script are maintained by hand so they can drift from the providers' data requirements. +These tests check that the obs data requested is covered by a download. -No network access is performed; nothing is fetched. +No network access is performed and nothing is fetched. """ import importlib.util @@ -61,9 +59,6 @@ def obs4mips_requirements() -> list[tuple[str, DataRequirement]]: climate_ref_ilamb.provider, ): for diagnostic in provider.diagnostics(): - # A diagnostic declares either a flat sequence of requirements or a sequence of - # alternative branches. Every branch is a way the diagnostic can run, so the data - # for all of them has to be fetchable. for item in diagnostic.data_requirements: branch = item if isinstance(item, (list, tuple)) else [item] for requirement in branch: @@ -130,8 +125,8 @@ def test_every_required_source_is_obtainable(script, obs4mips_requirements, obs4 """ Every `source_id` a diagnostic asks for is either in the obs4REF registry or fetched from ESGF. - This is the check that catches a new diagnostic naming a reference dataset that no one - can obtain, which would otherwise surface only as a diagnostic that never runs. + This is the check that catches a new diagnostic naming a reference dataset that no one can obtain, + which would otherwise surface only as a diagnostic that never runs. """ requested = {source_id for source_id, _ in _requested_pairs(script)} @@ -152,9 +147,10 @@ def test_every_selectable_dataset_is_fetched(script, obs4mips_requirements, esgf """ Every ESGF dataset a requirement can actually select is covered by a request. - Resolving against the catalog rather than taking the cross product of the requirement's - facets matters: ESGF intersects its facets, so a requirement naming four sources and eight - variables does not need all thirty-two combinations, only the ones that exist. + Resolving against the catalog rather than taking the cross product of the requirement's facets matters. + ESGF intersects its facets, + so a requirement naming four sources and eight variables does not need all thirty-two combinations, + only the ones that exist. """ requested = _requested_pairs(script) @@ -171,12 +167,11 @@ def test_every_selectable_dataset_is_fetched(script, obs4mips_requirements, esgf ) -def test_no_request_is_stale(script, obs4mips_requirements, obs4ref_source_ids): +def test_no_request_is_stale(script, obs4mips_requirements): """ Every requested `source_id` is still named by a requirement. - A request left behind after a diagnostic stops using a dataset silently downloads data - that nothing will read. + Avoid silently downloads data that nothing will read. """ required = { source_id From de8b51ed79a31378cd0f476ce55007950f2217fb Mon Sep 17 00:00:00 2001 From: Jared Lewis Date: Thu, 13 Aug 2026 15:29:27 +1000 Subject: [PATCH 4/4] docs: fix typos and finish the truncated obs4MIPs ingest sentence --- docs/getting-started/02-download-datasets.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/getting-started/02-download-datasets.md b/docs/getting-started/02-download-datasets.md index 964e1c92e..19945ecec 100644 --- a/docs/getting-started/02-download-datasets.md +++ b/docs/getting-started/02-download-datasets.md @@ -11,13 +11,13 @@ The Climate-REF requires local input datasets from CMIP6/CMIP6plus/CMIP7 to eval Depending on where you are running the REF, a local archive of CMIP6 datasets may be available already, if not the target datasets can be fetched from [ESGF](https://esgf-node.ornl.gov/search) directly. We have provided a script in [./scripts/fetch-esgf.py](https://github.com/Climate-REF/climate-ref/blob/main/scripts/fetch-esgf.py) for fetching the datasets that can be evaluated by the REF. -This involves a moderate volume of data, requireing more than 4TB of storage when assessing a single ensemble member per model. +This involves a moderate volume of data, requiring more than 4TB of storage when assessing a single ensemble member per model. Note that not all of these datasets are required. -The Climate-REF will determine which diagnostics can be evaluated according the datasets that are available. +The Climate-REF will determine which diagnostics can be evaluated according to the datasets that are available. The data used by the Climate-REF do not necessarily need to have been previously published to ESGF. -As long as the datasets match the data requirements of the diagnostics and they conform with the CMIP6 era cmorisation process they can be evaluated via the REF. +As long as the datasets match the data requirements of the diagnostics and they conform with the CMIP6 era cmorisation process, they can be evaluated via the REF. If you are preparing data for a modelling centre, start with the [modelling centre onboarding guide](modelling-centres.md) @@ -117,10 +117,10 @@ dataset holding *both* sets of files, which covers the record twice. `GPCP-Monthly-3-2` `pr` `v20231205` is the clearest case: obs4REF ships one file spanning 1983-2023 and ESGF ships 41 yearly files spanning the same period, giving one dataset of 42 files. A diagnostic reading it sees every time step twice. -Where the published copy carries a *newer* version there is no such problem: +Where the published copy carries a *newer* version, there is no such problem: the two ingest as separate datasets and the catalog uses the later version. -This is temporary until we split the obs4REF ingest from the obs4MIPs +This is temporary until we split the obs4REF ingest from the obs4MIPs ingest. /// ### Future work