Document the obs4MIPs reference data that must come from ESGF, and test the fetch script against the requirements

changelog/864.docs.md

@@ -0,0 +1,3 @@
+Documented the obs4MIPs reference datasets that must be fetched from ESGF because the obs4REF registry does not carry them
+(`20CR-V2`, `C3S-GTO-ECV-9-0`, `CERES-EBAF-4-2-1`, `ERA-5` and `NOAA-NCEI-LAI-AVHRR-5-0`),
+including which diagnostics require each one and the `scripts/fetch-esgf.py` commands that fetch them.

docs/getting-started/02-download-datasets.md

@@ -1,16 +1,23 @@
 # Download Required Datasets
 
-This tutorial covers how to fetch all reference datasets needed to run Climate-REF diagnostics. You may see references to *fetch*, *download*, and *retrieve* all of which refer to the process of copying data from one computer system to another. [Ingesting](../concepts.md) these datasets is covered in the next tutorial.
+This tutorial covers how to fetch all reference datasets needed to run Climate-REF diagnostics.
+You may see references to *fetch*, *download*, and *retrieve* all of which refer to the process of copying data from one computer system to another. [Ingesting](../concepts.md) these datasets is covered in the next tutorial.
 
 These commands should be rerun after new releases of Climate-REF to ensure you have the latest datasets.
 
 ## Input datasets
 
-The Climate-REF requires local input datasets from CMIP6/CMIP6plus to evaluate. Depending on where you are running the REF, a local archive of CMIP6 datasets may be available already, if not the target datasets can be fetched from [ESGF](https://esgf-node.ornl.gov/search) directly. We have provided a script in [./scripts/fetch-esgf.py](https://github.com/Climate-REF/climate-ref/blob/main/scripts/fetch-esgf.py) for fetching the datasets that can be evaluated by the REF. This involves a moderate volume of data, requireing more than 4TB of storage when assessing a single ensemble member per model.
+The Climate-REF requires local input datasets from CMIP6/CMIP6plus/CMIP7 to evaluate.
+Depending on where you are running the REF, a local archive of CMIP6 datasets may be available already,
+if not the target datasets can be fetched from [ESGF](https://esgf-node.ornl.gov/search) directly.
+We have provided a script in [./scripts/fetch-esgf.py](https://github.com/Climate-REF/climate-ref/blob/main/scripts/fetch-esgf.py) for fetching the datasets that can be evaluated by the REF.
+This involves a moderate volume of data, requiring more than 4TB of storage when assessing a single ensemble member per model.
 
-Note that not all of these datasets are required. The Climate-REF will determine which diagnostics can be evaluated according the datasets that are available.
+Note that not all of these datasets are required.
+The Climate-REF will determine which diagnostics can be evaluated according to the datasets that are available.
 
-The data used by the Climate-REF do not necessarily need to have been previously published to ESGF. As long as the datasets match the data requirements of the diagnostics and they conform with the CMIP6 era cmorisation process they can be evaluated via the REF.
+The data used by the Climate-REF do not necessarily need to have been previously published to ESGF.
+As long as the datasets match the data requirements of the diagnostics and they conform with the CMIP6 era cmorisation process, they can be evaluated via the REF.
 
 If you are preparing data for a modelling centre,
 start with the [modelling centre onboarding guide](modelling-centres.md)
@@ -57,6 +64,65 @@ ref datasets fetch-data --registry obs4ref --output-directory $REF_CONFIGURATION
 The command fetches up to four files concurrently by default.
 This can be overridden by setting the `REF_DATASET_FETCH_WORKERS` environment variable.
 
+[](){#fetch-obs4mips-datasets}
+
+## 2. Fetching obs4MIPs datasets from ESGF
+
+The obs4REF registry does not cover every reference dataset the diagnostics need.
+The following are already published to obs4MIPs on ESGF and must be fetched from there:
+
+| `source_id` | Variables | Required by |
+| --- | --- | --- |
+| `20CR-V2` | `psl` | `pmp/extratropical-modes-of-variability-{nam,nao,npo,pna,sam}` |
+| `C3S-GTO-ECV-9-0` | `toz` | `esmvaltool/ozone-{annual-cycle,lat-time,nh-mar,sh-oct}` |
+| `CERES-EBAF-4-2-1` | `rlut`, `rlutcs`, `rsut`, `rsutcs` | `esmvaltool/cloud-radiative-effects` |
+| `ERA-5` | `psl`, `ta`, `tas`, `ua` | `esmvaltool/cloud-scatterplots-reference`, `esmvaltool/regional-historical-{annual-cycle,timeseries,trend}` |
+| `NOAA-NCEI-LAI-AVHRR-5-0` | `lai` | `ilamb/lai-avh15c1` |
+
+A diagnostic whose reference data is missing simply plans no executions,
+so an incomplete fetch shows up as a diagnostic that never runs rather than as an error.
+
+The same [./scripts/fetch-esgf.py](https://github.com/Climate-REF/climate-ref/blob/main/scripts/fetch-esgf.py)
+script used for the CMIP6 input data fetches these.
+If you fetched the obs4REF registry in the previous step, ask for exactly these five:
+
+```bash
+python scripts/fetch-esgf.py --request-id pmp-modes-20cr-obs4mips
+python scripts/fetch-esgf.py --request-id esmvaltool-ozone-obs4mips
+python scripts/fetch-esgf.py --request-id esmvaltool-cloud-radiative-effects-obs4mips
+python scripts/fetch-esgf.py --request-id esmvaltool-cloud-scatterplots-obs4mips
+python scripts/fetch-esgf.py --request-id esmvaltool-historical-obs4mips
+python scripts/fetch-esgf.py --request-id ilamb-lai-obs4mips
+```
+
+(`ERA-5` is split across two requests, hence six commands for five datasets.)
+
+`--kind obs4mips` fetches all of the reference data in one go,
+but it also re-fetches four datasets the obs4REF registry already provides,
+so only use it if you are **not** using that registry — see the warning below.
+
+Files land in the [intake-esgf `local_cache`](https://intake-esgf.readthedocs.io/en/latest/configure.html),
+and are ingested with the `obs4mips` source type, the same as the obs4REF collection.
+
+/// admonition | Do not fetch these twice
+    type: warning
+
+The script also fetches `CERES-EBAF-4-2`, `GPCP-Monthly-3-2`, `HadISST-1-1` and `TropFlux-1-0`.
+These are the ESGF-published copies of datasets that were curated for the REF before publication,
+so the obs4REF registry ships them as well.
+
+**If you have already fetched the obs4REF registry, do not fetch these from ESGF as well.**
+Where the two copies carry the same version they share an `instance_id` and ingest as a single
+dataset holding *both* sets of files, which covers the record twice.
+`GPCP-Monthly-3-2` `pr` `v20231205` is the clearest case: obs4REF ships one file spanning 1983-2023 and ESGF ships 41 yearly files spanning the same period, giving one dataset of 42 files.
+A diagnostic reading it sees every time step twice.
+
+Where the published copy carries a *newer* version, there is no such problem:
+the two ingest as separate datasets and the catalog uses the later version.
+
+This is temporary until we split the obs4REF ingest from the obs4MIPs ingest.
+///
+
 ### Future work
 
 The Climate-REF team is working on providing a more integrated way to fetch and manage these datasets from the Next Generation ESGF infrastructure that in the process of being deployed.

scripts/fetch-esgf.py

@@ -129,7 +129,10 @@ def fetch(self, remove_ensembles: bool = True) -> PathDict:
 
 Request = CMIP6Request | Obs4MIPsRequest
 
-# TODO use the data requirements from the diagnostics directly
+# TODO use the data requirements from the diagnostics directly.
+# Until then, `tests/scripts/test_fetch_esgf_coverage.py`
+# checks the obs4MIPs requests below against those requirements,
+# so a diagnostic cannot start asking for reference data that nothing here fetches.
 requests: list[Request] = [
     CMIP6Request(
         id="esmvaltool-climate-at-global-warmings-levels",
@@ -409,7 +412,6 @@ def fetch(self, remove_ensembles: bool = True) -> PathDict:
             variable_id="ts",
         ),
     ),
-    # The PMP psl modes of variability request source_id "20CR", which is the obs4REF spelling.
     Obs4MIPsRequest(
         id="pmp-modes-20cr-obs4mips",
         facets=dict(

tests/scripts/test_fetch_esgf_coverage.py

@@ -0,0 +1,190 @@
+"""
+Tests that `scripts/fetch-esgf.py` fetches the obs4MIPs reference data the diagnostics ask for.
+
+The requests in that script are maintained by hand so they can drift from the providers' data requirements.
+These tests check that the obs data requested is covered by a download.
+
+No network access is performed and nothing is fetched.
+"""
+
+import importlib.util
+from pathlib import Path
+
+import pandas as pd
+import pytest
+
+from climate_ref_core.diagnostics import DataRequirement
+from climate_ref_core.source_types import SourceDatasetType
+
+REPO_ROOT = Path(__file__).parents[2]
+SCRIPT = REPO_ROOT / "scripts" / "fetch-esgf.py"
+OBS4MIPS_CATALOG = REPO_ROOT / "tests" / "test-data" / "esgf-catalog" / "obs4mips_catalog.parquet"
+OBS4REF_REGISTRY = (
+    REPO_ROOT
+    / "packages"
+    / "climate-ref"
+    / "src"
+    / "climate_ref"
+    / "dataset_registry"
+    / "obs4ref_reference.txt"
+)
+
+# Number of path parts in an obs4REF registry key:
+# obs4REF/{institution_id}/{source_id}/{frequency}/{variable_id}/{grid_label}/{version}/{filename}
+_OBS4REF_KEY_PARTS = 8
+_KEY_SOURCE_ID = 2
+_KEY_VARIABLE_ID = 4
+
+
+@pytest.fixture(scope="module")
+def script():
+    spec = importlib.util.spec_from_file_location("fetch_esgf", SCRIPT)
+    assert spec is not None and spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+@pytest.fixture(scope="module")
+def obs4mips_requirements() -> list[tuple[str, DataRequirement]]:
+    """Every obs4MIPs-typed data requirement across the default providers, tagged with its slug."""
+    climate_ref_esmvaltool = pytest.importorskip("climate_ref_esmvaltool")
+    climate_ref_ilamb = pytest.importorskip("climate_ref_ilamb")
+    climate_ref_pmp = pytest.importorskip("climate_ref_pmp")
+
+    requirements = []
+    for provider in (
+        climate_ref_esmvaltool.provider,
+        climate_ref_pmp.provider,
+        climate_ref_ilamb.provider,
+    ):
+        for diagnostic in provider.diagnostics():
+            for item in diagnostic.data_requirements:
+                branch = item if isinstance(item, (list, tuple)) else [item]
+                for requirement in branch:
+                    if requirement.source_type is SourceDatasetType.obs4MIPs:
+                        requirements.append((f"{provider.slug}/{diagnostic.slug}", requirement))
+    assert requirements, "no obs4MIPs requirements found; the providers failed to load"
+    return requirements
+
+
+@pytest.fixture(scope="module")
+def obs4ref_source_ids() -> set[str]:
+    """The `source_id`s the obs4REF registry provides, which need no ESGF request."""
+    source_ids = set()
+    for raw_line in OBS4REF_REGISTRY.read_text().splitlines():
+        line = raw_line.strip()
+        if not line or line.startswith("#"):
+            continue
+        parts = line.split()[0].split("/")
+        if len(parts) == _OBS4REF_KEY_PARTS:
+            source_ids.add(parts[_KEY_SOURCE_ID])
+    return source_ids
+
+
+@pytest.fixture(scope="module")
+def esgf_catalog() -> pd.DataFrame:
+    """
+    The ESGF-published part of the committed obs4MIPs catalog snapshot.
+
+    The snapshot is built from a local archive that also holds the obs4REF collection,
+    so rows whose path sits under `obs4REF/` are excluded: they are provided by the registry
+    rather than by a fetch from ESGF.
+    """
+    catalog = pd.read_parquet(OBS4MIPS_CATALOG)
+    return catalog[~catalog["path"].str.contains("/obs4REF/")]
+
+
+def _requested_pairs(script) -> dict[tuple[str, str], str]:
+    """Map every (source_id, variable_id) the script requests to the id of the request."""
+    pairs = {}
+    for request in script.requests:
+        if not isinstance(request, script.Obs4MIPsRequest):
+            continue
+        source_ids = request.facets["source_id"]
+        variable_ids = request.facets["variable_id"]
+        source_ids = (source_ids,) if isinstance(source_ids, str) else source_ids
+        variable_ids = (variable_ids,) if isinstance(variable_ids, str) else variable_ids
+        for source_id in source_ids:
+            for variable_id in variable_ids:
+                pairs[(source_id, variable_id)] = request.id
+    return pairs
+
+
+def _requirement_source_ids(requirement: DataRequirement) -> set[str]:
+    source_ids: set[str] = set()
+    for facet_filter in requirement.filters:
+        value = facet_filter.facets.get("source_id")
+        if value is None:
+            continue
+        source_ids.update((value,) if isinstance(value, str) else value)
+    return source_ids
+
+
+def test_every_required_source_is_obtainable(script, obs4mips_requirements, obs4ref_source_ids):
+    """
+    Every `source_id` a diagnostic asks for is either in the obs4REF registry or fetched from ESGF.
+
+    This is the check that catches a new diagnostic naming a reference dataset that no one can obtain,
+    which would otherwise surface only as a diagnostic that never runs.
+    """
+    requested = {source_id for source_id, _ in _requested_pairs(script)}
+
+    unobtainable = {
+        source_id: slug
+        for slug, requirement in obs4mips_requirements
+        for source_id in _requirement_source_ids(requirement)
+        if source_id not in obs4ref_source_ids and source_id not in requested
+    }
+
+    assert not unobtainable, (
+        "These source_ids are required by a diagnostic but are in neither the obs4REF registry "
+        f"nor an Obs4MIPsRequest in {SCRIPT.name}: {unobtainable}"
+    )
+
+
+def test_every_selectable_dataset_is_fetched(script, obs4mips_requirements, esgf_catalog):
+    """
+    Every ESGF dataset a requirement can actually select is covered by a request.
+
+    Resolving against the catalog rather than taking the cross product of the requirement's facets matters.
+    ESGF intersects its facets,
+    so a requirement naming four sources and eight variables does not need all thirty-two combinations,
+    only the ones that exist.
+    """
+    requested = _requested_pairs(script)
+
+    missing: dict[tuple[str, str], set[str]] = {}
+    for slug, requirement in obs4mips_requirements:
+        selected = requirement.apply_filters(esgf_catalog)
+        for source_id, variable_id in zip(selected["source_id"], selected["variable_id"]):
+            if (source_id, variable_id) not in requested:
+                missing.setdefault((source_id, variable_id), set()).add(slug)
+
+    assert not missing, (
+        f"These obs4MIPs datasets are selected by a diagnostic but no request in {SCRIPT.name} "
+        f"fetches them: { {pair: sorted(slugs) for pair, slugs in missing.items()} }"
+    )
+
+
+def test_no_request_is_stale(script, obs4mips_requirements):
+    """
+    Every requested `source_id` is still named by a requirement.
+
+    Avoid silently downloads data that nothing will read.
+    """
+    required = {
+        source_id
+        for _, requirement in obs4mips_requirements
+        for source_id in _requirement_source_ids(requirement)
+    }
+
+    stale = {
+        source_id: request_id
+        for (source_id, _), request_id in _requested_pairs(script).items()
+        if source_id not in required
+    }
+
+    assert not stale, (
+        f"These requests in {SCRIPT.name} fetch a source_id that no diagnostic requires: {stale}"
+    )