Add station-group stratification by lclanzi · Pull Request #259 · MeteoSwiss/evalml

lclanzi · 2026-09-07T15:14:56Z

This PR adds a new stratification axis, station group (all / holdout / holdin), alongside the existing region/season/init_hour stratifications, plus supporting infrastructure to make ground-truth retrieval more flexible. Note that this is a single fixed holdout/holdin split used for stratified evaluation, not k-fold cross-validation.

verification metrics can now be split by all / holdout / holdin stations, so skill can be evaluated separately on stations that were withheld from nudging (or other applications) vs. those that fed into it.
experiment.cross_validation in the top-level config (exclude_stations, or holdout_fraction + holdout_seed for a random split) is the single source of truth for which stations define the holdout group
verify() (src/verification/__init__.py) adds a station_group dimension to verification outputs, alongside the existing region/season/init_hour dimensions.
The dashboard (report_experiment_dashboard.py, template.html.jinja2, script.js) and scorecards (report_scorecard.py) are updated to expose and correctly aggregate this new dimension.
verif_hash() in common.smk now includes the station-group configuration, so changing it correctly invalidates cached verification outputs instead of silently reusing stale results.
Geographic station filtering for jretrieve-based truth: data_input.jretrieve.parse_selection() now supports filter_mode=switzerland (trim to stations within the real Swiss national border) and filter_mode=domain;domain_bbox=... (trim to an arbitrary bounding box), plus an optional use_limitation=<N> selector for data-usage-rights filtering. This lets a truth-root retrieve over a broad domain and then restrict to a precise region before verification

This PR depends on MeteoSwiss/anemoi-plugins-meteoswiss#40 and should be merged only afterwards.

…nd holdout

jonasbhend

Hi @lclanzi. Thanks for implementing this. I think the 'cross-validation' with holdin/out stations is a very nice feature. Given the additional way to select stations from jretrieve, I was wondering if we really want to support bbox selection and filtering down to Switzerland? What is the use case here that is not covered by selection by measurement network (e.g. 1,2) or bbox? If we don't have a strong argument, I would rather avoid this additional complexity.

Also, I am unable to check the implementation on linard, as mlflow is still being blocked. Will try to do a short test on Wednesday to confirm everything is working as expected.

lclanzi · 2026-09-08T06:59:31Z

Hi @lclanzi. Thanks for implementing this. I think the 'cross-validation' with holdin/out stations is a very nice feature. Given the additional way to select stations from jretrieve, I was wondering if we really want to support bbox selection and filtering down to Switzerland? What is the use case here that is not covered by selection by measurement network (e.g. 1,2) or bbox? If we don't have a strong argument, I would rather avoid this additional complexity.

Also, I am unable to check the implementation on linard, as mlflow is still being blocked. Will try to do a short test on Wednesday to confirm everything is working as expected.

Hi @jonasbhend, thanks a lot for the feedback!
I see your point, and it makes sense. However, I wasn’t able to retrieve SwissMetNet + partner stations (limited to Switzerland) directly through the jretrieve interface. Therefore, I currently retrieve all stations within the bounding box and filter them afterwards to keep only those in Switzerland. If you know how to achieve this directly with jretrieve, I will remove this additional complexity.

jonasbhend · 2026-09-08T09:35:36Z

I know that this doesn't work (easily), but my question is: why do you want to use only stations in Switzerland? I guess for the nudging, you wouldn't want to constrain, and for the evaluation we could argue similarly. If anything, I think for the evaluation we should implement it as a stratifiable region (in essence the hull of all forecasting regions), which would then be available for both station and analysis-based evaluations.

… for stratification

lclanzi · 2026-09-11T12:43:16Z

Thanks for the suggestions @jonasbhend!
I have added a new shapefile for Switzerland. This region is now available in the list and it is used for applying a region stratification over the country (instead of the filter in jretrieve)

dnerini

very cool feature! I left some comments

dnerini · 2026-10-05T12:12:09Z

this config can't be used by anyone else since in multiple instances it points to your personal folders, correct? Can we fix this? Otherwise remove the yaml

I removed the yaml. I will make a separate PR with varda-rapid related configs

dnerini · 2026-10-05T12:12:45Z

is adding an experiment yaml really necessary for this new feature PR?

I removed the yaml. I will make a separate PR with varda-rapid related configs

dnerini · 2026-10-05T12:14:42Z

    return xr.Dataset(data_vars=data_vars, coords=coords)


+def _trim_stations_xr(


why is this necessary? can't be handled directly through the jretrieve query?

In principle, yes. However, I have not found a way to query SwissMetNet + partner stations within Switzerland using jretrieve. What I currently do is to query stations over a rectangular domain containing Switzerland with use_limitation=40, and further trim all stations located out of the Swiss borders. @clairemerker and @jonasbhend also looked into this, and If I remember correctly, you also had issues with this operation?

As mentioned previously, I don't see the point in doing the subsetting on the jretrieve part. For evalml it seems sufficient to use bbox selection in jretrieve together with region stratification (which now includes Switzerland only as a separate region, right)? So I still suggest to drop this to simplify and keep the loading of obs clean(ish).

As far as I know, bbox selection only works with a rectangular domain, meaning that stations out of Switzerland will be included in the ground truth. Or am I wrong?

No that is correct. bbox selection is what it is. But if you wanted stations only from Switzerland, there is probably ways to do that by selecting measurement networks. And in any case for verification you could just use bbox selection and then stratify by Switzerland as a region which should be equivalent to bbox selection and trimming to Switzerland on the jretrieve call. My question here is if there is a use case (for nudging I guess) where we would want to use only stations from Switzerland and not the surrounding areas (other than selecting SwissMetNet say).

Yes, selecting Switzerland as a region would indeed work. Good point. I will remove this filter.

It would be nice though to have a better understanding of the jretrieve API. I would not know for instance how to select measurement networks

yep, in evalml, jretrieve:1,2 for example selects the measurement network 1: SMN and 2: SMN Niederschlag

I think stn_group_id doesn't provide enough options to retrieve the partner stations in Switzerland, at least I didn't manage to do so (either not implemented or not documented?). Maybe using meas_group_id could work, but the list is overwhelming to me, we will need help from MD to understand what to retrieve...
(You can have a look at the options further down on that page: https://service.meteoswiss.ch/jretrieve/api/v1/surface/help)

dnerini · 2026-10-05T12:15:19Z

+
+    # Per-variable station-coverage log, so ground-truth coverage can be compared
+    # against what was actually available to nudge.
+    _logged_params = ("T_2M", "TD_2M", "U_10M", "V_10M", "PMSL", "TOT_PREC", "VMAX_10M")


why hardcoding these variables here?

dnerini · 2026-10-05T12:16:15Z

    command += config.profile.parsable()
    command += ["--configfile", str(configfile)]
    command += ["--cores", str(cores)]
+    command += ["--rerun-incomplete"]


off topic? ;)

dnerini · 2026-10-05T12:24:07Z

+        the config contains a ``forward_transform_filter:
+        nudge_toward_observation`` block, in which case its
+        ``exclude_stations``/``holdout_fraction``/``holdout_seed`` are
+        injected into that block — see ``_inject_nudging_station_holdout``.


I think I need to be convinced about this approach as it seems to me a bit too fragile...
Wouldn't be more robust to use the station_holdout_cfg to write a list of stations that is then used downstream, eg for nudging? I know the nudging filter expects a list of stations in its config, but maybe we could change that a path to a list of stations?

Luca Lanzilao and others added 17 commits July 2, 2026 10:22

add config for nudging

bc5373a

update configs

6e13d16

update config and small edits

a81103c

update configs for nudging

6968349

chore: fix pre-commit

930ef89

add cross validation for nudging

9e43c40

fix bug in dashboard plotting

9f48857

small edits to dashboard

5b88610

add cross validation functionality: stations grouped in all, holdin a…

8c4bb45

…nd holdout

update configs

91b4c26

change selector layout in dashboard

b5c41df

edit configs

7ffe364

update cross validation methodology, add filtering to jretrieve output

0572149

Merge branch 'main' into feat/nudge-initial-condition-toward-observation

ce68e03

fix linting

3e8feb8

restore varda-single config

65e611f

fix linting

133e3cf

lclanzi requested review from dnerini and jonasbhend September 7, 2026 15:29

jonasbhend reviewed Sep 8, 2026

View reviewed changes

Comment thread src/data_input/__init__.py Outdated

lclanzi added 4 commits September 10, 2026 14:19

update configs

5955d8a

remove filtering from jretrieve and insert a new region (switzerland)…

e97998b

… for stratification

Merge branch 'main' into feat/nudge-initial-condition-toward-observation

21e480c

fix linting

24b5b02

dnerini reviewed Sep 14, 2026

View reviewed changes

Luca Lanzilao added 2 commits September 22, 2026 09:45

reply reviewer comments

460deea

Merge branch 'main' into feat/nudge-initial-condition-toward-observation

8ca4ed5

Luca Lanzilao added 4 commits September 22, 2026 09:59

fix linting

f662266

disentangle nudging logic from station stratification

88ee5c7

fix linting

4d62b4c

fix jretrieve path in config

ec031eb

dnerini reviewed Sep 23, 2026

View reviewed changes

Comment thread src/data_input/jretrieve.py Outdated

remove reference to previous version in comments

4658375

lclanzi requested review from dnerini and jonasbhend October 1, 2026 11:37

dnerini reviewed Oct 5, 2026

View reviewed changes

remove varda-rapid configs

f1cbf29

		return xr.Dataset(data_vars=data_vars, coords=coords)


		def _trim_stations_xr(

Conversation

lclanzi commented Sep 7, 2026 • edited by dnerini Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonasbhend left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lclanzi commented Sep 8, 2026

Uh oh!

jonasbhend commented Sep 8, 2026

Uh oh!

lclanzi commented Sep 11, 2026

Uh oh!

dnerini left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lclanzi commented Sep 7, 2026 •

edited by dnerini

Loading