Skip to content

Add verification support for surface variable added with post-processors - #265

Open
lclanzi wants to merge 7 commits into
mainfrom
feat/add-eval-for-postprocessor
Open

lclanzi wants to merge 7 commits into
mainfrom
feat/add-eval-for-postprocessor

Conversation

@lclanzi

@lclanzi lclanzi commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Adds wind speed (SP_10M), wind direction (DD_10M) and relative humidity (RELHUM_2M) as verifiable parameters,

  • Reads SP_10M/DD_10M/RELHUM_2M natively from ML-inference GRIB. Derives them from U_10M/V_10M/T_2M/TD_2M for baselines (ICON archive, INCA).
  • Fetches RELHUM_2M natively from the DWH (ure200h0) instead of deriving it from T_2M/TD_2M
  • Adds circular-aware BIAS/MSE/MAE/CORR for DD_10M: a plain difference near due-north (e.g. 355° vs 5°) previously measured a 350° error instead of the true 10°.
  • Adds a RELHUM_2M colormap default (viridis, 0-100%)

Luca Lanzilao added 4 commits September 18, 2026 18:20
# Conflicts:
#	src/verification/__init__.py
#	tests/unit/test_verification.py
#	workflow/rules/inference.smk

@jonasbhend jonasbhend left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @lclanzi. Very cool feature indeed.

I suggest we discuss if we want to ditch derived parameter computation in evalml in favor of using post-processors (and ignore sources for which derived parameters are unavailable). What do you think?

My second comment relates to direction of wind. For this, the standard verification metrics do not make sense (e.g. mean absolute error) and should be replaced with error metrics for circular quantities. I suggest to at least include a warning, or maybe ditch DD_10M for now and comment out the relevant sections.

@lclanzi

lclanzi commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the feedback @jonasbhend!

Yes, the idea is to compute all variables during inference, in order to minimize external operations.

For DD_10M:

  • Error metrics (bias, RMSE, MAE, correlation): before computing the error, I shift the observation to whichever "lap" around the circle is closest to the forecast. So forecast 355° vs. observed 5° reads as a 10° error, not 350°.
  • Mean/variance: instead of averaging degrees directly, I average the direction as a vector (its sin/cos components) and take the angle of that. mean(359°, 1°) comes out to ~0°.
  • Min/max: I removed these for wind direction

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants