Skip to content

feat(benchmark): Phase 4 validation pipeline integration, metric_CRPS… - #4062

Open
ayushman1210 wants to merge 4 commits into
PecanProject:developfrom
ayushman1210:phase4-benchmarking-integration
Open

feat(benchmark): Phase 4 validation pipeline integration, metric_CRPS…#4062
ayushman1210 wants to merge 4 commits into
PecanProject:developfrom
ayushman1210:phase4-benchmarking-integration

Conversation

@ayushman1210

Copy link
Copy Markdown
Contributor

Description

This PR implements Phase 4: Biogeochemistry MVP Integration and Initial Benchmarks (#4059), proving the decoupled PEcAn validation pipeline end-to-end against real model output and observations.

Key changes include:

  • metric_CRPS.R Implementation & Metric Registry: Added Continuous Ranked Probability Score in metric_CRPS.R using scoringRules::crps_sample() with pure-R analytical fallback. Registered CRPS in pecan_metric_registry and added unit test suite test-metric_CRPS.R (6/6 PASS).
  • Salinas SOCs Quarto HTML Scorecard: Created salinas-socs/ensemble_benchmark/run_benchmarks.R to ingest EFI-long ensemble outputs, align monthly model data to annual observations with align_by_time(), compute point metrics (RMSE, MAE, R2) & spread metrics (Coverage, CRPS), and compile a dynamic HTML report (Salinas_SOC_Validation_Report.html).
  • Fast Deterministic CI Integration Test: Added test-ensemble_integration.R in testthat (18/18 PASS) to verify time alignment, metric calculations, and scorecard column structures without heavy model execution in CI.
  • Visualization & Coverage Enhancements: Refactored metric_Coverage.R for direct in-band scoring and updated metric_timeseries_plot.R with numerical PMU scores and Status: PASS/FAIL annotations.

Motivation and Context

This change fulfills Phase 4 of the GSoC 2026 Validation Toolkit workplan. Building on the core data intake API (#4017), test coverage (#4032), and Quarto reporting layer (#4040, #4041), Phase 4 proves the pipeline on real biogeochemistry datasets (Salinas SOC ensemble & AmeriFlux MVP) and introduces ensemble spread metrics (CRPS, Coverage).

Closes #4059.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My change requires a change to the documentation.
  • My name is in the list of CITATION.cff
  • I agree that PEcAn Project may distribute my contribution under any or all of
    • the same license as the existing code,
    • and/or the BSD 3-clause license.
  • I have updated the CHANGELOG.md.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@ayushman1210
ayushman1210 force-pushed the phase4-benchmarking-integration branch from 6bb6f9a to 25e7854 Compare August 4, 2026 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

Phase 4: Biogeochemistry MVP Integration and Initial Benchmarks

1 participant