Add models.yaml as the single source of truth for leaderboard metadata

[shchur]

Summary

Submitters now declare how their model appears on the fev-bench leaderboard in benchmarks/fev_bench/models.yaml, in the same PR as their results CSV. Previously a maintainer filled in MODEL_CONFIG in the leaderboard Space by hand. After this, updating the leaderboard is a single command in the Space: uv run python save_tables.py [commit].

my-model:                    # = results CSV file name
  organization: My Org
  url: https://huggingface.co/my-org/my-model
  model_type: pretrained     # pretrained | task-specific | statistical | system | closed-api
  zero_shot: true
  commercial_use: true       # required, no default
  # display_name, hidden: optional

Changes

• One model per results CSV. The models.yaml key is the results CSV file name, so each key points to exactly one leaderboard row. toto-2_0.csv is split into one file per checkpoint size; the rows are unchanged (checked with assert_frame_equal).

• scripts/fev_bench_metadata.py defines the format as a pydantic model, ModelMetadata, and checks that entries and results files match one-to-one. It is the only place the format is documented; models.yaml, models/README.md, the tutorial and evaluate.py point to it. The submission tests and the leaderboard Space both load metadata through this module.

• model_type gains system and closed-api. The leaderboard hides models of these types until the reader opts in. closed-api takes precedence over system.

• hidden: true replaces the hardcoded Toto exclusion list in generate_fev_bench_figures.py.

• Tests (test_fev_bench_submissions.py): one new test loads all metadata, which validates every entry and the one-to-one match with the results files.

• evaluate.py points to the submission instructions after a run.

• Docs: submission steps updated in models/README.md and the add-your-model tutorial.

• Pairwise skill score heatmap color range narrowed from ±30% to ±15% (separate commit), since most pairwise skill scores are within ±10%. This changes the paper figures.

Review notes

• Licenses: commercial_use: false for TiRex, TimesFM-3, TabPFN-TS-3, CITRAS-FM, and Moirai-2.0 (its HF card says CC-BY-NC-4.0). All others are true. TS-ICL and TiRex-2 use custom licenses I haven't read; worth confirming.
• lingjiang2_api has url: TODO. Needs a real link before merge.
• The matching leaderboard Space change is committed locally and not yet pushed. It reads models.yaml through fev_bench_metadata.py, so it depends on this PR.

Testing

• uv run pytest test/test_fev_bench_submissions.py: 36 passed.
• Ran these suites locally: test_fev_bench_submissions.py, test_analysis.py, test_model.py, test_task.py, test_utils.py (168 passed). Not run locally: test_metrics.py (needs autogluon) and test_notebooks.py (needs nbformat), because those packages aren't installed here.
• generate_fev_bench_figures.py and render_fev_bench_leaderboard.py run cleanly; the figure script excludes the 3 hidden Toto sizes.
• Leaderboard Space run against this branch: save_tables.py --fev-repo .. succeeds, and the app has no errors with toggles on or off.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[abdulfatir]

Thanks Oleks! Only minor questions/comments.