Skip to content

fix(types): refuse NaN and out-of-range probabilities; stop the Brier floor indexing past a distribution - #78

Merged
TMHSDigital merged 1 commit into
mainfrom
fix/prediction-validation-and-brier-floor
Sep 23, 2026
Merged

TMHSDigital merged 1 commit into
mainfrom
fix/prediction-validation-and-brier-floor

Conversation

@TMHSDigital

Copy link
Copy Markdown
Owner

Fixes #31. Fixes #32.

#32: Prediction accepted impossible probabilities.

  • The check was that the distribution summed to about 1. It never checked the entries. {"a": nan, "b": 1.0} and {"a": 1.5, "b": -0.5} both passed, and so did a NaN latency.
  • Each distribution entry must now be finite and in [0, 1], and the error names the option. latency_ms must be finite.

#31: multiclass_brier_floor could index past the last option.

  • It drew the gold label with searchsorted over the raw cumulative sum. A distribution summing to 0.9995, which Prediction allows, then indexed one past the last option whenever the uniform draw landed above the sum.
  • The cumulative sum is now normalized and the index clamped. A distribution that sums to 1 draws exactly as before, so no existing figure moves. The site build confirms the example report still matches its command line for line.

Checked

  • New tests failed on main and pass now: NaN and negative distribution entries are refused, NaN and inf latency are refused, and the floor accepts a 0.9995 distribution.
  • ruff, mypy --strict, the full pytest suite, and the site build all pass.

🤖 Generated with Claude Code

…rier floor indexing past a distribution

Prediction checked that a distribution summed to about 1 but not that each
entry was a probability. {"a": nan, "b": 1.0} and {"a": 1.5, "b": -0.5}
both passed, as did a NaN latency; downstream, a NaN makes ECE and Brier
silently NaN and a negative entry corrupts the floors. Each entry must now
be finite and in [0, 1], and the error names the option. latency_ms must
be finite.

multiclass_brier_floor drew the gold label with searchsorted over the raw
cumulative sum, so a distribution summing to 0.9995 (which Prediction
allows) indexed one past the last option whenever the uniform draw landed
above the sum. The cumulative sum is now normalized and the index clamped;
a distribution that sums to 1 draws exactly as before, so no existing
figure moves.

Fixes #31. Fixes #32.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@TMHSDigital
TMHSDigital merged commit 22ea587 into main Sep 23, 2026
17 checks passed
@TMHSDigital
TMHSDigital deleted the fix/prediction-validation-and-brier-floor branch September 23, 2026 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant