Skip to content

feat(report): state the grid probabilities arrived on, and show it costs the floor nothing - #98

Merged
TMHSDigital merged 1 commit into
mainfrom
feat/report-probability-grid
Sep 25, 2026
Merged

TMHSDigital merged 1 commit into
mainfrom
feat/report-probability-grid

Conversation

@TMHSDigital

Copy link
Copy Markdown
Owner

Part of #9. It answers two of the three open questions.

Should the report state the observed grid? Yes. calibration.probability_grid(values) returns the coarsest of 0.1, 0.05, 0.01, 0.005 and 0.001 that every value sits on, or None. It needs at least 20 values, because a handful landing on a grid says nothing about the vendor. When there is a grid, the arm's section gets a line like this:

  • Resolution: all 80 probabilities this arm returned are multiples of 0.01, so no bin or threshold finer than 0.01 can mean anything, and a tie for the top option is ordinary rather than rare.

When the bins are narrower than the grid, the line adds a warning that the binning measures the rounding.

Is the achievable ECE floor higher than the one computed? No. scripts/quantization_floor.py simulates a perfectly calibrated model with continuous true probabilities, sends them rounded to the grid, and reads ECE on the rounded values against calibration_floor of those same values, as the report does. If rounding cost calibration that the floor doesn't model, the calibrated model would clear the floor's 95th percentile more than 5% of the time.

With 200 trials per cell:

rows bins grid mean ECE floor mean above p95
40 10 0.01 0.1141 0.1149 5.5%
500 10 0.01 0.0345 0.0339 5.5%
10,000 10 0.01 0.0074 0.0076 2.5%
10,000 10 0.05 0.0080 0.0075 6.5%
105 20 0.01 0.1002 0.0994 7.0%

The false-alarm rate stays at the nominal 5% (sampling error is about 1.5 points), even on a grid five times coarser than the vendor's. Rounding moves each value by at most half a step, in both directions, and since the floor is computed from the rounded values those errors cancel within a bin. METHODOLOGY has the full table, and the script reproduces it.

Still open on #9: whether the grid varies by model or by option count. That needs live runs against more than one model version.

Tests cover grid detection at 0.05, 0.01 and 0.001, no grid for values finer than that or for too few values, and the report line for an adapter on the 0.01 grid. docs/example-report.md is unchanged, since the mock's probabilities are continuous. The gate, the site checks and the prose checks all pass.

🤖 Generated with Claude Code

…sts the floor nothing

Hosted Jev rounds to two decimals. The report now says so at the point of
use: when every probability an arm returned sits on a grid of 0.001 or
coarser, over at least 20 values, a Resolution line names it, and warns
when the bins are narrower than the grid.

scripts/quantization_floor.py answers whether the rounding raises the ECE
floor. Across 40 to 10,000 rows, 10 or 20 bins, and grids of 0.01 or 0.05,
a calibrated model reported on the grid clears the floor's 95th percentile
at the nominal 5 percent, so the floor needs no correction. METHODOLOGY
carries the table.

Part of #9: whether the grid varies by model or option count needs live runs.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@TMHSDigital
TMHSDigital merged commit e81dfde into main Sep 25, 2026
23 checks passed
@TMHSDigital
TMHSDigital deleted the feat/report-probability-grid branch September 25, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant