feat(site): score your own pasted predictions against their floor - #95
Merged
Merged
Conversation
A new section on the explainer takes pasted rows, a probability and an outcome per line in CSV, tabs, or spaces with an optional header, and shows the ECE, the floor for exactly those predictions, and the verdict a report would print, word for word. parsePredictions in floor.js names each unreadable row by its line and nothing is scored until every row reads; the scoring runs in the worker, and nothing pasted is uploaded, stored, or put in the address. floor_golden.py exports four pasted texts scored by ece_figure, two with the ECE on and beside a rounding tie, and check_floor_parity.mjs parses and scores them and pins the parser's refusals. The smoke test pastes a bad row, then the worked example's rows, which must give the report's line. Fixes #56. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
TMHSDigital
force-pushed
the
feat/site-paste-predictions
branch
from
September 24, 2026 23:06
d9c1545 to
21e1a92
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #56.
What it does. A new "Your own predictions" section on the explainer, after the worked example. You paste rows, and it shows the measured ECE, the floor for exactly those predictions, and the verdict a report would print, word for word. The math reuses what
floor.jsalready has:ece,calibrationFloor(seed 0, 2,000 draws, equal-width bins at the chosen bin count) andstatement. That's the same path asece_figurein the Python.Input format.
1/0ortrue/false).Errors. The new
parsePredictionsinfloor.jsis a pure function, exported so it can be tested. It reports each problem as{ line, message }:NaN,Infinityand0x1)Nothing is scored until every row reads. The status line (
aria-live) states the first problem, and a list shows up to ten. The box getsaria-invalidand is described by both.Performance. Scoring runs only when you press "Score these rows", never on each keystroke, and it runs in the worker with progress and Cancel. It falls back to the main thread where a worker can't start, as the other tools do.
Privacy. The section says plainly that nothing pasted leaves the page: it isn't uploaded, stored, or put in the address. The copy button shares the result line only, never the rows.
A built-in self-check. A "Use the worked example's rows" button fills in the example's 105 rows. Scoring them reproduces the report's line exactly.
Parity.
floor_golden.pyexports four pasted texts: CSV with a header; tabs with CRLF,true/falseand a blank line; spaces with the ECE exactly on a rounding tie (0.03125, printed0.0312); and semicolons with quotes, beside a tie (0.0012499..., printed0.0012). Each is scored byece_figure, and--checkcovers them.check_floor_parity.mjsparses and scores each one. It requires the same row count, ECE, mean and p95 within 1e-9, and an identical verdict. It also pins 10 parser cases.trueas false), and the check failed with 3 disagreements.Smoke test. Two new browser checks:
Checks run. ruff, format, mypy --strict and pytest pass.
floor_golden.py --checkpasses. On a fresh build, the parity check (with the worked example), links, search, and smoke (8 of 8, no console error, no CSP violation) all pass, as do the CI prose checks. The fixture grows by about 20 KB.🤖 Generated with Claude Code