Skip to content

feat(site): score your own pasted predictions against their floor - #95

Merged
TMHSDigital merged 2 commits into
mainfrom
feat/site-paste-predictions
Sep 24, 2026
Merged

TMHSDigital merged 2 commits into
mainfrom
feat/site-paste-predictions

Conversation

@TMHSDigital

Copy link
Copy Markdown
Owner

Fixes #56.

What it does. A new "Your own predictions" section on the explainer, after the worked example. You paste rows, and it shows the measured ECE, the floor for exactly those predictions, and the verdict a report would print, word for word. The math reuses what floor.js already has: ece, calibrationFloor (seed 0, 2,000 draws, equal-width bins at the chosen bin count) and statement. That's the same path as ece_figure in the Python.

Input format.

  • One row per line: a probability, then an outcome (1/0 or true/false).
  • Separated by commas, semicolons, tabs, or spaces. Fields may be quoted.
  • A first line that names its columns is skipped as a header. So are blank lines and one trailing separator.
  • Up to 20,000 rows.

Errors. The new parsePredictions in floor.js is a pure function, exported so it can be tested. It reports each problem as { line, message }:

  • a probability that is not a plain number (this includes NaN, Infinity and 0x1)
  • a probability outside 0 to 1
  • an outcome that is not 0 or 1
  • the wrong number of columns
  • no rows at all
  • more than 20,000 rows

Nothing is scored until every row reads. The status line (aria-live) states the first problem, and a list shows up to ten. The box gets aria-invalid and is described by both.

Performance. Scoring runs only when you press "Score these rows", never on each keystroke, and it runs in the worker with progress and Cancel. It falls back to the main thread where a worker can't start, as the other tools do.

Privacy. The section says plainly that nothing pasted leaves the page: it isn't uploaded, stored, or put in the address. The copy button shares the result line only, never the rows.

A built-in self-check. A "Use the worked example's rows" button fills in the example's 105 rows. Scoring them reproduces the report's line exactly.

Parity.

  • floor_golden.py exports four pasted texts: CSV with a header; tabs with CRLF, true/false and a blank line; spaces with the ECE exactly on a rounding tie (0.03125, printed 0.0312); and semicolons with quotes, beside a tie (0.0012499..., printed 0.0012). Each is scored by ece_figure, and --check covers them.
  • check_floor_parity.mjs parses and scores each one. It requires the same row count, ECE, mean and p95 within 1e-9, and an identical verdict. It also pins 10 parser cases.
  • I broke the parser on purpose (read true as false), and the check failed with 3 disagreements.

Smoke test. Two new browser checks:

  • A bad row (line 3, probability 1.4) is named, the box is marked invalid, and no result is shown.
  • The example's rows, pasted, give exactly the report's line, and the address doesn't change.

Checks run. ruff, format, mypy --strict and pytest pass. floor_golden.py --check passes. On a fresh build, the parity check (with the worked example), links, search, and smoke (8 of 8, no console error, no CSP violation) all pass, as do the CI prose checks. The fixture grows by about 20 KB.

🤖 Generated with Claude Code

TMHSDigital and others added 2 commits September 24, 2026 19:04
A new section on the explainer takes pasted rows, a probability and an
outcome per line in CSV, tabs, or spaces with an optional header, and
shows the ECE, the floor for exactly those predictions, and the verdict a
report would print, word for word. parsePredictions in floor.js names each
unreadable row by its line and nothing is scored until every row reads;
the scoring runs in the worker, and nothing pasted is uploaded, stored, or
put in the address.

floor_golden.py exports four pasted texts scored by ece_figure, two with
the ECE on and beside a rounding tie, and check_floor_parity.mjs parses
and scores them and pins the parser's refusals. The smoke test pastes a
bad row, then the worked example's rows, which must give the report's
line.

Fixes #56.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@TMHSDigital
TMHSDigital force-pushed the feat/site-paste-predictions branch from d9c1545 to 21e1a92 Compare September 24, 2026 23:06
@TMHSDigital
TMHSDigital merged commit 0e5c63b into main Sep 24, 2026
23 checks passed
@TMHSDigital
TMHSDigital deleted the feat/site-paste-predictions branch September 24, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Site: paste your own predictions and see ECE against its floor

1 participant