diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c4fb30..db8d971 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,7 +29,16 @@ different event from one that moved because it was wrong. with `plumbline report` prints the same Dataset section, with the read, loaded, and refused counts, as the one written at run time (#58). Artifacts written before this still report, without the section. - +- The site scores your own predictions (#56). Paste a probability and an + outcome per line, as CSV, tabs, or spaces, with or without a header, and the + page computes the ECE, the floor for exactly those predictions, and the + verdict a report would print, word for word. A row that cannot be read is + named by its line, and nothing is scored until every row reads. Nothing + pasted leaves the browser: it is not uploaded, stored, or put in the address. + The parity check holds the path to `ece_figure` on pasted text in four + formats, two of them on and beside a rounding tie, and pins the parser's + refusals; the smoke test pastes a bad row and the worked example's rows, + which must reproduce the report's line. - A browser calculator for the ECE floor (`site/`), a JavaScript port of `synthetic_floor` that reproduces numpy's seeded random stream draw for draw. `scripts/floor_golden.py` exports golden values from the Python and diff --git a/scripts/check_floor_parity.mjs b/scripts/check_floor_parity.mjs index 4d4981f..ee6f5dd 100644 --- a/scripts/check_floor_parity.mjs +++ b/scripts/check_floor_parity.mjs @@ -53,6 +53,58 @@ for (const c of fixture.cases) { console.log(`${label.padEnd(36)} mean ${band.mean.toFixed(6)} p95 ${band.p95.toFixed(6)} ${elapsed}ms`); } +// Pasted predictions: the page parses the text, then scores it as a report +// does. Each case must parse to the rows the Python scored and print the same +// line, character for character. +for (const c of fixture.pasted) { + const label = `pasted, ${c.label}`; + const parsed = floor.parsePredictions(c.text); + if (parsed.errors.length) { + failures.push(`${label}: refused: ${JSON.stringify(parsed.errors.slice(0, 2))}`); + continue; + } + if (parsed.rows !== c.n) failures.push(`${label}: read ${parsed.rows} rows, the Python scored ${c.n}`); + const measured = floor.ece(parsed.probabilities, parsed.correct, c.n_bins); + const band = floor.calibrationFloor(parsed.probabilities, c.n_bins); + for (const [key, actual] of [["ece", measured], ["mean", band.mean], ["p95", band.p95]]) { + const gap = Math.abs(actual - c[key]); + worst = Math.max(worst, gap); + if (!(gap <= tolerance)) failures.push(`${label}: ${key} ${actual} against Python ${c[key]} (off by ${gap})`); + } + const line = floor.statement(measured, band); + if (line !== c.statement) failures.push(`${label}: verdict differs\n js: ${line}\n python: ${c.statement}`); +} +console.log(`${fixture.pasted.length} pasted texts parsed and scored`); + +// What the parser refuses, and the line it names. There is no Python to hold +// these to: the report never reads pasted text. They pin the page's own rules. +const refusals = [ + ["0.5,1\n1.2,0", [2, "the probability 1.2 is outside 0 to 1"]], + ["probability,outcome\n0.5,1\nabc,1", [3, 'the probability "abc" is not a number']], + ["NaN,1\n0.5,1", [1, 'the probability "NaN" is not a number']], + ["0.5,2", [1, 'the outcome "2" is not 0 or 1']], + ["0.5\n0.5,1", [1, "expected two values, a probability and an outcome, and found 1"]], + ["0.5,1,0.3", [1, "expected two values, a probability and an outcome, and found 3"]], + ["", [null, "there are no rows to score. Paste one row per line: a probability, then 0 or 1"]], + ["probability,outcome\n\n", [null, "there are no rows to score. Paste one row per line: a probability, then 0 or 1"]], +]; +for (const [text, [line, message]] of refusals) { + const { errors } = floor.parsePredictions(text); + const got = errors[0]; + if (!got || got.line !== line || got.message !== message) { + failures.push(`parser: ${JSON.stringify(text)} gave ${JSON.stringify(got)}, expected line ${line}: ${message}`); + } +} +const capped = floor.parsePredictions("0.5,1\n0.5,0\n0.5,1", { maxRows: 2 }).errors; +if (!(capped.length === 1 && capped[0].line === null && capped[0].message.startsWith("3 rows is more than this page scores"))) { + failures.push(`parser: a paste over the row cap gave ${JSON.stringify(capped)}`); +} +const lenient = floor.parsePredictions(' 0.5, 1,\n\n"0.25"\tfalse \n1e-1 TRUE\n'); +if (lenient.errors.length || lenient.probabilities.join() !== "0.5,0.25,0.1" || lenient.correct.join() !== "true,false,true") { + failures.push(`parser: lenient input read as ${JSON.stringify(lenient)}`); +} +console.log(`${refusals.length + 2} parser cases checked`); + // Every figure on the page is printed through fixed4, which must round as // Python's "%.4f" does, including on and beside a tie. const formats = Object.entries(fixture.fixed4); diff --git a/scripts/floor_golden.py b/scripts/floor_golden.py index 2129fe2..020290a 100644 --- a/scripts/floor_golden.py +++ b/scripts/floor_golden.py @@ -25,7 +25,8 @@ import numpy as np -from plumbline.metrics.calibration import CalibrationFigure, synthetic_floor +from plumbline.metrics.calibration import CalibrationFigure, ece_figure, synthetic_floor +from plumbline.types import ProbabilitySeries FIXTURE = Path(__file__).resolve().parent.parent / "site" / "floor-golden.json" @@ -93,6 +94,81 @@ def _formatting_cases() -> dict[str, str]: return {repr(value): f"{value:.4f}" for value in sorted(values)} +def _pasted_cases() -> list[dict[str, Any]]: + """Pasted text in the formats the page reads, with what a report prints for it. + + The page parses the text itself, so each case is written the way a person + might paste it, and the Python scores the rows the text was written from + through ``ece_figure``, the call a report makes. Probabilities are written + with ``repr``, which round-trips exactly, so both sides score the same + doubles. Two cases put the measured ECE on and beside a rounding tie. + """ + rng = np.random.default_rng(56) + confident = [float(p) for p in rng.beta(5.0, 2.0, 300)] + confident_hits = [bool(u < p * 0.9) for u, p in zip(rng.random(300), confident, strict=True)] + spread = [float(p) for p in rng.uniform(0.0, 1.0, 150)] + spread_hits = [bool(u < p) for u, p in zip(rng.random(150), spread, strict=True)] + tie = [0.75] * 32 + tie_hits = [True] * 23 + [False] * 9 # 23/32 against 0.75: ECE is exactly 0.03125 + near = [0.80125] * 400 + near_hits = [True] * 320 + [False] * 80 # 0.8 against 0.80125: beside 0.00125 + + def rows(ps: list[float], hits: list[bool], line: str) -> list[str]: + return [ + line.format(p=repr(p), y=int(y), word=str(y).lower()) + for p, y in zip(ps, hits, strict=True) + ] + + spread_lines = rows(spread, spread_hits, "{p}\t{word}") + spread_lines.insert(40, "") + series = [ + ( + "comma separated, with a header", + 10, + confident, + confident_hits, + "probability,outcome\n" + "\n".join(rows(confident, confident_hits, "{p},{y}")) + "\n", + ), + ( + "tab separated, CRLF line ends, true and false, a blank line", + 15, + spread, + spread_hits, + "\r\n".join(spread_lines), + ), + ( + "space separated, the ECE exactly on a rounding tie", + 10, + tie, + tie_hits, + "\n".join(rows(tie, tie_hits, " {p} {y}")), + ), + ( + "semicolons and quotes, the ECE beside a tie", + 10, + near, + near_hits, + 'p;"correct"\n' + "\n".join(rows(near, near_hits, '"{p}";"{y}"')), + ), + ] + cases = [] + for label, n_bins, ps, hits, text in series: + figure = ece_figure(ProbabilitySeries(tuple(ps), "calibrated_claim"), hits, n_bins=n_bins) + cases.append( + { + "label": label, + "n_bins": n_bins, + "text": text, + "n": figure.n, + "ece": figure.value, + "mean": figure.floor.mean, + "p95": figure.floor.p95, + "statement": figure.statement(), + } + ) + return cases + + def _pcg64_state(seed: int) -> dict[str, str]: state = np.random.default_rng(seed).bit_generator.state["state"] return {"state": str(state["state"]), "inc": str(state["inc"])} @@ -125,8 +201,9 @@ def build() -> dict[str, Any]: ) return { "about": ( - "Golden values from plumbline.metrics.calibration.synthetic_floor, " - "which site/floor.js must reproduce. Written by scripts/floor_golden.py." + "Golden values from plumbline.metrics.calibration (synthetic_floor for the " + "calculator, ece_figure for pasted rows), which site/floor.js must " + "reproduce. Written by scripts/floor_golden.py." ), "function": "synthetic_floor(n, n_bins, accuracy), all other arguments default", "defaults": {"binning": "equal_width", "n_boot": 2000, "concentration": 6.0, "seed": 0}, @@ -135,6 +212,7 @@ def build() -> dict[str, Any]: "pcg64_initial_state": {"0": _pcg64_state(0), "1": _pcg64_state(1)}, "cases": cases, "fixed4": _formatting_cases(), + "pasted": _pasted_cases(), } @@ -147,6 +225,16 @@ def _check(fresh: dict[str, Any]) -> list[str]: problems.append("numpy's seeded PCG64 states differ from the fixture") if committed.get("fixed4") != fresh["fixed4"]: problems.append("the :.4f formatting cases differ from the fixture") + old_pasted = committed.get("pasted", []) + if [c["text"] for c in old_pasted] != [c["text"] for c in fresh["pasted"]]: + problems.append("the pasted-text cases differ from the fixture; regenerate it") + else: + for old, new in zip(old_pasted, fresh["pasted"], strict=True): + for key in ("ece", "mean", "p95"): + if abs(old[key] - new[key]) > TOLERANCE: + problems.append(f"pasted, {new['label']}: {key} is {new[key]!r} now") + if old["statement"] != new["statement"]: + problems.append(f"pasted, {new['label']}: verdict wording changed") if len(committed["cases"]) != len(fresh["cases"]): problems.append("the fixture holds a different set of cases; regenerate it") return problems diff --git a/scripts/smoke_site.mjs b/scripts/smoke_site.mjs index 8a46e0d..c0df8ac 100644 --- a/scripts/smoke_site.mjs +++ b/scripts/smoke_site.mjs @@ -17,6 +17,8 @@ // field invalid; // - the calculator runs, and writes its inputs into the address; // - a link carrying those inputs runs the calculator as the page opens; +// - pasted predictions with a bad row name the line and score nothing, and the +// worked example's rows, pasted, reproduce the report's line; // - search, opened from the header on a doc page, finds a section of that doc. import { spawn } from "node:child_process"; @@ -313,6 +315,27 @@ await check("a link carrying the inputs runs the calculator as the page opens", await violations(); }); +await check("a pasted row that cannot be read names its line and scores nothing", async () => { + await until(`!${$("paste-fill")}.hidden`, "the example's rows to be offered"); + await evaluate(`${$("paste-data")}.value = "probability,outcome\\n0.5,1\\n1.4,0"; ${$("paste-go")}.click();`); + const status = await evaluate(`${$("paste-status")}.textContent`); + expect(status.startsWith("Line 3: the probability 1.4 is outside 0 to 1."), `status reads ${JSON.stringify(status)}`); + expect((await evaluate(`${$("paste-data")}.getAttribute("aria-invalid")`)) === "true", "the box is not marked invalid"); + expect(await evaluate(`${$("paste-result")}.hidden`), "a result was shown for rows that did not read"); +}); + +await check("the worked example's rows, pasted, give the report's line", async () => { + const before = await evaluate("location.search"); + await evaluate(`${$("paste-fill")}.click(); ${$("paste-go")}.click();`); + await until(`!${$("paste-result")}.hidden && !${$("paste-go")}.disabled`, "a scored paste"); + const verdict = await evaluate(`${$("paste-verdict")}.querySelector("span").textContent`); + const report = await evaluate(`${$("ex-report-line")}.lastElementChild.textContent`); + expect(verdict === report, `pasted verdict ${JSON.stringify(verdict.slice(0, 80))} is not the report's ${JSON.stringify(report.slice(0, 80))}`); + expect(!(await evaluate(`${$("paste-data")}.hasAttribute("aria-invalid")`)), "the box is still marked invalid"); + expect((await evaluate("location.search")) === before, "scoring pasted rows changed the address"); + await violations(); +}); + await check("search on a doc page finds a section of it", async () => { await open("docs/methodology.html"); await evaluate("document.querySelector('.search-open').click()"); diff --git a/site/explainer.css b/site/explainer.css index 9193a0f..ba4016b 100644 --- a/site/explainer.css +++ b/site/explainer.css @@ -6,12 +6,16 @@ section { padding: 2rem 0; border-top: 1px solid var(--line); } form { display: grid; grid-template-columns: repeat(auto-fit, minmax(9.5rem, 1fr)); gap: 1rem; margin: 1.25rem 0 1rem; } label { display: block; font-size: 0.9rem; font-weight: 600; margin-bottom: 0.25rem; } .hint { display: block; font-size: 0.8rem; font-weight: 400; color: var(--muted); } -input { width: 100%; font: 1rem var(--mono); padding: 0.5rem 0.6rem; color: var(--fg); background: var(--bg); border: 1px solid var(--control); border-radius: 4px; } +input, textarea { width: 100%; font: 1rem var(--mono); padding: 0.5rem 0.6rem; color: var(--fg); background: var(--bg); border: 1px solid var(--control); border-radius: 4px; } .actions { grid-column: 1 / -1; display: flex; gap: 1rem; align-items: center; flex-wrap: wrap; } button { font: 600 1rem var(--sans); padding: 0.55rem 1.1rem; color: var(--bg); background: var(--fg); border: 0; border-radius: 4px; cursor: pointer; } button[disabled] { opacity: 0.55; cursor: progress; } progress { flex: 1 1 8rem; height: 0.5rem; } .error { color: var(--error); } +textarea { display: block; min-height: 9rem; resize: vertical; font-size: 0.9rem; line-height: 1.4; } +form .wide { grid-column: 1 / -1; } +.note { border-left: 3px solid var(--line); padding: 0.2rem 0 0.2rem 0.8rem; } +.paste-errors { margin: 0 0 1rem; padding-left: 1.25rem; font-size: 0.9rem; } .result { border: 1px solid var(--line); border-radius: 6px; padding: 1rem; background: var(--panel); } .result[hidden], [hidden] { display: none !important; } @@ -98,5 +102,5 @@ a.button.secondary, button.secondary { color: var(--fg); background: var(--bg); a.button.secondary:hover, button.secondary:hover { border-color: var(--fg); } button.small-button { font-size: 0.85rem; font-weight: 400; padding: 0.3rem 0.7rem; } .share { display: flex; flex-wrap: wrap; gap: 0.5rem; margin: 0 0 1rem; } -input[aria-invalid="true"] { border-color: var(--error); outline: 1px solid var(--error); } +input[aria-invalid="true"], textarea[aria-invalid="true"] { border-color: var(--error); outline: 1px solid var(--error); } @media print { .rail, .cta, .share, .actions { display: none !important; } } diff --git a/site/explainer.js b/site/explainer.js index fd0356e..a0a0f6f 100644 --- a/site/explainer.js +++ b/site/explainer.js @@ -38,8 +38,11 @@ function runHere(job) { setTimeout(function () { try { - if (job.message.kind === "synthetic") { - job.resolve(F.syntheticFloor(job.message.n, job.message.nBins, job.message.accuracy)); + var m = job.message; + if (m.kind === "synthetic") { + job.resolve(F.syntheticFloor(m.n, m.nBins, m.accuracy)); + } else if (m.kind === "observed") { + job.resolve({ measured: F.ece(m.probabilities, m.correct, m.nBins), band: F.calibrationFloor(m.probabilities, m.nBins) }); } else { job.resolve(F.planRows(job.message.target, job.message.nBins, job.message.accuracy, { onStep: job.onStep })); } @@ -107,7 +110,7 @@ }, }; } - var runners = { calc: makeRunner(), plan: makeRunner() }; + var runners = { calc: makeRunner(), plan: makeRunner(), paste: makeRunner() }; /* ---- forms ------------------------------------------------------------ */ @@ -140,7 +143,7 @@ } function clearInvalid(form) { - Array.prototype.forEach.call(form.querySelectorAll("input"), function (input) { + Array.prototype.forEach.call(form.querySelectorAll("input, textarea"), function (input) { input.removeAttribute("aria-invalid"); input.removeAttribute("aria-describedby"); }); @@ -205,7 +208,7 @@ }); } - var lastResult = { calc: "", plan: "" }; + var lastResult = { calc: "", plan: "", paste: "" }; ["calc", "plan"].forEach(function (prefix) { function copyButton(id, what, valueOf) { $(id).addEventListener("click", function () { @@ -220,6 +223,16 @@ copyButton(prefix + "-copy-result", "Result", function () { return lastResult[prefix]; }); $(prefix + "-cancel").addEventListener("click", function () { runners[prefix].cancel(); }); }); + // Pasted rows are never put in the address, so this tool shares a result + // and not a link. + $("paste-cancel").addEventListener("click", function () { runners.paste.cancel(); }); + $("paste-copy-result").addEventListener("click", function () { + var status = $("paste-status"); + copy(lastResult.paste).then( + function () { status.classList.remove("error"); status.textContent = "Result copied."; }, + function () { status.textContent = "Could not copy; select the text instead."; } + ); + }); /* ---- calculator ------------------------------------------------------ */ @@ -348,6 +361,8 @@ } function showExample(ex) { + exampleRows = ex; + $("paste-fill").hidden = false; var p = ex.probabilities, correct = ex.correct, nBins = ex.n_bins, n = p.length; $("ex-report-line").replaceChildren(el("p", ex.report.accuracy_line), el("p", ex.report.ece_line)); @@ -405,6 +420,91 @@ $("ex-lesson").hidden = false; } + /* ---- your own predictions --------------------------------------------- */ + + // Nothing pasted here leaves the page: it is read below, scored in the + // worker, and never written to the address, storage, or the network. + var exampleRows = null; + var PASTE_ERRORS_SHOWN = 10; + + $("paste-fill").addEventListener("click", function () { + if (!exampleRows) return; + var lines = ["probability,outcome"]; + exampleRows.probabilities.forEach(function (p, i) { lines.push(p + "," + (exampleRows.correct[i] ? 1 : 0)); }); + $("paste-data").value = lines.join("\n") + "\n"; + $("paste-bins").value = String(exampleRows.n_bins); + busy("paste", false, "Filled in the worked example's " + exampleRows.probabilities.length + + " rows. Score them to see the report's line again."); + }); + + function showPasteErrors(errors) { + var list = $("paste-errors"); + list.replaceChildren(); + errors.slice(0, PASTE_ERRORS_SHOWN).forEach(function (e) { + list.appendChild(el("li", (e.line === null ? "" : "Line " + e.line + ": ") + e.message + ".")); + }); + if (errors.length > PASTE_ERRORS_SHOWN) { + list.appendChild(el("li", (errors.length - PASTE_ERRORS_SHOWN) + " more not shown.", "muted")); + } + list.hidden = !errors.length; + } + + $("paste").addEventListener("submit", function (event) { + event.preventDefault(); + var form = event.target; + clearInvalid(form); + showPasteErrors([]); + var bins = number(form.bins, isWhole(1, 1000), "Bins must be a whole number from 1 to 1,000."); + if (bins.message) { busy("paste", false); fail("paste", bins); return; } + var parsed = F.parsePredictions(form.data.value); + if (parsed.errors.length) { + // Nothing is scored until every row reads. The first problem is said in + // the status line; the list holds the rest, and the box points at both. + var first = parsed.errors[0]; + var more = parsed.errors.length - 1; + busy("paste", false); + fail("paste", { + message: (first.line === null ? "" : "Line " + first.line + ": ") + first.message + "." + + (more ? " " + more + " more " + (more === 1 ? "problem" : "problems") + " below." : "") + + " Nothing was scored.", + }); + showPasteErrors(parsed.errors); + form.data.setAttribute("aria-invalid", "true"); + form.data.setAttribute("aria-describedby", "paste-status paste-errors"); + form.data.focus(); + $("paste-result").hidden = true; + return; + } + var running = runners.paste.run( + { kind: "observed", probabilities: parsed.probabilities, correct: parsed.correct, nBins: bins.value }, + function (f) { $("paste-progress").value = f; }); + busy("paste", true, "Scoring " + parsed.rows.toLocaleString("en-US") + " rows and running 2,000 resamples..."); + running + .then(function (result) { busy("paste", false, showPaste(result, parsed)); }) + .catch(function (error) { busy("paste", false); fail("paste", error); }); + }); + + // Draws the result, and returns the one line the status reads out. + function showPaste(result, parsed) { + var band = result.band, measured = result.measured; + var clears = F.isDistinguishable(measured, band); + var statement = F.statement(measured, band); + text("paste-n", band.n.toLocaleString("en-US") + (parsed.header ? " (a header line was skipped)" : "")); + text("paste-ece", fmt(measured)); + text("paste-mean", fmt(band.mean)); + text("paste-p95", fmt(band.p95)); + var verdict = $("paste-verdict"); + verdict.className = "verdict " + (clears ? "distinguishable" : "inconclusive"); + verdict.replaceChildren( + el("strong", clears ? "Distinguishable from sampling noise" : "Inconclusive, which is not a pass"), + el("span", statement) + ); + lastResult.paste = statement; + $("paste-result").hidden = false; + return "ECE " + fmt(measured) + " against a floor 95th percentile of " + fmt(band.p95) + + (clears ? ": it clears it." : ": inconclusive."); + } + /* ---- planner --------------------------------------------------------- */ $("plan").addEventListener("submit", function (event) { diff --git a/site/floor-golden.json b/site/floor-golden.json index 20b0296..95ebb37 100644 --- a/site/floor-golden.json +++ b/site/floor-golden.json @@ -1,5 +1,5 @@ { - "about": "Golden values from plumbline.metrics.calibration.synthetic_floor, which site/floor.js must reproduce. Written by scripts/floor_golden.py.", + "about": "Golden values from plumbline.metrics.calibration (synthetic_floor for the calculator, ece_figure for pasted rows), which site/floor.js must reproduce. Written by scripts/floor_golden.py.", "function": "synthetic_floor(n, n_bins, accuracy), all other arguments default", "defaults": { "binning": "equal_width", @@ -1371,5 +1371,47 @@ "1.84375": "1.8438", "1.90625": "1.9062", "1.96875": "1.9688" - } + }, + "pasted": [ + { + "label": "comma separated, with a header", + "n_bins": 10, + "text": "probability,outcome\n0.3701902842671547,0\n0.6636748580851755,0\n0.4854677254876745,1\n0.9809095860229241,1\n0.5402041081123293,1\n0.5878420282243031,1\n0.6429783201852302,1\n0.6914959785439542,1\n0.8429774994285216,1\n0.6053664379951705,0\n0.5505226687317148,1\n0.47873665214418765,1\n0.9186624277149896,0\n0.32830736621921897,1\n0.8046851732099684,0\n0.8983728404473525,1\n0.7176103346111447,0\n0.6845520598937217,1\n0.8754085153219575,1\n0.8274990808507586,1\n0.8165098359026878,0\n0.7370462486190534,1\n0.8136063683786394,1\n0.8120125108755948,1\n0.7847554230365376,1\n0.7890675244463915,1\n0.5442676637935917,1\n0.7571337761499041,1\n0.9021548876500897,1\n0.637880062408436,1\n0.7557549754401213,1\n0.5083763821624885,1\n0.9141689143748283,1\n0.8253786551448449,1\n0.9130937777894381,1\n0.9339506644871263,1\n0.8784293563597917,1\n0.895670527899472,1\n0.8430529077285147,0\n0.9555244758421513,1\n0.6562334841941971,0\n0.44347198079416006,0\n0.5882951048748661,0\n0.6632882208074936,1\n0.7193471554059294,1\n0.4876146079345244,1\n0.953168038895694,0\n0.7174536271438119,0\n0.7623244598883582,1\n0.9509176055447596,1\n0.8831960622867566,1\n0.7383262444068864,1\n0.6705386587124225,1\n0.6742124297478073,0\n0.7473350702698447,1\n0.5874704389192519,1\n0.744815008138343,1\n0.8734280898226684,1\n0.8297216201264507,0\n0.500944448282882,1\n0.6082688643601383,1\n0.8503070978000848,1\n0.6915965862987412,1\n0.7453015147962215,1\n0.9149539864420778,0\n0.6881209119592449,0\n0.43798699977734107,1\n0.8914830297995384,0\n0.4105129789272475,0\n0.7472626135336042,1\n0.8677678395354927,0\n0.9018897593429304,1\n0.5551233908380523,1\n0.9424249586699289,1\n0.7490780290998588,0\n0.6472160899498884,1\n0.8677347123628618,1\n0.7553091985144603,1\n0.8320194462604729,0\n0.8075970235316438,0\n0.6339841866183518,1\n0.49726004536795126,0\n0.4105371902108986,1\n0.8417525930304077,1\n0.7536828171041157,1\n0.8064127928769447,1\n0.7854594038727644,1\n0.5963890082318981,0\n0.5529445130913578,0\n0.9267909229693727,1\n0.7515970304169407,1\n0.5593384722006013,0\n0.805210866278916,1\n0.7966683402613025,1\n0.5155310158049526,0\n0.4365883968646004,0\n0.7823467361944703,1\n0.7613503500913443,1\n0.626144209182107,1\n0.6694521261435745,1\n0.5866183003387513,1\n0.6000598269757904,0\n0.7172361276561078,1\n0.5527992496181331,0\n0.8590012221015682,1\n0.535433262307073,0\n0.8682061118295181,0\n0.9357722771399521,0\n0.8119736186824921,1\n0.7986123343128968,1\n0.8107896870003097,1\n0.8956489957173016,1\n0.7528890472068774,1\n0.7820916134883324,1\n0.9180589086435903,1\n0.7024068363271042,0\n0.7178566909297954,1\n0.7987040746674912,1\n0.8422369874987513,1\n0.7990381593486006,1\n0.734168410981497,1\n0.9148037540012764,0\n0.9512786598755222,1\n0.5115165194732113,0\n0.6680665924216662,0\n0.9035603892225401,1\n0.6278994800190909,1\n0.8197891964984755,0\n0.9647400144317287,0\n0.6972562919176344,1\n0.7684046697253971,0\n0.669647327062451,1\n0.7008342235525218,0\n0.5493029774934077,1\n0.829711646861401,0\n0.672083575156045,1\n0.8280829924700488,1\n0.7404069235501143,1\n0.9165172488447596,1\n0.2282370958780986,0\n0.5386581972300077,1\n0.7457243489462131,0\n0.5392385577908511,1\n0.7282403747702962,1\n0.8815371641998792,1\n0.9317419977093483,1\n0.6428026789281851,1\n0.430810850276538,0\n0.6506688477927165,1\n0.42472556381855225,0\n0.5521720366772187,0\n0.4962777211926458,0\n0.5521532981257298,0\n0.7796066865227138,0\n0.7969958630189081,0\n0.7912668283237764,1\n0.8779876427333928,1\n0.6233181788415615,1\n0.8342941272432719,1\n0.348248971986688,0\n0.836354674601855,0\n0.7486363732096073,1\n0.9022350531555352,1\n0.6235948805828943,0\n0.7412447353591232,1\n0.9513810859673139,1\n0.9228817311905471,1\n0.64916096179032,0\n0.9456648149743587,1\n0.5061343952388129,0\n0.8887209746062371,0\n0.79197745415576,0\n0.9066895906936023,0\n0.8363569154967893,1\n0.6975598059422804,0\n0.9110310794481414,1\n0.8845099497931554,1\n0.797249804568639,1\n0.8234974141447502,1\n0.8731182067678682,1\n0.6710415703971537,0\n0.6899353665103408,1\n0.9056899904434397,1\n0.5584019550584538,0\n0.6935218256066912,0\n0.5781141365894732,1\n0.5842441620215233,0\n0.8909719408973987,1\n0.47497656651502757,0\n0.30628057890831156,0\n0.6613257539649979,1\n0.9182098689670549,1\n0.8689256027144716,1\n0.7778303880886164,1\n0.6605842495341648,1\n0.7070577979094427,1\n0.5150620613284645,0\n0.7983389970158196,1\n0.44855480238672674,0\n0.6508648099224633,1\n0.9255033777884485,1\n0.394853735533111,0\n0.7867356781046114,1\n0.5862323242027035,1\n0.742574501218065,1\n0.8233157601980059,1\n0.9776754616159198,1\n0.8308139894160536,1\n0.6312823438927516,1\n0.7495943521641198,1\n0.6330588838243157,1\n0.805864860227732,0\n0.8887378106808479,1\n0.7164518065643302,1\n0.7346928331243497,1\n0.7964994861585364,0\n0.6452092959454038,1\n0.7887232244762341,1\n0.859688093786934,1\n0.7877313919164622,0\n0.8357431010713455,1\n0.761159160822158,1\n0.8012002285350062,0\n0.8342390553977881,0\n0.37358065987559713,0\n0.8399620235697624,0\n0.6093418214146425,1\n0.869359968523917,1\n0.7429978373115638,1\n0.9857316567739101,0\n0.8990737823380992,1\n0.7658433201169123,1\n0.8388790588565389,1\n0.13923361900684222,0\n0.7750684420668394,1\n0.5948658555396185,1\n0.4688512407957305,1\n0.9500019505154772,1\n0.6554731834099224,1\n0.7468502782246492,1\n0.6133090129809938,0\n0.6471700420621249,0\n0.9015034427957023,1\n0.918475057393014,1\n0.823568284769587,1\n0.8254780969183326,1\n0.8512121810403324,1\n0.7700545197165005,1\n0.715605522111546,1\n0.8236374691025015,0\n0.505662620472717,1\n0.7442785127318075,0\n0.5024241043670978,0\n0.5323628354510287,0\n0.49575715066601667,1\n0.8668514453238375,1\n0.6242494830337679,1\n0.7071417673637219,0\n0.46186034198055237,1\n0.8056259126568305,1\n0.3525147771800355,0\n0.842252928121914,1\n0.4873924680795274,1\n0.6706392345533406,0\n0.7386804567934326,1\n0.9086960236859598,1\n0.7764830732374907,1\n0.4095110358667931,1\n0.6654872908264021,1\n0.8415880166083428,1\n0.8703776906494938,0\n0.7233215805008749,0\n0.8981798649057454,1\n0.6446564490475859,1\n0.7307904831484313,1\n0.9337474280437011,1\n0.6230267811255795,1\n0.5360793418892547,1\n0.8660855378904423,1\n0.5271394905857312,0\n0.765759728799988,1\n0.6524696445069339,1\n0.9257168343443426,1\n0.5461358606182308,0\n0.7480755178612511,1\n0.8404996243858934,0\n0.46216000140207303,0\n0.939382629974081,1\n0.5308856836821055,1\n0.4263112730497274,1\n0.6318270693469151,0\n0.76546862303657,1\n0.8357280501480471,0\n0.710861165612637,1\n0.6217916985673697,0\n0.836028146464499,1\n0.8369846237927154,0\n0.8743917960956923,1\n0.8264623961107267,0\n0.8569061872208475,1\n", + "n": 300, + "ece": 0.08256759382915613, + "mean": 0.05060766943179064, + "p95": 0.07520325441503889, + "statement": "ECE 0.0826 over 300 rows (10 equal width bins), against a calibrated-model floor of 0.0506 (95th percentile 0.0752): miscalibration is distinguishable from sampling noise." + }, + { + "label": "tab separated, CRLF line ends, true and false, a blank line", + "n_bins": 15, + "text": "0.8092251254673093\tfalse\r\n0.4810475108402493\tfalse\r\n0.6150968308958921\ttrue\r\n0.16901035542560794\tfalse\r\n0.06109131264098788\tfalse\r\n0.12262722928878389\tfalse\r\n0.260205205042753\tfalse\r\n0.15746477725918107\tfalse\r\n0.09811156131315213\tfalse\r\n0.965616171921477\ttrue\r\n0.64024646917968\tfalse\r\n0.06283530440957164\tfalse\r\n0.6232950439881982\tfalse\r\n0.8174599894568946\ttrue\r\n0.4662516651391754\ttrue\r\n0.73528215473066\tfalse\r\n0.8560957030307116\ttrue\r\n0.7809314624426905\ttrue\r\n0.37469093722066327\tfalse\r\n0.8063052210769309\ttrue\r\n0.6797309842087644\tfalse\r\n0.5441444257254635\ttrue\r\n0.6640869675530805\ttrue\r\n0.8323301563131899\ttrue\r\n0.5994946338452651\ttrue\r\n0.3427956490611287\tfalse\r\n0.5574401539739183\ttrue\r\n0.936080333049103\ttrue\r\n0.6087827358989107\ttrue\r\n0.15745125025268203\tfalse\r\n0.6360353529084468\tfalse\r\n0.6914338569413809\ttrue\r\n0.42126265304268395\ttrue\r\n0.7833959783104201\ttrue\r\n0.8705602046249002\ttrue\r\n0.507211294929106\tfalse\r\n0.21790444828803646\tfalse\r\n0.6881032519474287\tfalse\r\n0.00411233163437319\tfalse\r\n0.3502617240759307\tfalse\r\n\r\n0.9352696943166048\ttrue\r\n0.6645994624576257\tfalse\r\n0.610127098529173\tfalse\r\n0.8208248356314631\ttrue\r\n0.92366253639984\ttrue\r\n0.6730912403808006\ttrue\r\n0.51129127814052\tfalse\r\n0.6817005717995038\ttrue\r\n0.8899861184206933\ttrue\r\n0.7140194045085091\tfalse\r\n0.9498130543981782\ttrue\r\n0.34520656461313304\tfalse\r\n0.21614137962877666\tfalse\r\n0.4783647743967546\ttrue\r\n0.8843747222028043\tfalse\r\n0.4061608268263315\tfalse\r\n0.0471967793994339\tfalse\r\n0.7220830451450202\ttrue\r\n0.7312581199981368\tfalse\r\n0.15257757557359208\tfalse\r\n0.6906773861917685\ttrue\r\n0.8159109793597631\tfalse\r\n0.9388029785643608\ttrue\r\n0.5196187225053817\ttrue\r\n0.005260178281515815\tfalse\r\n0.45596229708928215\tfalse\r\n0.6982282138608734\ttrue\r\n0.685603418704395\ttrue\r\n0.9719211064669567\ttrue\r\n0.15998676726080874\tfalse\r\n0.7850514335124563\ttrue\r\n0.5287084362267819\tfalse\r\n0.5118535986122918\tfalse\r\n0.3642922284946962\tfalse\r\n0.12612145219641568\tfalse\r\n0.8856591785934856\ttrue\r\n0.3500651481128494\tfalse\r\n0.8890678391009302\ttrue\r\n0.4778501255227805\tfalse\r\n0.2885141595192058\tfalse\r\n0.18314668398887002\tfalse\r\n0.8261608151809642\tfalse\r\n0.0670082083353638\tfalse\r\n0.699881437809391\tfalse\r\n0.000734218650418339\tfalse\r\n0.4763010765379868\ttrue\r\n0.7251867150355418\ttrue\r\n0.7643861262351269\ttrue\r\n0.3689060891729379\tfalse\r\n0.8407039905274829\ttrue\r\n0.1779477506507431\tfalse\r\n0.6525856250479789\ttrue\r\n0.11093291317605714\tfalse\r\n0.08169699773110173\tfalse\r\n0.8584770844072255\ttrue\r\n0.22227571522325595\tfalse\r\n0.37222805287748184\tfalse\r\n0.04131405978541758\tfalse\r\n0.18490454741039097\tfalse\r\n0.49893622010934036\tfalse\r\n0.5308800623040114\ttrue\r\n0.2124453901395038\tfalse\r\n0.8594821883376382\ttrue\r\n0.6141412143428835\ttrue\r\n0.049753841472731763\tfalse\r\n0.1270379908640027\tfalse\r\n0.3857686994077191\tfalse\r\n0.5290646649467213\ttrue\r\n0.23426514867427006\ttrue\r\n0.9721774387851962\ttrue\r\n0.8292813126608648\ttrue\r\n0.5643581363939282\tfalse\r\n0.7279323259932716\ttrue\r\n0.1733617511617992\tfalse\r\n0.5677147297527553\ttrue\r\n0.8417937724914958\ttrue\r\n0.7109214107389936\ttrue\r\n0.03928094042693042\ttrue\r\n0.5969846402370829\tfalse\r\n0.7096180600849119\ttrue\r\n0.06537837025450433\tfalse\r\n0.6958775599796297\ttrue\r\n0.1953942156617785\tfalse\r\n0.5557920253825632\ttrue\r\n0.9364734927925796\ttrue\r\n0.13308504047817282\tfalse\r\n0.9384531847054718\ttrue\r\n0.8059143590756168\ttrue\r\n0.1933381413865768\tfalse\r\n0.6408979420254313\ttrue\r\n0.09697128926361853\ttrue\r\n0.6062934627402057\tfalse\r\n0.31638374702764194\tfalse\r\n0.24657104375987926\tfalse\r\n0.05299896777931312\tfalse\r\n0.41718266278269156\tfalse\r\n0.9078211084000235\ttrue\r\n0.5898374349703307\tfalse\r\n0.2277324778059242\tfalse\r\n0.757064035744464\ttrue\r\n0.5688307708568273\tfalse\r\n0.00454309077941728\tfalse\r\n0.17939433257263737\tfalse\r\n0.08728479124377542\tfalse\r\n0.9373635851491866\ttrue\r\n0.21206081712181002\tfalse\r\n0.29868775220491417\tfalse\r\n0.6028032693583263\ttrue\r\n0.8043238663654723\tfalse\r\n0.9659887379506182\ttrue", + "n": 150, + "ece": 0.08763532023460592, + "mean": 0.097457440489558, + "p95": 0.13177332613847564, + "statement": "ECE 0.0876 over 150 rows (15 equal width bins), against a calibrated-model floor of 0.0975 (95th percentile 0.1318): INCONCLUSIVE at this sample size. A perfectly calibrated model would often score this badly on this many rows, so this dataset cannot tell the two apart. This is not a clean bill of health: nothing was established either way. Collect more rows to make the question answerable." + }, + { + "label": "space separated, the ECE exactly on a rounding tie", + "n_bins": 10, + "text": " 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 1\n 0.75 0\n 0.75 0\n 0.75 0\n 0.75 0\n 0.75 0\n 0.75 0\n 0.75 0\n 0.75 0\n 0.75 0", + "n": 32, + "ece": 0.03125, + "mean": 0.059703125, + "p95": 0.15625, + "statement": "ECE 0.0312 over 32 rows (10 equal width bins), against a calibrated-model floor of 0.0597 (95th percentile 0.1562): INCONCLUSIVE at this sample size. A perfectly calibrated model would often score this badly on this many rows, so this dataset cannot tell the two apart. This is not a clean bill of health: nothing was established either way. Collect more rows to make the question answerable." + }, + { + "label": "semicolons and quotes, the ECE beside a tie", + "n_bins": 10, + "text": "p;\"correct\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"1\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"\n\"0.80125\";\"0\"", + "n": 400, + "ece": 0.0012499999999998623, + "mean": 0.016173749999999997, + "p95": 0.03875000000000006, + "statement": "ECE 0.0012 over 400 rows (10 equal width bins), against a calibrated-model floor of 0.0162 (95th percentile 0.0388): INCONCLUSIVE at this sample size. A perfectly calibrated model would often score this badly on this many rows, so this dataset cannot tell the two apart. This is not a clean bill of health: nothing was established either way. Collect more rows to make the question answerable." + } + ] } diff --git a/site/floor-worker.js b/site/floor-worker.js index af0cc6f..7fca2d8 100644 --- a/site/floor-worker.js +++ b/site/floor-worker.js @@ -12,6 +12,11 @@ self.onmessage = function (event) { var result; if (job.kind === "synthetic") { result = F.syntheticFloor(job.n, job.nBins, job.accuracy, { onProgress: progress }); + } else if (job.kind === "observed") { + result = { + measured: F.ece(job.probabilities, job.correct, job.nBins), + band: F.calibrationFloor(job.probabilities, job.nBins, { onProgress: progress }), + }; } else if (job.kind === "plan") { result = F.planRows(job.target, job.nBins, job.accuracy, { onProgress: progress, diff --git a/site/floor.js b/site/floor.js index aed8aa1..f6b8e11 100644 --- a/site/floor.js +++ b/site/floor.js @@ -517,6 +517,111 @@ return { rows: rows, band: band, evaluations: evaluations }; } + /* ---------------------------------------------------------- pasted rows */ + + // The most rows the page scores. Each row is redrawn 2,000 times for the + // floor, so past this the wait grows long enough that the tool itself is the + // better place to run it. + var MAX_PASTED_ROWS = 20000; + var NUMBER = /^[+-]?(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?$/; + + function unquoted(raw) { + var value = raw.trim(); + if (value.length >= 2 && value.charAt(0) === '"' && value.charAt(value.length - 1) === '"') { + value = value.slice(1, -1).trim(); + } + return value; + } + + // A plain decimal or exponent number, or null. Number() alone would accept + // "", "0x1", and "Infinity", none of which is a probability someone meant. + function strictNumber(value) { + return NUMBER.test(value) ? Number(value) : null; + } + + function outcome(value) { + var lower = value.toLowerCase(); + if (lower === "true") return true; + if (lower === "false") return false; + var number = strictNumber(value); + return number === 0 || number === 1 ? number === 1 : null; + } + + // A first line is a header when it names its columns: some field is not a + // number, not an outcome, and not a spelling of NaN or infinity, which are + // mistakes to report rather than names to skip. + function isHeader(fields) { + return fields.some(function (value) { + return strictNumber(value) === null && outcome(value) === null && + /[a-z]/i.test(value) && !/^[+-]?(nan|inf|infinity)$/i.test(value); + }); + } + + function shown(value) { + return '"' + (value.length > 24 ? value.slice(0, 24) + "..." : value) + '"'; + } + + /** + * Reads pasted predictions: one row per line, a probability and an outcome + * (1 or true when the prediction was right, 0 or false when it was not), + * separated by a comma, semicolon, tab, or spaces. A first line that names + * its columns is skipped, and so are blank lines. + * + * Returns { probabilities, correct, rows, header, errors }. Each error is + * { line, message }, with line null for a problem with the whole paste. Rows + * are for scoring only when errors is empty. + * + * options: maxRows (default 20,000). + */ + function parsePredictions(text, options) { + options = options || {}; + var maxRows = options.maxRows || MAX_PASTED_ROWS; + var lines = String(text).split(/\r\n|\r|\n/); + var probabilities = []; + var correct = []; + var errors = []; + var header = false; + var first = true; + for (var i = 0; i < lines.length; i++) { + if (lines[i].trim() === "") continue; + var fields = lines[i].trim().split(/\s*[,;\t]\s*|\s+/).map(unquoted); + if (fields.length > 1 && fields[fields.length - 1] === "") fields.pop(); // a trailing separator + if (first) { + first = false; + if (isHeader(fields)) { header = true; continue; } + } + var line = i + 1; + if (fields.length !== 2) { + errors.push({ line: line, message: "expected two values, a probability and an outcome, and found " + fields.length }); + continue; + } + var p = strictNumber(fields[0]); + var y = outcome(fields[1]); + if (p === null) { + errors.push({ line: line, message: "the probability " + shown(fields[0]) + " is not a number" }); + } else if (!(p >= 0 && p <= 1)) { + errors.push({ line: line, message: "the probability " + fields[0] + " is outside 0 to 1" }); + } else if (y === null) { + errors.push({ line: line, message: "the outcome " + shown(fields[1]) + " is not 0 or 1" }); + } else { + probabilities.push(p); + correct.push(y); + } + } + var rows = probabilities.length; + if (!errors.length && !rows) { + errors.push({ line: null, message: "there are no rows to score. Paste one row per line: a probability, then 0 or 1" }); + } + if (rows > maxRows) { + errors.push({ + line: null, + message: rows.toLocaleString("en-US") + " rows is more than this page scores (" + + maxRows.toLocaleString("en-US") + "). Run plumbline itself on a set this large", + }); + } + return { probabilities: probabilities, correct: correct, rows: rows, header: header, errors: errors }; + } + /* -------------------------------------------------------------- wording */ // Python's "%.4f": round the exact binary value, half to even. toFixed also @@ -1064,6 +1169,8 @@ judgment: judgment, isDistinguishable: isDistinguishable, fixed4: fixed4, + parsePredictions: parsePredictions, + maxPastedRows: MAX_PASTED_ROWS, // Exposed for the parity check, not for the page. _internal: { Pcg64: Pcg64, diff --git a/site/index.html b/site/index.html index 74f7fd5..a88e8dd 100644 --- a/site/index.html +++ b/site/index.html @@ -52,6 +52,7 @@

A calibration claim has a floor

  • The argument
  • Floor calculator
  • Worked example
    • +
  • Your own predictions
  • Sample-size planner
  • What computes this
    • @@ -119,7 +120,7 @@

    Calibrated-model floor

    This is an estimate from summary numbers. Your real predictions can give a different floor at the same row count; the worked example - shows why.

    + shows why, and you can score your own predictions instead.

    @@ -191,7 +192,57 @@

    The same n and accuracy, from summary numbers only

    This is why plumbline needs your actual predictions. The calculator gives - an estimate from summary numbers. The tool gives the real floor, from your data.

    + an estimate from summary numbers. The tool gives the real floor, from your data, and so does + the next section.

    + + + +
    +

    Your own predictions

    +

    Paste the predictions a model made and whether each one was right, and this computes their + ECE, the floor for exactly those predictions, and the verdict a plumbline report would print, + word for word.

    +

    Nothing you paste leaves this page. It is read and scored by + the script on this page, in your browser. It is not uploaded, not stored, and not put in the + address, so reloading the page clears it.

    + +
    +
    + + +
    +
    + + +
    +
    + + + + + +
    +
    + + +
    @@ -258,6 +309,9 @@

    What computes these numbers

    at deploy time from the seeded mock adapter by the command the report records. The deploy fails unless the derived line matches the report character for character. No vendor's output is on this page. +
  • Your own predictions go through ece_figure, the call a report makes: the + ECE, then calibration_floor on the same rows. The check holds it to the Python + on pasted text, including a value that sits exactly on a rounding tie.
  • The check is scripts/check_floor_parity.mjs; the implementation is floor.js.