feat: score ordinal rows by rank, each figure against its own null

CHANGELOG.md

@@ -13,6 +13,19 @@ different event from one that moved because it was wrong.
 
 ### Added
 
+- Ordinal score rows are scored (#6). Each is read by three rank-aware
+  figures, in a score block of its own and never beside a choice figure: mean
+  absolute error of the expected score in levels, against a permutation null;
+  the ranked probability score; and a cumulative calibration error, the
+  predicted probability of being at or below each threshold against how often
+  it was, pooled and binned as ECE is. The last two are read against a
+  calibrated-model floor, built by redrawing each row's level from its own
+  distribution. `typesafe_wire` asks a real Score with the row's rubric, which
+  the JevBench loader now keeps as the levels' descriptions; the other arms
+  answer the levels as options and say so. A score row whose options are not
+  integer levels is refused. On the public fixture the six score rows now run,
+  so its report covers 111 rows, with the choice figures unchanged over 105.
+  METHODOLOGY's "Ordinal score questions are scored by rank" is the design.
 - A temperature per predicted label, tried only when the global fit is refused
   as the wrong shape or stops short of the floor (#4). It uses the global fit's
   split and verdict rule, leaves a label with under 100 fit rows as it came and

METHODOLOGY.md

@@ -186,15 +186,73 @@ Confidence on a noul is not reported, and the report says so in those words. A
 blank cell would suggest the vendor failed to send something; the statistic does
 not exist for an answer with no distribution.
 
-## Ordinal score questions are not scored in v0.1
+## Ordinal score questions are scored by rank
 
 Some datasets ask for a level rather than a label: 0, 1, 2, or 3 daily-rest
-violations. The levels are ordered, and every metric here is rank-blind: being
-wrong by one level and wrong by three score identically. Flattening the levels
-into unordered options would discard exactly the structure that makes the
-question a score, so plumbline loads those rows, marks them, and leaves them out
-of every figure. The load report and the run report both say how many were held
-back. Ordinal support is a v0.2 question, not a formatting one.
+violations. The levels are ordered, and every choice metric here is rank-blind:
+being wrong by one level and wrong by three score identically, and a
+distribution piled on the levels beside the answer is treated no differently
+from one spread to both ends. So a score row is never scored as a choice. It is
+asked as a score, answered as a distribution over its levels, and read by three
+figures of its own, in a block of its own, never beside a choice figure.
+
+### What an answer is
+
+A score answer is a probability for each level, from 0 to K minus 1. Its point
+answer is the expected score, the probability-weighted mean of the levels,
+which can fall between two of them. That is the answer a score vendor returns
+and the one a caller would act on, so it is read as given, never replaced by the
+most probable level. An arm that returns a single level and no distribution has
+that level as its expected score, and only the first figure below applies to it.
+
+### The three figures
+
+**Mean absolute error of the expected score, in levels.** The rank-aware
+accuracy: an answer one level off costs 1, three levels off costs 3. Lower is
+better. It is read against a permutation null: the arm's own expected scores
+shuffled across the rows, which keeps how the arm answers and breaks the link to
+what each row's level was. An arm clears the null only when its answers carry
+information about the gold level. A uniform guess would be the wrong null here,
+since an arm that always answered the middle level would beat it knowing
+nothing.
+
+**Ranked probability score.** For each threshold between two adjacent levels,
+the squared gap between the predicted probability that the answer is at or below
+it and whether it was, summed over the thresholds and divided by their number,
+then averaged over rows. It is the ordinal counterpart of the Brier score and a
+proper scoring rule: it is best in expectation when the probabilities are the
+true ones, and it charges mass by its distance from the answer, so a near miss
+costs less than a far one. Zero is perfect.
+
+**Cumulative calibration error.** What calibration means for an ordered
+distribution. A distribution is calibrated when, at every threshold, the
+predicted probability that the level is at or below it matches how often it is.
+Each row contributes one event per threshold, the predicted cumulative
+probability against whether the level was at or below the threshold, and the
+events are pooled and read exactly as ECE reads a choice column: ten equal-width
+bins and the count-weighted mean gap. The probability of the single most likely
+level is not used, because it ignores order: it cannot tell an answer spread
+over neighbouring levels from one split between the two ends.
+
+### The floors
+
+The ranked probability score and the cumulative calibration error are read
+against a calibrated-model floor built the way every floor here is built. The
+predicted distributions are held fixed and the gold level of each row is redrawn
+from its own distribution, two thousand times, so every resample is calibrated
+by construction. The spread of each figure across the resamples is the noise
+floor for this exact set of distributions at this row count. A figure above the
+floor's 95th percentile is distinguishable from sampling noise; one inside it is
+INCONCLUSIVE, with the same wording as every other figure, because nothing was
+established either way.
+
+### What is not done for scores
+
+Recalibration and the cascade are choice-question tools and are not applied to
+score rows. A temperature fitted to a distribution over ordered levels is a
+different correction, and a threshold on an expected score is a different
+decision; each would need its own design. The score block says so. Score rows
+are also left out of every choice figure, as they always were.
 
 ## A row that cannot be scored is refused, not scored
 

README.md

@@ -98,7 +98,7 @@ calibrated one both land there on too few rows, and the figure does not say
 which you have. Taking it as a clean bill of health inverts the conclusion, and
 it is the easiest mistake to make with this tool.
 
-On that same 105-row run, three of the four headline figures came back
+On that same run's 105 choice and yes/no rows, three of the four headline figures came back
 inconclusive and only accuracy cleared its null. A tool that printed the other
 three alone would be handing you numbers that look like findings and are not.
 That refusal is the product.
@@ -161,7 +161,7 @@ Expected output shape:
 
 ```
 111 rows read from datasets/public/jevbench-hard.jsonl, 111 loaded, 0 refused. ...
-artifact: results/20260921T222053+0000-mock-1b96dc91.json
+artifact: results/20260925T231349+0000-mock-420956a9.json
 report: results/report.md
 ```
 
@@ -343,10 +343,10 @@ groups side by side.
 
 Specific, and none of them are going to surprise you later.
 
-- **Choice and Noul only.** Ordinal Score rows load, are marked, and are excluded
-  from every figure. Flattening ordered levels into unordered options discards
-  the ordering that makes them a score, so v0.1 declines rather than
-  approximating.
+- **Score rows get three rank-aware figures and nothing else.** An ordinal level
+  is read by mean absolute error, the ranked probability score, and a cumulative
+  calibration error, each against its own null, in a block apart from the
+  choice figures. Recalibration and the cascade are not applied to them.
 - **One request per case, no batching.** Cost and latency figures are therefore
   conservative relative to batched use, where a single call carrying many
   questions against one shared state is materially cheaper and faster.

datasets/public/README.md

@@ -31,8 +31,9 @@ prompts differ, and the scoring differs:
   by a transport with no Noul. Those are different questions, so every record
   carries both what the row asks and how it was asked, and the report keeps them
   apart rather than averaging across the difference. The six `score` rows are
-  loaded, marked, and excluded from every figure: v0.1 has no ordinal support,
-  and flattening ordered levels into unordered options discards the ordering.
+  read by rank, in a block apart from the choice figures, and their rubric is
+  kept as the levels' descriptions; flattening ordered levels into unordered
+  options would discard the ordering.
 - JevBench's score combines intelligence, calibration, speed, and cost into one
   number. plumbline computes its own metrics, against its own calibrated-null
   floor, and deliberately publishes no combined score.

docs/PLAN.md

@@ -60,9 +60,11 @@ All three items are done. Kept here because the answers matter, not the list.
 
 ## v0.2
 
-- **Ordinal score questions.** The six score rows in the public fixture are
-  loaded, marked and excluded; scoring them needs rank-aware metrics, because
-  every metric here treats wrong-by-one and wrong-by-three identically.
+- **Ordinal score questions.** Landed (#6): mean absolute error against a
+  permutation null, the ranked probability score, and a cumulative calibration
+  error, each against a calibrated-model floor, in a score block of their own.
+  `typesafe_wire` asks a real Score with the row's rubric. Recalibration and the
+  cascade for scores are not designed yet.
 - **Batching.** One request per case today, so cost and latency are both
   conservative relative to batched use. The vendor's own documentation describes
   packing many questions against one shared state in a single call, which is a
@@ -105,7 +107,8 @@ Settled during the build. Reopen one only with a reason, not from scratch.
 - A yes/no row is asked as a Noul where the transport has one, and every record
   carries both what the row asks and how it was asked. A noul figure is never
   compared with a two-option-choice figure without that line between them.
-- Ordinal score rows are loaded, marked and excluded from every figure in v0.1.
+- Ordinal score rows are read by rank in a block of their own and left out of
+  every choice figure; before v0.2 they were excluded from every figure.
 - Artifacts never overwrite each other: the timestamp is only accurate to the
   second, so a repeated name gets a suffix rather than replacing user records.
 - A refused recalibration prints no number: the verdict, the split sizes, and

docs/datasets.md

@@ -26,9 +26,11 @@ Blank lines are skipped. Every other line is one case.
   `true`/`false`), because which option is the yes is a fact about your data,
   not something to guess. A transport that has a native yes/no question asks it
   as one, and gets back one probability rather than a distribution.
-- `score`: an ordinal level, such as 0 to 3. v0.1 loads these rows and excludes
-  them from every figure, because ordering is lost if levels are scored as
-  unordered options. The load summary says how many there were.
+- `score`: an ordinal level. Its `labels` are its levels, the integers 0 to
+  K minus 1 as strings, and a row whose options are not is refused. It is read
+  by rank, by three figures of its own, and never by the choice figures, which
+  would score wrong by one level and wrong by three the same. A level's
+  `label_descriptions` entry is its rubric, which a score transport sends.
 
 ## An example
 

docs/example-report.md

@@ -48,7 +48,7 @@ Running the same command against a real vendor produces the same shape. See
 
 # plumbline report
 
-Generated 2026-09-23 against dataset `1b96dc91`, 105 rows. 1 arm(s).
+Generated 2026-09-25 against dataset `420956a9`, 111 rows. 1 arm(s).
 
 ## How to read this
 
@@ -59,8 +59,7 @@ Generated 2026-09-23 against dataset `1b96dc91`, 105 rows. 1 arm(s).
 
 ## Dataset
 
-- 111 rows read from datasets\public\jevbench-hard.jsonl, 111 loaded, 0 refused. Translated from JevBench: the case text is the row's question above its state, and each row is asked as the question type it states. plumbline's harness, prompts and scoring differ from JevBench's, so these numbers are not comparable with theirs. 6 rows wrote the gold label as a JSON number against string options; each was matched to the option of the same name. 38 rows carried criteria that do not describe the options one for one, so their option descriptions were dropped rather than guessed. 6 rows ask for an ordinal score. plumbline v0.1 has no ordinal support: flattening levels into unordered options discards the ordering, so they are loaded, marked, and excluded from scored results.
-- 6 score rows are excluded from every figure below: plumbline v0.1 scores choice and yes/no questions only.
+- 111 rows read from datasets\public\jevbench-hard.jsonl, 111 loaded, 0 refused. Translated from JevBench: the case text is the row's question above its state, and each row is asked as the question type it states. plumbline's harness, prompts and scoring differ from JevBench's, so these numbers are not comparable with theirs. 6 rows wrote the gold label as a JSON number against string options; each was matched to the option of the same name. 38 rows carried criteria that do not describe the options one for one, so their option descriptions were dropped rather than guessed. 6 rows ask for an ordinal score. They are scored by rank, in a block of their own, and left out of every choice figure, since a rank-blind figure scores wrong by one and wrong by three the same.
 
 
 ---
@@ -72,15 +71,16 @@ The vendor asserts these probabilities are calibrated. Whether that survives con
 ### mock
 
 - **Model**: requested `mock-1`, reported `mock-1`.
-- **Option descriptions**: 67 rows carried option descriptions, and this adapter does not send them, so they had no effect on its answers.
-- **Asked**: 67 choice rows asked as choice, 38 noul rows asked as choice.
+- **Option descriptions**: 73 rows carried option descriptions, and this adapter does not send them, so they had no effect on its answers.
+- **Asked**: 67 choice rows asked as choice, 38 noul rows asked as choice, 6 score rows asked as choice.
   - 38 noul rows were asked as choice questions, which is a different question from the one the dataset states. Not comparable with an arm that asked them as noul.
+  - 6 score rows were asked as choice questions, which is a different question from the one the dataset states. Not comparable with an arm that asked them as score.
 - Accuracy 0.7714 over 105 rows, against a chance null of 0.3416 (95th percentile 0.4190): better than chance at this sample size.
 - ECE 0.0740 over 105 rows (10 equal width bins), against a calibrated-model floor of 0.0707 (95th percentile 0.1109): INCONCLUSIVE at this sample size. A perfectly calibrated model would often score this badly on this many rows, so this dataset cannot tell the two apart. This is not a clean bill of health: nothing was established either way. Collect more rows to make the question answerable.
 - Brier 0.1711 over 105 rows, against a calibrated-model floor of 0.1489 (95th percentile 0.1839): INCONCLUSIVE at this sample size. A perfectly calibrated model would often score this badly on this many rows, so this dataset cannot tell the two apart. This is not a clean bill of health: nothing was established either way. Collect more rows to make the question answerable.
 - **Confidence**: AUROC 0.6034 over 105 rows, against a permutation null of 0.4997 (95th percentile 0.6116): INCONCLUSIVE at this sample size. Permutation would often score this well on this many rows, so this dataset cannot tell the two apart. This is not a result in either direction. Collect more rows to make the question answerable.
-- **Cost**: not reported. No cost available. None of the 105 cases could be priced, so cost is not reported rather than being shown as zero. 105 rows: tokens were reported, but the model that answered is not priced.
-- **Latency**: p50 37.7ms, p95 85.0ms, p99 104.7ms over 105 live calls
+- **Cost**: not reported. No cost available. None of the 111 cases could be priced, so cost is not reported rather than being shown as zero. 111 rows: tokens were reported, but the model that answered is not priced.
+- **Latency**: p50 36.6ms, p95 85.0ms, p99 104.7ms over 111 live calls
 
 #### Recalibration
 
@@ -96,3 +96,11 @@ Read these only after the figures above. MCE is a maximum over bins, decided by
 
 - MCE 0.1588 over 105 rows (10 equal width bins), against a calibrated-model floor of 0.1251 (95th percentile 0.2198): INCONCLUSIVE at this sample size. A perfectly calibrated model would often score this badly on this many rows, so this dataset cannot tell the two apart. This is not a clean bill of health: nothing was established either way. Collect more rows to make the question answerable.
 - Multiclass Brier 0.3526 over 105 rows, against a calibrated-model floor of 0.3168 (95th percentile 0.3975): INCONCLUSIVE at this sample size. A perfectly calibrated model would often score this badly on this many rows, so this dataset cannot tell the two apart. This is not a clean bill of health: nothing was established either way. Collect more rows to make the question answerable.
+
+#### Score questions
+
+- 6 rows ask for an ordinal level and are read by rank, by the figures below and by none of the figures above. They are not comparable with a choice figure.
+- Mean absolute error 0.6746 levels over 6 rows, against a permutation null of 0.8199 (5th percentile 0.4664): INCONCLUSIVE at this sample size. The same answers shuffled across the rows would often score this well, so this dataset cannot tell whether they carry information about the level. This is not a result in either direction. Collect more rows to make the question answerable.
+- Ranked probability score 0.1577 over 6 rows, against a calibrated-model floor of 0.0829 (95th percentile 0.1930): INCONCLUSIVE at this sample size. A perfectly calibrated model would often score this badly on this many rows, so this dataset cannot tell the two apart. This is not a clean bill of health: nothing was established either way. Collect more rows to make the question answerable.
+- Cumulative calibration error 0.1796 over 6 rows (19 threshold events, 10 equal width bins), against a calibrated-model floor of 0.1420 (95th percentile 0.2323): INCONCLUSIVE at this sample size. A perfectly calibrated model would often score this badly on this many rows, so this dataset cannot tell the two apart. This is not a clean bill of health: nothing was established either way. Collect more rows to make the question answerable.
+- Recalibration and the cascade are not applied to score rows: each would be a different correction or decision on an ordered answer, and neither is designed yet.

site/index.html

@@ -125,7 +125,7 @@ <h3>Calibrated-model floor</h3>
   </section>
 
   <section id="example" aria-labelledby="example-h">
-    <h2 id="example-h">Worked example: the 105-row report</h2>
+    <h2 id="example-h">Worked example: the report's ECE line</h2>
     <p>The repository's <a href="docs/example-report.html">example report</a>
     prints this line:</p>
     <blockquote id="ex-report-line"><p class="muted">Loading the example...</p></blockquote>