feat(recalibration): a temperature per predicted label, when one temperature is the wrong shape

CHANGELOG.md

@@ -13,6 +13,15 @@ different event from one that moved because it was wrong.
 
 ### Added
 
+- A temperature per predicted label, tried only when the global fit is refused
+  as the wrong shape or stops short of the floor (#4). It uses the global fit's
+  split and verdict rule, leaves a label with under 100 fit rows as it came and
+  names it, prints its temperatures in a block of their own that says they are
+  not comparable with the global one, and names which correction to apply. On
+  the mock, from about 100 fit rows per label, it lands a per-label bias inside
+  the floor where one temperature leaves 1.2 to 3.3 times it; where one
+  temperature is the right shape its extra parameters cost 5 to 15 percent more
+  held-out ECE. `scripts/per_label_study.py` and METHODOLOGY carry the tables.
 - `--option-style letter` asks a local checkpoint its options as A, B, C and
   reads the letter tokens, so options of any length can be scored (#3). The
   default still reads each option's own token and refuses one that is several

METHODOLOGY.md

@@ -483,6 +483,73 @@ fitted on fewer rows carries uncertainty larger than the correction it claims to
 make, and it arrives looking authoritative. The report says the rule it failed
 and gives no temperature.
 
+## A temperature per predicted label, when one temperature is the wrong shape
+
+A model can be well calibrated on most labels and overconfident on one. One
+temperature cannot reach that: flattening enough for the skewed label
+over-flattens the honest ones, and the global fit's verdict says so, either as a
+refusal because the change was no larger than chance or as a partial fit whose
+residual stays above the floor. After exactly those two verdicts, and never
+otherwise, the report tries the smallest correction that can reach it: one
+temperature per predicted label.
+
+It is keyed on the label the model predicted, because that is all a caller knows
+when the correction is applied. Each label's temperature is fitted on the fit
+rows the model predicted that label for, in the same form as the global fit, and
+every held-out row is scaled by its own label's temperature. Scaling never
+changes which label is on top, so a row stays in its label's group. The split is
+the global fit's, so both are judged on the same held-out rows, and the verdict
+is the same rule: inside the floor is recommended, a material improvement that
+stops short is partial, and anything else is refused with no temperature
+printed. The report prints the per-label temperatures in a block of their own,
+says they are a different correction and not comparable with the global
+temperature, and names which of the two to apply.
+
+A label with fewer than 100 fit rows is not fitted. It keeps its probabilities as
+they came, and the report names it with its counts. A temperature fitted on a
+handful of rows is the noise of those rows, and a per-label fit has one of those
+per label.
+
+### When it helps, and what it costs
+
+`scripts/per_label_study.py` measures both on the seeded mock (accuracy 0.75,
+four options, five seeds). Each cell is post-scaling ECE on the held-out half
+over the floor's 95th percentile, so 1.0 or below is inside the floor, with how
+often each verdict would ship a correction:
+
+| rows | fit rows per label | one temperature | per label | per label ships |
+|---|---|---|---|---|
+| 500 | 49 | 1.05x | not fitted | 0% |
+| 1,000 | 103 | 1.22x | 0.77x | 60% |
+| 2,000 | 216 | 1.93x | 0.83x | 100% |
+| 4,000 | 461 | 2.09x | 0.86x | 100% |
+| 8,000 | 952 | 3.30x | 0.67x | 100% |
+
+That is a per-label bias: the model sharpened by a temperature of 0.45 whenever
+it says billing, and honest otherwise. From about 100 fit rows per label the
+per-label fit lands inside the floor, while the global fit falls further behind
+as rows are added, the same pattern as the top-line penalty above: a fixed bias
+that a shrinking floor exposes.
+
+The cost is measured where one temperature is the right shape: every answer
+sharpened by 0.5. There the four extra parameters can only add noise, and they
+leave 5 to 15 percent more ECE on the held-out rows than one temperature does
+(0.77x against 0.72x at 1,000 rows, 0.67x against 0.58x at 8,000), all inside the
+floor. The report never pays that cost, since it tries per-label only after one
+temperature was diagnosed as the wrong shape.
+
+The row gate is set more strictly than the mock requires. At 500 rows with the
+gate lowered to 40, the per-label fit still reached 0.69x, and the verdict
+shipped it on only two of five seeds. But the mock is per-label scaling's best
+case, since its distortion is exactly a per-label temperature and a real model's
+is not. So the default waits for 100 rows a label, where the result above is not
+in doubt.
+
+What it does not do: vector or matrix scaling, which fit more parameters than
+most datasets here can support; a correction keyed on the gold label, which no
+caller knows at inference time; or feeding the per-label temperatures into the
+cascade, which still scores on the global temperature when one was emitted.
+
 ## The binary Brier formulation
 
 The Brier score plumbline reports by default is

README.md

@@ -350,9 +350,10 @@ Specific, and none of them are going to surprise you later.
 - **One request per case, no batching.** Cost and latency figures are therefore
   conservative relative to batched use, where a single call carrying many
   questions against one shared state is materially cheaper and faster.
-- **Temperature scaling only.** Per-label and vector scaling are not fitted. When
-  the residual says temperature is the wrong correction, the tool refuses and
-  emits no temperature rather than returning one that does not fit.
+- **Temperature scaling, globally or per predicted label.** When one
+  temperature is the wrong shape, the report tries one per predicted label, and
+  refuses both rather than return a correction that does not fit. Vector and
+  matrix scaling are not fitted.
 - **Recalibration needs 200 held-out rows.** Below that it refuses. Most datasets
   people try first will not reach it.
 - **Cost requires a pricing table you supply.** plumbline ships no figures for

docs/PLAN.md

@@ -68,12 +68,13 @@ All three items are done. Kept here because the answers matter, not the list.
   packing many questions against one shared state in a single call, which is a
   materially different cost and latency profile and is the single largest
   measurement gap in v0.1.
-- **Per-label and vector scaling.** Only temperature is fitted. When the
-  residual says temperature is the wrong correction the tool refuses, which is
-  right but leaves the user with nothing to apply.
-- **Adapters do not receive `label_descriptions`**, so a dataset's per-option
-  criteria never reach the wire. A Choice takes them directly and a Noul takes
-  true/false descriptions.
+- **Per-label scaling.** Landed (#4): after a global fit refused as the wrong
+  shape or stopped short of the floor, a temperature per predicted label, on the
+  same split and verdict rule, with a 100-row gate per label. Vector and matrix
+  scaling stay out until per-label proves insufficient on real data.
+- **Option descriptions on the wire.** Landed in v0.1.1 (#39): `typesafe_wire`
+  sends a Choice's descriptions as its criteria, they join the cache key and the
+  dataset hash, and the report says when an adapter does not send them.
 
 ## Decisions
 

scripts/per_label_study.py

@@ -0,0 +1,110 @@
+"""When a temperature per predicted label helps, and what it costs (issue #4).
+
+Two cases on the seeded mock (accuracy 0.75, four options), at several sizes,
+over seeds. Every figure is on the held-out half, as the report's are.
+
+- A per-label bias: the model is sharpened (temperature 0.45) whenever it says
+  "billing" and honest otherwise. One temperature cannot reach this; the
+  question is how many rows a label needs before its own temperature can.
+- A global skew: every answer sharpened by the same temperature (0.5). Here one
+  temperature is the right shape, so any held-out loss from fitting four is the
+  price of the extra parameters, which is what overfitting means here.
+
+Each cell is post-scaling ECE over the floor's 95th percentile, averaged over
+seeds (at or below 1.0 is inside the floor), with how often each fit's verdict
+would ship a correction.
+
+    uv run python scripts/per_label_study.py [seeds]
+
+METHODOLOGY quotes the tables this prints at 5 seeds.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+import sys
+
+import numpy as np
+
+from plumbline.adapters.mock import MockAdapter
+from plumbline.metrics.recalibration import recalibrate, recalibrate_per_label
+from plumbline.runner import execute
+from plumbline.types import Case, Prediction, ProbabilitySeries, apply_temperature
+
+LABELS = ("billing", "returns", "shipping", "other")
+SEEDS = int(sys.argv[1]) if len(sys.argv) > 1 else 5
+SIZES = (500, 1000, 2000, 4000, 8000)
+BOOT = {"n_boot_ci": 100, "n_boot_floor": 300}
+
+
+def rows(n: int, seed: int) -> list[Case]:
+    rng = np.random.default_rng(seed)
+    return [
+        Case(id=f"r{i}", text=f"row {seed} {i}", labels=LABELS, gold_label=LABELS[g])
+        for i, g in enumerate(rng.integers(0, len(LABELS), size=n))
+    ]
+
+
+def sharpen(prediction: Prediction, temperature: float) -> Prediction:
+    assert prediction.distribution is not None
+    scaled = apply_temperature(prediction.distribution, temperature)
+    return dataclasses.replace(
+        prediction, distribution=scaled, prob_selected=scaled[prediction.label]
+    )
+
+
+def measured(n: int, seed: int, case: str) -> tuple[list[Prediction], list[bool], list[str]]:
+    cases = rows(n, seed)
+    adapter = MockAdapter(
+        {c.text: c.gold_label for c in cases},
+        accuracy=0.75,
+        calibration_temperature=0.5 if case == "global" else 1.0,
+        seed=seed,
+    )
+    result = execute.run(adapter, cases, workers=1)
+    predictions = [r.prediction for r in result.records if r.prediction is not None]
+    if case == "per-label":
+        predictions = [sharpen(p, 0.45) if p.label == "billing" else p for p in predictions]
+    gold = [r.gold_label for r in result.records if r.prediction is not None]
+    return predictions, [p.label == g for p, g in zip(predictions, gold, strict=True)], gold
+
+
+def study(n: int, seed: int, case: str) -> tuple[float, float, bool, bool, int]:
+    predictions, correct, gold = measured(n, seed, case)
+    series = ProbabilitySeries(
+        values=tuple(p.prob_selected or 0.0 for p in predictions), semantics="calibrated_claim"
+    )
+    common = {"distributions": [p.distribution for p in predictions], "gold_labels": gold}
+    one = recalibrate(series, correct, seed=seed, **common, **BOOT)
+    many = recalibrate_per_label(
+        series,
+        correct,
+        predicted_labels=[p.label for p in predictions],
+        seed=seed,
+        **common,
+        **BOOT,
+    )
+    smallest = min(entry.n_fit for entry in many.labels)
+    return (
+        one.after.ece / one.floor["ece"].p95,
+        many.after.ece / many.floor["ece"].p95,
+        one.recommendation != "refused",
+        many.recommendation != "refused",
+        smallest,
+    )
+
+
+for case in ("per-label", "global"):
+    print(f"\n{case} skew")
+    header = ("rows", 6), ("label rows", 10), ("one T", 7), ("per label", 10)
+    print(" ".join(f"{name:>{width}}" for name, width in header), f"{'ships 1':>8} {'ships K':>8}")
+    for n in SIZES:
+        results = [study(n, seed, case) for seed in range(SEEDS)]
+        one, many, ship_one, ship_many, smallest = (
+            np.array(column) for column in zip(*results, strict=True)
+        )
+        print(
+            f"{n:>6} {int(smallest.min()):>10} {one.mean():>6.2f}x {many.mean():>9.2f}x "
+            f"{ship_one.mean():>8.0%} {ship_many.mean():>8.0%}",
+            flush=True,
+        )