feat(local): ask a local checkpoint its options by letter

CHANGELOG.md

@@ -11,6 +11,18 @@ different event from one that moved because it was wrong.
 
 ## Unreleased
 
+### Added
+
+- `--option-style letter` asks a local checkpoint its options as A, B, C and
+  reads the letter tokens, so options of any length can be scored (#3). The
+  default still reads each option's own token and refuses one that is several
+  tokens. On the public fixture the default scores 39 of 105 rows; by letter,
+  all 105, at chance accuracy with an ECE of 0.349 against a floor of 0.087.
+  The style is part of the cache key only when it is `letter`, so existing
+  entries keep their keys, and the artifact and the report record it.
+  METHODOLOGY says why sequence probability was not used instead: longer
+  options would lose probability for being long.
+
 ## v0.1.1 (2026-09-25)
 
 The review release: every issue the repository review filed is fixed, and the

METHODOLOGY.md

@@ -425,6 +425,26 @@ moves, without anything about the case having changed. It can behave like a
 calibrated probability on a given workload, and whether it does is measurable,
 which is why plumbline measures it rather than assuming either way.
 
+The local arm reads that softmax in one of two ways, and the artifact and the
+report say which. By default it reads each option's own token, which means every
+option must be a single token for the checkpoint: an option that tokenizes into
+several pieces is refused by name, because a softmax over first tokens answers a
+different question than the dataset asks. Options written as identifiers, such
+as `pay_subject_to_10000_sublimit`, are almost never one token, so on a dataset
+of them this reading scores little beyond the yes/no rows.
+
+`--option-style letter` asks the same options by letter instead: the prompt lists
+them as A, B, C, and the softmax is over the letter tokens, each of which is one
+token. That scores options of any length and is the usual way multiple-choice
+questions are put to an open model. It is a different question, though, and is
+labelled as one. The letters bring position into the answer, which is why the
+cache keys the option order for this arm, and the numbers are conditional on the
+lettered option set exactly as the plain reading is on the words. Scoring each
+option by the probability of its whole token sequence was the other candidate,
+and was not taken: longer options would lose probability for being long, and
+every way of correcting for length is a choice the result would silently depend
+on.
+
 Citations, kept separate on purpose. SemIf is an independent project and says so:
 "not affiliated with or endorsed by TypeSafe". fastjev is an independently
 maintained fork of SemIf that preserves its history and MIT license, follows its

README.md

@@ -201,10 +201,16 @@ uv run plumbline run datasets/public/jevbench-hard.jsonl --format jevbench \
 
 `--device` defaults to `cpu`, which works anywhere and is slow. On Windows the
 torch that PyPI serves is CPU only; for `--device cuda`, install a CUDA build
-of torch from the PyTorch index into the same environment. The arm scores only
-options that are a single token for the checkpoint and refuses the rest by
-name, so on the public fixture, whose choice options are mostly multi-word
-identifiers, most choice rows are refused and the yes/no rows carry the run.
+of torch from the PyTorch index into the same environment.
+
+By default the arm reads each option's own token, so it scores only options
+that are a single token for the checkpoint and refuses the rest by name. The
+public fixture's choice options are mostly multi-word identifiers, so that way
+66 of its 67 choice rows are refused. Add `--option-style letter` to ask the
+options as A, B, C and read the letters instead: every row is scored, and the
+report says the question was lettered. On the fixture that run landed at chance
+accuracy with an ECE of 0.349 against a floor of 0.087: a small model that is
+confidently wrong, which is the case a calibration figure exists to catch.
 
 ### Running a hosted vendor
 
@@ -313,7 +319,7 @@ and self-hosted servers speak it, which is why one adapter covers all of them.
 | Adapter | Transport | Semantics | Run for real | Adding one |
 |---|---|---|---|---|
 | `typesafe_wire` | Jev wire format over HTTP | `calibrated_claim` | Yes, 40 rows against a hosted vendor | `--base-url`. Anything serving the same wire format is one flag, recorded in the artifact, including self-hosted endpoints and open models behind a compatible server. |
-| `local_logits` | Option-token logits from a local checkpoint | `restricted_softmax` | Yes, 105 rows against a pinned Qwen2.5-1.5B-Instruct on a GPU; 39 scored, 66 refused as multi-token | A HuggingFace model id and a pinned revision. Needs the optional `local` extra. |
+| `local_logits` | Option-token logits from a local checkpoint | `restricted_softmax` | Yes, 105 rows against a pinned Qwen2.5-1.5B-Instruct on a GPU; 39 scored by label, all 105 by letter | A HuggingFace model id and a pinned revision. Needs the optional `local` extra. |
 | `generative` | Chat completion, parsed | `none` | **No, tests only** | A model string. |
 | `mock` | None, seeded | configurable | Yes, it is the example report | Built in. A deterministic stand-in, not a system under test. |
 

src/plumbline/adapters/local_logits.py

@@ -34,13 +34,26 @@
 import threading
 import time
 from collections.abc import Mapping, Sequence
-from typing import Protocol
+from typing import Literal, Protocol, get_args
 
 from plumbline.adapters.base import Adapter
 from plumbline.types import CaseRefusedError, PlumblineError, Prediction, QuestionType
 
 DEFAULT_INSTRUCTIONS = "Which label best describes this text?"
 
+#: How the options are put to the checkpoint. ``label`` reads each option's own
+#: token, so every option must be a single token. ``letter`` lists the options
+#: as A, B, C... and reads the letters, so an option can be any text.
+OptionStyle = Literal["label", "letter"]
+OPTION_STYLES: tuple[OptionStyle, ...] = get_args(OptionStyle)
+
+#: The default question when options are asked by letter.
+DEFAULT_LETTER_INSTRUCTIONS = (
+    "Which option best describes this text? Answer with the letter of the option."
+)
+
+_LETTERS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+
 #: The one prompt shape this adapter sends. It is in ``call_params``, so editing
 #: it invalidates the cache rather than mixing two prompt shapes in one table.
 DEFAULT_PROMPT_TEMPLATE = "{instructions}\n\nText: {text}\n\nOptions: {options}\n\nAnswer:"
@@ -98,6 +111,11 @@ class LocalLogitsAdapter(Adapter):
             cannot be reproduced from the artifact alone.
         device: Passed to the default readout. Not part of ``call_params``: it
             moves arithmetic, not the question.
+        option_style: ``label`` reads each option's own token and refuses a
+            case whose options are not single tokens. ``letter`` lists the
+            options as A, B, C... and reads the letter tokens, so options of
+            any length can be asked, at the price of asking a lettered
+            question; the style is part of ``call_params``.
 
     ``probability_semantics`` is fixed at ``"restricted_softmax"`` and is not a
     constructor argument. There is no configuration under which this arm becomes
@@ -112,12 +130,17 @@ def __init__(
         model_requested: str,
         revision: str,
         readout: LogitReadout | None = None,
-        instructions: str = DEFAULT_INSTRUCTIONS,
+        instructions: str | None = None,
         prompt_template: str = DEFAULT_PROMPT_TEMPLATE,
         option_prefix: str = DEFAULT_OPTION_PREFIX,
         allow_unpinned_revision: bool = False,
         device: str = "cpu",
+        option_style: OptionStyle = "label",
     ) -> None:
+        if option_style not in OPTION_STYLES:
+            raise ValueError(
+                f"option_style must be one of {list(OPTION_STYLES)!r}, got {option_style!r}"
+            )
         if not revision or not revision.strip():
             raise ValueError(
                 "local_logits requires a revision. An unpinned checkpoint makes a result "
@@ -140,7 +163,10 @@ def __init__(
         self.pinned_revision = revision.strip()
         self.revision = self.pinned_revision
         self.probability_semantics = "restricted_softmax"
-        self.instructions = instructions
+        self.option_style: OptionStyle = option_style
+        self.instructions = instructions or (
+            DEFAULT_LETTER_INSTRUCTIONS if option_style == "letter" else DEFAULT_INSTRUCTIONS
+        )
         self.prompt_template = prompt_template
         self.option_prefix = option_prefix
         self.allow_unpinned_revision = allow_unpinned_revision
@@ -173,11 +199,16 @@ def call_params(self) -> Mapping[str, object]:
         The base key already covers the adapter name, the model, the revision,
         the case text, and the sorted labels. The prompt shape is added here.
         """
-        return {
+        params: dict[str, object] = {
             "instructions": self.instructions,
             "prompt_template": self.prompt_template,
             "option_prefix": self.option_prefix,
         }
+        # Added only when it differs from the original question, so every
+        # cache entry written before letters existed keeps its key.
+        if self.option_style != "label":
+            params["option_style"] = self.option_style
+        return params
 
     #: The prompt lists the options in the order given.
     label_order_matters = True
@@ -200,12 +231,18 @@ def classify(
         if len(labels) < 2:
             raise ValueError(f"need at least 2 labels, got {len(labels)}")
 
+        letters = self._letters(labels)
         readout = self.readout
-        token_ids = self._single_token_ids(readout, labels)
+        token_ids = self._single_token_ids(readout, letters or labels)
         self._check_revision(readout)
 
+        options = (
+            ", ".join(f"{letter}. {label}" for letter, label in zip(letters, labels, strict=True))
+            if letters
+            else ", ".join(labels)
+        )
         prompt = self.prompt_template.format(
-            instructions=self.instructions, text=text, options=", ".join(labels)
+            instructions=self.instructions, text=text, options=options
         )
 
         with self._forward_lock:
@@ -244,9 +281,22 @@ def classify(
                 "option_token_ids": dict(zip(labels, token_ids, strict=True)),
                 "option_logits": dict(zip(labels, logits, strict=True)),
                 "probability_semantics": self.probability_semantics,
+                "option_style": self.option_style,
+                **({"option_letters": dict(zip(labels, letters, strict=True))} if letters else {}),
             },
         )
 
+    def _letters(self, labels: list[str]) -> list[str]:
+        """The letter standing for each option, or none when options are read as themselves."""
+        if self.option_style != "letter":
+            return []
+        if len(labels) > len(_LETTERS):
+            raise CaseRefusedError(
+                f"this case has {len(labels)} options and there are {len(_LETTERS)} letters, "
+                "so it cannot be asked by letter. Ask it by label, or run it on another arm."
+            )
+        return list(_LETTERS[: len(labels)])
+
     def _single_token_ids(self, readout: LogitReadout, labels: list[str]) -> list[int]:
         """One token id per option, or a refusal naming what went wrong.
 
@@ -270,7 +320,9 @@ def _single_token_ids(self, readout: LogitReadout, labels: list[str]) -> list[in
                 "plumbline refuses the case rather than truncating an option to its "
                 "first token, because a softmax over truncated options answers a "
                 "different question than the one the dataset asks. Use options that are "
-                "single tokens for this checkpoint, or run this case on another arm."
+                "single tokens for this checkpoint, ask the options by letter "
+                "(option_style letter, --option-style letter), or run this case on another "
+                "arm."
             )
 
         duplicates = [

src/plumbline/cli.py

@@ -98,6 +98,15 @@ def run(
             "takes it. Recorded in the artifact.",
         ),
     ] = None,
+    option_style: Annotated[
+        str | None,
+        typer.Option(
+            "--option-style",
+            help="How a local checkpoint is asked its options: label reads each option's own "
+            "token and refuses options that are not one token; letter lists them as A, B, "
+            "C... and reads the letters.",
+        ),
+    ] = None,
     semantics: Annotated[
         str | None,
         typer.Option(
@@ -159,6 +168,7 @@ def run(
         base_url=base_url,
         timeout=timeout,
         device=device,
+        option_style=option_style,
         semantics=semantics,
         seed=seed,
         accuracy=accuracy,
@@ -317,6 +327,7 @@ def _build(
     base_url: str | None = None,
     timeout: float | None = None,
     device: str | None = None,
+    option_style: str | None = None,
     semantics: str | None,
     seed: int,
     accuracy: float,
@@ -333,6 +344,8 @@ def _build(
         config["timeout"] = timeout
     if device is not None:
         config["device"] = device
+    if option_style is not None:
+        config["option_style"] = option_style
     if semantics is not None:
         config["probability_semantics"] = _semantics(semantics)
 
@@ -374,6 +387,7 @@ def _build(
     "base_url": "--base-url",
     "timeout": "--timeout",
     "device": "--device",
+    "option_style": "--option-style",
     "probability_semantics": "--semantics",
 }
 
@@ -413,6 +427,11 @@ def _plan_text(
             if timeout
             else "- Timeout: the adapter's default.",
             *([f"- Device: {device}."] if device else []),
+            *(
+                ["- Options: asked as letters, A for the first, and read from the letter tokens."]
+                if getattr(adapter, "option_style", None) == "letter"
+                else []
+            ),
             f"- Probability semantics: {semantics}.",
             f"- Cost: {cost}",
             f"- Guard: {cost_limit}, {case_limit}; the run would start.",

src/plumbline/report/markdown.py

@@ -566,6 +566,13 @@ def _provenance(result: RunResult, options: ReportOptions) -> list[str]:
     if hits:
         line += f" {hits} of {len(result.records)} rows came from cache and cost nothing."
     lines = [line]
+    if result.config.get("option_style") == "letter":
+        lines.append(
+            "- **Options**: asked as letters, A for the first option and so on, and read from "
+            "the letter tokens. The softmax is over the letters, so it is still conditional on "
+            "the options supplied, and a lettered question is not the question an arm that "
+            "reads the option words is asked."
+        )
     if result.config.get("semantics_set_by") == "operator":
         lines.append(
             f"- **Probability semantics**: {_code(result.probability_semantics)}, set by the "

src/plumbline/runner/execute.py

@@ -641,6 +641,10 @@ def handle(index: int) -> None:
         # Where a local checkpoint ran. It does not change the question, but a
         # latency figure means nothing without it.
         "device": getattr(adapter, "device", None),
+        # How a local checkpoint was asked its options: by their own tokens, or
+        # by letter. The report says so, since a lettered question is a
+        # different question.
+        "option_style": getattr(adapter, "option_style", None),
         # Whether rows carried option descriptions and whether they were sent,
         # so the report can say when the dataset's descriptions went nowhere.
         "label_descriptions": {

tests/test_cli_run_options.py

@@ -282,3 +282,26 @@ def test_refused_cases_are_not_called_cache_hits_and_no_confidence_is_not_blamed
     assert "over 5 live calls" in latency_line
     assert "cache hits" not in latency_line
     assert "yes/no" not in confidence_line
+
+
+def test_an_option_style_goes_to_the_local_arm_and_is_refused_elsewhere(tmp_path: Path) -> None:
+    dataset = a_dataset(tmp_path / "d.jsonl")
+
+    refused = invoke("run", str(dataset), "--option-style", "letter", "--dry-run")
+    planned = invoke(
+        "run",
+        str(dataset),
+        "--adapter",
+        "local_logits",
+        "--model",
+        "some/checkpoint",
+        "--revision",
+        "c" * 40,
+        "--option-style",
+        "letter",
+        "--dry-run",
+    )
+
+    assert refused.exit_code == 1 and "does not take --option-style" in refused.stderr
+    assert planned.exit_code == 0, planned.stderr
+    assert "Options: asked as letters" in planned.stdout