fix(local): what running the local arm for real found

[TMHSDigital]

Part of #3. This covers the local half; the generative half is still open.

The run

local_logits ran against Qwen/Qwen2.5-1.5B-Instruct, pinned to commit 989aa7980e4cf806f80c7fef2b1adb7bc71aa306, on the vendored JevBench fixture. It used an RTX 4060 with CUDA torch 2.14 and transformers 5.17 on Windows.

It finished end to end: the artifact was written, every metric was computed against its null, and the report rendered. Two runs gave identical figures.

Scored	39 of 105: 38 yes/no rows and 1 choice row
Refused	66 choice rows, whose options are multi-token (pay_subject_to_10000_sublimit), each named
Accuracy	0.4615 against a chance null of 0.4957: INCONCLUSIVE
ECE	0.1257 against a floor of 0.1016 (p95 0.1869): INCONCLUSIVE
Brier	0.2762 against a floor of 0.2386 (p95 0.2662): distinguishable from noise
Latency	p50 71 ms, p95 467 ms, p99 488 ms (after the warm-up fix)

A 1.5B model at chance on this fixture is the null working as intended. The refusals are the designed behavior: truncating an option to its first token would change the question.

What it found

1. Every worker loaded its own checkpoint. The readout is built lazily on the first case, and the runner's 8 workers all reach that case together. A new test with a slow fake readout counted 8 loads, which is gigabytes each onto one GPU. The load is now behind a lock. Forward passes also run one at a time behind a second lock, with the latency timer started after the lock is acquired, so a pass that waited on another doesn't report the wait as model latency.
2. The CLI couldn't put the model on a GPU. The adapter took device, but the CLI never passed it, so every CLI run was on CPU. --device now passes it through, the dry run shows it, and the artifact records it next to the timeout.
3. Kernel setup was timed as latency. The first pass on a fresh CUDA context costs about a second, and under concurrent first calls one case took 10 s against a 66 ms median. TransformersReadout now makes one warm-up pass when it loads, and p99 fell from 10,042 ms to 487 ms.
4. Two report lines were wrong for any arm with failures.
   • The latency line counted len(records) - len(live) as cache hits, so the 66 refusals printed as "excluding 66 cache hits" on a run with no cache. It now counts only answers served from cache.
   • The confidence line said "a yes/no answer has no distribution to summarize" for every arm without a confidence. It now says so only when every row was asked as yes/no, and otherwise says the adapter returns none.

Not changed

• The multi-token refusal stays as designed. The fixture's choice options are mostly snake_case identifiers, so the local arm scores almost only the yes/no rows here. Scoring multi-token options by sequence log-probability is a methodology question, so I'll raise it on Run the local arm for real: local_logits and generative have never left the test suite #3 rather than decide it here.
• Loading took anywhere from about 100 s to 731 s on this machine, first import included. It isn't part of any figure, and it's noted on Run the local arm for real: local_logits and generative have never left the test suite #3.

Docs

README's adapters table now says the local arm has run for real, and the Limitations line narrows to the generative transport. The local section shows the pinned command and --device, notes that PyPI's Windows torch is CPU-only, and says to expect refusals on this fixture.

docs/example-report.md is unchanged. The full gate, the site build and its checks, and the prose checks all pass.

🤖 Generated with Claude Code