Measure reuse against a controlled baseline (story 4.4)

[fabbrik]

A pre-registered comparative experiment: memory-enabled against memory-disabled trials, one gate expression evaluated once, and NoDemonstratedBenefit when it does not hold.

What this does not claim

It measures the harness, not a model — that is line 3 of the report, not a footnote. There is no model credential in this repository and every IChatClient is a fake, so the size of any difference between conditions is a property of the fixture that produced it. The report lists what a real result would require, none of which exists here.

The gate has been observed saying no

arm	primary metric (failed attempts)	verdict
reference	0.500 vs 2.500	BenefitDemonstrated (harness-level, simulated agent)
negative control	2.500 vs 2.500	NoDemonstratedBenefit
wrong-strategy	3.000 vs 2.000	NoDemonstratedBenefit

The negative control injects records naming approaches the exploring agent would have reached anyway. The wrong-strategy arm injects an approach that does not resolve the task — memory that misleads costs an attempt, and the gate charges it. A harness that has never been observed to say no is not evidence when it says yes.

The harness refuses a measurement it cannot attribute

Before the gate runs, every trial's failed attempts are reconciled against what the task set independently implies for the strategies that trial actually read out of its injected block. An agent handed the answer by any route other than the block fails that reconciliation by name:

HarnessIntegrityException: Trial 7 … failed 0 attempt(s), and the task set implies 1 … This is exactly how a harness that handed the agent the answer would look.

This exists because a reviewer proved it was needed: a mutation that fed the agent the ground-truth answer — while still retrieving, injecting and recording the block, merely ignoring its content — passed all 85 original tests with both golden reports byte-identical.

Separation is in substance, not identifiers

The evaluation tasks originally shared their sentences with the learning tasks (content-word overlap 1.00 and 0.88) while passing an ID-only disjointness check. Retrieval is word-overlap, so the headline was measuring a near-verbatim lookup. They are rewritten to carry the same failure modes in a different system — overlap is now 0.00 on every pair — the task set refuses wording repeats, and the report prints the task text so a reader can judge for themselves. The primary metric moved from 0.000 to 0.500 as a result.

Honesty mechanisms

• Pre-registration is a checked-in file, digested by git blob id, re-read at render time; the report refuses to render if it changed after the trials ran.
• It records its own amendments — three, all made after results existed — and the report prints them above the numbers, in those words. Adding a control after seeing results is legitimate; a reader being unable to tell is not.
• Gate terms are derived from the file, not hardcoded, and refused unless they reproduce the declared expression.
• The report discloses what works against it: which gate terms could not fail in this arm, that the retrieval hit rate is designed in, and that the process is deterministic so the printed sd is not sampling variance.
• Elapsed time is excluded from the gate — two serial paths are charged only to the memory-enabled condition, and a biased measure must not decide a verdict.

Review

Three reviewers. 22 mutations; 11 survived the original suite, including the planted answer, a cached pre-registration reader defeating the tamper check, validation moved after all twelve trials, and every ledger row mislabelled. All now fail.

Tests: 1128 → 1295, zero warnings under TreatWarningsAsErrors.

🤖 Generated with Claude Code