Add TRACE-lite: typed verdicts, record store, and consumer contracts

.claude/commands/bench-checkers.md

@@ -0,0 +1,71 @@
+---
+description: Measure this project's own checkers with TRACE-Bench-lite — run the trace-bench Workflow over seeded-defect fixtures (loop verifier, single-pass reviewer, optional critic panel), score it deterministically, and report WrongAcceptRate, invariance, the external-judge gap and the pre-registered falsification flags. Use before raising a loop to L3, or after changing a checker prompt.
+argument-hint: [arms, e.g. "verifier,single-pass" | "panel" — default verifier,single-pass] [repeat N]
+disable-model-invocation: true
+---
+
+Run **TRACE-Bench-lite**: measure whether our checkers actually catch what they claim
+to, instead of assuming it. This is the measurement side of the TRACE-lite contract
+(`.claude/memory/topics/trace.md` → *Measurement & pre-registered falsification*).
+
+Arguments:
+$ARGUMENTS
+
+## Procedure
+1. **Check the fixtures still describe reality.** Preconditions: the files the fixtures
+   patch have no local changes, and (for E2E judges) nothing is listening on port 3000.
+   Run `pnpm trace bench-judge` (or `--skip-e2e` for a quick pass). It first runs every
+   judge command on the clean tree (a judge that already fails there aborts the run),
+   then refuses stale fixtures (patches that no longer apply). It refuses to start while
+   a fixture patch is still applied from a killed run — `pnpm trace bench-judge --recover`
+   reverses those. Ctrl+C / SIGTERM / SIGHUP stop it after the current patch is
+   reversed. Every row must be
+   `ok: true` — with `--skip-e2e`, the skipped E2E rows show `ok: null` and are the only
+   acceptable exception. A row that is `ok: false` means the manifest is stale; fix it
+   before benchmarking.
+2. **Build the workflow args** (fixtures carry only the work item + patch; the ground
+   truth never reaches the agents):
+   `pnpm -s trace bench-args --arms <arms> --repeat <N>` → save to a scratch file.
+   Default arms `verifier,single-pass`, repeat `2` (repeat ≥ 2 is needed for the
+   procedural-invariance metric). `panel` adds one agent per critic per fixture (~10×
+   fixtures) — run it deliberately, optionally with `--critics a,b`.
+3. **Run the Workflow** — `scriptPath: .claude/workflows/trace-bench.js`, `args` = the
+   step-2 JSON. This command opts into multi-agent orchestration (it spawns
+   fixtures × repeat verifier agents + fixtures single-pass agents); invoking Workflow
+   here is expected. Save the returned object to your **scratchpad**, never the repo —
+   the verdict prose names the defects, and anything committed is greppable by the next
+   run's agents.
+4. **Score it:** `pnpm trace bench-score <scratch>/run.json --transcripts <the workflow's
+   transcript dir> --record` — scans every agent transcript for leaked ground truth
+   (fixture ids, defect text) and excludes those runs (**F0**), prints the metrics, and
+   appends a counts-only `measured` TRACE record (`claim_id bench:trace-bench-lite`,
+   revising the previous run's record). `--record` refuses to run without
+   `--transcripts`: an unscanned run is not a measurement. `bench-args` mints a run id that
+   every agent label and the result carry, so a run is covered only by its OWN non-empty
+   transcripts — another run's transcript dir cannot vouch for it (F0 coverage). That
+   holds only with fresh args per Workflow invocation: re-run step 2 for every run, never
+   reuse an args file (`--record` refuses a run id that is already recorded).
+5. **Report** a table per arm (WrongAcceptRate, FalseHoldRate, defer rate, gate
+   downgrades, invariance; single-pass vs panel recall / false-block; n_eff), the
+   excluded runs (contaminated / no verdict), the external-judge gap (`judgeOnly` /
+   `agentOnly` / `both` / `neither` — `agentOnly` is what the agent checkers add beyond
+   tests; `neither` is undefended ground), and every fired flag with what it means for
+   the design (`bench-score` emits F0–F4; F5 comes from `pnpm trace metrics`). Say which
+   flags *could not* fire: F1 needs the `panel` arm, F3 needs `repeat ≥ 2`. Don't explain
+   a flag away after the fact — they are pre-registered.
+6. If a flag fires, propose (don't apply) the response: F1 → shrink the fan-out; F3 →
+   don't trust single-run verdicts / raise repeat; F4 → keep loops at ≤ L2 and tighten
+   the verifier via `/improve-skills`. Then run `/dream` to file any durable lesson.
+
+## Notes
+- The verifier arm runs the **exact** production `loop-iteration` checker (prompt,
+  schema, evidence gate and denylist are byte-identical copies — `pnpm test` fails on
+  drift), with `checksPassed` self-reported as true for every fixture (worst case: the
+  checker must not lean on the implementer's word).
+- Fixtures live **encoded** in `.claude/trace/bench/fixtures.bundle` (see that folder's
+  README): agents grep this repo, so ground truth must never be plain text in it. Edit via
+  `pnpm trace bench-unpack <dir-outside-repo>` → edit → `pnpm trace bench-pack <dir>`;
+  `pnpm test` checks the bundle, that patches apply, judge kinds vs the denylist, and that
+  no fixture id/defect text appears anywhere in the tree; `bench-judge` checks the rest.
+- When reporting, describe fixtures by count and category, not by id — the report text
+  may end up in the repo (log, records).

.claude/commands/critic-round.md

@@ -1,35 +1,58 @@
 ---
-description: Run the multi-agent critic round as a deterministic Workflow — critics fan out as parallel subagents (own context each), each blocker/important finding is adversarially verified by a separate skeptic, then a synthesized severity matrix is returned. Read-only; changes no code.
+description: Run the multi-agent critic round as a deterministic Workflow — critics fan out as parallel subagents (own context each; a code-only round uses one all-lens reviewer), each blocker/important finding gets a typed TRACE verdict (accept/qualify/revise/defer/reject) from a separate skeptic, then a synthesized severity matrix is returned and its verdict records appended. Read-only on code.
 argument-hint: [focus, e.g. "checkout flow on localhost"]
 ---
 
 Run a **genuine multi-agent critic round** — not a single agent simulating critics.
 Use the **Workflow tool** so each critic runs as its own subagent in its own context
-and every blocker/important finding is verified by a *separate* skeptic.
+and every blocker/important finding is verified by a *separate* skeptic. (One measured
+exception: a code-only round, `uiInScope: false`, runs one all-lens reviewer — see Notes.)
 
 Focus:
 $ARGUMENTS
 
 ## Procedure
 1. Make sure context exists for the critics: confirm `CLAUDE.md` and (if a UI flow
    is in scope) that localhost is running for the browser-driven critics.
-2. **Invoke the Workflow tool** with the predefined critic-panel workflow:
+2. **Gather the reuse inputs:** `pnpm -s trace query --latest --writer critic-panel --brief`
+   (prior verdicts) and `pnpm -s trace tree-id` (HEAD + uncommitted changes — reuse only
+   fires when the code is identical).
+3. **Invoke the Workflow tool** with the predefined critic-panel workflow:
    - `scriptPath: .claude/workflows/critic-panel.js`
-   - `args: { "focus": "$ARGUMENTS" }`
+   - `args: { "focus": "$ARGUMENTS", "priorRecords": <step-2 JSON>, "treeId": "<tree-id>" }`
    This command explicitly opts into multi-agent orchestration — calling Workflow
    here is expected.
-3. The workflow returns `{ focus, counts, confirmed, niceToHaves, refuted }`.
-   Render `confirmed` as the critic matrix and list `niceToHaves` separately:
+4. The workflow returns `{ focus, treeId, reviewMode, failedReviewers, counts, confirmed, deferred, revised, niceToHaves, refuted, traceRecords, reuseActions }`.
+   Render `confirmed` (verdict `accept` or `qualify`) as the critic matrix and list
+   `niceToHaves` separately:
 
-   | Critic | Severity | Finding | Evidence | Recommended action |
-   |---|---|---|---|---|
+   | Critic | Severity | Verdict | Finding | Evidence | Recommended action |
+   |---|---|---|---|---|---|
 
-4. Note any `refuted` findings (a skeptic could not substantiate them) so they are
-   not actioned. Do **not** edit files — this is review only.
+   Show a `qualify` verdict with its qualifier (the strength the claim actually holds
+   at). List `deferred` findings with their `missing` evidence and `revised` ones with
+   their `repair` — neither is actionable as stated. Note `refuted` findings so they
+   are not actioned. If `failedReviewers` is non-empty, say which lenses are missing —
+   the round is incomplete.
+5. **Record the verdicts** (memory, not code), per the consumer protocol in
+   `.claude/memory/topics/trace.md`: save the workflow result JSON to a scratchpad file,
+   then `pnpm trace write <file> --then-act HOLD --consumer critic-round --note
+   "review-only round"` (records the verdicts and that this round acted on none of them)
+   and `pnpm trace act --from <file>` (its `reuseActions`). If the writer rejects the
+   batch, report the violations and do not hand-edit drafts. These records are what the
+   next round reuses. Do **not** edit any other file — this is review only.
 
 ## Notes
 - This is the heavier, deterministic path (the workflow spawns ~10 critic agents +
-  one verifier per blocker/important finding). That is intended: real independent
-  review, not one context role-playing ten critics.
-- Other commands (`/multi-agent-dev`, `/multi-agent-e2e`, `/qa-pass`) reuse this same
+  one verifier per blocker/important finding, minus reused verdicts). That is intended:
+  real independent review, not one context role-playing ten critics.
+- Exception, measured: a code-only round (`uiInScope: false` — no browser lens in scope)
+  runs ONE all-lens reviewer instead of the critic fan-out (`reviewMode: 'single-pass'`):
+  TRACE-Bench-lite found it as good as the panel on code-only review at ~5x lower cost
+  (F1; qualified: one run on the 19-fixture bench, 5 of the 10 critics, measured as diff-level block/no-block with the bench's own single-pass prompt — not this round's production prompt). Each finding still names its lens and still gets its own separate
+  skeptic. Pass `reviewMode: 'panel'` to force the fan-out.
+- Agreement between critics is **not** extra evidence (they share a model); only a
+  critic bringing *different* evidence corroborates. Verdict semantics, the evidence
+  standard per claim type, and the consumer actions: `.claude/memory/topics/trace.md`.
+- Other commands (`/multi-agent-dev`, `/qa-pass`, `/loop`) reuse this same
   workflow for their critic rounds.