test(agent): golden-trace regression harness

[obvious-autobuild]

Why

The agent's ~930 lines of existing tests all hand-write their model scripts inline: loop behavior, argument shapes, and refusal wording are pinned per-test, but no persisted corpus of full sessions exists. A registry rename, a schema drift, or a gate-wording change that slips between the cracks of unit tests has nothing to catch it. This is deliverable BACKLOG-4 of the backlog program (issue #47).

What

A persisted, reviewed corpus of four full agent sessions in tests/fixtures/golden/, replayed offline in CI by tests/test_agent_golden.py — no API key, no anthropic SDK, no new dependency:

Trace	Session shape
01-read-only-chain.json	search_shots → load_shot → limit_distances, ends in a prose answer
02-run-whatif-proposal.json	confirm-then-apply: run_whatif proposes, the model asks, the loop ends before apply
03-gate-refusal.json	a run_whatif the gate refuses 422; the model explains the refusal verbatim and never retries
04-budget-exhaustion.json	six tool rounds; the cap ends the turn in error

Every trace replays at both levels — the loop (agent.turn) and the HTTP surface (POST /agent/chat through parse_sse) — and the shared assert_golden checks: ordered tool sequences with the recorded args; argument keys ⊆ the live tool's input_schema.properties with required keys present; no name outside agent_tools.TOOLS dispatched, with run_tool's unknown-tool text pinned against the live name list (names, never counts — a hardcoded count already drifted once); fixed event ordering (tool_call → tool_result pairs, exactly one terminal event); the 6-round budget and the TOOL_RESULT_MAX_CHARS truncation marker; confirm-then-apply invariants (the proposal reaches the sink and the action_proposed event; the model context carries "nothing is applied yet" and never the apply payload); streamed text reproduces the corpus exactly; honesty wording in every prose-final turn.

The corpus is hand-authored reviewed exemplars — that is the recording path: agent_fakes.RecordingClient exists for one-time key-bearing capture, and its serialization round-trip is itself tested — while CI only replays. The "model" in a golden trace is the fixture, so this pins loop + registry + gates, never model quality. Hermetic throughout: the suite runs with the key deleted, and the pristine-interpreter "app import never loads the anthropic SDK" check is extended to the new module. Also fixes the stale "eight allowlisted definitions" docstring in tests/test_agent_loop.py (the registry holds nine — the exact drift this harness's name-based assertions prevent).

How to Review

• tests/test_agent_golden.py — the harness; assert_golden is the contract, the per-trace tests below it are specifics.
• tests/fixtures/golden/*.json — the corpus; reviewed exemplars, not generated output.
• tests/agent_fakes.py — RecordingClient + to_trace() (the capture half, never constructed in CI).
• Intentionally excluded: no product-code or UI changes at all — this is a test-layer PR.

Test Evidence

Backend-only change (test layer): rendered UI surface unchanged, so visual dogfood evidence is not applicable.

• make test: 497 passed, 1 skipped (93 s) against the fresh pre-change baseline of 476 passed, 1 skipped — the +21 are the new golden tests.
• uv run ruff check clean; uv run mypy fusionlab clean (33 files).
• Mutated-fixture red test, verified once by hand during implementation: corrupting a fixture on disk (wrong arg key) fails with a readable diff — load_shot: argument key(s) ['shot_ids'] not in the live input_schema — and the restored fixture replays green; the same three corruptions (wrong arg key, out-of-allowlist name, dropped proposal round) are pinned permanently by test_the_harness_fails_loudly_on_a_mutated_trace.

Fixes #47

Human author: Tapan Ganatma Nakkina ([email protected])

🔗 Obvious Project · 🧵 Obvious Thread · 🔀 Obvious PR Artifact