fix(runtime): share the Claude Code stated-failure hook and label it in last_error

[aviggiano]

Follow-up to #1224 and #1225, which made ClaudeAgent and DeepSeekAgent record the failure Claude Code states in a failed result (#1084). Thanks @mrthankyou for the original work. This branch addresses the review of both.

Summary

• ClaudeAgent and DeepSeekAgent share one stated-failure hook, StatedFailureClaudeCodeAgent in claude.tsx, instead of two copies.
• The adapter keeps the stated text whole, so the controller redacts all of it before its 1,000-byte cap. A 16,384-character cut, tried during review, is dropped because it can leave part of a secret unredacted.
• last_error and the ledger's failure_message read Claude run failed See https://smithers.sh/reference/errors (agent stated: …).
• One Bun test per adapter reuses a single instance across every case, so it now fails without the per-generation reset or with a wider guard.
• The CHANGELOG entry is one bullet that names where the text shows (not status or why) and says projects must re-run ultrafuzz init.
• The adapter-boundary row names the upstream line; the Modal how-to lists the public diagnostic's failure_message.

Change

1. One base class for the hook (packages/runtime/src/templates/smithers/agents/claude.tsx, deepseek.tsx)

#1225 gave DeepSeekClaudeCodeAgent a copy of CompatibleClaudeCodeAgent's statedFailure field and its createOutputInterpreter() and generate() overrides, and claude.tsx exported GENERIC_CLAUDE_FAILURE, claudeResultText and attachStatedFailure only for that copy. StatedFailureClaudeCodeAgent extends SmithersClaudeCodeAgent now holds the field, both overrides and the #1084 comment, and both adapters extend it. DeepSeek does not extend CompatibleClaudeCodeAgent, whose buildCommand forces the "user" settings source; DeepSeek's sets "". The three helpers are module-private again, and deepseek.tsx imports the Smithers class with import type.

Nothing pins these templates' bytes: packages/runtime/src/controller-source.ts compares a project's .smithers/agents/*.ts with the packaged templates when ultrafuzz run plans a run, and the init test matches their text by pattern.

2. No cap on the kept text (claude.tsx: claudeResultText)

The review proposed capping the kept result at 16,384 characters, because the controller's normalizer (normalizeNodeAttemptFailureMessage in packages/artifacts/src/attempt-ledger.ts) redacts the whole string before its 1,000-byte cap. On this host redaction takes about 11 s per MiB of BIP39 words in punctuated runs, and 0.1 to 0.3 s per MiB of prose or base64.

This branch tried that cap and then dropped it. A cut runs before redaction, so a secret that crosses it reaches redaction only in part:

• With a plain slice, 15,200 newlines and then a 4096-bit PKCS#8 key left a full 64-character key line in the normalized message for 19 of 40 keys. Uncapped, none did.
• Cutting only at word and PEM-block boundaries fixed that case but not a secret made of several words. password: "correct horse battery staple" cut after horse kept horse, and the first 10 words of a 24-word mnemonic survived.

Claude Code's is_error results are short error messages, so the cap would only have bounded the cost of a statement of several megabytes. The adapter keeps the statement whole, as #1224 did. The reused-adapter test pins this: a 19,999-character statement arrives whole.

3. (agent stated: …) (packages/runtime/src/workflow-sync.ts, errorText)

SmithersError appends See https://smithers.sh/reference/errors to its message, so the ${text}: ${stated} join read …/reference/errors: Failed to refresh OAuth token: …. The join is now ${text} (agent stated: ${stated}). errorText feeds both last_error and the ledger's failure_message; the dashboard, inspect --json and the Modal diagnostics read last_error as written, so no other code joins the two. The sync test feeds the real thrown message and anchors the whole line.

4. One reused-instance adapter test (packages/runtime/test/runtime.test.ts)

The workflow can run several attempts of a task on one adapter: a retry that Smithers schedules before the workflow re-renders reuses the wrapper's executionAgent (executionAgent ??= admittedAgent() in workflow.tsx). The old tests built a new agent for every generation and had no banner or explicit-error case, so they could not see a missing reset or a wider guard. generated <adapter> adapter attaches a stated failure only to its own generation runs, on one instance per adapter:

1. stated failures: the Concurrent agents race on OAuth token refresh; three immediate retries all re-race and kill the run with an opaque "Claude run failed" #1084 OAuth refresh race and an expired login for Claude, and the 401 that fix(runtime): DeepSeekAgent records the failure Claude Code states, like ClaudeAgent #1225 observed for DeepSeek. Each keeps the message exactly Claude run failed See https://smithers.sh/reference/errors and the code AGENT_CLI_ERROR;
2. a crash that prints only stderr, which must not inherit the statement;
3. a session-limit banner (AGENT_QUOTA_EXCEEDED) and a result with an explicit error field, which get no statement;
4. a 19,999-character statement, which arrives whole;
5. a result that states nothing.

The per-call Claude and DeepSeek tests are gone; every case they had is a step here.

5. Normalizer test (packages/runtime/test/generated-workflow-verifier.test.ts)

The test passed execution: { agentCredentialEnv: [...] }, which the normalizer never reads: it calls sensitiveEnvironmentValues(process.env) with no names. The option is gone, and a comment says the value is redacted because its variable name looks like a credential. Redaction is unchanged.

6. CHANGELOG (CHANGELOG.md)

The two #1084 bullets are one, for ClaudeAgent and DeepSeekAgent. It drops "(so status and why show it)": ultrafuzz status relays the attempt error's message (attemptErrorSnippet in @smthrs/cli), and ultrafuzz why relays blockers that parseErrorSummary builds from the error's name and message. It names where the text shows: inspect <run-id> --json (data.state.nodes.<node-id>.last_error), the dashboard's latest error, the ledger's failure_message, and a public Modal eval worker's public-eval-diagnostics.json. ultrafuzz run refuses stock adapters from an earlier release (CONTROLLER_SOURCE_UNTRUSTED, "rerun ultrafuzz init"), so the entry says projects must re-run ultrafuzz init, which keeps ultrafuzz.toml, topology and prompts. A run launched by an earlier release records the statement only after resume --refresh-controller.

7. Adapter-boundary wording (docs/reference/agent-adapter-boundaries.md, packages/runtime/test/agent-adapter-boundaries.test.ts)

The claude.tsx row says the hook overrides generate as well as createOutputInterpreter, and names the upstream gap: @smthrs/agents 0.35.0 src/ClaudeCodeAgent.js builds a failed result's error as limitBannerText || resultError || "Claude run failed" (line 476) and drops payload.result. The deepseek.tsx row says it inherits the hook, and the gate keeps output-interpretation declared for it as inherited. #1084 stays the tracked reference.

Also: the Modal how-to (docs/how-to/run-evals-on-modal.md)

It said the public diagnostic's failed-node projection "never contains messages, paths, findings, or provider output". Since #415 a failed node's entry carries failure_message when the node has a last_error: that error, redacted and cut to at most 1,000 bytes (packages/modal/src/public-eval-diagnostics.ts). It can name paths and, since #1084, quote what an agent CLI stated, such as DeepSeek's 401. The paragraph now lists the field and what it can hold.

Verification

The branch merges current main (#1162 and #1245). The only conflict was in CHANGELOG.md, where #1162 had added the friction-log bullet directly above the two #1084 bullets this branch folds into one; the friction bullet stays above the folded one.

On the final head, after pnpm install --frozen-lockfile and pnpm -w build:

Command	Result
pnpm -w typecheck	pass
pnpm -w lint	pass
CI=1 ESLINT_PLUGIN_DIFF_COMMIT=<origin/main> pnpm -w lint:strict:ci	pass
pnpm -w knip	pass
pnpm -w format:check	pass
pnpm -w docs:check	pass; no prompt changed
bun test dist-test/test/runtime.test.js --test-name-pattern '^Bun adapter contract:' (Bun 1.3.14)	53 pass, 1 skip, 0 fail
node --test --test-name-pattern '#1084' dist-test/test/runtime.test.js	syncRun shows the failure an agent stated beside its generic error (#1084) passes

Before the merge and the cap removal, from a clean build of every package, the same static gates passed. So did test:release:supporting (Node 941/941, Bun adapter contracts 53 pass, 1 skip, 0 fail) and all four runtime shards (89 + 86 + 90 + 77 = 342 pass, 0 fail). CI reruns all of them on this head.

Mutation checks on claude.tsx: each change was applied on its own, the new test run with bun test dist-test/test/runtime.test.js --test-name-pattern 'attaches a stated failure only to its own generation', and the file restored. Each one fails both adapters' tests, at the step named:

Mutation	First failing step
this.statedFailure = undefined deleted	the stderr-only crash
guard widened to !event.ok	the limit banner
widened guard that spares limit banners	the explicit error field
a 16,384-character cut restored	the 19,999-character statement

Not in this PR

• Showing the stated cause in ultrafuzz why.
• Redacting the values of a task's configured credential variables (agentCredentialEnv) in the normalizer.
• Stripping C1 and bidi control characters from the stated text.
• Whether public eval diagnostics should carry agent or provider text at all. They have carried each failed node's redacted last_error since Retain bounded failed-node messages in durable diagnostics #415. Since Concurrent agents race on OAuth token refresh; three immediate retries all re-race and kill the run with an opaque "Claude run failed" #1084 that can include what an agent CLI stated, such as DeepSeek's 401, with the key redacted. Leaving it out would be a change to packages/modal/src/public-eval-diagnostics.ts.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no actionable issue was identified.

Summary

The PR consolidates Claude Code stated-failure handling for Claude and DeepSeek, labels the statement in persisted error text, and updates adapter tests and documentation.

• A reused-instance test checks that each generation receives only its own stated failure.
• Documentation clarifies where the resulting diagnostic is visible and how existing projects refresh their adapters.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Claude Code failed result] --> B[Shared stated-failure hook]
  B --> C[Error details]
  C --> D[Workflow synchronization]
  D --> E[Redacted state last_error]
  D --> F[Redacted and capped attempt failure_message]
  E --> G[Redacted and capped public diagnostic]

Loading

_{Reviews (1) · Last reviewed commit: "fix(runtime): keep the whole failure Cla..."}