fix(runtime): attempts abandoned by a crash are recorded, so stats and status agree

[aviggiano]

Problem

The campaign-resume e2e test (packages/cli/test/e2e/campaign-resume.test.ts) SIGKILLs the detached controller while summarize runs, then resumes the run. Afterwards ultrafuzz stats counted 3 agent attempts, while status (model_mix) and the stub agent counted 4. The test recorded this as a todo subtest. Smithers' own attempt rows, which ultrafuzz node node:summarize --attempts lists, hold two attempts for the node, the first one cancelled (observed in a run with this change; Ultrafuzz does not write those rows). attempts.jsonl held only the second.

Separately (#1087), stats counted a node interrupted by ultrafuzz cancel as failed, while status counted the same task as other.

Root cause

• Missing attempt. terminalWorkflowAttempts (packages/runtime/src/workflow-sync.ts) pairs each NodeStarted with a terminal NodeFinished, NodeFailed or NodeCancelled event. At each RunStarted it cleared the starts that were still open, and nothing recorded them. When a run resumes, Smithers 0.35.0 marks the dead activation's in-progress attempt rows cancelled in its resume-cancel-stale-attempt transaction but emits no event for them. So the attempt the kill interrupted never reached the ledger, and fix(runtime): attempt and usage ledgers never block run synchronization #1186's NodeCancelled handling had no event to handle.
• stats counts operator-cancelled nodes as failed while status counts them as other #1087. Run state has no cancelled status: statusFromWorkflowState maps a Smithers cancelled task to failed, and stats counted run state. status counts Smithers' own task states, where cancelled is other.

Change

Source and schema +88/−39, tests +260/−15, docs +22/−7.

1. fix(runtime): record attempts a stopped controller abandoned as canceled. A start still open at a RunStarted is kept as abandoned. It is recorded when its task has its next event, normally the replacement attempt's NodeStarted, with:

   • outcome and category canceled;
   • the message abandoned: the controller stopped during this attempt, and the resumed run cancelled it;
   • that event's sequence and timestamp as its terminal.

   It needs its own terminal event because the ledger identity is (workflow_run_id, source_event_sequence) and one RunStarted can abandon several attempts. The rest follows the existing rules:

   • Agent provenance comes from the smithers node attempt row that synchronization already fetches, so a pre-agent attempt is still skipped.
   • An abandoned attempt whose number a reset reuses is recorded without the agent block, like other superseded occurrences.
   • Nothing new can throw. An abandoned start whose task has no later event stays unrecorded, and a terminal stamped before its start is still skipped.

   Normal pairing and the abandoned path share one end() helper. The e2e todo is now a hard assertion. Docs: the attempt-ledger section of docs/reference/artifacts-reports.md, and the e2e description in docs/reference/development.md.

2. fix(cli): stats counts a node whose task Smithers cancelled as canceled (stats counts operator-cancelled nodes as failed while status counts them as other #1087). A node that run state records as failed is reported, and counted in totals.status_counts, as canceled when every failed task of it has Smithers state cancelled in its recorded provenance.workflow.state. That is the Smithers task state that status counts. The CLI result schema gains canceled in the stats node status enum and in statsStatusCounts. The envelope stays ultrafuzz.cli.result.v2 and stats stays ultrafuzz.stats.v1. Docs: docs/reference/cli.md.

3. fix(cli): stats takes a node's latest attempt in event order. aggregateAttemptOutcome took the last ledger row per strategy attempt in file order. After change 1, the next sync of a run that an earlier version already synchronized appends the abandoned row after the replacement's row. Without this change, that run's outcome column would read canceled for a node whose last attempt succeeded. Within one workflow run the higher source_event_sequence now wins. Rows from different workflow runs keep file order.

Deliberately not built

• No upstream Smithers event and no new compatibility patch. The Attempt and cancellation ledgers are incomplete after recovery #1139 analysis proposed asking Smithers to emit NodeCancelled{reason: "resumed"} for stale attempts. The events Ultrafuzz already reads are enough.
• No reconciliation from smithers node attempt rows. The spec offered it as an alternative source. Those rows carry no event sequence to key the ledger identity on, and using them would add a second pairing path.
• No cancelled status in run state (NODE_STATE_STATUSES). stats derives canceled from the recorded Smithers state instead, so state.json and event schemas are unchanged.
• stats counts operator-cancelled nodes as failed while status counts them as other #1087 does not use the attempt ledger. A task cancelled before Smithers selected an agent has no ledger row. A cancellation while the verifier runs is recorded as a failed agent attempt (the finalization overlay). A ledger-based rule would keep disagreeing with status in both cases.
• The attempt-source-event-join semantic gate is unchanged. It expects NodeFailed for every non-success outcome, so it would reject these rows, as it already rejects fix(runtime): attempt and usage ledgers never block run synchronization #1186's NodeCancelled-terminated rows. No production code runs it. Nothing supplies its eventLog context, and no production caller runs any of the node-attempt-ledger.schema.json gates (the ledger reader has its own checks). Only packages/artifacts/test/semantic-gates.test.ts runs it. Deleting this one gate would leave the other ledger-schema gates, which no production code runs either, so deleting them together is a separate cleanup.
• Only four events end an abandoned start. An abandoned start ends at the task's next NodeStarted, NodeFinished, NodeFailed or NodeCancelled. If the task's next event is some other type (NodeSkipped, NodePending, NodeRetrying) or never comes, the attempt stays unrecorded, as before.

Verification

Discriminating tests, each run against origin/integration/wave1 sources by restoring the base file, running, then re-applying the change:

• syncRun records each attempt that a killed controller abandoned as canceled (new, packages/runtime/test/runtime.test.ts). Two tasks are left in progress by one RunStarted and restarted as attempt 2. Two syncs must record one canceled row per task, with agent provenance, no diagnostics and retry_count 1. Base: the ledger is []. Fixed: passes.
• syncRun records an attempt abandoned at a run activation boundary even when a reset reuses its number. This replaces syncRun abandons an unterminated occurrence at a later run activation boundary, which asserted only ok and running. Base: []. Fixed: one canceled row without the agent block.
• stats counts a failed node whose task Smithers cancelled as canceled (new, packages/cli/test/run-statistics.test.ts). It covers a single-task node and a fan-out node. The fan-out node's canonical state aggregates two strategy tasks and has no Smithers state of its own. The test also validates the envelope against the closed CLI result schema. Base: [ 'failed', 1, undefined ]. Fixed: [ 'canceled', 0, 1 ]. With the fixed run-statistics.ts and the base schema it also fails; the schema is the only difference. Two simpler rules fail it too:
  • Requiring every failed state, the aggregate included, to carry Smithers state cancelled: the fan-out node whose tasks were both cancelled gets [ 'failed', 1, 0 ].
  • Accepting any cancelled task instead of every one: a fan-out node with one cancelled and one failed task gets [ 'canceled', 0, 1 ].
• stats reports a node's latest attempt outcome in event order, not ledger row order (new). With the base aggregateAttemptOutcome, the case where the abandoned row comes after its replacement gets 'canceled' instead of 'succeeded'. Fixed: passes.
• syncRun records an abandoned attempt of an already reported run and publishes the report again (new, packages/runtime/test/runtime.test.ts). The first synchronization sees the event log without the abandoned NodeStarted. That reproduces what an earlier version recorded for a finished, reported run: ledger [[2, 3, 4, 'succeeded']] (attempt, start, terminal, outcome), retry_count 0 and a verified report. The second synchronization sees the full log. It must append [1, 1, 3, 'canceled'] after the replacement's row and raise retry_count to 1. The report publication status that status reads must stay available and verified, and loadCurrentFinalReportSnapshot must still return verified-runtime-report. Base: the ledger keeps one row and retry_count stays 0. Two mutants fail it with terminal report receipt does not match current controller evidence: one skips the terminal-report republish once a publication status exists, and one stops an immutable node's retry_count from following the ledger.
• e2e campaign-resume.test.ts. Base: a probe copy of this test with the same flow printed per-node attempt_count 1/1/1 (3) against 4 status attempts, failure_categories [] for summarize, and the todo failing with 3 !== 4. Fixed: the test passes with the former todo as a hard assertion, on e9ee6dd (351 s) and on e64c6db (384 s), the last commit that changes source; the later commit adds tests only. The fixed run's ledger shows summarize attempt 1 canceled from sequence 39 (its NodeStarted) to 45 (attempt 2's NodeStarted), with agent chain index 0; attempt 2 succeeded 45 to 57. stats gave summarize attempt_count 2, retry_count 1 and failure_categories ["canceled"]. ultrafuzz node listed attempt 1 as cancelled, finishing 0.25 s before the ledger's finished_at.

Existing tests:

• The runtime tests matching attempt|ledger|occurrence|supersed|activation|abandon|cancel ran on the final HEAD: 48 tests, including the new one; 46 pass and 2 are skipped (Bun-only adapter contracts).
• CLI: the run-statistics, stats-command and cli-contracts test files pass (34 tests).
• Gates: prettier --check on the changed files, eslint, pnpm -w lint, CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/integration/wave1 pnpm -w lint:strict:ci, pnpm --filter @ultrafuzz/runtime --filter @ultrafuzz/cli typecheck, pnpm -w knip and node scripts/docs-check.mjs pass. The complexity ceiling is unchanged. The repository maximum is still 90 (synchronizeLinkedWorkflowRun, not touched), and the changed functions measure 2 to 37.

Risk / compatibility

• More ledger rows. attempts.jsonl gains a canceled row for each crash-abandoned attempt. Pre-agent attempts are skipped as before, unless a reset supersedes them. Those rows count toward retry_count, stats attempt_count, executed-attempt counts, ultrafuzz inspect attempt summaries and the eval recovery metrics (observed_node_attempts, repeated_model_backed_node_executions). All of these undercounted before.
• Durations include downtime. An abandoned attempt ends when the resumed run restarts its task, so its finished_at, and stats duration_ms for the node, include the time the controller was down. Smithers' own attempt row ends at the resume too: 0.25 s earlier in the e2e run.
• Runs synchronized before this change get their abandoned rows appended on the next sync, after later rows in the file. Change 3 keeps the outcome column correct. The other ledger readers I found (retry and attempt counts, inspect summaries, eval recovery metrics) count rows and do not depend on order.
• Finished runs with an abandoned attempt republish their report once. On such a run, the first synchronization after the upgrade (status, stats, inspect and why each run one) also raises retry_count in state.json for the node that lost an attempt. This happens even when the node's successful publication is immutable, because the retry count follows the ledger. The raised retry_count, and the workflow-synced event that the same synchronization appends, both make the published terminal report stale. The report publication status is keyed to a state fingerprint that includes retry_count, so the same synchronization publishes the report again: a new review/runtime-report/<generation>, a rewritten current.json, and a rewritten review/report-publication.json. The report stays verified and status keeps reporting it available. The retry_count change is what triggers that republish; the new runtime test fails without it.
• Schema. The stats JSON gains a canceled node status and a canceled key in totals.status_counts. A consumer that validates stats output against the previous closed schema will reject the new key.
• Scope of stats counts operator-cancelled nodes as failed while status counts them as other #1087. A failed node is reported canceled only when every failed task of it has recorded Smithers state cancelled. A failed node whose provenance has no Smithers state (the field is optional) keeps reporting failed.

Changelog entry

• [runtime] [cli] [docs] An agent attempt that was running when its controller stopped is now recorded in attempts.jsonl as canceled once the resumed run restarts or cancels its task. Smithers cancels such attempts on resume without emitting an event, so they were missing before, and after a crash and resume stats now counts the same agent attempts as status. stats also reports a failed node whose task Smithers cancelled, for example by ultrafuzz cancel, as canceled instead of failed, and takes a node's latest attempt outcome in event order (Attempt and cancellation ledgers are incomplete after recovery #1139, stats counts operator-cancelled nodes as failed while status counts them as other #1087).

Refs #1139, #1087

🤖 Generated with Claude Code

Confidence Score: 4/5

The PR should not merge until stats preserves failures when a fan-out task lacks cancellation evidence.

Findings

1.  Failed tasks can disappear ▶

Fix with agent prompt

### Issue 1
packages/cli/src/run-statistics.ts:491-495
When a fan-out node has one task marked `cancelled` and another failed task without a recorded workflow state, this check drops the second task. It then reports the whole node as `canceled`, hiding the failure in `totals.status_counts`. A failed task needs cancellation evidence before it can be counted as canceled.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR records controller-abandoned attempts on resume, aligns stats cancellation counts with Smithers task state, and selects latest attempt outcomes by event sequence.

• Adds runtime and CLI regression coverage, strengthens the resume end-to-end assertion, and updates the result schema and documentation.
• The cancellation aggregation still needs to account for failed tasks without recorded workflow state.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[RunStarted after controller stop] --> B[Retain open attempt by task]
  B --> C[Next task event]
  C --> D[Append canceled attempt to ledger]
  D --> E[stats attempt counts and outcome]
  F[Stored task workflow states] --> G[stats node status]
  G --> H[status counts]

Loading

_{Reviews (1) · Last reviewed commit: "test: cover a cancelled fan-out node and..."}

[greptile-apps]

Failed tasks can disappear When a fan-out node has one task marked cancelled and another failed task without a recorded workflow state, this check drops the second task. It then reports the whole node as canceled, hiding the failure in totals.status_counts. A failed task needs cancellation evidence before it can be counted as canceled.

Knowledge Base Used: CLI workflows

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/cli/src/run-statistics.ts
Line: 491-495

Comment:
**Failed tasks can disappear** When a fan-out node has one task marked `cancelled` and another failed task without a recorded workflow state, this check drops the second task. It then reports the whole node as `canceled`, hiding the failure in `totals.status_counts`. A failed task needs cancellation evidence before it can be counted as canceled.

**Knowledge Base Used:** [CLI workflows](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/cli-workflows.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.