fix(runtime): make run synchronization single-writer, idempotent, bounded and non-fatal for observers

[aviggiano]

Problem

Run state (state.json, the event/attempt/usage journals, run.json accounting) converges only when some command synchronizes the run: status, inspect, why, stats, the dashboard's inspect action, and the eval runner's 15 s poll. That pump had four stability defects:

• Preserve complete structured diagnostics when CLI output is truncated #1152 — no single writer. Every observer runs the full mutating pass and nothing serialized them. Two passes over one newly finished node both finalized it and raced on its manifest temp file, state.json and the journals. The analysis reproduced duplicate node-synced events, a node recorded failed in the immutable attempt ledger while it succeeded, and passes crashing midway through a write group.
• Make run status authoritative, bounded, and read-only #1145 — status was neither idempotent nor bounded. Every projection renewed the controller lease and advanced the concurrency clock, and any difference counted as a change, so each status poll rewrote state.json and a finished run kept an "active" lease renewed by whoever looked at it. Read-only runner queries had no default timeout, so one wedged runner process blocked status and the pumps indefinitely. An orphaned run's status told operators to run workflow runner supervise -r <id>, which is not a command.
• Observers died on bookkeeping. A failed or malformed runner events query made status return ok:false (exit 1, --watch stops) although the runner's health was available, and any error thrown by the refresh (for example a torn usage.jsonl append) escaped getRunHealth entirely.
• workflow_deadline_seconds is only enforced while an operator runs a CLI command; unattended runs have no wall clock #1110 — the deadline could cancel a paused run, and a failed cancel stopped the watch. A paused run executes nothing and resuming it records a new deadline, so cancelling it bounded nothing. WORKFLOW_DEADLINE_CANCEL_FAILED was an error, so status --watch stopped and nothing asked again.

Also: node-artifacts-missing was emitted for every gate failure with missing: [] for schema, semantic and authority failures, and a failed node's last_error joined messages without saying which artifact failed.

Root cause

The synchronization pass is written as if it were the only writer and as if every observation were a change. Neither holds: every observer is a writer, and most observations change nothing but the clock.

Change

• One pass per run at a time (workflow-sync.ts). synchronizeLinkedWorkflowRun runs each pass under a non-blocking proper-lockfile lock on <run>/.workflow-sync (lockfile <run>/.workflow-sync.lock). A pass that finds it held returns ok with an info WORKFLOW_SYNC_IN_PROGRESS diagnostic and writes nothing, so status never waits or fails on contention. A pass that cannot create the lock (a read-only, full or removed run root) fails with WORKFLOW_SYNC_LOCK_FAILED and writes nothing; it no longer throws out of inspect, syncRun or the dashboard. The target is not the run root, because proper-lockfile keys its in-process registry by target and the control lock already locks that path. onCompromised is a no-op, because the default throws from a timer and would kill an observer or the eval runner.
• Observations of an unchanged run leave state.json and the journals untouched (workflow-control.ts, workflow-sync.ts). The projection also reports observationOnly, which is true when its only changes are the lease renewed_at/expires_at and the concurrency observed_at and durations. An observe-only pass (status) and any pass over a terminal run skip writing such a projection. Each pass still creates and removes <run>/.workflow-sync.lock. Explicit passes over a live run (syncRun, inspect, why, stats) still renew the lease, so the eval runner and the Modal worker, which pump through syncRun and inspect, keep it current. eval status reads it. Durations stay exact: a skipped write means nothing they integrate over changed.
• Bounded runner queries (smithers.ts). runSmithersInspectionCommand bounds the runner process, never executable preparation, at 120 s by default. ULTRAFUZZ_RUNNER_QUERY_TIMEOUT_MS overrides it, capped at 600 s; 0, off or any other value that is not a positive integer means the default, so the bound cannot be disabled. A timeout becomes the existing failed-query diagnostic, with a message naming the bound.
• Status keeps reporting health (state-export.ts, stats.ts). The observe-only pass sets tolerateInvalidEventStreams and downgrades the transient code set stats already tolerated, now exported once from workflow-sync as TRANSIENT_SYNC_DIAGNOSTIC_CODES for both and extended with WORKFLOW_SYNC_LOCK_FAILED. Any error thrown by the refresh becomes a WORKFLOW_STATE_SYNC_FAILED warning (the snapshot race keeps WORKFLOW_STATE_SYNC_RACED). Other errors the refresh returns keep their severity (see below). getRunHealth passes the evidence it already verified into the first attempt, which skips one full sealed-snapshot verification per status; a retry after a snapshot race reads the evidence again, as every attempt did before.
• Deadline (workflow-control.ts, workflow-sync.ts). Paused runs are left out of deadlineExceeded. WORKFLOW_DEADLINE_CANCEL_FAILED is now a warning, and the next synchronization requests cancellation again.
• Journal and failure text (workflow-sync.ts, events.ts). node-artifacts-verified and node-artifacts-missing are no longer written. They duplicated node provenance and had no reader. Their schema variants stay, so existing journals replay. A node failed by host-side output validation now gets a last_error that prefixes each error with its run-relative path and JSON pointer, for example artifacts/project-discovery/findings.json#/0: must have required property 'status', and never with a host path. Runner-reported failures keep the runner's text, and a diagnostic with no in-run path keeps its bare message. The dead eventProvenanceForTask plumbing is deleted; createEventRecord never persisted it.
• Orphan remediation (state-export.ts). The runner's smithers supervise -r <id> text is rewritten to ultrafuzz resume <run-id>. Resume continues an orphaned run, because orphaned is not an active runner state. The generic "smithers" → "workflow runner" rename runs on the text around that command, so a chosen run ID such as smithers-upgrade-check stays intact.
• Docs. docs/reference/configuration.md gains the ULTRAFUZZ_RUNNER_QUERY_TIMEOUT_MS row and a paragraph on skipped passes, the lock file and its five-minute takeover, WORKFLOW_SYNC_LOCK_FAILED, status warnings and when state.json is left untouched. The workflow_deadline_seconds paragraph from docs: state that workflow_deadline_seconds is only enforced on sync #1157 now covers paused runs and the retried warning, and the supervisor paragraph no longer claims that reporting checks the deadline.

The lock sits in a small wrapper, synchronizeExclusively, that re-enters the unchanged 450-line pass body. The wrapper recognizes its own call through a module-private WeakSet of control objects. This keeps every caller on the locked path under the same public name. It also leaves the body's lines untouched apart from the few this PR changes. That limits textual conflicts with other in-flight edits of the body, and the diff-limited strict lint does not re-flag it (a review confirmed that a plain rename of the body fails lint:strict:ci on complexity and length). Once #1184's lint baseline lands, this can become an exported wrapper plus a private body. The lock is taken before the evidence read, not after it as the work-item spec and analysis proposed. That is safe because acquisition never waits, so it cannot form a cycle with the control lock, and a skipped pass then does no evidence work at all.

Deliberately not built (and why)

• No waiting lock, no dedup of events by id, no read-only status. A skipped pass leaves the run to the pass in progress, so there is nothing to wait for. status staying a pump is the documented way state converges (workflow_deadline_seconds is only enforced while an operator runs a CLI command; unattended runs have no wall clock #1110 depends on it).
• status does not downgrade every refresh error, although why does. A review suggested why's rule. fix(runtime): recover terminal pending resumes #707 deliberately did the opposite for status: it removed a blanket downgrade of failed refreshes and made ok false on any refresh error, and its test getRunHealth reports a terminal product status while workflow health is live requires WORKFLOW_TERMINAL_WITHOUT_FAILED_NODE to stay an error. I ran that test with the blanket downgrade and it fails (true !== false on health.ok). So only the transient set, the lock failure and thrown errors are downgraded, as scoped for this item. A failed runner node query (WORKFLOW_ATTEMPT_INSPECT_FAILED) therefore still fails status and stops --watch here; from its diff, fix(runtime): attempt and usage ledgers never block run synchronization #1186 (w02), which owns that path, makes it a warning where it is raised.
• No abort when the lock is compromised. A pass loses its lock only when its process does not run its 60 s refresh timer for 5 minutes (suspended, stopped, or an event loop blocked that long). Another pass then takes the lock over and both write, which is what happened on every overlap before this PR. Stopping the first pass would need a cancellation path through the whole pass and its writers, which is the kind of fencing layer this item was told not to add. The CHANGELOG states the five-minute condition.
• No revalidation of the workflow link under the lock. Lifecycle writers (resume, replay, fork) do not take the sync lock, so an observe-only pass could already read evidence and write after a link switch on origin/main. Passing getRunHealth's evidence into the first attempt adds only the time between that read and the start of the pass, which is a local divergence check. Retries now read the evidence again.
• The eval runner does not synchronize once more after a skipped pass. A review (not reproduced) noted this sequence: an eval poll is skipped while another pass holds the lock, that pass has written the terminal status but not yet published the terminal report, and then that pass dies. Nothing on the eval path would then publish the report. Scoring is a separate ultrafuzz eval score step that resolves the report when it runs (scoring.ts), so a holder that finishes publishes in time. The residual needs the holder to die inside that window. eval score then fails with EVAL_TERMINAL_REPORT_INVALID rather than silently, and any later synchronization of the run publishes the report. On origin/main the same outcome needed the eval runner's own pass to end just before another pass wrote the terminal status. The runner's final catch-up block is also being deleted by refactor(evals): delete the dead reporter/telemetry pipeline and publication leftovers #1174 (w20a), so a hunk there would conflict.
• Preserve complete structured diagnostics when CLI output is truncated #1152's "empty missing lists fail event-schema validation" is not implemented. The events that carried the empty lists are no longer emitted. Tightening their schema (for example minItems: 1 on missing) would make existing journals unreplayable, and a schema change rotates the validator build identity for in-flight runs. The analysis also found that the issue's "and N more" truncation loses nothing durable: the retained artifact, which ultrafuzz artifact validate re-checks, the sealed graph and tasks.json, and state.json keep the full data.
• No pricing re-fetch change. A model missing from an available catalog is still re-fetched on every sync. According to a review (by reading the code, not reproduced), each re-fetch stamps a new pricing_catalog.fetched_at, so run.json is rewritten on every status of such a run. Work item w02 owns accounting. This is one reason this PR only refers to Make run status authoritative, bounded, and read-only #1145.
• The pre-reset attempt-ledger hook (preserveFailedWorkflowAttemptsBeforeReset) does not take the lock. That area belongs to w02. Lifecycle writers (cancel, pause, resume) also stay outside it. This lock covers sync-versus-sync writes only.
• The deadline cancel command is not bounded. Only read-only queries got a timeout, and cancel is shared with ultrafuzz cancel.
• No change to the opt-in ULTRAFUZZ_OBSERVATION_SYNC_TIMEOUT_MS checkpoints. They are now partly redundant with the per-query bound, but removing them is a separate cut.

Verification

Discriminating evidence for the first round. I copied the new and changed tests onto an origin/main worktree (fbbcf6c5, same sync code as b6dd1da9), compiled them against main's src, and ran them there and on the branch. Two reviewers reproduced this table independently:

Test	origin/main	this branch
a paused run is not cancelled by an observation past its workflow deadline (unit)	fails	passes
re-projecting an unchanged run only advances its observation clock (unit)	fails (observationOnly absent)	passes
repeated observations leave an unchanged finished or orphaned run byte-identical	fails: "finished: a repeated status rewrote run state"	passes
status reports a runner query that stops answering instead of waiting on it	fails: "status waited on a runner query that stopped answering" (40 s guard)	passes (3 s bound)
a synchronization that finds another in progress skips it without waiting or writing	fails: "status waited on the synchronization in progress"	passes: skipped pass writes nothing, exactly one node-synced
a node whose outputs fail validation names each failing artifact in its last error	fails: last_error has no location; with the assertions reordered, node-artifacts-missing with "missing":[] is emitted	passes
a failed deadline cancel is a warning and the next status requests it again	fails: WORKFLOW_DEADLINE_CANCEL_FAILED has severity error	passes, and cancel is requested again on the second poll
an explicit synchronization keeps renewing a live run's controller lease	passes	passes (guard for the eval and Modal pumps)

Review round. Each new or changed test ran against origin/main (b6dd1da9) and against the previous head (689e92e0), both in separate worktrees with the final test file compiled against that commit's src:

Test	origin/main	previous head 689e92e	this branch
a synchronization that cannot take its lock is reported instead of thrown (new; run root chmod 0555, registered only when not running as root)	fails: no WORKFLOW_SYNC_LOCK_FAILED; main has no sync lock, and inspect returns only WORKFLOW_CONTROL_EVIDENCE_INVALID warnings	fails: getRunStatus rejects with EACCES: permission denied, mkdir '<run>/.workflow-sync.lock'	passes: inspect and status are ok with a WORKFLOW_SYNC_LOCK_FAILED warning, and syncRun returns ok:false with it
getRunHealth accepts strict 0.35 orphan, … (run ID now smithers-034-shapes)	fails: the reason still names supervise	fails: "ultrafuzz resume workflow runner-034-shapes"	passes
status keeps reporting runner health when run-state synchronization fails (now also asserts the WORKFLOW_STATE_SYNC_FAILED warning for the torn ledger)	fails: WORKFLOW_EVENTS_FAILED is an error	passes (the catch-all was already there; the assertion keeps the scenario honest)	passes
repeated observations … (precondition pinning a finished run's active lease removed)	—	—	passes

The retry change (a retry reads the evidence again) has no dedicated test; making a lifecycle command switch the link between two attempts of one pass is not something the fake runner can stage. The existing snapshot-race test covers the retry path and passes.

What else I ran on the final tree:

• 29 runtime.test.ts tests on the final commit, all passing: the nine new or changed ones above, every other getRunHealth* and getRunStatus* test (including fix(runtime): recover terminal pending resumes #707's getRunHealth reports a terminal product status while workflow health is live), the observer snapshot-race test (listRuns, observers and status re-read live run documents replaced while they were read), the four divergence-tolerance status tests, and both existing deadline tests (syncRun cancels a nonterminal workflow at its durable workflow deadline, syncRun honors cancellation and an overall deadline before terminal synchronization).
• workflow-control.test.js: all 14 pass.
• CLI: status --watch --json keeps a failing poll on one NDJSON line, status surfaces a terminal product and live workflow lifecycle divergence, status --watch stops immediately on a degraded verdict even while product state is nonterminal, stats falls back to unchanged local evidence when workflow event output is malformed, and status recommends ultrafuzz why instead of the engine command pass. In the first run, two of them failed at their ultrafuzz run launch step (exit 1, before any status or stats code ran; I did not capture its output) while the machine was heavily loaded; both passed when rerun. Separately, two runtime tests once failed inside startRun with WORKFLOW_SUBMISSION_FAILED ("workflow execution file dependencies/… changed while reading", the host flake a reviewer also hit) and passed when rerun.
• npx prettier --check and npx eslint on the changed files; CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/main pnpm -w lint:strict:ci; pnpm --filter @ultrafuzz/{runtime,cli,artifacts} typecheck; pnpm -w knip; node scripts/docs-check.mjs. All clean. The runtime src and tests also typecheck at each of this round's commits.

Run in the first round and not re-run on this head: the five finalization tests (syncRun accepts canonical findings…, …rejects a valid output swap…, …persists a schema-valid failed node when controller manifest publication fails, …marks task-output validation failures for terminal disposition, …rejects workspace-mirrored outputs…), lifecycle-inspection.test.ts tests 1–32, observation-deadline.test.ts and observation-snapshot.test.ts (12/12), dynamic-lifecycle.test.ts dynamic child failure, skip, and timeout keep strict joins blocked with durable terminal state, and artifacts event-record.test.ts (3/3, old node-artifacts-* fixtures still validate).

What I did not run: the full runtime, CLI, evals or Modal suites; the Bun adapter contracts (untouched); and a multi-process soak of concurrent syncs. The in-process gate test is deterministic; the analysis's 8/8 multi-process soak was on a prototype, not on this code.

CI: External static analysis (and release-gates, which depends on it) fails on this PR because Super-Linter's MD013 flags every long line in CHANGELOG.md, including lines that predate this PR. Every PR that edits CHANGELOG.md hits it; #1181 and #1184 address it, so this PR adds no third exemption and should be rebased after one of them lands.

Risk / compatibility

• Behaviour changes operators can see. status now exits 0 with warnings when its refresh hits a transient runner failure, a lock it cannot take, or a thrown bookkeeping error. A paused run is no longer cancelled at its deadline. A runner query slower than 120 s now fails unless ULTRAFUZZ_RUNNER_QUERY_TIMEOUT_MS is raised. events.jsonl stops receiving node-artifacts-* records. I found no reader of those records in the repo, and old journals still replay. stats now tolerates WORKFLOW_SYNC_LOCK_FAILED like its other transient codes, so on a run root it cannot write to it should fall back to local evidence with a warning (by reading the code; not run).
• Stale lock. If a process holding the lock is SIGKILLed, the lock directory stays until it goes stale after 5 minutes. Until then passes skip with WORKFLOW_SYNC_IN_PROGRESS, and status still reports runner health. A normal exit, SIGINT or SIGTERM removes it through proper-lockfile's exit hook.
• State freshness. status no longer renews the lease of a live run. Explicit passes and the eval/Modal pumps still do. Nothing in the runtime decides on the lease timestamps; evals/status.ts reads them for display heuristics, fed by the eval runner's explicit pass.
• In-flight runs. No schema, validator or sealed-template change. The new last_error format applies only to new finalizations; a node already finalized as an immutable output failure is not re-finalized.
• Merge order. fix(runtime): attempt and usage ledgers never block run synchronization #1186 (w02) deletes preserveFailedWorkflowAttemptsBeforeReset next to where synchronizeExclusively is inserted, a textual conflict to resolve by keeping the wrapper. From its diff (not run together with this branch), fix(runtime): attempt and usage ledgers never block run synchronization #1186 also catches accounting errors as a WORKFLOW_ACCOUNTING_FAILED warning. After both land, the torn-ledger scenario above would then probably stop reaching the thrown-error catch-all, and its new WORKFLOW_STATE_SYNC_FAILED assertion would fail loudly rather than pass without testing anything; whichever PR lands second needs to point that scenario at another thrown error.

Refs #1145
Closes #1152
Refs #1110

🤖 Generated with Claude Code

Confidence Score: 2/5

The PR does not appear safe to merge while the three outstanding synchronization and workflow-link findings remain unresolved.

Findings

1.  Lost lock permits overlapping writes ▶
2.  Stale workflow evidence gets written ▶
3.  Health can mix workflow links ▶

Fix with agent prompt

### Issue 1
packages/runtime/src/workflow-sync.ts:undefined-1386
If a synchronization process is suspended long enough for its five-minute lock to become stale, another process can acquire the lock while the first pass is still active. This no-op handler lets the first pass continue writing state and journals, so both passes can finalize the same node concurrently. Stop the pass when it loses lock ownership without throwing from the timer.

### Issue 2
packages/runtime/src/workflow-sync.ts:1435-1439
`getRunHealth` reads workflow evidence before acquiring the synchronization lock. If a concurrent resume or replay switches this run to a new workflow link in between, this pass uses the old evidence without checking the current link. It can then write the old workflow’s state into the new link’s `state.json` and journals. Revalidate the link under the lock before writing.

### Issue 3
packages/runtime/src/state-export.ts:undefined-386
If a concurrent resume, replay, or fork replaces run state during synchronization, this retry can reload and synchronize the new workflow link. `getRunHealth` then queries the original link for its verdict and counts, so one response can combine new-link local state with old-link health. Keep the synchronization and health query bound to the same link.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR serializes run-state synchronization, avoids observation-only state writes, bounds read-only runner queries, and keeps status reporting runner health when synchronization fails. It also changes deadline handling and improves failure and orphan-remediation text. Since the previous review, the author removed the changelog entry; the runtime changes are unchanged.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  O[Observer or explicit sync] --> L{Run sync lock available?}
  L -->|Held elsewhere| S[Skip pass; report in progress]
  L -->|Acquired| E[Read workflow evidence]
  E --> P[Project and persist eligible changes]
  P --> U[Release lock]

Loading

_{Reviews (3) · Last reviewed commit: "chore: move the changelog entry to the c..."}

[greptile-apps]

Lost lock permits overlapping writes If a synchronization process is suspended long enough for its five-minute lock to become stale, another process can acquire the lock while the first pass is still active. This no-op handler lets the first pass continue writing state and journals, so both passes can finalize the same node concurrently. Stop the pass when it loses lock ownership without throwing from the timer.

Knowledge Base Used: Runtime orchestration

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/runtime/src/workflow-sync.ts
Line: 1366

Comment:
**Lost lock permits overlapping writes** If a synchronization process is suspended long enough for its five-minute lock to become stale, another process can acquire the lock while the first pass is still active. This no-op handler lets the first pass continue writing state and journals, so both passes can finalize the same node concurrently. Stop the pass when it loses lock ownership without throwing from the timer.

**Knowledge Base Used:** [Runtime orchestration](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/runtime-orchestration.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Stale workflow evidence gets written getRunHealth reads workflow evidence before acquiring the synchronization lock. If a concurrent resume or replay switches this run to a new workflow link in between, this pass uses the old evidence without checking the current link. It can then write the old workflow’s state into the new link’s state.json and journals. Revalidate the link under the lock before writing.

Knowledge Base Used: Runtime orchestration

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/runtime/src/workflow-sync.ts
Line: 1415-1419

Comment:
**Stale workflow evidence gets written** `getRunHealth` reads workflow evidence before acquiring the synchronization lock. If a concurrent resume or replay switches this run to a new workflow link in between, this pass uses the old evidence without checking the current link. It can then write the old workflow’s state into the new link’s `state.json` and journals. Revalidate the link under the lock before writing.

**Knowledge Base Used:** [Runtime orchestration](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/runtime-orchestration.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Health can mix workflow links If a concurrent resume, replay, or fork replaces run state during synchronization, this retry can reload and synchronize the new workflow link. getRunHealth then queries the original link for its verdict and counts, so one response can combine new-link local state with old-link health. Keep the synchronization and health query bound to the same link.

Knowledge Base Used: Runtime orchestration

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/runtime/src/state-export.ts
Line: 386

Comment:
**Health can mix workflow links** If a concurrent resume, replay, or fork replaces run state during synchronization, this retry can reload and synchronize the new workflow link. `getRunHealth` then queries the original link for its verdict and counts, so one response can combine new-link local state with old-link health. Keep the synchronization and health query bound to the same link.

**Knowledge Base Used:** [Runtime orchestration](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/runtime-orchestration.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.