fix(artifacts): event appends and materialize keep working after the host clock steps back

[aviggiano]

Problem

• After a backward clock step, every event append fails, and materialize stops halfway (Greptile, P1, raised on fix: symlinked project roots, locale-independent ordering, clock-skew-tolerant audit journals and other small verified bugs #1195). fix: symlinked project roots, locale-independent ordering, clock-skew-tolerant audit journals and other small verified bugs #1195 let the clean, materialize and dashboard audit journals accept a clock step. It kept the run event journal's rule that timestamps never decrease, which dates from feat: add strict producer-visible JSON contracts with release-lane fix #558. Suppose the wall clock steps back behind a run's last event (an NTP step, a VM snapshot restore, a manual correction). Until the clock passes that event again, every appendEvent call throws event journal timestamps are not ordered at record N.
  • ultrafuzz materialize --confirm copies the files and appends its audit record, then throws on the event append. The run journal gets no materialize-selection event. A rerun returns MATERIALIZE_DESTINATION_EXISTS unless --allow-overwrite is given. The repro against origin/main shows exactly this (run on 2cacf4c; the artifacts and runtime sources are unchanged through 142ba80).
  • Sync, resume, replay, fork, pause, cancel and the workflow-link commit append events the same way. In that window, each of them fails as soon as it has an event to record.

Change

• appendEvent (packages/artifacts/src/events.ts). It still builds the record before reading the journal, as before. Once the read has found the journal's final event, the record is rebuilt one millisecond after that event when two things hold: the caller passed no timestamp, and the record's time is earlier than the final event's. Otherwise the prebuilt record is appended unchanged.
  • The record's time is earlier after a backward clock step. It is also earlier when another process appended an event between this build and the read. origin/main rejected that append as not ordered (see Verification).
  • Timestamps still never decrease, so the journal's ordering rule and its readers do not change.
  • The stamped event is later than every recorded one, so an event whose content matches the final event's still gets a new ID. Reusing the final timestamp would make it a rejected duplicate, because the event ID hashes the timestamp.
  • A timestamp the caller passes explicitly is still rejected when it is earlier. No production caller passes one. All 19 appendEvent calls in package sources omit it, checked by walking their argument objects with the TypeScript parser. The one spread argument is typed Omit<…, "nodeId" | "runId" | "timestamp">.
• Still one read per append, and the same fenced window. The final event comes from the read that already fences the write. refactor(artifacts): delete never-dispatched gates, production-dead code, and the unread event index #1181 found that this byte read is the part of an append's cost that still grows with the journal, so a second read would add to it.
  • appendStrictJsonlRecordsAfterTail becomes appendStrictJsonlRecordAfterTail (packages/artifacts/src/strict-jsonl.ts). It takes a builder in place of the records array, calls it with the final record, and returns the record it appended. appendEvent was its only caller (git grep).
  • The duplicate window is unchanged: the trailing records that share the prebuilt record's timestamp. A rebuilt record's time is later than the final event's, so the window then holds only the final event, and no recorded event can share the new time.
  • Building a record redacts its payload. That costs about 4 ms for a 1.8 KB materialize-selection payload. The canonicalization origin/main already runs between the read and the fenced write costs 0.06 ms for the same payload. Because the record is prebuilt, redaction stays out of that window, as on origin/main. Only a rebuilt record is built inside it.
• Docs. docs/reference/artifacts-reports.md says that event times can run ahead of the wall clock after a backward clock step.
• materialize.ts is unchanged.

Size: 2 source files +20/−11, 2 test files +48, 1 doc +6/−4.

Deliberately not fixed

• Explicit earlier timestamps. appendEvent still rejects a caller-supplied timestamp that is earlier than the final event. That timestamp is the caller's statement of when something happened, and rewriting it would record a false time. Only tests pass one, and an event append refuses a repeated or out-of-order event without changing the journal still pins the rejection.
• Materialize's write order. materializeSelection still copies, then appends the audit record, then appends the event. This change removes the finding's trigger. An event append that fails for another reason, such as a full disk or a torn journal, still leaves the copied files and the audit record behind. Making the three steps atomic is a separate change.
• Concurrent appends can lose or tear records (pre-existing). appendBytesDurableAt checks the file size, then writes at that offset, with no lock. Two processes can both pass the check and write at the same offset. The result is a lost record, or a leftover tail that makes every later append and replay of that journal fail. The entry points take different locks or none. Sync takes its .workflow-sync lock, and launch takes the control lock. Resume, replay and fork take the lifecycle lock, and cancelRun and materializeSelection take none. So their appends to one events.jsonl can overlap.
  • In the two-writer harness below, the journal tore in 5 of 7 rounds on origin/main and in 3 of 7 here. In the rounds that did not tear, 0 to 6 records per round were missing although both writers had reported success.
  • The same helper backs the other strict JSONL journals. The fix is a per-journal lock held across the read, validation and write, and belongs in its own change.
• The evals recovery-equivalence clock comparison. See Risk.

Verification

Both new tests fail on origin/main and pass on this branch. For the origin/main runs, the two source files came from origin/main and the @ultrafuzz/artifacts dist was rebuilt.

Test	On origin/main	Here
artifacts event appends take the wall clock and keep working after it steps back behind the journal	Error: event journal timestamps are not ordered at record 2	passes. An append after an hour-old final event takes the current time. The two new, identical events after a future-dated one are stamped 1 ms and 2 ms after it, and the whole journal replays.
runtime materializeSelection records its event after the host clock steps back behind the run's events	Error: event journal timestamps are not ordered at record 2, thrown by materializeSelection	passes: ok: true, and the journal's last event is the one the result names

Ablations, each run on this branch with one line changed:

• Dropping record.timestamp < final.timestamp from the condition, so every append after the first is stamped 1 ms after the final event: the artifacts test fails with stamped 2026-09-29T12:54:00.806Z, wall clock was 2026-09-29T13:54:00.805Z.
• Stamping the final event's own timestamp instead of 1 ms after it: the artifacts test fails with event journal contains a duplicate identity "evt-…" at record 2.

The verifier's repro script calls materializeSelection after a future-dated event. Against this branch's build, the first call returns ok: true and records its event, and a retry returns MATERIALIZE_DESTINATION_EXISTS. On origin/main, the first call throws event journal timestamps are not ordered at record 2 after copying.

Two-writer contention. Each round, two processes each appended 300 materialize-selection events with 1.8 KB payloads to one journal. There were 7 rounds per build, alternating builds.

• Fence failures (append path changed size) per round: 9 to 51 on origin/main, 15 to 52 here.
• timestamps are not ordered failures: every origin/main round had them (3 to 28 in the 4 rounds that counted them). No round here had any.

Suites run on this branch after merging origin/main 142ba80:

• artifacts: the full suite, 325/325.
• runtime:
  • materialize.test.js and audit-contracts.test.js: 9/9.
  • 18 runtime.test.js tests of sync, status, health, resume, launch and workflow linking. 15 of them name events.jsonl, replayEvents or appendEvent directly: 18/18.
  • verified-output.test.js, whose fixtures append events, and clean.test.js: 54/54.
  • lifecycle-inspection.test.js, including the cancelRun and getRunTimeline tests: 52/52.
  • From dynamic-lifecycle.test.js, the two tests that read events.jsonl: 2/2.
• Checks, all passing:
  • npx prettier --check and npx eslint on the changed files.
  • CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/main pnpm -w lint:strict:ci.
  • pnpm -w lint.
  • pnpm --filter @ultrafuzz/artifacts --filter @ultrafuzz/runtime typecheck.
  • pnpm -w knip.
  • node scripts/docs-check.mjs.

Risk / compatibility

• Journal format. Nothing changes. Existing journals replay as before. Journals written here still have timestamps that never decrease, so older builds read them.
• When behaviour changes. Only when the prebuilt record's time is earlier than the journal's final event: after a backward clock step, or when another process appended between the build and the read. Those appends used to throw. Now they are recorded 1 ms after the final event. After a clock step, that is ahead of the wall clock by up to the size of the step, until the clock catches up. Every other append is built and written as on origin/main.
• Copied event times. controller_invoked_at, lifecycle_result_at and link_event_at take the time from the record appendEvent returns. Their readers compare it for equality, so they still match. The controller-generation journal checks event_at ordering against wall-clock times, but nothing has written that journal since fix(runtime): restore native Smithers continuation #961.
• Evals recovery-equivalence scorer. Apart from that legacy check, it is the only reader found that compares an event time with another clock (grep for .timestamp, controller_invoked_at and link_event_at). It rejects a node attempt whose started_at (taken from the runner's NodeStarted event) is earlier than its generation's controller_invoked_at. Take an eval run that is resumed while the clock is behind. Its stamped controller_invoked_at can be later than the attempts that resume started, so the scorer rejects that run's lineage. On origin/main the resume itself failed. The eval runner never resumes runs itself: packages/evals/src has no resumeRun or replayRun call. The campaign path does not read this scorer.
• Filters and display. queryEvents since/until filters and the dashboard show the stamped time for such events.
• API. appendStrictJsonlRecordAfterTail replaces appendStrictJsonlRecordsAfterTail in @ultrafuzz/artifacts, a private workspace package. refactor(artifacts): delete never-dispatched gates, production-dead code, and the unread event index #1181 added the old name, and no release tag contains it.
• Overlap. No wave-3 branch (x01–x05) or open PR (feat(runtime)!: run lifecycle commands from the pnpm-patched install instead of per-command npm installs (#921 step 1) #1201, feat(topology)!: a failed property lens no longer skips the rest of the campaign #1198, refactor!: remove per-node cloud execution (execution.mode = "cloud") #1197, fix(runtime): record final-report producer selections in the run instead of querying smithers #1183, feat(runtime): let agents record roadblocks in a run-local friction log #1162) touches events.ts, strict-jsonl.ts, materialize.ts or the two test files. That was checked against each branch's diff and each PR's file list; fix(modal): keep the model-work flag when a model node has no run-state record #1202 has merged. x01, feat(topology)!: a failed property lens no longer skips the rest of the campaign #1198, refactor!: remove per-node cloud execution (execution.mode = "cloud") #1197 and fix(runtime): record final-report producer selections in the run instead of querying smithers #1183 edit other sections of docs/reference/artifacts-reports.md, from line 180 on. This branch edits lines 38–47. A trial git merge-tree against each merges that file cleanly.

Changelog entry

Run event appends no longer fail after the host clock steps back behind a run's last event. Until the clock catches up, a new event is stamped one millisecond after the last one. materialize --confirm now records its event instead of failing after it has copied files, and sync, resume and cancel keep working.

Greptile follow-up

• Kept: an event stamped after the final one can be ahead of the wall clock (comment). This happens only after the host clock steps back behind the journal. Without this change, the same append throws timestamps are not ordered, so the command fails outright. Greptile's scenario also needs an operator to resume an eval's run by hand in that window, because the eval runner never resumes runs (packages/evals/src has no resumeRun or replayRun). The recovery-equivalence risk is listed under Risk.
• Kept: final timestamps with a UTC offset (comment). Not reachable. Every production appendEvent call omits timestamp, so every recorded event carries a toISOString() value in Z form. The only explicit timestamps come from tests, and an earlier explicit timestamp is still rejected.

🤖 Generated with Claude Code

[greptile-apps]

Future timestamps reject valid attempts

When a watched eval run resumes after the host clock steps back, this branch can stamp the controller invocation ahead of the wall clock. Attempt started_at values still come from Smithers NodeStarted events. If an attempt starts before the clock catches up, the recovery-equivalence check treats it as preceding its controller submission, rejects valid lineage, and prevents the eval row from being recorded as terminal.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/artifacts/src/events.ts
Line: 1192-1193

Comment:
**Future timestamps reject valid attempts**

When a watched eval run resumes after the host clock steps back, this branch can stamp the controller invocation ahead of the wall clock. Attempt `started_at` values still come from Smithers `NodeStarted` events. If an attempt starts before the clock catches up, the recovery-equivalence check treats it as preceding its controller submission, rejects valid lineage, and prevents the eval row from being recorded as terminal.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Offset timestamps still block appends

If the journal ends with a valid timestamp using a positive UTC offset, this conversion can produce a UTC string that sorts before the final timestamp even though it represents a later instant. For example, 2026-09-30T16:00:00+05:00 becomes 2026-09-30T11:00:00.001Z. Event history compares the strings, so an append after a backward clock step still throws timestamps are not ordered.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/artifacts/src/events.ts
Line: 1193

Comment:
**Offset timestamps still block appends**

If the journal ends with a valid timestamp using a positive UTC offset, this conversion can produce a UTC string that sorts *before* the final timestamp even though it represents a later instant. For example, `2026-09-30T16:00:00+05:00` becomes `2026-09-30T11:00:00.001Z`. Event history compares the strings, so an append after a backward clock step still throws `timestamps are not ordered`.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.