Skip to content

MIT-680: pair a test pass with matched work on the hook's clock - #176

Merged
ZachFara merged 3 commits into
feature/eval-setupfrom
fix/MIT-680
Nov 12, 2026
Merged

ZachFara merged 3 commits into
feature/eval-setupfrom
fix/MIT-680

Conversation

@ZachFara

Copy link
Copy Markdown
Owner

What changed

  • Hooks stamp each test run with this machine's time (hook_client.on_tool_use). RemoteBackend sends it as test_run_at beside test_run, because older servers accept exactly {command, exit_code} inside test_run. The local backend stores it directly.
  • The hooks API validates test_run_at (an ISO time with a UTC offset) and stores it as at in the test_run event (web/hooks_api.py).
  • store.task_events.test_runs reads it as ObservedRun.client_at, and _follows_matched_work (service/hooks.py) compares matched_at with client_at. Runs from older hooks don't have client_at, so they fall back to the store's clock, which is how it worked before.
  • Decision 0036's "Known limit" note gets an amendment saying this is now fixed.
  • Files touched outside the ticket's area: domain/outcomes.py, store/task_events.py and web/hooks_api.py carry the new field, and test_hook_observe.py's events helper now pops (and checks) the run's at before comparing.

Why

Ticket: MIT-680. Root cause: a run's time was the store's clock (task_events.created_at, clock_timestamp()), but matched_at comes from the hook's spool, which uses this machine's clock. Pairing (decision 0036) credits a pass only if run.at > matched_at. When the store's clock is behind this machine's by more than the gap between the matched edit and the test run, the pass is dropped and the outcome stays unknown. Measured here: Postgres in Docker Desktop reads about 1 ms behind the Mac. In the end-to-end test that gap is only a few milliseconds, so it failed occasionally. Decision 0036 already listed this as a known limit and named this fix. A hosted server whose clock is seconds off would hit it all the time.

It is not pollution between tests: the failing run's log showed both tool-use calls posted, and the Bash call posts only when a test run was detected. With the run recorded and used = yes, clock pairing is the only path that leaves unknown.

How it was tested

  • New test_a_pass_after_the_fix_counts_when_this_machines_clock_runs_ahead: this machine's clock is set a minute ahead (spool and hook client). Before the fix it failed with exactly the ticket's ('yes', 'unknown') == ('yes', 'tests_passed'). It passes after.
  • New test_older_servers_still_accept_the_test_run: the wire's test_run keeps exactly {command, exit_code}, and test_run_at is an aware time.
  • Two new bad-input cases in test_only_derived_shapes_are_accepted: a non-string test_run_at and a naive test_run_at.
  • make check ran four times in total (once in pre-commit, then three times in a row), all passing: 1571 tests.

Learned

  • Postgres in Docker Desktop runs about 1 ms behind the Mac's clock. Any comparison between a client-side time (spool, datetime.now) and a database time (now(), clock_timestamp()) can flip in tests. Keep each comparison on one clock.
  • The hosted API's body validators check key sets exactly (set(value) == {...}). To send a new field to them without breaking older servers, put it at the top level of the body, where older servers ignore unknown keys (doc_lookup, test_run_at).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y

ZachFara and others added 3 commits November 11, 2026 17:33
matched_at is this machine's clock and a run's time was the store's
(clock_timestamp()), so a store clock behind this one dropped the pass's
credit and left the outcome unknown. Postgres in Docker Desktop runs about
a millisecond behind the Mac, which failed the hosted end-to-end test once.

Hooks now stamp each test run with their own time, sent as test_run_at
beside test_run (older servers check test_run's keys exactly), and pairing
compares that with matched_at. Runs from older hooks keep the store's time.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
test_run_at kept the client's offset. Python accepts offsets with
fractional seconds (+05:00:30.5) that Postgres can't cast to timestamptz,
so a stored one would fail every later Stop for the task. Normalize to UTC
before storing, and refuse times that overflow when shifted to UTC at the
year edges (0001-01-01T00:00:00+01:00) with a 400.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
@ZachFara
ZachFara merged commit b4af7db into feature/eval-setup Nov 12, 2026
2 checks passed
@ZachFara
ZachFara deleted the fix/MIT-680 branch November 12, 2026 02:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant