[kyle-sexton]

No related issue: GITHUB_TOKEN trip 05:38Z attribution (reconcile-lost-runs catch-up)

Summary

GITHUB_TOKEN counter B tripped at 05:38Z on 2026-11-03. In that hour maintenance-reconcile-lost-runs spent 356 calls over 4 runs. The lanes kill switch was on 04:34-04:55Z, so the 04:15Z and 04:38Z runs stopped with 1 call each: run 41579246488 logs check-kill-switch: proceed=false reason=kill-switch and the lanes kill switch stops this run; nothing read or re-run. They were not paced.

• Run 41580864978 (04:56Z, 163 calls) did the recovery work the switch had left. It made 13 rerun-failed-jobs calls for verdict runs (by design these take no place under the cap), 3 lane and review re-runs, and about 24 first reads of runs the switch had cancelled before they started, which it then settled. It also read 12 re-runnable pr-verify runs and deferred them with all 3 re-run places taken.
• Run 41581313634 (05:02Z, 99 calls) was a workflow_dispatch by hand, so it did not go through pace.
• Run 41585026094 (05:38Z, 105 calls) read the same deferred runs again: attempts/{n}/jobs=15, of which about 12 were pr-verify runs still waiting for a place (10 deferred, 2 re-run).

The defect: holdReason reads a lane run's attempt-1 jobs before placement. Only a held run is settled, so a re-runnable run that is deferred is read again on every pass until it gets a place. The file's own rule says attempt-1 jobs of a completed run never change.

Fix

holdReason now takes a lane run still at attempt 1 from the settled-runs file's existing job carry (jobs, keyed id:attempt) while the run listing shows the same head and updated_at. It carries a whole read whose every job's steps listed, using the same checks as the PR-head job carry.

• A re-run changes the attempt and updated_at, which drops the carry.
• Act still re-reads each run before it writes.
• The settled-run rules are unchanged, so SETTLED_VERSION stays 3. The entries keep the existing shape, and a lane run's id never collides with a CI run's.

Call delta:

• Per pass while a lane backlog drains: about -10 to -12 calls, one attempts/1/jobs read per deferred run, plus annotation reads for any cut job.
• Trip hour: about -24 of 356 (around 7%), from the 05:02Z and 05:38Z passes.
• The 04:56Z peak is unchanged, because first reads and verdict re-runs are needed work.
• Steady state with no deferred lane runs: 0.

Verification

• New test a lane run deferred for want of a place is not read again while it stays at attempt 1:
  • Pass 1 re-runs 3 of 5 lost pr-verify runs and reads all 5.
  • Pass 2 makes 0 attempts/1/jobs reads and re-runs the 2 deferred runs.
  • A changed updated_at reads the run again.
  • Without the change, pass 2 reads [201, 202] and the test fails.
• node --test over reconcile.test.mjs, infra-failure.test.mjs and component-status.test.mjs: 177 pass, 0 fail.

Related

• Per-run readings: the gh-job-calls monitor (local).
• Not changed here: each pass has a fixed cost of about 21 workflows/{x}/runs listings (9 statuses x 2 lanes, plus intake, verdict and the carry lookup) and 22 head_sha listings. That is the larger steady-state cost. The status split is deliberate (LANE_LIST_STATUSES).

🤖 Generated with Claude Code

https://claude.ai/code/session_01VcupomBPg7kuobwLoNpxEW

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T09:11:45.140569Z	3dcf9a4	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-review: 1 inline finding(s) on 3dcf9a4deaa491e90493079dfe3d8b4abf3a29d4; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 3dba34543a1a52a39912c7d88657519e9fa67c9c: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #12752 (incremental, 3dcf9a4d..3dba3454)

Scope: 1 of 1 file in scope, read in full. The only change is in .github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs: one hunk at L2253-L2267 that adds a pass in which the deferred runs stay deferred.

Earlier finding: fixed. The new pass adds the case the earlier advisory said was missing (discussion_r4455949030): a run deferred twice. Checked against reconcile.mjs at head:

• No room for re-runs. running(203..205) has status: "in_progress" and run_attempt: 2, so each one counts toward inFlight (reconcile.mjs#L1055). That leaves room at 0 (L1185), so 201 and 202 stay deferred and held.reruns is [].
• The carry-hit branch now runs. lost(201) and lost(202) keep the same head_sha/updated_at at attempt 1, so the pass uses the carried read. The assertion on held.settled.jobs (ids 201 and 202 at attempt === 1) matches the { id, attempt: 1, ... } entry shape (L604). Removing keep.set(key, carried) would now fail that assertion.
• A third pass reads nothing. The third pass is fed held.settled and asserts no job reads, which checks that the kept entry still works one pass later.
• The new pass uses the right carry. It is fed first.settled, not second.settled, so it models pass 2 going the other way. The later "listing changed" check still uses first.settled, so it is not affected.

Findings: none.

Not verified: this lane has no shell, so I did not run the tests. The owner's reply says the new assertion fails when that line is commented out and passes with it. That is author-claimed and unverified.

Other checks:

• Settled threads: the one settled thread is the fix-now reply to the finding above. This push answers it.
• Plugin manifests: none changed.
• Trusted paths: the changed file is not on the trusted-path list.
• Spend: the change is test-only, so it adds no spend.

[github-actions]

claude-security-review reviewed 3dba34543a1a52a39912c7d88657519e9fa67c9c: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #12752 (incremental since 3dcf9a4): no security issues found

Scope. Since 3dcf9a4deaa491e90493079dfe3d8b4abf3a29d4, this push changed one file, .github/scripts/maintenance-reconcile-lost-runs/reconcile.test.mjs. I read its whole hunk (+15 lines at L2253). None of the trusted-path globs match this file. The only listed path in that directory is infra-failure.mjs.

What changed. The new lines are test code only. They extend the existing test a lane run deferred for want of a place is not read again while it stays at attempt 1 with a case where a run is deferred twice:

• Three runs (203–205) are re-runs still in progress at attempt 2.
• The deferred runs 201 and 202 make no attempts/1/jobs reads and get no re-runs.
• 201 and 202 keep their carried attempt-1 entries in settled.jobs.
• A third pass, fed that carry, also makes no reads.

Security surface. None. No production code, workflow, token request, kill-switch read, merge hold, trusted-actor list or instruction surface changed. There are no plugin manifest changes. The test only checks the carry and freshness behaviour that the prior round reviewed in reconcile.mjs, and it changes none of that behaviour. Skip gate 2 applies.

The prior-findings file holds only this lane's earlier no-findings summary, so nothing is repeated here. No settled threads apply.

Spend: none. The change is test-only and adds no API calls or lane runs.

critical-findings: 0

[github-actions]

Post-merge mutation testing

• Range: dcae0f5..daaefff
• Merged pull request: #12752
• Outcome: reported

Killed 5, survived 2, timeout 0, no coverage 0, invalid 0, not run 0.
Covered-code score 71%; mutation score 71%.

File	Killed	Survived	Timeout	No coverage	Invalid	Not run
.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs	5	2	0	0	0	0

Surviving mutants

Location	Operator	Disposition	Cause
.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:593	other	equivalent	none
.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:603	ROR	productive	input-gap

The run's summary has the full report.