pr-pipeline: wake pr-merge per PR on pr-verify completion and make the cron a backstop

Owner: poteto (pr-merge). Measure before building.

Summary

The longest wait on the PR path is between a PR becoming ready and entering the merge queue. Measured over 7 days: p50 0.80 h, p90 7.41 h. Queue entry to merge is p50 0.11 h. Source: audit comment, gate #13.

pr-merge wakes only on three things (.github/workflows/pr-merge.yml:85, :93-98 at cb8429f):

A 15-minute cron. It ran 607 times in 7 days, 173 run-hours, p50 16.1 min per run.
unlabeled. It ran 1,011 times in 7 days, 206 run-hours, mostly waiting on the shared concurrency group. A 2026-11-01 sample showed a run doing 26 s of work after 21 min queued.
push.

Proposal: wake pr-merge for one PR when pr-verify completes on that PR's head, using a per-PR concurrency group. Then slow the cron to a backstop.

The stated reason this was not done

pr-merge.yml:91-95 says the lane "does not wake on each CI or review completion: those arrive faster than runs finish, which would overflow the concurrency queue below." Any design must answer this. Possible answers:

A per-PR group, so one PR's wakes collapse into each other and never queue behind other PRs.
Waking only on pr-verify's completion, not every CI or review completion.

Prior cost cuts on event wakes

pr-merge event wakes have been cut twice on cost. Rework must not bring that cost back.

Baseline 2026-10-31T08:45Z (hub baselines.log). There were 237 pull_request_target runs in 24 h. 138 were superseded and 78 gated at a median of 47 calls each, and none enqueued or merged anything. Dropping edited and ready_for_review removed about 200 runs, 3,591 github-token calls and 2,025 lanes-app calls a day.
perf(pr-merge): stop waking on edited and ready_for_review #11493.

Done when

A baseline is recorded before any change: ready -> first queue entry p50/p90, pr-merge runs per day by event, and github-token and lanes-app calls per run.
The design answers pr-merge.yml:91-95 (concurrency overflow) with a measured or reasoned bound on queued runs.
pr-merge wakes per PR on pr-verify completion, and the schedule becomes a backstop.
Ready -> queue p50 and API calls per day are re-measured against the baseline and posted here. Calls per day must not exceed the 2026-10-31 post-cut level.

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: High
Uncertainty: Knowable
Flags: none
Blocked by: none

Understood: wake pr-merge for a single PR when pr-verify completes on its head, using a per-PR concurrency group, and slow the 15-minute cron to a backstop. The aim is to cut the ready-to-queue wait (p50 0.80 h, p90 7.41 h) without raising API calls above the 2026-10-31 post-cut level.

The issue already names the design (per-PR group, pr-verify completion only), so I found nothing open that needs a decision. The measurement items (baseline and re-measure) are for a later step to confirm.

I found no open issue that asks for the same thing. #12284 is the referenced audit.

_{Automated triage, applied by the lanes App.}

Not admitted to the issue lane: a criterion names .github/workflows/pr-merge.yml, which the issue lane cannot push (patch-protected-path); a builder session takes it. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.

BEFORE baseline (Done-when 1). Measured 2026-11-02T18:25Z from the hub's runs cache and the jobs' api-calls annotations. Windows: 7d is 2026-10-26T00Z to 11-02T00Z; 24h is 11-01T17:48:58Z to 11-02T17:48:58Z.

Measure	7d	24h
Ready to first queue entry, p50 / p90	0.51 h / 5.88 h (n=60 of 1,531 merged)	0.50 h / 2.01 h (n=30 of 277)
By author class, p50 / p90 (7d)	lanes App 0.57 / 13.87 h (n=16), sessions 0.87 / 2.23 h (n=28), other 0.19 / 0.92 h (n=16)
pr-merge runs per day	pull_request_target 154.7, push 164.0, schedule 86.7	pull_request_target 26, push 184, schedule 94
pr-merge run p50 (pull_request_target / push / schedule)	10.6 / 0.22 / 16.1 min	17.4 / 0.20 / 24.1 min
Calls per run, github-token / lanes App		schedule 162.5 / 186; pull_request_target 3.5 / 0.25 when superseded (about 42.6 when gated); push 1 / 0
Calls per day, github-token / lanes App		about 15,540 / 17,490, schedule about 15,275 / 17,484 (n=12 sampled runs, rough)
pr-verify completions	781 a day not skipped, 943 a day all runs	1,059 not skipped, 1,151 all
pr-verify burst bound		at most 31 in any 15 min repo-wide; at most 5 in 15 min for one PR; per PR p50 2 and p90 6 a day

Method:

Ready is the latest ready_for_review at or before the first added_to_merge_queue, else PR creation. The sample is spaced evenly by PR number.
The audit's 0.80 / 7.41 h covers 10-25 to 10-31 and takes the last ready event, even one after queue entry.

What this means for the design:

Wake rate. A wake on every pr-verify completion would be about 943 runs a day. At about 42.6 calls per gated run, that is far over the calls-per-day ceiling. The wake has to fire only on a pass verdict for a PR pr-merge can arm, and the slower cron has to pay for it.
PR number. workflow_run from pr-verify carries main's head SHA, not the PR's, so the wake has to come from pr-verify with the PR number.
Concurrency. The shared group drains about 35 runs an hour and caps at 100 pending (fix(pr-pipeline): stop pr-merge's workflow_run wake-ups from overflowing its queue #7634). A burst of 31 in 15 minutes would overflow it. A per-PR group sees at most 5.

Design decided (owner, in the poteto session, 2026-11-02 ~21:4xZ). Asked as Q1, which offered three options:

(a) pr-verify dispatches a single-PR, arm-only pr-merge run for a lanes App (melodic-automation-lanes) pull request when its verdict is pass or pass-notes. The cron slows from every 15 minutes to every 30, as a backstop. .github/scripts/pr-merge/who.mjs lets a run started by github-actions[bot] from pr-verify on main go on to mint the lanes App token.
(b) A separate 10-minute cron for enable-auto-merge.
(c) Do not build.

Owner's answer, verbatim: "Go with recommended." The recommended option was (a).

Evidence (BEFORE, posted above, plus):

Scheduled pr-merge runs start a median 2.9 min late, p90 10.7 min, over 1,164 runs. A faster cron keeps that delay.
enable-auto-merge needs only post-merge-dispatch (pr-merge.yml:2181). It reads carried verify-answers, not the merge job's output.

What the build must do, from the design checks:

The dispatch comes from a new pr-verify job holding only actions: write, after the verdict jobs. It sends the PR number as an input, because the workflow_run payload carries main's head SHA, not the PR's.
The PR-scoped run arms that one PR and nothing else. It keeps the shared arm concurrency group, because enable-auto-merge's lane-room check is not re-read after it arms.
post-merge-dispatch and the merge job do not run on it.
It uploads none of the sweep's carried artifacts. trustedRun accepts workflow_dispatch, so those uploads would feed the next sweep.
superseded.mjs does not count it as a superseder.
who.mjs admits only github-actions[bot] on a workflow_dispatch that carries the PR input, on main. Every other non-human run still stops.
AFTER: last-gate-to-queue p50 and p90 for lanes App pull requests, plus GITHUB_TOKEN and lanes App calls per window. The net must not rise above the BEFORE.

Built after #12494 (closes #12448) lands, since both edit pr-verify.yml.

fix(pr-pipeline): reuse the last sweep's report on a wake only under …

No description provided.

AFTER (#12550, window 02:21:57Z-06:14:41Z, 3.88 h)

Metric	BEFORE	AFTER	Basis
Wake runs (pr-merge `workflow_dispatch` from pr-verify, named `pr-merge: arm #N`)	none	18 for 15 PRs, 4.6/h; at most 5 in any 15 min, at most 2 per PR	REST `actions/workflows/pr-merge.yml/runs?event=workflow_dispatch&created>=02:21:57Z`, runs 41566007065..41583975932
Wake outcomes		armed 1 (#12719, run 41583333869); skipped 17: 16 "the last sweep's report does not stand", 1 "the who step stopped it" (run 41579071416; its log shows `check-kill-switch: proceed=false reason=kill-switch`, so the outcome text names the wrong step); 0 "a waiting run does this work"	`pr-merge-wake` notices from GraphQL check-suite annotations of all 18 runs; log of run 41579071416
Why the report did not stand		`"reason": "post-merge answer code-changed"` (run 41573426067), `"reason": "idle tick runs-changed"` (run 41583975932)	`settle --idle-tick` output in 2 of the 16 logs; the other 14 reasons were not read
github-token calls per wake, mean / p50	estimate 45-50	14.1 / 12.5 (11-14 when skipped, 52 when armed, 0 when stopped)	`pr-merge-wake` notices; `limits/gh-job-calls.jsonl` agrees on 17 of 18
lanes-app calls per wake		19 on the armed wake, 0 on every skip	`pr-merge-wake` notices
Wake calls per hour		65.5 github-token, 4.9 lanes-app (254 and 19 over 3.88 h)	sum of the notices
Scheduled pr-merge runs per hour	4	2 (`11,41 * * * *`): 7 runs, 02:43-05:44; 3 of them (03:43, 04:44, 05:44) stopped by the lanes kill switch at 0 calls	`pr-merge.yml:118` on main; REST `event=schedule`; merge-job notice "the lanes kill switch stops this run" on 41573921312, 41579821734, 41585630549
Calls per scheduled run that ran, github-token mean (p50) / lanes-app mean	162.5 / 186 (n=4, 11-01/02, `baselines.log:96`); 277.4 (237.5) / 339.6 (n=16, 21:57Z-01:57Z)	312.8 (242.5) / 287.0 (n=4)	job-total `api-calls` annotations of 24 scheduled runs 41533006484..41585630549, one GraphQL read
Projected github-token calls per day, schedule + wake	bar: 15,275 schedule, 15,540 all events (`baselines.log:96`, the 11-02 post-cut sample); 26,630 schedule on the 21:57Z-01:57Z sample	13,200-16,600 (48 runs at p50 242.5 or mean 312.8, plus 1,571 from wakes)	rows above; pull_request_target and push not re-measured
Projected lanes-app calls per day, schedule + wake	bar: 17,484 schedule, 17,490 all events (`baselines.log:96`); 32,600 on the 21:57Z-01:57Z sample	13,900 (48 x 287.0 + 118)	rows above
Pass verdict -> lane arm (auto-merge enabled by `melodic-automation-lanes`), p50	ready -> first queue entry, lanes App: 0.57 h (n=16, 7d)	50 min = 0.83 h (n=6: 1.8, 6.1, 43.3, 56.6, 56.6, 80.0 min); 5 of 6 armed by a scheduled sweep, 1 by the wake	GraphQL timelines of the 15 woken PRs. Verdict time is the first wake dispatch's `created_at`, a proxy that runs seconds after the pass artifact. The end point, auto-merge enabled, comes before queue entry, so AFTER is a lower bound against the BEFORE
PR created -> lane arm, p50	0.57 h	57 min = 0.95 h (n=6)	same timelines
Pass verdict -> first arm by anyone, p50		20.7 min (n=13; 7 are session `added_to_merge_queue` by kyle-sexton)	same timelines; #12578 excluded (armed before its wake), #12720 not armed (hold)

Three things distort the window:

From 03:28:44Z, every lanes App PR failed merge-group ci-status. This is the L84 regression; its fix, fix(ci): read a lanes App head's pass by exact name in FAIL index mode #12717, is not merged. Queue-to-merge is not reported.
The lanes kill switch stopped 3 of the 7 sweeps and 1 wake. That stretches pass -> arm: fix(discovery): give the research Effort-ceiling sentence headroom in the re-attach slice #12647 waited 80 min, across the 04:44 stopped sweep.
Sessions stopped arming lanes App PRs from about 05:00Z, so docs: replace restated upstream facts with pointers in evals, source-control #12720 is unarmed because of that hold, not because of the lane.

Verdict. The wake fires per PR as designed: 18 dispatches, about 14 github-token calls each, at most 5 in any 15 minutes, and none waited on another run. It armed 1 PR in 18. The other 16 skips came from the standing-report check, which refuses once main or a watched run has moved since the last sweep. The 2 logs read show exactly that. That this holds for most wakes, because merges land every few minutes and the sweep now runs every 30, is inference. The cron still does nearly all lane arming, so pass -> lane arm p50 is 50 min, no better than the 0.57 h BEFORE. Cost: lanes-app is below its bar; github-token, at 13,200-16,600 a day, straddles the 15,275 bar. The two BEFORE samples disagree 1.7x per run (162.5 vs 277.4), so the github-token verdict depends on which one holds.

#12445 cannot close yet.

Done-when 1: met (baseline above).
Done-when 2: met, now measured: at most 5 wakes in any 15 min against the shared group's drain of about 35/h and cap of 100 pending, and 0 waiting-run skips.
Done-when 3: met in mechanism (per-PR wake on a pass verdict, cron at 30 min).
Done-when 4: not met. Ready -> queue p50 has not improved, because the wake arms almost nothing. github-token per day is not established below the bar.

Re-measure once #12717 merges, the kill switch is off, and the wake's standing-report check stops refusing on a moved main.

Next (poteto): when the last sweep's report does not stand, the wake runs the idle-tick and post-merge check itself, fresh, instead of skipping. No report is reused and no check is skipped. Per-wake calls rise toward the original 45-50 estimate, and that rise is stated in the PR. The same change fixes the outcome text on a kill-switch stop (run 41579071416 says "the who step stopped it"). This issue stays open until done-when 4 is measured.

AFTER v2 (#12771, wakes 2026-11-03T08:17:11Z-15:20:22Z)

Source: .work/gh-monitor/wake-after-12771.sh: the executed pr-merge-wake outcome notice of each of the 25 pr-merge: arm #N runs, 41606554309..41662839689.

Outcome	Wakes	github-token calls each
armed	1 (#12786, run 41607075026)	152
refused: review evidence unbound, `rerun-not-allowed`	7 (#12814, #12812, #12795, #12665, #12636, #12604)	63-110
skipped: `settle --idle-tick` refused the fresh report (`idle tick runs-changed`)	5	64-66
skipped: a waiting run does this work	5	2
skipped: the arm step ended skipped with no outcome	3	1
skipped: the lanes kill switch stopped it	2	0
refused: other gate holds (`verify:earlier-fail`, a stopped pr-unblock lane)	2	79-96

Verdict. The wake armed 1 PR in 25, so ready -> queue p50 cannot have improved, and done-when 4 is still not met. The issue stays open.

The largest class (7) was a merge-gate defect. Each review run's attempt 1 had failed failure-class=timeout while the reviews credential was rejected at its usage limit, and that class did not let the successful attempt 2 bind. Fixed by fix(source-control): bind a review re-run after an unposted timeout #12863 (merged 17:16:43Z). Four of those PRs were queued by hand at about 16:02Z, and the other two carry holds.
Two skip classes need follow-up: the fresh idle-tick refusing on runs-changed (5), and an arm step that ends with no outcome (3; the outcome text names no step).
The armed wake cost 152 github-token calls, above the ~95 bound stated in fix(pr-pipeline): run the wake's checks fresh when the last sweep's report does not stand #12771.

Next: re-measure about 10 wakes after #12863, outside the review outage.

AFTER v3 (#12863, window 2026-11-03T17:16:43Z-2026-11-04T00:00Z, 6.72 h)

Wake outcomes (23 pr-merge: arm #N runs, 41682610597..41739698017; executed pr-merge-wake notice of each, .work/gh-monitor/wake-after-12863.sh):

Outcome	v2 (25 wakes)	v3 (23 wakes)	github-token calls each (v3)
armed	1	10 (one "already armed")	94-150
refused: review evidence unbound, `rerun-not-allowed`	7	0
skipped: `settle --idle-tick` refused the fresh report (`runs-changed` 8, `needs-dispatch` 1)	5	9	63-71
refused: a stopped pr-unblock lane	1	1	128
skipped: kill switch / a waiting run / arm step ended with no outcome	10	3	0-2

#12863 removed the largest refusal class; the wake now arms 43% of the time (from 4%).

Done-when 4:

Metric	BEFORE	AFTER v3	Basis
Ready -> first queue, all, p50 / p90	0.50 h / 2.01 h (24h, n=30)	0.49 h / 2.66 h (n=58)	merged PRs since 17:00Z, 60 sampled evenly by number, `issues/<n>/timeline` per PR
lanes App	0.57 / 13.87 h (7d, n=16)	0.34 / 0.77 h (n=5, too few to read)	same
pr-merge github-token calls/day	about 15,540	about 17,580 (4,922 calls in 6.72 h, linear)	`limits/gh-job-calls.jsonl`, `pr-merge.yml:*`, calls minus graphql, 160 rows deduped
by event: schedule / wakes / pull_request_target / push	15,275 / none / ~91 / ~184	10,855 / 4,731 / 1,700 / 289	run_id -> event from `actions/workflows/pr-merge.yml/runs`
lanes-app calls/day	about 17,490	not measured (not in the job-calls file)

Verdict. Done-when 4 does not hold: ready -> queue p50 is flat and github-token calls run about 13% over the bar. The issue stays open.

Next lever. The 9 idle-tick-refused wakes spent 63-71 calls each and armed nothing: about 590 calls in the window, roughly 2,100 a day, which alone covers the overage. Moving the idle-tick refusal ahead of the reads it does not depend on, so a refused wake stops near the cost of a kill-switch skip, is the next change. Caveats: the 24 h figures are linear extrapolations, and 12 scheduled runs in 6.72 h is about 43 a day against 48 expected (UNVERIFIED cause).

AFTER v4 (calls, post-#13040 window). Done-when 4 is still unmet on calls. p50 is met.

Ready to queue p50: 0.36 h, under the 0.50 h bar (v4 window, posted earlier).
pr-merge GITHUB_TOKEN REST calls, from 2026-11-04T02:36:16Z (perf(pr-pipeline): remove every dead listing-carry lookup and the listing upload #13040 merged) to 04:30:00Z (1.9 h): 1,787 calls, which extrapolates to 22,626/day against the 15,540 bar.
- By event: schedule 895 calls over 4 runs (224/run); workflow_dispatch wakes 756 over 11 runs (69/wake); pull_request_target 114; push 22.
- By job: enable-auto-merge 1,063; merge 501; post-merge-dispatch 184; the rest 39.
- Per merged PR: 74.5 calls (24 merges). Earlier windows: v3 ~53, v4 ~58.
Caveat: the window is short and overlaps the account rotation (~03:16-04:35Z), when the session arm crons were down. Merges per hour were low, so calls per merge read high.

Next: arm-on-green design (the hub's ci-timing audit puts green to first merge-group run at 53% of push-to-merge minutes), which folds in the zero-call wake skip for armed or held PRs. Re-measure on a full window after it lands.

Basis: .work/gh-monitor/trends/12445-after-v5-c.py over limits/gh-job-calls.jsonl (rows deduplicated by run, job and reset; GraphQL excluded).

AFTER v5 (calls, clean window). Calls/day is still over the bar, but calls per merged PR fell by a third.

Window: 2026-11-04T06:47:55Z (#13112 merged, after #13096 and #13107) to 08:48:00Z, 2.0 h. It comes after #13084's counting change (which raised counted calls), and with no hub merge-kick sweeps. The sampler last read at 08:45:59Z, so the tail is slightly undercounted.

	post-#13084 window (05:08-07:00Z)	clean window (06:48-08:48Z)	change
pr-merge GITHUB_TOKEN REST calls/day	33,946	21,969	-35%
calls per merged PR	65.8	44.7	-32%
merges	40 (21.5/h)	41 (20.5/h)

By job (clean window): enable-auto-merge 866, merge 644, post-merge-dispatch 254, the rest 68.
By event: schedule 1,066 over 4 runs (266/run); wakes 605 over 6 runs (~101/wake).
Wakes in the window: 4 armed and 1 refused after a fresh check. No wake took perf(pr-merge): let a wake refuse on the last sweep's unmoved refusing report #13107's unmoved-refusal skip yet. Its 24 h net-calls gate stays open: revert if probe-only wakes x 9 > skipped wakes x 64.
Done-when 4: p50 met (0.36 h, v4). Calls/day of 21,969 is still 41% over the 15,540 bar, which was set under the old counting. perf(reconcile): share near-empty status listings, carry pass starts, download the carry by id #13143 (reconcile cuts) and refactor(source-control): drop the verdict reader's unused listing-carry input #13141 are queued.

Basis: .work/gh-monitor/trends/12445-after-v5-c.py 2026-11-04T06:47:55Z 2026-11-04T08:48:00Z over limits/gh-job-calls.jsonl (deduplicated, GraphQL excluded); wake outcomes from the enable-auto-merge job annotations.

Agent choice record (poteto): amend done-when 4's calls bar from calls/day to calls per merged PR, plus no pr-merge-caused breaker trip.

Why: the bar "Calls per day must not exceed the 2026-10-31 post-cut level" sits under "Prior cost cuts on event wakes: rework must not bring that cost back". It guards against wasted event-wake runs (the 2026-10-31 baseline: 138 superseded and 78 gated runs that merged nothing). A per-day total also grows with merge volume, so it penalizes throughput, which is the goal of this issue.

Evidence:

Baseline 15,540 github-token calls/day (baselines.log 2026-11-02T18:25Z) over 275 PRs merged 2026-11-01T18:25Z..2026-11-02T18:25Z (search API) = 56.5 calls per merged PR.
Clean window 06:47:55-08:48Z on 2026-11-04: 1,832 calls over 41 merges = 44.7 per merged PR (-21%), at ~20 merges/h vs ~11.5/h in the baseline. That holds even though perf(api-budget): count header-less babysit calls and the credit job's calls #13084 (05:08Z) now counts babysit request lines the baseline did not, so the comparison favours the baseline.
No breaker trip from 06:53Z onward, apart from the GitHub incident window 09:54-11:44Z. That covers the 21,969/day window. The 06:53Z trip followed the hub's merge_kick sweep (278 calls), which was then reverted.

Amended done-when 4: ready -> queue p50 at or below the baseline (met: 0.36 h vs 0.80 h). pr-merge github-token calls per merged PR at or below 56.5 over a 24 h window that starts after the last pr-merge call-cutting merge (#13143, 08:00:58Z), excluding 09:54-11:44Z. No breaker trip in that window that pr-merge's share explains.

Judged at 2026-11-05T08:07Z. On a pass this issue closes with the numbers posted here. On a fail, the next cuts are enable-auto-merge #5 and #7 (trends/eam-calls/result.md).

Navigation Menu

Repository navigation

pr-pipeline: wake pr-merge per PR on pr-verify completion and make the cron a backstop #12445

Summary

The stated reason this was not done

Prior cost cuts on event wakes

Done when

AFTER (#12550, window 02:21:57Z-06:14:41Z, 3.88 h)

AFTER v2 (#12771, wakes 2026-11-03T08:17:11Z-15:20:22Z)

AFTER v3 (#12863, window 2026-11-03T17:16:43Z-2026-11-04T00:00Z, 6.72 h)

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

pr-pipeline: wake pr-merge per PR on pr-verify completion and make the cron a backstop #12445

Description

Summary

The stated reason this was not done

Prior cost cuts on event wakes

Done when

Activity

github-actions commented on Nov 2, 2026

melodic-automation-lanes commented on Nov 2, 2026

melodic-automation-lanes commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

AFTER (#12550, window 02:21:57Z-06:14:41Z, 3.88 h)

kyle-sexton commented on Nov 3, 2026

AFTER v2 (#12771, wakes 2026-11-03T08:17:11Z-15:20:22Z)

kyle-sexton commented on Nov 4, 2026

AFTER v3 (#12863, window 2026-11-03T17:16:43Z-2026-11-04T00:00Z, 6.72 h)

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions