Skip to content

pr-pipeline: static friction audit of the unattended merge flow and the 2026-10-18 gaps checklist (child of #7531) #12284

Closed
4 of 7 issues completed

Description

@kyle-sexton

Refs: #7531

Problem

The 2026-10-26 record on #7531 split #7531. The data retro stays there, gated on live merges. The static friction audit and the 2026-10-18 gaps checklist move to a child issue, because they need no live merge data. That record's first criterion is this child.

The 2026-10-26 amendment changed the deliverable. The findings are posted as a comment on this issue, plus follow-up issues, and any lasting conclusion goes into docs/conventions/pr-pipeline/README.md. No report file is committed (AGENTS.md: one-off investigations stay in the scratchpad, and only the conclusion is committed). The revert-triggered stop is a criterion on #7528's record, so the audit cites #7528 for it.

Routing: host-run. Posting findings as a comment and filing follow-up issues are tracker writes, and the issue lane can only read the tracker. Both #7531 lane runs say so (6536975267, 6537727272).

Scope, from #7531's body: CI/CD performance, the autonomous Actions lanes, and the merge path; find the friction points.

The 2026-10-18 gaps checklist (comment), from that comment:

That comment's last item, the candidates to evaluate on #7531's data, stays with #7531's data retro.

Folded in from #7612 (comment): its items 2, 3, 5 and 6, the label-normalization restrictions on admission.

Done when

  • A findings comment on this issue covers CI/CD performance, the autonomous Actions lanes and the merge path, and names each friction point with the file:line or run that shows it.
  • Each of the six gaps-checklist items above, and each of pr-pipeline: harden issue-lane admission after the ratification marker #7612's items 2, 3, 5 and 6, is answered in that comment as keep, drop, or a follow-up issue number.
  • Every follow-up the findings recommend is a filed issue, cited by number.
  • Any lasting conclusion is written into docs/conventions/pr-pipeline/README.md through a PR. No report file is committed, and no lane behavior changes in this issue.
  • The revert-triggered stop is cited as pr-merge: GITHUB_TOKEN-dispatched post-merge runs file no demotion issue #7528's criterion, not re-tracked here.

Activity

  1. added theissue type on Nov 2, 2026
  2. added
    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 2, 2026
  3. github-actions commented on Nov 2, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; the lanes App applied this class.

    Size: Large
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    I understood this as a static audit of the unattended merge flow. It covers CI/CD performance, the autonomous Actions lanes and the merge path. It also answers the six 2026-10-18 gaps-checklist items and #7612's items 2, 3, 5 and 6. The deliverable is a findings comment, follow-up issues for each recommendation, and any lasting conclusion written into the pr-pipeline README through a PR. The done-when is exact, and the tracker writes are already routed to host-run. The audit spans several subsystems, so it may need splitting. I found no likely duplicates among the open issues.

    Automated triage, applied by the lanes App.

  4. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.
    on Nov 2, 2026
  5. melodic-automation-lanes commented on Nov 2, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: it carries host-run; Size large: the split into sub-issues at intake is pending (owner Q93). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  6. melodic-automation-lanes commented on Nov 2, 2026

    @melodic-automation-lanes
    Contributor

    The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.

  7. kyle-sexton commented on Nov 2, 2026

    @kyle-sexton
    MemberAuthor

    Claim. Backlog HITL host-run worker is taking this now (read-only static friction audit; findings comment follows).

  8. kyle-sexton commented on Nov 2, 2026

    @kyle-sexton
    MemberAuthor

    Host run. A read-only static friction audit of the unattended path from a ready PR to merge, with run costs measured from the hub's local runs cache and a sampled read of failing ci-status steps.

    What ran

    • Code read at origin/main cb8429f3cf (2026-11-02T15:30Z). Every file:line below is at that commit.
    • Live ruleset read: GET repos/.../rules/branches/main.
    • Run costs come from the hub's local runs cache, which holds every workflow run with its created, started and updated times, conclusion and attempt. Two windows: 7d (2026-10-26T00Z to 11-02T00Z) and 1d (2026-11-01). This used no API calls. "Run-h" means run wall hours (updated_at - run_started_at) over non-skipped runs. On this public repository hosted runners do not bill, so run-h measures runner contention and, for the model lanes, subscription usage. It is not a charge (costs).
    • Failure causes come from a seeded random sample of failed pr-require-checks runs on 11-01, read through actions/runs/{id}/jobs:
      • 25 pull_request runs, of which 24 were readable (one returned HTTP 502).
      • 12 of the 24 merge_group runs.
      • Three runs' ci-status annotations.
    • Prior measurements are reused where they still hold: the 2026-11-01 lifecycle baseline, the 2026-10-28 owner-touchpoint audit, and the 2026-11-01 pr-merge call counts. Their figures are labeled as quoted.
    • About 50 GitHub calls in total, with no rate-limit answer.
    • No gate, workflow, label rule or config was changed, and no PR was opened.

    Where the time goes (measured)

    Workflow (event) Runs 7d Run-h 7d p50 / p90 min Fail share 7d 1d (11-01) runs / run-h
    work-resolve-issue (issues) 1,182 non-skipped of 4,733 694 27.1 / 77.4 227 failed + 71 cancelled (25%) 233 / 110
    pr-require-checks (pull_request) 5,515 291 3.2 / 6.6 1,143 (21%); 790 re-run attempts (14%) 1,162 / 56 (192 re-runs)
    pr-merge (pull_request_target unlabeled) 1,011 206 10.6 / 23.3 0.3% 38 / 7.5
    pr-require-checks (merge_group) 1,885 200 5.4 / 9.5 134 (7.1%) 363 / 38
    pr-merge (15-min schedule) 607 173 16.1 / 31.3 2.6% 94 / 27
    pr-unblock (dispatch) 1,024 133 7.0 / 13.1 118 (12%) 230 / 35
    post-merge-test-mutation 482 123 12.9 / 31.7 30 (6%) 89 / 21
    pr-review-hosted + pr-review-security-hosted 3,089 each 118 + 105 2.0 / 3.6 and 1.7 / 3.2 under 0.5% 648 each / 53
    pr-verify (workflow_run) 5,470 of 6,604 114 0.9 / 2.0 0.1% 1,257 / 19
    pr-feedback-dispatch (workflow_run) 12,345 of 19,756 101 0.3 / 0.8 0.6% 2,786 / 26
    pr-refine (dispatch) 1,035 68 2.6 / 6.9 59 (6%) 221 / 17
    pr-review-main (workflow_run) 3,939 64 0.5 / 2.3 0.1% 1,372 / 20
    pr-require-checks full main (dispatch) 466 68 8.6 / 9.5 13 54 / 9

    About 336 PRs merged through the queue on 11-01 (successful merge_group runs). On that day the merge path after a PR is ready used roughly 300 run-h. The lifecycle baseline (2026-11-01, quoted) puts the PR-side wait at ready -> first queue entry p50 0.80 h / p90 7.41 h over 7d, and queue entry -> merged at p50 0.11 h. The wait before the queue is the binding stage.

    Why ci-status is red (sample)

    Of 24 failed pull_request runs:

    • 7 were real lane failures (lint-repo, lint-shell, check-plugins/check-catalog, test-bash, test-node).
    • 17 (71%) were holds.
    Cause PR runs (of 24) Merge-group runs (of 12) Example runs
    do-not-merge label (intended cross-lane hold) 8 1 41342742932, 41319120018
    pr-unblock failure status on the head 6 1 41332979407, 41344950103
    needs-human label 3 0 41306188307
    Review lanes not yet complete on the head 2 n/a 41327400610
    Lanes App body hold or member veto n/a 6 41348597754, 41324239904
    Hold label added after the PR was enqueued n/a 4 41361037999, 41317739074
    Real lane failure 7 2 41280047305, 41278145401

    A red ci-status from do-not-merge or needs-human is the hold doing its job, not a defect. The friction rows are the others:

    • The pr-unblock stop status.
    • The lanes App body and member hold.
    • Holds that land after enqueue: 10 of the 12 sampled merge-group failures are holds, not code. Each one ejects the PR and rebuilds the group for its peers.
    • The review-pending re-run loop.

    Gate by gate

    Owner direction s10: a hold against our own actors is friction. The only HIGH/CRITICAL boundary is a token or secret leak, an account compromise, or billing. No verdict below loosens one of those.

    # Gate or step Risk it covers Existing control on the same risk Measured cost Verdict Basis
    1 ci-status, the one required check (ruleset required_status_checks, strict_required_status_checks_policy: false) Broken code reaching main Merge-queue re-run (#14) 291 run-h/7d on PR runs; 21% red, 71% of the red sample being holds Keep pr-require-checks.yml:4003; ruleset read
    2 Contract step: title and the do-not-merge label A held or misnamed PR merging None: this is the one documented cross-lane hold 8 of 24 red PR runs Keep pr-require-checks.yml:4795; loop-lane/README.md:460
    3 needs-human label holds A PR a person must see Overlaps #2 for own-actor holds 3 of 24 Keep for the review lanes' cost and critical-security holds. Simplify other uses to do-not-merge (#17) pr-require-checks.yml:5445; pr-pipeline/README.md:2804
    4 Only a trusted person lifts a review-lane hold A bot clearing a critical-security or cost hold None Not seen in the sample Keep (boundary: secret leak, billing) pr-require-checks.yml:5551; pr-pipeline/README.md:2806
    5 Cost-only review hold cleared by an owner decision A real charge merging unseen #4 Not seen in the sample Keep (billing) pr-require-checks.yml:5122
    6 human-merge-hold label, lifted only by a trusted person acting directly A lanes App PR whose body says "Hold: a human merges this" #2 does the same job with one label Not seen in the sample Simplify: fold into do-not-merge. It is an own-actor hold. pr-require-checks.yml:5686; pr-pipeline/README.md:2812
    7 Lanes App body hold or member veto, read by the verdict reader in ci-status A lanes PR a member asked to wait on #2. The babysit gate already reads body holds (AGENTS.md) 6 of 12 merge-group failures. The code comment quotes a cold verdict read in the queue at "about 860" calls and 6-11 min (measured 2026-10-25) Simplify: when a member vetoes, the lanes App adds do-not-merge, and ci-status reads the label only. That removes the verdict reader from ci-status. pr-require-checks.yml:6247, comment at :4029-4051
    8 pr-unblock failure commit status holds the head Merging a head a fixer lane gave up on #2, and the ci-lanes result for real failures 6 of 24 red PR runs, 1 of 12 merge-group. Recovery waits for the half-hourly reconcile re-run Simplify: post do-not-merge (or nothing, when ci-lanes is already red) instead of a separate status context. That drops one hold mechanism and its re-run dependency. pr-require-checks.yml:6186; pr-pipeline/README.md:2863-2869; maintenance-reconcile-lost-runs.yml:93
    9 Both review lanes must have run on the head An unreviewed head merging; the security lane catches token and secret exposure None for security. pr-review-main (#11) for lane-source PRs 223 run-h/7d of review runs; 790 ci-status re-runs/7d (14% of PR runs), driven by rerun-ci-status after each review completes Keep both lanes: the code-review lane also raises the cost hold (pr-review-hosted.yml:242-250), so it is a billing control. Simplify the ordering so ci-status does not start red on "review pending". Options: start the review lanes so they gate pr-verify instead of ci-status, or fold the review check into pr-merge's verdict. Either removes the re-run loop. pr-require-checks.yml:6755; pr-feedback-dispatch.yml:575; pr-pipeline/README.md:2872
    10 Hold checks inside the merge group (labels, pr-verify-failed, pr-unblock status, lanes body hold) A hold added after enqueue #2-#8 on the PR run Holds are 10 of 12 sampled merge-group failures; 134 failed groups/7d at p50 5.4 min, plus peer rebuilds Keep the check. Simplify the order: the earlier select-tests step (:958) already fails a held PR "before the lanes", so make it the only queue-time hold read and drop the duplicates in ci-status (:5779, :6186, :6247). Better still, dequeue the PR on labeled instead of waiting for the group run. pr-require-checks.yml:958, :5779
    11 pr-review-main: re-review from main for PRs that change a lane source A head forging its own review verdict None 3,939 runs and 64 run-h/7d. It fires on every pr-require-checks completion, contract-only runs and re-runs included (1,372 runs on 11-01 for 745 distinct PR heads, about 1.8 per head), and most runs exit early Keep the gate (forgery of a security review). Simplify-later the trigger: skip completions whose ci-status already ran on that head. pr-review-main.yml:3, :59
    12 pr-verify: verifier verdict bound to the head Merging a head the verdict did not judge None 5,470 runs and 114 run-h/7d; also fires on every completion (workflow_run), about 1.8 per head Keep. Simplify-later as #11 pr-verify.yml:67-71
    13 pr-merge lane wake: 15-min cron, the unlabeled event and push Missing a PR that just became mergeable none Ready -> queue p50 0.80 h / p90 7.41 h (7d, quoted). Schedule: 607 runs, 173 run-h. unlabeled: 1,011 runs and 206 run-h, mostly waiting on the shared concurrency group (2026-11-01 sample, quoted: "26 s after 21 min queued"). The file says it "does not wake on each CI or review completion" Simplify: wake per PR on pr-verify completion (per-PR concurrency group). Drop unlabeled as a wake once that event exists, and slow the cron to a backstop. This is the largest PR-side wait. pr-merge.yml:85, :93, :98
    14 Merge queue (ruleset: ALLGREEN, max 5, check_response_timeout_minutes 30) re-runs full CI per group Semantic conflicts between PRs merged with no rebase (strict is false) None 1,885 groups and 200 run-h/7d; 7.1% fail, mostly holds (#10) Keep. Release merge groups widen to the whole tree (quoted baseline: about 38 a day); scope them to the release diff. That is the owner's call, per the baseline. ruleset read; lifecycle baseline 2026-11-01
    15 Ruleset required_review_thread_resolution: true A review finding left unanswered Lanes settle bot threads (pr-unblock / publish-feedback) Owner resolves were about 30 threads over 16 PRs in 23 h on 10-27/28 (quoted touchpoint audit T4); not re-measured Keep. Simplify: let the lane resolve outdated, answered, non-CRITICAL bot threads on owner-login PRs (touchpoint T4). ruleset read
    16 Ruleset: two pull_request rules, one with dismiss_stale_reviews_on_push and 0 required approvals; signatures; linear history Unsigned or forged commits; approval drift none No measured cost; dismissal only affects the trust-path approvals for Dependabot Keep signatures (supply chain). Remove-later the duplicate pull_request rule once someone confirms which ruleset owns the Dependabot approval path (UNVERIFIED which). ruleset read
    17 needs-human used as a general park by lanes (pr-unblock escalation routes) A disputed finding merging unjudged #2 123 PRs with a pr-unblock stop comment in 7d (quoted, 10-28) Simplify: keep person routes only for CRITICAL security or cost. Send other feedback-disputed/feedback-uncertain routes to a bounded retry (touchpoint T5). pr-pipeline/README.md:2804; touchpoint audit T5
    18 Fixer lanes pr-unblock and pr-refine Red CI, review threads, docs drift none 133 + 68 run-h/7d; pr-unblock 12% failed Keep. A failed pr-unblock run is a source of #8's holds. runs cache
    19 pr-feedback-dispatch router (workflow_run on 8 workflows) Routing completions to the next lane none 19,756 runs/7d (7,411 skipped), 101 run-h Simplify-later: narrow workflows: to the completions it acts on. It ran about 11 times per merged PR over 7d (19,756 runs, 1,727 successful merge groups). pr-feedback-dispatch.yml:99
    20 Post-merge: full main run, demotion alert, red-main revert or fix-forward, mutation tests A bad merge staying on main Merge queue (#14) Full main runs 466/7d at p50 8.6 min (now about one per 27 min on 11-01, inside the 30-min cap; the baseline's 14-min gap no longer holds); mutation 123 run-h/7d Keep pr-pipeline/README.md:3682, :3726-3728
    21 Owner-login PR adoption idle wait A person's PR merging while they still work on it Draft state Was 24 h on 10-28; now idle-hours: 2 Already simplified (touchpoint T3) pr-pipeline.yaml:1051

    Top findings by measured cost

    1. The PR-side wait from ready to queue (ci: add Conventional Commits PR-title gate #13). p50 0.80 h and p90 7.41 h over 7d, against 0.11 h from queue entry to merge. pr-merge wakes on a 15-min cron and unlabeled only (pr-merge.yml:93-98). Its unlabeled runs alone are 206 run-h/7d, mostly concurrency wait. Fix: wake per PR on pr-verify completion.

    2. ci-status is red mainly for holds, and holds are enforced in seven places. 71% of sampled red PR runs and 10 of 12 sampled red merge groups were holds, not code. loop-lane/README.md:460 calls do-not-merge "the only cross-lane hold", yet ci-status enforces several more:

      • needs-human
      • human-merge-hold
      • review-lane hold lifts
      • pr-unblock statuses
      • lanes App body holds and vetoes
      • pr-verify-failed in the queue

      Collapsing the own-actor ones into do-not-merge (perf(markdown-formatter): unwired producer builds telemetry payload before the opt-in guard (~90ms/edit on Windows) #6, docs(playbook): document the local plugin development loop #7, build(dependabot): fast-track first-party melodic-software/* actions #8) removes the verdict reader from ci-status and the half-hourly re-run dependency.

    3. Review-pending re-runs. 790 re-run attempts in 7d (14% of PR CI runs) come from rerun-ci-status after each review completes (chore: adopt widened comment-hygiene policy and re-pin ci-workflows @9567b8b #9). Move the review requirement off ci-status's first pass.

    4. Trigger fan-out. pr-verify, pr-review-main and pr-feedback-dispatch together ran 21,754 times in 7d (279 run-h), firing on every completion including re-runs and contract-only runs (fix(markdown-formatter): harden hook + simplify plumbing #11, fix(comment-hygiene): exclude '.' from the owner/repo#N leading boundary #12, feat(biome-format): add Biome format/lint-on-edit plugin #19).

    5. Queue ejections on late holds (build(deps): bump ci-workflows pin to b6431a1 (PSSA RULE_ERROR retry fix) #10). 134 failed groups/7d, most of them holds. Dequeue on labeled, and read holds once, in select-tests.

    Outside the merge path but in scope, work-resolve-issue is the largest lane: 694 run-h/7d, 25% failed or cancelled. The baseline's retry-tail lever, a shorter stranded re-fire backoff for transient classes, still applies (lifecycle baseline 2026-11-01, lever 3). turn-cap-restarts stays untouched (pr-pipeline.yaml:958).

    2026-10-18 gaps checklist

    Item Answer Basis
    No owner for post-merge bookkeeping (track done, container close-out) Follow-up needed. Merged -> issue closed is p50 0 h (Closes links work). Parent close-out lags: 25 of 67 open parents had every sub-issue closed on 2026-11-01 (quoted baseline; UNVERIFIED today). The owner is work-unblock's parent auto-close; the follow-up is why it skips them. lifecycle baseline 2026-11-01 §2
    Triage in two places Keep, answered. Intake triage owns the stop classification for issues, and admission re-judges before admitting (loop-lane/README.md:326-329, :342). A local /work-items:triage verdict is input that intake re-judges. cited lines
    Merge authority switch and rung lowering (#6202, Q505) Keep, no gap. One authority (authority: pr-merge, rung: merge, pr-pipeline.yaml:1097-1098). The rung still exists and falls automatically after a lane merge breaks main (pr-pipeline/README.md:1026-1027). No class is read (:1015). cited lines
    Post-merge failure: revert or fix forward Keep, decided. The bot reverts only when every guard holds; otherwise fix issues fix main forward through the issue lane (pr-pipeline/README.md:3726-3728). The revert-triggered stop is #7528's criterion and is not re-tracked here. cited lines; #7528
    Post-merge smoke check against a deployed environment (#6202) Drop. This repository deploys no environment. The full main run after merges plus post-merge-test-mutation cover main (#20). pr-pipeline/README.md:3682; runs cache
    Stages outside the plan (build/package, post-deploy monitoring, incidents, postmortems, migrations, a release definition) Drop for this repository, with one exception: "release" is the chore(release) version-bump PR from release-publish-plugins.yml. Rollout is not in scope while there are no feature flags. judgment; workflow list

    #7612 items 2, 3, 5 and 6

    Item Answer Basis
    2. Lookalike-Unicode work-class labels Drop. Admission now matches the exact work-class: untrusted-provenance string (admit.mjs:175, :995-998), and no merge authority reads a class (pr-pipeline/README.md:1015). A lookalike label holds nothing and unlocks nothing. Creating one needs repository write access, and an actor with write access could simply remove the real label, which untrustedHold catches when a non-human does it. cited lines
    3. Who applied a work-class label Drop. A non-human's removal of the untrusted label keeps the hold (admit.mjs:995-998), and an add by anyone holds, failing closed. No other class gates admission or merge. cited lines
    5. report-outcome.mjs exact held-class match Drop (obsolete). report-outcome.mjs no longer matches work classes (git grep at cb8429f finds none). grep
    6. predicate.mjs normalization Drop (obsolete). "No key reads a work class" (predicate.mjs:3). cited line

    Done-when status

    Target labels: add none; remove none.

  9. kyle-sexton commented on Nov 3, 2026

    @kyle-sexton
    MemberAuthor

    Follow-ups for the three findings recommendations that had no issue (criterion 3): row 14, release merge groups scoped to the release diff, is #12633; row 15, the lane resolving answered non-CRITICAL bot threads on owner-login PRs, is #12634; the work-resolve-issue retry-tail lever is #12635. Row 14 is not an owner decision: ADR 0048:124-128 fixes only how a release merges, and narrowing which suites a release group runs touches none of the three owner stop reasons. Agent choice, not an owner answer; written by the issue-triage session.

  10. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    MemberAuthor

    Delivered: closing as completed.
    Agent choice, not an owner answer.

    The open slices (#12445, #12447, #12456, #12438) carry the remaining work as their own issues.

  11. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    MemberAuthor

    Closing as completed: all five done-when items are met (record above); its open follow-ups stand on their own.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.priority: mediumReal value, no hard deadline; normal backlog flow.status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    Work status

    confirmed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions