[ML] Fix backport-pending label handling for auto-backports

[edsavage]

Summary

The remove-backport-pending job clears backport-pending from a source PR once all of its backports have merged. It never runs for auto-merged backports: auto-merge was armed with GITHUB_TOKEN, so GitHub attributes the deferred merge to github-actions[bot], and merges attributed to github-actions[bot] do not trigger workflows — GitHub's built-in recursion guard (docs: "events triggered by the GITHUB_TOKEN will not create a new workflow run"). So backport-pending got stuck on the source PR and had to be removed by hand (seen on #3078 and #3115).

Fix (direct, not a workaround)

Arm auto-merge with the ephemeral App token (elastic-vault-github-plugin-prod[bot]) that already authors the backport PRs, instead of GITHUB_TOKEN. GitHub attributes a deferred auto-merge to whichever identity enabled it, and merges by a GitHub App identity do trigger downstream workflows — so remove-backport-pending fires normally on merge.

• The approval still comes from github-actions[bot] (step-level GITHUB_TOKEN), a distinct identity from the ephemeral-token author, so it satisfies the required review (a token can't approve its own PR).
• Falls back to GITHUB_TOKEN only when the ephemeral token is unavailable (Token Policy not registered); in that degraded case cleanup still needs a manual touch, but backports still open and can be merged by a human.

This is the GitHub-recommended remedy (use an App/PAT identity for the merge, per the docs above) and reuses the exact infrastructure we already rely on to make CI trigger on backport PRs. It's the same thing elastic/kibana gets for free by authoring/merging backports as the kibanamachine user rather than github-actions[bot].

Supersedes #3127 (scheduled-sweep backstop), which treated the symptom rather than the cause.

Test plan

• Merge to main.
• Merge a >bug/auto-backport PR with version labels; confirm backports open (ephemeral-authored), are approved by github-actions[bot], and auto-merge on green.
• Confirm that when the backports auto-merge, remove-backport-pending runs (check the Actions tab for the backport PR's merged event) and backport-pending is cleared from the source PR without manual intervention.
• Confirm the fallback path (no ephemeral token) still opens backports and arms auto-merge.

Fixes the GITHUB_TOKEN-suppression cleanup gap from the auto-backport-and-merge work (#3094, #3103, #3122).

---

Implications for the auto-backport-and-merge workflow — no change to the merge gate

This is strictly additive to the auto-backport-and-merge flow validated in #3122/#3115. The "merge only when CI is green" guarantee is enforced by branch protection, not by the token that arms auto-merge, so it is untouched:

• All release branches (9.5/9.4/8.19) require the buildkite/ml-cpp-pr-builds rollup + CLA status checks and 1 approving review, with no push restrictions (restrictions: none). gh pr merge --auto honours these regardless of which identity enabled it — GitHub still holds the merge until CI is green and the review is satisfied.
• What changes is only who is recorded as enabling auto-merge (ephemeral App identity vs github-actions[bot]), which is what un-suppresses the on-merge remove-backport-pending run. Timing/behaviour of the merge itself is identical.

Nothing new is triggered that shouldn't be:

• The backport job on a backport PR's merge event stays skipped (!contains(labels,'backport')), so no recursive backporting.
• Buildkite branch/snapshot builds already fire on any merge commit (Buildkite webhooks aren't subject to GitHub's Actions recursion guard) — unchanged.
• Only remove-backport-pending newly fires, which is the goal.

Permissions: release branches have restrictions: none, and the TokenPolicy (elastic/catalog-info #4297) grants the ephemeral identity contents: write + pull_requests: write, so it can complete the merge once the gate passes. If it ever couldn't, gh pr merge errors are swallowed (|| echo ::warning) and the PR simply stays open — fail-safe: the worst case is a stuck-open PR, never a premature or unreviewed merge.

Fallback path unchanged: if the ephemeral token isn't minted, MERGE_TOKEN falls back to GITHUB_TOKEN, exactly matching today's behaviour (backports open, auto-merge armed, cleanup needs a manual touch).

Note: backports created before this merges (e.g. #3125/#3126/#3118) were armed under the old GITHUB_TOKEN path, so that round still needs one manual backport-pending clear on the source PR. This fix applies to backport rounds created after it merges.

---

Also: stop copying backport-pending / auto-backport onto the backport PRs

.backportrc.json's copySourcePRLabels copied every source label except backport and version labels onto the backports the action creates, so backport-pending and auto-backport leaked onto them (observed on #3125/#3126). backport-pending on a backport is misleading (it isn't itself awaiting backports) and pollutes "pending backports" queries; auto-backport is meaningless there (the backport label already guards against re-backporting).

Fix: exclude both from the copy regex —

-  "copySourcePRLabels": "^(?!backport$)(?!v\\d).*$",
+  "copySourcePRLabels": "^(?!backport(?:-pending)?$)(?!auto-backport$)(?!v\\d).*$",

:ml, >type and other labels still propagate. main-only, same as the merge-identity change (the backport action runs from main's config when a source PR merges to main).

Test plan (label copy)

• After the next auto-backport round, confirm the created backport PRs carry backport + :ml + >type + their version label, but not backport-pending or auto-backport.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Hi @edsavage, I've created a changelog YAML for you.

[Copilot]

Pull request overview

Updates the auto-backport automation so that source PR cleanup (backport-pending removal) runs even when backport PRs are auto-merged, by ensuring the eventual merge is attributed to an App identity (not github-actions[bot], which is subject to GitHub Actions recursion-guard suppression). Also tightens label propagation so backport PRs don’t inherit source-only labels.

Changes:

• Arm gh pr merge --auto using the ephemeral App token when available, while still approving via GITHUB_TOKEN to satisfy required-review rules.
• Update .backportrc.json to stop copying backport-pending and auto-backport onto backport PRs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
.github/workflows/backport.yml	Uses the ephemeral App token to enable auto-merge so the merge event triggers remove-backport-pending as intended.
.backportrc.json	Excludes backport-pending and auto-backport from copied labels so backport PRs aren’t misleadingly labeled.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[edsavage]

Verification note — remaining test-plan items await the next auto-backport round

- [x] Merge to main is done. The other four items are behavioural and can only be ticked once the backport GitHub Action actually runs — i.e. the next time a >bug/auto-backport PR with version labels merges to main. (Manual backports such as #3130 do not exercise the action, so they don't count.)

When that next round happens, check off:

1. Backports open + approval + auto-merge — the created backport PRs are authored by elastic-vault-github-plugin-prod[bot] (ephemeral App token, so CI triggers), approved by github-actions[bot], and have auto-merge armed. Confirm they merge on green.
2. remove-backport-pending fires automatically — on each backport's merged event, the Remove backport-pending label job runs (Actions tab of the backport PR), and backport-pending clears off the source PR with no manual touch. This is the core fix (previously suppressed by GitHub's GITHUB_TOKEN recursion guard).
3. Fallback path — only reachable if the Token Policy is unavailable (ephemeral token unminted); not safely forceable, so verify opportunistically if it ever happens.
4. Label copy — the created backport PRs carry backport + :ml + >type + their version label, but not backport-pending or auto-backport. (The .backportrc.json regex change is already on main; this confirms the observed outcome.)

Static confirmation already in place on main: the merge-identity change in .github/workflows/backport.yml (MERGE_TOKEN="${EPHEMERAL_TOKEN:-$GH_TOKEN}") and the tightened copySourcePRLabels regex.

[edsavage]

Non-destructive verification (mechanism in place)

Ahead of the next live auto-backport round, I verified the merged code/config on main implements the fix. This doesn't tick the test-plan boxes (those need a real round's runtime observation), but confirms the mechanism is correct so only the live observation remains.

1. backport.yml — auto-merge identity, approval, fallback (items 2–4)

• Auto-merge armed with MERGE_TOKEN="${EPHEMERAL_TOKEN:-$GH_TOKEN}" — ephemeral App identity, falling back to GITHUB_TOKEN when the token isn't minted.
• Approval performed by github-actions[bot] (GITHUB_TOKEN) only when the ephemeral token is present (distinct identity → approval counts); otherwise it emits a ::warning:: that manual approval is needed.
• remove-backport-pending job present and gated on the merged event.

2. .backportrc.json copySourcePRLabels (item 5) — unit-tested the live regex ^(?!backport(?:-pending)?$)(?!auto-backport$)(?!v\d).*$:

Label	Result
backport	excluded
backport-pending	excluded
auto-backport	excluded
v9.5.0, v8.19.20	excluded
:ml, >bug, >enhancement, >refactoring, team:ml	copied

3. Live evidence status — no genuine auto-backport round has occurred since this merged (#3130 is a manual backport, not action-created, so it doesn't exercise the ephemeral-token path).

Next validation opportunity: #3131 carries v9.5.1/v9.4.5/v8.19.20, so its merge to main will trigger a real round (9.5/9.4 clean auto-backports). That round should let us tick items 2, 3 and 5 from real evidence, with the 8.19 conflict exercising the degraded/fallback path (item 4).