[pull] trunk from cli:trunk - #207
Merged
Merged
Conversation
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 127aa8fc-0468-4e3d-afec-4587cda12d7a
Co-authored-by: Copilot App <[email protected]> Copilot-Session: cd6441de-bed4-4adb-88c3-904b349ba16f
…e-guidance Tell agents to use the PR template in AGENTS.md
Set GH_EXTENSION=1 when gh invokes an extension
The triage workflow spent an LLM turn deciding it had nothing to do. A no-op scheduled run cost 50-75 AI Credits because the agent walked the dedup protocol itself: fetching open PRs, reading every prior triage comment, and comparing head SHAs. That work is entirely deterministic, and the most expensive single call was the agent re-ingesting its own past comments, so the cost grew every time the workflow commented. Move that comparison into a shell step that runs after checkout but before the engine starts. It writes a work list to /tmp/gh-aw/dependabot-worklist.json, and when the list is empty it emits a `noop` safe output, which makes the harness exit before any inference is billed. This also hardens scope. The agent no longer decides which PRs are in range, so it cannot be talked into assessing a PR outside the work list by content in a PR it is reading. `issues: read` is needed because PR conversation comments are served by the issues API, and the timeout moves to 30 minutes because the runs that do have work now do strictly more evidence gathering per PR. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 190d246d-0b1f-4ce6-9aa6-dee10d3d4cf8
The triager almost never returned High confidence, and when its prose disagreed with a human reviewer it was usually because it had guessed at something it could have read. It named workflow files that do not exist in this repository, and it called a direct `go.mod` requirement indirect. Both mistakes share a cause: the skill never told the agent to look at the PR's own diff or at the checked-out source tree. It had access to both the whole time. So replace inference with five required evidence items - the diff, the dependency's position in the manifest, the repository's actual import surface, CI state, and upstream release evidence - and make High confidence conditional on having gathered them. The old definition of High was unreachable by construction. It asked for the upstream change to be read "end to end" while a separate instruction capped confidence at Medium rather than reading indefinitely, so any non-trivial bump fell through to Medium no matter how clear it was. Redefine High as decision-relevant completeness: a four-release bump that touches nothing this repository imports is High once you have verified that, because reading the rest could not change the answer. Also drop the dedup protocol, which the pre-flight step now performs deterministically, and add an in-repo coherence check for bumps that edit generated files without updating the version each file records - the gh-aw lock files being the case that prompted it. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 190d246d-0b1f-4ce6-9aa6-dee10d3d4cf8
The evidence rule told the agent that a dependency in the first `require` block is direct. That is a `go mod tidy` formatting convention, not the semantics. What actually marks a requirement indirect is the trailing `// indirect` comment on its own line, and Go's parser reads it that way regardless of block: put a commented and an uncommented require in the same block and `go mod edit -json` still reports Indirect true and false respectively. The two agree in this repository today, so nothing was misclassified. But the rule would break on a reorganised or hand-edited file, and misreporting a direct dependency as indirect is precisely the error the required-evidence section exists to prevent. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 190d246d-0b1f-4ce6-9aa6-dee10d3d4cf8
Adding a custom pre-agent `steps:` block made the compiler wrap it in a DIFC proxy, because a guard policy is configured. That proxy applies `min-integrity` but not `trusted-users`, which are resolved at runtime after it starts. The pre-flight finds its dedup marker by reading back its own `cli-triage[bot]` comments, and those are precisely what `min-integrity: approved` filters out - the app posts with author_association NONE, which is why `trusted-users` exists here at all. So the marker was never found, every open Dependabot PR looked unassessed on every run, and the workflow would have re-triaged and re-commented on all of them hourly: the exact failure this design was written to prevent, moved from the agent to a place with no model to notice it. Turning the proxy off does not widen the injection surface. The pre-flight hands nothing it reads to the model - it extracts PR numbers, head SHAs and CI states - and it matches the marker only inside comments already narrowed to the app's own login. That login check, not integrity, is what stops a third party forging a marker. The agent still runs under the full policy via the MCP gateway. Verified against cli/cli: all seven open Dependabot PRs are correctly recognised as already assessed at their current head, and a run with the login filter pointed at a non-existent bot correctly reports them as needing assessment. Three smaller corrections ride along, all fallout from the same review: - Silence no-op issue reporting. gh-aw posts a comment to a shared "no-op runs" issue on every noop, and noop is now the routine idle outcome of an hourly reconciler, so that would have been roughly 24 comments a day forever. The run log already records why a run did nothing. - Drop CI state from the required-evidence count. The pre-flight now guarantees terminal CI, so the agent never gathers it and it could never be the missing item that caps confidence. Counting it made the gate for High confidence four items dressed up as five. - Log which PRs the terminal-CI gate excluded. A check that never reports would otherwise keep a PR out of triage permanently and silently. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 18bc01f9-9498-4bd8-9fd0-70308491b695
`statusCheckRollup` contexts are CheckRun and StatusContext objects, which sit behind the `checks` and `statuses` scopes. The workflow token had neither, so the field would have come back unreadable at runtime even though it reads fine with a developer token, which is what I tested with. The dangerous part was not the missing permission but how the gate reacted to it. `[.statusCheckRollup[]? | select(pending)] | length == 0` cannot tell "this PR has no checks" from "I could not read this PR's checks", so an unreadable rollup counted as terminal CI and the PR would have been assessed while its CI was still running. Silently wrong beats loudly broken only from the outside. So the classification now treats a null rollup as pending and names it in the skip log, and the permissions are granted. The gate fails safe if either is ever dropped again. Also warn when the listing hits the 100-PR cap. gh truncates silently and the ordering is stable, so PRs past the cap would never be reached on a later run either. Paginating for a case that far outside anything this repository has seen, and well above the 20-comment safe-output cap, is not worth the extra requests, but the condition should not be invisible. Reported by Copilot review on #14079. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 18bc01f9-9498-4bd8-9fd0-70308491b695
Make Dependabot triage cheaper and more decisive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )