[WIP] Test repair agent Part 2: update the pulse listener - #6421
Draft
evgenyrp wants to merge 12 commits into
Draft
[WIP] Test repair agent Part 2: update the pulse listener#6421evgenyrp wants to merge 12 commits into
evgenyrp wants to merge 12 commits into
Conversation
Contributor
Author
|
This is preliminary as I'm still reviewing and testing myself. Also, it might change when the agent Part 1 lands. |
Brings in the build-repair action-task and push-age gates alongside the Treeherder-based rewrite of the listener's failure filtering. Conflict resolution: - regression.py: kept the Treeherder-backed regression gate (per-push status from Treeherder, ancestor chain from mozci) and added master's is_stale_push, whose param is named `project` to match the module. - test_regression.py / test_consumer.py: master's build-path tests, which use the new get_task seam, plus the test-path suite. - README.md: one pipeline covering both paths, including master's action-task and push-age steps. Two changes outside the conflicts: - consumer.py auto-merged cleanly but was left broken: master changed taskcluster.get_hg_revision to take a task definition rather than a task id, while _process_test still passed the id, which would raise AttributeError on every test-failure message. It now fetches the task once and passes it on. - The same two gates master added to builds now also apply to test failures, since the signature change was made for them and the reasoning carries over: a backfill is not a new failure, and a failure surfacing days after its push is not worth repairing. Both run before the ancestor walk, which can block. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Contributor
Author
|
Current state of things: Over ~86 minutes: ~116 failures rejected at the Treeherder gate, 4 runs triggered: |
Collaborator
Looking at treeherder for this: https://treeherder.mozilla.org/jobs?repo=autoland&revision=0faca76bc591445f1de70a0081901faea33c8242&searchStr=mochitest-browser-chrome&selectedTaskRun=c2YUKOVWS8KEqC3kvBqYVg.0. Can you also look at the other 3 triggered runs to see if they were successfully finding what needed to be found? |
marco-c
reviewed
Jul 30, 2026
| 1: "not classified", | ||
| 2: "fixed by commit", | ||
| 3: "expected fail", | ||
| 4: "intermittent", |
Collaborator
There was a problem hiding this comment.
The "intermittent" classification is manual, not sure how quick usually sheriffs are at applying it. CC @Archaeopteryx
marco-c
reviewed
Jul 30, 2026
Comment on lines
+35
to
+42
| # Treeherder classifies a failing job shortly after we see the failure, so the | ||
| # gate waits for the job to be ingested before reading that verdict. | ||
| treeherder_ingest_poll_seconds: int = 30 | ||
| treeherder_ingest_max_wait_seconds: int = 240 | ||
| # How long to wait for a verdict once the job is ingested. Classification lands a | ||
| # few minutes after ingestion, and most test failures turn out to be intermittent | ||
| # or expected-fail, so waiting here rejects them before the ancestor walk. | ||
| treeherder_classification_wait_seconds: int = 300 |
Collaborator
There was a problem hiding this comment.
@Archaeopteryx if we wait this long and then the agent takes a few more minutes to analyze, do we risk sheriffs will already have run a backfill by the time the analysis is done?
Contributor
Author
This was a dry run. I added treeherder URLs to the logs for easier inspection. I'm going to deploy it with a daily limit and then we can look at what actual agent runs look like. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tested only locally with a dry run. I will redeploy it when the agent is landed
depends on #6375
fixes #6399