Skip to content

coordinator: a master push with only no-runner jobs is reported as a code break #29

Closed

Description

@vpacher

The coordinator treats a master push that failed only because a job got no runner as a code break. This is step 1 of the CI changes agreed in obs-rust/obs-studio#3633. It changes only how the coordinator reacts, so it can land alone.

What happens

On 11-04, four master pushes of obs-rust/obs-studio in a row were red (41874818392, 41885535539, 41888783212, 41897411183). Every failed job was a macOS job with conclusion cancelled, no runner and no steps, and GitHub's message "The job was not acquired by Runner of type hosted even after multiple attempts". Every job that ran passed.

For such a push:

  1. red_branches in workflow/gh.py sees conclusion == failure, so the work list has fix-master.
  2. The fix-master text says to find the PR that broke it, and fix or revert it first. No PR broke it.
  3. gh run view --log-failed returns 0 lines for these runs, so "read the failed job log" gives nothing.
  4. tasks() skips fix-ci for a repo that is red, so unrelated PR failures wait.

The flake lists in the coordinator skills do not name this case.

GitHub's status page does not cover it alone. Of the runs on 11-04 with such a job, only one started inside an Actions incident (16:05-17:01 and 17:40-18:16 UTC). The check must use the job's own data.

Scope

  1. Classify in workflow/: a failed push run where every failed job is cancelled with no runner and no steps gets a rerun task, not fix-master, and does not count as red for fix-ci. A run with any other failed job stays fix-master.
  2. Show GitHub's status: workflow.coordinator prints unresolved GitHub incidents at the top of the work list (https://www.githubstatus.com/api/v2/incidents/unresolved.json, no login). If that read fails, the list still prints and says so.
  3. Skills: obs-org-coordinator, and obs-coordinator in obs-rust/obs-studio, add this case to the flake lists: the log is empty, rerun the job, and during an incident do not revert or escalate.

Not in scope

Which jobs run where (the other steps of obs-rust/obs-studio#3633), and no-runner failures on a PR or in the queue. The author can rerun on a PR, and the queue already prints a runner-capacity message (ci/gate_jobs.py).

Acceptance criteria

  • A test in tests/test_coordinator.py, RED first: a push run with only no-runner jobs gives the rerun task and no fix-master, and fix-ci tasks for that repo stay on the list.
  • A test: a push run with one no-runner job and one failed job with steps stays fix-master.
  • A test: an unresolved incident prints at the top of the list, and a failed status read does not stop the list.
  • --sim still works with no status read.

Activity

  1. vpacher commented on Nov 5, 2026

    @vpacher
    MemberAuthor

    Claiming: this issue, as scoped above

    1. One PR here: the classification, the GitHub status lines, and the obs-org-coordinator skill.
    2. One small PR in obs-rust/obs-studio: the same case in the obs-coordinator skill.

    The first PR is ready to open today.

  2. moved this from Triage to In review in obs-rust Multiviewon Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

readyMeets the definition of Ready; claimable

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions