Skip to content

Reduce scheduler contention after lost assignments - #2954

Merged
MarcusSorealheis merged 3 commits into
TraceMachina:mainfrom
MarcusSorealheis:codex/tra-1475-matcher-contention
Oct 18, 2026
Merged

MarcusSorealheis merged 3 commits into
TraceMachina:mainfrom
MarcusSorealheis:codex/tra-1475-matcher-contention

Conversation

@MarcusSorealheis

@MarcusSorealheis MarcusSorealheis commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

What and why

When schedulers race to assign the same queued action, the loser currently keeps competing for every following action with the same platform properties. After a lost assignment, this change skips that property shape for the remainder of the pass, including parked actions revisited when capacity opens. Other shapes remain eligible, and the next pass considers all shapes again.

Diagnostic logging now samples up to five oldest queued actions from the matcher's existing listing, removing the additional queued and executing database listings. Matching diagnostics remain disabled by default. The platform-properties documentation describes the new behavior.

The executing sweep, running-operation reports, and default logging change already landed in #2923, #2925, and #2924. This preserves their reviewed decisions: lost results consume a retry, and worker keepalives do not extend the execution ceiling. Reliable result resends require an acknowledgment protocol and are outside this change.

How was this verified?

  • cargo test -p nativelink-scheduler --locked: 246 passed, 1 ignored on macOS.
  • Both new tests fail against the original matcher and pass with this change. They force a peer to win an assignment, check that only the matching property shape waits, and confirm it runs on the next pass. The second case loses the assignment while replaying parked actions after capacity opens.
  • The logging-enabled test gates database listings so an extra diagnostic queue query cannot complete the matching pass.
  • cargo clippy -p nativelink-scheduler --test simple_scheduler_test --locked -- -D warnings passed.
  • Repository pre-commit hooks passed for the changed files. Docs anchor, snippet, navigation, and corpus generation checks passed after generating the changelog prerequisite.

Bazel and live-cluster validation were not run.

Risk

A contested property shape may wait for the next matching pass while local capacity is available. If fallback matching is disabled, that pass needs a task or worker notification. No retry budget, worker reservation size, or persistent state changes. Diagnostic consumers should account for the new Oldest actions in the matching pass message, which samples queued actions rather than logging a separate executing list.

AI assistance

Automated assistance was used to draft the changes and run the listed checks.


This change is Reviewable

@vercel

vercel Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 17, 2026 11:11pm UTC
nativelink-aidm Ready Ready Preview Oct 17, 2026 11:11pm UTC

Request Review

@MarcusSorealheis
MarcusSorealheis merged commit 8431150 into TraceMachina:main Oct 18, 2026
41 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — 23d87ef9 Deployed Oct 17, 2026 by vercel[bot]
Preview – nativelink-aidm — 23d87ef9 Deployed Oct 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants