Hold a worker for a queued action whose reservation only fits an idle one

[amankrx]

What and why

An action whose reservation only fits an idle worker starves under load. Every matching pass tries it first, finds each worker partly busy, then fills the room that opens with newer, smaller actions, so no worker ever drains for it while the fleet reads as free. One deployment had 10,610 of 23,866 targets in a 16 GiB class on 20 GiB workers; one replica logged 27,202 live-memory vetoes for them in 30 minutes while the queue grew.

reservation_hold (off by default) holds a worker for such an action once it has waited hold_after_s and fits no busy worker until more than one of that worker's actions finish; an action one completion would make room for is only waiting its turn, which the pass order already gives it. The worker chosen fits the action idle, has not declined that much while idle, and has the fewest actions running and then the most memory free as last reported. It is offered nothing else, and the action is placed there as soon as the ledger and the live veto allow. The hold stays through the dispatch until the worker takes the action; a decline sends the action back with the hold kept, unless the worker declined it idle, in which case the next pass holds another. The hold ends when a worker takes the action anywhere, when it leaves the queue, when the worker leaves or drains, or after max_hold_s; an expired hold is not repeated until as long again has passed, twice as long for each expiry in a row on an idle worker. At most max_held_workers are held per replica, and replicas sharing a queue take turns per action by a hash over the replicas whose workers would fit it, so one worker is held fleet-wide. Both durations must be above zero. Counter scheduler.worker.holds{scheduler.hold.outcome}, gauge scheduler.worker.held.

How was this verified?

reservation_hold_test.rs: held then placed when the worker drains, no hold for an action one completion makes room for, expiry and cooldown, cooldown doubling on an idle expiry under the live veto, a declined dispatch keeping the hold, an idle-declined worker passed over and a hold moving after an idle decline, the memory tiebreak, release on drain, leave and when the action leaves the queue, placement elsewhere, the cap, and replicas taking turns. Scheduler, service, util and config suites and clippy pass.

Risk

Off unless configured; a held worker runs below capacity for at most max_hold_s, and the cap bounds how many per replica.

AI assistance

An agent drafted the change and I reviewed every line.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
nativelink	Ready	Preview	Oct 13, 2026 9:01pm UTC
nativelink-aidm	Ready	Preview	Oct 13, 2026 9:01pm UTC