Retry an action's input fetch on the worker after a transient store failure

[amankrx]

What and why

A worker that fails an input fetch hands the action back, and the scheduler sends it to another worker to fetch the same inputs from the same store. Against an object store answering 503 for minutes, one customer's workers failed 788,690 fetches in 18 hours that way, each spending a scheduler retry on a store that would not answer anyone.

The worker now fetches the Command and the input tree again itself after a transient store error, from an empty work directory, pausing 1 s doubling to 30 s, while max_download_timeout_s leaves room for another attempt or the action's own timeout when shorter. The scheduler is not told; its ceilings still bound the action, and the max_download_timeout_s doc says to stay below them. Only store failures are retried: an error the worker's own OS raised (full disk, link limit, read-only mount) hands the action back to land on a healthy worker. Counter worker.prepare.retries{outcome}: retried per attempt, then one of recovered, exhausted, failed, abandoned.

How was this verified?

Tests under paused time: recovery after 1 s and 2 s; a partly built tree cleared and restaged; a store that never recovers stopping at the download deadline and at a shorter action timeout; a kill during the pause. Unit tests cover the classifier, OS errors and the outcome tally. Worker suite and clippy pass.

Risk

Low. A permanent store error still fails at once; a transient one costs the worker slot up to the window instead of a scheduler retry.

AI assistance

An agent drafted the change and I reviewed every line.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
nativelink	Ready	Preview	Oct 13, 2026 5:29pm UTC
nativelink-aidm	Ready	Preview	Oct 13, 2026 5:29pm UTC

[corcillo]

Separate from this PR, but this adds a call site: fs::remove_dir_all (nativelink-util/src/fs.rs:541) chmods regular files to 0o600 once a directory in the tree returns PermissionDenied. Staged inputs are hardlinks to CAS blobs (0o444) and executable variants (0o555), so that chmod changes the shared inode (the #2347 class): an executable variant loses its execute bit for every other action using it. It only triggers on a tree containing a read-only directory, but clearing the work dir mid-prepare during a throttle makes it more likely. Worth an issue: clear trees the way directory-cache eviction does (make directories writable, unlink files, never chmod files).