Repository navigation
Conversation
BatchReadBlobs applies a hardcoded 30 s deadline to each blob read so a single stalled read cannot pin the whole batch until the client's overall deadline. Under cold large-tree reads against a slow backing tier -- e.g. an R2/S3 object store behind a fast store doing writeback backfill on read -- large blobs legitimately take longer than 30 s to materialize, and the hardcoded deadline fails the read with DeadlineExceeded even though the transfer is making progress, failing the entire batch. Add an optional per-instance `batch_read_per_blob_timeout_s` field to the CAS service config. Unset preserves today's 30 s default exactly; deployments with slow cold tiers can raise it. The simplest defensible design -- a configurable base rather than a size-scaled formula -- was chosen deliberately: the operator knows their tier's throughput floor, and a scaling heuristic baked into the server is hard to tune and harder to remove later. BatchUpdateBlobs keeps the fixed default. The new regression test stalls the store for 2 minutes and shows the read succeeds with a configured 3 minute timeout; with the fix reverted the configured value is ignored and the read fails with DeadlineExceeded at 30 s (the existing default-timeout test continues to cover the unset case). Co-Authored-By: Claude Opus 4.8 <[email protected]>
…out_s` Regenerated web/apps/docs/content/docs/reference/nativelink-config/main.mdx from the nativelink-config crate (build-schema + schema-to-mdx), as AGENTS.md requires when a config field or its doc comment changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
feat(cas): make the BatchReadBlobs per-blob timeout configurable
BatchReadBlobs applies a hardcoded 30 s deadline to each blob read so a
single stalled read cannot pin the whole batch until the client's
overall deadline. Under cold large-tree reads against a slow backing
tier -- e.g. an R2/S3 object store behind a fast store doing writeback
backfill on read -- large blobs legitimately take longer than 30 s to
materialize, and the hardcoded deadline fails the read with
DeadlineExceeded even though the transfer is making progress, failing
the entire batch.
Add an optional per-instance
batch_read_per_blob_timeout_sfield tothe CAS service config. Unset preserves today's 30 s default exactly;
deployments with slow cold tiers can raise it. The simplest defensible
design -- a configurable base rather than a size-scaled formula -- was
chosen deliberately: the operator knows their tier's throughput floor,
and a scaling heuristic baked into the server is hard to tune and
harder to remove later. BatchUpdateBlobs keeps the fixed default.
How was this verified?
The new regression test stalls the store for 2 minutes and shows the
read succeeds with a configured 3 minute timeout; with the fix
reverted the configured value is ignored and the read fails with
DeadlineExceeded at 30 s (the existing default-timeout test continues
to cover the unset case).
Also part of the soak branch (#73): upstream HEAD plus every pending fix, run through the railgun fleet for an hour of continuous remote builds with no errors.
docs(config-reference): regenerate main forbatch_read_per_blob_timeout_s``: regenerated withgen:config-reference, never hand-edited.Risk
Unset keeps today's 30 s default exactly; only deployments that set
batch_read_per_blob_timeout_schange behaviour. BatchUpdateBlobs is untouched.AI assistance
Written with Claude Code; b7r6 reviewed the change. Draft until the author has read the CI results.
🤖 Generated with Claude Code