Stop Azure downloads from retrying a failed destination

[b7r6]

What and why

When an Azure download loses its destination, the failed send becomes Aborted, which the default policy retries. Each retry retains the broken writer and starts another SDK download. Stop retrying once the writer is known to be broken, preserving the original error.

Two commits: first the failing Rust regressions, then the production fix and backend documentation. The production change adds one terminal-condition check.

How was this verified?

The larger regression uses the real AzureBlobStore, Azure SDK, retrier and buffer channel. Only HTTP is simulated: HEAD, ranged GET, paced response bodies, and capacity-driven 503s with Retry-After. It first exhausts the account's token budget and completes a read after capacity recovers. It then compares a complete read with a consumer that closes after receiving 4 MiB of a 128 MiB object.

The measurements below use the original archived conditional log-duration fit: its 64,810 records were later found to contain 3,522 duplicate completed bodies, including evaluation responses under a training label. The original point fit sampled 500 of those leaked responses. The calibration correction and new four-worker batch in #70 use 61,288 distinct completed bodies across 23 capture groups. This two-commit regression history and its original measured byte counts remain unchanged; its timing profile must not be presented as independently validated. Both profiles use 129 empirical residual quantiles from 1,728 validation observations. Token budgets and consumer closure are explicit stress conditions, not inferred production behavior; no sampled throttle percentage or programmed egress multiplier is used.

Thirty seeds (2026101700..2026101729), paired across both commits, with alternating healthy/closed arms and paused Tokio time. The process saw 24 CPUs. The byte ledger counts body chunks actually yielded to the SDK, including detached work after return; overlapping byte offsets identify re-purchases. Every run checks byte conservation.

Closed-consumer results (mean ± sample SD; 95% t intervals over seeds):

Measurement	Before fix	After fix
Delivered to consumer	4 MiB	4 MiB
HTTP payload transferred	770.0 ± 27.6 MiB [759.7, 780.3]	121.9 ± 8.1 MiB [118.8, 124.9]
Re-purchased bytes	642.7 ± 26.7 MiB [632.7, 652.6]	0 MiB
Retries after failed delivery	6	0
Time until read returns	38.60 ± 2.98 s [37.49, 39.71]	0.100 ± 0.045 s [0.083, 0.117]

Paired payload reduction: 648.1 ± 25.4 MiB, 95% CI [638.6, 657.6]. Re-purchase violations fall from 30/30 to 0/30 (Wilson 95% intervals [0.886, 1] and [0, 0.114]). Healthy reads transfer exactly 128 MiB with no retries in both versions. All probe requests succeed at the HTTP boundary: the repeated transfers occur after the account has recovered.

Payload median/p95/p99: before 766.0/828.8/844.5 MiB; after 128/128/128 MiB. Return-time median/p95/p99: before 39.289/42.931/43.473 s; after 0.091/0.137/0.259 s. These describe virtual fixture runs, not estimates of production incidence.

Each run prints its seed and 100 ms transfer histogram. For seed 2026101700, the baseline buys 784 MiB in successive retry bursts through 42.6 s; the fixed read buys 128 MiB once. The smaller regression independently checks the terminal decision both directly and after an initial 503.

Reproduce either commit with:

cargo test -p nativelink-store --test azure_blob_store_test --test azure_download_egress_test -- --nocapture
bazel test --test_env=PATH //nativelink-store:integration_tests/azure_blob_store_test_test //nativelink-store:integration_tests/azure_download_egress_test_test

The first commit intentionally fails both new regressions; the second passes all 23 tests in Cargo and both Bazel targets. Strict Cargo Clippy, rustfmt, pre-commit hooks, and documentation checks pass. --test_env=PATH supplies Bash to the Bazel test runner on this Nix host.

A four-worker end-to-end follow-up is in #70, with actual scheduler/worker processes, real ENOSPC, SDK HTTP traffic, and complete archived evidence. Forty real-clock invocations cover ten paired seeds in each of control and failed-fill scenarios. With identical completed work and the corrected timing profile, the failed-fill comparison saves 309.2 MiB of emulator HTTP payload (95% CI [271.8, 346.6]) and reduces mean work-drain time from 41.66 s to 6.22 s. The original four-worker batch is preserved separately rather than pooled with the corrected batch. All original workers release their reservations and reuse their warm caches. Faster recovery increases burst traffic; this demonstrates a finite retry tail, not the historical hours-long silent wedge.

Risk

Transient Azure failures remain retryable while the destination is usable. This stops new NativeLink retries; it does not cancel requests the SDK already started. The fixture still measures those bytes after closure (mean 51.9 MiB with the fix). It establishes this read-amplification mechanism, not its share of the customer incident. The calibrated response draws omit within-run correlation, and the test's fixed 128 MiB object does not represent a fitted customer blob-size distribution.

AI assistance

Codex implemented the reproduction, calibration export, fix and verification at b7r6's request, building on the joint investigation and its captured Azure stress data. Codex drafted this description.