Reproduce Azure failed-fill egress across four real workers

[b7r6]

What and why

Adds a four-worker reproduction of the failed-destination retry defect fixed in #69. Real LocalWorker processes, the NativeLink scheduler over gRPC, filesystem/fast-slow/verify stores, and the Azure Blob SDK share a loopback HTTP account. Filling two private tmpfs filesystems causes real ENOSPC; removing the filler repairs them without replacing the workers. The baseline removes only #69's terminal-writer guard.

This companion PR is based on fix/azure-terminal-read; #69 remains two commits. Seeded workload distributions drive actual SDK requests, and account capacity admission causes every simulated 503. No traffic multiplier or sampled throttling percentage is injected.

The calibration audit removed 3,522 duplicated completed bodies caused by ingesting partial capture snapshots as independent runs. Those included 1,929 evaluation responses under a training label, of which 500 entered the old sampled point fit. Corrected ingestion merges snapshots before computing overlap or assigning splits and rejects conflicting copies. The regenerated fit uses 61,288 distinct completed bodies across 23 capture groups. Evaluation remains retrospective on previously inspected captures, not fresh prospective validation. Original results and artifacts are preserved alongside the corrected batch.

How was this verified?

Forty new real-clock invocations under the corrected timing profile: ten paired seeds (2026101800..2026101809) in each of healthy-control and failed-fill scenarios, alternating which version runs first. Both arms complete identical work and return matching output paths/digests.

Failed-fill measurement	Baseline mean ± SD	Fixed mean ± SD	Paired fixed − baseline, 95% CI
HTTP body payload	1458.8 ± 43.5 MiB	1149.6 ± 74.0 MiB	−309.2 MiB [−346.6, −271.8]
Time to drain submitted work	41.66 ± 2.61 s	6.22 ± 1.31 s	−35.44 s [−37.21, −33.66]

Both failed-fill arms reach ENOSPC on all 20 deliberately filled filesystems. The baseline exhausts retries against a closed writer on all 20; the fixed arm does so on none. Healthy-control differences include zero: payload +12.4 MiB [−8.7, +33.5], work drain −0.18 s [−0.65, +0.28]. These intervals are not equivalence tests.

All 836 work items completed, plus 160 warmups and 160 sentinels. Every planned command ran once. All sentinels reused warm caches in the original worker processes; final scheduler snapshots retained original membership with zero reservations. No reports or client outcomes are missing or censored. All 44 failed internal preparations remain in the logs. Independent audits replayed admission decisions, reconstructed byte-range unions, checked lifecycle and scheduler accounting, and checked process/mount cleanup.

The README contains full statistics, boundaries, paired figures, and reproduction commands. The new 4.1 MiB evidence archive preserves every request/action ledger and worker log, measured source snapshots, three audit reports, corrected timing observations, fit/export code and a SHA-256 inventory. All 367 payload hashes verify; analysis after relocation reproduces all 40 runs and 20 paired comparisons. The original archive remains unchanged. The binary-hook exceptions name only the two evidence archives.

Validation:

• Fleet Cargo/Bazel tests, including real SDK HTTP behavior, admission/recovery, dropped-body accounting, interval unions and filesystem hardlinks.
• Both Azure store regression Bazel targets; strict Clippy/rustfmt and pre-commit hooks.
• Nine standard-library calibration-checker tests, including refused requests, incomplete bodies and unknown denominators.
• Corrected ingestion/export tests, seven Rust closed-form tests, strict Clippy and numerical verification over all 61,288 reads.

The new calibration checker compares measurement boundaries explicitly. The stress fixture does not match the customer's full request profile: pooled HEAD/GET is 0.131–0.185 versus about 0.40, every complete GET body is 4.194304 MB versus the reported rest band of 2.7 MB ±25%, and tiny command outputs make body out/in ratios millions rather than about 15. These mismatches remain visible; the fixture is a targeted failed-fill experiment, not a fitted customer workload.

Risk

This adds a Linux experiment and development dependencies. It requires user/mount namespaces and creates private tmpfs filesystems. Four processes share one host, not four Azure VMs. Scheduler state is in memory. The emulator implements the Blob operations exercised here, not the entire Azure service. Bytes count HTTP body production, not Azure billing; the measured fork revision is recorded rather than presented as an exact historical release.

Faster recovery increases burst traffic: mean throttled GETs rise from 2.6 to 19.3. Already-started SDK transfers and legitimate retries remain. The fix reduces total bytes and delayed recovery here, not every traffic metric. Independent timing residuals omit within-run correlation. The original and corrected profiles were collected sequentially; their difference cannot be attributed solely to calibration, nor pooled as independent seed repetitions.

This reproduces a finite retry tail, not the customer's hours-long silent wedge or the full production plateau. GET-only duration and Azure successful all-API latency have different denominators and are not interchangeable calibration targets. The read plateau's production trigger and incidence remain unassigned.

AI assistance

Codex and three delegated agents implemented and reviewed the harness and calibration corrections, preserved both measurement batches, and performed independent evidence, accounting and lifecycle audits at b7r6's request. Codex drafted this description. The response-time fit uses the investigation's captured Azure stress data; workload distributions remain declared stress conditions.