[vjovanov]

Closes #1191

A benchmark shard's install now tries each network command up to three times, each attempt bounded at four minutes, and says which command it is on. Both shard jobs are bounded at twenty minutes. A 503 or 504 from snapshot.debian.org is tried again instead of turning cargo bench (shard N) red, and a stalled fetch now fails the shard instead of holding a merge-queue group until its 90-minute timeout.

Before, a shard either failed on one mirror error:

E: Failed to fetch https://snapshot.debian.org/archive/debian/20261001T000000Z/pool/main/v/valgrind/valgrind_3.19.0-1_amd64.deb  503  No healthy backends [IP: 151.101.162.132 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

or went silent in the install step until the queue or a person cancelled it, with nothing to say which command was running (job 126984740865, 88 minutes):

2026-11-05T01:23:22.6539417Z ##[endgroup]
2026-11-05T02:51:31.4252726Z ##[error]The operation was canceled.

After, every network command names itself before each attempt. This is shard 4 of hosted run 42920806655 on this branch, where every command succeeded on its first attempt:

23:16:54.808Z bench-shard: apt-get update, attempt 1 of 3
23:16:56.324Z bench-shard: apt-get install, attempt 1 of 3
23:16:58.432Z bench-shard: cargo install, attempt 1 of 3
23:17:45.544Z bench-shard: cargo fetch, attempt 1 of 3

A 503 is now one failed attempt, not a failed step. Below is the committed wrapper, copied byte for byte from the action, run locally against a stand-in apt-get that answers 503 once (with sleep stubbed out):

bench-shard: apt-get install, attempt 1 of 3
E: Failed to fetch https://snapshot.debian.org/archive/debian/20261001T000000Z/pool/main/v/valgrind/valgrind_3.19.0-1_amd64.deb  503  No healthy backends
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
bench-shard: apt-get install failed with exit 100, attempt 1 of 3
bench-shard: apt-get install, attempt 2 of 3
step-exit=0

What changed

• The cause. Both ci.yml's bench job and bench-nightly.yml's bench job use .github/actions/bench-shard. Its step Install Valgrind and iai-callgrind-runner ran apt-get update, apt-get install, cargo install iai-callgrind-runner and cargo fetch once each, with no time bound, and neither job had a timeout-minutes. That caused both failure shapes in the issue:
  • A 503/504 that lasted longer than apt's per-item Acquire::Retries=5 turned the shard red.
  • A connection that kept trickling bytes held the shard until the merge queue's 90-minute check timeout. Triage reproduced this in the shard's own image (rust:1.95.0-bookworm@sha256:6258907a…): it ran the shard's apt-get -o Acquire::Retries=5 update -qq against a loopback mirror that sends one byte every 20 s. apt stayed on its first connection until the outer bound killed it, and one run held it for 1320 s. A mirror that accepts and never answers does not stall apt: apt gives up after 211 s. Only the trickle does. The live stalls' logs end before dpkg's first line, so the stall was in apt's fetch. That is inferred, not shown, because -qq and --quiet hid the command.
• The fix. The step now defines one wrapper, fetch LABEL COMMAND..., and all four network commands go through it (§AR-ci.5.6):
  • each attempt runs under its own timeout --kill-after=10 240;
  • up to three attempts, with sleep 30 only after a failed one, so a healthy install never waits;
  • bench-shard: <label>, attempt <n> of 3 is printed before each attempt, and bench-shard: <label> failed with exit <status>, attempt <n> of 3 after a failed one;
  • after the third failure the step exits with the command's own status (100 for apt, 101 for cargo, 124/137 when the bound ended it), not a generic 1.
• The job bound. Both shard jobs carry timeout-minutes: 20. This catches a stall that no per-command bound covers, including one in cargo bench itself. Even if one command fails all three attempts at full length (3 × 250 s + 2 × 30 s = 810 s), the step fails and names that command with more than five minutes to spare. A command that recovers on its last attempt adds about 9.3 minutes. On the durations triage measured (56 passing pull-request shards on 2026-11-10, 70 nightly shard jobs from 2026-11-01 to 2026-11-10), that still ends a pull-request shard inside 15 minutes and a nightly shard inside 20.
• What did not move. Every pin stays as it was: the snapshot timestamp 20261001T000000Z, valgrind=1:3.19.0-1, the libc6-dbg="$(dpkg-query …)" expression, runner 0.16.1 and the image digest. A retry asks the same snapshot for the same versions. -qq and --quiet stay too, as §AR-ci.requirements.5.4 requires. The marker lines name the command instead.

Specification points

All are architecture points in docs/architecture/AR-ci.md, because this is the repository's own CI. No FS or REQ point moves, and nothing a user of grund sees changes.

• §AR-ci.5.6 is new. It holds the wrapper, the numbers and why they are what they are, the job bound, the rule that no pin moves, and a pointer to the test that holds it.
• §AR-ci.5: one sentence changed. "If the snapshot is unreachable the step fails; rerun it" now reads: a snapshot unreachable for a while is tried again, and one that stays unreachable fails the step in bounded time. The one-snapshot rule is otherwise word for word.
• §AR-ci.requirements.2: one sentence added. The two shard jobs already carry the twenty-minute bound. The mark stays deferred, because only these two jobs are bounded, not every ci.yml job.
• docs/file-size-agent-exceptions.toml: AR-ci.md's ceiling goes from 1199 to the 1200 non-blank lines it now measures, with a reason paragraph in the file's usual form.

Found and deliberately not fixed

• R1-01 (minor, deferred). The 240 s bound also covers apt-get install's dpkg phase, which takes about 2 s. apt runs dpkg in its own session, so a kill that lands there leaves dpkg locked or interrupted, and both retries fail with exit 100. Review reproduced this in the shard's image with a shortened bound. It needs the bound to fire in the last ~2 s of a download that would otherwise have finished. The result is a red shard in bounded time: no worse than before, and not the 90-minute hold. The follow-up is to put only apt-get install --download-only under fetch and install from the cache outside it. A kill during the download or during apt-get update does recover: review saw attempt 2 resume the download with a Range: request and install.
• R1-02 (adjacent). Other jobs install pinned crates from crates.io with no bound or retry: ci.yml (lychee, fissile), ci-leg.yml, macos-suites.yml and links-nightly.yml. This is out of scope. §AR-ci.5.6 governs the shard's install alone, every 503/504 on record names snapshot.debian.org, and the repository's flaky-test history has no crates.io install flake.
• R1-03 (adjacent). grund does not scan .github/actions/, so grund check never resolves the action's §AR-ci.5.6 citation, nor its older ones. The citation is correct. The gap predates this change and belongs to a change of its own.
• Deferred by design: a cache of the two pinned .debs. §AR-ci.5.6 names it as the next step if an outage ever outlasts the three attempts.

How it was verified

• tests/integration/test_bench_shard_install.py (new, cites §AR-ci.5.6) does not read the install as text. It runs it. It inlines the composite action into each shard job and runs the install step under bash --noprofile --norc -eo pipefail, as the step's shell: bash does. apt-get, cargo, timeout and sleep are replaced by stand-ins. For each of the four commands, in both ci.yml and bench-nightly.yml, it runs four scenarios: healthy, fails once, stalls once, and down for good. It also reads each job's timeout-minutes. Before the fix, its two shard tests failed with thirteen problems each, one per missing property. Its ten fixture checks run the same checks against altered copies of the install, so a check that can no longer fail is caught: the old install fails every property, and a retry that bash -e ends, one bound around all attempts, unwrapped cargo, a retry with no message or pause, and a bound too short or too long for the job are each caught. The module runs in about 1.7 s.
• Locally, test_bench_shard_install, test_ci_bench_jobs and test_bench_runtime_pins ran 22 tests: OK, none skipped. Triage's reproducer exits 0 on the branch head 7f425602de. pre-commit run --all-files, the whole CI gate, exits 0 in 20m58s.
• Hosted run 42920806655 on 7f425602de: the run succeeded, including cargo bench (build) and all seven shards. Each shard ran every command once, on attempt 1. The retry path has run only against the test's stand-ins, not against the real mirror, because the mirror did not fail during this run.

How this ran: the simple path, with triage, specify, implement, one review round (approve, no blocker or major), then green. Fix round 1 and the second review and fix rounds were cancelled, and no tier was raised. The green step is a program, so it does not appear in the list of agent invocations below.

AI workflow: `rhei`, 14 agent invocations across 1 model; 6 tasks completed, 10 in progress.

1. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket supervising (visit 1) — cld, anthropic/claude-opus-5-5 — 1.0s — 0 in / 0 out
2. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket supervising (visit 1) — cld, anthropic/claude-opus-5-5 — 2.0s — 0 in / 0 out
3. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket supervising (visit 1) — cld, anthropic/claude-opus-5-5 — 1m48s — 1.3M in / 10.3k out
4. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket.triage assessing (visit 1) — cld, anthropic/claude-opus-5-5 — 30m02s
5. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket.triage assessing (visit 1) — cld, anthropic/claude-opus-5-5 — 15m14s — 4.0M in / 32.7k out
6. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket.triage reproducing (visit 1) — cld, anthropic/claude-opus-5-5 — 8m54s — 4.6M in / 34.3k out
7. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket supervising (visit 2) — cld, anthropic/claude-opus-5-5 — 2m52s — 2.3M in / 16.1k out
8. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket.specify specify — cld, anthropic/claude-opus-5-5 — 57m23s — 13.4M in / 107.6k out
9. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket supervising (visit 3) — cld, anthropic/claude-opus-5-5 — 1m16s — 715.7k in / 7.7k out
10. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket.implement implement — cld, anthropic/claude-opus-5-5 — 74m48s — 4.2M in / 20.2k out
11. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket supervising (visit 4) — cld, anthropic/claude-opus-5-5 — 57.0s — 507.0k in / 6.0k out
12. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket.review-1 review — cld, anthropic/claude-opus-5-5 — 10m22s — 5.2M in / 54.2k out
13. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket supervising (visit 5) — cld, anthropic/claude-opus-5-5 — 1m07s — 864.9k in / 5.9k out
14. github-issues-agent-grounds-grund-1191-implement-6567849c.ticket supervising (visit 6) — cld, anthropic/claude-opus-5-5 — 42.0s — 638.4k in / 4.5k out

Accounting	Value
cost	$19.74
total tokens	38.0M
input tokens (incl. cache)	37.7M
input cache read	36.4M
input cache write	1.3M
output tokens (incl. cache)	299.6k
output cache read	-
output cache write	-
coverage	Partial