CI: conformance-smoke shards hit the 60-min timeout-minutes cap; slowest shard cancels and blocks merges #6456
Description
The conformance-smoke gap-suite shards have timeout-minutes: 60, and the slowest shard is running right up against it — intermittently (sometimes) or consistently (some PRs) exceeding 60 min and getting cancelled. Because conformance-smoke-complete requires every shard to succeed, one cancelled shard fails the whole check, blocking merges on green code.
Evidence (this week)
| PR | symptom |
|---|---|
| #6449 | shard 1 cancelled at ~60 min on the original run, 2 reruns, AND a fresh commit — 4× consistently |
| #6452 | shard 1 cancelled on original + rerun (2×) |
| #6444 | a shard cancelled on the original run; a fresh full re-shard passed → borderline, not always-over |
| #6434, #6387 | one shard cancelled at the cap while the other 7 passed |
In every case the completed shards had zero untriaged failures — the code is fine; the shard just doesn't finish in 60 min.
Why it's not concurrency-cancellation
The cancellations happen on runs that were not superseded by a newer push (e.g. #6449's current head with nothing after it), and the durations sit at ~55–60 min (the timeout-minutes: 60 value), so these are job timeouts, not cancel-in-progress.
Cost
A rerun helps only when the shard is borderline (#6444). For a consistently-over shard (#6449) reruns never clear it, so the PR can't merge without an admin bypass. This is now a recurring per-PR tax.
Options
- Rebalance the 8 shards — the split is by test-name hash, so a few slow tests can pile onto one shard. Balance by measured duration, or split into more shards.
- Raise
timeout-minutesforconformance-smoke(e.g. 60 → 90) as a stopgap. - Speed up the slow tests — CI: stop truncating gap-suite output, add per-test summary #796 (closed) noted an O(n²)
normalize_outputin the harness; if that's still in the diff path it inflates the slow shard.
Not blaming any single PR — this is CI-infra. Filing so it's tracked rather than absorbed as rerun friction. Related: the chronic cargo-test 120-min-cap timeouts.
Activity
ci: raise conformance-smoke shard timeout 55->75 min (#6456) (#6478)
proggeramlug commentedon Jul 16, 2026
Interim fix landed in #6478 — raised the per-shard
timeout-minutes55→75, which unblocks merges immediately (the deterministic shard-1 55:18 timeout now has ~20 min headroom).Reopening to track the durable fix: raise the shard count
8 → 12in theconformance-smokematrix so no single--shard N/8slice approaches the cap as the gap suite keeps growing. The timeout bump is a headroom band-aid; re-sharding is the scalable fix (per-shard wall time drops ~33%, back into the comfortable 22-30 min range).proggeramlug commentedon Jul 30, 2026
The reported timeout mechanism was removed by #6478: conformance-smoke shards now have a 75-minute limit, giving the measured ~55-minute heavy shard about 20 minutes of headroom. If a shard again approaches the new ceiling, that should be tracked with fresh timing data rather than keeping the obsolete 60-minute-cap report open.