fix(coding-agent): retry provider stream stalls on the shared bounded budget - #845
Conversation
Provider stream watchdog failures ("Provider stream start timed out after
<n>ms" and "Idle timeout waiting for provider stream after <n>ms") were
special-cased: the second consecutive stall escalated straight to the
fallback chain, bypassing settings.retry.maxRetries. Without a configured
chain the turn surrendered after a single same-model probe and surfaced
"Retry failed after 1 attempts".
Stalls are ordinary transient failures (isRetryableErrorMessage already
classifies them so). They now consume the same bounded same-model budget as
every other retryable class and escalate to the chain only on exhaustion.
The retry path capped both provider guards with retry.provider.streamRetryTimeoutMs (default 30s), so a configured 90s stream-start budget expired after 30s on the retry - the reported "Provider stream start timed out after 30000ms" that immediately followed a 90000ms stall. A slow-but-alive provider was judged dead on a deadline it was never given. Retries now send the configured timeouts unchanged. streamRetryTimeoutMs keeps bounding the retry continuation itself, which still cancels a wedged retry without shortening the provider's own guards.
Adds the fork-change entry and the CHANGELOG [Unreleased] note for the two retry fixes, and updates the real-CLI stall QA driver to assert the shared same-model budget (1 + maxRetries primary requests) before fallback.
CI triage: all 9 failures are pre-existing on
|
| Job | Cause | Also red on main |
|---|---|---|
| Terminal tools (ubuntu, windows) | Cannot find native binding -> @rolldown/binding-* (npm/cli#4828) |
yes |
| Inspector handoff (ubuntu, windows) | same rolldown binding | yes |
| Test (coding-agent 1/3, 2/3, 3/3) | same rolldown binding | yes |
| Test (workspaces + scripts) | scripts/build-all.test.mjs:32 expects a stale package build order (protocol/client placement) |
yes, byte-identical diff |
The rolldown failures are vitest Startup Errors - no test file executes at all, so no assertion from this PR is involved. The build-all.test.mjs assertion is about workspace build ordering; this PR touches no scripts/ file.
Local verification on this branch (macOS, where the binding resolves):
retry-fallbacksuites: 17 files, 196 tests greenagent-session-retry-events: 21 green ·packages/ai/test/retry.test.ts: 45 green- agent stream-watchdog suites: 51 green ·
settings-manager: 69 green - new + updated suites:
retry-fallback-stall-shared-budget4,provider-timeout-retry4,provider-idle-{recovery,steering}6 - root
npm run check: rc=0 · Changelog gate: PASS - real-CLI QA:
primary=4 (expected 4) fallback=1, exit 0
Both main failures need their own fix (a lockfile/optional-dependency repair and a build-all.test.mjs order update); they are out of scope here.
…ompaction provider-retry-recompaction pinned the retry request to the 30s streamRetryTimeoutMs cap that this branch removes. The queue-first assertions this regression exists for - queueAwareContinue call count, per-request user texts, and the absence of continuation_error - are unchanged; only the two capped-timeout expectations now read the configured guards.
Merge rationale: this PR introduces zero new failures
New failures introduced by this PR: none. Those six One failure genuinely was mine and is fixed here: Green on this PRStatic checks · Changelog gate · Cubic · GitGuardian · Terminal tools (macOS/Linux/Windows) · Local evidence
|
1 participant
Problem
A session hitting a hung provider ended the turn after a single same-model retry:
Two behaviors combined to produce this, and both are reversed here.
1. Stalls bypassed the shared retry budget.
agent-session.tstracked consecutiveprovider-stream stalls and escalated the second one straight to the fallback chain,
skipping
settings.retry.maxRetries. With no fallback chain configured the retry loopsurrendered - hence
Retry failed after 1 attemptswhilemaxRetrieswas 3.2. Retries shrank the configured timeouts.
provider-timeout-retry.tsclamped bothprovider guards with
Math.min(configured, retry.provider.streamRetryTimeoutMs)(default30s), so a configured 90s stream-start budget became 30s on the retry. That is the second
error line: the retry was judged dead on a deadline the operator never configured.
Change
isRetryableErrorMessagealreadyclassifies both watchdog wordings as retryable, so they now consume the same bounded
same-model budget as every other transient class and escalate to the fallback chain only on
exhaustion. The streak counter and its escalation branch are gone.
timeoutMs/streamStartTimeoutMs. No timeout valueis reduced.
streamRetryTimeoutMsstill bounds the retry continuation(
runBoundedRetryContinuation), so a wedged retry is still cancelled - without lying to theprovider about its deadline. Disabled guards are still never re-enabled.
This intentionally reverses the two 2026-07-29 entries in
src/changes.md; both are recordedin the new entry.
Verification
Failing-first, then green (TDD).
retry-fallback-stall-shared-budget.test.tscallCount2, expected 4provider-timeout-retry.test.tsstreamStartTimeoutMs90000 -> 30000Regression sweep - 313 tests green:
test/suite/retry-fallback(17 files, 196),agent-session-retry-events(21),packages/ai/test/retry.test.ts(45), agent stream-watchdogsuites (51),
settings-manager(69),provider-idle-{recovery,steering}(6).Real-CLI QA (channel 3).
mock-loop-stall-fallback.mjsdrives the built CLI against an HTTPserver that writes SSE headers and then sends nothing:
primary=4is 1 initial request + 3 retries; it was 2 before this change.mock-loop-stream-retry.mjsalso passes 2/2.Gates. Root
npm run checkrc=0; changelog gate PASS.Note:
mock-loop-transport-timeout-recovery.mjsfails identically on untouchedmain(it mkdirs
process.execPathfrom a stray argv flag) - pre-existing, untouched by this PR.Tests updated, with reasons
retry-fallback-stall-escalation.test.ts->retry-fallback-stall-shared-budget.test.ts:the old file pinned the 2-stall escalation this PR removes. The contract worth keeping
(full same-model budget) is asserted for both stall wordings in the replacement.
provider-idle-{recovery,steering}.test.ts: only the capped-timeout arithmetic changed(
30000-> the configured value). Every steering, queue-retention, and continuation-boundassertion is unchanged - the recovery suite still proves the continuation expires at 30s
while the provider request keeps its full 300s/90s guards.
Plan:
.omo/plans/stall-shared-retry.mdSummary by cubic
Retries provider-stream stalls on the shared same‑model budget and preserves configured provider timeouts, avoiding early surrender and misreported 30s retries. Previously the second consecutive stall bypassed
retry.maxRetries, and retries shranktimeoutMs/streamStartTimeoutMstoretry.provider.streamRetryTimeoutMs(30s).Review notes
packages/coding-agent/src/core/agent-session.ts: remove stall streak/escalation; treat stalls as ordinary transient errors; exclude from 429-tier routing; fallback only when the shared same-model budget is exhausted.packages/coding-agent/src/core/provider-timeout-retry.ts: forward configuredtimeoutMs/streamStartTimeoutMs; never re-enable disabled guards; setwatchdogTimeoutMs = streamRetryTimeoutMsto bound the retry continuation only.provider-timeout-retryunit; update steering/recovery/recompaction regressions to expect unchanged provider guards; real-CLI QA script asserts 1 +maxRetriesprimary requests before fallback.CHANGELOG.mdandsrc/changes.md.Rollout
retry.maxRetriesor configure a fallback chain. Otherwise no action; expect up to 1 +retry.maxRetriesprimary attempts before fallback on stalls.Written for commit c5f894c. Summary will update on new commits.