[ranxianglei]

Summary

The upstream 429 governor (upstreamThrottleLocalReject / noteUpstreamThrottle429 / noteUpstreamThrottleResolved, #4204) was wired only on the relay forwarding path. Three loop-lane egress chokepoints bypassed it, so when the upstream continuously returns 429 for compress requests, the governor never arms on the lane most prone to storms. This wires them in.

What changed

Each site now reports its verdict to the governor's observe pair — a 429 arms the egress hold for the next client turn; a settled non-429 clears it:

Site	File	Fetch primitive	Observation
Responses-wire compress loop (round-2+)	src/compress-loop-responses.ts	fetchWithRetry	terminal boundary: settled ok -> resolved; thrown UpstreamHttpError(429) -> arm
Preflight refold summary	src/preflight.ts	fetchWithTimeout	per response, like relay
Reasoning-guard continuation rounds	src/reasoning-guard.ts	fetchWithTimeout	per response, like relay

Observe-only, no local gate — matching the main-loop fix in #4725. A mid-loop local reject would change re-request behavior under load; the issue's ask ("stop amplifying") is met by arming the hold so the NEXT turn's relay forward answers locally instead of storming again.

The observation split follows each site's fetch primitive: fetchWithRetry resolves only ok responses and throws UpstreamHttpError(status) on a terminal non-ok, so those sites report at the boundary; fetchWithTimeout hands back the raw response, so those sites report per response exactly as the relay path does.

Coverage map (#4743 family)

• Main Anthropic/OpenAI-wire loop -> src/loop/core.ts ([PR] chore: empty PR (no changes) #4725, in flight)
• Degenerate-turn continuation retry -> src/degenerate-retry.ts ([PR] fix: review-findings wave — 7 fixes from the #4725 pre-release audit (P1×2 + P2×3 + P3×2) #4752, in flight, tagged [wiring] 429 governor is invisible to the loop lane (compress round-2+ resends do not go through gate/observe) #4743)
• This PR: Responses-wire loop, preflight refold, reasoning-guard rounds

These three together close the loop-lane coverage for #4743; they touch disjoint files.

Tests

tests/issue4743-loop-lane-429-observe.test.ts — three wiring pins that drive the real code path against a 429 upstream and assert the hold armed (and cleared on a later 200). Unlike the #4725 governor-contract pin, these FAIL on the unpatched tree (verified locally: 0/3 pass without the fix, 3/3 with). Targeted regression suites for the touched paths (reasoning-guard, preflight-transport-recovery, preflight-transient-empty-retry, responses-guard): 25/25 green locally.

Closes #4743

Review fixes (agent review, #4772 thread)

• 50b2bf859 — loop-lane boundary report: a terminal NON-429 UpstreamHttpError (e.g. fail-fast 400) now reports resolved to the governor, matching the Provider 429 (concurrency-style limit): turn re-driven every few seconds for minutes, keeps the account pinned at the limit #4204 contract ("ANY resolved non-429 … clears the hold") and the relay / [PR] fix: review-findings wave — 7 fixes from the #4725 pre-release audit (P1×2 + P2×3 + P3×2) #4752 sites; transport deaths stay unobserved ([bug] 上游在会话中途不可用时，bili 把一次故障放大成四重代价：固定节奏 20 连击全部冷发、输出预算被静默钳制 83 次（最狠 65536→1360）、紧急收缩与 nudge 停摆均不可见 #3148 territory). The responses-wire pin gains a re-arm → fail-fast-400 phase asserting the hold lifts (verified failing without the fix line).

• c1e0b726b — self-block regression found in review and fixed. The per-response 429 arm in the preflight walk violated the fix: honor anthropic-beta context-1m to size the context window #301 contract: the fits-case forward runs AFTER preflight and passes the relay gate, so a summary-probe 429 armed the hold that then answered the SAME request's own forward locally (CI red: tests/preflight-fail-fast.test.ts fix: honor anthropic-beta context-1m to size the context window #301 e2e, reproduced locally 429 !== 200). Fix: the walk reports resolved per response only; the arm moves to the caller's withhold verdict in server.ts (kind === "upstream" && status === 429 → note with null retry-after). Nothing is forwarded on a withheld request, so the armed hold answers only the NEXT turn's relay forward locally — the [wiring] 429 governor is invisible to the loop lane (compress round-2+ resends do not go through gate/observe) #4743 ask is preserved. Doomed-413 exits stay un-armed. New e2e pin (503 withhold arms → next small turn gets local 429 stage:forward, zero egress) verified failing without the fix line; issue4743 preflight pin now asserts the anti-self-block direction.

  Behavior change vs this PR's first revision (not vs master): preflight summary 429 no longer arms the hold mid-request; it arms at the withhold verdict instead. Net effect on master behavior: unchanged for the fits-case (still forwards); the withhold-case now arms where master did not (that is the feature).

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

这是 agent PR（ework-agent-pr 标记），不重做——按 review 流程处理：拉分支核对代码、对照 governor 契约验证接线、本地跑定向测试、跟踪 CI。

查重：不重复。 与 #4725（主循环，现为空壳 PR）、#4752（degenerate-retry，在途）同属 #4743 覆盖家族；本 PR 覆盖的三个文件（responses-wire compress loop / preflight refold / reasoning-guard rounds）与它们互不相交，无其他重叠条目。

标签： bug / scope/wiring / p2（wiring 遗漏；#4204 已覆盖主转发路径，此处是 429 风暴下 loop 车道放大问题，非整体不可用）

本地核验（PR head e7e38d4，base = 当前 master，落后 0 commit）：

• 新 wiring pin 3/3 过；把三个 src 文件还原到 master 后 0/3 挂 → 确认真实 wiring pin，不是恒真断言
• 定向回归（reasoning-guard / preflight-transport-recovery / preflight-transient-empty-retry / responses-guard / Provider 429 (concurrency-style limit): turn re-driven every few seconds for minutes, keeps the account pinned at the limit #4204 governor 契约 pin）：35/35 绿
• tsc --noEmit 干净
• CI：23 个 check run 全部 queued，出结果后跟进

review 中发现一个小偏差，稍后补独立标注 commit：compress-loop 站点在 fetchWithRetry 抛出非 429 的 UpstreamHttpError（如 fail-fast 400）时没有上报 resolved——而 governor 契约（#4204 模块头注释「ANY resolved non-429 … clears the hold」）以及 relay 和 #4752 的写法都是任何已决定的非 429 都清 hold。一行对齐 + 一行测试钉住。

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

#4772 审核结论(agent 自报 PR,审核模式:不重做,核对代码 + 盯 CI)。

做了什么

1. 逐行核对 diff(3 个 site + 测试文件),对照 src/upstream-throttle.ts governor 契约与 relay 规范模式(relay.ts:816 gate / :1584-1585 observe):各 site hold key 一致性(host+credentialFingerprint)、observe 落点(响应体读完后 / catch 边界)、重试语义不变——均正确。
2. 发现一处契约缺口,直接修掉(50b2bf859,独立 commit):responses-wire 循环的 catch 分支只对 429 arm,对其它终态非 429(如 fail-fast 400)静默——与 Provider 429 (concurrency-style limit): turn re-driven every few seconds for minutes, keeps the account pinned at the limit #4204 契约「任何 settled 非 429 都清 hold」及 relay/[PR] fix: review-findings wave — 7 fixes from the #4725 pre-release audit (P1×2 + P2×3 + P3×2) #4752 同款 site 不一致。修法:抛出的非 429 状态补报 resolved;传输层死亡保持不观测([bug] 上游在会话中途不可用时，bili 把一次故障放大成四重代价：固定节奏 20 连击全部冷发、输出预算被静默钳制 83 次（最狠 65536→1360）、紧急收缩与 nudge 停摆均不可见 #3148 断路器域)。测试加 re-arm→fail-fast-400 阶段,已验证去掉修复行即红。
3. 预防性 merge ci-fixes(66c46df32):master head 带着 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 已记录的 8 个红 shard(storage-lane/ledger/CRLF 簇),先把 origin/ci-fixes 合进 PR 分支(无冲突),不让继承红白烧一轮 CI;已在 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 留一行。PR diff 因此暂时多带 8 个 ci-fixes 提交(它们落 master 后自动消失)。
4. 本地定向验证:新 pin 无补丁 0/3 → 有补丁 3/3;reasoning-guard / preflight-transport-recovery / preflight-transient-empty-retry / responses-guard / governor 套件 35/35 绿;tsc 干净(merge 后复跑仍干净)。

八问

1. 是什么问题:上游 429 governor(Provider 429 (concurrency-style limit): turn re-driven every few seconds for minutes, keeps the account pinned at the limit #4204)只接在 relay 转发路径上,loop 车道三个出站点(Responses-wire 压缩循环、preflight refold、reasoning-guard 续跑轮次)绕过了它——上游对压缩请求持续 429 时,最易风暴的车道 governor 根本不武装,每个 client turn 都在重新轰炸。
2. 什么原因导致的:接线遗漏——Provider 429 (concurrency-style limit): turn re-driven every few seconds for minutes, keeps the account pinned at the limit #4204 引入 governor 时只给 relay 路径加了 observe,loop 车道 egress 站点从未接入([wiring] 429 governor is invisible to the loop lane (compress round-2+ resends do not go through gate/observe) #4743 的覆盖图列出的就是这些)。不是逻辑错误。
3. 影响范围与严重程度:仅影响 loop 车道请求遭遇持续上游 429 的会话;未武装期间每个 client turn 放大对已限流端点的压力。无数据损坏、不影响 bili 自身可用性;p2。
4. 怎么修复的:observe-only 接线,对齐 relay——每个 settled 响应(或边界错误)上报 governor observe 对:429 arm egress hold,settled 非 429 清 hold。不在 mid-loop 重请求上加本地闸门(会改变负载下的行为);hold 让下一 turn 的 relay forward 就地应答,不再重新风暴。审核另补齐边界 site「非 429 也清」的半边契约。
5. 是否有回归问题:无。逐项过:fetchWithRetry/fetchWithTimeout 语义不变(observe 是对 url/headers/status 的纯旁路读取);重试次数/延迟/backoff 不变;reasoning-guard/preflight 错误路径与 body 处理不变;upstreamThrottleLocalReject 仍只在 relay 路径调用,无新增本地拒绝面;跨 site hold key 一致性已验证。
6. 是否破坏缓存:不涉及——governor 状态是进程级 throttle 台账,与前缀缓存/压缩缓存正交。
7. 是否改了设计方向:不涉及——按 [wiring] 429 governor is invisible to the loop lane (compress round-2+ resends do not go through gate/observe) #4743 家族计划把既有 Provider 429 (concurrency-style limit): turn re-driven every few seconds for minutes, keeps the account pinned at the limit #4204 机制扩展到未覆盖站点;observe-only 的选择与 [PR] chore: empty PR (no changes) #4725 主循环先例一致。
8. 是否引入新配置:无——复用现有 BILI_THROTTLE_429* 旋钮与默认值。

状态与判定

• CI:66c46df32 的 24 个 run 仍在排队(runner 拥塞,已等约半小时);继承性 master 红已通过 ci-fixes merge 前置处理。
• 判定:代码审核通过,CI 绿即可合。CI 出结果后若红,按 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 流程跟进(继承红→merge ci-fixes;自红→自修),有结论再回贴一行。

[github-actions]

📦 Built Package Artifact

Branch: 2026-10-26_loop-lane-429-observe (a41f9fa)

Latest build — install this exact version

npm install -g [email protected]

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit	version	build
a41f9fa	[email protected]	build
c1e0b72	[email protected]	build
66c46df	[email protected]	build

Download artifact

1. Download the artifact from the Actions run
2. Extract the tarball and install:

tar xzf billion-context-pr4772.tgz
npm install -g package

---

Install instructions are refreshed on each push.

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review update — found and fixed a self-block regression in this PR's preflight site. My earlier "mergeable once CI green" verdict is superseded: CI went red exactly where predicted, and it was this PR's code, not inherited master red. Fix is pushed as c1e0b726b (labeled review-fix commit, PR description updated).

Mechanism. The per-response 429 arm added in the preflight walk (src/preflight.ts) fired before the same request's own forward: when the payload fits the window, preflight returns outbound, the relay forward runs the #4204 gate (src/server/relay.ts:816) — and sees the hold armed by its own summary probe's 429, answering locally instead of forwarding. That breaks the established #301 contract ("payload fits + preflight 429 → still forwarded, no false positive"). CI: tests/preflight-fail-fast.test.ts e2e red on windows-22 shard 1; reproduced locally on Linux (429 !== 200). (The other red shard — issue3846-local-refusal-evidence cancelledByParent — passes 6/6 locally with the patch; flake/inherited, not this PR.)

Fix (c1e0b726b). Split the observation at the preflight site:

• the walk now reports resolved per response (settled non-429 clears the hold — endpoint demonstrably processing again);
• the arm moves to the caller's withhold verdict in src/server.ts (failure.kind === "upstream" && status === 429 → noteUpstreamThrottle429(upstreamUrl, headers, null)). Nothing is forwarded on a withheld request, so the armed hold only answers the next turn's relay forward locally — precisely the [wiring] 429 governor is invisible to the loop lane (compress round-2+ resends do not go through gate/observe) #4743 ask. Doomed-413 exits stay un-armed (they carry no 429 evidence).
• No new local gate introduced — the PR's observe-only design is preserved (a gate on summary egress would be a design decision for you, not a review fix).

Pins. New e2e in tests/preflight-fail-fast.test.ts: over-window turn → 503 withhold arms the hold → next small turn gets local 429 stage:"forward" with zero egress upstream; verified failing without the fix line. The issue4743 preflight pin now asserts the anti-self-block direction (429 probe alone does NOT arm) plus cleared-on-200. Local: both touched files 12/12, targeted suites 34/34, tsc --noEmit clean.

Structured 八问:

1. 是什么问题:PR 在 preflight 折叠摘要的每个响应上武装 429 hold——但「payload 装得下」的请求会在 preflight 之后继续转发,被自己刚武装的 hold 在 relay 闸门处本地拦截,违反了 fix: honor anthropic-beta context-1m to size the context window #301 既有契约(装得下+preflight 429 → 照常转发)。
2. 什么原因导致的:观测点放错了层——arm 放在了 walk 内部的逐响应位置,而该请求的真实出站发生在 walk 之后、闸门之前,同一请求内自锁。
3. 影响范围与严重程度:上游摘要端点限流(429)且 payload 恰好装得下的所有会话——本该转发的请求被本地 429 拒掉。限流风暴场景下的可用性回归(CI 已红 + 本地复现)。
4. 怎么修复的:观测拆分——walk 只报 resolved(非 429 清除 hold),arm 移到调用方的 withhold 判决处(server.ts,kind=upstream 且 status=429 才武装,retry-after 不可得传 null)。withhold 请求不产生任何出站,hold 只会拦截下一轮转发——正是 [wiring] 429 governor is invisible to the loop lane (compress round-2+ resends do not go through gate/observe) #4743 要的语义。doomed-413 出口不武装(无 429 证据)。
5. 是否有回归问题:逐条过既有行为——fix: honor anthropic-beta context-1m to size the context window #301 fits 契约恢复(e2e 绿)、fix: honor anthropic-beta context-1m to size the context window #301 overflow-429 fail-fast 契约不变(503 路径未动)、Provider 429 (concurrency-style limit): turn re-driven every few seconds for minutes, keeps the account pinned at the limit #4204 治理者契约(任何 settled 非 429 清除 hold)三个站点全部保持、responses-loop 与 reasoning-guard 两站点本 fix 未触碰且经核查无自锁路径(两者都在 relay 闸门通过之后执行、且是当轮最后一次出站)。定向套件 34/34 + tsc 干净。
6. 是否破坏缓存:不涉及——只改了 hold 状态在何处被设置,缓存/重试/游标/去重逻辑零改动。
7. 是否改了设计方向:不改——observe-only 设计保持(没有新增本地闸门)。一个已知残留限制如实披露:持续溢出型风暴(每轮都超窗)仍会每轮花一次摘要调用才走 fail-fast,因为 withhold 判决在 relay 闸门之前;这是本 PR 选定设计的固有属性,不是本次 review-fix 的范围,留作后续候选。
8. 是否引入新配置:无。

行为变更披露(vs 本 PR 第一版,非 vs master):preflight 摘要 429 不再在请求中途武装 hold,改在 withhold 判决时武装。相对 master 的净效果:fits 场景行为不变(照常转发);withhold 场景从「不武装」变为「武装下一轮闸门」——后者即本 PR 的功能本身。

判定:代码审核完成(两轮 review fixes 均已落地:50b2bf859 边界上报补 resolved、c1e0b726b self-block 修复),CI 在 c1e0b726b 上绿即可合。