Skip to content

[PR] fix(server): closeAllConnections() cuts through the post-response linger (#4380) - #4384

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-23_closeall-linger-cut
Oct 23, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-23_closeall-linger-cut

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Fixes #4380

问题

installPostResponseLinger(#1982/#2895)对已完成响应的 socket 包装了 destroy()、推迟真正的销毁以等 peer FIN。这静默挫败了 server.closeAllConnections()——该 API 的契约是「立刻切断所有连接」,但 armed 的 socket 实际上要等满 linger 预算(默认 5s,实测约 3s 直到 undici 连接池过期发 FIN)才死。

影响与实测

每个嵌入式 startServer/close 周期(测试 teardown、优雅停机)都按次支付这笔等待。两个直接受害者:

  • tests/lane-decisions-matrix.test.ts:194s → 2.6s(56 行矩阵 × 3s = 168s,占全文件 87%)
  • tests/plugin-fork.test.ts:150s(测量超时) → 8.9s

这也是 windows CI 测试车道 13–37 分钟长尾的主要成分(#4360)。

修复

包装 server.closeAllConnections:先跑 #2895 的 force-close seam(forceCloseLingeringSockets():给整个 ledger 盖 restartForced 戳 + 直接 destroy),再委托原始实现。seam 同时覆盖 node 自身迭代触达不到的 ledger socket(armed 包装、CONNECT 隧道、claimed upgrade);orig 调用作为兜底。普通 close() 的 linger 保护不变(负向测试钉死)。

测试

  • 新文件 tests/server-close-all-connections.test.ts(双向):
    • 正向:closeAllConnections() 后 close 事件 <2500ms;
    • 负向:纯 close() 后 1200ms 内无 close 事件(linger 保护仍在),再显式 cut 收尾。
  • 已登记 scripts/live-files.mjs(尾部追加,不 renumber 端口带)。
  • 双向验证过:修复前正向断言精确失败(3.0s),修复后 1ms。

不在范围

  • dsh-native / dsh-settings-ui 本地慢与失败为环境问题(需真实 dsh 二进制),CI ubuntu 全绿,与本修复无关。

…ger (#4380)

The linger wrap (#1982/#2895) defers socket.destroy() for
completed-response sockets, which silently defeated
server.closeAllConnections(): the API promises 'cut every tracked
connection NOW', but armed sockets sat out the linger budget (default
5s, in practice ~3s until the peer's pool expires) before dying. Every
embedded startServer/close cycle — test teardown above all — paid that
per stop: the lane decision matrix alone burned 168s of its 194s wall
time on 56 x 3s waits.

Wrap closeAllConnections to run the #2895 force-close seam (stamp
restartForced on the whole ledger + destroy) before delegating to the
original; the seam also covers ledgered sockets node's own iteration
reaches late or never (armed wraps, CONNECT tunnels, claimed upgrades).
Plain close() keeps the linger protection unchanged (negative test).

Measured: tests/lane-decisions-matrix.test.ts 194s -> 2.6s,
tests/plugin-fork.test.ts 150s(timeout) -> 8.9s.

Fixes #4380
… stays pinned)

Narrow the #4380 cut to the linger-protected class (completed at least
one response, never errored): held CONNECT tunnels and claimed upgrades
stay unreachable to plain closeAllConnections() — exactly the #2895
semantics the auto-restart repro pins. The restart force-close seam
still owns those, unchanged.

tests/auto-restart.test.ts 16/16 green again.
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zai/glm-5.3)

f5aacf0 收窄了切断范围:只切「完成过至少一次响应且无错」的 linger 保护类;held CONNECT 隧道/claimed upgrade 依旧不受普通 closeAllConnections() 影响(#2895 repro 语义保持,auto-restart 16/16 绿;restart force-close seam 仍专属负责它们)。窄化后 lane matrix 2.8s / 新回归测试双绿保持。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zai/glm-5.3)

状态说明:本 PR 的 28 个 workflow 在排队中被整体取消(当前 runner 队列被多个并行分支的 workflow 占满)。内容与测试均就绪——需要时在 Actions 里 re-run 或推一个空 commit 即可触发。与本修复同链的 #4358(句柄泄漏)也待合并;两者都进 master 后我会 rebase #4361(分片权重)并用修复后的实测耗时出终版。

@ranxianglei ranxianglei added bug Something isn't working p2 scope/wiring labels Oct 23, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复——open PR 里只有 #3563 也出现 closeAllConnections 字样,但那是测试 harness 有界关闭 + CI drain-cancel 重试形状(#3356),机制与失败域都不同;本 PR 是 #4380 的唯一修复。

标签: bug / scope/wiring / p2(缺陷类;服务请求管道生命周期面;确定性成本问题,与 #4380 定级一致)。

已完成的核对与推进(接力自 zai/glm-5.3 的会话):

  1. 代码复核:diff 仅 3 个文件(+121 行)——src/server.ts 包装 closeAllConnections(只切 lastResponseEndAt !== null && errored === null 的 linger 保护类,盖 restartForced 戳后经 ledger destroy,再委托原始实现兜底)、scripts/live-files.mjs 尾部追加新测试文件(不 renumber 端口带)、新增双向回归测试。包装逻辑与既有 forceCloseLingeringSockets seam 的 ledger 语义一致,未触碰 seam 本身。
  2. 本地定向验证(分支 f5aacf0,已对齐当前 master 07888d7):
  3. CI 已重新触发:head f5aacf0 上 14 个 workflow 此前在排队中被整体取消,我已通过 API 全部 re-run(均 201 受理),现在排队等 runner——最终全局判定以远程 CI 为准。
  4. 链状态:同链 [PR] fix(storage): close the sqlite handle when quick_check throws on a corrupt header (#4357) #4358(sqlite 句柄泄漏)仍 open/behind,不阻塞本 PR 自身 CI;两者进 master 后 [PR] ci(#4360): content-hash caching for pretest dist + bolder Windows sharding — test lane down to ~6.5 min | ci(#4360): pretest dist 内容哈希缓存 + windows 分片加粗 — 测试车道压到 ~6.5 分钟 #4361(分片权重)再 rebase 出终版,维持原计划。

结构化八问

  1. 是什么问题:installPostResponseLinger(建议:客户端侧连接请优雅关闭(close_notify+FIN),避免向下游客户端发出 RST #1982/[Bug] autoRestartOnUpdate 在 HTTPS_PROXY/MITM 接入下几乎必然失败:closeAllConnections() 关不掉 CONNECT 隧道 → "connections stuck after forced close" #2895)延迟已完成响应 socket 的销毁以等 peer FIN,使 server.closeAllConnections() 的「立刻切断」契约被静默挫败——armed socket 要等满 ~3s(linger 预算 5s)才死,每个嵌入式 startServer/close 周期(测试 teardown、优雅停机)都按次支付。
  2. 什么原因导致的:linger 包装改写的是 socket.destroy() 的时机,而 node 自身的 closeAllConnections 迭代触达不到 ledger 里 off-parser 类的 idle 池化 socket;linger 保护与切断 API 之间没有任何协调点,保护方单方面赢了。
  3. 影响范围与严重程度:所有嵌入式 startServer/close 周期;确定性时间成本(CI lane matrix 194s、plugin-fork 150s、windows CI 长尾主成分),无数据丢失、无不可用 → p2。
  4. 怎么修复的:在 startServer 内包装 closeAllConnections,先按 ledger 精确切掉 linger 保护类(完成过至少一次响应且无错)并盖 intentional-cut 戳,再委托原始实现兜底 node 追踪而 ledger 未记录的连接;held CONNECT 隧道/claimed upgrade 明确排除,仍归 restart force-close seam 专属负责。
  5. 是否有回归问题:审过的既有行为——普通 close() 的 linger 保护不变(负向测试钉死,本地 auto-restart 16/16 绿);[Bug] autoRestartOnUpdate 在 HTTPS_PROXY/MITM 接入下几乎必然失败:closeAllConnections() 关不掉 CONNECT 隧道 → "connections stuck after forced close" #2895 repro 语义保持(隧道/upgrade 依旧不能被普通 closeAllConnections 切);force-close seam 本体未动;live-files 端口带不漂移(尾部追加)。有意行为变更(已披露):closeAllConnections() 对 linger 保护类 idle 池化 socket 从「等 ~3–5s 才断」变为「立即断」——这正是修复目标,且符合该 API 的文档契约。
  6. 是否破坏缓存:不涉及——未触碰缓存/重试/游标/去重逻辑。
  7. 是否改了设计方向:不涉及——startServer 内的窄生命周期修复,无架构转向。
  8. 是否引入了新的配置:不涉及。

收工判定:代码与双向测试本地全部核实通过,diff 干净(每行都与 #4380 目的相关),CI 已重新入队——待远程 CI 全绿即可合并(合并由人执行)。若队列再次拥堵导致取消,告诉我一声我再触发或推空 commit。

@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-23_closeall-linger-cut (f5aacf0)

Option A — Install this exact build (recommended)

npm install -g [email protected]

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4384.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei
ranxianglei merged commit 6104078 into master Oct 23, 2026
89 of 181 checks passed
ranxianglei pushed a commit that referenced this pull request Oct 23, 2026
Line-count LPT balanced bytes, not seconds: process-spawning suites
cost orders of magnitude more per line than pure assertion files, and
the 'perfectly balanced' layouts still produced 5min..37min legs from
equal byte counts (windows 22-2/22-3 vs 22-1/22-4 on the same run).
With #4384's linger cut landed the per-file wall times are stable
enough to weight the partition by MEASURED duration:

- scripts/ci-shard-partition.mjs: fileWeight = last measured ms per
  file (scripts/ci-shard-weights.json), new files fall back to
  lineCount x msPerLineFallback (median ms/line); same deterministic
  total LPT otherwise.
- scripts/ci-shard-weights.json: 840-file table measured post-#4384
  (single-file node --test runs, VM, ~66min serial total); local-env
  hangs capped at the 150s timeout and listed under timedOut, local
  rc=1 env-fails excluded. Regenerate via
  scripts/measure-test-durations.mjs when shard balance degrades.
- run-ci-tests.mjs delegates partitionAll to the module (this script
  has top-level side effects and cannot be imported by tests).
- ci.yml: windows-22 CORE 3->5 shards, windows FULL 3->5 (both node
  versions) — measured max shard drops 1984s->794s serial (concurrency
  c=4 per shard => ~7min worst-leg target incl. setup).

Closes #4360 (with #4377's dist cache + #4384's linger cut, the three
levers of the 8-minute goal)
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.764-master.6104078 (rolling master tag).

npm install -g [email protected]  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit that referenced this pull request Oct 23, 2026
Line-count LPT balanced bytes, not seconds: process-spawning suites
cost orders of magnitude more per line than pure assertion files, and
the 'perfectly balanced' layouts still produced 5min..37min legs from
equal byte counts (windows 22-2/22-3 vs 22-1/22-4 on the same run).
With #4384's linger cut landed the per-file wall times are stable
enough to weight the partition by MEASURED duration:

- scripts/ci-shard-partition.mjs: fileWeight = last measured ms per
  file (scripts/ci-shard-weights.json), new files fall back to
  lineCount x msPerLineFallback (median ms/line); same deterministic
  total LPT otherwise.
- scripts/ci-shard-weights.json: 840-file table measured post-#4384
  (single-file node --test runs, VM, ~66min serial total); local-env
  hangs capped at the 150s timeout and listed under timedOut, local
  rc=1 env-fails excluded. Regenerate via
  scripts/measure-test-durations.mjs when shard balance degrades.
- run-ci-tests.mjs delegates partitionAll to the module (this script
  has top-level side effects and cannot be imported by tests).
- ci.yml: windows-22 CORE 3->5 shards, windows FULL 3->5 (both node
  versions) — measured max shard drops 1984s->794s serial (concurrency
  c=4 per shard => ~7min worst-leg target incl. setup). Four NEW check
  names appear (windows 22/24 shards 4/5); branch protection must add
  them before they gate merges.

Cherry-picked from closed PR #4361 (commit 0569781, stranded when
the PR was superseded by #4377; picked up per owner invitation in the
#4361 thread). The sibling LIVE_FILES commit (d06d192) is already on
master via #4416 and is not included.

Closes #4437
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/wiring

Projects

None yet

1 participant