Skip to content

[PR] test: realign five stale suites to #3900 always-reserve-headroom window math - #4208

Closed
ranxianglei wants to merge 1 commit into
masterfrom
2026-10-22_realign-post3900-window-expectations
Closed

ranxianglei wants to merge 1 commit into
masterfrom
2026-10-22_realign-post3900-window-expectations

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #4194

What

4d50f9c2 (#4184, implementing #3900 "reserve output headroom on every wire") dropped the Anthropic exemption, so every wire's effective window now carries min(max_tokens, cap × window). It realigned anthropic-beta-window / context-overflow / preflight-max-output-tokens but left five other suites asserting the pre-#3900 numbers. This PR realigns those suites — no source changes.

Per-suite mechanism (verified by local repro on fe5ba59, then fixed)

Suite Failure Mechanism
tests/codex-window-align.test.ts 0/8 — every limit exactly −1024 (= fixture max_tokens) The clamped/declared window surfaces carry the per-request reservation. All assertions now <base> − FIXTURE_MAX_TOKENS (new named constant).
tests/side-request-isolation.test.ts 1 — anthropic row expected blocked=false under tolerance The shared body carried max_tokens, so anthropic now reserves and the boundary flips. Pure guard-math rows now run without an output-budget field; added an explicit anthropic reservation boundary pair mirroring the OpenAI case (both directions, #3617 gate discipline).
tests/external-summary-autofold.test.ts 3 — nudge never fired Proxy-direct requests miss the route table → window resolves as fallback → #3900 reservation runs → the pre-existing #282 fallback floor (handle.ts: nativeFromFallback && reserved < 100_000) re-inflates the post-reservation window (48976 → 100000) → usage judged at 42% instead of 85.8% → pressure band cold. Fix: give the fixture model an authoritative registry window (setRegistryForTest) so the window is non-fallback and no floor applies.
tests/fork-nudge-deadzone.test.ts 1 — lane proof Not window math: the new [headroom] … nudge/preflight judge … log line contains the word "nudge" and logs before the diagNudge decision line, so the selector picked it up first. Selector now pins the [<id>] nudge prefix. (Probe dump confirmed the decision line itself reads nudge idle: usage=32% — same as pre-#3900.)
tests/plugin-fork.test.ts 1 — 398976 !== 400000 Status contextLimit is session.metadata.effectiveContextLimit (src/plugin.ts:1611) — the window nudge/preflight judge against — which carries the reservation by design. Assertion now 400000 − 1024.

Behavior-change disclosure (from #3900 itself, surfaced here for the record)

On the Anthropic wire, small fallback-sourced windows (< 100K after reservation) are now effectively floored to 100K by the pre-existing #282 rule — e.g. a 50K guessed window judges usage against 100K instead of 48976. This is a consequence of #3900 × #282, ships with master already, and is out of scope for this test-only PR — flagged per the behavior-drift rule. If the owner wants that interaction changed, it belongs in a follow-up against #3900.

Verification

  • Local targeted runs on this branch: 101/101 across all five suites (codex-window-align 8, side-request-isolation 33, external-summary-autofold 8, fork-nudge-deadzone 3, plugin-fork 49).
  • npm run typecheck clean.
  • Full-suite verdict = remote CI test lanes on this head.

#4194)

4d50f9c (#4184/#3900) dropped the Anthropic output-headroom exemption, so
every wire's effective window now carries min(max_tokens, cap x window):

- codex-window-align: every asserted effective limit is now base - max_tokens
  (272000/400000/200000/828400/95000/800000/100000); named fixture constant.
- side-request-isolation: pure guard-math rows run without an output-budget
  field; added the explicit anthropic reservation boundary pair (both
  directions, #3617 gate discipline).
- external-summary-autofold: proxy-direct fixtures miss the route table, so
  their windows resolved as fallback and #282's floor re-inflated the
  post-reservation window back to 100K, keeping the pressure band cold; give
  the fixture model an authoritative registry window instead.
- fork-nudge-deadzone: pin the log selector to the diagNudge "[<id>] nudge"
  prefix — the new [headroom] line also contains "nudge" and logs first.
- plugin-fork: status contextLimit is metadata.effectiveContextLimit, which
  carries the reservation by design -> 400000 - 1024.

No source changes. Local targeted runs: 101/101 across the five suites.
@ranxianglei ranxianglei changed the title test: realign five stale suites to #3900 always-reserve-headroom window math [PR] test: realign five stale suites to #3900 always-reserve-headroom window math Oct 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-22_realign-post3900-window-expectations (bc23a5e)

Option A — Install this exact build (recommended)

npm install -g [email protected]

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4208.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

Closing as superseded: while this PR was in flight, #4198 merged (2026-10-21T16:37Z) reverting #4184/4d50f9c2 entirely — master is back to the pre-#3900 reserve-on-every-wire semantics, and v0.1.204 shipped from that state. This PR realigned five suites to the POST-#3900 numbers, which is now the wrong direction: the CI failure on this head (codex-window-align actual=272000 vs expected 270976 etc.) is the merge of these expectations with the reverted code, not a bug in either side.

Per #4198's stated plan, #3900 stays open for a redo with different semantics (record the un-reserved value in effectiveContextLimit, apply the reservation at the band layer, update all affected tests + goldens in one pass). If that redo lands, the per-suite mechanisms worked out here are still reusable reference:

No source changes were involved; the branch 2026-10-22_realign-post3900-window-expectations remains available for reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

master test lanes red after #4184 (#3900 always-reserve-headroom): stale expectations across codex-window-align + side-request-isolation (+3 more)

1 participant