Skip to content

feat(streaming): add opt-in per-key stream cost breaker - #66

Merged
ALikesToCode merged 4 commits into
feat/realtimefrom
feat/stream-cost-breaker
Oct 9, 2026
Merged

ALikesToCode merged 4 commits into
feat/realtimefrom
feat/stream-cost-breaker

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds an opt-in per-key stream cost breaker. When a key has a cap, the gateway tracks a stream's running cost and stops it with the protocol's real error before it emits a chunk that would cross the cap. With STREAM_COST_BREAKER_ENABLED unset or off (the default), or a key's cap left null, streams, account queries, writes, key serialization and storage are byte-for-byte unchanged. Every new hook checks the flag first and falls through to the existing code.

  • Settings:
    • STREAM_COST_BREAKER_ENABLED: empty means off; a malformed value is off with one warning that leaves the value out.
    • A key's max_stream_cost_usd is set through the existing administrator key controls and stored as integer micro-USD. It must be non-negative, finite, at most 1,000,000 USD and exact to the micro-USD. Zero is valid.
    • Null (the default for every existing key) disables the breaker for that key. An explicit null clears a cap; omitting the field in an update or key rotation keeps it.
  • Pricing:
    • Running cost uses the existing provider-qualified MODEL_PRICING_USD_PER_MILLION input, cache-read, cache-write and output buckets.
    • A requested model, or any eligible auto-route candidate, without a known price returns 503 stream_cost_unpriced before handoff. An input estimate already over the cap returns 429 stream_cost_cap_exceeded.
    • Complete provider usage is labelled measured. Otherwise a conservative_estimate counts each emitted UTF-8 byte as an output token, which deliberately overestimates.
  • Stopping a stream:
    • The upstream is aborted and closed once, and the protocol's real error is emitted: Chat SSE data: {"error": {"code": "stream_cost_cap_exceeded", …}}, Anthropic event: error, or Responses response.failed.
    • The stream never ends with a finish_reason: stop, a success body or a successful [DONE].
  • Accounting: a stopped stream is recorded with status 499. If final provider usage is missing, the conservative unresolved usage hold stays in place.
  • Parsing: SSE frames split across reads and UTF-8 boundaries are handled. Errors and logs contain no secrets or content.
  • Not an invoice ceiling: buffered provider work, hidden reasoning, multimedia and provider-added structure can push the real cost past the cap. The cap is an observation threshold.

Details are in docs/stream-cost-breaker.md.

Turning it on

Apply migration 0030 before setting the flag. With the flag on and the column missing, D1 account reads and writes return 503 stream_cost_storage_unavailable rather than dropping controls or falling back. In practice, that means key authentication fails until the migration is applied.

Wiring

  • services/auth_service.py gets small call sites only. Account reads and writes, local SQLite schema setup, cached user rows and administrator updates pass through stream_cost_breaker helpers, which return the existing results while the flag is off.
  • services/key_controls.py accepts max_stream_cost_usd only while the flag is on, and validates it, stores it and shows it in the public key record.
  • services/request_accounting.py prepares the breaker for capped streaming requests, wraps event streams and records stopped streams with their hold.
  • services/upstream_transport.py binds the upstream cancellation so the breaker can close it.
  • worker/control-users-d1.mjs reads and writes the cap column only while the flag is on.
  • worker/stream-cost-breaker.mjs provides the same breaker for native Worker streams. It is connected to the native generation path with the Wave 6 integration.

Shared registrations

  • tests/test_stream_cost_breaker_worker.mjs in test:worker.
  • worker/stream-cost-breaker.mjs in the edge module inventory.
  • STREAM_COST_BREAKER_ENABLED in the Container env keys.
  • Migration 0030_stream_cost_caps.sql adds the nullable control_users.max_stream_cost_microusd column (ALTER TABLE … ADD COLUMN). No new tables.

Stacked on #62

Migration numbers must be contiguous in each PR, so this branch merges feat/realtime (#62, migration 0029). Merge #62 first. The learned cooldown and usage receipt PRs stack on this one.

Tests

  • tests/test_stream_cost_breaker.py and tests/test_stream_cost_breaker_worker.mjs cover:
    • flag off or null cap leaving streams, account rows, key records and storage unchanged;
    • cap validation, storage round trips, explicit clearing, and preservation on update and rotation;
    • the 503 for unpriced models and the 429 for an input estimate over the cap;
    • measured and conservative running cost across cache buckets;
    • the Chat, Anthropic and Responses errors with no success terminator;
    • single upstream cancellation and the unresolved hold when final usage is missing;
    • split SSE frames and UTF-8 boundaries;
    • the missing-column 503;
    • no secrets or content in errors.
  • The key control, auth, user store, accounting, reservation, cancellation and managed suites (185 tests) pass with the new call sites.
  • Full Python suite: 4,482 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,667 passed. npm run test:integration: 7 passed.
  • cf:migrations:check reports 30 contiguous, additive migrations. test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

Test fixture fixes

Two existing tests build a partly migrated D1 database by skipping 0003_control_users.sql together with every migration that alters that table. Migration 0030 also alters control_users, so both now skip it as well: tests/test_d1_schema_worker.mjs (the /ready schema test) and tests/test_d1_integration.mjs (the unmigrated-accounts integration test).

Apply the migration before enabling the flag: npx wrangler d1 migrations apply multillm-intelligence --remote.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 6. The README line comes with the Wave 6 integration PR.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 46bd317a-98e3-4ad8-868a-d2b84319d8f9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy 470a092 Oct 09 2026, 05:53 AM

@ALikesToCode
ALikesToCode merged commit ae2a0ab into feat/realtime Oct 9, 2026
4 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant