Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions docs/stream-cost-breaker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Stream cost breaker

The stream cost breaker is off by default. Set `STREAM_COST_BREAKER_ENABLED=true`
and set a key's `max_stream_cost_usd` through its existing administrator controls
endpoint to enforce a cap. Empty or malformed flags disable enforcement; malformed
flags produce one warning without their value.

Disabled examples:

```json
{"max_stream_cost_usd": null}
```

With the flag off, existing streams, account queries, writes and public key records
keep their original shape. With a null cap, the body and public key record are
unchanged. Existing keys remain disabled after migration. Clearing a cap uses an
explicit null; omitting it during an account update or key rotation preserves it.

Enabled example (five cents):

```json
{"max_stream_cost_usd": 0.05}
```

Caps use integer micro-USD in storage. Values must be nonnegative, finite, no more
than 1,000,000 USD, and exactly representable in whole micro-USD. Zero is a valid
cap. Apply `intelligence-migrations/0030_stream_cost_caps.sql` to D1 before enabling
the flag. It adds a nullable `control_users.max_stream_cost_microusd` column.
The local SQL account store adds its corresponding nullable column when enabled.
Enabled D1 account reads and writes return HTTP 503
`stream_cost_storage_unavailable` if the column is missing, without dropping
controls or falling back to another store.

## Prices and running cost

Configure the existing `MODEL_PRICING_USD_PER_MILLION` catalog with provider-qualified
models and input, cache-read, cache-write and output prices. Existing
`PROMPT_CACHE_PRICE_METADATA_JSON` can supplement missing cache prices.
For example:

```json
{"openai:example-model": {"input": 2, "cache_read": 0.5, "cache_write": 3, "output": 8}}
```

All eligible automatic-route candidates must have a known bound. Missing pricing
returns HTTP 503 `stream_cost_unpriced` before provider handoff. An input estimate
already above the cap returns HTTP 429 `stream_cost_cap_exceeded`. These checks
never grant retry permission.

Provider usage uses the existing inclusive OpenAI and exclusive Anthropic cache
bucket conventions. Complete usable observations are labeled `measured`.
Otherwise, `conservative_estimate` uses the highest configured input/cache rate
and a byte-count bound for input structures and emitted deltas. Tool arguments,
reasoning and text all contribute. Each UTF-8 byte is treated as an output token
when a tokenizer or provider count is unavailable. This deliberately overestimates
ordinary text. Input framing, hidden reasoning, multimedia and provider-added
structures can exceed this estimate; the cap is an observation threshold, not an
invoice ceiling. Pricing is captured for each breaker before streaming.

## Stream errors and cleanup

The breaker buffers complete SSE frames across socket reads and UTF-8 boundaries.
It checks each frame before emitting it and withholds terminal success frames until
the trailer is checked. Frames and the terminal trailer are each limited to
64 KiB, plus the current upstream read. A malformed, incomplete or oversized
frame fails closed with the protocol's error envelope.

A crossing stops upstream consumption and closes its owner once:

- Chat emits `data: {"error":{"code":"stream_cost_cap_exceeded", ...}}`.
- Anthropic emits `event: error` with an error object.
- Responses emits `event: response.failed` with a failed response object.

A rejected stream never emits a fabricated stop reason or successful `[DONE]`.
After headers are sent, the HTTP status can remain 200; clients must inspect the
SSE error event. Ledger finalization records an aborted or unresolved stream as
499. Error fields and logs contain no prompt, output, key, cap value or provider
error text. Raw streams bypass the wrapper when enforcement is disabled.

Upstream read buffering is at most 64 KiB in the shared HTTP transport, but a
provider can generate or queue more work before cancellation reaches it. Native
Worker buffering also includes the current provider read. No bound on the
provider's undisclosed queued work or invoice overshoot is promised.

## Usage holds and retention

Without final complete provider usage, do not settle a reservation as measured.
Existing durable usage reservations retain the conservative hold in the
`unknown` state for reconciliation; they are not released by stream cleanup.
Enable `USAGE_RESERVATIONS_ENABLED` for durable monetary holds. Legacy process
reservations remain held while the process retains them, but cannot survive a
restart and retain their existing review/expiry behavior. A key with no daily or
monthly budget has no pre-existing monetary reservation.

The breaker retains only bounded parsing buffers during a request and content-free
cost/usage state afterward. It does not persist prompt or response content.
Existing retention and ledger policies continue to apply.

## Native Worker interface

Before dispatch, call `prepareStreamCostBreaker(env, user, models, inputTokens,
protocol)` with all eligible provider-qualified models and a conservative input
token bound. It returns null for disabled keys and throws errors with
`status` and `code` on failed admission.

After upstream dispatch, wrap the SSE body with `wrapStreamCost(body, state,
{abort, onFinal})`. Pass the upstream abort callback and an accounting finalizer.
The finalizer receives `finalUsage`, `runningMicrousd`, `basis` and `exceeded`;
retain the existing hold when `finalUsage` is null. Supply the environment to
`activeUsersByPrefix(db, prefix, env)` when authenticating native streams so
the cap column is read and schema errors fail closed.
2 changes: 2 additions & 0 deletions intelligence-migrations/0030_stream_cost_caps.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
-- NULL disables stream cost enforcement for existing keys.
ALTER TABLE control_users ADD COLUMN max_stream_cost_microusd INTEGER CHECK (max_stream_cost_microusd BETWEEN 0 AND 1000000000000);
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs tests/test_realtime_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs tests/test_stream_cost_breaker_worker.mjs tests/test_realtime_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
Loading
Loading