diff --git a/docs/stream-cost-breaker.md b/docs/stream-cost-breaker.md new file mode 100644 index 00000000..cfcffa23 --- /dev/null +++ b/docs/stream-cost-breaker.md @@ -0,0 +1,111 @@ +# Stream cost breaker + +The stream cost breaker is off by default. Set `STREAM_COST_BREAKER_ENABLED=true` +and set a key's `max_stream_cost_usd` through its existing administrator controls +endpoint to enforce a cap. Empty or malformed flags disable enforcement; malformed +flags produce one warning without their value. + +Disabled examples: + +```json +{"max_stream_cost_usd": null} +``` + +With the flag off, existing streams, account queries, writes and public key records +keep their original shape. With a null cap, the body and public key record are +unchanged. Existing keys remain disabled after migration. Clearing a cap uses an +explicit null; omitting it during an account update or key rotation preserves it. + +Enabled example (five cents): + +```json +{"max_stream_cost_usd": 0.05} +``` + +Caps use integer micro-USD in storage. Values must be nonnegative, finite, no more +than 1,000,000 USD, and exactly representable in whole micro-USD. Zero is a valid +cap. Apply `intelligence-migrations/0030_stream_cost_caps.sql` to D1 before enabling +the flag. It adds a nullable `control_users.max_stream_cost_microusd` column. +The local SQL account store adds its corresponding nullable column when enabled. +Enabled D1 account reads and writes return HTTP 503 +`stream_cost_storage_unavailable` if the column is missing, without dropping +controls or falling back to another store. + +## Prices and running cost + +Configure the existing `MODEL_PRICING_USD_PER_MILLION` catalog with provider-qualified +models and input, cache-read, cache-write and output prices. Existing +`PROMPT_CACHE_PRICE_METADATA_JSON` can supplement missing cache prices. +For example: + +```json +{"openai:example-model": {"input": 2, "cache_read": 0.5, "cache_write": 3, "output": 8}} +``` + +All eligible automatic-route candidates must have a known bound. Missing pricing +returns HTTP 503 `stream_cost_unpriced` before provider handoff. An input estimate +already above the cap returns HTTP 429 `stream_cost_cap_exceeded`. These checks +never grant retry permission. + +Provider usage uses the existing inclusive OpenAI and exclusive Anthropic cache +bucket conventions. Complete usable observations are labeled `measured`. +Otherwise, `conservative_estimate` uses the highest configured input/cache rate +and a byte-count bound for input structures and emitted deltas. Tool arguments, +reasoning and text all contribute. Each UTF-8 byte is treated as an output token +when a tokenizer or provider count is unavailable. This deliberately overestimates +ordinary text. Input framing, hidden reasoning, multimedia and provider-added +structures can exceed this estimate; the cap is an observation threshold, not an +invoice ceiling. Pricing is captured for each breaker before streaming. + +## Stream errors and cleanup + +The breaker buffers complete SSE frames across socket reads and UTF-8 boundaries. +It checks each frame before emitting it and withholds terminal success frames until +the trailer is checked. Frames and the terminal trailer are each limited to +64 KiB, plus the current upstream read. A malformed, incomplete or oversized +frame fails closed with the protocol's error envelope. + +A crossing stops upstream consumption and closes its owner once: + +- Chat emits `data: {"error":{"code":"stream_cost_cap_exceeded", ...}}`. +- Anthropic emits `event: error` with an error object. +- Responses emits `event: response.failed` with a failed response object. + +A rejected stream never emits a fabricated stop reason or successful `[DONE]`. +After headers are sent, the HTTP status can remain 200; clients must inspect the +SSE error event. Ledger finalization records an aborted or unresolved stream as +499. Error fields and logs contain no prompt, output, key, cap value or provider +error text. Raw streams bypass the wrapper when enforcement is disabled. + +Upstream read buffering is at most 64 KiB in the shared HTTP transport, but a +provider can generate or queue more work before cancellation reaches it. Native +Worker buffering also includes the current provider read. No bound on the +provider's undisclosed queued work or invoice overshoot is promised. + +## Usage holds and retention + +Without final complete provider usage, do not settle a reservation as measured. +Existing durable usage reservations retain the conservative hold in the +`unknown` state for reconciliation; they are not released by stream cleanup. +Enable `USAGE_RESERVATIONS_ENABLED` for durable monetary holds. Legacy process +reservations remain held while the process retains them, but cannot survive a +restart and retain their existing review/expiry behavior. A key with no daily or +monthly budget has no pre-existing monetary reservation. + +The breaker retains only bounded parsing buffers during a request and content-free +cost/usage state afterward. It does not persist prompt or response content. +Existing retention and ledger policies continue to apply. + +## Native Worker interface + +Before dispatch, call `prepareStreamCostBreaker(env, user, models, inputTokens, +protocol)` with all eligible provider-qualified models and a conservative input +token bound. It returns null for disabled keys and throws errors with +`status` and `code` on failed admission. + +After upstream dispatch, wrap the SSE body with `wrapStreamCost(body, state, +{abort, onFinal})`. Pass the upstream abort callback and an accounting finalizer. +The finalizer receives `finalUsage`, `runningMicrousd`, `basis` and `exceeded`; +retain the existing hold when `finalUsage` is null. Supply the environment to +`activeUsersByPrefix(db, prefix, env)` when authenticating native streams so +the cap column is read and schema errors fail closed. diff --git a/intelligence-migrations/0030_stream_cost_caps.sql b/intelligence-migrations/0030_stream_cost_caps.sql new file mode 100644 index 00000000..ff61b6e2 --- /dev/null +++ b/intelligence-migrations/0030_stream_cost_caps.sql @@ -0,0 +1,2 @@ +-- NULL disables stream cost enforcement for existing keys. +ALTER TABLE control_users ADD COLUMN max_stream_cost_microusd INTEGER CHECK (max_stream_cost_microusd BETWEEN 0 AND 1000000000000); diff --git a/package.json b/package.json index 1bf970cf..b10757f5 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,7 @@ "lint": "ruff check . --select E9,F63,F7,F82", "typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming", "test": "python -m pytest -q", - "test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs tests/test_realtime_worker.mjs", + "test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs tests/test_stream_cost_breaker_worker.mjs tests/test_realtime_worker.mjs", "db:migrate": "python scripts/validate_sqlite_schema.py --apply", "db:validate": "python scripts/validate_sqlite_schema.py", "security:secrets": "python scripts/check_static_secrets.py", diff --git a/services/auth_service.py b/services/auth_service.py index 07e6a06a..9c910254 100644 --- a/services/auth_service.py +++ b/services/auth_service.py @@ -47,7 +47,7 @@ serialize_scopes, usable_credential, ) -from services import key_controls +from services import key_controls, stream_cost_breaker from services.nanogpt_key_pool import configured_nanogpt_keys from services import user_store from services.sqlite_store import connect, storage_path @@ -122,6 +122,7 @@ def _ensure_users_schema(cls, connection: sqlite3.Connection) -> None: ).fetchone() if not table_exists: cls._create_users_table(connection) + stream_cost_breaker.ensure_sql_cap(connection) cls._ensure_users_indexes(connection) return @@ -162,6 +163,7 @@ def _ensure_users_schema(cls, connection: sqlite3.Connection) -> None: f"ALTER TABLE users ADD COLUMN {column_name} {column_definition}" ) + stream_cost_breaker.ensure_sql_cap(connection) cls._backfill_user_metadata(connection) cls._ensure_users_indexes(connection) @@ -290,6 +292,7 @@ def _row_to_user(cls, row: sqlite3.Row) -> Dict[str, Any]: "created_by": row["created_by"], "rotated_at": deserialize_datetime(row["rotated_at"]), "revoked_at": deserialize_datetime(row["revoked_at"]), + **stream_cost_breaker.storage_control(row), **key_controls.from_storage( {name: key_controls.row_value(row, name) for name in key_controls.CONTROL_FIELDS} ), @@ -298,12 +301,12 @@ def _row_to_user(cls, row: sqlite3.Row) -> Dict[str, Any]: @classmethod def _reload_user_cache(cls) -> None: if user_store.using_d1(): - rows = user_store.list_users() + rows = stream_cost_breaker.list_users() else: with cls._storage_lock: with closing(cls._connect()) as connection: rows = connection.execute( - """ + stream_cost_breaker.sql_select(""" SELECT username, api_key_hash, api_key_prefix, scopes, is_admin, created_at, last_login, last_used_at, last_used_ip, @@ -311,7 +314,7 @@ def _reload_user_cache(cls) -> None: monthly_budget_usd, allowed_models, allowed_ips, expires_at, secret_scan_mode, shadow_eval_rate FROM users ORDER BY username - """ + """) ).fetchall() cls._users = { row["username"]: cls._row_to_user(row) @@ -322,13 +325,13 @@ def _reload_user_cache(cls) -> None: @classmethod def _load_user_by_username(cls, username: str) -> Optional[Dict[str, Any]]: if user_store.using_d1(): - row = user_store.get_user(username) + row = stream_cost_breaker.get_user(username) else: with cls._storage_lock: with closing(cls._connect()) as connection: cls._ensure_users_schema(connection) row = connection.execute( - """ + stream_cost_breaker.sql_select(""" SELECT username, api_key_hash, api_key_prefix, scopes, is_admin, created_at, last_login, last_used_at, last_used_ip, @@ -336,7 +339,7 @@ def _load_user_by_username(cls, username: str) -> Optional[Dict[str, Any]]: monthly_budget_usd, allowed_models, allowed_ips, expires_at, secret_scan_mode, shadow_eval_rate FROM users WHERE username = ? - """, + """), (username,), ).fetchone() if not row: @@ -352,13 +355,13 @@ def _load_user_by_username(cls, username: str) -> Optional[Dict[str, Any]]: @classmethod def _load_users_by_api_key_prefix(cls, api_key_prefix: str) -> List[tuple[str, Dict[str, Any]]]: if user_store.using_d1(): - rows = user_store.users_by_prefix(api_key_prefix) + rows = stream_cost_breaker.users_by_prefix(api_key_prefix) else: with cls._storage_lock: with closing(cls._connect()) as connection: cls._ensure_users_schema(connection) rows = connection.execute( - """ + stream_cost_breaker.sql_select(""" SELECT username, api_key_hash, api_key_prefix, scopes, is_admin, created_at, last_login, last_used_at, last_used_ip, @@ -367,7 +370,7 @@ def _load_users_by_api_key_prefix(cls, api_key_prefix: str) -> List[tuple[str, D FROM users WHERE api_key_prefix = ? AND revoked_at IS NULL ORDER BY username - """, + """), (api_key_prefix,), ).fetchall() @@ -407,7 +410,7 @@ def _persist_user( cls._forget_verified_keys() stored_controls = {**key_controls.empty(), **(controls or {})} if user_store.using_d1(): - user_store.upsert_user({ + stream_cost_breaker.upsert_user({ "username": username, "api_key_hash": api_key_hash, "api_key_prefix": api_key_prefix, @@ -421,6 +424,8 @@ def _persist_user( "rotated_at": serialize_datetime(rotated_at), "revoked_at": serialize_datetime(revoked_at), **{name: stored_controls[name] for name in key_controls.CONTROL_FIELDS}, + **({stream_cost_breaker.CAP_FIELD: stored_controls[stream_cost_breaker.CAP_FIELD]} + if stream_cost_breaker.CAP_FIELD in stored_controls else {}), }) cls._reload_user_cache() return @@ -472,6 +477,7 @@ def _persist_user( *(stored_controls[name] for name in key_controls.CONTROL_FIELDS), ), ) + stream_cost_breaker.persist_sql_cap(connection, username, stored_controls) connection.commit() cls._reload_user_cache() @@ -913,7 +919,8 @@ def _public_user(cls, username: str, user: Dict[str, Any]) -> Dict[str, Any]: @staticmethod def _stored_controls(user: Dict[str, Any]) -> Dict[str, Any]: - return {name: user.get(name) for name in key_controls.CONTROL_FIELDS} + return {**{name: user.get(name) for name in key_controls.CONTROL_FIELDS}, + **stream_cost_breaker.storage_control(user)} @classmethod def _update_login(cls, username: str, last_login: datetime) -> None: @@ -1190,7 +1197,7 @@ def set_key_controls(cls, username: str, payload: Dict[str, Any]) -> Dict[str, A user = cls._load_user_by_username(username) if not user: raise APIError("User not found", status_code=404) - controls = key_controls.validate(payload) + controls = {**stream_cost_breaker.storage_control(user), **key_controls.validate(payload)} if ( username == normalized_username(os.environ.get("ADMIN_USERNAME", "admin")) and (controls["expires_at"] or controls["allowed_ips"]) diff --git a/services/key_controls.py b/services/key_controls.py index bd8d3276..633db632 100644 --- a/services/key_controls.py +++ b/services/key_controls.py @@ -18,7 +18,7 @@ from werkzeug.security import check_password_hash from error_handlers import APIError -from services import user_store +from services import user_store, stream_cost_breaker from services.auth_primitives import build_api_key_prefix, normalized_username CONTROL_FIELDS = ("daily_budget_usd", "monthly_budget_usd", "allowed_models", "allowed_ips", "expires_at", "secret_scan_mode", "shadow_eval_rate") @@ -118,7 +118,7 @@ def validate(payload: Mapping[str, Any]) -> dict[str, Any]: """Normalize an administrator's controls; unknown fields are refused.""" if not isinstance(payload, Mapping): raise APIError("Key controls must be a JSON object", 400) - unknown = set(payload) - set(CONTROL_FIELDS) + unknown = set(payload) - set(CONTROL_FIELDS) - ({"max_stream_cost_usd"} if stream_cost_breaker.enabled() else set()) if unknown: raise APIError(f"Unknown key control: {sorted(unknown)[0][:64]}", 400) mode = payload.get("secret_scan_mode") @@ -147,6 +147,7 @@ def validate(payload: Mapping[str, Any]) -> dict[str, Any]: "expires_at": expires_at.isoformat() if expires_at else None, "secret_scan_mode": mode, "shadow_eval_rate": rate, + **stream_cost_breaker.validated_control(payload), } @@ -163,6 +164,7 @@ def from_storage(row: Mapping[str, Any]) -> dict[str, Any]: rate = row.get("shadow_eval_rate") if hasattr(row, "get") else row_value(row, "shadow_eval_rate") if type(rate) in (int, float) and 0 <= rate <= 0.2: controls["shadow_eval_rate"] = float(rate) + controls.update(stream_cost_breaker.storage_control(row)) return controls @@ -184,6 +186,7 @@ def public(controls: Mapping[str, Any]) -> dict[str, Any]: "expires_at": controls.get("expires_at"), "secret_scan_mode": controls.get("secret_scan_mode"), "shadow_eval_rate": controls.get("shadow_eval_rate"), + **stream_cost_breaker.public_control(controls), } @@ -283,9 +286,9 @@ def client_ip(request) -> Optional[str]: def _security_user(auth, row): """Reject malformed authority records before permissive legacy decoding.""" - if not isinstance(row, dict) or set(row) != set(user_store.USER_FIELDS): + if not isinstance(row, dict) or set(row) != set(user_store.USER_FIELDS) | ({stream_cost_breaker.CAP_FIELD} if stream_cost_breaker.enabled() else set()): raise ValueError("Incomplete security account") - user_store._row(row) + stream_cost_breaker.d1_row(row) if normalized_username(row["username"]) != row["username"]: raise ValueError("Invalid security account identity") if not row["api_key_hash"] or not row["api_key_prefix"] or not re.fullmatch(r"[A-Za-z0-9:_.,-]{0,512}", row["scopes"]): diff --git a/services/request_accounting.py b/services/request_accounting.py index 2624570d..ee363315 100644 --- a/services/request_accounting.py +++ b/services/request_accounting.py @@ -22,7 +22,7 @@ from werkzeug.exceptions import HTTPException from error_handlers import APIError -from services import key_controls, prompt_cache_cost, reservation_store, telemetry_export, usage_ledger +from services import key_controls, prompt_cache_cost, reservation_store, telemetry_export, usage_ledger, stream_cost_breaker from services.budget_service import BudgetService, budgeted from services.cost_service import CostService from services.usage_types import StreamUsageObserver, UsageObservation @@ -93,6 +93,7 @@ class UsageContext: # Served from the response cache: no provider was called, so nothing is charged. cached: bool = False ambiguous: bool = False + stream_cost: Any = None def classify() -> Optional[str]: @@ -270,6 +271,16 @@ def begin() -> Optional[Response]: start_ns=time.time_ns(), units=_image_units(kind, payload), trace=telemetry_export.trace_context(request.headers.get("traceparent"))) context.input_tokens, context.output_tokens = _token_estimates(payload) + if payload and payload.get("stream") is True and stream_cost_breaker.enabled() and ( + user.get(stream_cost_breaker.CAP_FIELD) is not None or user.get("max_stream_cost_usd") is not None): + protocol = ("anthropic" if request.path.endswith("/messages") else + "responses" if request.path.endswith("/responses") else "chat") + qualified = [_qualified(candidate, provider) for model in models for candidate in _candidates(model)] + try: + context.stream_cost = stream_cost_breaker.prepare( + user, qualified, max(context.input_tokens, stream_cost_breaker.input_bound(payload)), protocol) + except APIError as error: + return _error(error.status_code, error.payload["error"], "Stream cost enforcement refused this request.") if budgeted(user): qualified = [_qualified(model, provider) for model in models] estimate = (reservation_price(qualified, context.input_tokens, context.output_tokens, context.units) @@ -385,7 +396,9 @@ def cancellation_observer(): def observe(outcome): if outcome.ambiguous and not context.finished: - context.ambiguous = True + state = context.stream_cost + if state is None or state.final_usage is None: + context.ambiguous = True return observe @@ -437,7 +450,8 @@ def _record(context: UsageContext, status: int, usage: Optional[UsageObservation logger.warning("Usage could not be recorded (%s)", type(error).__name__) finally: # The settled cost is counted before the in-flight estimate is released. - BudgetService.settle(context.reservation) + if context.stream_cost is None or context.stream_cost.final_usage is not None or reservation_store.enabled(): + BudgetService.settle(context.reservation) def _row(context: UsageContext, status: int, usage: Optional[UsageObservation], units: Optional[int]) -> dict: @@ -460,6 +474,10 @@ def _row(context: UsageContext, status: int, usage: Optional[UsageObservation], cost = price([selected] if selected else qualified, observation.input_tokens, observation.output_tokens, context.units) basis = observation.storage_basis(cost) + if context.stream_cost is not None and usage is not None: + breakdown = CostService.price_buckets(selected or requested, usage, requests=context.units) + cost = breakdown["cost_usd"] + basis = usage.storage_basis(cost) prefix = context.user.get("api_key_prefix") row = { "at": usage_ledger.utc_timestamp(), @@ -496,6 +514,10 @@ def _capture(context: UsageContext) -> None: context.request_id = request_id if isinstance(request_id, str) and REQUEST_ID.fullmatch(request_id) else None selected = getattr(g, "multillm_model", None) context.selected = selected if isinstance(selected, str) else None + if context.stream_cost is not None: + owner = getattr(g, "gateway_cancellation", None) + if owner is not None: + context.stream_cost.cancel = owner.cancel class _SniffedStream: @@ -542,6 +564,8 @@ def finish(result: Any) -> Any: _record(context, 500, None, None) return result context.cached = response.headers.get("X-MultiLLM-Cache") == "hit" + if context.cached: + context.stream_cost = None if response.is_streamed: event_stream = (response.mimetype or "") == "text/event-stream" json_stream = (response.mimetype or "").endswith("json") @@ -551,8 +575,14 @@ def closed(tail: bytes, observed: Optional[UsageObservation]) -> None: usage = observed if event_stream else _json_tail_usage(tail) if json_stream else None if event_stream and context.selected is None: context.selected = _sse_model(tail) - _record(context, status, usage, None) - + if context.stream_cost is not None: + usage = context.stream_cost.final_usage + if usage is None: + context.ambiguous = True + _record(context, 499 if context.stream_cost is not None and context.stream_cost.exceeded else status, usage, None) + + if event_stream and not context.cached: + response.response = stream_cost_breaker.wrap_stream(response.response, context.stream_cost) response.response = _SniffedStream(response.response, closed, event_stream=event_stream) return response body = _json_body(response) diff --git a/services/stream_cost_breaker.py b/services/stream_cost_breaker.py new file mode 100644 index 00000000..93029ecc --- /dev/null +++ b/services/stream_cost_breaker.py @@ -0,0 +1,372 @@ +"""Opt-in per-key stream caps, bucket bounds and protocol-aware cleanup.""" +from __future__ import annotations + +import json +import logging +import os +import re +from dataclasses import dataclass +from decimal import Decimal, InvalidOperation, ROUND_CEILING +from typing import Any + +from error_handlers import APIError +from services.cost_service import CostService +from services.prompt_cache_cost import CacheObservation, buckets_from, COST_FIELDS + +logger = logging.getLogger(__name__) +CAP_FIELD = "max_stream_cost_microusd" +MAX_CAP = 1_000_000_000_000 +FRAME_LIMIT = 65536 +_warned: set[str] = set() +FRAME_END = re.compile(br"\r\n\r\n|\n\n|\r\r") +FALSE = {"", "false", "0", "no", "off"} + +def enabled() -> bool: + flag = os.environ.get("STREAM_COST_BREAKER_ENABLED", "").strip().lower() + if flag in FALSE: + return False + if flag in {"true", "1", "yes", "on"}: + return True + if "flag" not in _warned: + _warned.add("flag") + logger.warning("Invalid STREAM_COST_BREAKER_ENABLED; stream cost breaker disabled") + return False + +def failure(code: str, status: int = 503) -> APIError: + return APIError("Stream cost enforcement could not admit this request", status, {"error": code}) + +def valid_cap(value: Any) -> bool: + return value is None or (type(value) is int and 0 <= value <= MAX_CAP) + +def validate_cap(value: Any) -> int | None: + if value is None or value == "": + return None + try: + number = Decimal(str(value)) * 1_000_000 + except (InvalidOperation, ValueError, TypeError): + raise APIError("max_stream_cost_usd must be a nonnegative whole number of micro-USD", 400) from None + if isinstance(value, bool) or not number.is_finite() or number < 0 or number > MAX_CAP or number != number.to_integral_value(): + raise APIError("max_stream_cost_usd must be a nonnegative whole number of micro-USD", 400) + return int(number) + +def storage_control(row) -> dict: + if not enabled(): + return {} + try: + cap = row[CAP_FIELD] + except (KeyError, IndexError): + return {} + if not valid_cap(cap): + raise failure("stream_cost_storage_unavailable") + return {CAP_FIELD: cap} if cap is not None else {} + +def cap_for(user): + if CAP_FIELD in user: + return user[CAP_FIELD] + if user.get("max_stream_cost_usd") is None: + return None + try: + return validate_cap(user["max_stream_cost_usd"]) + except APIError: + raise failure("stream_cost_storage_unavailable") from None + +def public_control(row) -> dict: + cap = row.get(CAP_FIELD) + return {"max_stream_cost_usd": cap / 1_000_000} if enabled() and cap is not None else {} + +def validated_control(payload) -> dict: + return {CAP_FIELD: validate_cap(payload["max_stream_cost_usd"])} if enabled() and "max_stream_cost_usd" in payload else {} + +def sql_select(query: str) -> str: + return query.replace("shadow_eval_rate\n", "shadow_eval_rate, max_stream_cost_microusd\n") if enabled() else query + +def ensure_sql_cap(connection) -> None: + if enabled() and CAP_FIELD not in {row["name"] for row in connection.execute("PRAGMA table_info(users)")}: + connection.execute("ALTER TABLE users ADD COLUMN max_stream_cost_microusd INTEGER CHECK (max_stream_cost_microusd BETWEEN 0 AND 1000000000000)") + +def persist_sql_cap(connection, username, controls) -> None: + if enabled() and CAP_FIELD in controls: + cap = controls[CAP_FIELD] + if not valid_cap(cap): + raise failure("stream_cost_storage_unavailable") + connection.execute("UPDATE users SET max_stream_cost_microusd=? WHERE username=?", (cap, username)) + +def d1_row(value): + from services import user_store + if not enabled(): + return user_store._row(value) + if not isinstance(value, dict) or CAP_FIELD not in value or not valid_cap(value[CAP_FIELD]): + raise failure("stream_cost_storage_unavailable") + base = user_store._row({name: item for name, item in value.items() if name != CAP_FIELD}) + return {**base, CAP_FIELD: value[CAP_FIELD]} + +def d1_read(operation, **values): + from services import user_store + response = user_store._call(operation, **values) + if operation == "get": + if "user" not in response: + raise failure("stream_cost_storage_unavailable") + return None if response["user"] is None else d1_row(response["user"]) + rows = response.get("users") + if not isinstance(rows, list) or len(rows) > user_store.PAGE_SIZE: + raise failure("stream_cost_storage_unavailable") + return [d1_row(row) for row in rows] + +def list_users(): + from services import user_store + if not enabled(): + return user_store.list_users() + rows, after = [], None + while True: + page = d1_read("list", after=after, limit=user_store.PAGE_SIZE) + rows.extend(page) + if len(page) < user_store.PAGE_SIZE: + return rows + after = page[-1]["username"] + +def get_user(username): + from services import user_store + return d1_read("get", username=username) if enabled() else user_store.get_user(username) + +def users_by_prefix(prefix): + from services import user_store + return d1_read("by_prefix", prefix=prefix) if enabled() else user_store.users_by_prefix(prefix) + +def upsert_user(user): + from services import user_store + if not enabled(): + return user_store.upsert_user(user) + fields = {name: user[name] for name in user_store.USER_FIELDS} + if CAP_FIELD in user: + if not valid_cap(user[CAP_FIELD]): + raise failure("stream_cost_storage_unavailable") + fields[CAP_FIELD] = user[CAP_FIELD] + if user_store._call("upsert", user=fields).get("stored") is not True: + raise failure("stream_cost_storage_unavailable") + +def input_bound(payload) -> int: + values = [payload[name] for name in ("messages", "input", "prompt", "contents", "system", + "instructions", "tools", "functions", "tool_choice", "response_format", "response_schema") if name in payload] + return len(json.dumps(values, ensure_ascii=False).encode("utf-8")) if values else 0 + +@dataclass(frozen=True) +class Prices: + model: str + bound_input: Decimal + output: Decimal + flat: Decimal + rates: tuple[Decimal, ...] + +def prices_for(model: str) -> Prices: + probe = CacheObservation(3, 1, "provider", ("provider",), 1, 1, 1, "anthropic") + result = CostService.price_buckets(model, probe) + if result["cost_usd"] is None or any(result[name] is None for name in COST_FIELDS): + raise failure("stream_cost_unpriced") + rates = [Decimal(str(result[name])) for name in COST_FIELDS] + catalog = CostService.pricing_for(model) or {} + flat = catalog.get("request", Decimal(0)) * 1_000_000 + return Prices(model, max(rates[:3]), rates[3], flat, tuple(rates)) + +def _strings(value) -> int: + if isinstance(value, str): + return len(value.encode("utf-8")) + if isinstance(value, dict): + return sum(_strings(v) for v in value.values()) + if isinstance(value, list): + return sum(_strings(v) for v in value) + return 0 + +def _output_bytes(value) -> int: + choices = value.get("choices") + if isinstance(choices, list): + return sum(_strings(choice.get("delta", {})) for choice in choices if isinstance(choice, dict)) + delta = value.get("delta") + if isinstance(delta, (str, dict)): + return _strings(delta) + return 0 + +def _terminal(value) -> bool: + return value.get("type") in {"message_stop", "response.completed", "response.failed", "response.incomplete"} or ( + isinstance(value.get("choices"), list) and any(isinstance(c, dict) and c.get("finish_reason") is not None for c in value["choices"])) + +class StreamCostState: + def __init__(self, cap, prices, input_tokens, protocol): + self.cap, self.prices, self.input_tokens, self.protocol = cap, prices, input_tokens, protocol + self.running_microusd = 0 + self.basis = "conservative_estimate" + self.exceeded = False + self.observation = None + self.final_usage = None + self.output_bound = 0 + self.output_at_measurement = 0 + self.cancel = None + self.terminal = False + self._cost() + + def _cost(self): + measured = self.observation + additional = self.output_bound - self.output_at_measurement + costs, fully_measured = [], measured is not None and additional == 0 + counts = buckets_from(measured) + amounts = [counts[name] for name in ( + "ordinary_input_tokens", "cache_read_input_tokens", "cache_write_input_tokens", "output_tokens")] + for price in self.prices: + if all(amount is not None or rate == 0 for amount, rate in zip(amounts, price.rates)): + amount = price.flat + sum((Decimal(value or 0) * rate for value, rate in zip(amounts, price.rates)), Decimal(0)) + amount += additional * price.output + else: + fully_measured = False + known_input = sum(v for v in amounts[:3] if v is not None) + output = measured.output_tokens if measured is not None and measured.output_tokens is not None else 0 + amount = price.flat + max(self.input_tokens, known_input) * price.bound_input + max(output + additional, self.output_bound) * price.output + costs.append(amount) + self.running_microusd = int(max(costs).to_integral_value(rounding=ROUND_CEILING)) + self.basis = "measured" if fully_measured else "conservative_estimate" + + def observe(self, frame: bytes) -> bool: + try: + text = frame.decode("utf-8") + data = "\n".join(line[5:].lstrip(" ") for line in text.splitlines() if line.startswith("data:")) + if not data: + return True + if data == "[DONE]": + self.terminal = True + if self.basis == "measured": + self.final_usage = self.observation + return True + value = json.loads(data) + if not isinstance(value, dict): + raise ValueError + except (UnicodeError, ValueError, RecursionError): + self.exceeded = True + return False + self.output_bound += _output_bytes(value) + usage = CacheObservation.from_body(value) + if usage is not None: + self.observation = self.observation.merge(usage) if self.observation is not None else usage + if usage.output_tokens is not None: + self.output_at_measurement = self.output_bound + self._cost() + if _terminal(value): + self.terminal = True + if self.terminal and self.basis == "measured": + self.final_usage = self.observation + elif self.basis != "measured": + self.final_usage = None + if self.running_microusd > self.cap: + self.exceeded = True + return False + return True + + def error_frame(self) -> bytes: + error = {"code": "stream_cost_cap_exceeded", "type": "stream_cost_cap_exceeded", + "message": "The stream cost cap was exceeded.", "cost_basis": self.basis} + if self.protocol == "anthropic": + value, prefix = {"type": "error", "error": error}, "event: error\n" + elif self.protocol == "responses": + value, prefix = {"type": "response.failed", "response": {"object": "response", "status": "failed", "error": error}}, "event: response.failed\n" + else: + value, prefix = {"error": error}, "" + return (prefix + "data: " + json.dumps(value) + "\n\n").encode() + +def prepare(user, models, input_tokens, protocol): + if not enabled(): + return None + cap = cap_for(user) + if cap is None: + return None + if not valid_cap(cap): + raise failure("stream_cost_storage_unavailable") + if not models: + raise failure("stream_cost_unpriced") + state = StreamCostState(cap, [prices_for(model) for model in models], input_tokens, protocol) + if state.running_microusd > cap: + raise failure("stream_cost_cap_exceeded", 429) + return state + +def bind_upstream(context): + from flask import g, has_request_context + if has_request_context(): + usage = getattr(g, "usage_context", None) + state = getattr(usage, "stream_cost", None) + if state is not None: + owner = getattr(g, "gateway_cancellation", None) + state.cancel = owner.cancel if owner is not None else context.cancel + +class CostStream: + """Buffer one bounded SSE frame; close even before the first read.""" + def __init__(self, source, state): + self.source, self.iterator, self.state = source, iter(source), state + self.pending = b"" + self.stopped = self.closed = False + self.trailer = b"" + + def __iter__(self): + return self + + def _close_source(self): + if self.closed: + return + self.closed = True + try: + if self.state.cancel is not None: + self.state.cancel() + finally: + close = getattr(self.source, "close", None) + if callable(close): + try: + close() + except Exception as error: + logger.warning("Stream cost cleanup failed type=%s", type(error).__name__) + + def _reject(self, keep_usage=False): + self.state.exceeded = True + if not keep_usage: + self.state.final_usage = None + self.stopped = True + self.pending = self.trailer = b"" + self._close_source() + return self.state.error_frame() + + def __next__(self): + if self.stopped: + raise StopIteration + while True: + match = FRAME_END.search(self.pending) + if match: + frame, self.pending = self.pending[:match.end()], self.pending[match.end():] + if len(frame) > FRAME_LIMIT: + return self._reject() + if not self.state.observe(frame): + return self._reject(keep_usage=self.state.running_microusd > self.state.cap) + if self.state.terminal: + self.trailer += frame + if len(self.trailer) > FRAME_LIMIT: + return self._reject() + continue + return frame + if len(self.pending) > FRAME_LIMIT: + return self._reject() + try: + chunk = next(self.iterator) + except StopIteration: + if self.pending: + return self._reject() + self.stopped = True + self._close_source() + if self.trailer: + trailer, self.trailer = self.trailer, b"" + return trailer + raise + except BaseException: + self.close() + raise + self.pending += chunk.encode("utf-8") if isinstance(chunk, str) else bytes(chunk) + + def close(self): + self.stopped = True + self.pending = b"" + self._close_source() + +def wrap_stream(source, state): + return CostStream(source, state) if state is not None else source diff --git a/services/upstream_transport.py b/services/upstream_transport.py index 5301bd1e..0cc9fb74 100644 --- a/services/upstream_transport.py +++ b/services/upstream_transport.py @@ -6,6 +6,7 @@ from streaming.sse import iter_sse_data from services.request_cancellation import CancellationIterator, bind_cancellation +from services.stream_cost_breaker import bind_upstream def close_retry_response(response: requests.Response) -> None: """Release a retryable response without masking the original result.""" @@ -14,7 +15,9 @@ def close_retry_response(response: requests.Response) -> None: def iter_stream_content(response: requests.Response, *, pii_context=None) -> Iterator[bytes]: """Yield decoded upstream bytes as soon as the socket exposes them.""" - source = CancellationIterator(lambda: _stream_content(response), bind_cancellation(response)) + context = bind_cancellation(response) + bind_upstream(context) + source = CancellationIterator(lambda: _stream_content(response), context) if pii_context is None: return source from services.pii_stream import RehydratingIterator diff --git a/tests/fixtures/secret_firewall_coverage.json b/tests/fixtures/secret_firewall_coverage.json index 21690fb8..c4696421 100644 --- a/tests/fixtures/secret_firewall_coverage.json +++ b/tests/fixtures/secret_firewall_coverage.json @@ -397,6 +397,7 @@ "worker/shadow-eval-schedule.mjs", "worker/sse-heartbeat.mjs", "worker/status-page.mjs", + "worker/stream-cost-breaker.mjs", "worker/upstream-cancellation.mjs", "worker/usage-buckets-d1.mjs", "worker/usage-ledger-d1.mjs" diff --git a/tests/test_d1_integration.mjs b/tests/test_d1_integration.mjs index 6b419cba..16642b95 100644 --- a/tests/test_d1_integration.mjs +++ b/tests/test_d1_integration.mjs @@ -70,10 +70,11 @@ test("key controls and the usage ledger persist through the Worker in D1", async }); test("a D1 without the account migration fails closed for dashboard keys but not for the environment admin", async t => { - // 0007, 0011 and 0013 alter control_users, so they are skipped with the migration that creates the table; + // 0007, 0011, 0013 and 0030 alter control_users, so they are skipped with the migration that creates the table; // 0016 alters 0007's usage_events. const { url } = await privateStore(t, ["0003_control_users.sql", "0004_control_user_audit.sql", "0005_auto_routes.sql", - "0007_usage_ledger.sql", "0011_secret_firewall.sql", "0013_shadow_eval.sql", "0016_usage_buckets.sql"]); + "0007_usage_ledger.sql", "0011_secret_firewall.sql", "0013_shadow_eval.sql", "0016_usage_buckets.sql", + "0030_stream_cost_caps.sql"]); assert.deepEqual(await drive(url, "unmigrated"), { dashboard_key: 503, admin: "admin", route: ["gguu:gpt-image-2.5-sunburst", "gguu:gpt-image-2.5"], save_route: 503 }); }); diff --git a/tests/test_d1_schema_worker.mjs b/tests/test_d1_schema_worker.mjs index 44701153..ff824e08 100644 --- a/tests/test_d1_schema_worker.mjs +++ b/tests/test_d1_schema_worker.mjs @@ -29,8 +29,8 @@ test("/ready reports a D1 schema that was never migrated instead of forwarding", const worker = (await loadWorkerModule()).default; let forwarded = 0; const container = { getByName: () => ({ fetch: async () => { forwarded += 1; return Response.json({ status: "healthy" }); } }) }; - // 0007 alters control_users, so skipping 0003 skips it too; 0016 alters 0007's usage_events. - const partial = await database(t, { skip: ["0003_control_users.sql", "0005_auto_routes.sql", "0007_usage_ledger.sql", "0011_secret_firewall.sql", "0013_shadow_eval.sql", "0016_usage_buckets.sql"] }); + // 0007 and 0030 alter control_users, so skipping 0003 skips them too; 0016 alters 0007's usage_events. + const partial = await database(t, { skip: ["0003_control_users.sql", "0005_auto_routes.sql", "0007_usage_ledger.sql", "0011_secret_firewall.sql", "0013_shadow_eval.sql", "0016_usage_buckets.sql", "0030_stream_cost_caps.sql"] }); assert.deepEqual(await d1Readiness(partial), { ready: false, checked: true, missing: ["control_users", "auto_routes", "usage_events", "usage_daily", "usage_batches", "shadow_eval_samples", "shadow_eval_results", "shadow_eval_config", "shadow_eval_policy_backups"] }); const blocked = await worker.fetch(new Request("https://gateway.example/ready"), { INTELLIGENCE_DB: partial, MULTILLM_PROXY_CONTAINER: container }); assert.equal(blocked.status, 503); diff --git a/tests/test_stream_cost_breaker.py b/tests/test_stream_cost_breaker.py new file mode 100644 index 00000000..b5f565f0 --- /dev/null +++ b/tests/test_stream_cost_breaker.py @@ -0,0 +1,241 @@ +"""Offline cost breaker fixtures.""" +import json +import sqlite3 +from pathlib import Path +import pytest +from flask import Flask, Response, g +from error_handlers import APIError +from services import stream_cost_breaker as breaker, request_accounting as accounting, key_controls +from services.upstream_transport import iter_stream_content + +@pytest.fixture(autouse=True) +def settings(monkeypatch): + for name, value in {"STREAM_COST_BREAKER_ENABLED":"true", + "MODEL_PRICING_USD_PER_MILLION":json.dumps({"openai:m":{"input":1,"cache_read":0.25,"cache_write":2,"output":1}}), + "PROMPT_CACHE_PRICE_METADATA_JSON":"{}", "PROMPT_CACHE_USAGE_BUCKETS_ENABLED":"false", + "USAGE_RESERVATIONS_ENABLED":"false","AUTH_STORAGE_BACKEND":"sql", + "CONTROL_PLANE_DATABASE_URL":"","CONFIG_REVISION_SYNC_ENABLED":"false"}.items(): + monkeypatch.setenv(name,value) + +def frame(value): + return ("data: "+json.dumps(value,ensure_ascii=False)+"\n\n").encode() +def delta(text): + return frame({"choices":[{"delta":{"content":text}}]}) +class Source: + def __init__(self,chunks): + self.chunks=iter(chunks) + self.closes=self.reads=0 + def __iter__(self): return self + def __next__(self): + self.reads+=1 + return next(self.chunks) + def close(self): self.closes+=1 + +@pytest.mark.parametrize("flag",["","false","private-invalid-value"]) +def test_defaults(monkeypatch,caplog,flag): + monkeypatch.setenv("STREAM_COST_BREAKER_ENABLED",flag) + breaker._warned.clear() + source=Source([b"\xff",b"data: malformed\n",b"\n"]) + assert breaker.wrap_stream(source,None) is source + assert "max_stream_cost_usd" not in key_controls.public({}) + assert not breaker.enabled() and not breaker.enabled() + assert "private-invalid-value" not in caplog.text + assert len(caplog.records)==(1 if flag.startswith("private") else 0) + +@pytest.mark.parametrize("protocol,marker",[("chat",b'data: {"error"'),("anthropic",b"event: error"),("responses",b"event: response.failed")]) +def test_crossing(protocol,marker): + state=breaker.prepare({"max_stream_cost_microusd":4},["openai:m"],0,protocol) + source=Source([delta("four"),delta("secret"),b"data: [DONE]\n\n"]) + wrapped=breaker.wrap_stream(source,state) + chunks=list(wrapped) + wrapped.close() + assert chunks[0]==delta("four") + assert marker in chunks[1] and b"stream_cost_cap_exceeded" in chunks[1] + assert b"[DONE]" not in b"".join(chunks) and b"finish_reason" not in chunks[1] + assert source.closes==1 and source.reads==2 and state.exceeded + +def test_unicode_and_crlf(): + data=delta("เคน").replace(b"\n",b"\r\n")+b"data: [DONE]\r\n\r\n" + state=breaker.prepare({"max_stream_cost_microusd":3},["openai:m"],0,"chat") + assert b"".join(breaker.wrap_stream(Source([data[i:i+1] for i in range(len(data))]),state))==data + assert state.running_microusd==3 + +def test_cache_and_unknown_final(): + state=breaker.prepare({"max_stream_cost_microusd":100},["openai:m"],0,"anthropic") + data=frame({"type":"message","usage":{"input_tokens":4,"cache_read_input_tokens":8,"cache_creation_input_tokens":2,"output_tokens":1}}) + list(breaker.wrap_stream(Source([data,frame({"type":"message_stop"})]),state)) + assert state.basis=="measured" and state.running_microusd==11 and state.final_usage is not None + unknown=breaker.prepare({"max_stream_cost_microusd":100},["openai:m"],3,"chat") + list(breaker.wrap_stream(Source([delta("x"),b"data: [DONE]\n\n"]),unknown)) + assert unknown.final_usage is None and unknown.basis=="conservative_estimate" + +def test_preflight(): + with pytest.raises(APIError) as error: + breaker.prepare({"max_stream_cost_microusd":20},["openai:m","missing"],0,"chat") + assert error.value.status_code==503 and error.value.payload["error"]=="stream_cost_unpriced" + with pytest.raises(APIError) as error: + breaker.prepare({"max_stream_cost_microusd":1},["openai:m"],2,"chat") + assert error.value.payload["error"]=="stream_cost_cap_exceeded" + +@pytest.mark.parametrize("value",[True,-1,"nan","inf",0.0000001,{},1000000001]) +def test_invalid_cap(value): + with pytest.raises(APIError): key_controls.validate({"max_stream_cost_usd":value}) + +def test_cap_and_old_key_migration(): + assert key_controls.validate({"max_stream_cost_usd":"0.000012"})["max_stream_cost_microusd"]==12 + assert key_controls.public({"max_stream_cost_microusd":12})["max_stream_cost_usd"]==0.000012 + assert "max_stream_cost_usd" not in key_controls.public({"max_stream_cost_microusd":None}) + db=sqlite3.connect(":memory:") + db.execute("CREATE TABLE control_users (username TEXT PRIMARY KEY)") + db.execute("INSERT INTO control_users VALUES (?)",("old",)) + db.executescript(Path("intelligence-migrations/0030_stream_cost_caps.sql").read_text()) + assert db.execute("SELECT max_stream_cost_microusd FROM control_users").fetchone()==(None,) + +def test_accounting_path(monkeypatch): + app=Flask(__name__) + rows=[] + monkeypatch.setattr(accounting.usage_ledger.LEDGER,"record",rows.append) + monkeypatch.setattr(accounting.telemetry_export.EXPORTER,"submit",lambda row:None) + monkeypatch.setattr(accounting.BudgetService,"record_cost",lambda row:None) + monkeypatch.setattr(accounting.BudgetService,"settle",lambda *a,**k:None) + class Upstream(Source): + headers={"content-type":"text/event-stream"} + status_code=200 + def iter_content(self,chunk_size): yield from self + upstream=Upstream([delta("ok"),delta("secret-response"),b"data: [DONE]\n\n"]) + @app.post("/v1/chat/completions") + def chat(): + g.authenticated_user={"username":"reader","max_stream_cost_usd":0.000003} + g.rate_limit={"input_tokens":0,"output_tokens":50} + denied=accounting.begin() + if denied is not None: return denied + return accounting.finish(Response(iter_stream_content(upstream),mimetype="text/event-stream")) + with app.test_client() as client: + response=client.post("/v1/chat/completions",json={"model":"openai:m","stream":True}) + data=response.get_data() + response.close() + assert b"stream_cost_cap_exceeded" in data and b"secret-response" not in data and b"[DONE]" not in data + assert upstream.closes==1 and len(rows)==1 and rows[0]["status"]==499 and rows[0]["cost_usd"] is None + with app.test_request_context("/v1/chat/completions",method="POST",json={"model":"missing","stream":True}): + g.authenticated_user={"username":"reader","max_stream_cost_usd":0.000003} + denied=accounting.begin() + assert denied.status_code==503 and denied.get_json()["error"]=="stream_cost_unpriced" + +def test_trailing_measured_overrun_never_emits_success_finish(): + state=breaker.prepare({"max_stream_cost_microusd":4},["openai:m"],0,"chat") + stop=frame({"choices":[{"delta":{},"finish_reason":"stop"}]}) + usage=frame({"usage":{"prompt_tokens":0,"prompt_tokens_details":{"cached_tokens":0},"completion_tokens":5}}) + data=b"".join(breaker.wrap_stream(Source([delta("four"),stop,usage,b"data: [DONE]\n\n"]),state)) + assert b"stream_cost_cap_exceeded" in data and b"finish_reason" not in data and b"[DONE]" not in data + assert state.final_usage is not None and state.final_usage.output_tokens==5 + +def test_multiline_and_oversized_frames(): + value=b'data: {"choices":\n' + b'data: [{"delta":{"content":"ok"}}]}\n\n' + state=breaker.prepare({"max_stream_cost_microusd":2},["openai:m"],0,"chat") + assert b"".join(breaker.wrap_stream(Source([value]),state))==value + state=breaker.prepare({"max_stream_cost_microusd":100000},["openai:m"],0,"chat") + source=Source([b"data: "+b"x"*breaker.FRAME_LIMIT]) + result=b"".join(breaker.wrap_stream(source,state)) + assert b"stream_cost_cap_exceeded" in result and source.closes==1 + +def test_cancel_before_read_and_concurrent_state_isolation(): + state=breaker.prepare({"max_stream_cost_microusd":2},["openai:m"],0,"chat") + source=Source([delta("x")]) + body=breaker.wrap_stream(source,state) + body.close(); body.close() + assert source.closes==1 and source.reads==0 + other=breaker.prepare({"max_stream_cost_microusd":2},["openai:m"],0,"chat") + assert b"".join(breaker.wrap_stream(Source([delta("ok")]),other))==delta("ok") + assert state.output_bound==0 and other.output_bound==2 + +def test_sql_accounts_keep_cap_through_updates_rotation_and_disabled_writes(tmp_path,monkeypatch): + from unittest.mock import patch + from services.auth_service import AuthService + monkeypatch.setenv("AUTH_DB_PATH",str(tmp_path/"auth.sqlite3")) + monkeypatch.setenv("ADMIN_API_KEY","synthetic-stream-cost-admin") + monkeypatch.setenv("ADMIN_USERNAME","admin") + for name,value in (("_storage_path",None),("_users",{}),("_api_key_prefix_index",{}),("_verified_keys",{})): + monkeypatch.setattr(AuthService,name,value) + AuthService.initialize() + with patch.object(AuthService,"get_current_user",return_value={"username":"admin","is_admin":True}): + AuthService.create_user("reader",scopes=["chat"]) + AuthService.set_key_controls("reader",{"max_stream_cost_usd":"0.000004"}) + AuthService.set_key_controls("reader",{"allowed_models":["openai:*"]}) + key=AuthService.rotate_api_key("reader")["api_key"] + verified=AuthService.verify_api_key(key) + assert verified["max_stream_cost_usd"]==0.000004 + with Flask(__name__).test_request_context("/v1/chat/completions",method="POST", + json={"model":"openai:missing","stream":True}): + g.authenticated_user=verified + denied=accounting.begin() + assert denied.status_code==503 and denied.get_json()["error"]=="stream_cost_unpriced" + monkeypatch.setenv("STREAM_COST_BREAKER_ENABLED","false") + AuthService.rotate_api_key("reader") + monkeypatch.setenv("STREAM_COST_BREAKER_ENABLED","true") + assert AuthService._load_user_by_username("reader")["max_stream_cost_microusd"]==4 + AuthService.set_key_controls("reader",{"max_stream_cost_usd":None}) + assert "max_stream_cost_usd" not in AuthService._public_user("reader",AuthService._users["reader"]) + +def test_strict_d1_rows_missing_schema_and_off_wire(monkeypatch): + from services import user_store + row={name:None for name in user_store.USER_FIELDS} + row.update(username="reader",api_key_hash="synthetic-hash",api_key_prefix="synthetic-prefix", + scopes="chat",is_admin=0,created_at="2026-10-09T00:00:00Z") + monkeypatch.setattr(user_store,"_call",lambda *a,**k:{"user":row}) + with pytest.raises(APIError) as error: breaker.get_user("reader") + assert error.value.status_code==503 and error.value.payload["error"]=="stream_cost_storage_unavailable" + row["max_stream_cost_microusd"]=4 + assert breaker.get_user("reader")["max_stream_cost_microusd"]==4 + row["unknown"]=None + with pytest.raises(APIError): breaker.get_user("reader") + row.pop("unknown"); row.pop("max_stream_cost_microusd") + monkeypatch.setenv("STREAM_COST_BREAKER_ENABLED","false") + captured=[] + monkeypatch.setattr(user_store,"upsert_user",captured.append) + breaker.upsert_user(row) + assert captured==[row] + +def test_real_durable_unknown_hold(monkeypatch,tmp_path): + from services import reservation_store + from services.budget_service import BudgetService + monkeypatch.setenv("USAGE_RESERVATIONS_ENABLED","true") + monkeypatch.setenv("USAGE_HOLD_REVIEW_AFTER_SECONDS","") + store=reservation_store.SqlReservationStore(str(tmp_path/"holds.sqlite3")) + monkeypatch.setattr(reservation_store,"get_store",lambda:store) + identity="a"*32 + store.reserve(identity,"reader",0.001,1,None,0,0) + BudgetService.mark_dispatched(identity) + state=breaker.prepare({"max_stream_cost_microusd":4},["openai:m"],0,"chat") + list(breaker.wrap_stream(Source([delta("four"),delta("over")]),state)) + context=accounting.UsageContext("chat",["openai:m"],None,{"username":"reader"}, + accounting.time.perf_counter(),1,reservation=identity,path="/v1/chat/completions",trace=(None,None),stream_cost=state) + monkeypatch.setattr(accounting.usage_ledger.LEDGER,"record",lambda row:None) + monkeypatch.setattr(accounting.telemetry_export.EXPORTER,"submit",lambda row:None) + accounting._record(context,499,None,None) + assert store.get(identity)["state"]=="unknown" + assert store.summary("reader")["held_usd"]==0.001 + +def test_cached_stream_keeps_zero_charge_and_releases_hold(monkeypatch): + app=Flask(__name__) + rows,settled=[],[] + monkeypatch.setattr(accounting.usage_ledger.LEDGER,"record",rows.append) + monkeypatch.setattr(accounting.telemetry_export.EXPORTER,"submit",lambda row:None) + monkeypatch.setattr(accounting.BudgetService,"record_cost",lambda row:None) + monkeypatch.setattr(accounting.BudgetService,"settle",lambda identity,**kwargs:settled.append(identity)) + with app.test_request_context("/v1/chat/completions",method="POST"): + g.usage_context=accounting.UsageContext("chat",["openai:m"],None,{"username":"reader"}, + accounting.time.perf_counter(),1,reservation="cached",trace=(None,None), + stream_cost=breaker.prepare({"max_stream_cost_microusd":0},["openai:m"],0,"chat")) + original=delta("cached")+b"data: [DONE]\n\n" + response=accounting.finish(Response(iter([original]),mimetype="text/event-stream", + headers={"X-MultiLLM-Cache":"hit"})) + assert b"".join(response.response)==original + response.close() + assert rows[0]["cost_usd"]==0 and rows[0]["status"]==200 and rows[0]["cost_basis"]=="cache" + assert settled==["cached"] + +def test_fractional_prices_do_not_create_a_rounding_fee(monkeypatch): + monkeypatch.setenv("MODEL_PRICING_USD_PER_MILLION",json.dumps({"openai:m":{ + "input":0.1,"cache_read":0.1,"cache_write":0.1,"output":0.1}})) + state=breaker.prepare({"max_stream_cost_microusd":0},["openai:m"],0,"chat") + assert state.running_microusd==0 diff --git a/tests/test_stream_cost_breaker_worker.mjs b/tests/test_stream_cost_breaker_worker.mjs new file mode 100644 index 00000000..1f83f953 --- /dev/null +++ b/tests/test_stream_cost_breaker_worker.mjs @@ -0,0 +1,107 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { prepareStreamCostBreaker, wrapStreamCost, streamCostEnabled, validStreamCap } from "../worker/stream-cost-breaker.mjs"; +const env={STREAM_COST_BREAKER_ENABLED:"true",MODEL_PRICING_USD_PER_MILLION:JSON.stringify({"openai:m":{input:1,cache_read:0.25,cache_write:2,output:1}})}; +const enc=new TextEncoder(); +const frame=v=>enc.encode("data: "+JSON.stringify(v)+"\n\n"); +const delta=text=>frame({choices:[{delta:{content:text}}]}); +function source(chunks) { + let closes=0; + return {stream:new ReadableStream({pull(c){chunks.length?c.enqueue(chunks.shift()):c.close();},cancel(){closes++;}},{highWaterMark:0}),closes:()=>closes}; +} +const read=stream=>new Response(stream).text(); +test("off and null preserve identity",()=>{ + for(const flag of ["","false","garbage"]) assert.equal(streamCostEnabled({STREAM_COST_BREAKER_ENABLED:flag}),false); + const original=source([delta("x")]).stream; + assert.equal(wrapStreamCost(original,null),original); + assert.equal(prepareStreamCostBreaker(env,{},["missing"],0,"chat"),null); + for(const cap of [-1,0.5,true,NaN,1e16]) assert.equal(validStreamCap(cap),false); +}); +for(const [protocol,marker] of [["chat",'data: {"error"'],["anthropic","event: error"],["responses","event: response.failed"]]) { + test(protocol+" crossing closes once",async()=>{ + const state=prepareStreamCostBreaker(env,{max_stream_cost_microusd:4},["openai:m"],0,protocol); + const upstream=source([delta("four"),delta("secret-response"),enc.encode("data: [DONE]\n\n")]); + const text=await read(wrapStreamCost(upstream.stream,state)); + assert.ok(text.startsWith(new TextDecoder().decode(delta("four"))) && text.includes(marker)); + assert.ok(text.includes("stream_cost_cap_exceeded") && !text.includes("secret-response") && !text.includes("[DONE]")); + assert.equal(upstream.closes(),1); assert.equal(state.basis,"conservative_estimate"); + }); +} +test("split unicode SSE",async()=>{ + const bytes=delta("เคน"),state=prepareStreamCostBreaker(env,{max_stream_cost_microusd:3},["openai:m"],0,"chat"); + assert.equal(await read(wrapStreamCost(source([...bytes].map(v=>Uint8Array.of(v))).stream,state)),new TextDecoder().decode(bytes)); + assert.equal(state.runningMicrousd,3); +}); +test("measured buckets and unknown final",async()=>{ + const state=prepareStreamCostBreaker(env,{max_stream_cost_microusd:100},["openai:m"],0,"anthropic"); + await read(wrapStreamCost(source([frame({type:"message",usage:{input_tokens:4,cache_read_input_tokens:8,cache_creation_input_tokens:2,output_tokens:1}}),frame({type:"message_stop"})]).stream,state)); + assert.equal(state.runningMicrousd,11); assert.equal(state.basis,"measured"); assert.ok(state.finalUsage); + const unknown=prepareStreamCostBreaker(env,{max_stream_cost_microusd:100},["openai:m"],0,"chat"); + await read(wrapStreamCost(source([delta("x")]).stream,unknown)); assert.equal(unknown.finalUsage,null); +}); +test("unpriced candidate rejects",()=>{ + assert.throws(()=>prepareStreamCostBreaker(env,{max_stream_cost_microusd:20},["openai:m","missing"],0,"chat"),e=>e.code==="stream_cost_unpriced"&&e.status===503); +}); + +import { USER_FIELDS, validUser, handleControlUsersRequest } from "../worker/control-users-d1.mjs"; +const account = cap => ({...Object.fromEntries(USER_FIELDS.map(name=>[name,null])), + username:"reader",api_key_hash:"synthetic-hash",api_key_prefix:"synthetic-prefix",scopes:"chat",is_admin:0, + created_at:"2026-10-09T00:00:00Z",...cap}); +function fakeDb({missing=false,stored={}}={}) { + const statements=[],rows=new Map(Object.entries(stored)); + const db={prepare(sql){ + if(missing && sql.includes("max_stream_cost_microusd")) throw new Error("no such column: max_stream_cost_microusd"); + const query={sql,args:[],bind(...args){this.args=args;return this;}, + async first(){return sql.includes("LIMIT 1")?null:rows.get(this.args[0])??null;}, + async all(){return {results:[...rows.values()]};}}; + statements.push(query); return query; + },async batch(items){ + for(const item of items) if(item.sql.startsWith("INSERT INTO control_users")) { + const names=item.sql.match(/\(([^)]+)\)/)[1].split(",").map(v=>v.trim()); + const row=Object.fromEntries(names.map((name,i)=>[name,item.args[i]])); + rows.set(row.username,{...rows.get(row.username),...row}); + } + return []; + }}; + return {db,rows,statements}; +} +async function call(db,body,flags=env) { + return handleControlUsersRequest(new Request("http://intelligence.internal/v1/users",{method:"POST", + headers:{"content-type":"application/json"},body:JSON.stringify({version:1,...body})}),{...flags,INTELLIGENCE_DB:db}); +} +test("strict caps and omitted updates preserve existing column",async()=>{ + for(const value of [-1,0.5,true,NaN,1e16]) assert.equal(validUser(account({max_stream_cost_microusd:value})),false); + const {db,rows,statements}=fakeDb({stored:{reader:account({max_stream_cost_microusd:4})}}); + assert.equal((await call(db,{operation:"upsert",user:account({})})).status,200); + assert.equal(rows.get("reader").max_stream_cost_microusd,4); + assert.equal((await call(db,{operation:"upsert",user:account({max_stream_cost_microusd:null})})).status,200); + assert.equal(rows.get("reader").max_stream_cost_microusd,null); + const legacy=fakeDb(); + assert.equal((await call(legacy.db,{operation:"upsert",user:account({})},{})).status,200); + assert.ok(legacy.statements.every(s=>!s.sql.includes("max_stream_cost_microusd"))); + assert.equal(Object.keys(legacy.rows.get("reader")).length,USER_FIELDS.length); +}); +test("enabled missing column fails closed before writes",async()=>{ + const {db,statements}=fakeDb({missing:true}); + for(const body of [{operation:"get",username:"reader"},{operation:"list",after:null,limit:5}, + {operation:"by_prefix",prefix:"synthetic-prefix"},{operation:"upsert",user:account({max_stream_cost_microusd:4})}]) { + const response=await call(db,body); assert.equal(response.status,503); + assert.equal((await response.json()).error.code,"stream_cost_storage_unavailable"); + } + assert.ok(statements.every(s=>!s.sql.startsWith("INSERT"))); +}); +test("late provider usage overrun withholds successful completion",async()=>{ + const state=prepareStreamCostBreaker(env,{max_stream_cost_microusd:4},["openai:m"],0,"chat"); + const text=await read(wrapStreamCost(source([delta("four"), + frame({choices:[{delta:{},finish_reason:"stop"}]}), + frame({usage:{prompt_tokens:0,prompt_tokens_details:{cached_tokens:0},completion_tokens:5}}), + enc.encode("data: [DONE]\n\n")]).stream,state)); + assert.ok(text.includes("stream_cost_cap_exceeded")&&!text.includes("finish_reason")&&!text.includes("[DONE]")); + assert.equal(state.finalUsage?.output,5); +}); +test("cancel before reading invokes cleanup and finalization once",async()=>{ + const state=prepareStreamCostBreaker(env,{max_stream_cost_microusd:4},["openai:m"],0,"chat"),upstream=source([delta("x")]); + let aborts=0,finals=0; + const wrapped=wrapStreamCost(upstream.stream,state,{abort(){aborts++;},onFinal(){finals++;}}); + await wrapped.cancel(); assert.equal(upstream.closes(),1);assert.equal(aborts,1);assert.equal(finals,1); +}); diff --git a/worker/container-env.mjs b/worker/container-env.mjs index dd44801f..d5116a3d 100644 --- a/worker/container-env.mjs +++ b/worker/container-env.mjs @@ -42,6 +42,7 @@ const DIRECT_ENV_KEYS = [ "CONFIG_SYNC_TTL_SECONDS", "REALTIME_ENABLED", "REALTIME_PROVIDERS_JSON", + "STREAM_COST_BREAKER_ENABLED", "BANDIT_MODE", "BATCH_SPILLOVER_ENABLED", "CANARY_TRAFFIC_ENABLED", diff --git a/worker/control-users-d1.mjs b/worker/control-users-d1.mjs index ef364728..363ef2b7 100644 --- a/worker/control-users-d1.mjs +++ b/worker/control-users-d1.mjs @@ -2,6 +2,7 @@ * Durable dashboard accounts (username, key hash and metadata) in D1, reachable only * through the Container's private outbound handler. Fixed statements, no client SQL. */ +import { STREAM_CAP_FIELD, streamCostEnabled, validStreamCap } from "./stream-cost-breaker.mjs"; import { commitRevision, revisionSyncSettings } from "./config-revision.mjs"; import { logFailure } from "./log.mjs"; import { AUDIT_OPERATIONS, handleAuditOperation } from "./control-audit-d1.mjs"; @@ -32,7 +33,15 @@ const reply = (value, status = 200) => Response.json(value.error : value, { status, headers: { "cache-control": "no-store" } }); const fields = (body, names) => Object.keys(body).length === names.length && names.every(key => Object.hasOwn(body, key)); +export function streamUserFields(env) { + return streamCostEnabled(env) ? [...USER_FIELDS, STREAM_CAP_FIELD] : USER_FIELDS; +} + export function validUser(user) { + if (user && Object.hasOwn(user, STREAM_CAP_FIELD)) { + if (!validStreamCap(user[STREAM_CAP_FIELD])) return false; + user = Object.fromEntries(Object.entries(user).filter(([name]) => name !== STREAM_CAP_FIELD)); + } if (user && !Object.hasOwn(user, "shadow_eval_rate")) user = { ...user, shadow_eval_rate: null }; if (user && !Object.hasOwn(user, "secret_scan_mode")) user = { ...user, secret_scan_mode: null }; return user !== null && typeof user === "object" && !Array.isArray(user) && fields(user, USER_FIELDS) @@ -129,7 +138,8 @@ async function read(query) { } /** Run an account query with every column, or with the pre-0007 columns and no controls. */ -async function readUsers(query) { +async function readUsers(query, env = {}) { + if (streamCostEnabled(env)) return read(() => query(streamUserFields(env).join(", "))); for (const end of [USER_FIELDS.length, USER_FIELDS.length - 1, USER_FIELDS.length - 2, 12]) { try { const rows = await read(() => query(USER_FIELDS.slice(0, end).join(", "))); @@ -155,9 +165,9 @@ function audit(db, operation, outcome, user) { } /** Unrevoked accounts for a key prefix, shared by the Container RPC and the edge. */ -export async function activeUsersByPrefix(db, prefix) { +export async function activeUsersByPrefix(db, prefix, env = {}) { return readUsers(async columns => (await db.prepare(`SELECT ${columns} FROM control_users - WHERE api_key_prefix=? AND revoked_at IS NULL ORDER BY username LIMIT ${MAX_PAGE}`).bind(prefix).all()).results); + WHERE api_key_prefix=? AND revoked_at IS NULL ORDER BY username LIMIT ${MAX_PAGE}`).bind(prefix).all()).results, env); } export async function boundedBody(request, maxBytes = MAX_BODY_BYTES) { @@ -209,27 +219,38 @@ export async function handleControlUsersRequest(request, env) { ? db.prepare(`SELECT ${columns} FROM control_users ORDER BY username LIMIT ?`).bind(body.limit) : db.prepare(`SELECT ${columns} FROM control_users WHERE username > ? ORDER BY username LIMIT ?`).bind(body.after, body.limit) ).all()).results; - const users = revisionSyncSettings(env).enabled ? await read(() => query(COLUMNS)) : await readUsers(query); + const users = revisionSyncSettings(env).enabled ? await read(() => query(streamUserFields(env).join(", "))) : await readUsers(query, env); return reply({ version: 1, users }); } case "get": { if (!fields(body, ["version", "operation", "username"]) || !text(body.username, 128)) break; const user = await readUsers(columns => db.prepare(`SELECT ${columns} FROM control_users WHERE username=?`) - .bind(body.username).first()); + .bind(body.username).first(), env); return reply({ version: 1, user: user ?? null }); } case "by_prefix": { if (!fields(body, ["version", "operation", "prefix"]) || !text(body.prefix, 64)) break; - return reply({ version: 1, users: await activeUsersByPrefix(db, body.prefix) }); + return reply({ version: 1, users: await activeUsersByPrefix(db, body.prefix, env) }); } case "upsert": { if (!fields(body, ["version", "operation", "user"]) || !validUser(body.user)) break; + if (!streamCostEnabled(env) && body.user[STREAM_CAP_FIELD] != null) return reply({ error: "invalid_request" }, 400); const user = { shadow_eval_rate: null, secret_scan_mode: null, ...body.user }; if (grantsAdmin(user) && !adminUsernames(env).has(user.username)) { logFailure("account_admin_refused", new Error("Administration is not configured for this username")); await audit(db, "upsert", "refused", user).run(); return reply({ error: "admin_not_allowed" }, 403); } + if (streamCostEnabled(env)) { + // Verify schema even when an omitted cap must be preserved. + await db.prepare("SELECT max_stream_cost_microusd FROM control_users LIMIT 1").first(); + const columns = Object.hasOwn(user, STREAM_CAP_FIELD) ? streamUserFields(env) : USER_FIELDS; + const statements = [upsertStatement(db, user, columns), audit(db, "upsert", "stored", user)]; + if (revisionSyncSettings(env).enabled) { + if (!await commitRevision(db, ["key_controls", "model_grants"], statements)) return reply({ error: "revision_conflict" }, 409); + } else await db.batch(statements); + return reply({ version: 1, stored: true }); + } // The write and its audit row commit together, or neither does. for (const end of [USER_FIELDS.length, USER_FIELDS.length - 1, USER_FIELDS.length - 2, 12]) { if (USER_FIELDS.slice(end).some(name => user[name] !== null)) throw new Error("Account controls require migration"); @@ -277,6 +298,9 @@ export async function handleControlUsersRequest(request, env) { } return reply({ error: "invalid_request" }, 400); } catch (error) { + if (streamCostEnabled(env) && missingColumn(error)) { + return reply({ error: "stream_cost_storage_unavailable" }, 503); + } logFailure("account_storage_failed", error, { operation: OPERATIONS.has(body.operation) ? body.operation : "unknown" }); return reply({ error: "storage_unavailable" }, 503); } diff --git a/worker/stream-cost-breaker.mjs b/worker/stream-cost-breaker.mjs new file mode 100644 index 00000000..80283326 --- /dev/null +++ b/worker/stream-cost-breaker.mjs @@ -0,0 +1,242 @@ +/** Opt-in stream cost enforcement for native Worker responses. */ +const CAP = "max_stream_cost_microusd"; +export const STREAM_CAP_FIELD = CAP; +const MAX_CAP = 1_000_000_000_000; +const LIMIT = 65536; +const encoder = new TextEncoder(); +const warned = new Set(); +export function streamCostEnabled(env) { + const flag = String(env?.STREAM_COST_BREAKER_ENABLED ?? "").trim().toLowerCase(); + if (["", "false", "0", "no", "off"].includes(flag)) return false; + if (["true", "1", "yes", "on"].includes(flag)) return true; + if (!warned.has("flag")) { + warned.add("flag"); + console.warn("Invalid STREAM_COST_BREAKER_ENABLED; stream cost breaker disabled"); + } + return false; +} +export const validStreamCap = cap => cap === null || (Number.isSafeInteger(cap) && cap >= 0 && cap <= MAX_CAP); +function failure(code, status = 503) { + const error = new Error("Stream cost enforcement could not admit this request"); + error.code = code; error.status = status; + return error; +} +// Decimal rates stay rational until the final micro-USD ceiling. +function decimal(value) { + if (!["number", "string"].includes(typeof value)) throw failure("stream_cost_unpriced"); + const text = String(value), match = /^(\d+)(?:\.(\d+))?(?:e([+-]?\d+))?$/i.exec(text); + if (!match || text.length > 64) throw failure("stream_cost_unpriced"); + const exponent = Number(match[3] ?? 0) - (match[2]?.length ?? 0); + if (Math.abs(exponent) > 30) throw failure("stream_cost_unpriced"); + const n = BigInt(match[1] + (match[2] ?? "")); + return exponent >= 0 ? { n: n * 10n ** BigInt(exponent), d: 1n } : { n, d: 10n ** BigInt(-exponent) }; +} +const add = (a, b) => ({ n: a.n * b.d + b.n * a.d, d: a.d * b.d }); +const mul = (a, n) => ({ n: a.n * BigInt(n), d: a.d }); +const greater = (a, b) => a.n * b.d > b.n * a.d; +const zero = { n: 0n, d: 1n }; +function priceFor(env, model) { + let table, metadata; + try { + table = JSON.parse(env.MODEL_PRICING_USD_PER_MILLION || "{}"); + metadata = JSON.parse(env.PROMPT_CACHE_PRICE_METADATA_JSON || "{}"); + } catch { throw failure("stream_cost_unpriced"); } + if (!table || Array.isArray(table) || typeof table !== "object") throw failure("stream_cost_unpriced"); + table = Object.fromEntries(Object.entries(table).map(([key, value]) => [key.trim().toLowerCase(), value])); + model = model.trim().toLowerCase(); + const provider = model.includes(":") ? model.split(":")[0] : ""; + const entry = [model, provider ? provider + ":*" : "", "*"].map(key => table[key]).find(v => v && typeof v === "object" && !Array.isArray(v)); + if (!entry) throw failure("stream_cost_unpriced"); + const names = ["input", "cache_read", "cache_write", "output"]; + const flatOnly = "request" in entry && !names.some(name => name in entry) && !("input_cost_per_million" in entry) && !("output_cost_per_million" in entry); + const aliases = { input: "input_cost_per_million", output: "output_cost_per_million" }; + const supplement = metadata?.[model] ?? {}; + const rates = names.map(name => decimal(flatOnly ? 0 : entry[name] ?? entry[aliases[name]] ?? supplement[name])); + const bound = rates.slice(0, 3).reduce((a, b) => greater(a, b) ? a : b); + return { model, rates, bound, flat: mul(decimal(entry.request ?? 0), 1_000_000) }; +} +const count = value => Number.isSafeInteger(value) && value >= 0 ? value : null; +function observation(value) { + const native = value.usage ? value : value.response?.usage ? value.response : value.message?.usage ? value.message : null; + if (!native?.usage || typeof native.usage !== "object") return null; + const usage = native.usage; + const anthropic = native.type === "message" || Boolean(value.message) || String(value.type ?? "").startsWith("message_") + || "cache_read_input_tokens" in usage || "cache_creation_input_tokens" in usage; + const raw = count(usage.prompt_tokens ?? usage.input_tokens); + const details = usage.prompt_tokens_details ?? usage.input_tokens_details ?? {}; + return { + raw, read: count(anthropic ? usage.cache_read_input_tokens : details.cached_tokens), + write: count(anthropic ? usage.cache_creation_input_tokens : details.cache_write_tokens ?? (raw !== null ? 0 : null)), + output: count(usage.completion_tokens ?? usage.output_tokens), + source: anthropic ? "anthropic" : raw !== null || Object.keys(details).length ? "openai" : "unknown", + }; +} +function merge(old, next) { + if (!old) return next; + return { raw: next.raw ?? old.raw, read: next.read ?? old.read, + write: next.source !== "unknown" ? next.write ?? old.write : old.write, + output: next.output ?? old.output, source: next.source === "unknown" ? old.source : next.source }; +} +function buckets(usage) { + if (!usage) return [null, null, null, null]; + const ordinary = usage.source === "anthropic" ? usage.raw : + usage.raw !== null && usage.read !== null && usage.write !== null ? count(usage.raw - usage.read - usage.write) : null; + return [ordinary, usage.read, usage.write, usage.output]; +} +function strings(value) { + if (typeof value === "string") return encoder.encode(value).byteLength; + if (Array.isArray(value)) return value.reduce((sum, v) => sum + strings(v), 0); + if (value && typeof value === "object") return Object.values(value).reduce((sum, v) => sum + strings(v), 0); + return 0; +} +function outputBytes(value) { + if (Array.isArray(value.choices)) return value.choices.reduce((sum, c) => sum + strings(c?.delta ?? {}), 0); + return strings(value.delta); +} +function terminal(value) { + return ["message_stop", "response.completed", "response.failed", "response.incomplete"].includes(value.type) + || (Array.isArray(value.choices) && value.choices.some(c => c?.finish_reason != null)); +} +class StreamCostState { + constructor(cap, prices, input, protocol) { + this.cap = cap; this.prices = prices; this.input = input; this.protocol = protocol; + this.runningMicrousd = 0; this.basis = "conservative_estimate"; this.exceeded = false; + this.terminal = false; this.usage = null; this.finalUsage = null; this.output = 0; this.outputAtMeasurement = 0; + this.cost(); + } + cost() { + const amounts = buckets(this.usage), additional = this.output - this.outputAtMeasurement; + let measured = this.usage !== null && additional === 0; + const costs = this.prices.map(price => { + let result = price.flat; + if (amounts.every((amount, i) => amount !== null || price.rates[i].n === 0n)) { + amounts.forEach((amount, i) => { result = add(result, mul(price.rates[i], amount ?? 0)); }); + return add(result, mul(price.rates[3], additional)); + } + measured = false; + const knownInput = amounts.slice(0, 3).reduce((sum, n) => sum + (n ?? 0), 0); + result = add(result, mul(price.bound, Math.max(this.input, knownInput))); + return add(result, mul(price.rates[3], Math.max((this.usage?.output ?? 0) + additional, this.output))); + }); + const maximum = costs.reduce((a, b) => greater(a, b) ? a : b); + this.runningMicrousd = Number((maximum.n + maximum.d - 1n) / maximum.d); + this.basis = measured ? "measured" : "conservative_estimate"; + } + observe(bytes) { + let data, value; + try { + const text = new TextDecoder("utf-8", { fatal: true }).decode(bytes); + data = text.split(/\r\n|\r|\n/).filter(line => line.startsWith("data:")).map(line => line.slice(5).replace(/^ +/, "")).join("\n"); + if (!data) return true; + if (data === "[DONE]") { + this.terminal = true; + if (this.basis === "measured") this.finalUsage = this.usage; + return true; + } + value = JSON.parse(data); + if (!value || Array.isArray(value) || typeof value !== "object") throw new Error("Invalid frame"); + } catch { this.exceeded = true; return false; } + this.output += outputBytes(value); + const found = observation(value); + if (found) { + this.usage = merge(this.usage, found); + if (found.output !== null) this.outputAtMeasurement = this.output; + } + this.cost(); + if (terminal(value)) this.terminal = true; + if (this.terminal && this.basis === "measured") this.finalUsage = this.usage; + else if (this.basis !== "measured") this.finalUsage = null; + if (this.runningMicrousd > this.cap) { this.exceeded = true; return false; } + return true; + } + errorFrame() { + const error = { code: "stream_cost_cap_exceeded", type: "stream_cost_cap_exceeded", + message: "The stream cost cap was exceeded.", cost_basis: this.basis }; + let value = { error }, prefix = ""; + if (this.protocol === "anthropic") { value = { type: "error", error }; prefix = "event: error\n"; } + if (this.protocol === "responses") { + value = { type: "response.failed", response: { object: "response", status: "failed", error } }; + prefix = "event: response.failed\n"; + } + return encoder.encode(prefix + "data: " + JSON.stringify(value) + "\n\n"); + } +} +export function prepareStreamCostBreaker(env, user, models, inputTokens, protocol = "chat") { + const cap = user?.[CAP]; + if (!streamCostEnabled(env) || cap == null) return null; + if (!validStreamCap(cap)) throw failure("stream_cost_storage_unavailable"); + if (!models.length || count(inputTokens) === null) throw failure("stream_cost_unpriced"); + const state = new StreamCostState(cap, models.map(model => priceFor(env, model)), inputTokens, protocol); + if (state.runningMicrousd > cap) throw failure("stream_cost_cap_exceeded", 429); + return state; +} +function frameEnd(bytes) { + for (let i = 0; i < bytes.length - 1; i++) { + if ((bytes[i] === 10 && bytes[i + 1] === 10) || (bytes[i] === 13 && bytes[i + 1] === 13)) return i + 2; + if (bytes[i] === 13 && bytes[i + 1] === 10 && bytes[i + 2] === 13 && bytes[i + 3] === 10) return i + 4; + } + return 0; +} +/** onFinal receives content-free state; retain the hold when finalUsage is null. */ +export function wrapStreamCost(stream, state, { abort, onFinal } = {}) { + if (!state) return stream; + const reader = stream.getReader(); + let pending = new Uint8Array(), trailer = [], trailerSize = 0, closed = false, finalized = false; + const finalize = async () => { + if (finalized) return; + finalized = true; + try { await onFinal?.(state); } catch { console.warn("Stream cost finalization failed"); } + }; + const close = async reason => { + if (closed) return; + closed = true; + try { await abort?.(); } catch { console.warn("Stream cost abort failed"); } + try { await reader.cancel(reason); } catch { console.warn("Stream cost cleanup failed"); } + finally { reader.releaseLock(); await finalize(); } + }; + return new ReadableStream({ + async pull(controller) { + try { + while (!closed) { + const end = frameEnd(pending); + if (end) { + const frame = pending.slice(0, end); pending = pending.slice(end); + if (end > LIMIT || !state.observe(frame)) { + state.exceeded = true; + if (end > LIMIT || state.runningMicrousd <= state.cap) state.finalUsage = null; + pending = new Uint8Array(); + await close("stream_cost_cap_exceeded"); controller.enqueue(state.errorFrame()); controller.close(); return; + } + if (state.terminal) { + trailer.push(frame); trailerSize += frame.length; + if (trailerSize > LIMIT) { + state.exceeded = true; state.finalUsage = null; + await close("stream_cost_cap_exceeded"); controller.enqueue(state.errorFrame()); controller.close(); return; + } + continue; + } + controller.enqueue(frame); return; + } + if (pending.length > LIMIT) { + state.exceeded = true; state.finalUsage = null; + await close("stream_cost_cap_exceeded"); controller.enqueue(state.errorFrame()); controller.close(); return; + } + const { value, done } = await reader.read(); + if (done) { + closed = true; reader.releaseLock(); + if (pending.length) { state.exceeded = true; state.finalUsage = null; controller.enqueue(state.errorFrame()); } + if (!pending.length) for (const frame of trailer) controller.enqueue(frame); + trailer = []; await finalize(); controller.close(); return; + } + const next = new Uint8Array(pending.length + value.byteLength); + next.set(pending); next.set(value, pending.length); pending = next; + } + } catch { + state.finalUsage = null; + await close("interrupted"); + controller.error(failure("stream_cost_interrupted", 502)); + } + }, + async cancel() { state.finalUsage = null; await close("cancelled"); }, + }, { highWaterMark: 0 }); +}