diff --git a/docs/usage-reservations.md b/docs/usage-reservations.md new file mode 100644 index 00000000..68265edd --- /dev/null +++ b/docs/usage-reservations.md @@ -0,0 +1,82 @@ +# Durable usage reservations + +Dollar-budgeted keys can retain conservative usage holds after uncertain provider +handoff. The default is `USAGE_RESERVATIONS_ENABLED=false`; existing accounting, +responses, headers, and storage remain unchanged. + +Enable with: + +```text +USAGE_RESERVATIONS_ENABLED=true +USAGE_HOLD_REVIEW_AFTER_SECONDS=259200 +``` + +An empty value uses the default. Invalid settings disable reservations and emit one +warning per setting, without its value. Review age must be 1–2147483647 seconds. +Review age marks an unresolved hold as `needs_review`; it never releases money. + +A reservation moves from `reserved` to `dispatched`, then to `settled` with measured +provider cost or to `unknown` when completion or usage is uncertain. Disconnects after +handoff retain the estimate. Confirmed cancellation before dispatch and cache hits +before dispatch settle at zero with a release basis. A successful response with missing +or partial billable usage retains its hold; missing counters remain null rather than +becoming measured zero. Requests without a configured monetary budget are unchanged. + +Admission and settlement use atomic storage transactions. All unresolved holds count +against both current UTC budget periods, even when created in an older period. +Settled charges belong to their admission day and month. Settlement and transition +identifiers reject conflicting replays; these identifiers grant no provider retry +permission. An estimate must fit every configured budget. Any unpriced eligible model +refuses admission with HTTP 503 `unpriced_reservation`. A priced request that exceeds +a budget returns HTTP 429 `budget_exceeded`. Holds round upwards and budget caps round downwards. Money is represented in integer units of +USD 0.0000000001 with a maximum representable amount of USD 900719.9254740991. +Measured spend can exceed the estimate; it is recorded without truncation to the budget. + +`/v1/usage` includes content-free `budget.reservations` counters: `held_usd`, `reserved`, +`dispatched`, `unknown`, and `needs_review`. `in_flight_usd` includes unresolved durable +holds, and remaining budgets subtract them. Unavailable storage reports unavailable +totals and null remaining budgets, rather than reporting spendable money. + +Unknown reservations require explicit reconciliation. A trusted administrator supplies +the reservation ID, current revision, measured cost, a provider evidence reference and +an audit reason. An explicitly authorized adjustment instead uses an adjustment basis +and audit reason. References and reasons are bounded identifiers, not provider response +bodies or free-form content. Concurrent reconciliations use compare-and-swap; one claim +wins and the other returns `reservation_conflict`. Reconciliation preserves previously +observed counters and retains transition history. No public reconciliation endpoint is +provided by the storage module; the caller must verify administrator authorization +before invoking the reconciliation contract. The private handler is not a public API. + +SQLite uses the usage database path (`USAGE_DB_PATH`) and initializes additive tables. +Keep this file on durable shared storage for processes using that authority. D1 uses +`INTELLIGENCE_DB` and the additive `0020_usage_reservations.sql` migration. Its private +JSON endpoint is `http://intelligence.internal/v1/reservations`. Container access uses +an injected `D1ReservationStore` transport; it must not fall back to ephemeral SQLite +when D1 is configured. An absent D1 binding, missing tables, or transport failure returns +HTTP 503 `usage_reservations_unavailable` before provider submission. No new binding +or public route is needed. Rehearse and apply the migration before enabling the feature. + +Request owners must persist handoff immediately before provider submission and finalize +with classified, measured accounting. Native owners use `createReservationLifecycle` +with verified identity, all eligible prices and initial legacy spend. They supply measured +cost and nullable counters through the existing flat finalization event (or its `usage` +field), plus the classified outcome and cancellation outcome. +Flask owners use `request_accounting.mark_dispatched` before submission; the existing +cancellation observer also records handoff when it binds an upstream response. Failed +finalization retains money, including a crashed reservation still in `reserved` or +`dispatched`. Review these stale records by an explicit unknown transition before +reconciliation; never assume that lack of a completion proves no provider charge. + +Before first enabling, drain existing requests and flush the legacy usage ledger across +all replicas. The first reservation snapshots existing spend for each principal and UTC +period once; subsequent charges come from the durable reservation authority, while the +usage ledger continues to report measured costs without charging them again. Use one +authority and consistent settings on every replica. Do not change the flag during +in-flight requests or disable it with unresolved holds. Enabling after mixed legacy and +durable writers requires an operator-led reconciliation of the baseline. Automatic +external invoice matching, payments, and PostgreSQL reservation storage are not provided. + +Hold and audit records contain no prompts, responses, credentials or provider bodies. +They are retained without automatic expiry or deletion, including after review age. +Their storage must be monitored and backed up. This accounting does not establish that +a provider invoice matches local prices, token estimates, or reported metering. diff --git a/intelligence-migrations/0020_usage_reservations.sql b/intelligence-migrations/0020_usage_reservations.sql new file mode 100644 index 00000000..6391d5a6 --- /dev/null +++ b/intelligence-migrations/0020_usage_reservations.sql @@ -0,0 +1,37 @@ +CREATE TABLE IF NOT EXISTS usage_reservation_budgets ( + principal TEXT NOT NULL, + period TEXT NOT NULL, + baseline_units INTEGER NOT NULL CHECK (baseline_units >= 0), + PRIMARY KEY (principal, period) +); +CREATE TABLE IF NOT EXISTS usage_reservations ( + id TEXT PRIMARY KEY, + principal TEXT NOT NULL, + day TEXT NOT NULL, + month TEXT NOT NULL, + amount_units INTEGER NOT NULL CHECK (amount_units >= 0), + charged_units INTEGER, + state TEXT NOT NULL CHECK (state IN ('reserved', 'dispatched', 'settled', 'unknown', 'reconciled')), + basis TEXT, + input_tokens INTEGER, + output_tokens INTEGER, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + handoff_at INTEGER, + revision INTEGER NOT NULL DEFAULT 0, + transition_id TEXT NOT NULL, + settlement_id TEXT UNIQUE +); +CREATE INDEX IF NOT EXISTS usage_reservations_principal ON usage_reservations (principal, state, day, month); +CREATE TABLE IF NOT EXISTS usage_reservation_transitions ( + transition_id TEXT PRIMARY KEY, + reservation_id TEXT NOT NULL REFERENCES usage_reservations(id), + previous_state TEXT, + state TEXT NOT NULL, + at INTEGER NOT NULL, + basis TEXT, + evidence TEXT, + reason TEXT, + document TEXT NOT NULL +); +CREATE INDEX IF NOT EXISTS usage_reservation_transitions_history ON usage_reservation_transitions (reservation_id, at); diff --git a/package.json b/package.json index 470abf5a..d2eb2fa9 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,7 @@ "lint": "ruff check . --select E9,F63,F7,F82", "typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming", "test": "python -m pytest -q", - "test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_deferred_tools_worker.mjs", + "test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs", "db:migrate": "python scripts/validate_sqlite_schema.py --apply", "db:validate": "python scripts/validate_sqlite_schema.py", "security:secrets": "python scripts/check_static_secrets.py", diff --git a/services/budget_service.py b/services/budget_service.py index b34adbe6..a58afc88 100644 --- a/services/budget_service.py +++ b/services/budget_service.py @@ -19,7 +19,7 @@ from datetime import datetime, timedelta, timezone from typing import Any, Mapping, Optional -from services import usage_ledger +from services import reservation_store, usage_ledger logger = logging.getLogger(__name__) @@ -199,11 +199,13 @@ def _spent(cls, state: _Principal, period: dict) -> tuple[float, float, float]: return day, month, sum(amount for _, _, amount in state.inflight.values()) @classmethod - def check_and_reserve(cls, user: Mapping[str, Any], estimate_usd: float, + def check_and_reserve(cls, user: Mapping[str, Any], estimate_usd: Optional[float], now: Optional[datetime] = None) -> BudgetDecision: daily, monthly = limits(user) if daily is None and monthly is None: return BudgetDecision(True) + if reservation_store.enabled(): + return cls._reserve_durable(user, estimate_usd, now) principal = str(user.get("username") or user.get("id") or "") period = periods(now) state = cls._state(principal) @@ -239,10 +241,86 @@ def check_and_reserve(cls, user: Mapping[str, Any], estimate_usd: float, ) @classmethod - def settle(cls, reservation: Optional[str]) -> None: + def _durable_totals(cls, user, now=None): + principal = str(user.get("username") or user.get("id") or "") + summary = reservation_store.get_store().summary(principal, now, reservation_store.review_seconds()) + if not summary["day_seeded"] or not summary["month_seeded"]: + state, period = cls._state(principal), periods(now) + cls._refresh(principal, state, period) + with cls._lock: + day, month, _ = cls._spent(state, period) + if not summary["day_seeded"]: + summary["spent_today_usd"] = day + if not summary["month_seeded"]: + summary["spent_this_month_usd"] = month + return summary + + @classmethod + def _reserve_durable(cls, user, estimate, now=None): + if estimate is None: + return BudgetDecision(False, error="unpriced_reservation", status_code=503, + message="Configure pricing before dispatching a request with a monetary budget.") + try: + summary = cls._durable_totals(user, now) + identity = uuid.uuid4().hex + daily, monthly = limits(user) + reservation_store.get_store().reserve(identity, str(user.get("username") or user.get("id") or ""), + estimate, daily, monthly, summary["spent_today_usd"], summary["spent_this_month_usd"], now) + return BudgetDecision(True, reservation=identity) + except reservation_store.ReservationError as error: + return BudgetDecision(False, error=error.code, status_code=error.status, + message=("This request exceeds the key's budget including durable holds." + if error.code == "budget_exceeded" else "The durable usage reservation could not be made.")) + except (BudgetUnavailable, OSError): + return BudgetDecision(False, error="usage_reservations_unavailable", status_code=503, + message="Durable usage reservations are unavailable.") + + @classmethod + def mark_dispatched(cls, reservation): + """Persist before provider submission; failure must prevent submission.""" + if not reservation or not reservation_store.enabled(): + return + store = reservation_store.get_store() + row = store.get(reservation) + if row["state"] == "reserved": + store.transition(reservation, row["revision"], "dispatched", transition_id=uuid.uuid4().hex) + elif row["state"] != "dispatched": + raise reservation_store.ReservationError("reservation_conflict", 409) + + @classmethod + def complete(cls, reservation, row, *, cached=False): + """Replace a hold only with measured cost; uncertain completion retains it.""" + if not reservation or not reservation_store.enabled(): + return + store = reservation_store.get_store() + current = store.get(reservation) + if current["state"] not in {"reserved", "dispatched"}: + return + if cached and current["state"] == "reserved": + cls.settle(reservation, before_dispatch=True) + return + if current["state"] == "reserved": + cls.mark_dispatched(reservation) + current = store.get(reservation) + measured = row.get("cost_usd") is not None and row.get("cost_basis") == "usage" and not cached + fields = {"input_tokens": row.get("input_tokens"), "output_tokens": row.get("output_tokens")} + if measured: + fields.update(cost_usd=row["cost_usd"], basis="provider", settlement_id=reservation) + store.transition(reservation, current["revision"], "settled" if measured else "unknown", + transition_id=uuid.uuid4().hex, **fields) + + @classmethod + def settle(cls, reservation: Optional[str], *, before_dispatch=False) -> None: """Release an in-flight estimate; the settled cost arrives through `record_cost`.""" if not reservation: return + if reservation_store.enabled(): + if before_dispatch: + store = reservation_store.get_store() + row = store.get(reservation) + store.transition(reservation, row["revision"], "settled", cost_usd=0, basis="released", + settlement_id=reservation, transition_id=uuid.uuid4().hex) + return with cls._lock: entry = cls._reservations.pop(reservation, None) if entry is None: @@ -274,17 +352,23 @@ def status(cls, user: Mapping[str, Any], now: Optional[datetime] = None) -> dict period = periods(now) state = cls._state(principal) available = True + durable = None try: - cls._refresh(principal, state, period) - except BudgetUnavailable: + if reservation_store.enabled(): + durable = cls._durable_totals(user, now) + else: + cls._refresh(principal, state, period) + except (BudgetUnavailable, reservation_store.ReservationError, OSError): available = False with cls._lock: day, month, inflight = cls._spent(state, period) + if durable is not None: + day, month, inflight = durable["spent_today_usd"], durable["spent_this_month_usd"], durable["held_usd"] def remaining(limit, spent): return None if limit is None else round(max(0.0, limit - spent - inflight), 10) - return { + result = { "totals_available": available, "day": period["day"], "month": period["month"], @@ -299,5 +383,13 @@ def remaining(limit, spent): "monthly_resets_at": period["month_resets_at"].isoformat(), } + if reservation_store.enabled(): + result["reservations"] = ({key: durable[key] for key in + ("held_usd", "reserved", "dispatched", "unknown", "needs_review")} if durable is not None + else {"error": "usage_reservations_unavailable"}) + if not available: + result["daily_remaining_usd"] = result["monthly_remaining_usd"] = None + return result + usage_ledger.LEDGER.add_listener(BudgetService.on_flushed, BudgetService.on_dropped) diff --git a/services/request_accounting.py b/services/request_accounting.py index 949c4aed..2624570d 100644 --- a/services/request_accounting.py +++ b/services/request_accounting.py @@ -15,13 +15,14 @@ import re import time from dataclasses import dataclass, field, replace +from datetime import datetime, timezone from typing import Any, Iterable, Optional from flask import Response, current_app, g, jsonify, request from werkzeug.exceptions import HTTPException from error_handlers import APIError -from services import key_controls, prompt_cache_cost, telemetry_export, usage_ledger +from services import key_controls, prompt_cache_cost, reservation_store, telemetry_export, usage_ledger from services.budget_service import BudgetService, budgeted from services.cost_service import CostService from services.usage_types import StreamUsageObserver, UsageObservation @@ -214,6 +215,19 @@ def price(models: list[str], input_tokens: Optional[int], output_tokens: Optiona return max(known) if known else None +def reservation_price(models, input_tokens, output_tokens, units): + costs = [CostService.estimate(candidate, input_tokens, output_tokens, requests=units) + for model in models for candidate in _candidates(model)] + return max(costs) if costs and all(cost is not None for cost in costs) else None + + +def mark_dispatched(): + """Injected pre-submission hook for managed request owners.""" + context = getattr(g, "usage_context", None) + if isinstance(context, UsageContext): + BudgetService.mark_dispatched(context.reservation) + + def estimate_cost(models: list[str], input_tokens: int, output_tokens: int, units: int) -> float: return price(models, input_tokens, output_tokens, units) or 0.0 @@ -257,9 +271,11 @@ def begin() -> Optional[Response]: trace=telemetry_export.trace_context(request.headers.get("traceparent"))) context.input_tokens, context.output_tokens = _token_estimates(payload) if budgeted(user): - context.estimate = estimate_cost([_qualified(model, provider) for model in models], - context.input_tokens, context.output_tokens, context.units) - decision = BudgetService.check_and_reserve(user, context.estimate) + qualified = [_qualified(model, provider) for model in models] + estimate = (reservation_price(qualified, context.input_tokens, context.output_tokens, context.units) + if reservation_store.enabled() else estimate_cost(qualified, context.input_tokens, context.output_tokens, context.units)) + context.estimate = estimate or 0.0 + decision = BudgetService.check_and_reserve(user, estimate) if not decision.allowed: return _error(decision.status_code, decision.error, decision.message, decision.retry_after, **({"budget": decision.details} if decision.details else {})) @@ -364,12 +380,29 @@ def cancellation_observer(): if not isinstance(context, UsageContext): return None + if reservation_store.enabled(): + BudgetService.mark_dispatched(context.reservation) + def observe(outcome): if outcome.ambiguous and not context.finished: context.ambiguous = True return observe +def _record_budget(context, row, budget_row=None): + if reservation_store.enabled() and context.reservation: + held = reservation_store.get_store().get(context.reservation) + # Admission owns the UTC budget period, including completion after midnight. + row = {**row, "at": datetime.fromtimestamp(held["created_at"] / 1000, timezone.utc).isoformat( + timespec="milliseconds").replace("+00:00", "Z")} + if row["cost_basis"] == "estimate": + row = {**row, "cost_usd": None, "cost_basis": None} + BudgetService.complete(context.reservation, row, cached=context.cached) + else: + BudgetService.record_cost(budget_row if budget_row is not None else row) + return row + + def _record(context: UsageContext, status: int, usage: Optional[UsageObservation], units: Optional[int]) -> None: if context.finished: return @@ -377,7 +410,10 @@ def _record(context: UsageContext, status: int, usage: Optional[UsageObservation if context.ambiguous: status, usage, units = 499, None, None if context.path in DEFERRED_PATHS: - BudgetService.settle(context.reservation) + if reservation_store.enabled(): + BudgetService.settle(context.reservation, before_dispatch=True) + else: + BudgetService.settle(context.reservation) return try: row = _row(context, status, usage, units) @@ -392,7 +428,7 @@ def _record(context: UsageContext, status: int, usage: Optional[UsageObservation if prompt_cache_cost.enabled() else price(models, estimated.input_tokens, estimated.output_tokens, context.units)) budget_row = {**row, "cost_usd": estimate} - BudgetService.record_cost(budget_row) + row = _record_budget(context, row, budget_row) usage_ledger.LEDGER.record(row) telemetry_export.EXPORTER.submit({**row, "method": context.method, "trace_id": context.trace[0], "parent_span_id": context.trace[1], "start_ns": context.start_ns, @@ -550,7 +586,9 @@ def admit_item(user: dict, model: str, units: int) -> tuple[Optional[dict], Opti "message": f"This API key is not allowed to use {str(model)[:128]}."}, None if not budgeted(user): return None, None - decision = BudgetService.check_and_reserve(user, estimate_cost([model], 0, 0, max(1, units))) + estimate = (reservation_price([model], 0, 0, max(1, units)) if reservation_store.enabled() + else estimate_cost([model], 0, 0, max(1, units))) + decision = BudgetService.check_and_reserve(user, estimate) if not decision.allowed: return {"status": decision.status_code, "code": decision.error, "message": decision.message}, None return None, decision.reservation @@ -568,7 +606,10 @@ def record_item(user: dict, *, endpoint: str, requested: str, selected: Optional def _record_row(context: UsageContext, status: int, units: int) -> None: try: row = _row(context, status, None, units) - BudgetService.record_cost(row) + if reservation_store.enabled() and context.reservation and status < 400 and row["cost_usd"] is not None: + # Flat-price media has measured units on success, even without tokens. + row = {**row, "cost_basis": "usage"} + row = _record_budget(context, row) usage_ledger.LEDGER.record(row) except Exception as error: # Accounting must never fail the work it describes. logger.warning("Usage could not be recorded (%s)", type(error).__name__) diff --git a/services/reservation_store.py b/services/reservation_store.py new file mode 100644 index 00000000..212f372e --- /dev/null +++ b/services/reservation_store.py @@ -0,0 +1,402 @@ +"""Atomic, content-free dollar reservations; uncertain handoffs never expire money.""" +from __future__ import annotations + +import json +import logging +import os +import re +import sqlite3 +import threading +import uuid +from contextlib import closing +from datetime import datetime, timezone +from decimal import Decimal, InvalidOperation, ROUND_CEILING, ROUND_FLOOR +from pathlib import Path +from typing import Any + +from services.sqlite_store import storage_path + +logger = logging.getLogger(__name__) +SCALE = 10_000_000_000 +MAX_UNITS = 2**53 - 1 +DEFAULT_REVIEW_SECONDS = 259200 +ACTIVE = ("reserved", "dispatched", "unknown") +MIGRATION = "0020_usage_reservations.sql" +_ID = re.compile(r"[0-9a-f]{32}\Z") +_LABEL = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.:-]{0,127}\Z") +_warned: set[str] = set() +_warn_lock = threading.Lock() +_store: Any = None + + +class ReservationError(Exception): + def __init__(self, code="usage_reservations_unavailable", status=503): + super().__init__(code) + self.code, self.status = code, status + + +def _warn(name): + with _warn_lock: + if name not in _warned: + _warned.add(name) + logger.warning("Invalid %s; usage reservations disabled", name) + + +def review_seconds() -> int | None: + raw = os.environ.get("USAGE_HOLD_REVIEW_AFTER_SECONDS", "").strip() + if not raw: + return DEFAULT_REVIEW_SECONDS + if len(raw) > 10 or not raw.isascii() or not raw.isdecimal() or not 1 <= int(raw) <= 2**31 - 1: + _warn("USAGE_HOLD_REVIEW_AFTER_SECONDS") + return None + return int(raw) + + +def enabled() -> bool: + raw = os.environ.get("USAGE_RESERVATIONS_ENABLED", "").strip().lower() + if raw in {"", "false", "0", "no", "off"}: + return False + if raw not in {"true", "1", "yes", "on"}: + _warn("USAGE_RESERVATIONS_ENABLED") + return False + return review_seconds() is not None + + +def units(value, *, limit=False) -> int: + if type(value) not in (int, float): + raise ReservationError("invalid_reservation_request", 400) + try: + amount = Decimal(str(value)) * SCALE + if not amount.is_finite() or not 0 <= amount <= MAX_UNITS: + raise ValueError() + return int(amount.to_integral_value(rounding=ROUND_FLOOR if limit else ROUND_CEILING)) + except (InvalidOperation, ValueError, OverflowError): + raise ReservationError("invalid_reservation_request", 400) from None + + +def _identifier(value): + if not isinstance(value, str) or not _ID.fullmatch(value): + raise ReservationError("invalid_reservation_request", 400) + return value + + +def _principal(value): + if (not isinstance(value, str) or not 1 <= len(value) <= 256 + or any(ord(character) < 32 or ord(character) == 127 for character in value)): + raise ReservationError("invalid_reservation_request", 400) + return value + + +def clock(now=None): + value = (now or datetime.now(timezone.utc)).astimezone(timezone.utc) + return int(value.timestamp() * 1000), value.strftime("%Y-%m-%d"), value.strftime("%Y-%m") + + +def public(row): + value = dict(row) + value["estimate_usd"] = value.pop("amount_units") / SCALE + charged = value.pop("charged_units") + value["cost_usd"] = charged / SCALE if charged is not None else None + return value + + +def _document(fields): + return json.dumps(fields, sort_keys=True, separators=(",", ":")) + + +def transition_fields(state, *, cost_usd=None, basis=None, input_tokens=None, output_tokens=None, + settlement_id=None, admin=False, evidence=None, authorized_adjustment=False, reason=None): + if state not in {"dispatched", "unknown", "settled", "reconciled"}: + raise ReservationError("invalid_reservation_request", 400) + for value in (input_tokens, output_tokens): + if value is not None and (type(value) is not int or not 0 <= value <= MAX_UNITS): + raise ReservationError("invalid_reservation_request", 400) + charged = None if cost_usd is None else units(cost_usd) + if state in {"settled", "reconciled"}: + _identifier(settlement_id) + if charged is None or basis not in {"provider", "released", "adjustment"}: + raise ReservationError("invalid_reservation_request", 400) + elif any(value is not None for value in (cost_usd, basis, settlement_id, evidence, reason)): + raise ReservationError("invalid_reservation_request", 400) + if state == "settled" and (basis == "adjustment" or (basis == "released" and charged != 0)): + raise ReservationError("invalid_reservation_request", 400) + if state == "reconciled": + if admin is not True: + raise ReservationError("admin_required", 403) + if not isinstance(reason, str) or not _LABEL.fullmatch(reason): + raise ReservationError("reconciliation_evidence_required", 400) + if basis == "provider": + if not isinstance(evidence, str) or not _LABEL.fullmatch(evidence): + raise ReservationError("reconciliation_evidence_required", 400) + elif basis != "adjustment" or authorized_adjustment is not True or evidence is not None: + raise ReservationError("reconciliation_evidence_required", 400) + elif evidence is not None or reason is not None or admin or authorized_adjustment: + raise ReservationError("invalid_reservation_request", 400) + return dict(charged_units=charged, basis=basis, input_tokens=input_tokens, output_tokens=output_tokens, + settlement_id=settlement_id, evidence=evidence, reason=reason) + + +def reconcile(store, identity, revision, cost_usd, *, admin, reason, evidence=None, + authorized_adjustment=False, transition_id=None): + return store.transition(identity, revision, "reconciled", cost_usd=cost_usd, + admin=admin, reason=reason, evidence=evidence, + authorized_adjustment=authorized_adjustment, + basis="adjustment" if authorized_adjustment else "provider", + transition_id=transition_id or uuid.uuid4().hex, settlement_id=identity) + + +class SqlReservationStore: + """SQLite authority; BEGIN IMMEDIATE serializes admission and terminal claims.""" + def __init__(self, path=None, *, initialize=True): + self.path = Path(path) if path is not None else storage_path("USAGE_DB_PATH", "usage.sqlite3") + if initialize: + try: + with closing(self._connect()) as db: + db.executescript((Path(__file__).resolve().parents[1] / "intelligence-migrations" / MIGRATION).read_text()) + except (sqlite3.Error, OSError): + raise ReservationError() from None + + def _connect(self): + self.path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) + existed = self.path.exists() + db = sqlite3.connect(self.path, timeout=10) + if not existed: + self.path.chmod(0o600) + db.row_factory = sqlite3.Row + db.execute("PRAGMA foreign_keys = ON") + return db + + def _run(self, operation): + try: + with closing(self._connect()) as db, db: + db.execute("BEGIN IMMEDIATE") + return operation(db) + except (sqlite3.Error, OSError): + raise ReservationError() from None + + def get(self, identity): + _identifier(identity) + def read(db): + row = db.execute("SELECT * FROM usage_reservations WHERE id = ?", (identity,)).fetchone() + if row is None: + raise ReservationError("reservation_not_found", 404) + return public(row) + return self._run(read) + + def audit(self, identity): + _identifier(identity) + return self._run(lambda db: [dict(row) for row in db.execute( + "SELECT * FROM usage_reservation_transitions WHERE reservation_id = ? ORDER BY rowid", (identity,))]) + + def reserve(self, identity, principal, estimate_usd, daily_budget_usd, monthly_budget_usd, + day_spent_usd, month_spent_usd, now=None): + _identifier(identity) + _principal(principal) + if daily_budget_usd is None and monthly_budget_usd is None: + raise ReservationError("invalid_reservation_request", 400) + if estimate_usd is None: + raise ReservationError("unpriced_reservation", 503) + amount = units(estimate_usd) + daily = None if daily_budget_usd is None else units(daily_budget_usd, limit=True) + monthly = None if monthly_budget_usd is None else units(monthly_budget_usd, limit=True) + baseline = (units(day_spent_usd), units(month_spent_usd)) + at, day, month = clock(now) + document = _document(dict(id=identity, principal=principal, amount_units=amount, day=day, month=month)) + def write(db): + existing = db.execute("SELECT * FROM usage_reservations WHERE id = ?", (identity,)).fetchone() + if existing is not None: + audit = db.execute("SELECT document FROM usage_reservation_transitions WHERE transition_id = ?", (identity,)).fetchone() + if existing["state"] != "reserved" or not audit or audit[0] != document: + raise ReservationError("reservation_conflict", 409) + return public(existing) + if db.execute("SELECT 1 FROM usage_reservation_transitions WHERE transition_id = ?", (identity,)).fetchone(): + raise ReservationError("reservation_conflict", 409) + for period, spent in zip((day, month), baseline): + db.execute("INSERT OR IGNORE INTO usage_reservation_budgets VALUES (?, ?, ?)", (principal, period, spent)) + summary = self._summary(db, principal, at, day, month, DEFAULT_REVIEW_SECONDS) + for limit, spent in ((daily, summary["day_units"]), (monthly, summary["month_units"])): + if limit is not None and (spent + summary["held_units"] >= limit + or spent + summary["held_units"] + amount > limit): + raise ReservationError("budget_exceeded", 429) + db.execute("""INSERT INTO usage_reservations + (id, principal, day, month, amount_units, state, created_at, updated_at, transition_id) + VALUES (?, ?, ?, ?, ?, 'reserved', ?, ?, ?)""", (identity, principal, day, month, amount, at, at, identity)) + db.execute("""INSERT INTO usage_reservation_transitions + (transition_id, reservation_id, state, at, document) VALUES (?, ?, 'reserved', ?, ?)""", + (identity, identity, at, document)) + return public(db.execute("SELECT * FROM usage_reservations WHERE id = ?", (identity,)).fetchone()) + return self._run(write) + + def transition(self, identity, revision, state, *, transition_id, now=None, **values): + _identifier(identity) + _identifier(transition_id) + if type(revision) is not int or not 0 <= revision <= MAX_UNITS: + raise ReservationError("invalid_reservation_request", 400) + fields = transition_fields(state, **values) + document = _document(dict(id=identity, revision=revision, state=state, **fields)) + at, _, _ = clock(now) + def write(db): + row = db.execute("SELECT * FROM usage_reservations WHERE id = ?", (identity,)).fetchone() + if row is None: + raise ReservationError("reservation_not_found", 404) + previous = db.execute("SELECT document FROM usage_reservation_transitions WHERE transition_id = ?", (transition_id,)).fetchone() + if previous: + if previous[0] != document: + raise ReservationError("reservation_conflict", 409) + return {"applied": False, "reservation": public(row)} + legal = ((row["state"] == "reserved" and state == "dispatched") + or (row["state"] == "reserved" and state == "settled" and fields["basis"] == "released") + or (row["state"] == "dispatched" and state in {"unknown", "settled"} and fields["basis"] != "released") + or (row["state"] == "unknown" and state == "reconciled")) + if row["revision"] != revision or not legal: + raise ReservationError("reservation_conflict", 409) + if fields["settlement_id"] and db.execute("SELECT 1 FROM usage_reservations WHERE settlement_id = ?", + (fields["settlement_id"],)).fetchone(): + raise ReservationError("reservation_conflict", 409) + db.execute("""UPDATE usage_reservations SET state = ?, revision = revision + 1, + charged_units = ?, basis = ?, input_tokens = COALESCE(?, input_tokens), + output_tokens = COALESCE(?, output_tokens), updated_at = ?, + handoff_at = CASE WHEN ? = 'dispatched' THEN ? ELSE handoff_at END, + transition_id = ?, settlement_id = ? WHERE id = ? AND revision = ?""", + (state, fields["charged_units"], fields["basis"], fields["input_tokens"], fields["output_tokens"], + at, state, at, transition_id, fields["settlement_id"], identity, revision)) + db.execute("""INSERT INTO usage_reservation_transitions + (transition_id, reservation_id, previous_state, state, at, basis, evidence, reason, document) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""", (transition_id, identity, row["state"], state, at, + fields["basis"], fields["evidence"], fields["reason"], document)) + return {"applied": True, "reservation": public(db.execute( + "SELECT * FROM usage_reservations WHERE id = ?", (identity,)).fetchone())} + return self._run(write) + + @staticmethod + def _summary(db, principal, at, day, month, review): + totals = dict(db.execute("""SELECT + COALESCE(SUM(CASE WHEN state IN ('reserved','dispatched','unknown') THEN amount_units ELSE 0 END),0) AS held_units, + COALESCE(SUM(CASE WHEN state IN ('settled','reconciled') AND day = ? THEN charged_units ELSE 0 END),0) AS day_units, + COALESCE(SUM(CASE WHEN state IN ('settled','reconciled') AND month = ? THEN charged_units ELSE 0 END),0) AS month_units, + COUNT(CASE WHEN state = 'reserved' THEN 1 END) AS reserved, + COUNT(CASE WHEN state = 'dispatched' THEN 1 END) AS dispatched, + COUNT(CASE WHEN state = 'unknown' THEN 1 END) AS unknown, + COUNT(CASE WHEN state IN ('reserved','dispatched','unknown') AND created_at <= ? THEN 1 END) AS needs_review + FROM usage_reservations WHERE principal = ?""", (day, month, at - review * 1000, principal)).fetchone()) + for name, period in (("day", day), ("month", month)): + base = db.execute("SELECT baseline_units FROM usage_reservation_budgets WHERE principal = ? AND period = ?", + (principal, period)).fetchone() + totals[name + "_seeded"] = base is not None + totals[name + "_units"] += base[0] if base else 0 + return totals + + def summary(self, principal, now=None, review_after=DEFAULT_REVIEW_SECONDS): + _principal(principal) + if type(review_after) is not int or not 1 <= review_after <= 2**31 - 1: + raise ReservationError("invalid_reservation_request", 400) + at, day, month = clock(now) + result = self._run(lambda db: self._summary(db, principal, at, day, month, review_after)) + for target, source in (("held_usd", "held_units"), ("spent_today_usd", "day_units"), + ("spent_this_month_usd", "month_units")): + result[target] = result.pop(source) / SCALE + return result + + +def _validate_response(operation, result): + try: + expected = {"version", "reservation"} if operation in {"get", "reserve"} else { + "transition": {"version", "reservation", "applied"}, "summary": {"version", "summary"}, + "audit": {"version", "transitions"}}[operation] + if set(result) != expected: + raise ReservationError() + if operation in {"get", "reserve", "transition"}: + row = result["reservation"] + fields = {"id", "principal", "day", "month", "estimate_usd", "cost_usd", "state", "basis", + "input_tokens", "output_tokens", "created_at", "updated_at", "handoff_at", "revision", + "transition_id", "settlement_id"} + if not isinstance(row, dict) or set(row) != fields: + raise ReservationError() + _identifier(row["id"]) + _identifier(row["transition_id"]) + _principal(row["principal"]) + units(row["estimate_usd"]) + if row["cost_usd"] is not None: + units(row["cost_usd"]) + if row["state"] not in {"reserved", "dispatched", "unknown", "settled", "reconciled"}: + raise ReservationError() + if operation == "transition" and type(result["applied"]) is not bool: + raise ReservationError() + elif operation == "summary": + summary = result["summary"] + fields = {"day_seeded", "month_seeded", "held_usd", "spent_today_usd", "spent_this_month_usd", + "reserved", "dispatched", "unknown", "needs_review"} + if not isinstance(summary, dict) or set(summary) != fields: + raise ReservationError() + for field in ("day_seeded", "month_seeded"): + if type(summary[field]) is not bool: + raise ReservationError() + for field in ("held_usd", "spent_today_usd", "spent_this_month_usd"): + units(summary[field]) + for field in ("reserved", "dispatched", "unknown", "needs_review"): + if type(summary[field]) is not int or summary[field] < 0: + raise ReservationError() + elif not isinstance(result["transitions"], list) or len(result["transitions"]) > 100: + raise ReservationError() + except (ReservationError, KeyError, TypeError, ValueError): + raise ReservationError() from None + + +class D1ReservationStore: + """Injected private transport; no retry or alternate local authority on failure.""" + def __init__(self, call): + self.call = call + + def _call(self, operation, **values): + try: + result = self.call({"version": 1, "operation": operation, **values}) + if not isinstance(result, dict) or result.get("version") != 1: + raise ReservationError() + if "error" in result: + code = result["error"].get("code") + statuses = {"budget_exceeded": 429, "reservation_conflict": 409, "admin_required": 403, + "reconciliation_evidence_required": 400, "invalid_reservation_request": 400, + "reservation_not_found": 404, "unpriced_reservation": 503} + raise ReservationError(code if code in statuses else "usage_reservations_unavailable", statuses.get(code, 503)) + _validate_response(operation, result) + return result + except ReservationError: + raise + except Exception as error: + code = getattr(error, "code", None) + statuses = {"budget_exceeded": 429, "reservation_conflict": 409, "admin_required": 403, + "reconciliation_evidence_required": 400, "invalid_reservation_request": 400, + "reservation_not_found": 404, "unpriced_reservation": 503} + raise ReservationError(code if code in statuses else "usage_reservations_unavailable", statuses.get(code, 503)) from None + + def reserve(self, identity, principal, estimate_usd, daily_budget_usd, monthly_budget_usd, + day_spent_usd, month_spent_usd, now=None): + return self._call("reserve", id=identity, principal=principal, estimate_usd=estimate_usd, + daily_budget_usd=daily_budget_usd, monthly_budget_usd=monthly_budget_usd, + day_spent_usd=day_spent_usd, month_spent_usd=month_spent_usd)["reservation"] + + def get(self, identity): + return self._call("get", id=identity)["reservation"] + + def transition(self, identity, revision, state, *, transition_id, now=None, **values): + result = self._call("transition", id=identity, revision=revision, state=state, transition_id=transition_id, **values) + return {"applied": result["applied"], "reservation": result["reservation"]} + + def summary(self, principal, now=None, review_after=DEFAULT_REVIEW_SECONDS): + return self._call("summary", principal=principal)["summary"] + + def audit(self, identity): + return self._call("audit", id=identity)["transitions"] + + +def configure_store(store): + global _store + _store = store + + +def get_store(): + if _store is not None: + return _store + if os.environ.get("INTELLIGENCE_STORAGE_BACKEND", "").strip() != "": + raise ReservationError() + return SqlReservationStore() diff --git a/tests/fixtures/secret_firewall_coverage.json b/tests/fixtures/secret_firewall_coverage.json index d918327c..dac82242 100644 --- a/tests/fixtures/secret_firewall_coverage.json +++ b/tests/fixtures/secret_firewall_coverage.json @@ -303,6 +303,7 @@ "worker/prometheus.mjs", "worker/prompt-templates-d1.mjs", "worker/request-telemetry.mjs", + "worker/reservations-d1.mjs", "worker/retention-policy.mjs", "worker/roleplay/candidate-context.mjs", "worker/roleplay/capacity.mjs", diff --git a/tests/test_usage_reservations.py b/tests/test_usage_reservations.py new file mode 100644 index 00000000..468cbac2 --- /dev/null +++ b/tests/test_usage_reservations.py @@ -0,0 +1,356 @@ +"""Durable dollar holds and accounting preserve uncertain observations.""" +import json +import sqlite3 +from concurrent.futures import ThreadPoolExecutor +from datetime import datetime, timedelta, timezone +from pathlib import Path +from unittest.mock import patch + +import pytest +from flask import Flask, g + +from services import budget_service, request_accounting as accounting, usage_ledger +from services import reservation_store as reservations + +MIGRATION = "0020_usage_reservations.sql" +NOW = datetime(2026, 10, 9, tzinfo=timezone.utc) +USER = {"username": "alice", "daily_budget_usd": 1.0, "monthly_budget_usd": 5.0} + + +@pytest.fixture +def store(tmp_path, monkeypatch): + monkeypatch.setenv("USAGE_RESERVATIONS_ENABLED", "true") + monkeypatch.setenv("USAGE_HOLD_REVIEW_AFTER_SECONDS", "259200") + monkeypatch.setenv("USAGE_LEDGER_ENABLED", "false") + monkeypatch.setenv("INTELLIGENCE_STORAGE_BACKEND", "") + monkeypatch.setenv("CONTROL_PLANE_DATABASE_URL", "") + monkeypatch.setenv("PROMPT_CACHE_USAGE_BUCKETS_ENABLED", "false") + monkeypatch.setenv("MODEL_PRICING_USD_PER_MILLION", json.dumps({"priced": {"input": 1000, "output": 1000}})) + instance = reservations.SqlReservationStore(tmp_path / "usage.sqlite3") + monkeypatch.setattr(reservations, "_store", instance) + monkeypatch.setattr(usage_ledger.LEDGER, "record", lambda row: None) + monkeypatch.setattr(accounting.telemetry_export.EXPORTER, "submit", lambda row: None) + budget_service.BudgetService.reset() + yield instance + budget_service.BudgetService.reset() + + +def reserve(store, identity="a" * 32, estimate=0.6): + return store.reserve(identity, USER["username"], estimate, 1.0, 5.0, 0.0, 0.0, NOW) + + +def move(store, row, state, **fields): + return store.transition(row["id"], row["revision"], state, transition_id=fields.pop("transition_id", "b" * 32), + now=NOW, **fields)["reservation"] + + +def test_disconnect_keeps_hold_across_restart_and_review(store): + row = reserve(store) + row = move(store, row, "dispatched", transition_id="c" * 32) + row = move(store, row, "unknown", input_tokens=0) + reopened = reservations.SqlReservationStore(store.path) + status = reopened.summary("alice", NOW + timedelta(days=40), 259200) + assert status["held_usd"] == 0.6 and status["needs_review"] == 1 + assert reopened.get(row["id"])["output_tokens"] is None + assert reopened.get(row["id"])["input_tokens"] == 0 + with pytest.raises(reservations.ReservationError, match="budget_exceeded"): + reserve(reopened, "d" * 32, 0.5) + + +def test_before_dispatch_release_and_terminal_exactly_once(store): + row = move(store, reserve(store), "settled", cost_usd=0, basis="released", settlement_id="e" * 32) + assert store.summary("alice", NOW)["held_usd"] == 0 + with pytest.raises(reservations.ReservationError, match="reservation_conflict"): + move(store, row, "dispatched", transition_id="f" * 32) + + +def test_measured_settlement_is_idempotent_and_not_double_charged(store): + row = move(store, reserve(store), "dispatched", transition_id="c" * 32) + values = dict(transition_id="d" * 32, settlement_id="e" * 32, cost_usd=0.2, basis="provider", + input_tokens=100, output_tokens=100, now=NOW) + first = store.transition(row["id"], 1, "settled", **values) + assert first["applied"] + assert not store.transition(row["id"], 1, "settled", **values)["applied"] + with pytest.raises(reservations.ReservationError, match="reservation_conflict"): + store.transition(row["id"], 1, "settled", **{**values, "cost_usd": 0.1}) + summary = store.summary("alice", NOW) + assert summary["spent_today_usd"] == 0.2 and summary["held_usd"] == 0 + assert len(store.audit(row["id"])) == 3 + + +def test_reconcile_requires_evidence_admin_reason_and_cas(store): + row = move(store, reserve(store), "dispatched", transition_id="c" * 32) + row = move(store, row, "unknown") + with pytest.raises(reservations.ReservationError, match="admin_required"): + reservations.reconcile(store, row["id"], 2, 0.2, admin=False, evidence="receipt_1", reason="provider_receipt") + with pytest.raises(reservations.ReservationError, match="reconciliation_evidence_required"): + reservations.reconcile(store, row["id"], 2, 0.2, admin=True, reason="provider_receipt") + def claim(index): + try: + return reservations.reconcile(store, row["id"], 2, 0.2, admin=True, evidence="receipt_1", + reason="provider_receipt", transition_id=f"{index:032x}")["applied"] + except reservations.ReservationError as error: + assert error.code == "reservation_conflict" + return False + with ThreadPoolExecutor(max_workers=4) as executor: + assert sum(executor.map(claim, range(10, 18))) == 1 + assert store.summary("alice", NOW)["spent_today_usd"] == 0.2 + + +def test_authorized_adjustment_retains_audit_and_unknown_tokens(store): + row = move(store, reserve(store), "dispatched", transition_id="c" * 32) + row = move(store, row, "unknown", input_tokens=0) + result = reservations.reconcile(store, row["id"], 2, 0.1, admin=True, authorized_adjustment=True, + reason="approved_adjustment") + assert result["reservation"]["state"] == "reconciled" + assert result["reservation"]["input_tokens"] == 0 + assert result["reservation"]["output_tokens"] is None + assert store.audit(row["id"])[-1]["basis"] == "adjustment" + + +def test_concurrent_reserves_enforce_shared_budget(store): + def admit(index): + try: + reserve(store, f"{index:032x}", 0.4) + return True + except reservations.ReservationError as error: + assert error.code == "budget_exceeded" + return False + with ThreadPoolExecutor(max_workers=6) as executor: + assert sum(executor.map(admit, range(1, 15))) == 2 + + +def test_budget_admission_and_status_use_durable_authority(store): + service = budget_service.BudgetService + first = service.check_and_reserve(USER, 0.6, NOW) + assert first.allowed + service.mark_dispatched(first.reservation) + service.complete(first.reservation, {"cost_usd": None, "cost_basis": None, "input_tokens": 0, "output_tokens": None}) + service.settle(first.reservation) + service.reset() + status = service.status(USER, NOW) + assert status["in_flight_usd"] == 0.6 and status["reservations"]["unknown"] == 1 + assert not service.check_and_reserve(USER, 0.5, NOW).allowed + refused = service.check_and_reserve(USER, None, NOW) + assert refused.status_code == 503 and refused.error == "unpriced_reservation" + assert service.check_and_reserve({"username": "unlimited"}, None, NOW).allowed + + +def test_legacy_baseline_is_saved_once_and_new_measured_cost_is_counted_once(store, monkeypatch): + service = budget_service.BudgetService + monkeypatch.setenv("USAGE_LEDGER_ENABLED", "true") + class Totals: + def totals(self, *args): + return {"day_usd": 0.3, "month_usd": 0.5} + monkeypatch.setattr(usage_ledger.LEDGER, "store", Totals) + first = service.check_and_reserve(USER, 0.6, NOW) + service.complete(first.reservation, {"cost_usd": 0.2, "cost_basis": "usage", "input_tokens": 100, "output_tokens": 100}) + service.reset() + assert service.status(USER, NOW)["spent_today_usd"] == 0.5 + assert service.check_and_reserve(USER, 0.5, NOW).allowed + + +@pytest.mark.parametrize("flag", ["", "false", "malformed-value"]) +def test_disabled_never_touches_storage_or_changes_legacy_status(store, monkeypatch, flag): + monkeypatch.setenv("USAGE_RESERVATIONS_ENABLED", flag) + monkeypatch.setattr(reservations, "get_store", lambda: pytest.fail("disabled storage access")) + service = budget_service.BudgetService + first = service.check_and_reserve(USER, 0.2, NOW) + service.settle(first.reservation) + assert "reservations" not in service.status(USER, NOW) + + +@pytest.mark.parametrize("value,enabled", [("", True), ("bad", False), ("0", False), ("-1", False)]) +def test_review_config_is_default_or_fail_safe(store, monkeypatch, value, enabled): + monkeypatch.setenv("USAGE_HOLD_REVIEW_AFTER_SECONDS", value) + assert reservations.enabled() is enabled + + +def test_missing_schema_fails_closed_and_additive_migration_keeps_old_rows(store, monkeypatch, tmp_path): + missing = reservations.SqlReservationStore(tmp_path / "missing.sqlite3", initialize=False) + monkeypatch.setattr(reservations, "_store", missing) + decision = budget_service.BudgetService.check_and_reserve(USER, 0.2, NOW) + assert decision.status_code == 503 and decision.error == "usage_reservations_unavailable" + with sqlite3.connect(store.path) as db: + db.execute("CREATE TABLE old_rows (value TEXT)") + db.execute("INSERT INTO old_rows VALUES ('keep')") + db.executescript((Path(__file__).parents[1] / "intelligence-migrations" / MIGRATION).read_text()) + assert db.execute("SELECT value FROM old_rows").fetchone()[0] == "keep" + + +def test_accounting_unknown_is_not_converted_to_zero_or_an_estimate(store): + app = Flask(__name__) + with app.test_request_context("/v1/chat/completions", method="POST", json={"model": "priced", "messages": [], "max_tokens": 100}): + g.authenticated_user = USER + assert accounting.begin() is None + ctx = g.usage_context + observer = accounting.cancellation_observer() + from services.request_cancellation import CancellationContext + owner = CancellationContext(lambda: None, on_outcome=observer) + owner.handoff() + owner.cancel() + rows = [] + with patch.object(usage_ledger.LEDGER, "record", rows.append): + accounting.finish(app.response_class(b"unchanged", status=200)) + row = store.get(ctx.reservation) + assert row["state"] == "unknown" and row["cost_usd"] is None + assert row["input_tokens"] is None and row["output_tokens"] is None + assert rows[0]["cost_usd"] is None + + +def test_registered_accounting_boundary_preserves_body_and_unpriced_fails_before_view(store): + app = Flask(__name__) + calls = [] + @app.post("/v1/chat/completions") + def chat(): + g.authenticated_user = USER + refused = accounting.begin() + if refused is not None: + return refused + calls.append(True) + return accounting.finish(app.response_class(b'{"usage":{"input_tokens":10,"output_tokens":20}}', mimetype="application/json")) + client = app.test_client() + response = client.post("/v1/chat/completions", json={"model": "priced", "max_tokens": 100}) + assert response.data == b'{"usage":{"input_tokens":10,"output_tokens":20}}' + assert budget_service.BudgetService.status(USER)["spent_today_usd"] == 0.03 + response = client.post("/v1/chat/completions", json={"model": "unpriced"}) + assert response.status_code == 503 and response.json["error"] == "unpriced_reservation" + assert len(calls) == 1 + + +def test_failed_finalization_retains_reservation_and_cached_completion_releases_before_handoff(store, monkeypatch): + service = budget_service.BudgetService + first = service.check_and_reserve(USER, 0.5, NOW) + service.complete(first.reservation, {"cost_usd": 0, "cost_basis": "cache"}, cached=True) + assert store.get(first.reservation)["basis"] == "released" + second = service.check_and_reserve(USER, 0.5, NOW) + monkeypatch.setattr(store, "transition", lambda *args, **kw: (_ for _ in ()).throw(reservations.ReservationError())) + with pytest.raises(reservations.ReservationError): + service.complete(second.reservation, {"cost_usd": 0.2, "cost_basis": "usage"}) + service.settle(second.reservation) + assert store.get(second.reservation)["state"] == "reserved" + assert service.status(USER, NOW)["in_flight_usd"] == 0.5 + + +def test_estimated_success_keeps_unknown_cost_and_admission_period_in_ledger(store, monkeypatch): + ctx = accounting.UsageContext("chat", ["priced"], None, USER, 0, 0, input_tokens=10, output_tokens=20) + ctx.path = "/v1/chat/completions" + ctx.reservation = budget_service.BudgetService.check_and_reserve(USER, 0.03, NOW).reservation + rows = [] + monkeypatch.setattr(usage_ledger.LEDGER, "record", rows.append) + accounting._record(ctx, 200, None, None) + assert rows[0]["at"] == "2026-10-09T00:00:00.000Z" + assert rows[0]["cost_usd"] is None and rows[0]["cost_basis"] is None + assert store.get(ctx.reservation)["state"] == "unknown" + + +def test_pricing_refuses_any_unpriced_auto_candidate(store, monkeypatch): + monkeypatch.setattr(accounting, "_candidates", lambda model: ["priced", "unpriced"]) + assert accounting.reservation_price(["auto:chat"], 10, 20, 1) is None + + +def test_period_rollover_preserves_unknown_hold_and_settles_original_period(store): + row = reserve(store) + row = move(store, row, "dispatched", transition_id="c" * 32) + later = NOW + timedelta(days=31) + assert store.summary("alice", later)["held_usd"] == 0.6 + store.transition(row["id"], 1, "settled", transition_id="f" * 32, settlement_id="e" * 32, + cost_usd=0.2, basis="provider", now=later) + assert store.summary("alice", later)["spent_this_month_usd"] == 0 + assert store.summary("alice", NOW)["spent_this_month_usd"] == 0.2 + + +def test_identifiers_monetary_limits_and_settlement_claims_are_bounded(store): + for estimate in [-1, float("nan"), float("inf"), True, "1"]: + with pytest.raises(reservations.ReservationError, match="invalid_reservation_request"): + reserve(store, estimate=estimate) + first = move(store, reserve(store, estimate=0.3), "dispatched", transition_id="c" * 32) + first = move(store, first, "settled", transition_id="f" * 32, settlement_id="e" * 32, + cost_usd=0.2, basis="provider") + second = reserve(store, "1" * 32, 0.3) + second = move(store, second, "dispatched", transition_id="2" * 32) + with pytest.raises(reservations.ReservationError, match="reservation_conflict"): + move(store, second, "settled", transition_id="3" * 32, settlement_id="e" * 32, cost_usd=0.2, basis="provider") + assert store.get(second["id"])["state"] == "dispatched" + + +from tests.unified_api_test_case import UnifiedApiTestCase + + +class DurableUsageRoutesTest(UnifiedApiTestCase): + def setUp(self): + self.env_patch = patch("config.load_runtime_env") + self.http_patch = patch("requests.sessions.Session.request", side_effect=AssertionError("unexpected HTTP")) + self.env_patch.start() + self.http_patch.start() + self.addCleanup(self.env_patch.stop) + self.addCleanup(self.http_patch.stop) + super().setUp() + import os + os.environ.update({"USAGE_RESERVATIONS_ENABLED": "true", "USAGE_HOLD_REVIEW_AFTER_SECONDS": "259200", + "USAGE_LEDGER_ENABLED": "false", "INTELLIGENCE_STORAGE_BACKEND": "", "CONTROL_PLANE_DATABASE_URL": "", + "PROMPT_CACHE_USAGE_BUCKETS_ENABLED": "false", "USAGE_DB_PATH": str(Path(self.temp_dir.name) / "usage.sqlite3"), + "MODEL_PRICING_USD_PER_MILLION": json.dumps({"opencode:*": {"input": 1000, "output": 1000}})}) + self.store = accounting.reservation_store.SqlReservationStore() + self.store_patch = patch.object(accounting.reservation_store, "_store", self.store) + self.store_patch.start() + self.addCleanup(self.store_patch.stop) + accounting.BudgetService.reset() + self.addCleanup(accounting.BudgetService.reset) + self.rows = [] + self.ledger_patch = patch.object(accounting.usage_ledger.LEDGER, "record", self.rows.append) + self.ledger_patch.start() + self.addCleanup(self.ledger_patch.stop) + with patch.object(self.app_module.AuthService, "get_current_user", return_value={"username": "admin", "is_admin": True}): + self.key = self.app_module.AuthService.create_user("bounded", scopes=["chat", "models"])["api_key"] + self.app_module.AuthService.set_key_controls("bounded", {"daily_budget_usd": 1.0, "monthly_budget_usd": 5.0}) + self.headers = {"Authorization": f"Bearer {self.key}"} + + def test_real_chat_and_usage_paths_show_durable_settlement_without_body_changes(self): + upstream = self._chat_response("ok") + upstream._content = b'{"choices":[],"usage":{"input_tokens":10,"output_tokens":20}}' + with patch.object(self.app_module.ProxyService, "make_request", return_value=upstream): + response = self.client.post("/v1/chat/completions", headers=self.headers, + json={"model": "opencode:test", "messages": [{"role": "user", "content": "hi"}], "max_tokens": 100}) + assert response.status_code == 200 and response.data == upstream.content + status = self.client.get("/v1/usage", headers=self.headers) + assert status.status_code == 200 + assert status.json["budget"]["spent_today_usd"] == 0.03 + assert status.json["budget"]["reservations"]["held_usd"] == 0 + + def test_missing_durable_table_refuses_chat_before_upstream(self): + missing = reservations.SqlReservationStore(Path(self.temp_dir.name) / "missing.sqlite3", initialize=False) + with patch.object(accounting.reservation_store, "_store", missing), patch.object(self.app_module.ProxyService, "make_request") as upstream: + response = self.client.post("/v1/chat/completions", headers=self.headers, + json={"model": "opencode:test", "messages": [], "max_tokens": 100}) + assert response.status_code == 503 and response.json["error"] == "usage_reservations_unavailable" + upstream.assert_not_called() + + +def test_malformed_private_transport_response_and_bounded_error_fail_closed(store): + for body in [{"version": 1}, {"version": 1, "reservation": {}}, {"version": 1, "summary": {}}]: + remote = reservations.D1ReservationStore(lambda request: body) + with pytest.raises(reservations.ReservationError): + remote.get("a" * 32) + class RemoteError(Exception): + code = "budget_exceeded" + def unavailable(request): + raise RemoteError() + with pytest.raises(reservations.ReservationError) as failure: + reservations.D1ReservationStore(unavailable).get("a" * 32) + assert failure.value.status == 429 and failure.value.code == "budget_exceeded" + + +def test_extremely_long_review_setting_disables_without_exposing_value(store, monkeypatch, caplog): + reservations._warned.clear() + value = "9" * 5000 + monkeypatch.setenv("USAGE_HOLD_REVIEW_AFTER_SECONDS", value) + assert not reservations.enabled() and not reservations.enabled() + assert value not in caplog.text + assert sum("USAGE_HOLD_REVIEW_AFTER_SECONDS" in message for message in caplog.messages) == 1 + + +def test_fractional_budget_does_not_round_up_a_hold_above_the_configured_limit(store): + with pytest.raises(reservations.ReservationError, match="budget_exceeded"): + store.reserve("a" * 32, "alice", 0.00000000001, 0.00000000001, None, 0, 0, NOW) + assert store.summary("alice", NOW)["held_usd"] == 0 diff --git a/tests/test_usage_reservations_worker.mjs b/tests/test_usage_reservations_worker.mjs new file mode 100644 index 00000000..d3f832d4 --- /dev/null +++ b/tests/test_usage_reservations_worker.mjs @@ -0,0 +1,245 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { readFileSync } from "node:fs"; +import { DatabaseSync } from "node:sqlite"; +import { spawnSync } from "node:child_process"; +import { handleReservationsRequest, createReservationLifecycle } from "../worker/reservations-d1.mjs"; + +const MIGRATION = "0020_usage_reservations.sql"; +const NOW = Date.UTC(2026, 9, 9); +const id = n => n.toString(16).padStart(32, "0"); +// A local SQLite-backed D1 contract fake exercises the actual SQL without workerd. +function database(t, schema = true) { + const sql = new DatabaseSync(":memory:"); + t.after(() => sql.close()); + if (schema) sql.exec(readFileSync(new URL(`../intelligence-migrations/${MIGRATION}`, import.meta.url), "utf8")); + const db = { prepare(query) { + let values = []; + const statement = { bind(...args) { values = args; return statement; }, + run() { const result = sql.prepare(query).run(...values); return { success: true, meta: { changes: Number(result.changes) } }; }, + all() { + const prepared = sql.prepare(query); + if (!prepared.columns().length) { + const result = prepared.run(...values); + return { success: true, results: [], meta: { changes: Number(result.changes) } }; + } + return { success: true, results: prepared.all(...values), meta: { changes: 0 } }; + }, + first() { return sql.prepare(query).get(...values) ?? null; } }; + return statement; + }, async batch(statements) { + sql.exec("BEGIN IMMEDIATE"); + try { + const results = statements.map(s => s.all()); + sql.exec("COMMIT"); + return results; + } catch (error) { sql.exec("ROLLBACK"); throw error; } + } }; + const env = { USAGE_RESERVATIONS_ENABLED: "true", INTELLIGENCE_DB: db }; + const call = async body => { + const response = await handleReservationsRequest(new Request("http://intelligence.internal/v1/reservations", { + method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ version: 1, ...body }), + }), env, { now: () => NOW }); + return { status: response.status, ...await response.json() }; + }; + return { call, db, env, sql }; +} +const reserve = (n = 1, changes = {}) => ({ operation: "reserve", id: id(n), principal: "alice", estimate_usd: 0.6, + daily_budget_usd: 1, monthly_budget_usd: 5, day_spent_usd: 0, month_spent_usd: 0, ...changes }); +const transition = (state, revision, n, changes = {}) => ({ operation: "transition", id: id(1), revision, + state, transition_id: id(n), ...changes }); + +test("unknown handoff holds survive review expiry; unknown fields remain null", async t => { + const { call } = database(t); + assert.equal((await call(reserve())).status, 200); + assert.equal((await call(transition("dispatched", 0, 2))).status, 200); + assert.equal((await call(transition("unknown", 1, 3, { input_tokens: 0 }))).status, 200); + const row = (await call({ operation: "get", id: id(1) })).reservation; + assert.equal(row.input_tokens, 0); + assert.equal(row.output_tokens, null); + assert.equal((await call(reserve(4, { estimate_usd: 0.5 }))).status, 429); + const summary = (await call({ operation: "summary", principal: "alice" })).summary; + assert.equal(summary.held_usd, 0.6); +}); + +test("pre-dispatch release, idempotent measured settlement and CAS reconciliation", async t => { + const { call } = database(t); + await call(reserve()); + await call(transition("dispatched", 0, 2)); + const measured = transition("settled", 1, 3, { cost_usd: 0.2, basis: "provider", settlement_id: id(4), input_tokens: 100, output_tokens: 100 }); + assert.equal((await call(measured)).applied, true); + assert.equal((await call(measured)).applied, false); + assert.equal((await call({ ...measured, cost_usd: 0.1 })).status, 409); + assert.equal((await call({ operation: "summary", principal: "alice" })).summary.spent_today_usd, 0.2); + await call(reserve(5, { estimate_usd: 0.4 })); + const release = { ...transition("settled", 0, 6, { cost_usd: 0, basis: "released", settlement_id: id(7) }), id: id(5) }; + assert.equal((await call(release)).status, 200); + assert.equal((await call({ ...transition("dispatched", 1, 8), id: id(5) })).status, 409); +}); + +test("reconciliation requires verified admin evidence and one claim wins", async t => { + const { call } = database(t); + await call(reserve()); + await call(transition("dispatched", 0, 2)); + await call(transition("unknown", 1, 3)); + const reconciliation = transition("reconciled", 2, 4, { cost_usd: 0.1, basis: "provider", settlement_id: id(5), + admin: true, evidence: "receipt_1", reason: "provider_receipt" }); + assert.equal((await call({ ...reconciliation, admin: false })).status, 403); + assert.equal((await call({ ...reconciliation, evidence: null })).status, 400); + const results = await Promise.all([call(reconciliation), call({ ...reconciliation, transition_id: id(6), settlement_id: id(7) })]); + assert.equal(results.filter(r => r.applied).length, 1); + assert.equal(results.filter(r => r.status === 409).length, 1); +}); + +test("off and invalid settings do not touch D1; missing table fails with a bounded 503", async t => { + const { call, env } = database(t, false); + assert.equal((await call(reserve())).status, 503); + for (const flag of [undefined, "", "false", "invalid"]) { + const response = await handleReservationsRequest(new Request("http://intelligence.internal/v1/reservations", { method: "POST" }), + { USAGE_RESERVATIONS_ENABLED: flag, INTELLIGENCE_DB: { prepare() { assert.fail("disabled storage"); } } }); + assert.equal(response.status, 404); + } + const hook = createReservationLifecycle(env, { call: () => assert.fail("disabled hook"), identity: () => assert.fail("disabled identity") }); + env.USAGE_RESERVATIONS_ENABLED = "false"; + await hook.admit({}); await hook.before_dispatch({}); await hook.finalize({}); +}); + +test("private RPC rejects content, malformed money and unpriced bounded admissions", async t => { + const { call, env } = database(t); + for (const changes of [{ prompt: "secret" }, { estimate_usd: -1 }, { daily_budget_usd: null, monthly_budget_usd: null }]) { + assert.equal((await call(reserve(1, changes))).status, 400); + } + assert.equal((await call(reserve(1, { estimate_usd: null }))).error.code, "unpriced_reservation"); + assert.equal((await call(reserve(1, { estimate_usd: 2 }))).status, 429); + const response = await handleReservationsRequest(new Request("https://example.test/v1/reservations", { method: "POST" }), env); + assert.equal(response.status, 404); +}); + + +test("review threshold never changes held money or history and baselines seed once", async t => { + const { call, db } = database(t); + await call(reserve(1, { day_spent_usd: 0.1, month_spent_usd: 0.3 })); + await call(transition("dispatched", 0, 2)); + await call(transition("unknown", 1, 3)); + const { reservationSummary } = await import("../worker/reservations-d1.mjs"); + const later = await reservationSummary(db, "alice", NOW + 40 * 86400_000, 259200); + assert.equal(later.held_usd, 0.6); + assert.equal(later.needs_review, 1); + assert.equal((await call({ operation: "audit", id: id(1) })).transitions.length, 3); + assert.equal((await call(reserve(4, { estimate_usd: 0.3, day_spent_usd: 0.9 }))).status, 200); + assert.equal((await call({ operation: "summary", principal: "alice" })).summary.spent_today_usd, 0.1); +}); + +test("native request lifecycle owns admission, handoff, settlement and no pre-dispatch charge", async t => { + const { call, env } = database(t); + const hooks = createReservationLifecycle(env, { call, identity: () => { + const { operation, id: ignored, ...values } = reserve(); + return values; + } }); + await hooks.admit({}); + await hooks.before_dispatch({}); + await hooks.finalize({ usage: { cost_usd: 0.2, cost_basis: "usage", input_tokens: 100, output_tokens: 100 } }); + await hooks.finalize({ usage: { cost_usd: 0.2, cost_basis: "usage" } }); + assert.equal((await call({ operation: "summary", principal: "alice" })).summary.spent_today_usd, 0.2); + const release = createReservationLifecycle(env, { call, identity: () => { + const { operation, id: ignored, ...values } = reserve(2, { estimate_usd: 0.4 }); + return values; + } }); + await release.admit({}); + await release.finalize({ outcome: "canceled" }); + assert.equal((await call({ operation: "summary", principal: "alice" })).summary.held_usd, 0); +}); + +test("native cancellation discards apparent usage and retains the estimate exactly once", async t => { + const { call, env } = database(t); + const hooks = createReservationLifecycle(env, { call, identity: () => { + const { operation, id: ignored, ...values } = reserve(); + return values; + } }); + await hooks.admit({}); await hooks.before_dispatch({}); + const event = { cancellationOutcome: { ambiguous: true }, usage: { cost_usd: 0, cost_basis: "usage", input_tokens: 0, output_tokens: 0 } }; + await hooks.finalize(event); await hooks.finalize(event); + const summary = (await call({ operation: "summary", principal: "alice" })).summary; + assert.equal(summary.held_usd, 0.6); assert.equal(summary.unknown, 1); +}); + +test("claim identifier collisions cannot commit an unaudited terminal update", async t => { + const { call } = database(t); + await call(reserve(1, { estimate_usd: 0.3 })); + await call(reserve(4, { estimate_usd: 0.3 })); + await call(transition("dispatched", 0, 2)); + await call({ ...transition("dispatched", 0, 5), id: id(4) }); + const first = transition("settled", 1, 6, { cost_usd: 0.1, basis: "provider", settlement_id: id(7) }); + const second = { ...first, id: id(4), settlement_id: id(8) }; + const results = await Promise.all([call(first), call(second)]); + assert.equal(results.filter(result => result.status === 200).length, 1); + assert.equal(results.filter(result => result.status === 409).length, 1); + const rows = [(await call({ operation: "get", id: id(1) })).reservation, (await call({ operation: "get", id: id(4) })).reservation]; + assert.equal(rows.filter(row => row.state === "settled").length, 1); +}); + + +test("SQLite and the D1 contract return identical reservation and summary shapes", async t => { + const { call } = database(t); + await call(reserve()); + await call(transition("dispatched", 0, 2)); + await call(transition("unknown", 1, 3, { input_tokens: 0 })); + const observed = [(await call({ operation: "get", id: id(1) })).reservation, + (await call({ operation: "summary", principal: "alice" })).summary]; + const code = `import os,sys,json,tempfile +from pathlib import Path +from datetime import datetime,timezone +sys.path.insert(0,os.getcwd()) +from services.reservation_store import SqlReservationStore +with tempfile.TemporaryDirectory() as directory: + store=SqlReservationStore(Path(directory)/'usage.sqlite3') + now=datetime(2026,10,9,tzinfo=timezone.utc) + identity='${id(1)}' + store.reserve(identity,'alice',0.6,1,5,0,0,now) + store.transition(identity,0,'dispatched',transition_id='${id(2)}',now=now) + store.transition(identity,1,'unknown',transition_id='${id(3)}',input_tokens=0,now=now) + print(json.dumps([store.get(identity),store.summary('alice',now)]))`; + const python = spawnSync("/home/mysterious/storage/github/MultiLLM-Proxy/.venv/bin/python", ["-I", "-c", code], { + cwd: new URL("..", import.meta.url), encoding: "utf8", timeout: 10000, + }); + assert.equal(python.status, 0, python.stderr); + assert.deepEqual(observed, JSON.parse(python.stdout)); +}); + +test("registered native dispatch preserves response bytes and completes the injected hold", async t => { + const { call, env } = database(t); + env.MODEL_PRICING_USD_PER_MILLION = JSON.stringify({ "openai:test": { input: 1000, output: 1000 } }); + const hooks = createReservationLifecycle(env, { call, identity: () => { + const { operation, id: ignored, ...values } = reserve(); + return values; + } }); + const { nativeGenerationFetch } = await import("../worker/gateway-extensions.mjs"); + const body = '{"choices":[],"usage":{"prompt_tokens":10,"completion_tokens":20}}'; + const response = await nativeGenerationFetch(new Request("http://localhost/openai/v1/chat/completions", { + method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ model: "test" }), + }), env, {}, { route: "/openai/v1/chat/completions", provider: "openai", principal: { id: "admin" } }, + async () => new Response(body, { headers: { "content-type": "application/json" } }), [hooks]); + assert.equal(await response.text(), body); + const summary = (await call({ operation: "summary", principal: "alice" })).summary; + assert.equal(summary.spent_today_usd, 0.03); assert.equal(summary.held_usd, 0); +}); + +test("enabled hook failures expose a JSON 503 and prohibit provider dispatch", async t => { + const { call, env } = database(t, false); + const hooks = createReservationLifecycle(env, { call, identity: () => { + const { operation, id: ignored, ...values } = reserve(); + return values; + } }); + try { await hooks.admit({}); assert.fail("missing schema admitted"); } + catch (error) { + assert.equal(error.status, 503); + assert.equal((await error.response().json()).error.code, "usage_reservations_unavailable"); + } +}); + + +test("fractional budgets never round admission above the configured cap", async t => { + const { call } = database(t); + assert.equal((await call(reserve(1, { estimate_usd: 1e-11, daily_budget_usd: 1e-11, monthly_budget_usd: null }))).status, 429); + assert.equal((await call({ operation: "summary", principal: "alice" })).summary.held_usd, 0); +}); diff --git a/worker/container-env.mjs b/worker/container-env.mjs index b3b27f75..b6315525 100644 --- a/worker/container-env.mjs +++ b/worker/container-env.mjs @@ -30,6 +30,8 @@ const DIRECT_ENV_KEYS = [ "DEFERRED_TOOLS_ENABLED", "GENERATION_CACHE_BACKEND", "GENERATION_CACHE_SHARED_ENABLED", + "USAGE_HOLD_REVIEW_AFTER_SECONDS", + "USAGE_RESERVATIONS_ENABLED", "CONNECTION_PROFILES_DB_PATH", "CONFIG_REVISION_SYNC_ENABLED", "CONFIG_SYNC_TTL_SECONDS", diff --git a/worker/d1-schema.mjs b/worker/d1-schema.mjs index 7ba673b5..28a31eed 100644 --- a/worker/d1-schema.mjs +++ b/worker/d1-schema.mjs @@ -10,7 +10,7 @@ export const REQUIRED_D1_TABLES = Object.freeze(["intelligence_policy", "intelli "usage_events", "usage_daily", "usage_batches", "media_jobs", "media_job_items", "shadow_eval_samples", "shadow_eval_results", "shadow_eval_config", "shadow_eval_policy_backups", "prompt_templates", - "config_snapshot_revisions", "config_snapshots", "config_snapshot_applications", "control_revisions", "tool_grants", "generation_cache"]); + "config_snapshot_revisions", "config_snapshots", "config_snapshot_applications", "control_revisions", "tool_grants", "generation_cache", "usage_reservation_budgets", "usage_reservations", "usage_reservation_transitions"]); /** * Readiness for the D1 schema. A deploy that skipped `wrangler d1 migrations apply` reports diff --git a/worker/reservations-d1.mjs b/worker/reservations-d1.mjs new file mode 100644 index 00000000..51b32199 --- /dev/null +++ b/worker/reservations-d1.mjs @@ -0,0 +1,288 @@ +/** Private, atomic monetary holds and injected request lifecycle. */ +const SCALE = 10_000_000_000; +const MAX_UNITS = Number.MAX_SAFE_INTEGER; +const DEFAULT_REVIEW = 259200; +const ACTIVE = new Set(["reserved", "dispatched", "unknown"]); +const ID = /^[a-f0-9]{32}$/; +const LABEL = /^[A-Za-z0-9][A-Za-z0-9_.:-]{0,127}$/; +const warned = new Set(); +const FIELDS = { + reserve: ["id", "principal", "estimate_usd", "daily_budget_usd", "monthly_budget_usd", "day_spent_usd", "month_spent_usd"], + transition: ["id", "revision", "state", "transition_id", "settlement_id", "cost_usd", "basis", "input_tokens", "output_tokens", + "admin", "evidence", "authorized_adjustment", "reason"], + get: ["id"], audit: ["id"], summary: ["principal"], +}; +export class ReservationError extends Error { + constructor(code = "usage_reservations_unavailable", status = 503) { super(code); this.code = code; this.status = status; } + response() { return reply({ error: { code: this.code, message: "The usage reservation operation could not be completed." } }, this.status); } +} +function fail(code = "invalid_reservation_request", status = 400) { throw new ReservationError(code, status); } +function warn(name) { + if (!warned.has(name)) { warned.add(name); console.warn(`Invalid ${name}; usage reservations disabled`); } +} +export function reservationSettings(env) { + const flag = String(env.USAGE_RESERVATIONS_ENABLED ?? "").trim().toLowerCase(); + if (["", "false", "0", "off", "no"].includes(flag)) return { enabled: false, review: DEFAULT_REVIEW }; + if (!["true", "1", "on", "yes"].includes(flag)) { warn("USAGE_RESERVATIONS_ENABLED"); return { enabled: false }; } + const raw = String(env.USAGE_HOLD_REVIEW_AFTER_SECONDS ?? "").trim(); + const review = raw ? Number(raw) : DEFAULT_REVIEW; + if ((raw && !/^\d+$/.test(raw)) || !Number.isSafeInteger(review) || review < 1 || review > 2 ** 31 - 1) { + warn("USAGE_HOLD_REVIEW_AFTER_SECONDS"); return { enabled: false }; + } + return { enabled: true, review }; +} +function identifier(value) { if (typeof value !== "string" || value.length !== 32 || !ID.test(value)) fail(); return value; } +function principal(value) { + if (typeof value !== "string" || ![...value].length || [...value].length > 256 || /[\x00-\x1f\x7f]/.test(value)) fail(); + return value; +} +function integer(value) { return Number.isSafeInteger(value) && value >= 0; } +export function reservationUnits(value, { limit = false } = {}) { + if (typeof value !== "number" || !Number.isFinite(value) || value < 0 || value > MAX_UNITS / SCALE) fail(); + // Parse the decimal representation, including exponents, without binary rounding. + const [mantissa, exponent = "0"] = String(value).toLowerCase().split("e"); + const [whole, fraction = ""] = mantissa.split("."); + const digits = BigInt(whole + fraction); + const power = 10 + Number(exponent) - fraction.length; + let result; + if (power >= 0) result = digits * 10n ** BigInt(power); + else { + const divisor = 10n ** BigInt(-power); + result = limit ? digits / divisor : (digits + divisor - 1n) / divisor; + } + if (result > BigInt(MAX_UNITS)) fail(); + return Number(result); +} +function document(value) { return JSON.stringify(Object.fromEntries(Object.entries(value).sort(([a], [b]) => a.localeCompare(b)))); } +function periods(at) { + const day = new Date(at).toISOString().slice(0, 10); + return { at, day, month: day.slice(0, 7) }; +} +function publicRow(row) { + if (!row) fail("reservation_not_found", 404); + const { amount_units, charged_units, ...value } = row; + return { ...value, estimate_usd: amount_units / SCALE, cost_usd: charged_units === null ? null : charged_units / SCALE }; +} +async function batch(db, statements) { + const results = await db.batch(statements); + if (results.some(r => !r.success)) throw new ReservationError(); + return results; +} +const TOTALS = `SELECT + COALESCE(SUM(CASE WHEN state IN ('reserved','dispatched','unknown') THEN amount_units ELSE 0 END),0) AS held_units, + COALESCE(SUM(CASE WHEN state IN ('settled','reconciled') AND day = ? THEN charged_units ELSE 0 END),0) AS day_units, + COALESCE(SUM(CASE WHEN state IN ('settled','reconciled') AND month = ? THEN charged_units ELSE 0 END),0) AS month_units, + COUNT(CASE WHEN state = 'reserved' THEN 1 END) AS reserved, + COUNT(CASE WHEN state = 'dispatched' THEN 1 END) AS dispatched, + COUNT(CASE WHEN state = 'unknown' THEN 1 END) AS unknown, + COUNT(CASE WHEN state IN ('reserved','dispatched','unknown') AND created_at <= ? THEN 1 END) AS needs_review + FROM usage_reservations WHERE principal = ?`; +export async function reservationSummary(db, identity, now, review = DEFAULT_REVIEW) { + principal(identity); + const { at, day, month } = periods(now); + const results = await batch(db, [db.prepare(TOTALS).bind(day, month, at - review * 1000, identity), + db.prepare("SELECT period, baseline_units FROM usage_reservation_budgets WHERE principal = ? AND period IN (?, ?)").bind(identity, day, month)]); + const totals = results[0].results[0]; + const bases = new Map(results[1].results.map(row => [row.period, row.baseline_units])); + const { held_units, day_units, month_units, ...counts } = totals; + return { ...counts, day_seeded: bases.has(day), month_seeded: bases.has(month), held_usd: held_units / SCALE, + spent_today_usd: (day_units + (bases.get(day) ?? 0)) / SCALE, + spent_this_month_usd: (month_units + (bases.get(month) ?? 0)) / SCALE }; +} +const RESERVE = `INSERT INTO usage_reservations + (id,principal,day,month,amount_units,state,created_at,updated_at,transition_id) + SELECT ?1,?2,?3,?4,?5,'reserved',?6,?6,?1 + WHERE NOT EXISTS (SELECT 1 FROM usage_reservation_transitions WHERE transition_id=?1) + AND (?7 IS NULL OR + (COALESCE((SELECT baseline_units FROM usage_reservation_budgets WHERE principal=?2 AND period=?3),0) + + COALESCE((SELECT SUM(CASE WHEN state IN ('reserved','dispatched','unknown') THEN amount_units + WHEN day=?3 THEN charged_units ELSE 0 END) FROM usage_reservations WHERE principal=?2),0)) < ?7 + AND (COALESCE((SELECT baseline_units FROM usage_reservation_budgets WHERE principal=?2 AND period=?3),0) + + COALESCE((SELECT SUM(CASE WHEN state IN ('reserved','dispatched','unknown') THEN amount_units + WHEN day=?3 THEN charged_units ELSE 0 END) FROM usage_reservations WHERE principal=?2),0)) <= ?7-?5) + AND (?8 IS NULL OR + (COALESCE((SELECT baseline_units FROM usage_reservation_budgets WHERE principal=?2 AND period=?4),0) + + COALESCE((SELECT SUM(CASE WHEN state IN ('reserved','dispatched','unknown') THEN amount_units + WHEN month=?4 THEN charged_units ELSE 0 END) FROM usage_reservations WHERE principal=?2),0)) < ?8 + AND (COALESCE((SELECT baseline_units FROM usage_reservation_budgets WHERE principal=?2 AND period=?4),0) + + COALESCE((SELECT SUM(CASE WHEN state IN ('reserved','dispatched','unknown') THEN amount_units + WHEN month=?4 THEN charged_units ELSE 0 END) FROM usage_reservations WHERE principal=?2),0)) <= ?8-?5) + ON CONFLICT(id) DO NOTHING`; +export async function reserveUsage(db, body, now) { + identifier(body.id); principal(body.principal); + if (body.daily_budget_usd == null && body.monthly_budget_usd == null) fail(); + if (body.estimate_usd == null) fail("unpriced_reservation", 503); + const amount = reservationUnits(body.estimate_usd); + const daily = body.daily_budget_usd == null ? null : reservationUnits(body.daily_budget_usd, { limit: true }); + const monthly = body.monthly_budget_usd == null ? null : reservationUnits(body.monthly_budget_usd, { limit: true }); + const dayBase = reservationUnits(body.day_spent_usd), monthBase = reservationUnits(body.month_spent_usd); + const { at, day, month } = periods(now); + const doc = document({ id: body.id, principal: body.principal, amount_units: amount, day, month }); + const results = await batch(db, [ + db.prepare("INSERT OR IGNORE INTO usage_reservation_budgets VALUES (?, ?, ?)").bind(body.principal, day, dayBase), + db.prepare("INSERT OR IGNORE INTO usage_reservation_budgets VALUES (?, ?, ?)").bind(body.principal, month, monthBase), + db.prepare(RESERVE).bind(body.id, body.principal, day, month, amount, at, daily, monthly), + db.prepare(`INSERT OR IGNORE INTO usage_reservation_transitions + (transition_id,reservation_id,state,at,document) SELECT id,id,'reserved',created_at,? + FROM usage_reservations WHERE id=? AND revision=0`).bind(doc, body.id), + db.prepare("SELECT * FROM usage_reservations WHERE id=?").bind(body.id), + db.prepare("SELECT document FROM usage_reservation_transitions WHERE transition_id=?").bind(body.id), + ]); + const row = results[4].results[0]; + if (!row) fail(results[5].results.length ? "reservation_conflict" : "budget_exceeded", results[5].results.length ? 409 : 429); + if (row.state !== "reserved" || results[5].results[0]?.document !== doc) fail("reservation_conflict", 409); + return publicRow(row); +} +function transitionFields(body) { + const state = body.state; + if (!["dispatched", "unknown", "settled", "reconciled"].includes(state) || !integer(body.revision)) fail(); + identifier(body.id); identifier(body.transition_id); + const { cost_usd = null, basis = null, input_tokens = null, output_tokens = null, settlement_id = null, + evidence = null, reason = null } = body; + for (const count of [input_tokens, output_tokens]) if (count !== null && !integer(count)) fail(); + const charged_units = cost_usd === null ? null : reservationUnits(cost_usd); + if (["settled", "reconciled"].includes(state)) { + identifier(settlement_id); + if (charged_units === null || !["provider", "released", "adjustment"].includes(basis)) fail(); + } else if ([cost_usd, basis, settlement_id, evidence, reason].some(value => value !== null)) fail(); + if (state === "settled" && (basis === "adjustment" || (basis === "released" && charged_units !== 0))) fail(); + if (state === "reconciled") { + if (body.admin !== true) fail("admin_required", 403); + if (typeof reason !== "string" || /[\r\n]/.test(reason) || !LABEL.test(reason)) fail("reconciliation_evidence_required"); + if (basis === "provider") { + if (typeof evidence !== "string" || /[\r\n]/.test(evidence) || !LABEL.test(evidence)) fail("reconciliation_evidence_required"); + } else if (basis !== "adjustment" || body.authorized_adjustment !== true || evidence !== null) fail("reconciliation_evidence_required"); + } else if (evidence !== null || reason !== null || body.admin || body.authorized_adjustment) fail(); + return { charged_units, basis, input_tokens, output_tokens, settlement_id, evidence, reason }; +} +export async function transitionUsage(db, body, now) { + const fields = transitionFields(body); + const doc = document({ id: body.id, revision: body.revision, state: body.state, ...fields }); + const row = await db.prepare("SELECT * FROM usage_reservations WHERE id=?").bind(body.id).first(); + if (!row) fail("reservation_not_found", 404); + const previous = await db.prepare("SELECT document FROM usage_reservation_transitions WHERE transition_id=?").bind(body.transition_id).first(); + if (previous) { + if (previous.document !== doc) fail("reservation_conflict", 409); + return { applied: false, reservation: publicRow(row) }; + } + const legal = (row.state === "reserved" && body.state === "dispatched") + || (row.state === "reserved" && body.state === "settled" && fields.basis === "released") + || (row.state === "dispatched" && ["unknown", "settled"].includes(body.state) && fields.basis !== "released") + || (row.state === "unknown" && body.state === "reconciled"); + if (row.revision !== body.revision || !legal) fail("reservation_conflict", 409); + const { charged_units, basis, input_tokens, output_tokens, settlement_id, evidence, reason } = fields; + const results = await batch(db, [db.prepare(`UPDATE usage_reservations SET state=?1, revision=revision+1, + charged_units=?2,basis=?3,input_tokens=COALESCE(?4,input_tokens),output_tokens=COALESCE(?5,output_tokens), + updated_at=?6,handoff_at=CASE WHEN ?1='dispatched' THEN ?6 ELSE handoff_at END,transition_id=?7,settlement_id=?8 + WHERE id=?9 AND revision=?10 AND state=?11 + AND NOT EXISTS (SELECT 1 FROM usage_reservation_transitions WHERE transition_id=?7) AND NOT EXISTS + (SELECT 1 FROM usage_reservations WHERE settlement_id=?8 AND id!=?9)`) + .bind(body.state, charged_units, basis, input_tokens, output_tokens, now, body.transition_id, settlement_id, body.id, body.revision, row.state), + db.prepare(`INSERT OR IGNORE INTO usage_reservation_transitions + (transition_id,reservation_id,previous_state,state,at,basis,evidence,reason,document) + SELECT ?1,id,?2,state,updated_at,basis,?3,?4,?5 FROM usage_reservations WHERE id=?6 AND transition_id=?1`) + .bind(body.transition_id, row.state, evidence, reason, doc, body.id), + db.prepare("SELECT * FROM usage_reservations WHERE id=?").bind(body.id), + db.prepare("SELECT document FROM usage_reservation_transitions WHERE transition_id=?").bind(body.transition_id)]); + if (results[3].results[0]?.document !== doc) fail("reservation_conflict", 409); + return { applied: results[0].meta.changes === 1, reservation: publicRow(results[2].results[0]) }; +} +function validate(body) { + if (!body || typeof body !== "object" || Array.isArray(body) || body.version !== 1 || !Object.hasOwn(FIELDS, body.operation)) fail(); + const allowed = new Set(["version", "operation", ...FIELDS[body.operation]]); + if (Object.keys(body).some(key => !allowed.has(key))) fail(); + if (["get", "audit"].includes(body.operation)) identifier(body.id); + if (body.operation === "summary") principal(body.principal); + return body; +} +async function readBody(request) { + if (request.headers.get("content-type")?.split(";", 1)[0].trim().toLowerCase() !== "application/json") fail(); + const length = request.headers.get("content-length"); + if (length !== null && (!/^\d+$/.test(length) || Number(length) > 32768)) fail(); + if (!request.body) fail(); + const reader = request.body.getReader(); + const chunks = []; + let size = 0, finished = false; + try { + while (true) { + const item = await reader.read(); + if (item.done) { finished = true; break; } + size += item.value.byteLength; + if (size > 32768) fail(); + chunks.push(item.value); + } + } finally { if (!finished) void reader.cancel().catch(() => {}); reader.releaseLock(); } + const bytes = new Uint8Array(size); + let offset = 0; + for (const chunk of chunks) { bytes.set(chunk, offset); offset += chunk.byteLength; } + try { return validate(JSON.parse(new TextDecoder("utf-8", { fatal: true }).decode(bytes))); } + catch (error) { if (error instanceof ReservationError) throw error; fail(); } +} +function reply(value, status = 200) { + return Response.json({ version: 1, ...value }, { status, headers: { "cache-control": "no-store", "x-content-type-options": "nosniff" } }); +} +/** Must be mounted only by the trusted private outbound dispatcher. */ +export async function handleReservationsRequest(request, env, { now = Date.now } = {}) { + const settings = reservationSettings(env); + if (!settings.enabled) return reply({ error: { code: "not_found" } }, 404); + try { + const url = new URL(request.url); + if (url.origin !== "http://intelligence.internal" || url.pathname !== "/v1/reservations" || url.search || url.hash + || url.username || url.password) fail("store_route_not_found", 404); + if (request.method !== "POST") fail("store_method_not_allowed", 405); + const body = await readBody(request); + const db = env.INTELLIGENCE_DB; + if (!db || typeof db.prepare !== "function" || typeof db.batch !== "function") throw new ReservationError(); + let result; + if (body.operation === "reserve") result = { reservation: await reserveUsage(db, body, now()) }; + else if (body.operation === "transition") result = await transitionUsage(db, body, now()); + else if (body.operation === "summary") result = { summary: await reservationSummary(db, body.principal, now(), settings.review) }; + else if (body.operation === "audit") { + const rows = await db.prepare("SELECT * FROM usage_reservation_transitions WHERE reservation_id=? ORDER BY rowid LIMIT 100") + .bind(body.id).all(); + result = { transitions: rows.results }; + } else result = { reservation: publicRow(await db.prepare("SELECT * FROM usage_reservations WHERE id=?").bind(body.id).first()) }; + return reply(result); + } catch (error) { + const failure = error instanceof ReservationError ? error : new ReservationError(); + return reply({ error: { code: failure.code, message: "The usage reservation operation could not be completed." } }, failure.status); + } +} +/** One instance per native request. Identity and pricing come from verified admission. */ +export function createReservationLifecycle(env, { call, identity }) { + let reservation; + const transition = async (state, fields = {}) => { + const result = await call({ version: 1, operation: "transition", id: reservation.id, revision: reservation.revision, + state, transition_id: crypto.randomUUID().replaceAll("-", ""), ...fields }); + if (result.error) throw new ReservationError(result.error.code, result.error.code === "reservation_conflict" ? 409 : 503); + reservation = result.reservation; + }; + return { + enabled: env => reservationSettings(env).enabled, + async admit(context) { + if (!reservationSettings(env).enabled) return; + const values = await identity(context); + if (values.daily_budget_usd == null && values.monthly_budget_usd == null) return; + const result = await call({ version: 1, operation: "reserve", id: crypto.randomUUID().replaceAll("-", ""), ...values }); + if (result.error) throw new ReservationError(result.error.code, result.error.code === "budget_exceeded" ? 429 : 503); + reservation = result.reservation; + }, + async before_dispatch() { + if (reservation?.state === "reserved") await transition("dispatched"); + }, + async finalize(context) { + if (!reservation || !ACTIVE.has(reservation.state) || reservation.state === "unknown") return; + if (reservation.state === "reserved") { + await transition("settled", { cost_usd: 0, basis: "released", settlement_id: reservation.id }); + return; + } + const row = context.usage ?? context; + const measured = (!context.outcome || context.outcome === "success") && !context.cancellationOutcome?.ambiguous && row.cost_usd != null && row.cost_basis === "usage"; + await transition(measured ? "settled" : "unknown", measured + ? { cost_usd: row.cost_usd, basis: "provider", settlement_id: reservation.id, + input_tokens: row.input_tokens ?? null, output_tokens: row.output_tokens ?? null } + : { input_tokens: context.cancellationOutcome?.ambiguous ? null : row.input_tokens ?? null, + output_tokens: context.cancellationOutcome?.ambiguous ? null : row.output_tokens ?? null }); + }, + }; +}