diff --git a/docs/credits-ledger.md b/docs/credits-ledger.md new file mode 100644 index 00000000..5ace9fb7 --- /dev/null +++ b/docs/credits-ledger.md @@ -0,0 +1,157 @@ +# Gateway credits + +Credits are disabled unless `CREDITS_ENABLED=true`. Empty values use the defaults. +`CREDITS_CURRENCY` defaults to USD; other currencies disable credits. Invalid +settings log one warning without their values. Disabled routes return 404. + +Apply the additive credits migration to the chosen database before enabling the +feature. It creates `credits_balances`, `credits_entries` and `credits_audit`. +Existing usage records are retained. The runtime does not apply migrations. +Missing tables or required columns return JSON 503 with `credits_unavailable`; +credit admission then stops before provider dispatch. Storage outages have the +same result, without retrying writes or switching to another balance authority. +The local store uses the existing `USAGE_DB_PATH`; the Container uses the private +D1 transport when `INTELLIGENCE_STORAGE_BACKEND=d1`. + +Amounts are integer micro-USD, bounded to ±1,000,000,000,000,000. One USD is +1,000,000 micro-USD. Fractional amounts, booleans and overflow are rejected, +without rounding or conversion from usage estimates. Revisions have the same +nonnegative bound. No account receives automatic credit, bonus or top-up. + +The ledger stores `credit`, `reserve`, `commit`, `release`, `adjust` and +`compensate` events. Only the materialized balance row changes. Balance is the +sum of credits and adjustments, minus committed costs, plus compensations. +Held credit is the sum of open estimates; available credit is balance minus +held. Every write checks the balance revision. New reservations and negative +adjustments must not make available credit negative. Each owner has its own +operation ID namespace. Repeating an identical +operation, including its original revision, returns its original entry. Reusing +the ID with different content returns 409 `credits_conflict`. + +Use a distinct `scoped_id` for each provider attempt, including hedges, shadows +and batch items, and a distinct operation ID for each transition. Reserving an +estimate holds credit. Committing the actual integer cost removes that attempt's +whole hold and debits only the actual cost. The commit entry's `held_delta` +records the released hold atomically. An explicit release removes a hold without +a charge; its amount must match the original estimate. Terminal attempts cannot +reopen. + +A commit records the actual cost even when it exceeds the hold and available +credit, and closes only its own hold. Balance and available credit can become +negative within the integer bounds. New reservations are refused while available +credit is zero or negative, until credits or positive adjustments cover the +shortfall and leave available credit for admission. + +Unknown reconciliation appends a linked reserve marker with zero balance and +hold deltas. Its zero amount is a marker, not a zero-cost settlement. The original +hold remains indefinitely; there is no automatic expiry. A later known cost can +settle it. A compensation references an original credit, adjustment or commit +operation and applies the exact inverse balance delta once. It never edits the +original entry, reopens a hold or modifies a usage receipt. + +Gateway prices are an explicit operator tariff, separate from provider usage +estimates and invoices. `register_credit_authority(store, tariff=...)` and +`registerCreditAuthority(db, {tariff, env})` implement the enterprise +`CreditAuthority` contract. The tariff callback receives `(operation, phase)`; +it must verify the operation's integer amount against the reviewed tariff and +return that tariff's integer revision. Return `None`/`null` for unpriced admission +or unknown reconciliation. Unpriced reserve or commit raises `AuthorityDenied` +with `credits_unpriced`; insufficient funds raise `credits_insufficient`. +Registration performs no storage writes. The authority also checks the feature +flag before evaluating prices or touching storage. + +For every authority transition, `AuthorityOperation.revision` must be the owner's +current balance revision. On 412 `credits_revision_mismatch`, re-read that revision +and retry with the same operation ID and unchanged cost and attempt fields, using +a bounded retry limit. A rejected revision writes nothing, and operation IDs make +retries idempotent. After a successful write, replay the original revision to +retrieve its entry; reusing the ID with changed content is a conflict. + +`GET /v1/credits` requires a gateway key with `models` scope. It reads only the +authenticated owner; query parameters cannot choose another owner. Verified +tenant context produces a stable organisation/team/principal namespace; +legacy principals keep their existing namespace. The response is: + +```text +{currency: "USD", balance_microusd: 100, held_microusd: 60, + available_microusd: 40, entries: [...], next_cursor: null} +``` + +Use `?limit=20&cursor=0` for pagination. The limit is 1–100 (default 100); cursors +are nonnegative revision strings. Follow `next_cursor` until it is null. Pages +are ordered by increasing ledger revision. All responses use `Cache-Control: +no-store`. Reads and writes include an `ETag` containing the balance revision. +Entries contain operation IDs, integer deltas, attempt/reference IDs and tariff +revisions. They omit actor and adjustment reason. No prompts, completions, +provider credentials or payment details are collected. + +`GET /admin/credits/` requires an administrator dashboard session and +includes the current revision. URL-encode tenant owner IDs. +`POST /admin/credits/adjust` requires the same session, a CSRF token and an +`If-Match` header copied from the owner's current ETag. For example: + +```text +If-Match: "0" +X-CSRFToken: +{"owner":"alice","amount_microusd":1000000, + "operation_id":"allocation-1","reason":"Reviewed allocation"} +``` + +Missing `If-Match` returns 428 `credits_revision_required`; a stale revision +returns 412 `credits_revision_mismatch`. Negative adjustments are allowed only +within available credit. Reasons must be 1–512 printable ASCII characters. +Audit rows atomically record owner, operation ID, revision, actor and kind, with +no reason or request content. Treat reason text as administrative metadata and +keep user content out of it. Ledger and audit history are retained; this feature +provides no deletion, automatic payment, currency conversion or invoice service. + +Application integration uses `register_credits_routes(app, csrf)` in +`app.py:create_app`. Bind verified context as `g.tenant_context`. Supply the +registered credit authority to `services/enterprise_contract.py: +register_enterprise_adapters`, preserving the other authority collaborators. +Reservation integration calls reserve from `services/budget_service.py: +BudgetService._reserve_durable` / `services/request_accounting.py:begin`, commit +from `BudgetService.complete` / `services/request_accounting.py:_record_budget`, +and reconcile from `services/reservation_store.py:reconcile`. Use independent +attempt identities and the current balance revision. The private dispatcher call site is +`worker/managed-state-dispatch.mjs:handleManagedStateRequest`, mapped to +`handleCreditsRequest` for `/v1/managed-state/credits`. The Container transport +uses the `credits` domain, allowlisted as `/v1/managed-state/credits` in +`services/intelligence_d1_store.py:_ENDPOINTS` on the private internal origin. +For native requests, bind the authority at the reserve/settle/reconcile call +sites in `worker/reservations-d1.mjs:createReservationLifecycle` through the +fixed registration in `worker/gateway-extensions.mjs`. These call sites must +classify unknown costs rather than translating them to zero. `CreditDenied` +extends `AuthorityDenied` with bounded `code` and `status`; Flask route +registration installs its JSON error handler. Native request integration uses +the denial's `response()` method through `generationErrorResponse` so schema +failures remain JSON 503 denials. + +Usage receipts may include an optional ledger operation ID and revision as +provenance. Credit admission and settlement do not require receipt storage, +and receipt errors must never trigger a second charge. Local tests exercise +SQLite and a SQLite-backed D1 contract fake; they do not validate deployed D1, +live provider metering or an operator's tariff. + +Shared integer and idempotency vectors: + +```json +[ + {"invalid_amount": true}, + {"invalid_amount": 0.5}, + {"invalid_amount": 1000000000000001}, + {"invalid_amount": -1000000000000001}, + {"request": {"owner":"alice","kind":"credit","amount_microusd":100,"operation_id":"fund","revision":0},"revision":1}, + {"request": {"owner":"alice","kind":"credit","amount_microusd":100,"operation_id":"fund","revision":0},"revision":1}, + {"request": {"owner":"alice","kind":"credit","amount_microusd":101,"operation_id":"fund","revision":0},"error":"credits_conflict"}, + {"request": {"owner":"alice","kind":"reserve","amount_microusd":60,"operation_id":"hold","revision":1,"scoped_id":"attempt","tariff_revision":1},"revision":2}, + {"request": {"owner":"alice","kind":"reserve","amount_microusd":50,"operation_id":"second","revision":2,"scoped_id":"other","tariff_revision":1},"error":"credits_insufficient"}, + {"request": {"owner":"alice","kind":"commit","amount_microusd":25,"operation_id":"charge","revision":2,"scoped_id":"attempt","tariff_revision":1},"revision":3}, + {"request":{"owner":"overrun","kind":"credit","amount_microusd":100,"operation_id":"fund","revision":0},"revision":1}, + {"request":{"owner":"overrun","kind":"reserve","amount_microusd":60,"operation_id":"hold","revision":1,"scoped_id":"attempt","tariff_revision":1},"revision":2}, + {"request":{"owner":"overrun","kind":"commit","amount_microusd":150,"operation_id":"charge","revision":2,"scoped_id":"attempt","tariff_revision":1},"revision":3,"summary":{"balance_microusd":-50,"held_microusd":0,"available_microusd":-50}}, + {"request":{"owner":"overrun","kind":"reserve","amount_microusd":1,"operation_id":"denied","revision":3,"scoped_id":"next","tariff_revision":1},"error":"credits_insufficient"}, + {"request":{"owner":"overrun","kind":"credit","amount_microusd":100,"operation_id":"refill","revision":3},"revision":4,"summary":{"balance_microusd":50,"held_microusd":0,"available_microusd":50}}, + {"request":{"owner":"overrun","kind":"reserve","amount_microusd":1,"operation_id":"next","revision":4,"scoped_id":"next","tariff_revision":1},"revision":5,"summary":{"balance_microusd":50,"held_microusd":1,"available_microusd":49}} +] +``` diff --git a/intelligence-migrations/0037_credits_ledger.sql b/intelligence-migrations/0037_credits_ledger.sql new file mode 100644 index 00000000..528b9ed5 --- /dev/null +++ b/intelligence-migrations/0037_credits_ledger.sql @@ -0,0 +1,33 @@ +CREATE TABLE IF NOT EXISTS credits_balances ( + owner TEXT PRIMARY KEY, + balance_microusd INTEGER NOT NULL DEFAULT 0 CHECK (balance_microusd BETWEEN -1000000000000000 AND 1000000000000000), + held_microusd INTEGER NOT NULL DEFAULT 0 CHECK (held_microusd BETWEEN 0 AND 1000000000000000), + revision INTEGER NOT NULL DEFAULT 0 CHECK (revision BETWEEN 0 AND 1000000000000000) +); +CREATE TABLE IF NOT EXISTS credits_entries ( + owner TEXT NOT NULL, + operation_id TEXT NOT NULL, + revision INTEGER NOT NULL, + kind TEXT NOT NULL CHECK (kind IN ('credit','reserve','commit','release','adjust','compensate')), + amount_microusd INTEGER NOT NULL CHECK (amount_microusd BETWEEN -1000000000000000 AND 1000000000000000), + balance_delta INTEGER NOT NULL, + held_delta INTEGER NOT NULL, + scoped_id TEXT, + reference TEXT, + tariff_revision INTEGER, + actor TEXT, + reason TEXT, + document TEXT NOT NULL, + PRIMARY KEY (owner, operation_id), + UNIQUE (owner, revision) +); +CREATE UNIQUE INDEX IF NOT EXISTS credits_compensation_once ON credits_entries(owner, reference) WHERE kind = 'compensate'; +CREATE INDEX IF NOT EXISTS credits_attempt_entries ON credits_entries(owner, scoped_id, revision); +CREATE TABLE IF NOT EXISTS credits_audit ( + owner TEXT NOT NULL, + operation_id TEXT NOT NULL, + revision INTEGER NOT NULL, + actor TEXT NOT NULL, + kind TEXT NOT NULL, + PRIMARY KEY (owner, operation_id) +); diff --git a/package.json b/package.json index ecfe77f2..a5dd4c0b 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,7 @@ "lint": "ruff check . --select E9,F63,F7,F82", "typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming", "test": "python -m pytest -q", - "test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs tests/test_usage_receipts_worker.mjs tests/test_learned_cooldown_worker.mjs tests/test_stream_cost_breaker_worker.mjs tests/test_realtime_worker.mjs tests/test_latency_slo_worker.mjs tests/test_context_canary_worker.mjs tests/test_enterprise_contracts_worker.mjs tests/test_wave6_worker.mjs tests/test_scim_provisioning_worker.mjs tests/test_saml_federation_worker.mjs tests/test_tenant_governance_worker.mjs tests/test_tenant_hierarchy_worker.mjs", + "test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs tests/test_usage_receipts_worker.mjs tests/test_learned_cooldown_worker.mjs tests/test_stream_cost_breaker_worker.mjs tests/test_realtime_worker.mjs tests/test_latency_slo_worker.mjs tests/test_context_canary_worker.mjs tests/test_enterprise_contracts_worker.mjs tests/test_wave6_worker.mjs tests/test_credits_ledger_worker.mjs tests/test_scim_provisioning_worker.mjs tests/test_saml_federation_worker.mjs tests/test_tenant_governance_worker.mjs tests/test_tenant_hierarchy_worker.mjs", "db:migrate": "python scripts/validate_sqlite_schema.py --apply", "db:validate": "python scripts/validate_sqlite_schema.py", "security:secrets": "python scripts/check_static_secrets.py", diff --git a/routes/credits.py b/routes/credits.py new file mode 100644 index 00000000..d4b42852 --- /dev/null +++ b/routes/credits.py @@ -0,0 +1,94 @@ +"""Owner reads and explicit administrator CAS adjustments for gateway credits.""" +import re + +from flask import g, jsonify, request + +from route_helpers import api_authenticate_only, login_required +from routes.config_snapshots import json_errors +from routes.core import require_admin_dashboard_user +from services import credits_ledger as ledger + + +def register_credits_routes(app, csrf, *, store=None): + if app.extensions.get('credits_routes_registered'): + return + app.extensions['credits_routes_registered'] = True + + def disabled_credits(): + if request.path == '/v1/credits' or request.path.startswith('/admin/credits/'): + if not ledger.enabled(): + return failed(ledger.CreditsError('not_found', 404)) + return None + + app.before_request_funcs.setdefault(None, []).insert(0, disabled_credits) + + @app.after_request + def private_credits(result): + if request.path == '/v1/credits' or request.path.startswith('/admin/credits/'): + result.headers['Cache-Control'] = 'no-store' + return result + + def authority(): + if not ledger.enabled(): + raise ledger.CreditsError('not_found', 404) + return store if store is not None else ledger.open_store() + + def failed(error): + return jsonify({'error': {'code': error.code, 'message': 'Credits operation unavailable.'}}), error.status, {'Cache-Control': 'no-store'} + + app.register_error_handler(ledger.CreditDenied, failed) + + def response(value, *, owner_view=False): + result = jsonify({key: item for key, item in value.items() if not owner_view or key != 'revision'}) + result.headers['Cache-Control'] = 'no-store' + if 'revision' in value: + result.headers['ETag'] = '"' + str(value['revision']) + '"' + return result + + def page(owner): + limit = request.args.get('limit', '100') + if not re.fullmatch(r'[0-9]{1,3}', limit): + raise ledger.CreditsError('invalid_credits_request', 400) + return authority().read(owner, cursor=request.args.get('cursor', '0'), limit=int(limit)) + + @app.route('/v1/credits', methods=['GET', 'OPTIONS']) + @csrf.exempt + @api_authenticate_only(required_scope='models') + def owner_credits(): + try: + context = getattr(g, 'tenant_context', None) + user = g.authenticated_user + owner = ledger.context_owner(context) if context is not None else str(user.get('username') or user.get('id') or '') + return response(page(owner), owner_view=True) + except ledger.CreditsError as error: + return failed(error) + + @app.get('/admin/credits/') + @json_errors + @login_required + def admin_credits(owner): + try: + require_admin_dashboard_user() + return response(page(owner)) + except ledger.CreditsError as error: + return failed(error) + + @app.post('/admin/credits/adjust') + @json_errors + @login_required + def adjust_credits(): + try: + actor = require_admin_dashboard_user() + match = request.headers.get('If-Match') + if match is None: + raise ledger.CreditsError('credits_revision_required', 428) + if not re.fullmatch(r'"[0-9]{1,16}"', match): + raise ledger.CreditsError('invalid_credits_request', 400) + body = request.get_json(silent=True) + if not isinstance(body, dict) or set(body) != {'owner', 'amount_microusd', 'operation_id', 'reason'}: + raise ledger.CreditsError('invalid_credits_request', 400) + entry = authority().append(body['owner'], 'adjust', body['amount_microusd'], body['operation_id'], + int(match[1:-1]), reason=body['reason'], actor=str(actor.get('username') or actor.get('id') or '')) + return response(entry) + except ledger.CreditsError as error: + return failed(error) diff --git a/services/credits_ledger.py b/services/credits_ledger.py new file mode 100644 index 00000000..63a85e71 --- /dev/null +++ b/services/credits_ledger.py @@ -0,0 +1,401 @@ +"""Immutable integer credits and revision-bound, fail-closed spending authority.""" +from __future__ import annotations + +import json +import logging +import os +import re +import sqlite3 +import threading +from contextlib import closing +from pathlib import Path +from typing import Mapping + +from services.enterprise_contract import (AuthorityDenied, AuthorityOperation, AuthorityResult, + TenantContext, register_enterprise_adapters) +from services.sqlite_store import storage_path + +MAX_AMOUNT = 10**15 +MIGRATION = '0037_credits_ledger.sql' +TABLES = ('credits_balances', 'credits_entries', 'credits_audit') +_warned: set[str] = set() +_warn_lock = threading.Lock() +logger = logging.getLogger(__name__) + + +class CreditsError(Exception): + def __init__(self, code='credits_unavailable', status=503): + super().__init__(code) + self.code, self.status = code, status + + +class CreditDenied(AuthorityDenied): + """Bounded error metadata lets request admission return a JSON denial.""" + def __init__(self, code, status=503): + super().__init__(code) + self.code, self.status = code, status + + +def _warn(name): + with _warn_lock: + if name not in _warned: + _warned.add(name) + logger.warning('Invalid %s; credits disabled', name) + + +def enabled(env: Mapping | None = None) -> bool: + values = os.environ if env is None else env + raw = values.get('CREDITS_ENABLED', '') + flag = raw.strip().lower() if isinstance(raw, str) else 'invalid' + if flag in {'', '0', 'false', 'no', 'off'}: + return False + if flag not in {'1', 'true', 'yes', 'on'}: + _warn('CREDITS_ENABLED') + return False + currency = values.get('CREDITS_CURRENCY', '') + if currency == '': + currency = 'USD' + if currency != 'USD': + _warn('CREDITS_CURRENCY') + return False + return True + + +def integer(value, *, nonnegative=False): + if type(value) is not int or not (-MAX_AMOUNT <= value <= MAX_AMOUNT) or (nonnegative and value < 0): + raise CreditsError('invalid_credits_request', 400) + return value + + +def label(value): + if not isinstance(value, str) or not re.fullmatch(r'[A-Za-z0-9_:.\-]{1,128}', value): + raise CreditsError('invalid_credits_request', 400) + return value + + +def reason_text(value): + if not isinstance(value, str) or not 1 <= len(value) <= 512 or any(ord(c) < 32 or ord(c) > 126 for c in value): + raise CreditsError('invalid_credits_request', 400) + return value + + +def owner_id(value): + if not isinstance(value, str) or not 1 <= len(value) <= 512 or any(ord(c) < 32 or ord(c) == 127 for c in value): + raise CreditsError('invalid_credits_request', 400) + return value + + +def context_owner(context: TenantContext) -> str: + if type(context) is not TenantContext: + raise CreditsError('invalid_credits_request', 400) + if context.org_id is None: + return owner_id(context.principal_id) + # JSON tuple delimiters cannot collide with legacy opaque principal IDs. + return 'tenant:' + json.dumps([context.org_id, context.team_id, context.principal_id], separators=(',', ':')) + + +def request_fields(owner, kind, amount_microusd, operation_id, revision, *, scoped_id=None, + reference=None, tariff_revision=None, actor=None, reason=None, unknown=False): + owner_id(owner) + label(operation_id) + integer(revision, nonnegative=True) + integer(amount_microusd) + if kind not in {'credit', 'reserve', 'commit', 'release', 'adjust', 'compensate'} or type(unknown) is not bool: + raise CreditsError('invalid_credits_request', 400) + for value in (scoped_id, reference, actor): + if value is not None: + label(value) + if kind in {'reserve', 'commit', 'release'}: + label(scoped_id) + integer(amount_microusd, nonnegative=True) + if kind in {'reserve', 'commit'} and not unknown: + if tariff_revision is None: + raise CreditsError('credits_unpriced', 503) + integer(tariff_revision, nonnegative=True) + elif tariff_revision is not None: + integer(tariff_revision, nonnegative=True) + if kind == 'credit': + integer(amount_microusd, nonnegative=True) + if kind in {'adjust', 'compensate'}: + label(actor) + reason_text(reason) + elif reason is not None: + reason_text(reason) + if kind == 'compensate': + label(reference) + elif reference is not None: + raise CreditsError('invalid_credits_request', 400) + if unknown and (kind != 'reserve' or amount_microusd != 0): + raise CreditsError('invalid_credits_request', 400) + return dict(owner=owner, kind=kind, amount_microusd=amount_microusd, operation_id=operation_id, + revision=revision, scoped_id=scoped_id, reference=reference, tariff_revision=tariff_revision, + actor=actor, reason=reason, unknown=unknown) + + +def deltas(fields, entries): + kind, amount = fields['kind'], fields['amount_microusd'] + balance_delta, held_delta = 0, 0 + if kind in {'credit', 'adjust'}: + balance_delta = amount + elif kind == 'compensate': + original = next((row for row in entries if row['operation_id'] == fields['reference']), None) + if (not original or original['kind'] not in {'credit', 'adjust', 'commit'} + or amount != -original['balance_delta'] + or any(row['kind'] == 'compensate' and row['reference'] == fields['reference'] for row in entries)): + raise CreditsError('credits_conflict', 409) + balance_delta = amount + else: + attempt = [row for row in entries if row['scoped_id'] == fields['scoped_id']] + hold = next((row for row in attempt if row['kind'] == 'reserve' and row['reference'] is None), None) + terminal = any(row['kind'] in {'commit', 'release'} for row in attempt) + if kind == 'reserve' and not fields['unknown']: + if hold: + raise CreditsError('credits_conflict', 409) + held_delta = amount + else: + if not hold or terminal: + raise CreditsError('credits_conflict', 409) + fields['reference'] = hold['operation_id'] + if kind == 'commit': + balance_delta, held_delta = -amount, -hold['amount_microusd'] + elif kind == 'release': + if amount != hold['amount_microusd']: + raise CreditsError('invalid_credits_request', 400) + held_delta = -amount + return balance_delta, held_delta + + +def pagination(cursor, limit): + if (not isinstance(cursor, str) or not re.fullmatch(r'[0-9]{1,16}', cursor) + or int(cursor) > MAX_AMOUNT or type(limit) is not int or not 1 <= limit <= 100): + raise CreditsError('invalid_credits_request', 400) + return int(cursor) + + +def public_entry(row): + return {key: row[key] for key in ('operation_id', 'revision', 'kind', 'amount_microusd', + 'balance_delta', 'held_delta', 'scoped_id', 'reference', 'tariff_revision')} + + +class SqlCreditsLedger: + """SQLite transactions serialize CAS and preserve event/balance atomicity.""" + def __init__(self, path=None, *, initialize=False): + self.path = Path(path) if path is not None else storage_path('USAGE_DB_PATH', 'usage.sqlite3') + if initialize: + self._run(lambda db: db.executescript((Path(__file__).resolve().parents[1] / 'intelligence-migrations' / MIGRATION).read_text()), check=False) + + def _run(self, action, *, check=True): + try: + if check and not self.path.is_file(): + raise CreditsError() + target = self.path.resolve().as_uri() + '?mode=rw' if check else str(self.path) + with closing(sqlite3.connect(target, timeout=10, uri=check)) as db, db: + db.row_factory = sqlite3.Row + db.execute('BEGIN IMMEDIATE') + if check: + db.execute('SELECT owner,operation_id,revision,kind,amount_microusd,balance_delta,held_delta,scoped_id,reference,tariff_revision,actor,reason,document FROM credits_entries LIMIT 0') + db.execute('SELECT owner,operation_id,revision,actor,kind FROM credits_audit LIMIT 0') + db.execute('SELECT balance_microusd, held_microusd, revision FROM credits_balances LIMIT 0') + return action(db) + except (sqlite3.Error, OSError): + raise CreditsError() from None + + def append(self, owner, kind, amount_microusd, operation_id, revision, **kwargs): + fields = request_fields(owner, kind, amount_microusd, operation_id, revision, **kwargs) + document = json.dumps(fields, sort_keys=True, ensure_ascii=False, separators=(',', ':')) + def write(db): + existing = db.execute('SELECT * FROM credits_entries WHERE owner=? AND operation_id=?', (owner, operation_id)).fetchone() + if existing: + if existing['document'] != document: + raise CreditsError('credits_conflict', 409) + return public_entry(existing) + row = db.execute('SELECT * FROM credits_balances WHERE owner=?', (owner,)).fetchone() + balance, held, current = (row['balance_microusd'], row['held_microusd'], row['revision']) if row else (0, 0, 0) + if current != revision: + raise CreditsError('credits_revision_mismatch', 412) + entries = [dict(item) for item in db.execute('SELECT * FROM credits_entries WHERE owner=? AND (scoped_id=? OR operation_id=? OR reference=?)', + (owner, fields['scoped_id'], fields['reference'], fields['reference']))] + bd, hd = deltas(fields, entries) + integer(balance + bd) + integer(held + hd, nonnegative=True) + integer(revision + 1, nonnegative=True) + available = balance + bd - held - hd + reservation = kind == 'reserve' and not fields['unknown'] + enforced_spend = reservation or (kind == 'adjust' and amount_microusd < 0) + if enforced_spend and (available < 0 or (reservation and balance - held <= 0)): + raise CreditsError('credits_insufficient', 409) + db.execute('INSERT OR IGNORE INTO credits_balances(owner) VALUES (?)', (owner,)) + changed = db.execute('UPDATE credits_balances SET balance_microusd=?, held_microusd=?, revision=revision+1 WHERE owner=? AND revision=?', + (balance + bd, held + hd, owner, revision)).rowcount + if changed != 1: + raise CreditsError('credits_revision_mismatch', 412) + db.execute('''INSERT INTO credits_entries + (owner,operation_id,revision,kind,amount_microusd,balance_delta,held_delta,scoped_id,reference,tariff_revision,actor,reason,document) + VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)''', + (owner, operation_id, revision + 1, kind, amount_microusd, bd, hd, fields['scoped_id'], fields['reference'], + fields['tariff_revision'], fields['actor'], fields['reason'], document)) + if fields['actor']: + db.execute('INSERT INTO credits_audit VALUES (?,?,?,?,?)', (owner, operation_id, revision + 1, fields['actor'], kind)) + return public_entry(db.execute('SELECT * FROM credits_entries WHERE owner=? AND operation_id=?', (owner, operation_id)).fetchone()) + return self._run(write) + + def read(self, owner, *, cursor='0', limit=100): + owner_id(owner) + after = pagination(cursor, limit) + def read(db): + row = db.execute('SELECT * FROM credits_balances WHERE owner=?', (owner,)).fetchone() + balance, held, revision = (row['balance_microusd'], row['held_microusd'], row['revision']) if row else (0, 0, 0) + rows = db.execute('SELECT * FROM credits_entries WHERE owner=? AND revision>? ORDER BY revision LIMIT ?', (owner, after, limit + 1)).fetchall() + page = rows[:limit] + return dict(currency='USD', balance_microusd=balance, held_microusd=held, available_microusd=balance-held, + revision=revision, entries=[public_entry(item) for item in page], + next_cursor=str(page[-1]['revision']) if len(rows) > limit else None) + return self._run(read) + + +class D1CreditsLedger: + """Private transport without retries or local fallback on authority failure.""" + def __init__(self, call): + self.call = call + + def _call(self, operation, **values): + try: + result = self.call({'version': 1, 'operation': operation, **values}) + if not isinstance(result, dict) or type(result.get('version')) is not int or result.get('version') != 1: + raise CreditsError() + if 'error' in result: + code = result['error'].get('code') + statuses = {'credits_conflict': 409, 'credits_insufficient': 409, 'credits_revision_mismatch': 412, + 'invalid_credits_request': 400, 'credits_unpriced': 503} + raise CreditsError(code if code in statuses else 'credits_unavailable', statuses.get(code, 503)) + try: + return _validate_private_result(operation, result, values) + except CreditsError: + raise CreditsError() from None + except CreditsError: + raise + except Exception as error: + statuses = {'credits_conflict': 409, 'credits_insufficient': 409, 'credits_revision_mismatch': 412, + 'invalid_credits_request': 400, 'credits_unpriced': 503} + code = getattr(error, 'code', None) + raise CreditsError(code if code in statuses else 'credits_unavailable', statuses.get(code, 503)) from None + + def append(self, owner, kind, amount_microusd, operation_id, revision, **kwargs): + fields = request_fields(owner, kind, amount_microusd, operation_id, revision, **kwargs) + return self._call('append', **fields) + + def read(self, owner, *, cursor='0', limit=100): + owner_id(owner) + pagination(cursor, limit) + return self._call('read', owner=owner, cursor=cursor, limit=limit) + + +def _validate_entry(value): + fields = {'operation_id', 'revision', 'kind', 'amount_microusd', 'balance_delta', 'held_delta', + 'scoped_id', 'reference', 'tariff_revision'} + if not isinstance(value, dict) or set(value) != fields: + raise CreditsError() + label(value['operation_id']) + integer(value['revision'], nonnegative=True) + for name in ('amount_microusd', 'balance_delta', 'held_delta'): + integer(value[name]) + for name in ('scoped_id', 'reference'): + if value[name] is not None: + label(value[name]) + if value['tariff_revision'] is not None: + integer(value['tariff_revision'], nonnegative=True) + kind, amount, bd, hd = (value[name] for name in ('kind', 'amount_microusd', 'balance_delta', 'held_delta')) + if kind in {'credit', 'adjust', 'compensate'}: + valid = bd == amount and hd == 0 and (kind != 'credit' or amount >= 0) + elif kind == 'reserve': + valid = amount >= 0 and bd == 0 and hd in {0, amount} and value['scoped_id'] is not None + elif kind in {'commit', 'release'}: + valid = amount >= 0 and hd <= 0 and bd == (-amount if kind == 'commit' else 0) and value['reference'] is not None + else: + valid = False + if not valid: + raise CreditsError() + + +def _validate_private_result(operation, result, values): + name = 'entry' if operation == 'append' else 'summary' + if set(result) != {'version', name}: + raise CreditsError() + value = result[name] + if operation == 'append': + _validate_entry(value) + if (any(value[key] != values[key] for key in ('operation_id', 'kind', 'amount_microusd', 'scoped_id', 'tariff_revision')) + or value['revision'] != values['revision'] + 1): + raise CreditsError() + return value + if not isinstance(value, dict) or set(value) != {'currency', 'balance_microusd', 'held_microusd', 'available_microusd', 'revision', 'entries', 'next_cursor'}: + raise CreditsError() + for field in ('balance_microusd', 'held_microusd', 'available_microusd', 'revision'): + integer(value[field], nonnegative=field in {'held_microusd', 'revision'}) + if (value['currency'] != 'USD' or value['available_microusd'] != value['balance_microusd'] - value['held_microusd'] + or not isinstance(value['entries'], list) or len(value['entries']) > values['limit']): + raise CreditsError() + after = int(values['cursor']) + for entry in value['entries']: + _validate_entry(entry) + if not after < entry['revision'] <= value['revision']: + raise CreditsError() + after = entry['revision'] + if value['next_cursor'] is not None and (not value['entries'] or value['next_cursor'] != str(after) or len(value['entries']) != values['limit']): + raise CreditsError() + return value + + +def open_store(): + from services import control_state_d1 + if control_state_d1.using_d1(): + def call(document): + values = {key: value for key, value in document.items() if key not in {'version', 'operation'}} + return control_state_d1.call('credits', document['operation'], **values) + return D1CreditsLedger(call) + if os.environ.get('INTELLIGENCE_STORAGE_BACKEND', '').strip(): + raise CreditsError() + return SqlCreditsLedger() + + +class LedgerCreditAuthority: + """Tariff returns its reviewed revision, or None when cost is unknown/unpriced.""" + def __init__(self, store, tariff): + self.store, self.tariff = store, tariff + + def _apply(self, operation, phase): + if type(operation) is not AuthorityOperation: + raise TypeError('AuthorityOperation required') + if not enabled(): + raise CreditDenied('credits_disabled', 404) + try: + tariff_revision = self.tariff(operation, phase) + unknown = phase == 'reconcile' and tariff_revision is None + if tariff_revision is None and not unknown: + raise CreditDenied('credits_unpriced') + kind = 'reserve' if phase == 'reserve' or unknown else 'commit' + entry = self.store.append(context_owner(operation.context), kind, 0 if unknown else operation.amount, + operation.operation_id, operation.revision, scoped_id=operation.scoped_id, + tariff_revision=tariff_revision, unknown=unknown) + return AuthorityResult(operation.context, operation.scoped_id, entry['revision'], operation.operation_id, True) + except CreditsError as error: + raise CreditDenied(error.code, error.status) from None + + def reserve(self, operation): + return self._apply(operation, 'reserve') + + def commit(self, operation): + return self._apply(operation, 'commit') + + def reconcile(self, operation): + return self._apply(operation, 'reconcile') + + +def register_credit_authority(store, *, tariff): + if not callable(tariff): + raise TypeError('Explicit operator tariff required') + return LedgerCreditAuthority(store, tariff) + + +def register_enterprise_credit(authority): + return register_enterprise_adapters(credit=authority) diff --git a/tests/fixtures/secret_firewall_coverage.json b/tests/fixtures/secret_firewall_coverage.json index 63220171..a5d9d554 100644 --- a/tests/fixtures/secret_firewall_coverage.json +++ b/tests/fixtures/secret_firewall_coverage.json @@ -40,6 +40,11 @@ "app.route('/users//rotate-key', methods=['POST'])", "app.route('/v1/metrics/prometheus', methods=['GET', 'OPTIONS'])" ], + "routes/credits.py": [ + "app.get('/admin/credits/')", + "app.post('/admin/credits/adjust')", + "app.route('/v1/credits', methods=['GET', 'OPTIONS'])" + ], "routes/dashboard_security.py": [ "app.route('/admin/audit', endpoint='audit_log')", "app.route(dashboard_sso.ACCESS_LOGIN_PATH, methods=['GET', 'POST'])" @@ -286,6 +291,7 @@ "worker/control-state-d1.mjs", "worker/control-users-d1.mjs", "worker/cors-policy.mjs", + "worker/credits-d1.mjs", "worker/d1-schema.mjs", "worker/enterprise-contract.mjs", "worker/exact-generation-cache.mjs", diff --git a/tests/test_credits_ledger.py b/tests/test_credits_ledger.py new file mode 100644 index 00000000..4d3cb02f --- /dev/null +++ b/tests/test_credits_ledger.py @@ -0,0 +1,398 @@ +"""Local ledger, registered HTTP routes and immutable authority contracts.""" +import json +import sqlite3 +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +import pytest + +from services import credits_ledger as ledger +from services.enterprise_contract import AuthorityOperation, TenantContext, AuthorityDenied, call_authority + + +def vectors(): + text = (Path(__file__).resolve().parents[1] / 'docs/credits-ledger.md').read_text() + return json.loads(text.split('```json\n', 1)[1].split('```', 1)[0]) + + +@pytest.fixture +def store(tmp_path, monkeypatch): + monkeypatch.setenv('CREDITS_ENABLED', 'true') + monkeypatch.setenv('CREDITS_CURRENCY', 'USD') + return ledger.SqlCreditsLedger(tmp_path / 'credits.sqlite3', initialize=True) + + +def append(store, kind, amount, op, revision, **fields): + return store.append('alice', kind, amount, op, revision, **fields) + + +def test_shared_vectors(store): + for vector in vectors(): + if 'invalid_amount' in vector: + with pytest.raises(ledger.CreditsError) as error: + append(store, 'credit', vector['invalid_amount'], 'bad', 0) + assert error.value.status == 400 + continue + values = vector['request'] + if vector.get('error'): + with pytest.raises(ledger.CreditsError) as error: + store.append(**values) + assert error.value.code == vector['error'] + else: + result = store.append(**values) + assert result['revision'] == vector['revision'] + if 'summary' in vector: + summary = store.read(values['owner']) + assert {key: summary[key] for key in vector['summary']} == vector['summary'] + summary = store.read('alice') + assert (summary['balance_microusd'], summary['held_microusd'], summary['available_microusd']) == (75, 0, 75) + + +def test_concurrent_reserves_and_overflow(store): + append(store, 'credit', 100, 'fund', 0) + def reserve(op): + try: + return append(store, 'reserve', 80, op, 1, scoped_id=op, tariff_revision=1) + except ledger.CreditsError as error: + return error.code + with ThreadPoolExecutor(max_workers=2) as pool: + results = list(pool.map(reserve, ['hedge', 'shadow'])) + assert sum(isinstance(result, dict) for result in results) == 1 + assert store.read('alice')['available_microusd'] == 20 + with pytest.raises(ledger.CreditsError): + append(store, 'credit', ledger.MAX_AMOUNT, 'overflow', 2) + assert store.read('alice')['revision'] == 2 + + +def test_unknown_hold_and_each_attempt_charge(store): + append(store, 'credit', 100, 'fund', 0) + authority = ledger.register_credit_authority(store, tariff=lambda op, phase: None if phase == 'reconcile' else 7) + adapters = ledger.register_enterprise_credit(authority) + def op(rev, name, amount, attempt='hedge'): + return AuthorityOperation(TenantContext('alice'), attempt, rev, name, amount) + call_authority(adapters, 'credit', 'reserve', op(1, 'hold1', 50)) + call_authority(adapters, 'credit', 'reconcile', op(2, 'unknown1', 0)) + assert store.read('alice')['held_microusd'] == 50 + call_authority(adapters, 'credit', 'commit', op(3, 'charge1', 20)) + call_authority(adapters, 'credit', 'reserve', op(4, 'hold2', 40, 'batch')) + call_authority(adapters, 'credit', 'commit', op(5, 'charge2', 30, 'batch')) + assert store.read('alice')['balance_microusd'] == 50 + denied = ledger.register_credit_authority(store, tariff=lambda *_: None) + with pytest.raises(AuthorityDenied, match='credits_unpriced'): + denied.reserve(op(6, 'unpriced', 0, 'new')) + + +def test_compensation_is_linked_and_original_unchanged(store): + original = append(store, 'credit', 100, 'fund', 0) + append(store, 'adjust', -20, 'debit', 1, reason='reviewed', actor='admin') + append(store, 'compensate', 20, 'correction', 2, reference='debit', reason='correction', actor='admin') + assert store.read('alice')['balance_microusd'] == 100 + assert store.read('alice')['entries'][0] == original + with pytest.raises(ledger.CreditsError): + append(store, 'compensate', 20, 'double', 3, reference='debit', reason='correction', actor='admin') + assert store.read('bob')['entries'] == [] + assert store.read('alice', limit=1)['next_cursor'] == '1' + assert store.read('alice', cursor='1', limit=1)['entries'][0]['operation_id'] == 'debit' + + +def test_missing_schema_and_default_off(tmp_path, caplog): + missing = ledger.SqlCreditsLedger(tmp_path / 'missing.sqlite3') + with pytest.raises(ledger.CreditsError, match='credits_unavailable'): + missing.append('alice', 'credit', 1, 'fund', 0) + assert not missing.path.exists() + for flag in ['', 'false', '0', 'bad']: + assert not ledger.enabled({'CREDITS_ENABLED': flag}) + assert ledger.enabled({'CREDITS_ENABLED': 'true', 'CREDITS_CURRENCY': ''}) + for _ in range(2): + assert not ledger.enabled({'CREDITS_ENABLED': 'true', 'CREDITS_CURRENCY': 'EUR'}) + assert caplog.text.count('Invalid CREDITS_CURRENCY') == 1 + + +def test_migration_preserves_old_rows_and_missing_columns_fail_closed(tmp_path): + path = tmp_path / 'old.sqlite3' + migration = (Path(__file__).resolve().parents[1] / 'intelligence-migrations' / ledger.MIGRATION).read_text() + with sqlite3.connect(path) as db: + db.execute('CREATE TABLE old_usage (cost INTEGER)') + db.execute('INSERT INTO old_usage VALUES (NULL)') + db.executescript(migration) + db.executescript(migration) + assert db.execute('SELECT cost FROM old_usage').fetchone() == (None,) + broken = tmp_path / 'broken.sqlite3' + with sqlite3.connect(broken) as db: + db.execute('CREATE TABLE credits_balances (owner TEXT PRIMARY KEY, balance_microusd INTEGER, held_microusd INTEGER, revision INTEGER)') + db.execute('CREATE TABLE credits_entries (owner TEXT)') + db.execute('CREATE TABLE credits_audit (owner TEXT)') + store = ledger.SqlCreditsLedger(broken) + for action in (lambda: store.read('alice'), lambda: append(store, 'credit', 1, 'fund', 0)): + with pytest.raises(ledger.CreditsError, match='credits_unavailable'): + action() + with sqlite3.connect(broken) as db: + assert db.execute('SELECT count(*) FROM credits_balances').fetchone()[0] == 0 + + +def test_spend_above_estimate_closes_only_its_hold(store): + append(store, 'credit', 100, 'fund', 0) + append(store, 'reserve', 30, 'hedge', 1, scoped_id='hedge', tariff_revision=1) + append(store, 'reserve', 60, 'shadow', 2, scoped_id='shadow', tariff_revision=1) + append(store, 'commit', 41, 'charge', 3, scoped_id='hedge', tariff_revision=1) + summary = store.read('alice') + assert (summary['balance_microusd'], summary['held_microusd'], summary['available_microusd']) == (59, 60, -1) + append(store, 'reserve', 0, 'unknown', 4, scoped_id='shadow', unknown=True) + assert store.read('alice')['held_microusd'] == 60 + append(store, 'release', 60, 'release', 5, scoped_id='shadow') + with pytest.raises(ledger.CreditsError, match='credits_conflict'): + append(store, 'commit', 1, 'late', 6, scoped_id='shadow', tariff_revision=1) + assert store.read('alice')['balance_microusd'] == 59 + + +@pytest.mark.parametrize('recovery', ['credit', 'adjust']) +def test_commit_overrun_blocks_new_spend_until_funded(store, recovery): + append(store, 'credit', 100, 'fund', 0) + authority = ledger.register_credit_authority(store, tariff=lambda *_: 1) + def op(revision, name, amount, attempt='attempt'): + return AuthorityOperation(TenantContext('alice'), attempt, revision, name, amount) + authority.reserve(op(1, 'hold', 60)) + with pytest.raises(ledger.CreditDenied, match='credits_revision_mismatch') as error: + authority.commit(op(1, 'charge', 150)) + assert error.value.status == 412 + charged = authority.commit(op(2, 'charge', 150)) + assert authority.commit(op(2, 'charge', 150)) == charged + summary = store.read('alice') + assert (summary['balance_microusd'], summary['held_microusd'], summary['available_microusd']) == (-50, 0, -50) + assert summary['revision'] == 3 + assert ledger.D1CreditsLedger(lambda _: dict(version=1, summary=summary)).read('alice') == summary + for amount in (0, 1): + with pytest.raises(ledger.CreditDenied, match='credits_insufficient'): + authority.reserve(op(3, f'denied{amount}', amount, f'new{amount}')) + with pytest.raises(ledger.CreditsError, match='credits_insufficient'): + append(store, 'adjust', -1, 'debit', 3, actor='admin', reason='reviewed') + with pytest.raises(ledger.CreditDenied, match='credits_conflict'): + authority.commit(op(3, 'late', 1)) + with pytest.raises(ledger.CreditDenied, match='credits_conflict'): + authority.commit(op(2, 'charge', 151)) + extra = dict(actor='admin', reason='reviewed') if recovery == 'adjust' else {} + append(store, recovery, 25, 'partial', 3, **extra) + assert store.read('alice')['available_microusd'] == -25 + append(store, recovery, 25, 'zero', 4, **extra) + with pytest.raises(ledger.CreditDenied, match='credits_insufficient'): + authority.reserve(op(5, 'zero-denied', 0, 'zero-attempt')) + append(store, recovery, 50, 'recovered', 5, **extra) + assert store.read('alice')['available_microusd'] == 50 + authority.reserve(op(6, 'next', 1, 'next')) + assert store.read('alice')['available_microusd'] == 49 + + +@pytest.mark.parametrize('balance,held', [(-ledger.MAX_AMOUNT, 0), (-50, 10), (ledger.MAX_AMOUNT, 0)]) +def test_private_summary_accepts_signed_balances(store, balance, held): + summary = dict(currency='USD', balance_microusd=balance, held_microusd=held, + available_microusd=balance-held, revision=0, entries=[], next_cursor=None) + assert ledger.D1CreditsLedger(lambda _: dict(version=1, summary=summary)).read('alice') == summary + invalid = [dict(held_microusd=-1), dict(revision=-1), dict(balance_microusd=-ledger.MAX_AMOUNT-1), + dict(available_microusd=-ledger.MAX_AMOUNT-1), dict(balance_microusd=True)] + for change in invalid: + with pytest.raises(ledger.CreditsError, match='credits_unavailable'): + ledger.D1CreditsLedger(lambda _: dict(version=1, summary={**summary, **change})).read('alice') + + +def test_duplicate_concurrency_pagination_and_content_free_audit(store): + def fund(_): + return append(store, 'adjust', 100, 'fund', 0, actor='admin', reason='Reviewed allocation') + with ThreadPoolExecutor(max_workers=2) as pool: + results = list(pool.map(fund, range(2))) + assert results[0] == results[1] + for revision in range(1, 102): + append(store, 'adjust', 1, f'grant{revision}', revision, actor='admin', reason='Reviewed allocation') + page = store.read('alice') + assert len(page['entries']) == 100 + assert page['next_cursor'] == '100' + assert len(store.read('alice', cursor='100')['entries']) == 2 + with sqlite3.connect(store.path) as db: + assert [field[1] for field in db.execute('PRAGMA table_info(credits_audit)')] == ['owner', 'operation_id', 'revision', 'actor', 'kind'] + assert db.execute('SELECT count(*) FROM credits_audit').fetchone()[0] == 102 + balance, held = db.execute('SELECT SUM(balance_delta),SUM(held_delta) FROM credits_entries').fetchone() + assert (balance, held) == (page['balance_microusd'], page['held_microusd']) + for kwargs in ({'limit': 101}, {'limit': True}, {'cursor': '-1'}, {'cursor': '1.5'}): + with pytest.raises(ledger.CreditsError): + store.read('alice', **kwargs) + + +def test_tenant_namespace_and_d1_outage_have_no_local_fallback(store): + context = TenantContext('alice', org_id='org', team_id='team') + owner = ledger.context_owner(context) + store.append(owner, 'credit', 100, 'fund', 0) + assert store.read('alice')['balance_microusd'] == 0 + assert owner != ledger.context_owner(TenantContext('alice', org_id='other', team_id='team')) + d1 = ledger.D1CreditsLedger(lambda _: (_ for _ in ()).throw(OSError('private transport unavailable'))) + with pytest.raises(ledger.CreditsError, match='credits_unavailable'): + d1.read('alice') + assert store.read('alice')['balance_microusd'] == 0 + + +def test_private_transport_validates_response_and_preserves_errors(store): + entry = append(store, 'credit', 100, 'fund', 0) + response = dict(version=1, entry=entry) + d1 = ledger.D1CreditsLedger(lambda _: response) + assert d1.append('alice', 'credit', 100, 'fund', 0) == entry + for altered in ({**entry, 'amount_microusd': True}, {**entry, 'operation_id': 'foreign'}, + {**entry, 'revision': 2}, {**entry, 'prompt': 'private'}, {**entry, 'balance_delta': 99}): + with pytest.raises(ledger.CreditsError) as error: + ledger.D1CreditsLedger(lambda _: dict(version=1, entry=altered)).append('alice', 'credit', 100, 'fund', 0) + assert error.value.code == 'credits_unavailable' and error.value.status == 503 + summary = store.read('alice') + assert ledger.D1CreditsLedger(lambda _: dict(version=1, summary=summary)).read('alice') == summary + for altered in ({**summary, 'available_microusd': 101}, {**summary, 'next_cursor': 'bogus'}, {**summary, 'prompt': 'private'}): + with pytest.raises(ledger.CreditsError, match='credits_unavailable'): + ledger.D1CreditsLedger(lambda _: dict(version=1, summary=altered)).read('alice') + with pytest.raises(ledger.CreditsError) as error: + ledger.D1CreditsLedger(lambda _: dict(version=1, error={'code': 'credits_revision_mismatch'})).read('alice') + assert error.value.status == 412 + + +def test_disabled_authority_never_prices_or_changes_storage(store, monkeypatch): + def forbidden(*args): + pytest.fail('disabled tariff evaluated') + authority = ledger.register_credit_authority(store, tariff=forbidden) + monkeypatch.setenv('CREDITS_ENABLED', '') + with pytest.raises(AuthorityDenied, match='credits_disabled'): + authority.reserve(AuthorityOperation(TenantContext('alice'), 'attempt', 0, 'disabled', 0)) + assert store.read('alice')['revision'] == 0 + + +def test_registered_authority_denial_stops_provider_before_handoff(tmp_path, monkeypatch): + from flask import Flask + from types import SimpleNamespace + from routes import credits + monkeypatch.setenv('CREDITS_ENABLED', 'true') + monkeypatch.setenv('CREDITS_CURRENCY', 'USD') + store = ledger.SqlCreditsLedger(tmp_path / 'missing.sqlite3') + authority = ledger.register_credit_authority(store, tariff=lambda *_: 1) + app = Flask('credits-admission') + credits.register_credits_routes(app, SimpleNamespace(exempt=lambda fn: fn), store=store) + provider_calls = [] + def generation(): + authority.reserve(AuthorityOperation(TenantContext('alice'), 'attempt', 0, 'hold', 1)) + provider_calls.append('called') + return 'provider result' + app.add_url_rule('/generation', 'generation', generation) + response = app.test_client().get('/generation') + assert response.status_code == 503 and response.json['error']['code'] == 'credits_unavailable' + assert provider_calls == [] and not store.path.exists() + + +def test_registered_real_auth_scope_and_missing_schema(store, tmp_path, monkeypatch): + from flask import Flask + from types import SimpleNamespace + from routes import credits + helpers = credits.api_authenticate_only.__globals__ + def authenticate(key, address): + if key == 'models-test': + return {'username': 'alice', 'scopes': ['models']} + if key == 'chat-test': + return {'username': 'alice', 'scopes': ['chat']} + return None + monkeypatch.setattr(helpers['AuthService'], 'verify_api_key', authenticate) + monkeypatch.setattr(helpers['request_accounting'], 'check_key_controls', lambda user: None) + monkeypatch.setattr(helpers['request_accounting'], 'begin', lambda: None) + app = Flask(__name__) + app.config['SECRET_KEY'] = 'synthetic-test-secret' + credits.register_credits_routes(app, SimpleNamespace(exempt=lambda fn: fn), store=store) + client = app.test_client() + append(store, 'credit', 100, 'fund', 0) + assert client.get('/v1/credits', headers={'Authorization': 'Bearer chat-test'}).status_code == 403 + assert client.get('/v1/credits').status_code == 401 + assert client.get('/v1/credits?owner=bob', headers={'Authorization': 'Bearer models-test'}).json['balance_microusd'] == 100 + assert client.get('/v1/credits?limit=101', headers={'Authorization': 'Bearer models-test'}).status_code == 400 + assert client.get('/v1/credits?cursor=-1', headers={'Authorization': 'Bearer models-test'}).status_code == 400 + missing = ledger.SqlCreditsLedger(tmp_path / 'absent.sqlite3') + other = Flask('missing-credits') + other.config['SECRET_KEY'] = 'synthetic-test-secret' + credits.register_credits_routes(other, SimpleNamespace(exempt=lambda fn: fn), store=missing) + response = other.test_client().get('/v1/credits', headers={'Authorization': 'Bearer models-test'}) + assert response.status_code == 503 and response.json['error']['code'] == 'credits_unavailable' + assert not missing.path.exists() + + +def test_real_admin_session_csrf_audit_and_disabled_response_bytes(store, monkeypatch): + from flask import Flask + from flask_wtf.csrf import CSRFProtect, generate_csrf + from routes import credits + session_authority = credits.login_required.__globals__['AuthService'] + admin_authority = credits.require_admin_dashboard_user.__globals__['AuthService'] + signed_in = {'value': False} + user = {'username': 'member', 'is_admin': False} + monkeypatch.setattr(session_authority, 'is_authenticated', lambda: signed_in['value']) + monkeypatch.setattr(admin_authority, 'get_current_user', lambda: user) + app = Flask('credits-admin-security') + app.config.update(SECRET_KEY='synthetic-test-secret', TESTING=True) + app.add_url_rule('/login', 'login', lambda: 'login') + app.add_url_rule('/csrf-token', 'token', lambda: {'csrf_token': generate_csrf()}) + app.add_url_rule('/existing', 'existing', lambda: ('unchanged', 201, {'X-Existing': 'value'})) + csrf = CSRFProtect(app) + credits.register_credits_routes(app, csrf, store=store) + client = app.test_client() + assert client.get('/admin/credits/alice').status_code == 302 + signed_in['value'] = True + assert client.get('/admin/credits/alice').status_code == 403 + token = client.get('/csrf-token').json['csrf_token'] + body = dict(owner='alice', amount_microusd=100, operation_id='fund', reason='Reviewed allocation') + assert client.post('/admin/credits/adjust', json=body, headers={'If-Match': '"0"', 'X-CSRFToken': token}).status_code == 403 + user.update(username='admin', is_admin=True) + assert client.post('/admin/credits/adjust', json=body, headers={'If-Match': '"0"'}).status_code == 400 + result = client.post('/admin/credits/adjust', json=body, headers={'If-Match': '"0"', 'X-CSRFToken': token}) + assert result.status_code == 200 and result.headers['ETag'] == '"1"' + assert client.post('/admin/credits/adjust', json=body, headers={'If-Match': '"0"', 'X-CSRFToken': token}).json == result.json + with sqlite3.connect(store.path) as db: + assert db.execute('SELECT actor FROM credits_audit').fetchone() == ('admin',) + before = client.get('/existing') + monkeypatch.setenv('CREDITS_ENABLED', '') + after = client.get('/existing') + assert (after.data, after.status_code, dict(after.headers)) == (before.data, before.status_code, dict(before.headers)) + assert client.post('/admin/credits/adjust', json=body).status_code == 404 + assert store.read('alice')['balance_microusd'] == 100 + + +def test_registered_routes_auth_cas_and_no_store(store, monkeypatch): + from flask import Flask, g + from flask_wtf.csrf import CSRFProtect + from routes import credits + import route_helpers + # Decorator collaborators are injected before route registration. + def auth(required_scope): + assert required_scope == 'models' + def decorate(fn): + def wrapped(*args, **kwargs): + from flask import request + if request.headers.get('Authorization') != 'Bearer synthetic': + return {'error': 'unauthorized'}, 401 + g.authenticated_user = {'username': 'alice'} + return fn(*args, **kwargs) + wrapped.__name__ = fn.__name__ + return wrapped + return decorate + monkeypatch.setattr(credits, 'api_authenticate_only', auth) + monkeypatch.setattr(credits, 'login_required', lambda fn: fn) + monkeypatch.setattr(credits, 'require_admin_dashboard_user', lambda: {'username': 'admin'}) + monkeypatch.setattr(route_helpers, 'login_required', lambda fn: fn) + monkeypatch.setenv('CREDITS_ENABLED', 'false') + monkeypatch.setenv('CREDITS_CURRENCY', 'USD') + app = Flask(__name__) + app.config.update(SECRET_KEY='synthetic-test-secret', TESTING=True) + csrf = CSRFProtect(app) + credits.register_credits_routes(app, csrf, store=store) + client = app.test_client() + assert client.get('/v1/credits').status_code == 404 + assert client.post('/admin/credits/adjust', json={}).status_code == 404 + monkeypatch.setenv('CREDITS_ENABLED', 'true') + assert client.get('/v1/credits').status_code == 401 + response = client.get('/v1/credits?owner=bob', headers={'Authorization': 'Bearer synthetic'}) + assert response.json['entries'] == [] + assert response.headers['Cache-Control'] == 'no-store' + assert client.post('/admin/credits/adjust', json={}).status_code == 400 + app.config['WTF_CSRF_ENABLED'] = False + body = dict(owner='alice', amount_microusd=100, operation_id='grant', reason='reviewed') + assert client.post('/admin/credits/adjust', json=body).status_code == 428 + assert client.post('/admin/credits/adjust', json=body, headers={'If-Match': '"1"'}).status_code == 412 + assert client.post('/admin/credits/adjust', json=body, headers={'If-Match': '"0"'}).status_code == 200 + assert client.get('/admin/credits/alice').headers['ETag'] == '"1"' + monkeypatch.setattr(credits, 'require_admin_dashboard_user', lambda: (_ for _ in ()).throw(ledger.CreditsError('admin_required', 403))) + assert client.get('/admin/credits/alice').status_code == 403 diff --git a/tests/test_credits_ledger_worker.mjs b/tests/test_credits_ledger_worker.mjs new file mode 100644 index 00000000..e99b2f01 --- /dev/null +++ b/tests/test_credits_ledger_worker.mjs @@ -0,0 +1,226 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { readFileSync } from 'node:fs'; +import { DatabaseSync } from 'node:sqlite'; +import { spawnSync } from 'node:child_process'; +import { handleCreditsRequest, appendCredit, readCredits, registerCreditAuthority, creditsEnabled, contextOwner } from '../worker/credits-d1.mjs'; +import { AuthorityOperation, TenantContext, callAuthority, registerEnterpriseAdapters } from '../worker/enterprise-contract.mjs'; + +const migration = readFileSync(new URL('../intelligence-migrations/0037_credits_ledger.sql', import.meta.url), 'utf8'); +const vectors = JSON.parse(readFileSync(new URL('../docs/credits-ledger.md', import.meta.url), 'utf8').split('```json\n')[1].split('```')[0]); + +function database(t, schema = true) { + const sql = new DatabaseSync(':memory:'); + t.after(() => sql.close()); + if (schema) sql.exec(migration); + const db = { prepare(query) { + let values = []; + const statement = { + bind(...args) { values = args; return statement; }, + first() { return sql.prepare(query).get(...values) ?? null; }, + all() { + const prepared = sql.prepare(query); + if (prepared.columns().length) return { results: prepared.all(...values), success: true }; + return { results: [], success: true, meta: { changes: Number(prepared.run(...values).changes) } }; + }, + }; + return statement; + }, async batch(statements) { + sql.exec('BEGIN IMMEDIATE'); + try { + const rows = statements.map(statement => statement.all()); + sql.exec('COMMIT'); + return rows; + } catch (error) { sql.exec('ROLLBACK'); throw error; } + } }; + const env = { CREDITS_ENABLED: 'true', CREDITS_CURRENCY: 'USD', INTELLIGENCE_DB: db }; + const call = async body => { + const response = await handleCreditsRequest(new Request('http://intelligence.internal/v1/credits', { + method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ version: 1, ...body }), + }), env); + return { status: response.status, ...await response.json() }; + }; + return { sql, db, env, call }; +} + +const entry = (kind, amount, operation_id, revision, extra = {}) => ({ owner: 'alice', kind, + amount_microusd: amount, operation_id, revision, ...extra }); + +test('shared integer, insufficient and duplicate operation vectors', async t => { + const { call } = database(t); + for (const vector of vectors) { + const request = vector.request ?? entry('credit', vector.invalid_amount, 'bad', 0); + const result = await call({ operation: 'append', ...request }); + if ('invalid_amount' in vector) assert.equal(result.status, 400); + else if (vector.error) assert.equal(result.error.code, vector.error); + else assert.equal(result.entry.revision, vector.revision); + if (vector.summary) { + const { summary } = await call({ operation: 'read', owner: request.owner }); + for (const [key, value] of Object.entries(vector.summary)) assert.equal(summary[key], value); + } + } + const summary = (await call({ operation: 'read', owner: 'alice' })).summary; + assert.equal(summary.balance_microusd, 75); + assert.equal(summary.held_microusd, 0); + assert.equal(summary.available_microusd, 75); +}); + +test('concurrent CAS reserves and duplicate requests spend once', async t => { + const { db } = database(t); + await appendCredit(db, entry('credit', 100, 'fund', 0)); + const reserve = name => appendCredit(db, entry('reserve', 80, name, 1, { scoped_id: name, tariff_revision: 1 })); + const results = await Promise.allSettled([reserve('hedge'), reserve('shadow')]); + assert.equal(results.filter(result => result.status === 'fulfilled').length, 1); + assert.equal((await readCredits(db, 'alice')).available_microusd, 20); + const { db: other } = database(t); + const fund = entry('credit', 100, 'fund', 0); + const duplicates = await Promise.all([appendCredit(other, fund), appendCredit(other, fund)]); + assert.deepEqual(duplicates[0], duplicates[1]); + assert.equal((await readCredits(other, 'alice')).balance_microusd, 100); +}); + +test('unknown reconciliation retains hold and all attempts settle independently', async t => { + const { db, env } = database(t); + await appendCredit(db, entry('credit', 100, 'fund', 0)); + const authority = registerCreditAuthority(db, { env, tariff: (_op, phase) => phase === 'reconcile' ? null : 7 }); + const adapters = registerEnterpriseAdapters({ credit: authority }); + const op = (revision, operation_id, amount, scoped_id = 'hedge') => new AuthorityOperation({ + context: new TenantContext({ principal_id: 'alice' }), revision, operation_id, amount, scoped_id }); + await callAuthority(adapters, 'credit', 'reserve', op(1, 'hold1', 50)); + await callAuthority(adapters, 'credit', 'reconcile', op(2, 'unknown1', 0)); + assert.equal((await readCredits(db, 'alice')).held_microusd, 50); + await callAuthority(adapters, 'credit', 'commit', op(3, 'charge1', 20)); + await callAuthority(adapters, 'credit', 'reserve', op(4, 'hold2', 40, 'batch')); + await callAuthority(adapters, 'credit', 'commit', op(5, 'charge2', 30, 'batch')); + assert.equal((await readCredits(db, 'alice')).balance_microusd, 50); + await assert.rejects(registerCreditAuthority(db, { env, tariff: () => null }).reserve(op(6, 'unpriced', 0, 'new')), /credits_unpriced/); + const disabled = registerCreditAuthority({ prepare() { assert.fail('disabled storage'); } }, { tariff: () => assert.fail('disabled tariff') }); + await assert.rejects(disabled.reserve(op(6, 'disabled', 0, 'new')), /credits_disabled/); +}); + +test('compensation, releases, overflow and cursor owner isolation', async t => { + const { db } = database(t); + const original = await appendCredit(db, entry('credit', 100, 'fund', 0)); + await appendCredit(db, entry('adjust', -20, 'debit', 1, { actor: 'admin', reason: 'reviewed' })); + await appendCredit(db, entry('compensate', 20, 'correction', 2, { actor: 'admin', reason: 'reviewed', reference: 'debit' })); + await assert.rejects(appendCredit(db, entry('compensate', 20, 'twice', 3, { actor: 'admin', reason: 'reviewed', reference: 'debit' })), /credits_conflict/); + await assert.rejects(appendCredit(db, entry('credit', 1000000000000000, 'overflow', 3)), /invalid_credits_request/); + assert.deepEqual((await readCredits(db, 'alice')).entries[0], original); + assert.deepEqual((await readCredits(db, 'bob')).entries, []); + assert.equal((await readCredits(db, 'alice', { limit: 1 })).next_cursor, '1'); + assert.equal((await readCredits(db, 'alice', { limit: 1, cursor: '1' })).entries[0].operation_id, 'debit'); + await appendCredit(db, entry('reserve', 40, 'hold', 3, { scoped_id: 'new', tariff_revision: 1 })); + await appendCredit(db, entry('release', 40, 'release', 4, { scoped_id: 'new' })); + assert.equal((await readCredits(db, 'alice')).held_microusd, 0); +}); + +test('private boundary, disabled traffic and missing schema fail closed', async t => { + const { call, env, db } = database(t, false); + assert.equal((await call({ operation: 'append', ...entry('credit', 1, 'fund', 0) })).error.code, 'credits_unavailable'); + assert.equal((await call({ operation: 'read', owner: 'alice' })).status, 503); + for (const flag of [undefined, '', 'false', 'bad']) { + const response = await handleCreditsRequest(new Request('http://intelligence.internal/v1/credits', { method: 'POST' }), { + CREDITS_ENABLED: flag, INTELLIGENCE_DB: { prepare() { assert.fail('disabled storage'); } }, + }); + assert.equal(response.status, 404); + } + const foreign = await handleCreditsRequest(new Request('https://example.test/v1/credits', { method: 'POST' }), env); + assert.equal(foreign.status, 404); + const { call: healthy } = database(t); + assert.equal((await healthy({ operation: 'append', ...entry('credit', 1, 'fund', 0), prompt: 'content' })).status, 400); + assert.equal((await healthy({ operation: 'append', ...entry('reserve', 1, 'hold', 0, { scoped_id: 'attempt' }) })).error.code, 'credits_unpriced'); + for (const currency of ['EUR', false, null]) assert.equal(creditsEnabled({ CREDITS_ENABLED: 'true', CREDITS_CURRENCY: currency }), false); + assert.equal(creditsEnabled({ CREDITS_ENABLED: 'true', CREDITS_CURRENCY: '' }), true); + let dispatched = false; + const operation = new AuthorityOperation({ context: new TenantContext({ principal_id: 'alice' }), + scoped_id: 'attempt', operation_id: 'hold', revision: 0, amount: 1 }); + try { + await registerCreditAuthority(db, { env, tariff: () => 1 }).reserve(operation); + dispatched = true; + } catch (error) { + assert.equal(error.status, 503); + assert.equal((await error.response().json()).error.code, 'credits_unavailable'); + } + assert.equal(dispatched, false); +}); + +test('actual spend above estimate closes only its hold', async t => { + const { db } = database(t); + await appendCredit(db, entry('credit', 100, 'fund', 0)); + await appendCredit(db, entry('reserve', 30, 'hedge', 1, { scoped_id: 'hedge', tariff_revision: 1 })); + await appendCredit(db, entry('reserve', 60, 'shadow', 2, { scoped_id: 'shadow', tariff_revision: 1 })); + await appendCredit(db, entry('commit', 41, 'charge', 3, { scoped_id: 'hedge', tariff_revision: 1 })); + const summary = await readCredits(db, 'alice'); + assert.deepEqual([summary.balance_microusd, summary.held_microusd, summary.available_microusd], [59, 60, -1]); + await appendCredit(db, entry('reserve', 0, 'unknown', 4, { scoped_id: 'shadow', unknown: true })); + assert.equal((await readCredits(db, 'alice')).held_microusd, 60); + await appendCredit(db, entry('release', 60, 'release', 5, { scoped_id: 'shadow' })); + await assert.rejects(appendCredit(db, entry('commit', 1, 'late', 6, { scoped_id: 'shadow', tariff_revision: 1 })), /credits_conflict/); + assert.equal((await readCredits(db, 'alice')).balance_microusd, 59); +}); + +for (const recovery of ['credit', 'adjust']) test(`commit overrun blocks new spend until ${recovery} funding`, async t => { + const { db, env, call } = database(t); + await appendCredit(db, entry('credit', 100, 'fund', 0)); + const authority = registerCreditAuthority(db, { env, tariff: () => 1 }); + const op = (revision, operation_id, amount, scoped_id = 'attempt') => new AuthorityOperation({ + context: new TenantContext({ principal_id: 'alice' }), revision, operation_id, amount, scoped_id }); + await authority.reserve(op(1, 'hold', 60)); + await assert.rejects(authority.commit(op(1, 'charge', 150)), error => error.code === 'credits_revision_mismatch' && error.status === 412); + const charged = await authority.commit(op(2, 'charge', 150)); + assert.deepEqual(await authority.commit(op(2, 'charge', 150)), charged); + const { status, summary } = await call({ operation: 'read', owner: 'alice' }); + assert.equal(status, 200); + assert.deepEqual([summary.balance_microusd, summary.held_microusd, summary.available_microusd], [-50, 0, -50]); + assert.equal(summary.revision, 3); + for (const amount of [0, 1]) await assert.rejects(authority.reserve(op(3, `denied${amount}`, amount, `new${amount}`)), /credits_insufficient/); + await assert.rejects(appendCredit(db, entry('adjust', -1, 'debit', 3, { actor: 'admin', reason: 'reviewed' })), /credits_insufficient/); + await assert.rejects(authority.commit(op(3, 'late', 1)), /credits_conflict/); + await assert.rejects(authority.commit(op(2, 'charge', 151)), /credits_conflict/); + const extra = recovery === 'adjust' ? { actor: 'admin', reason: 'reviewed' } : {}; + await appendCredit(db, entry(recovery, 25, 'partial', 3, extra)); + assert.equal((await readCredits(db, 'alice')).available_microusd, -25); + await appendCredit(db, entry(recovery, 25, 'zero', 4, extra)); + await assert.rejects(authority.reserve(op(5, 'zero-denied', 0, 'zero-attempt')), /credits_insufficient/); + await appendCredit(db, entry(recovery, 50, 'recovered', 5, extra)); + assert.equal((await readCredits(db, 'alice')).available_microusd, 50); + await authority.reserve(op(6, 'next', 1, 'next')); + assert.equal((await readCredits(db, 'alice')).available_microusd, 49); +}); + +test('migration rehearsal, incomplete columns, audit and bounded pagination', async t => { + const { db, sql } = database(t); + sql.exec('CREATE TABLE old_usage (cost INTEGER); INSERT INTO old_usage VALUES (NULL)'); + sql.exec(migration); + assert.equal(sql.prepare('SELECT cost FROM old_usage').get().cost, null); + for (let revision = 0; revision < 102; revision++) { + await appendCredit(db, entry('adjust', 1, `grant${revision}`, revision, { actor: 'admin', reason: 'Reviewed allocation' })); + } + const first = await readCredits(db, 'alice'); + assert.equal(first.entries.length, 100); + assert.equal(first.next_cursor, '100'); + assert.equal((await readCredits(db, 'alice', { cursor: '100' })).entries.length, 2); + assert.equal(sql.prepare('SELECT count(*) AS n FROM credits_audit').get().n, 102); + assert.deepEqual(sql.prepare('PRAGMA table_info(credits_audit)').all().map(row => row.name), ['owner', 'operation_id', 'revision', 'actor', 'kind']); + for (const options of [{ limit: 101 }, { limit: true }, { cursor: '-1' }, { cursor: '1.5' }]) await assert.rejects(readCredits(db, 'alice', options), /invalid_credits_request/); + const { db: broken, sql: partial } = database(t, false); + partial.exec(`CREATE TABLE credits_balances (owner TEXT PRIMARY KEY,balance_microusd INTEGER,held_microusd INTEGER,revision INTEGER); + CREATE TABLE credits_entries (owner TEXT); CREATE TABLE credits_audit (owner TEXT)`); + await assert.rejects(readCredits(broken, 'alice'), /credits_unavailable/); + await assert.rejects(appendCredit(broken, entry('credit', 1, 'fund', 0)), /credits_unavailable/); + assert.equal(partial.prepare('SELECT count(*) AS n FROM credits_balances').get().n, 0); + const org = new TenantContext({ principal_id: 'alice', org_id: 'org', team_id: 'team' }); + assert.equal(contextOwner(org), 'tenant:["org","team","alice"]'); + assert.notEqual(contextOwner(org), contextOwner(new TenantContext({ principal_id: 'alice', org_id: 'other', team_id: 'team' }))); +}); + +test('Python and D1 return identical ledger vectors', async t => { + const { db } = database(t); + for (const vector of vectors.filter(vector => vector.request && !vector.error)) await appendCredit(db, vector.request); + const code = `import os,sys,json,tempfile\nfrom pathlib import Path\nsys.path.insert(0,os.getcwd())\nfrom services.credits_ledger import SqlCreditsLedger\ntext=Path('docs/credits-ledger.md').read_text()\nvectors=json.loads(text.split('\x60\x60\x60json\\n',1)[1].split('\x60\x60\x60',1)[0])\nwith tempfile.TemporaryDirectory() as d:\n store=SqlCreditsLedger(Path(d)/'credits.sqlite3',initialize=True)\n for v in vectors:\n if 'request' in v and not v.get('error'): store.append(**v['request'])\n print(json.dumps(store.read('alice')))\n`; + const python = spawnSync('/home/mysterious/storage/github/MultiLLM-Proxy/.venv/bin/python', ['-I', '-c', code], { + cwd: new URL('..', import.meta.url), encoding: 'utf8', timeout: 10000, + }); + assert.equal(python.status, 0, python.stderr); + assert.deepEqual(await readCredits(db, 'alice'), JSON.parse(python.stdout)); +}); diff --git a/worker/container-env.mjs b/worker/container-env.mjs index 017e9d73..054dfd01 100644 --- a/worker/container-env.mjs +++ b/worker/container-env.mjs @@ -40,6 +40,8 @@ const DIRECT_ENV_KEYS = [ "CONNECTION_PROFILES_DB_PATH", "CONFIG_REVISION_SYNC_ENABLED", "CONFIG_SYNC_TTL_SECONDS", + "CREDITS_CURRENCY", + "CREDITS_ENABLED", "ORGANISATIONS_ENABLED", "SAML_BROKER_URL", "SAML_CALLBACK_URL", diff --git a/worker/credits-d1.mjs b/worker/credits-d1.mjs new file mode 100644 index 00000000..7e1f0588 --- /dev/null +++ b/worker/credits-d1.mjs @@ -0,0 +1,273 @@ +/** Private append-only integer credits with atomic D1 balance revisions. */ +import { AuthorityDenied, AuthorityOperation, AuthorityResult, TenantContext } from './enterprise-contract.mjs'; + +export const MAX_AMOUNT = 1_000_000_000_000_000; +const warned = new Set(); +const entryKeys = ['operation_id', 'revision', 'kind', 'amount_microusd', 'balance_delta', + 'held_delta', 'scoped_id', 'reference', 'tariff_revision']; + +export class CreditsError extends Error { + constructor(code = 'credits_unavailable', status = 503) { + super(code); this.code = code; this.status = status; + } +} + +export class CreditDenied extends AuthorityDenied { + constructor(code, status = 503) { super(code); this.code = code; this.status = status; } + response() { + return Response.json({ error: { code: this.code, message: 'Credits operation unavailable.' } }, { + status: this.status, headers: { 'Cache-Control': 'no-store' }, + }); + } +} + +export function creditsEnabled(env = {}, warn = message => console.warn(message)) { + const raw = env.CREDITS_ENABLED; + const flag = raw === undefined ? '' : typeof raw === 'string' ? raw.trim().toLowerCase() : 'invalid'; + const invalid = name => { + if (!warned.has(name)) { warned.add(name); warn(`Invalid ${name}; credits disabled`); } + return false; + }; + if (['', '0', 'false', 'no', 'off'].includes(flag)) return false; + if (!['1', 'true', 'yes', 'on'].includes(flag)) return invalid('CREDITS_ENABLED'); + if ((env.CREDITS_CURRENCY === undefined || env.CREDITS_CURRENCY === '' ? 'USD' : env.CREDITS_CURRENCY) !== 'USD') { + return invalid('CREDITS_CURRENCY'); + } + return true; +} + +function integer(value, nonnegative = false) { + if (!Number.isSafeInteger(value) || Math.abs(value) > MAX_AMOUNT || nonnegative && value < 0) { + throw new CreditsError('invalid_credits_request', 400); + } + return value; +} +function label(value) { + if (typeof value !== 'string' || !/^[A-Za-z0-9_:.\-]{1,128}$/.test(value) || /\n|\r/.test(value)) { + throw new CreditsError('invalid_credits_request', 400); + } + return value; +} +function ownerId(value) { + if (typeof value !== 'string' || value.length < 1 || value.length > 512 || /[\x00-\x1f\x7f]/.test(value)) { + throw new CreditsError('invalid_credits_request', 400); + } + return value; +} +function reasonText(value) { + if (typeof value !== 'string' || value.length < 1 || value.length > 512 || /[^\x20-\x7e]/.test(value)) { + throw new CreditsError('invalid_credits_request', 400); + } + return value; +} +export function contextOwner(context) { + if (!(context instanceof TenantContext)) throw new CreditsError('invalid_credits_request', 400); + return context.org_id === null ? ownerId(context.principal_id) + : `tenant:${JSON.stringify([context.org_id, context.team_id, context.principal_id])}`; +} +function requestFields(data) { + const { owner, kind, amount_microusd, operation_id, revision, scoped_id = null, reference = null, + tariff_revision = null, actor = null, reason = null, unknown = false } = data; + ownerId(owner); label(operation_id); integer(amount_microusd); integer(revision, true); + if (!['credit', 'reserve', 'commit', 'release', 'adjust', 'compensate'].includes(kind) || typeof unknown !== 'boolean') { + throw new CreditsError('invalid_credits_request', 400); + } + for (const value of [scoped_id, reference, actor]) if (value !== null) label(value); + if (['reserve', 'commit', 'release'].includes(kind)) { label(scoped_id); integer(amount_microusd, true); } + if (['reserve', 'commit'].includes(kind) && !unknown) { + if (tariff_revision === null) throw new CreditsError('credits_unpriced'); + integer(tariff_revision, true); + } else if (tariff_revision !== null) integer(tariff_revision, true); + if (kind === 'credit') integer(amount_microusd, true); + if (['adjust', 'compensate'].includes(kind)) { label(actor); reasonText(reason); } + else if (reason !== null) reasonText(reason); + if (kind === 'compensate') label(reference); + else if (reference !== null) throw new CreditsError('invalid_credits_request', 400); + if (unknown && (kind !== 'reserve' || amount_microusd !== 0)) throw new CreditsError('invalid_credits_request', 400); + return { owner, kind, amount_microusd, operation_id, revision, scoped_id, reference, tariff_revision, actor, reason, unknown }; +} +const documentFor = fields => JSON.stringify(Object.fromEntries(Object.entries(fields).sort(([a], [b]) => a.localeCompare(b)))); +const publicEntry = row => Object.fromEntries(entryKeys.map(key => [key, row[key]])); + +function deltas(fields, entries) { + const { kind, amount_microusd: amount } = fields; + let bd = 0, hd = 0; + if (['credit', 'adjust'].includes(kind)) bd = amount; + else if (kind === 'compensate') { + const original = entries.find(row => row.operation_id === fields.reference); + if (!original || !['credit', 'adjust', 'commit'].includes(original.kind) || amount !== -original.balance_delta || + entries.some(row => row.kind === 'compensate' && row.reference === fields.reference)) throw new CreditsError('credits_conflict', 409); + bd = amount; + } else { + const attempt = entries.filter(row => row.scoped_id === fields.scoped_id); + const hold = attempt.find(row => row.kind === 'reserve' && row.reference === null); + if (kind === 'reserve' && !fields.unknown) { + if (hold) throw new CreditsError('credits_conflict', 409); + hd = amount; + } else { + if (!hold || attempt.some(row => ['commit', 'release'].includes(row.kind))) throw new CreditsError('credits_conflict', 409); + fields.reference = hold.operation_id; + if (kind === 'commit') { bd = -amount; hd = -hold.amount_microusd; } + else if (kind === 'release') { + if (amount !== hold.amount_microusd) throw new CreditsError('invalid_credits_request', 400); + hd = -amount; + } + } + } + return [bd, hd]; +} + +async function schema(db) { + if (!db || typeof db.prepare !== 'function' || typeof db.batch !== 'function') throw new CreditsError(); + // Probe all required columns before any owner initialization or write. + await db.batch([ + db.prepare('SELECT owner,balance_microusd,held_microusd,revision FROM credits_balances LIMIT 0'), + db.prepare('SELECT owner,operation_id,revision,kind,amount_microusd,balance_delta,held_delta,scoped_id,reference,tariff_revision,actor,reason,document FROM credits_entries LIMIT 0'), + db.prepare('SELECT owner,operation_id,revision,actor,kind FROM credits_audit LIMIT 0'), + ]); +} + +export async function appendCredit(db, data) { + const fields = requestFields(data); + const document = documentFor(fields); + try { + await schema(db); + const find = () => db.prepare('SELECT * FROM credits_entries WHERE owner=? AND operation_id=?') + .bind(fields.owner, fields.operation_id).first(); + const replay = row => { + if (row.document !== document) throw new CreditsError('credits_conflict', 409); + return publicEntry(row); + }; + const existing = await find(); + if (existing) return replay(existing); + const row = await db.prepare('SELECT * FROM credits_balances WHERE owner=?').bind(fields.owner).first(); + const balance = row?.balance_microusd ?? 0, held = row?.held_microusd ?? 0, revision = row?.revision ?? 0; + if (revision !== fields.revision) { + const duplicate = await find(); + if (duplicate) return replay(duplicate); + throw new CreditsError('credits_revision_mismatch', 412); + } + const entries = (await db.prepare('SELECT * FROM credits_entries WHERE owner=? AND (scoped_id=? OR operation_id=? OR reference=?)') + .bind(fields.owner, fields.scoped_id, fields.reference, fields.reference).all()).results; + const [bd, hd] = deltas(fields, entries); + integer(balance + bd); integer(held + hd, true); integer(revision + 1, true); + const available = balance + bd - held - hd; + const reservation = fields.kind === 'reserve' && !fields.unknown; + const enforcedSpend = reservation || (fields.kind === 'adjust' && fields.amount_microusd < 0); + if (enforcedSpend && (available < 0 || (reservation && balance - held <= 0))) { + throw new CreditsError('credits_insufficient', 409); + } + const { owner, operation_id, kind, amount_microusd, scoped_id, reference, tariff_revision, actor, reason } = fields; + try { + await db.batch([ + db.prepare('INSERT OR IGNORE INTO credits_balances(owner) VALUES (?)').bind(owner), + db.prepare(`INSERT INTO credits_entries + (owner,operation_id,revision,kind,amount_microusd,balance_delta,held_delta,scoped_id,reference,tariff_revision,actor,reason,document) + SELECT ?,?,?,?,?,?,?,?,?,?,?,?,? FROM credits_balances WHERE owner=? AND revision=?`) + .bind(owner, operation_id, revision + 1, kind, amount_microusd, bd, hd, scoped_id, reference, + tariff_revision, actor, reason, document, owner, revision), + db.prepare(`UPDATE credits_balances SET balance_microusd=?,held_microusd=?,revision=revision+1 + WHERE owner=? AND revision=? AND EXISTS + (SELECT 1 FROM credits_entries WHERE owner=? AND operation_id=? AND revision=? AND document=?)`) + .bind(balance + bd, held + hd, owner, revision, owner, operation_id, revision + 1, document), + db.prepare(`INSERT OR IGNORE INTO credits_audit SELECT owner,operation_id,revision,actor,kind + FROM credits_entries WHERE owner=? AND operation_id=? AND actor IS NOT NULL`) + .bind(owner, operation_id), + ]); + } catch (error) { + const duplicate = await find(); + if (duplicate) return replay(duplicate); + throw error; + } + const saved = await find(); + if (!saved) throw new CreditsError('credits_revision_mismatch', 412); + return replay(saved); + } catch (error) { throw error instanceof CreditsError ? error : new CreditsError(); } +} + +export async function readCredits(db, owner, { cursor = '0', limit = 100 } = {}) { + ownerId(owner); + if (typeof cursor !== 'string' || !/^[0-9]{1,16}$/.test(cursor) || /\n|\r/.test(cursor) || + Number(cursor) > MAX_AMOUNT || !Number.isInteger(limit) || limit < 1 || limit > 100) { + throw new CreditsError('invalid_credits_request', 400); + } + try { + await schema(db); + const results = await db.batch([ + db.prepare('SELECT * FROM credits_balances WHERE owner=?').bind(owner), + db.prepare('SELECT * FROM credits_entries WHERE owner=? AND revision>? ORDER BY revision LIMIT ?').bind(owner, Number(cursor), limit + 1), + ]); + const row = results[0].results[0], rows = results[1].results, page = rows.slice(0, limit); + const balance = row?.balance_microusd ?? 0, held = row?.held_microusd ?? 0; + return { currency: 'USD', balance_microusd: balance, held_microusd: held, available_microusd: balance - held, + revision: row?.revision ?? 0, entries: page.map(publicEntry), next_cursor: rows.length > limit ? String(page.at(-1).revision) : null }; + } catch (error) { throw error instanceof CreditsError ? error : new CreditsError(); } +} + +const reply = (body, status = 200) => Response.json({ version: 1, ...body }, { + status, headers: { 'Cache-Control': 'no-store' }, +}); + +async function boundedBody(request) { + const reader = request.body?.getReader(); + if (!reader) throw new CreditsError('invalid_credits_request', 400); + const parts = []; let size = 0; + try { + while (true) { + const { done, value } = await reader.read(); + if (done) break; + size += value.byteLength; + if (size > 16384) throw new CreditsError('invalid_credits_request', 400); + parts.push(value); + } + const bytes = new Uint8Array(size); let offset = 0; + for (const part of parts) { bytes.set(part, offset); offset += part.byteLength; } + return JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(bytes)); + } catch (error) { throw error instanceof CreditsError ? error : new CreditsError('invalid_credits_request', 400); } + finally { void reader.cancel().catch(() => {}); } +} + +export async function handleCreditsRequest(request, env) { + const url = new URL(request.url); + if (url.origin !== 'http://intelligence.internal' || url.username || url.password || url.search || url.hash || + !['/v1/credits', '/v1/managed-state/credits'].includes(url.pathname) || !creditsEnabled(env)) { + return reply({ error: { code: 'not_found', message: 'Credits operation unavailable.' } }, 404); + } + try { + if (request.method !== 'POST') return reply({ error: { code: 'method_not_allowed' } }, 405); + const body = await boundedBody(request); + if (!body || typeof body !== 'object' || Array.isArray(body) || body.version !== 1 || !['append', 'read'].includes(body.operation)) { + throw new CreditsError('invalid_credits_request', 400); + } + const required = body.operation === 'append' ? ['owner', 'kind', 'amount_microusd', 'operation_id', 'revision'] : ['owner']; + const optional = body.operation === 'append' ? ['scoped_id', 'reference', 'tariff_revision', 'actor', 'reason', 'unknown'] : ['cursor', 'limit']; + if (required.some(key => !(key in body)) || Object.keys(body).some(key => !['version', 'operation', ...required, ...optional].includes(key))) { + throw new CreditsError('invalid_credits_request', 400); + } + const db = env.INTELLIGENCE_DB; + return body.operation === 'append' ? reply({ entry: await appendCredit(db, body) }) + : reply({ summary: await readCredits(db, body.owner, body) }); + } catch (error) { + const failure = error instanceof CreditsError ? error : new CreditsError(); + return reply({ error: { code: failure.code, message: 'Credits operation unavailable.' } }, failure.status); + } +} + +export function registerCreditAuthority(db, { tariff, env = {} }) { + if (typeof tariff !== 'function') throw new TypeError('Explicit operator tariff required'); + const apply = async (operation, phase) => { + if (!(operation instanceof AuthorityOperation)) throw new TypeError('AuthorityOperation required'); + if (!creditsEnabled(env)) throw new CreditDenied('credits_disabled', 404); + try { + const tariff_revision = await tariff(operation, phase); + const unknown = phase === 'reconcile' && tariff_revision === null; + if (tariff_revision === null && !unknown) throw new CreditDenied('credits_unpriced'); + const row = await appendCredit(db, { owner: contextOwner(operation.context), kind: phase === 'reserve' || unknown ? 'reserve' : 'commit', + amount_microusd: unknown ? 0 : operation.amount, operation_id: operation.operation_id, revision: operation.revision, + scoped_id: operation.scoped_id, tariff_revision, unknown }); + return new AuthorityResult({ context: operation.context, scoped_id: operation.scoped_id, + operation_id: operation.operation_id, revision: row.revision, allowed: true }); + } catch (error) { throw error instanceof CreditsError ? new CreditDenied(error.code, error.status) : error; } + }; + return Object.freeze({ reserve: op => apply(op, 'reserve'), commit: op => apply(op, 'commit'), reconcile: op => apply(op, 'reconcile') }); +} diff --git a/worker/d1-schema.mjs b/worker/d1-schema.mjs index 0d77910f..49694b2d 100644 --- a/worker/d1-schema.mjs +++ b/worker/d1-schema.mjs @@ -10,7 +10,7 @@ export const REQUIRED_D1_TABLES = Object.freeze(["intelligence_policy", "intelli "usage_events", "usage_daily", "usage_batches", "media_jobs", "media_job_items", "shadow_eval_samples", "shadow_eval_results", "shadow_eval_config", "shadow_eval_policy_backups", "prompt_templates", - "config_snapshot_revisions", "config_snapshots", "config_snapshot_applications", "control_revisions", "tool_grants", "generation_cache", "usage_reservation_budgets", "usage_reservations", "usage_reservation_transitions", "session_tiers", "gateway_alert_rules", "gateway_alert_events", "managed_idempotency", "context_pages", "canary_traffic", "semantic_generation_cache", "gateway_batch_files", "gateway_batches", "gateway_batch_items", "hosted_responses", "realtime_tickets", "realtime_sessions", "learned_cooldown", "usage_receipt_keys", "usage_receipt_heads", "usage_receipts", "tenant_organisations", "tenant_teams", "tenant_memberships", "tenant_bindings", "tenant_audit", "tenant_governance_policies", "tenant_governance_reservations", "tenant_governance_components", "tenant_governance_baselines", "tenant_governance_audit", "saml_requests", "saml_subject_links", "saml_audit", "scim_resources", "scim_group_members", "scim_token_digests", "scim_audit"]); + "config_snapshot_revisions", "config_snapshots", "config_snapshot_applications", "control_revisions", "tool_grants", "generation_cache", "usage_reservation_budgets", "usage_reservations", "usage_reservation_transitions", "session_tiers", "gateway_alert_rules", "gateway_alert_events", "managed_idempotency", "context_pages", "canary_traffic", "semantic_generation_cache", "gateway_batch_files", "gateway_batches", "gateway_batch_items", "hosted_responses", "realtime_tickets", "realtime_sessions", "learned_cooldown", "usage_receipt_keys", "usage_receipt_heads", "usage_receipts", "tenant_organisations", "tenant_teams", "tenant_memberships", "tenant_bindings", "tenant_audit", "tenant_governance_policies", "tenant_governance_reservations", "tenant_governance_components", "tenant_governance_baselines", "tenant_governance_audit", "saml_requests", "saml_subject_links", "saml_audit", "scim_resources", "scim_group_members", "scim_token_digests", "scim_audit", "credits_balances", "credits_entries", "credits_audit"]); /** * Readiness for the D1 schema. A deploy that skipped `wrangler d1 migrations apply` reports