diff --git a/docs/usage-receipts.md b/docs/usage-receipts.md new file mode 100644 index 00000000..e4390595 --- /dev/null +++ b/docs/usage-receipts.md @@ -0,0 +1,141 @@ +# Signed usage receipts + +Usage receipts prove that the gateway signed a particular usage record. They do +not prove provider-billed truth, correct pricing, complete metering, payment, +invoicing, or external anchoring. Unknown costs and token counts remain `null`; +estimated and unknown evidence remains explicitly labelled. Records contain +usage metadata only. They do not contain prompts, completions, or credentials. + +## Configuration and storage + +`USAGE_RECEIPTS_ENABLED` is off by default. An empty value is off. Accepted true +values are `true`, `1`, `yes`, and `on`; false values are `false`, `0`, `no`, and +`off`, case-insensitively. An invalid value disables receipts with one warning +that does not print the value. While off, usage rows, storage and responses keep +their existing shape, and receipt routes return 404. + +For example, an enabled configuration names a key and a secret reference: + +```text +USAGE_RECEIPTS_ENABLED=true +USAGE_RECEIPTS_KEY_ID=usage-2026-10 +USAGE_RECEIPTS_SIGNING_KEY_REF=USAGE_RECEIPT_PRIVATE +``` + +The referenced environment variable or Worker secret binding contains a +base64-encoded DER PKCS#8 Ed25519 private key. The reference setting contains only +its binding name. Provision private material through the deployment's secret +mechanism. Neither responses nor errors expose it. Python uses `cryptography`; +the Worker uses WebCrypto Ed25519. Unsupported signing runtimes return 503. + +Apply `0032_usage_receipts.sql` to the usage database before enabling receipts. +It creates `usage_receipt_heads`, `usage_receipts`, and `usage_receipt_keys`. +Local SQL uses the ledger's database path. D1 uses the private managed-state +authority. Schema application and public-key review are explicit operator steps. +The migration is additive and does not rewrite existing usage rows. + +Independently review the public key, then insert its unique key ID, raw 32-byte +public key encoded as base64, and `reviewed=1` into `usage_receipt_keys`. An +unreviewed key is not published or used to sign. A reviewed active public key +must match the private key. Never reuse an existing key ID for different material. +To rotate, add a newly reviewed public key and change the active key ID/reference. +Keep the old reviewed public keys so old receipts remain verifiable. + +With receipts enabled, missing tables, missing or unusable private material, +or an unreviewed/mismatched active key makes receipt reads and writes unavailable +with a JSON 503 error. Usage recording continues. Receipt failures produce only +a fixed warning; they never authorize another provider request. + +## Reads and immutable corrections + +`GET /v1/usage/receipts/` requires authenticated model-read scope and returns +only a receipt owned by the caller. Its ID is the hexadecimal `record_hash`. +A missing receipt or another principal's receipt returns 404, including for an +administrator. `GET /v1/usage/receipt-keys` requires the same scope and returns +`{"keys":[{"key_id":"...","public_key_base64":"..."}]}`, with reviewed keys +only. Responses use `Cache-Control: no-store`. + +A receipt has exactly these fields: + +```json +{ + "record": {"cost_usd": null, "cost_basis": "unknown"}, + "canonical_bytes_base64": "...", + "signature_ed25519": "...", + "key_id": "usage-2026-10", + "previous_hash": null, + "record_hash": "...", + "sequence": 1 +} +``` + +Sequences start at 1 for each principal. Atomic compare-and-swap advances the +head and stores the receipt in one transaction. A stable event ID is unique +within its principal: repeating the same event and metadata returns the same +receipt, while changing that metadata under the same ID returns 409. Ledger +flush events use `:`, distinct from a provider request ID. +Use a new event ID for a settlement correction and include the old receipt's +hash as metadata such as `corrects`. The correction appends to the chain; it +never replaces the original record. Reservation reconciliation uses that same +settled-event interface with its immutable settlement/transition identity. + +## Canonical bytes and verification + +The decoded `canonical_bytes_base64` is the UTF-8 canonical JSON object containing +`version=1`, `principal`, `event_id`, `key_id`, `record`, `sequence`, and +`previous_hash`. Every unknown metadata field and every exact JSON null is kept. +Object keys sort by Unicode codepoint; arrays keep their order. Strings use JSON +escaping with scalar Unicode encoded directly in UTF-8. Lone surrogates are +rejected. Booleans remain distinct from numbers. + +Finite floating-point numbers use the exact decimal expansion of their IEEE-754 +value, without exponent notation. Negative zero becomes `0`. Integer values +must fit in the JavaScript safe-integer range. This is a versioned gateway +canonical format, rather than the shortest-number encoding used by JSON JCS. +For example, `0.1` becomes +`0.1000000000000000055511151231257827021181583404541015625`. + +Signed bytes are the ASCII prefix `MultiLLM usage receipt v1` followed by the +two literal characters backslash and lowercase `n`, followed by canonical JSON +bytes. `signature_ed25519` is the base64-encoded Ed25519 signature of those bytes; +`record_hash` is their SHA-256 digest in lowercase hexadecimal. The signed +context binds the key ID, principal, event, sequence and previous hash. + +Verify the signature with a trusted reviewed public key, recompute canonical +bytes and the hash, and compare all displayed fields with the signed payload. +For a complete chain, require sequence 1 and a null previous hash, followed by +consecutive sequence numbers and matching previous hashes. To verify a suffix, +supply an independently trusted starting sequence and hash. Reordering, missing +records and modified metadata fail verification. A chain cannot prove that a +tail was not truncated without an independently trusted latest head. Public key +delivery still depends on trusting the gateway or separately distributing keys. + +Both runtime test files check this public RFC 8032 test vector: + +```text +public_key_base64: 11qYAYKxCrfVS/7TyWQHOg7hcvPapiMlrwIaaPcHURo= +key_id: test-1 +canonical UTF-8: +{"event_id":"settlement-1","key_id":"test-1","previous_hash":null,"principal":"alice","record":{"cost_basis":"unknown","cost_usd":null,"extra":{"fraction":0.5,"nullable":null,"text":"é","zero":0}},"sequence":1,"version":1} +signature_ed25519: bj6tfJIYOfMT4NdUChDUckhAsA4PLn+ZRTkFoMgFkJ688dug9EXuyw01kUt39d17bJJ+qAP7rzhdROvEgJ9ADg== +record_hash: 500716026bf9836219938f4e6bba0cb435e940bcf6db76910f1ee9f6279ad256 +``` + +## Operational bounds and retention + +Canonical documents are limited to 64 KiB, 32 nested levels, and 4,096 visited +values/keys. The private request body is limited to 128 KiB. Content-bearing or +credential-bearing field names are rejected recursively; unknown fields must +still be content-free metadata. Event IDs and key IDs are limited to 128 safe +ASCII characters, principals to 256 Unicode characters. The Worker makes at +most 16 metadata CAS attempts before returning 409 on contention. + +Receipts are separate from usage rollups and raw-ledger pruning. There is no +automatic receipt or public-key deletion. Plan storage growth and access with +your retention policy; this feature does not create an immutable external +archive or override a legal deletion requirement. Receipts are written after +successful ledger storage, so a crash or unavailable signing authority can +leave a stored usage event without a receipt. Replaying its stable metadata +event ID can fill that gap without double metering or calling a provider. +Historic ledger rows are not automatically backfilled. Raw/passthrough request +or response bodies are never rewritten. diff --git a/intelligence-migrations/0032_usage_receipts.sql b/intelligence-migrations/0032_usage_receipts.sql new file mode 100644 index 00000000..c424647a --- /dev/null +++ b/intelligence-migrations/0032_usage_receipts.sql @@ -0,0 +1,22 @@ +CREATE TABLE IF NOT EXISTS usage_receipt_keys ( + key_id TEXT PRIMARY KEY, + public_key_base64 TEXT NOT NULL, + reviewed INTEGER NOT NULL DEFAULT 0 CHECK (reviewed IN (0, 1)) +); +CREATE TABLE IF NOT EXISTS usage_receipt_heads ( + principal TEXT PRIMARY KEY, + sequence INTEGER NOT NULL DEFAULT 0 CHECK (sequence >= 0), + record_hash TEXT +); +CREATE TABLE IF NOT EXISTS usage_receipts ( + id TEXT PRIMARY KEY, + principal TEXT NOT NULL, + event_id TEXT NOT NULL, + sequence INTEGER NOT NULL CHECK (sequence > 0), + previous_hash TEXT, + key_id TEXT NOT NULL REFERENCES usage_receipt_keys(key_id), + receipt_json TEXT NOT NULL, + UNIQUE (principal, sequence), + UNIQUE (principal, event_id) +); +CREATE INDEX IF NOT EXISTS idx_usage_receipts_principal ON usage_receipts(principal, sequence); diff --git a/package.json b/package.json index 445e4732..c9dedb41 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,7 @@ "lint": "ruff check . --select E9,F63,F7,F82", "typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming", "test": "python -m pytest -q", - "test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs tests/test_learned_cooldown_worker.mjs tests/test_stream_cost_breaker_worker.mjs tests/test_realtime_worker.mjs", + "test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs tests/test_usage_receipts_worker.mjs tests/test_learned_cooldown_worker.mjs tests/test_stream_cost_breaker_worker.mjs tests/test_realtime_worker.mjs", "db:migrate": "python scripts/validate_sqlite_schema.py --apply", "db:validate": "python scripts/validate_sqlite_schema.py", "security:secrets": "python scripts/check_static_secrets.py", diff --git a/routes/usage_receipts.py b/routes/usage_receipts.py new file mode 100644 index 00000000..4ae00300 --- /dev/null +++ b/routes/usage_receipts.py @@ -0,0 +1,42 @@ +"""Authenticated reads of immutable usage receipts and reviewed public keys.""" +from flask import g, jsonify + +from route_helpers import api_authenticate_only +from services import usage_receipts + + +def register_usage_receipt_routes(app, csrf, *, store=None): + def authority(): + if not usage_receipts.enabled(): + raise usage_receipts.ReceiptError("not_found", 404) + return store or usage_receipts.open_store() + + def failed(error): + return jsonify({"error": {"code": error.code, "message": "Usage receipt operation unavailable."}}), error.status + + @app.route("/v1/usage/receipts/", methods=["GET", "OPTIONS"]) + @csrf.exempt + @api_authenticate_only(required_scope="models") + def usage_receipt(id): + try: + user = g.authenticated_user + principal = str(user.get("username") or user.get("id") or "") + receipt = authority().get(principal, id) + if receipt is None: + raise usage_receipts.ReceiptError("not_found", 404) + response = jsonify(receipt) + response.headers["Cache-Control"] = "no-store" + return response + except usage_receipts.ReceiptError as error: + return failed(error) + + @app.route("/v1/usage/receipt-keys", methods=["GET", "OPTIONS"]) + @csrf.exempt + @api_authenticate_only(required_scope="models") + def usage_receipt_keys(): + try: + response = jsonify({"keys": authority().keys()}) + response.headers["Cache-Control"] = "no-store" + return response + except usage_receipts.ReceiptError as error: + return failed(error) diff --git a/services/usage_ledger.py b/services/usage_ledger.py index 75573dde..d8c5552d 100644 --- a/services/usage_ledger.py +++ b/services/usage_ledger.py @@ -20,7 +20,7 @@ from datetime import datetime, timedelta, timezone from typing import Any, Callable, Optional -from services import usage_store +from services import usage_store, usage_receipts logger = logging.getLogger(__name__) @@ -189,6 +189,7 @@ def flush_once(self) -> int: self._stats["recorded"] += len(batch["rows"]) self._stats["last_flush_at"] = utc_timestamp() self._notify(0, batch["rows"]) + usage_receipts.record_flushed_usage(batch["id"], batch["rows"]) return len(batch["rows"]) def flush(self, timeout: float = 5.0) -> bool: diff --git a/services/usage_receipts.py b/services/usage_receipts.py new file mode 100644 index 00000000..36819de2 --- /dev/null +++ b/services/usage_receipts.py @@ -0,0 +1,327 @@ +"""Opt-in, immutable Ed25519 evidence for content-free usage metadata.""" +from __future__ import annotations + +import base64 +import hashlib +import json +import logging +import math +import os +import re +from contextlib import closing +from decimal import Decimal +from functools import lru_cache +from pathlib import Path + +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey, Ed25519PublicKey + +from services import intelligence_d1_store, usage_store +from services.sqlite_store import connect + +logger = logging.getLogger(__name__) +DOMAIN = b"MultiLLM usage receipt v1\\n" +MAX_BYTES = 65536 +MAX_INTEGER = 2**53 - 1 +_LABEL = re.compile(r"[A-Za-z0-9_.:-]{1,128}\Z") +_REF = re.compile(r"[A-Za-z_][A-Za-z0-9_]{0,127}\Z") +_HASH = re.compile(r"[0-9a-f]{64}\Z") +_CONTENT_FIELDS = frozenset({"prompt", "messages", "content", "input", "output", "response", "api_key", + "authorization", "password", "secret", "private_key", "signing_key"}) + + +class ReceiptError(Exception): + def __init__(self, code="usage_receipts_unavailable", status=503): + super().__init__(code) + self.code, self.status = code, status + + +@lru_cache(maxsize=1) +def _warn_invalid_flag(): + logger.warning("Invalid USAGE_RECEIPTS_ENABLED; usage receipts disabled") + + +def enabled() -> bool: + flag = os.environ.get("USAGE_RECEIPTS_ENABLED", "").strip().lower() + if flag not in {"", "0", "false", "no", "off", "1", "true", "yes", "on"}: + _warn_invalid_flag() + return False + return flag in {"1", "true", "yes", "on"} + + +def canonical_bytes(value) -> bytes: + """Exact finite IEEE-754 decimals, scalar Unicode keys in codepoint order. + + Unlike runtime-specific shortest-number printers, exact decimal expansion is + identical in Python and WebCrypto callers. Integer values must be JS-safe. + """ + nodes, scalar_bytes = 0, 0 + def scalar(text): + nonlocal scalar_bytes + scalar_bytes += len(text.encode("utf-8")) + if scalar_bytes > MAX_BYTES: + raise ValueError() + return text + def encode(item, depth=0): + nonlocal nodes + nodes += 1 + if depth > 32 or nodes > 4096: + raise ValueError() + if item is None or type(item) is bool: + return scalar(json.dumps(item)) + if type(item) in (int, float): + if not math.isfinite(item) or (item == int(item) and abs(item) > MAX_INTEGER): + raise ValueError() + if item == 0: + return scalar("0") + if type(item) is int: + return scalar(str(item)) + text = format(Decimal.from_float(item), "f").rstrip("0").rstrip(".") if not item.is_integer() else str(int(item)) + return scalar(text) + if type(item) is str: + if len(item) > MAX_BYTES or any(0xD800 <= ord(char) <= 0xDFFF for char in item): + raise ValueError() + return scalar(json.dumps(item, ensure_ascii=False)) + if type(item) is list: + return "[" + ",".join(encode(child, depth + 1) for child in item) + "]" + if type(item) is dict and all(type(key) is str for key in item): + return "{" + ",".join(encode(key, depth + 1) + ":" + encode(item[key], depth + 1) + for key in sorted(item)) + "}" + raise ValueError() + try: + result = encode(value).encode("utf-8") + if len(result) > MAX_BYTES: + raise ValueError() + return result + except (ValueError, TypeError, OverflowError, UnicodeError, RecursionError): + raise ReceiptError("invalid_usage_receipt", 400) from None + + +def _identifier(value): + if not isinstance(value, str) or not _LABEL.fullmatch(value): + raise ReceiptError("invalid_usage_receipt", 400) + return value + + +def _principal(value): + if (not isinstance(value, str) or not 1 <= len(value) <= 256 + or any(ord(char) < 32 or ord(char) == 127 for char in value)): + raise ReceiptError("invalid_usage_receipt", 400) + canonical_bytes(value) + return value + + +def checked_record(record): + if type(record) is not dict: + raise ReceiptError("invalid_usage_receipt", 400) + def check(value): + if isinstance(value, dict): + for key, child in value.items(): + if not isinstance(key, str) or key.lower() in _CONTENT_FIELDS: + raise ReceiptError("invalid_usage_receipt", 400) + check(child) + elif isinstance(value, list): + for child in value: + check(child) + encoded = canonical_bytes(record) + check(record) + return json.loads(encoded) + + +def _signer(): + if not enabled(): + raise ReceiptError("not_found", 404) + key_id = os.environ.get("USAGE_RECEIPTS_KEY_ID", "") + reference = os.environ.get("USAGE_RECEIPTS_SIGNING_KEY_REF", "") + try: + if not _LABEL.fullmatch(key_id) or not _REF.fullmatch(reference): + raise ValueError() + material = os.environ.get(reference, "") + if not material or len(material) > 8192: + raise ValueError() + private = serialization.load_der_private_key(base64.b64decode(material, validate=True), password=None) + if not isinstance(private, Ed25519PrivateKey): + raise ValueError() + public = base64.b64encode(private.public_key().public_bytes( + serialization.Encoding.Raw, serialization.PublicFormat.Raw)).decode("ascii") + return key_id, private, public + except Exception: + raise ReceiptError() from None + + +def make_receipt(principal, event_id, record, sequence, previous_hash, signer): + key_id, private, _ = signer + payload = {"version": 1, "principal": principal, "event_id": event_id, "record": record, + "sequence": sequence, "previous_hash": previous_hash, "key_id": key_id} + canonical = canonical_bytes(payload) + signed = DOMAIN + canonical + return {"record": record, "canonical_bytes_base64": base64.b64encode(canonical).decode("ascii"), + "signature_ed25519": base64.b64encode(private.sign(signed)).decode("ascii"), + "key_id": key_id, "previous_hash": previous_hash, + "record_hash": hashlib.sha256(signed).hexdigest(), "sequence": sequence} + + +def verify_chain(receipts, public_keys, *, principal, sequence=0, previous_hash=None) -> bool: + """A suffix needs an independently trusted starting sequence and hash.""" + try: + for receipt in receipts: + canonical = base64.b64decode(receipt["canonical_bytes_base64"], validate=True) + payload = json.loads(canonical) + if (canonical != canonical_bytes(payload) or payload["version"] != 1 + or payload["principal"] != principal or payload["sequence"] != sequence + 1 + or payload["previous_hash"] != previous_hash + or any(canonical_bytes(payload[name]) != canonical_bytes(receipt[name]) + for name in ("record", "key_id", "sequence", "previous_hash"))): + return False + signed = DOMAIN + canonical + if hashlib.sha256(signed).hexdigest() != receipt["record_hash"]: + return False + public = Ed25519PublicKey.from_public_bytes(base64.b64decode(public_keys[receipt["key_id"]], validate=True)) + public.verify(base64.b64decode(receipt["signature_ed25519"], validate=True), signed) + sequence, previous_hash = receipt["sequence"], receipt["record_hash"] + return True + except Exception: + return False + + +class SqlReceiptStore: + """Uses the ledger database; schema application is an explicit operator step.""" + def __init__(self, path=None): + self.path = Path(path) if path is not None else usage_store.SqlUsageStore.path() + + def _run(self, operation): + if not enabled(): + raise ReceiptError("not_found", 404) + try: + with closing(connect(self.path)) as db, db: + db.execute("BEGIN IMMEDIATE") + return operation(db) + except ReceiptError: + raise + except Exception: + raise ReceiptError() from None + + @staticmethod + def _ready(db): + signer = _signer() + row = db.execute("SELECT public_key_base64 FROM usage_receipt_keys WHERE key_id = ? AND reviewed = 1", + (signer[0],)).fetchone() + if row is None or row["public_key_base64"] != signer[2]: + raise ReceiptError() + # Reads also fail closed if any part of the migration is absent. + db.execute("SELECT principal FROM usage_receipt_heads LIMIT 0") + db.execute("SELECT id FROM usage_receipts LIMIT 0") + return signer + + @staticmethod + def _public(row): + return json.loads(row["receipt_json"]) if row else None + + def keys(self): + def read(db): + self._ready(db) + return [dict(row) for row in db.execute("SELECT key_id, public_key_base64 FROM usage_receipt_keys " + "WHERE reviewed = 1 ORDER BY key_id")] + return self._run(read) + + def get(self, principal, identity): + _principal(principal) + if not isinstance(identity, str) or not _HASH.fullmatch(identity): + return None + def read(db): + self._ready(db) + return self._public(db.execute("SELECT receipt_json FROM usage_receipts WHERE principal = ? AND id = ?", + (principal, identity)).fetchone()) + return self._run(read) + + def find_event(self, principal, event_id): + def read(db): + self._ready(db) + return self._public(db.execute("SELECT receipt_json FROM usage_receipts WHERE principal = ? AND event_id = ?", + (_principal(principal), _identifier(event_id))).fetchone()) + return self._run(read) + + def append(self, principal, event_id, record): + _principal(principal) + _identifier(event_id) + record = checked_record(record) + def write(db): + signer = self._ready(db) + existing = self._public(db.execute("SELECT receipt_json FROM usage_receipts WHERE principal = ? AND event_id = ?", + (principal, event_id)).fetchone()) + if existing: + if canonical_bytes(existing["record"]) != canonical_bytes(record): + raise ReceiptError("usage_receipt_event_conflict", 409) + return existing + db.execute("INSERT INTO usage_receipt_heads (principal, sequence, record_hash) VALUES (?, 0, NULL) " + "ON CONFLICT(principal) DO NOTHING", (principal,)) + head = db.execute("SELECT sequence, record_hash FROM usage_receipt_heads WHERE principal = ?", (principal,)).fetchone() + receipt = make_receipt(principal, event_id, record, head["sequence"] + 1, head["record_hash"], signer) + db.execute("INSERT INTO usage_receipts (id, principal, event_id, sequence, previous_hash, key_id, receipt_json) " + "VALUES (?, ?, ?, ?, ?, ?, ?)", (receipt["record_hash"], principal, event_id, receipt["sequence"], + receipt["previous_hash"], receipt["key_id"], json.dumps(receipt, ensure_ascii=False))) + changed = db.execute("UPDATE usage_receipt_heads SET sequence = ?, record_hash = ? " + "WHERE principal = ? AND sequence = ? AND COALESCE(record_hash, '') = ?", + (receipt["sequence"], receipt["record_hash"], principal, head["sequence"], head["record_hash"] or "")).rowcount + if changed != 1: + raise ReceiptError("usage_receipt_conflict", 409) + return receipt + return self._run(write) + + +class D1ReceiptStore: + """The Worker owns signing and CAS; the private transport can be injected.""" + def __init__(self, *, call=None): + self.call = call or self._private + + @staticmethod + def _private(body): + return intelligence_d1_store.request_private_intelligence(body, endpoint="usage-receipts") + + def _call(self, body, field): + if not enabled(): + raise ReceiptError("not_found", 404) + try: + result = self.call(body) + if not isinstance(result, dict) or result.get("version") != 1 or field not in result: + raise ReceiptError() + return result[field] + except ReceiptError: + raise + except Exception: + raise ReceiptError() from None + + def append(self, principal, event_id, record): + return self._call({"operation": "append", "principal": _principal(principal), + "event_id": _identifier(event_id), "record": checked_record(record)}, "receipt") + + def get(self, principal, identity): + return self._call({"operation": "get", "principal": _principal(principal), "id": identity}, "receipt") + + def keys(self): + return self._call({"operation": "keys"}, "keys") + + +def open_store(): + return D1ReceiptStore() if usage_store.selected_backend() == "d1" else SqlReceiptStore() + + +def record_settled_usage(principal, event_id, record, *, store=None): + """Static settlement/reconciliation callback; failures never change metering.""" + if not enabled(): + return None + try: + return (store or open_store()).append(principal, event_id, record) + except Exception: + logger.warning("Usage receipt unavailable") + return None + + +def record_flushed_usage(batch_id, rows): + if not enabled(): + return + for index, row in enumerate(rows): + record = dict(row) + if record.get("cost_basis") is None: + record.setdefault("usage_basis", "unknown") + record_settled_usage(row.get("principal"), f"{batch_id}:{index}", record) diff --git a/tests/fixtures/secret_firewall_coverage.json b/tests/fixtures/secret_firewall_coverage.json index e48643c6..16606c2c 100644 --- a/tests/fixtures/secret_firewall_coverage.json +++ b/tests/fixtures/secret_firewall_coverage.json @@ -194,6 +194,10 @@ "app.route('/users//controls', methods=['PUT'])", "app.route('/v1/usage', methods=['GET', 'OPTIONS'])" ], + "routes/usage_receipts.py": [ + "app.route('/v1/usage/receipt-keys', methods=['GET', 'OPTIONS'])", + "app.route('/v1/usage/receipts/', methods=['GET', 'OPTIONS'])" + ], "routes/workbench.py": [ "app.get('/admin/workbench/catalog')", "app.get('/admin/workbench/deployment')", @@ -401,7 +405,8 @@ "worker/stream-cost-breaker.mjs", "worker/upstream-cancellation.mjs", "worker/usage-buckets-d1.mjs", - "worker/usage-ledger-d1.mjs" + "worker/usage-ledger-d1.mjs", + "worker/usage-receipts.mjs" ], "edge_route_literals": { "cloudflare-worker.mjs": [ diff --git a/tests/test_usage_receipts.py b/tests/test_usage_receipts.py new file mode 100644 index 00000000..316576a1 --- /dev/null +++ b/tests/test_usage_receipts.py @@ -0,0 +1,218 @@ +"""Immutable receipt storage, canonical evidence and authenticated read routes.""" +import base64 +import json +import sqlite3 +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from types import SimpleNamespace + +import pytest +from flask import Flask, g + +from services import usage_receipts as receipts, usage_ledger +from routes import usage_receipts as receipt_routes + +# Public RFC 8032 test material, never a deployment signing identity. +PRIVATE = "MC4CAQAwBQYDK2VwBCIEIJ1hsZ3v/VpguoRK9JLsLMREScVpezJpGXA7rAMcrn9g" +PUBLIC = "11qYAYKxCrfVS/7TyWQHOg7hcvPapiMlrwIaaPcHURo=" +RECORD = {"cost_usd": None, "cost_basis": "unknown", "extra": { + "fraction": 0.5, "zero": 0, "text": "é", "nullable": None}} +CANONICAL = '{"event_id":"settlement-1","key_id":"test-1","previous_hash":null,"principal":"alice","record":{"cost_basis":"unknown","cost_usd":null,"extra":{"fraction":0.5,"nullable":null,"text":"é","zero":0}},"sequence":1,"version":1}' +SIGNATURE = "bj6tfJIYOfMT4NdUChDUckhAsA4PLn+ZRTkFoMgFkJ688dug9EXuyw01kUt39d17bJJ+qAP7rzhdROvEgJ9ADg==" +HASH = "500716026bf9836219938f4e6bba0cb435e940bcf6db76910f1ee9f6279ad256" + + +@pytest.fixture +def store(tmp_path, monkeypatch): + monkeypatch.setenv("USAGE_RECEIPTS_ENABLED", "true") + monkeypatch.setenv("USAGE_RECEIPTS_KEY_ID", "test-1") + monkeypatch.setenv("USAGE_RECEIPTS_SIGNING_KEY_REF", "RECEIPT_TEST_PRIVATE") + monkeypatch.setenv("RECEIPT_TEST_PRIVATE", PRIVATE) + monkeypatch.setenv("CONTROL_PLANE_DATABASE_URL", "") + monkeypatch.setenv("USAGE_LEDGER_BACKEND", "sql") + path = tmp_path / "usage.sqlite3" + monkeypatch.setenv("USAGE_DB_PATH", str(path)) + with sqlite3.connect(path) as db: + db.execute("CREATE TABLE old_usage (cost REAL)") + db.execute("INSERT INTO old_usage VALUES (NULL)") + db.executescript((Path(__file__).resolve().parents[1] / "intelligence-migrations/0032_usage_receipts.sql").read_text()) + db.executescript((Path(__file__).resolve().parents[1] / "intelligence-migrations/0032_usage_receipts.sql").read_text()) + db.execute("INSERT INTO usage_receipt_keys VALUES (?, ?, 1)", ("test-1", PUBLIC)) + assert db.execute("SELECT cost FROM old_usage").fetchone() == (None,) + return receipts.SqlReceiptStore(path) + + +def test_cross_runtime_vector_and_unknown_fields(store): + result = store.append("alice", "settlement-1", RECORD) + assert base64.b64decode(result["canonical_bytes_base64"]) == CANONICAL.encode() + assert result["signature_ed25519"] == SIGNATURE + assert result["record_hash"] == HASH + assert result["record"] == RECORD + assert receipts.verify_chain([result], {"test-1": PUBLIC}, principal="alice") + + +@pytest.mark.parametrize("value", [float("nan"), float("inf"), 2**53, "\ud800", {1: None}]) +def test_non_interoperable_json_rejected(value): + with pytest.raises(receipts.ReceiptError): + receipts.canonical_bytes(value) + + +def test_exact_numbers_unicode_order_and_negative_zero(): + assert receipts.canonical_bytes({"z": -0.0, "n": 0.1, "😀": 1, "\ue000": 2}) == ( + '{"n":0.1000000000000000055511151231257827021181583404541015625,"z":0,"\ue000":2,"😀":1}'.encode()) + + +def test_concurrent_cas_duplicate_and_corrections(store): + with ThreadPoolExecutor(max_workers=4) as pool: + rows = list(pool.map(lambda n: store.append("alice", f"event-{n}", RECORD), range(12))) + ordered = sorted(rows, key=lambda row: row["sequence"]) + assert [row["sequence"] for row in ordered] == list(range(1, 13)) + assert receipts.verify_chain(ordered, {"test-1": PUBLIC}, principal="alice") + first = store.append("alice", "duplicate", RECORD) + assert store.append("alice", "duplicate", RECORD) == first + with pytest.raises(receipts.ReceiptError) as error: + store.append("alice", "duplicate", {**RECORD, "cost_usd": 0.5}) + assert error.value.status == 409 + correction = store.append("alice", "correction", {**RECORD, "cost_usd": 0.5, + "corrects": first["record_hash"]}) + assert correction["previous_hash"] == first["record_hash"] + assert store.get("alice", first["record_hash"])["record"]["cost_usd"] is None + assert store.get("bob", first["record_hash"]) is None + + +def test_tamper_reorder_chain_gap_and_cross_principal_fail(store): + a = store.append("alice", "a", RECORD) + b = store.append("alice", "b", RECORD) + assert not receipts.verify_chain([b, a], {"test-1": PUBLIC}, principal="alice") + assert not receipts.verify_chain([b], {"test-1": PUBLIC}, principal="alice") + assert not receipts.verify_chain([a, {**b, "sequence": 3}], {"test-1": PUBLIC}, principal="alice") + assert not receipts.verify_chain([{**a, "record": {"cost_usd": 0}}], {"test-1": PUBLIC}, principal="alice") + assert not receipts.verify_chain([{**a, "sequence": True}], {"test-1": PUBLIC}, principal="alice") + assert not receipts.verify_chain([a], {"test-1": PUBLIC}, principal="bob") + + +def test_reviewed_key_history_rotation_and_unusable_key(store, monkeypatch): + first = store.append("alice", "old", RECORD) + with sqlite3.connect(store.path) as db: + db.execute("INSERT INTO usage_receipt_keys VALUES ('unreviewed', ?, 0)", (PUBLIC,)) + db.execute("INSERT INTO usage_receipt_keys VALUES ('test-2', ?, 1)", (PUBLIC,)) + monkeypatch.setenv("USAGE_RECEIPTS_KEY_ID", "test-2") + second = store.append("alice", "new", RECORD) + keys = store.keys() + assert [key["key_id"] for key in keys] == ["test-1", "test-2"] + assert receipts.verify_chain([first, second], {key["key_id"]: key["public_key_base64"] for key in keys}, principal="alice") + for value in ("", "invalid"): + monkeypatch.setenv("RECEIPT_TEST_PRIVATE", value) + with pytest.raises(receipts.ReceiptError) as error: + store.append("alice", "missing", RECORD) + assert error.value.status == 503 + assert str(error.value) == "usage_receipts_unavailable" + + +def test_registered_owner_routes_disabled_and_unavailable(store, monkeypatch): + app = Flask(__name__) + def auth(*, required_scope): + def decorate(fn): + def wrapped(**values): + g.authenticated_user = {"username": "bob" if g.get("other") else "alice"} + return fn(**values) + wrapped.__name__ = fn.__name__ + return wrapped + return decorate + monkeypatch.setattr(receipt_routes, "api_authenticate_only", auth) + receipt_routes.register_usage_receipt_routes(app, SimpleNamespace(exempt=lambda fn: fn), store=store) + client = app.test_client() + item = store.append("alice", "route", RECORD) + assert client.get("/v1/usage/receipts/" + item["record_hash"]).get_json() == item + assert client.get("/v1/usage/receipts/" + "0" * 64).status_code == 404 + assert client.get("/v1/usage/receipt-keys").get_json() == {"keys": store.keys()} + with app.test_request_context(): + g.other = True + assert app.view_functions["usage_receipt"](id=item["record_hash"])[1] == 404 + monkeypatch.setenv("RECEIPT_TEST_PRIVATE", "") + assert client.get("/v1/usage/receipt-keys").status_code == 503 + monkeypatch.setenv("USAGE_RECEIPTS_ENABLED", "") + assert client.get("/v1/usage/receipt-keys").status_code == 404 + + +def test_registered_routes_use_real_authentication_and_scope_boundary(store, monkeypatch): + app = Flask(__name__) + helper_objects = receipt_routes.api_authenticate_only.__globals__ + accounting = helper_objects["request_accounting"] + def authenticate(key, address): + if key == "models-test": + return {"username": "alice", "scopes": ["models"]} + if key == "chat-test": + return {"username": "alice", "scopes": ["chat"]} + return None + monkeypatch.setattr(helper_objects["AuthService"], "verify_api_key", authenticate) + monkeypatch.setattr(accounting, "check_key_controls", lambda user: None) + monkeypatch.setattr(accounting, "begin", lambda: None) + monkeypatch.setattr(accounting, "finish", lambda response: response) + monkeypatch.setitem(helper_objects, "authorize_integration_route", lambda user: None) + receipt_routes.register_usage_receipt_routes(app, SimpleNamespace(exempt=lambda fn: fn), store=store) + client = app.test_client() + assert client.get("/v1/usage/receipt-keys").status_code == 401 + assert client.get("/v1/usage/receipt-keys", headers={"Authorization": "Bearer invalid-test"}).status_code == 401 + assert client.get("/v1/usage/receipt-keys", headers={"Authorization": "Bearer chat-test"}).status_code == 403 + assert client.get("/v1/usage/receipt-keys", headers={"Authorization": "Bearer models-test"}).status_code == 200 + + +def test_off_flags_no_storage_and_one_warning(monkeypatch, caplog): + for flag in ("", "false", "malformed", "malformed"): + monkeypatch.setenv("USAGE_RECEIPTS_ENABLED", flag) + receipts.record_settled_usage("alice", "event", RECORD, store=SimpleNamespace( + append=lambda *args: pytest.fail("disabled storage"))) + assert sum("Invalid USAGE_RECEIPTS_ENABLED" in row.message for row in caplog.records) == 1 + + +def test_default_ledger_rows_storage_and_response_unchanged(monkeypatch): + monkeypatch.setenv("USAGE_RECEIPTS_ENABLED", "") + monkeypatch.setenv("USAGE_LEDGER_ENABLED", "true") + monkeypatch.setattr(receipts, "open_store", lambda: pytest.fail("disabled receipt storage")) + ledger = usage_ledger.UsageLedger() + batches = [] + monkeypatch.setattr(ledger, "store", lambda: SimpleNamespace(record=lambda batch, rows: batches.extend(rows))) + monkeypatch.setattr(ledger, "_ensure_thread", lambda: None) + record = {"principal": "alice", "cost_usd": None, "cost_basis": None} + original = json.dumps(record, separators=(",", ":")) + assert ledger.record(record) is True and ledger.flush_once() == 1 + assert json.dumps(batches[0], separators=(",", ":")) == original + + +def test_missing_tables_and_content_limits_fail_closed(tmp_path, store): + missing = receipts.SqlReceiptStore(tmp_path / "missing.sqlite3") + with pytest.raises(receipts.ReceiptError) as error: + missing.append("alice", "a", RECORD) + assert error.value.status == 503 + for record in ({"prompt": "secret"}, {"extra": {"api_key": "secret"}}, {"extra": "x" * 65536}): + with pytest.raises(receipts.ReceiptError) as error: + store.append("alice", "invalid", record) + assert error.value.status == 400 + + +def test_ledger_flush_hook_does_not_change_usage_or_fail_it(store, monkeypatch): + ledger = usage_ledger.UsageLedger() + batches = [] + monkeypatch.setattr(ledger, "store", lambda: SimpleNamespace(record=lambda batch, rows: batches.append((batch, rows)))) + monkeypatch.setattr(ledger, "_ensure_thread", lambda: None) + monkeypatch.setenv("USAGE_LEDGER_ENABLED", "true") + monkeypatch.setattr(receipts, "open_store", lambda: store) + row = {"principal": "alice", "cost_usd": None, "cost_basis": None} + assert ledger.record(row) and ledger.flush_once() == 1 + event_id = f"{batches[0][0]}:0" + receipt = store.find_event("alice", event_id) + assert receipt["record"] == {**row, "usage_basis": "unknown"} + assert batches[0][1] == [row] + monkeypatch.setenv("RECEIPT_TEST_PRIVATE", "") + assert ledger.record(row) and ledger.flush_once() == 1 + + +def test_d1_transport_is_injected_and_dedicated(store): + calls = [] + def call(body): + calls.append(body) + return {"version": 1, "receipt": store.append(body["principal"], body["event_id"], body["record"])} + d1 = receipts.D1ReceiptStore(call=call) + assert d1.append("alice", "d1", RECORD)["sequence"] == 1 + assert calls == [{"operation": "append", "principal": "alice", "event_id": "d1", "record": RECORD}] diff --git a/tests/test_usage_receipts_worker.mjs b/tests/test_usage_receipts_worker.mjs new file mode 100644 index 00000000..847caa99 --- /dev/null +++ b/tests/test_usage_receipts_worker.mjs @@ -0,0 +1,192 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { readFileSync } from "node:fs"; +import { DatabaseSync } from "node:sqlite"; +import { convertV4MiniflareOptions, Miniflare } from "miniflare"; +import { canonicalBytes, appendReceipt, verifyChain, handleUsageReceiptRequest, handleUsageReceiptStoreRequest } from "../worker/usage-receipts.mjs"; +import { handleManagedStateRequest } from "../worker/managed-state-dispatch.mjs"; +import { handleUsageLedgerRequest } from "../worker/usage-ledger-d1.mjs"; + +// Public RFC 8032 vector; never a deployment identity. +const PRIVATE = "MC4CAQAwBQYDK2VwBCIEIJ1hsZ3v/VpguoRK9JLsLMREScVpezJpGXA7rAMcrn9g"; +const PUBLIC = "11qYAYKxCrfVS/7TyWQHOg7hcvPapiMlrwIaaPcHURo="; +const RECORD = { cost_usd: null, cost_basis: "unknown", extra: { fraction: 0.5, zero: 0, text: "é", nullable: null } }; +const CANONICAL = '{"event_id":"settlement-1","key_id":"test-1","previous_hash":null,"principal":"alice","record":{"cost_basis":"unknown","cost_usd":null,"extra":{"fraction":0.5,"nullable":null,"text":"é","zero":0}},"sequence":1,"version":1}'; +const SIGNATURE = "bj6tfJIYOfMT4NdUChDUckhAsA4PLn+ZRTkFoMgFkJ688dug9EXuyw01kUt39d17bJJ+qAP7rzhdROvEgJ9ADg=="; +const HASH = "500716026bf9836219938f4e6bba0cb435e940bcf6db76910f1ee9f6279ad256"; + +function database(t, schema = true) { + const sql = new DatabaseSync(":memory:"); + t.after(() => sql.close()); + if (schema) { + sql.exec(readFileSync(new URL("../intelligence-migrations/0032_usage_receipts.sql", import.meta.url), "utf8")); + sql.prepare("INSERT INTO usage_receipt_keys VALUES (?, ?, 1)").run("test-1", PUBLIC); + } + const db = { prepare(query) { + let values = []; + const statement = { bind(...args) { values = args; return statement; }, + first() { return sql.prepare(query).get(...values) ?? null; }, + all() { + const prepared = sql.prepare(query); + if (prepared.columns().length) return { results: prepared.all(...values), meta: { changes: 0 } }; + return { results: [], meta: { changes: Number(prepared.run(...values).changes) } }; + } }; + return statement; + }, async batch(statements) { + sql.exec("BEGIN IMMEDIATE"); + try { const result = statements.map(statement => statement.all()); sql.exec("COMMIT"); return result; } + catch (error) { sql.exec("ROLLBACK"); throw error; } + } }; + const env = { USAGE_RECEIPTS_ENABLED: "true", USAGE_RECEIPTS_KEY_ID: "test-1", + USAGE_RECEIPTS_SIGNING_KEY_REF: "RECEIPT_TEST_PRIVATE", RECEIPT_TEST_PRIVATE: PRIVATE, INTELLIGENCE_DB: db }; + const call = async body => { + const response = await handleManagedStateRequest(new Request("http://intelligence.internal/v1/managed-state/usage-receipts", { + method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ version: 1, ...body }), + }), env); + return { status: response.status, ...await response.json() }; + }; + return { db, env, sql, call }; +} + +test("cross-runtime signed canonical vector keeps unknown metadata and nulls", async t => { + const { env } = database(t); + const row = await appendReceipt(env, "alice", "settlement-1", RECORD); + assert.equal(Buffer.from(row.canonical_bytes_base64, "base64").toString(), CANONICAL); + assert.equal(row.signature_ed25519, SIGNATURE); + assert.equal(row.record_hash, HASH); + assert.deepEqual(row.record, RECORD); + assert.equal(await verifyChain([row], { "test-1": PUBLIC }, { principal: "alice" }), true); +}); + +test("canonical finite exact numbers, scalar Unicode and bounded JSON", () => { + assert.equal(new TextDecoder().decode(canonicalBytes({ z: -0, n: 0.1, "😀": 1, "\ue000": 2 })), + '{"n":0.1000000000000000055511151231257827021181583404541015625,"z":0,"\ue000":2,"😀":1}'); + for (const value of [NaN, Infinity, 2 ** 53, "\ud800", undefined, { value: undefined }, { x: "a".repeat(65536) }]) { + assert.throws(() => canonicalBytes(value)); + } +}); + +test("CAS serializes concurrent appends and duplicate events, corrections append", async t => { + const { env, call } = database(t); + const rows = await Promise.all(Array.from({ length: 6 }, (_, n) => appendReceipt(env, "alice", `event-${n}`, RECORD))); + rows.sort((a, b) => a.sequence - b.sequence); + assert.deepEqual(rows.map(row => row.sequence), [1, 2, 3, 4, 5, 6]); + assert.equal(await verifyChain(rows, { "test-1": PUBLIC }, { principal: "alice" }), true); + const a = await appendReceipt(env, "alice", "duplicate", RECORD); + const duplicates = await Promise.all(Array.from({ length: 6 }, () => appendReceipt(env, "alice", "same-event", RECORD))); + assert.equal(new Set(duplicates.map(item => item.record_hash)).size, 1); + assert.deepEqual(await appendReceipt(env, "alice", "duplicate", RECORD), a); + const changed = await call({ operation: "append", principal: "alice", event_id: "duplicate", record: { cost_usd: 0 } }); + assert.equal(changed.status, 409); + const b = await appendReceipt(env, "alice", "correction", { ...RECORD, cost_usd: 0.5, corrects: a.record_hash }); + assert.equal(b.previous_hash, duplicates[0].record_hash); + assert.equal(b.record.corrects, a.record_hash); + assert.deepEqual((await call({ operation: "get", principal: "alice", id: a.record_hash })).receipt, a); + assert.equal((await call({ operation: "get", principal: "bob", id: a.record_hash })).receipt, null); +}); + +test("verification detects tampering, reordering, gaps and principal substitution", async t => { + const { env } = database(t); + const a = await appendReceipt(env, "alice", "a", RECORD), b = await appendReceipt(env, "alice", "b", RECORD); + for (const rows of [[b, a], [b], [a, { ...b, sequence: 3 }], [{ ...a, record: { cost_usd: 0 } }]]) { + assert.equal(await verifyChain(rows, { "test-1": PUBLIC }, { principal: "alice" }), false); + } + assert.equal(await verifyChain([a], { "test-1": PUBLIC }, { principal: "bob" }), false); +}); + +test("reviewed rotated keys are retained; missing, unreviewed or unsupported signing fails closed", async t => { + const { env, sql, call } = database(t); + sql.prepare("INSERT INTO usage_receipt_keys VALUES (?, ?, ?)").run("unreviewed", PUBLIC, 0); + sql.prepare("INSERT INTO usage_receipt_keys VALUES (?, ?, ?)").run("test-2", PUBLIC, 1); + const a = await appendReceipt(env, "alice", "a", RECORD); + env.USAGE_RECEIPTS_KEY_ID = "test-2"; + const b = await appendReceipt(env, "alice", "b", RECORD); + const keys = (await call({ operation: "keys" })).keys; + assert.deepEqual(keys.map(key => key.key_id), ["test-1", "test-2"]); + assert.equal(await verifyChain([a, b], Object.fromEntries(keys.map(key => [key.key_id, key.public_key_base64])), { principal: "alice" }), true); + env.USAGE_RECEIPTS_KEY_ID = "unreviewed"; + assert.equal((await call({ operation: "keys" })).status, 503); + env.USAGE_RECEIPTS_KEY_ID = "test-1"; + for (const value of ["", "invalid"]) { + env.RECEIPT_TEST_PRIVATE = value; + assert.equal((await call({ operation: "keys" })).status, 503); + } + env.RECEIPT_TEST_PRIVATE = PRIVATE; + const original = crypto.subtle.importKey.bind(crypto.subtle); + t.mock.method(crypto.subtle, "importKey", (...args) => args[2] === "Ed25519" ? Promise.reject(new Error("unsupported")) : original(...args)); + assert.equal((await call({ operation: "keys" })).status, 503); +}); + +test("registered public routes authenticate owners and return unavailable safely", async t => { + const { env } = database(t); + const a = await appendReceipt(env, "alice", "a", RECORD); + const request = new Request(`https://gateway.test/v1/usage/receipts/${a.record_hash}`); + assert.equal((await handleUsageReceiptRequest(request, env, { principal: "alice" })).status, 200); + assert.equal((await handleUsageReceiptRequest(request, env, { principal: "bob" })).status, 404); + assert.equal((await handleUsageReceiptRequest(request, env, null)).status, 401); + env.RECEIPT_TEST_PRIVATE = ""; + assert.equal((await handleUsageReceiptRequest(request, env, { principal: "alice" })).status, 503); +}); + +test("private dispatcher flag, target, missing table and size bounds precede storage", async t => { + const { env, call } = database(t, false); + assert.equal((await call({ operation: "append", principal: "alice", event_id: "a", record: RECORD })).status, 503); + for (const flag of [undefined, "", "false", "invalid"]) { + const response = await handleManagedStateRequest(new Request("http://intelligence.internal/v1/managed-state/usage-receipts", { method: "POST" }), + { USAGE_RECEIPTS_ENABLED: flag, INTELLIGENCE_DB: { prepare() { assert.fail("disabled storage"); } } }); + assert.equal(response.status, 404); + } + assert.equal((await handleUsageReceiptStoreRequest(new Request("https://external.test/v1/managed-state/usage-receipts", { method: "POST" }), env)).status, 404); + const oversized = new Request("http://intelligence.internal/v1/managed-state/usage-receipts", { + method: "POST", headers: { "content-type": "application/json" }, body: "x".repeat(131073) }); + assert.equal((await handleManagedStateRequest(oversized, env)).status, 400); +}); + +test("ledger flush records once, default rows and responses stay unchanged, missing key preserves usage", async t => { + const { env, sql } = database(t); + sql.exec("CREATE TABLE control_users (id TEXT PRIMARY KEY)"); + for (const migration of ["0007_usage_ledger.sql", "0016_usage_buckets.sql"]) { + sql.exec(readFileSync(new URL(`../intelligence-migrations/${migration}`, import.meta.url), "utf8")); + } + const row = { at: "2026-10-09T00:00:00Z", principal: "alice", key_prefix: null, kind: "chat", endpoint: "/v1/chat/completions", + requested_model: "test:model", selected_model: "test:model", status: 200, latency_ms: 1, + input_tokens: null, output_tokens: null, cost_usd: null, cost_basis: null, request_id: "test" }; + const call = async (batch, currentEnv) => { + const response = await handleUsageLedgerRequest(new Request("http://intelligence.internal/v1/usage", { + method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ version: 1, operation: "record", batch, rows: [row] }) }), currentEnv); + return await response.json(); + }; + assert.deepEqual(await call("a".repeat(32), { ...env, USAGE_RECEIPTS_ENABLED: "false" }), { version: 1, recorded: 1, duplicate: false }); + assert.equal(sql.prepare("SELECT COUNT(*) AS count FROM usage_receipts").get().count, 0); + assert.deepEqual(await call("b".repeat(32), env), { version: 1, recorded: 1, duplicate: false }); + assert.deepEqual(await call("b".repeat(32), env), { version: 1, recorded: 0, duplicate: true }); + assert.equal(sql.prepare("SELECT COUNT(*) AS count FROM usage_receipts").get().count, 1); + env.RECEIPT_TEST_PRIVATE = ""; + assert.deepEqual(await call("c".repeat(32), env), { version: 1, recorded: 1, duplicate: false }); + assert.equal(sql.prepare("SELECT COUNT(*) AS count FROM usage_events").get().count, 3); +}); + +test("workerd Ed25519 signs the same receipt through the real D1 authority", async t => { + const source = readFileSync(new URL("../worker/usage-receipts.mjs", import.meta.url), "utf8") + .replace('import { boundedBody } from "./control-users-d1.mjs";', + 'async function boundedBody(request, maximum) { const text = await request.text(); if (new TextEncoder().encode(text).length > maximum) throw new Error("body limit"); return text; }'); + const mf = new Miniflare(convertV4MiniflareOptions({ modules: true, compatibilityDate: "2026-08-18", + script: source + '\nexport default { fetch(request, env) { return handleUsageReceiptStoreRequest(new Request("http://intelligence.internal/v1/managed-state/usage-receipts", request), env); } };', + d1Databases: ["INTELLIGENCE_DB"], bindings: { USAGE_RECEIPTS_ENABLED: "true", USAGE_RECEIPTS_KEY_ID: "test-1", + USAGE_RECEIPTS_SIGNING_KEY_REF: "RECEIPT_TEST_PRIVATE", RECEIPT_TEST_PRIVATE: PRIVATE } })); + t.after(() => mf.dispose()); + const call = () => mf.dispatchFetch("http://receipt.test/", { method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ version: 1, operation: "append", principal: "alice", event_id: "settlement-1", record: RECORD }) }); + assert.equal((await call()).status, 503); + const db = await mf.getD1Database("INTELLIGENCE_DB"); + const migration = readFileSync(new URL("../intelligence-migrations/0032_usage_receipts.sql", import.meta.url), "utf8"); + await db.batch(migration.split(";").filter(sql => sql.trim()).map(sql => db.prepare(sql))); + await db.prepare("INSERT INTO usage_receipt_keys VALUES (?, ?, 1)").bind("test-1", PUBLIC).run(); + const response = await call(); + assert.equal(response.status, 200); + const row = (await response.json()).receipt; + assert.equal(row.signature_ed25519, SIGNATURE); + assert.equal(row.record_hash, HASH); + assert.equal(Buffer.from(row.canonical_bytes_base64, "base64").toString(), CANONICAL); + assert.deepEqual((await (await call()).json()).receipt, row); +}); diff --git a/worker/container-env.mjs b/worker/container-env.mjs index 87b67ddd..004b9941 100644 --- a/worker/container-env.mjs +++ b/worker/container-env.mjs @@ -45,6 +45,9 @@ const DIRECT_ENV_KEYS = [ "REALTIME_ENABLED", "REALTIME_PROVIDERS_JSON", "STREAM_COST_BREAKER_ENABLED", + "USAGE_RECEIPTS_ENABLED", + "USAGE_RECEIPTS_KEY_ID", + "USAGE_RECEIPTS_SIGNING_KEY_REF", "BANDIT_MODE", "BATCH_SPILLOVER_ENABLED", "CANARY_TRAFFIC_ENABLED", diff --git a/worker/d1-schema.mjs b/worker/d1-schema.mjs index c9ce40f9..fb20de73 100644 --- a/worker/d1-schema.mjs +++ b/worker/d1-schema.mjs @@ -10,7 +10,7 @@ export const REQUIRED_D1_TABLES = Object.freeze(["intelligence_policy", "intelli "usage_events", "usage_daily", "usage_batches", "media_jobs", "media_job_items", "shadow_eval_samples", "shadow_eval_results", "shadow_eval_config", "shadow_eval_policy_backups", "prompt_templates", - "config_snapshot_revisions", "config_snapshots", "config_snapshot_applications", "control_revisions", "tool_grants", "generation_cache", "usage_reservation_budgets", "usage_reservations", "usage_reservation_transitions", "session_tiers", "gateway_alert_rules", "gateway_alert_events", "managed_idempotency", "context_pages", "canary_traffic", "semantic_generation_cache", "gateway_batch_files", "gateway_batches", "gateway_batch_items", "hosted_responses", "realtime_tickets", "realtime_sessions", "learned_cooldown"]); + "config_snapshot_revisions", "config_snapshots", "config_snapshot_applications", "control_revisions", "tool_grants", "generation_cache", "usage_reservation_budgets", "usage_reservations", "usage_reservation_transitions", "session_tiers", "gateway_alert_rules", "gateway_alert_events", "managed_idempotency", "context_pages", "canary_traffic", "semantic_generation_cache", "gateway_batch_files", "gateway_batches", "gateway_batch_items", "hosted_responses", "realtime_tickets", "realtime_sessions", "learned_cooldown", "usage_receipt_keys", "usage_receipt_heads", "usage_receipts"]); /** * Readiness for the D1 schema. A deploy that skipped `wrangler d1 migrations apply` reports diff --git a/worker/managed-state-dispatch.mjs b/worker/managed-state-dispatch.mjs index 929ae856..6c6da481 100644 --- a/worker/managed-state-dispatch.mjs +++ b/worker/managed-state-dispatch.mjs @@ -2,6 +2,7 @@ import { handleIdempotencyRequest } from "./idempotency-d1.mjs"; import { handleResponsesStateRequest } from "./responses-state-d1.mjs"; import { handleContextPageStateRequest } from "./context-pages-d1.mjs"; +import { handleUsageReceiptStoreRequest } from "./usage-receipts.mjs"; export function handleManagedStateRequest(request, env, { reservations, toolGrants } = {}) { const url = new URL(request.url); @@ -12,6 +13,7 @@ export function handleManagedStateRequest(request, env, { reservations, toolGran if (url.pathname === "/v1/managed-state/idempotency") return handleIdempotencyRequest(request, env); if (url.pathname === "/v1/managed-state/responses") return handleResponsesStateRequest(request, env); if (url.pathname === "/v1/managed-state/context-pages") return handleContextPageStateRequest(request, env); + if (url.pathname === "/v1/managed-state/usage-receipts") return handleUsageReceiptStoreRequest(request, env); if (url.pathname === "/v1/managed-state/reservations" && reservations) return reservations(request, env); if (url.pathname === "/v1/managed-state/tool-grants" && toolGrants) return toolGrants(request, env); return Response.json({ version: 1, error: { code: "not_found", message: "Managed storage operation not found." } }, { status: 404 }); diff --git a/worker/usage-ledger-d1.mjs b/worker/usage-ledger-d1.mjs index b80704ad..cf3e0b1a 100644 --- a/worker/usage-ledger-d1.mjs +++ b/worker/usage-ledger-d1.mjs @@ -7,6 +7,7 @@ import { boundedBody } from "./control-users-d1.mjs"; import { logFailure } from "./log.mjs"; import { submitNativeEvent } from "./observability-export.mjs"; +import { recordFlushedUsage } from "./usage-receipts.mjs"; import { BUCKET_FIELDS, bucketInsertStatement, recordUsageWithBuckets, usageBucketsEnabled, validBaseUsageRow as validRow } from "./usage-buckets-d1.mjs"; // Upper bounds in milliseconds of the usage_daily latency buckets; the last bucket is open. @@ -100,6 +101,7 @@ async function record(db, env, body, insertEvents = INSERT_EVENTS) { ]); const duplicate = applied.meta.changes !== 1; if (!duplicate) mirror(env, body.rows); + await recordFlushedUsage(env, body.batch, body.rows); return { version: 1, recorded: duplicate ? 0 : rows.length, duplicate }; } diff --git a/worker/usage-receipts.mjs b/worker/usage-receipts.mjs new file mode 100644 index 00000000..3ab9bf4a --- /dev/null +++ b/worker/usage-receipts.mjs @@ -0,0 +1,242 @@ +/** Immutable, opt-in usage provenance with reviewed Ed25519 verification history. */ +import { boundedBody } from "./control-users-d1.mjs"; + +const DOMAIN = new TextEncoder().encode("MultiLLM usage receipt v1\\n"); +const MAX_BYTES = 65536, MAX_BODY_BYTES = 131072; +const LABEL = /^[A-Za-z0-9_.:-]{1,128}$/, REF = /^[A-Za-z_][A-Za-z0-9_]{0,127}$/, HASH = /^[0-9a-f]{64}$/; +const CONTENT_FIELDS = new Set(["prompt", "messages", "content", "input", "output", "response", "api_key", + "authorization", "password", "secret", "private_key", "signing_key"]); +let warned = false; + +class ReceiptError extends Error { + constructor(code = "usage_receipts_unavailable", status = 503) { super(code); this.code = code; this.status = status; } +} +const invalid = () => { throw new ReceiptError("invalid_usage_receipt", 400); }; +const reply = (value, status = 200) => Response.json(value, { status, headers: { "cache-control": "no-store" } }); +const failed = error => reply({ version: 1, error: { code: error instanceof ReceiptError ? error.code : "usage_receipts_unavailable", + message: "Usage receipt operation unavailable." } }, error instanceof ReceiptError ? error.status : 503); +const b64 = bytes => btoa(String.fromCharCode(...bytes)); +function unb64(value) { + if (typeof value !== "string" || value.length > 131072 || !/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(value)) invalid(); + return Uint8Array.from(atob(value), char => char.charCodeAt(0)); +} +const signedBytes = bytes => { const result = new Uint8Array(DOMAIN.length + bytes.length); result.set(DOMAIN); result.set(bytes, DOMAIN.length); return result; }; +const digest = async bytes => Array.from(new Uint8Array(await crypto.subtle.digest("SHA-256", bytes)), byte => byte.toString(16).padStart(2, "0")).join(""); + +export function usageReceiptsEnabled(env) { + const flag = String(env.USAGE_RECEIPTS_ENABLED ?? "").trim().toLowerCase(); + if (!["", "0", "false", "no", "off", "1", "true", "yes", "on"].includes(flag)) { + if (!warned) { warned = true; console.warn("Invalid USAGE_RECEIPTS_ENABLED; usage receipts disabled"); } + return false; + } + return ["1", "true", "yes", "on"].includes(flag); +} + +function exactNumber(value) { + if (!Number.isFinite(value) || (Number.isInteger(value) && !Number.isSafeInteger(value))) invalid(); + if (value === 0) return "0"; + if (Number.isInteger(value)) return String(value); + const view = new DataView(new ArrayBuffer(8)); + view.setFloat64(0, Math.abs(value)); + const bits = view.getBigUint64(0), exponent = Number((bits >> 52n) & 2047n); + let numerator = bits & ((1n << 52n) - 1n); + if (exponent) numerator += 1n << 52n; + let denominatorPower = 1074 - (exponent ? exponent - 1 : 0); + while (denominatorPower > 0 && numerator % 2n === 0n) { numerator /= 2n; denominatorPower--; } + if (denominatorPower <= 0) return (value < 0 ? "-" : "") + String(numerator << BigInt(-denominatorPower)); + const digits = String(numerator * 5n ** BigInt(denominatorPower)).padStart(denominatorPower + 1, "0"); + return (value < 0 ? "-" : "") + digits.slice(0, -denominatorPower) + "." + digits.slice(-denominatorPower); +} + +function scalarString(value) { + if (value.length > MAX_BYTES) invalid(); + for (const char of value) { const point = char.codePointAt(0); if (point >= 0xd800 && point <= 0xdfff) invalid(); } + return JSON.stringify(value); +} +function codepointOrder(a, b) { + const left = Array.from(a, char => char.codePointAt(0)), right = Array.from(b, char => char.codePointAt(0)); + for (let i = 0; i < Math.min(left.length, right.length); i++) if (left[i] !== right[i]) return left[i] - right[i]; + return left.length - right.length; +} +export function canonicalBytes(value) { + let nodes = 0, scalarBytes = 0; + function scalar(text) { + scalarBytes += new TextEncoder().encode(text).length; + if (scalarBytes > MAX_BYTES) invalid(); + return text; + } + function encode(item, depth = 0) { + if (++nodes > 4096 || depth > 32) invalid(); + if (item === null || typeof item === "boolean") return scalar(JSON.stringify(item)); + if (typeof item === "number") return scalar(exactNumber(item)); + if (typeof item === "string") return scalar(scalarString(item)); + if (Array.isArray(item)) { + if (Object.keys(item).length !== item.length) invalid(); + return "[" + item.map(child => encode(child, depth + 1)).join(",") + "]"; + } + if (item && typeof item === "object" && [Object.prototype, null].includes(Object.getPrototypeOf(item))) { + if (Object.getOwnPropertySymbols(item).length) invalid(); + return "{" + Object.keys(item).sort(codepointOrder).map(key => encode(key, depth + 1) + ":" + encode(item[key], depth + 1)).join(",") + "}"; + } + invalid(); + } + const bytes = new TextEncoder().encode(encode(value)); + if (bytes.length > MAX_BYTES) invalid(); + return bytes; +} +const canonicalText = value => new TextDecoder().decode(canonicalBytes(value)); +function checkedRecord(record) { + if (!record || Array.isArray(record) || typeof record !== "object") invalid(); + const text = canonicalText(record); + function check(value) { + if (!value || typeof value !== "object") return; + for (const [key, child] of Object.entries(value)) { + if (CONTENT_FIELDS.has(key.toLowerCase())) invalid(); + check(child); + } + } + check(record); + return JSON.parse(text); +} +function principal(value) { + if (typeof value !== "string" || !value.length || Array.from(value).length > 256 || /[\x00-\x1f\x7f]/.test(value)) invalid(); + scalarString(value); return value; +} +function identifier(value) { if (typeof value !== "string" || !LABEL.test(value)) invalid(); return value; } + +async function ready(env) { + if (!usageReceiptsEnabled(env)) throw new ReceiptError("not_found", 404); + try { + if (!env.INTELLIGENCE_DB || typeof env.USAGE_RECEIPTS_KEY_ID !== "string" || !LABEL.test(env.USAGE_RECEIPTS_KEY_ID) + || typeof env.USAGE_RECEIPTS_SIGNING_KEY_REF !== "string" || !REF.test(env.USAGE_RECEIPTS_SIGNING_KEY_REF)) throw new ReceiptError(); + const material = env[env.USAGE_RECEIPTS_SIGNING_KEY_REF]; + if (typeof material !== "string" || !material.length || material.length > 8192) throw new ReceiptError(); + const key = await crypto.subtle.importKey("pkcs8", unb64(material), "Ed25519", true, ["sign"]); + const jwk = await crypto.subtle.exportKey("jwk", key); + const publicKey = b64(unb64(jwk.x.replaceAll("-", "+").replaceAll("_", "/") + "=")); + const reviewed = await env.INTELLIGENCE_DB.prepare("SELECT public_key_base64 FROM usage_receipt_keys WHERE key_id = ? AND reviewed = 1") + .bind(env.USAGE_RECEIPTS_KEY_ID).first(); + if (!reviewed || reviewed.public_key_base64 !== publicKey) throw new ReceiptError(); + await env.INTELLIGENCE_DB.prepare("SELECT principal FROM usage_receipt_heads LIMIT 0").all(); + await env.INTELLIGENCE_DB.prepare("SELECT id FROM usage_receipts LIMIT 0").all(); + return { key, key_id: env.USAGE_RECEIPTS_KEY_ID }; + } catch { throw new ReceiptError(); } +} +async function eventReceipt(db, owner, eventId) { + const row = await db.prepare("SELECT receipt_json FROM usage_receipts WHERE principal = ? AND event_id = ?").bind(owner, eventId).first(); + return row ? JSON.parse(row.receipt_json) : null; +} +async function makeReceipt(owner, eventId, record, sequence, previousHash, signer) { + const canonical = canonicalBytes({ version: 1, principal: owner, event_id: eventId, record, sequence, previous_hash: previousHash, key_id: signer.key_id }); + const signed = signedBytes(canonical); + return { record, canonical_bytes_base64: b64(canonical), signature_ed25519: b64(new Uint8Array(await crypto.subtle.sign("Ed25519", signer.key, signed))), + key_id: signer.key_id, previous_hash: previousHash, record_hash: await digest(signed), sequence }; +} + +export async function appendReceipt(env, owner, eventId, metadata) { + if (!usageReceiptsEnabled(env)) throw new ReceiptError("not_found", 404); + principal(owner); identifier(eventId); + const record = checkedRecord(metadata), signer = await ready(env), db = env.INTELLIGENCE_DB; + await db.batch([db.prepare("INSERT INTO usage_receipt_heads (principal, sequence, record_hash) VALUES (?, 0, NULL) ON CONFLICT(principal) DO NOTHING").bind(owner)]); + for (let attempt = 0; attempt < 16; attempt++) { + const existing = await eventReceipt(db, owner, eventId); + if (existing) { + if (canonicalText(existing.record) !== canonicalText(record)) throw new ReceiptError("usage_receipt_event_conflict", 409); + return existing; + } + const head = await db.prepare("SELECT sequence, record_hash FROM usage_receipt_heads WHERE principal = ?").bind(owner).first(); + const receipt = await makeReceipt(owner, eventId, record, head.sequence + 1, head.record_hash, signer); + // The guarded insert and head advance share a D1 transaction. Losing CAS does + // not write a receipt; a retry only signs metadata, never calls a provider. + const [inserted] = await db.batch([ + db.prepare("INSERT INTO usage_receipts (id, principal, event_id, sequence, previous_hash, key_id, receipt_json) " + + "SELECT ?, ?, ?, ?, ?, ?, ? WHERE EXISTS (SELECT 1 FROM usage_receipt_heads WHERE principal = ? AND sequence = ? AND COALESCE(record_hash, '') = ?) " + + "ON CONFLICT(principal, event_id) DO NOTHING").bind(receipt.record_hash, owner, eventId, receipt.sequence, receipt.previous_hash, + receipt.key_id, JSON.stringify(receipt), owner, head.sequence, head.record_hash ?? ""), + db.prepare("UPDATE usage_receipt_heads SET sequence = ?, record_hash = ? WHERE principal = ? AND sequence = ? AND COALESCE(record_hash, '') = ? " + + "AND EXISTS (SELECT 1 FROM usage_receipts WHERE id = ? AND principal = ? AND sequence = ?)") + .bind(receipt.sequence, receipt.record_hash, owner, head.sequence, head.record_hash ?? "", receipt.record_hash, owner, receipt.sequence), + ]); + if (inserted.meta.changes === 1) return receipt; + } + throw new ReceiptError("usage_receipt_conflict", 409); +} + +export async function verifyChain(receipts, keys, { principal: owner, sequence = 0, previous_hash: previousHash = null }) { + try { + for (const receipt of receipts) { + const canonical = unb64(receipt.canonical_bytes_base64), payload = JSON.parse(new TextDecoder("utf-8", { fatal: true }).decode(canonical)); + if (b64(canonical) !== b64(canonicalBytes(payload)) || payload.version !== 1 || payload.principal !== owner + || payload.sequence !== sequence + 1 || payload.previous_hash !== previousHash + || ["record", "key_id", "sequence", "previous_hash"].some(name => canonicalText(payload[name]) !== canonicalText(receipt[name]))) return false; + const signed = signedBytes(canonical); + if (await digest(signed) !== receipt.record_hash) return false; + const key = await crypto.subtle.importKey("raw", unb64(keys[receipt.key_id]), "Ed25519", false, ["verify"]); + if (!await crypto.subtle.verify("Ed25519", key, unb64(receipt.signature_ed25519), signed)) return false; + sequence = receipt.sequence; previousHash = receipt.record_hash; + } + return true; + } catch { return false; } +} +async function readReceipt(env, owner, id) { + principal(owner); + await ready(env); + if (typeof id !== "string" || !HASH.test(id)) return null; + const row = await env.INTELLIGENCE_DB.prepare("SELECT receipt_json FROM usage_receipts WHERE principal = ? AND id = ?").bind(owner, id).first(); + return row ? JSON.parse(row.receipt_json) : null; +} +async function reviewedKeys(env) { + await ready(env); + return (await env.INTELLIGENCE_DB.prepare("SELECT key_id, public_key_base64 FROM usage_receipt_keys WHERE reviewed = 1 ORDER BY key_id").all()).results; +} + +export async function handleUsageReceiptStoreRequest(request, env) { + if (!usageReceiptsEnabled(env)) return failed(new ReceiptError("not_found", 404)); + const url = new URL(request.url); + if (url.origin !== "http://intelligence.internal" || url.pathname !== "/v1/managed-state/usage-receipts" + || url.search || url.hash || url.username || url.password || request.method !== "POST") return failed(new ReceiptError("not_found", 404)); + let body; + try { + if (request.headers.get("content-type")?.split(";", 1)[0].trim().toLowerCase() !== "application/json") invalid(); + const length = request.headers.get("content-length"); + if (length !== null && (!/^\d+$/.test(length) || Number(length) > MAX_BODY_BYTES)) invalid(); + body = JSON.parse(await boundedBody(request, MAX_BODY_BYTES)); + if (!body || Array.isArray(body) || body.version !== 1) invalid(); + const fields = { append: ["version", "operation", "principal", "event_id", "record"], get: ["version", "operation", "principal", "id"], keys: ["version", "operation"] }; + if (!Object.hasOwn(fields, body.operation) || Object.keys(body).length !== fields[body.operation].length + || !fields[body.operation].every(name => Object.hasOwn(body, name))) invalid(); + } catch { return failed(new ReceiptError("invalid_usage_receipt", 400)); } + try { + if (body.operation === "append") return reply({ version: 1, receipt: await appendReceipt(env, body.principal, body.event_id, body.record) }); + if (body.operation === "get") return reply({ version: 1, receipt: await readReceipt(env, body.principal, body.id) }); + return reply({ version: 1, keys: await reviewedKeys(env) }); + } catch (error) { return failed(error); } +} + +/** Caller supplies identity only after existing authentication/scope enforcement. */ +export async function handleUsageReceiptRequest(request, env, identity) { + const url = new URL(request.url); + if (!url.pathname.startsWith("/v1/usage/receipts/") && url.pathname !== "/v1/usage/receipt-keys") return null; + if (!usageReceiptsEnabled(env)) return failed(new ReceiptError("not_found", 404)); + if (!identity?.principal) return failed(new ReceiptError("authentication_required", 401)); + if (request.method !== "GET") return failed(new ReceiptError("method_not_allowed", 405)); + try { + if (url.pathname === "/v1/usage/receipt-keys") return reply({ keys: await reviewedKeys(env) }); + const receipt = await readReceipt(env, identity.principal, url.pathname.slice("/v1/usage/receipts/".length)); + return receipt ? reply(receipt) : failed(new ReceiptError("not_found", 404)); + } catch (error) { return failed(error); } +} + +export async function recordSettledUsage(env, owner, eventId, record) { + if (!usageReceiptsEnabled(env)) return null; + try { return await appendReceipt(env, owner, eventId, record); } + catch { console.warn("Usage receipt unavailable"); return null; } +} +export async function recordFlushedUsage(env, batchId, rows) { + if (!usageReceiptsEnabled(env)) return; + for (let index = 0; index < rows.length; index++) { + const row = rows[index], record = { ...row }; + if (record.cost_basis == null && !Object.hasOwn(record, "usage_basis")) record.usage_basis = "unknown"; + await recordSettledUsage(env, row.principal, `${batchId}:${index}`, record); + } +}