diff --git a/docs/session-tiers.md b/docs/session-tiers.md new file mode 100644 index 00000000..e0f9b4c2 --- /dev/null +++ b/docs/session-tiers.md @@ -0,0 +1,107 @@ +# Role-scoped sticky session tiers + +Session tiers keep an approved route stable across tool calls in the `main`, +`delegation` and `aux` lanes. They are disabled by default: + +```text +SESSION_TIER_MODE=off +SESSION_TIER_TTL_SECONDS=1800 +``` + +With the feature disabled, existing routing, response headers and storage are +unchanged. An empty setting uses its default. A malformed mode or TTL disables +the feature and logs one warning without its value. The TTL must be 1–86400 seconds. + +Enable sticky routing with `SESSION_TIER_MODE=sticky`. Requests without tier +metadata continue to use ordinary routing. An explicit model or pinned roleplay +route remains caller-selected and never acquires a tier binding. + +Managed Chat Completions requests use this metadata alongside the normal body: + +```json +{ + "model": "auto:intelligence", + "messages": [{"role": "user", "content": "Continue the task."}], + "session_tier": { + "session": "example-session", + "lane": "main", + "approved_model": "openai:reviewed-model", + "approved_tier": 2 + } +} +``` + +The managed dispatcher must bind the session to its authenticated principal, +remove this metadata before request parsing and upstream dispatch, and supply +the current reviewed policy revision. `prepare_session_tier` accepts only already +eligible candidates and an optional credential/cooldown admission predicate. +`select_candidates(..., session_tier=turn)` preserves the existing eligibility +checks and applies the lane's ordering. Finalization records the actual candidate +and tool-call IDs after classified completion. It does not authorize another attempt. + +Public roleplay requests carry the approval inside `routing`: + +```json +{ + "model": "roleplay:intelligence", + "session_id": "example-session", + "messages": [{"role": "user", "content": "Continue."}], + "routing": { + "session_tier": { + "lane": "main", + "approved_model": "opencode:glm-5.3-flash", + "approved_tier": 1 + } + } +} +``` + +Roleplay storage belongs to the existing authenticated session's Durable Object. +The tier is an explicit numeric label, not a model-quality measurement. Roleplay +has no reviewed numeric tier catalogue, so fallback can use only an identical +native model name on another eligible provider. Provider aliases with different +native names do not qualify. General managed candidates must match both the approved +native model and reviewed `quality_tier`. Billing, entitlement, capability, context limits, +credential availability and cooldown always take precedence over stickiness. + +A different approved model or tier takes effect only at a user-turn boundary +after successful completion and all matching tool results. Partial or unrelated +results leave the outstanding calls unresolved. A continuation without a new user +turn cannot change the approval. Failures, cancellations and ambiguous results +cannot open a safe boundary. Streaming roleplay observes bounded tool-call IDs +without changing response bytes; incomplete or oversized observations prevent a +safe boundary. Unknown streamed tool state stays unresolved until expiry or revision +invalidation. Roleplay automatic continuation, refusal fallback and output-contract +repair are disabled for an opted-in tier turn; existing pre-output fallback rules +still apply within the constrained candidate list. + +The approved route and actual fallback route are separate fields. A fallback is +recorded honestly and preferred on subsequent turns without rewriting the approved +route. Expiry, a policy revision change or confirmed credential failure invalidates +only the affected lane. An in-flight request holds a lease; another request for +that lane receives HTTP 409 `session_tier_busy`. Late completion cannot overwrite +a replacement lease. A lease expires with the lane's TTL, including after a crash. + +Bindings retain hashed principal/session identities, lane, route identifiers, a +hashed revision, expiry, a completed-generation marker, hashed outstanding tool +IDs and a lease. They add no prompt, response, tool arguments or result content +retention. Existing zero-retention restrictions and response headers still apply. +Live bindings expire after the configured idle TTL; the local adapter prunes +expired entries during access and the Durable Object reuses its bounded three-lane +storage. This feature does not create a transcript or change existing conversation +memory behavior. + +The local Python adapter is process-local and permits at most 10,000 live lanes. +Use the injected SQLite adapter when persistence across restarts is required; +it also limits live rows to 10,000. Apply the additive `session_tiers` migration +before using that adapter. It never creates tables during a request. Missing or +unavailable enabled storage returns HTTP 503 `session_tier_storage_unavailable`. +Capacity exhaustion and unavailable approved routes return HTTP 503; invalid +approval metadata returns HTTP 400. Tool markers are limited to 128 outstanding +calls, with identifiers at most 200 characters. Worker storage needs no D1 table +because roleplay retains its existing Durable Object ownership. + +Stickiness does not reserve spend, guarantee a provider's capabilities or metering, +grant paid access, or promote a model autonomously. Every approval still passes +the existing operator policy and budget checks. A remote classifier cannot choose +the approved tier. Storage failure is an error, never a synthetic completion. diff --git a/intelligence-migrations/0021_session_tiers.sql b/intelligence-migrations/0021_session_tiers.sql new file mode 100644 index 00000000..bf255e68 --- /dev/null +++ b/intelligence-migrations/0021_session_tiers.sql @@ -0,0 +1,14 @@ +-- Scoped hashes, reviewed route identifiers and bounded turn markers only. +CREATE TABLE IF NOT EXISTS session_tiers ( + scope_hash TEXT PRIMARY KEY CHECK(length(scope_hash) = 64), + lane TEXT NOT NULL CHECK(lane IN ('main', 'delegation', 'aux')), + tier INTEGER NOT NULL CHECK(tier >= 0 AND tier <= 100), + approved_model TEXT NOT NULL CHECK(length(approved_model) <= 256), + actual_model TEXT NOT NULL CHECK(length(actual_model) <= 256), + policy_revision TEXT NOT NULL CHECK(length(policy_revision) = 64), + expires_at REAL NOT NULL, + safe_turn INTEGER NOT NULL CHECK(safe_turn IN (0, 1)), + pending_tools TEXT NOT NULL CHECK(json_valid(pending_tools) AND length(pending_tools) <= 9000), + lease TEXT NOT NULL CHECK(length(lease) <= 64) +); +CREATE INDEX IF NOT EXISTS session_tiers_expiry ON session_tiers(expires_at); diff --git a/package.json b/package.json index d2eb2fa9..56f892f0 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,7 @@ "lint": "ruff check . --select E9,F63,F7,F82", "typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming", "test": "python -m pytest -q", - "test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs", + "test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_session_tiers_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs", "db:migrate": "python scripts/validate_sqlite_schema.py --apply", "db:validate": "python scripts/validate_sqlite_schema.py", "security:secrets": "python scripts/check_static_secrets.py", diff --git a/services/intelligence_policy.py b/services/intelligence_policy.py index d1ffe0e8..07a46b4f 100644 --- a/services/intelligence_policy.py +++ b/services/intelligence_policy.py @@ -216,7 +216,7 @@ def expected_reply_ms(candidate, request, now=None): return first_ms + tokens * 1000 / max(rate, 1) -def select_candidates(policy, request, config): +def select_candidates(policy, request, config, *, session_tier=None): candidates = [] for priority, candidate in enumerate(policy["candidates"]): if request.explicit and candidate["model"] != request.payload["model"]: @@ -252,28 +252,33 @@ def rank(item): return (-score, -candidate.get("quality_tier", 0), *speed, priority) return (priority, -score, *speed) + def apply_session_tier(choices): + if session_tier is not None and not request.explicit: + return session_tier.select(choices) + return choices + ranked = sorted(candidates, key=rank) if request.explicit or request.profile == "balanced": - return [candidate for _, candidate in ranked] + return apply_session_tier([candidate for _, candidate in ranked]) preference = ( validate_preference(policy["precision_preference"]) if "precision_preference" in policy else configured_preference() ) if not preference or len(ranked) < 2: - return [candidate for _, candidate in ranked] + return apply_session_tier([candidate for _, candidate in ranked]) metadata = { f"{model.provider}:{model.model_id}": model.metadata for model in ProviderCatalogService.list_models() } # Every original ranking dimension precedes precision. A quality tier also # separates fast-profile ties without altering their existing relative order. - return prefer_precision( + return apply_session_tier(prefer_precision( ranked, preference, lambda item: (*rank(item)[:-1], item[1].get("quality_tier", 0)), metadata, - ) + )) def model_advertisement(policy, config=None): diff --git a/services/session_tiers.py b/services/session_tiers.py new file mode 100644 index 00000000..21f7f1e9 --- /dev/null +++ b/services/session_tiers.py @@ -0,0 +1,259 @@ +"""Approved session lanes; retain hashes and policy markers, never content.""" +from __future__ import annotations + +import copy +import hashlib +import json +import logging +import os +import secrets +import sqlite3 +import threading +import time +from dataclasses import dataclass + +from services.intelligence_contract import GatewayError, identifier + +logger = logging.getLogger(__name__) +_warned = set() +_warning_lock = threading.Lock() +MAX_ENTRIES = 10000 +MAX_TOOLS = 128 +LANES = {"main", "delegation", "aux"} + + +@dataclass(frozen=True) +class SessionTierSettings: + enabled: bool = False + ttl_seconds: int = 1800 + + +def settings(environ=None): + source = os.environ if environ is None else environ + mode = (source.get("SESSION_TIER_MODE") or "off").strip() or "off" + ttl = (source.get("SESSION_TIER_TTL_SECONDS") or "1800").strip() or "1800" + try: + if not ttl.isascii() or not ttl.isdigit(): + raise ValueError + seconds = int(ttl) + if mode not in {"off", "sticky"} or not 1 <= seconds <= 86400: + raise ValueError + except (TypeError, ValueError): + with _warning_lock: + if "settings" not in _warned: + _warned.add("settings") + logger.warning("Invalid session tier settings; session tiers disabled") + return SessionTierSettings() + return SessionTierSettings(mode == "sticky", seconds) + + +def _error(code, message, status=503): + return GatewayError(code, message, status) + + +def _hash(*parts): + return hashlib.sha256(json.dumps(parts, separators=(",", ":"), ensure_ascii=True).encode()).hexdigest() + + +def _text(value, name, maximum=256): + if not isinstance(value, str) or not value.strip() or len(value) > maximum: + raise _error("invalid_session_tier", f"Invalid {name}", 400) + return value + + +def validate_selection(lane, model, tier): + if not isinstance(lane, str) or lane not in LANES or type(tier) is not int or not 0 <= tier <= 100: + raise _error("invalid_session_tier", "Invalid lane or approved tier", 400) + try: + identifier(model) + except (TypeError, ValueError): + raise _error("invalid_session_tier", "Invalid approved model", 400) from None + + +def _tool_hash(call_id): + return _hash("tool", _text(call_id, "tool call identifier", 200)) + + +def _pending_after_messages(pending, messages): + if not isinstance(messages, list) or len(messages) > 256: + raise _error("invalid_session_tier", "Invalid session messages", 400) + outstanding = set(pending) + for message in messages: + if not isinstance(message, dict): + raise _error("invalid_session_tier", "Invalid session message", 400) + if message.get("role") == "assistant": + calls = message.get("tool_calls") or [] + if not isinstance(calls, list) or len(calls) > MAX_TOOLS: + raise _error("session_tier_tool_limit", "Invalid tool call markers", 400) + for call in calls: + if not isinstance(call, dict): + raise _error("invalid_session_tier", "Invalid tool call marker", 400) + outstanding.add(_tool_hash(call.get("id"))) + elif message.get("role") == "tool": + outstanding.discard(_tool_hash(message.get("tool_call_id"))) + if len(outstanding) > MAX_TOOLS: + raise _error("session_tier_tool_limit", "Too many outstanding tool calls", 400) + return sorted(outstanding) + + +class LocalSessionTierStore: + """Bounded process-local adapter; inject durable storage when needed.""" + def __init__(self, max_entries=MAX_ENTRIES): + self.rows = {} + self.max_entries = min(MAX_ENTRIES, max(1, max_entries)) + self.lock = threading.RLock() + + def transaction(self, scope, now, update): + with self.lock: + self.rows = {k: v for k, v in self.rows.items() if v["expires_at"] > now} + row, result = update(copy.deepcopy(self.rows.get(scope))) + if row is None: + self.rows.pop(scope, None) + else: + if scope not in self.rows and len(self.rows) >= self.max_entries: + raise _error("session_tier_capacity", "Session tier capacity unavailable") + self.rows[scope] = copy.deepcopy(row) + return result + + +class SQLiteSessionTierStore: + """Injected connection adapter; never migrates tables on traffic.""" + def __init__(self, connection): + self.connection = connection + self.lock = threading.RLock() + + def transaction(self, scope, now, update): + with self.lock: + try: + self.connection.execute("BEGIN IMMEDIATE") + self.connection.execute("DELETE FROM session_tiers WHERE expires_at <= ?", (now,)) + cursor = self.connection.execute("SELECT * FROM session_tiers WHERE scope_hash = ?", (scope,)) + saved = cursor.fetchone() + row = dict(zip([col[0] for col in cursor.description], saved)) if saved else None + if row: + row["pending_tools"] = json.loads(row["pending_tools"]) + row["safe_turn"] = bool(row["safe_turn"]) + row, result = update(row) + if row is None: + self.connection.execute("DELETE FROM session_tiers WHERE scope_hash = ?", (scope,)) + else: + count = self.connection.execute("SELECT COUNT(*) FROM session_tiers").fetchone()[0] + if not saved and count >= MAX_ENTRIES: + raise _error("session_tier_capacity", "Session tier capacity unavailable") + self._save(scope, row) + self.connection.commit() + return result + except sqlite3.Error: + self.connection.rollback() + raise _error("session_tier_storage_unavailable", "Session tier storage unavailable; apply its migration") from None + except Exception: + self.connection.rollback() + raise + + def _save(self, scope, row): + columns = ("lane", "tier", "approved_model", "actual_model", "policy_revision", "expires_at", "safe_turn", "pending_tools", "lease") + values = [json.dumps(row[k]) if k == "pending_tools" else row[k] for k in columns] + self.connection.execute( + """INSERT INTO session_tiers (scope_hash, lane, tier, approved_model, actual_model, + policy_revision, expires_at, safe_turn, pending_tools, lease) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(scope_hash) DO UPDATE SET lane=excluded.lane, tier=excluded.tier, + approved_model=excluded.approved_model, actual_model=excluded.actual_model, + policy_revision=excluded.policy_revision, expires_at=excluded.expires_at, + safe_turn=excluded.safe_turn, pending_tools=excluded.pending_tools, lease=excluded.lease""", + (scope, *values), + ) + + +class SessionTierTurn: + def __init__(self, service, scope, row, candidates): + self.service, self.scope, self.row = service, scope, row + self.models = {c["model"] for c in candidates} + + def select(self, candidates): + allowed = [c for c in candidates if c["model"] in self.models] + return sorted(allowed, key=lambda c: c["model"] != self.row["actual_model"]) + + def finish(self, candidate, *, success, tool_calls=(), credential_failed=False): + def update(current): + if not current or current["lease"] != self.row["lease"]: + return current, None + if credential_failed: + return None, None + if candidate is not None: + if candidate["model"] not in self.models: + raise _error("session_tier_ineligible", "Actual route is outside approved tier") + current["actual_model"] = candidate["model"] + pending = set(current["pending_tools"]) + if not isinstance(tool_calls, (list, tuple)) or len(tool_calls) > MAX_TOOLS: + raise _error("session_tier_tool_limit", "Too many outstanding tool calls") + if any(not isinstance(call, dict) for call in tool_calls): + raise _error("invalid_session_tier", "Invalid tool call marker") + pending.update(_tool_hash(call.get("id")) for call in tool_calls) + if len(pending) > MAX_TOOLS: + raise _error("session_tier_tool_limit", "Too many outstanding tool calls") + current.update(pending_tools=sorted(pending), safe_turn=bool(candidate and success), lease="") + return current, None + self.service.store.transaction(self.scope, self.service.clock(), update) + + +class SessionTiers: + def __init__(self, store, *, clock=time.time, ttl_seconds=1800): + self.store, self.clock, self.ttl_seconds = store, clock, ttl_seconds + + def begin(self, *, principal, session, lane, approved_model, approved_tier, + revision, messages, candidates, unavailable=lambda c: False): + validate_selection(lane, approved_model, approved_tier) + scope = _hash("session-tier", _text(principal, "principal"), _text(session, "session"), lane) + revision = _hash("revision", _text(revision, "policy revision", 512)) + now = self.clock() + def update(row): + if row and row["lease"]: + raise _error("session_tier_busy", "A session lane already has an unresolved request", 409) + if row and row["policy_revision"] != revision: + row = None + pending = _pending_after_messages(row["pending_tools"] if row else [], messages) + safe = not pending and bool(messages) and messages[-1].get("role") == "user" + change = row is None or (safe and row["safe_turn"] and ( + row["approved_model"] != approved_model or row["tier"] != approved_tier + )) + model = approved_model if change else row["approved_model"] + tier = approved_tier if change else row["tier"] + native_model = model.partition(":")[2] + allowed = [c for c in candidates if c.get("quality_tier", 0) == tier + and c["model"].partition(":")[2] == native_model and not unavailable(c)] + if change and not any(c["model"] == model for c in allowed): + raise _error("session_tier_ineligible", "Approved model is not eligible") + if not allowed: + return None, _error("session_tier_ineligible", "No eligible route remains in approved tier") + actual = model if change else row["actual_model"] + row = {"lane": lane, "tier": tier, "approved_model": model, "actual_model": actual, + "policy_revision": revision, "expires_at": now + self.ttl_seconds, + "safe_turn": False, "pending_tools": pending, "lease": secrets.token_hex(16)} + return row, SessionTierTurn(self, scope, row, allowed) + result = self.store.transaction(scope, now, update) + if isinstance(result, GatewayError): + raise result + return result + + +local_store = LocalSessionTierStore() + + +def prepare_session_tier(payload, *, principal, revision, candidates, store=None, environ=None, unavailable=lambda c: False): + """Strip managed metadata before parsing; explicit routes never acquire a lane.""" + configured = settings(environ) + if not configured.enabled or "session_tier" not in payload: + return payload, None + cleaned = {k: v for k, v in payload.items() if k != "session_tier"} + if payload.get("model") != "auto:intelligence": + return cleaned, None + raw = payload["session_tier"] + fields = {"session", "lane", "approved_model", "approved_tier"} + if not isinstance(raw, dict) or set(raw) != fields: + raise _error("invalid_session_tier", "Session tier requires session, lane and explicit approval", 400) + service = SessionTiers(store if store is not None else local_store, ttl_seconds=configured.ttl_seconds) + ticket = service.begin(principal=principal, revision=revision, candidates=candidates, + session=raw["session"], lane=raw["lane"], approved_model=raw["approved_model"], + approved_tier=raw["approved_tier"], messages=payload.get("messages", []), unavailable=unavailable) + return cleaned, ticket diff --git a/tests/fixtures/secret_firewall_coverage.json b/tests/fixtures/secret_firewall_coverage.json index dac82242..6610c1a9 100644 --- a/tests/fixtures/secret_firewall_coverage.json +++ b/tests/fixtures/secret_firewall_coverage.json @@ -341,6 +341,7 @@ "worker/roleplay/routing-policy.mjs", "worker/roleplay/session-metrics.mjs", "worker/roleplay/session-storage.mjs", + "worker/roleplay/session-tier-policy.mjs", "worker/roleplay/sse-collector.mjs", "worker/roleplay/state-runtime.mjs", "worker/roleplay/stream-validation-gate.mjs", diff --git a/tests/test_session_tiers.py b/tests/test_session_tiers.py new file mode 100644 index 00000000..9ba12fbc --- /dev/null +++ b/tests/test_session_tiers.py @@ -0,0 +1,186 @@ +"""Session lane policy and content-free storage contracts.""" +import json +import sqlite3 +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +import pytest + +from services.intelligence_contract import GatewayError +from services.session_tiers import LocalSessionTierStore, SQLiteSessionTierStore, SessionTiers, settings + +MIGRATION = "0021_session_tiers.sql" +A = {"model": "openai:a", "quality_tier": 1} +B = {"model": "openai:b", "quality_tier": 2} +F = {"model": "other:a", "quality_tier": 1} +USER = [{"role": "user", "content": "private prompt"}] + + +def make(store=None): + return SessionTiers(store or LocalSessionTierStore(), clock=lambda: 100, ttl_seconds=1800) + + +def begin(service, **kw): + return service.begin(principal="authenticated", session="private session", lane="main", + approved_model="openai:a", approved_tier=1, revision="r1", + messages=USER, candidates=[A, B, F], **kw) + + +@pytest.mark.parametrize("env", [{}, {"SESSION_TIER_MODE": ""}, {"SESSION_TIER_MODE": "bad"}, + {"SESSION_TIER_MODE": "sticky", "SESSION_TIER_TTL_SECONDS": "bad"}]) +def test_default_empty_and_malformed_are_off(env, caplog): + assert not settings(env).enabled + assert "bad" not in caplog.text + + +def test_defaults_and_bounds(): + assert settings({"SESSION_TIER_MODE": "sticky", "SESSION_TIER_TTL_SECONDS": ""}).ttl_seconds == 1800 + assert not settings({"SESSION_TIER_MODE": "sticky", "SESSION_TIER_TTL_SECONDS": "86401"}).enabled + + +def test_role_and_principal_isolation(): + s = make() + first = begin(s) + other = s.begin(principal="other", session="private session", lane="main", approved_model=B["model"], + approved_tier=2, revision="r1", messages=USER, candidates=[A, B]) + delegated = s.begin(principal="authenticated", session="private session", lane="delegation", approved_model=B["model"], + approved_tier=2, revision="r1", messages=USER, candidates=[A, B]) + assert first.select([B, A, F]) == [A, F] + assert other.select([A, B]) == [B] == delegated.select([A, B]) + + +def test_tool_loop_requires_all_matching_results_before_tier_change(): + s = make() + t = begin(s) + t.finish(A, success=True, tool_calls=[{"id": "one"}, {"id": "two"}]) + t = s.begin(principal="authenticated", session="private session", lane="main", approved_model=B["model"], + approved_tier=2, revision="r1", candidates=[B, A], + messages=[{"role": "tool", "tool_call_id": "one", "content": "private result"}, *USER]) + assert t.select([B, A]) == [A] + t.finish(A, success=True) + t = s.begin(principal="authenticated", session="private session", lane="main", approved_model=B["model"], + approved_tier=2, revision="r1", candidates=[B, A], + messages=[{"role": "tool", "tool_call_id": "wrong", "content": "x"}, *USER]) + assert t.select([B, A]) == [A] + t.finish(A, success=True) + t = s.begin(principal="authenticated", session="private session", lane="main", approved_model=B["model"], + approved_tier=2, revision="r1", candidates=[B, A], + messages=[{"role": "tool", "tool_call_id": "two", "content": "x"}, *USER]) + assert t.select([B, A]) == [B] + + +def test_unsafe_continuation_cannot_change_and_actual_fallback_is_honest(): + s = make() + t = begin(s) + t.finish(F, success=True) + assert next(iter(s.store.rows.values()))["actual_model"] == F["model"] + t = s.begin(principal="authenticated", session="private session", lane="main", approved_model=B["model"], + approved_tier=2, revision="r1", candidates=[A, B, F], messages=[{"role": "assistant", "content": "x"}]) + assert t.select([B, A, F]) == [F, A] + t.finish(F, success=False) + assert begin(s).select([A, F]) == [F, A] + + +def test_revision_expiry_and_credential_failure_clear_only_lane(): + s = make() + t = begin(s) + t.finish(A, success=True, tool_calls=[{"id": "one"}]) + s.clock = lambda: 1900 + t = begin(s) + assert not t.row["pending_tools"] + t.finish(A, success=True) + t = s.begin(principal="authenticated", session="private session", lane="main", approved_model=B["model"], + approved_tier=2, revision="r2", messages=USER, candidates=[B]) + t.finish(None, success=False, credential_failed=True) + assert not s.store.rows + + +def test_grants_and_cooldown_admission_win(): + s = make() + t = begin(s) + t.finish(A, success=True) + assert begin(s).select([B, F]) == [F] + with pytest.raises(GatewayError) as error: + begin(make(), unavailable=lambda c: True) + assert error.value.status == 503 + + +def test_same_tier_different_native_model_cannot_enter_the_tool_loop(): + other = {"model": "other:different", "quality_tier": 1} + t = make().begin(principal="authenticated", session="session", lane="main", + approved_model=A["model"], approved_tier=1, revision="r1", + messages=USER, candidates=[A, F, other]) + assert t.select([other, F, A]) == [A, F] + + +def test_concurrent_turns_fail_closed_and_late_finish_cannot_replace_new_binding(): + s = make() + def attempt(_): + try: + return begin(s) + except GatewayError as e: + assert e.code == "session_tier_busy" + return None + with ThreadPoolExecutor(max_workers=8) as pool: + turns = [t for t in pool.map(attempt, range(8)) if t] + assert len(turns) == 1 + s.clock = lambda: 1900 + current = begin(s) + turns[0].finish(F, success=True) + assert next(iter(s.store.rows.values()))["lease"] == current.row["lease"] + + +def test_store_contains_no_content_or_raw_identity(): + s = make() + t = begin(s) + t.finish(A, success=True, tool_calls=[{"id": "private call", "function": {"arguments": "private args"}}]) + saved = json.dumps(s.store.rows) + assert "private" not in saved and "authenticated" not in saved + assert len(next(iter(s.store.rows))) == 64 + + +def test_sqlite_migration_additive_old_rows_and_missing_table(): + conn = sqlite3.connect(":memory:") + conn.execute("CREATE TABLE prior (value TEXT)") + conn.execute("INSERT INTO prior VALUES ('old')") + conn.commit() + s = make(SQLiteSessionTierStore(conn)) + with pytest.raises(GatewayError) as error: + begin(s) + assert error.value.code == "session_tier_storage_unavailable" and error.value.status == 503 + sql = (Path(__file__).parents[1] / "intelligence-migrations" / MIGRATION).read_text() + conn.executescript(sql) + conn.executescript(sql) + t = begin(s) + t.finish(A, success=True) + assert begin(make(SQLiteSessionTierStore(conn))).select([B, A]) == [A] + assert conn.execute("SELECT value FROM prior").fetchone()[0] == "old" + + +def test_capacity_fail_closed(): + s = make(LocalSessionTierStore(max_entries=1)) + begin(s) + with pytest.raises(GatewayError) as error: + s.begin(principal="other", session="session", lane="aux", approved_model=A["model"], + approved_tier=1, revision="r1", messages=USER, candidates=[A]) + assert error.value.status == 503 + + +def test_managed_policy_hook_default_and_explicit_unchanged(monkeypatch): + import services.intelligence_policy as policy + from types import SimpleNamespace + from services.session_tiers import prepare_session_tier + monkeypatch.setattr(policy, "eligible", lambda *args: True) + monkeypatch.setattr(policy, "expected_reply_ms", lambda *args: None) + c = {**A, "capabilities": [], "max_output_tokens": 100, "context_window": 1000} + request = SimpleNamespace(explicit=False, required=set(), input_tokens=1, output_tokens=1, + task="chat", profile="balanced", allow_paid=False, payload={"model": "auto:intelligence"}) + choices = policy.select_candidates({"candidates": [c]}, request, {}) + t = begin(make()) + assert policy.select_candidates({"candidates": [c]}, request, {}, session_tier=t) == choices + body = {"model": "auto:intelligence", "session_tier": {"invalid": "private"}} + assert prepare_session_tier(body, principal="p", revision="r", candidates=[], environ={}) == (body, None) + body = {"model": "openai:a", "session_tier": {"invalid": "private"}} + cleaned, ticket = prepare_session_tier(body, principal="p", revision="r", candidates=[], + environ={"SESSION_TIER_MODE": "sticky"}) + assert cleaned == {"model": "openai:a"} and ticket is None diff --git a/tests/test_session_tiers_worker.mjs b/tests/test_session_tiers_worker.mjs new file mode 100644 index 00000000..000c2fc5 --- /dev/null +++ b/tests/test_session_tiers_worker.mjs @@ -0,0 +1,294 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { readFileSync } from "node:fs"; +import { SessionTierPolicy, sessionTierSettings, parseSessionTier } from "../worker/roleplay/session-tier-policy.mjs"; +import { parseRoutingPolicy } from "../worker/roleplay/routing-policy.mjs"; +import { completionResponse, makeRoleplayEnv, roleplayRequest, handleRoleplayEdgeRequest, + withGlobalFetch } from "./helpers/roleplay_fixture.mjs"; + +const MIGRATION = "0021_session_tiers.sql"; +const a = { provider: "opencode", model: "a", key: "opencode:a" }; +const b = { provider: "opencode", model: "b", key: "opencode:b" }; +const f = { provider: "other", model: "a", key: "other:a" }; +const user = [{ role: "user", content: "private prompt" }]; +function fixture() { + const rows = new Map(); + const storage = { async get(k) { return structuredClone(rows.get(k)); }, + async put(k, v) { rows.set(k, structuredClone(v)); }, async delete(k) { rows.delete(k); } }; + const policy = new SessionTierPolicy(storage, { enabled: true, ttlSeconds: 1800 }, () => 100); + return { policy, rows }; +} +function start(policy, options = {}) { + return policy.begin({ lane: "main", approved_model: "opencode:a", approved_tier: 1 }, [a, b, f], user, "r1", options); +} + +test("default, empty and malformed settings disable without content logging", () => { + for (const value of [undefined, "", "off", "private"]) assert.equal(sessionTierSettings({ SESSION_TIER_MODE: value }).enabled, false); + assert.equal(sessionTierSettings({ SESSION_TIER_MODE: "sticky", SESSION_TIER_TTL_SECONDS: "" }).ttlSeconds, 1800); + assert.equal(sessionTierSettings({ SESSION_TIER_MODE: "sticky", SESSION_TIER_TTL_SECONDS: "0" }).enabled, false); +}); + +test("routing defaults unchanged and explicit metadata validated only when enabled", () => { + assert.deepEqual(parseRoutingPolicy(), { mode: "provider-priority", provider: "", model: "", billing: "configured", fallback: "safe" }); + assert.throws(() => parseRoutingPolicy({ session_tier: {} })); + const tier = { lane: "main", approved_model: "opencode:a", approved_tier: 1 }; + assert.deepEqual(parseRoutingPolicy({ session_tier: tier }, { sessionTierEnabled: true }).session_tier, tier); + for (const value of [{ ...tier, lane: "unknown" }, { ...tier, approved_tier: true }, { ...tier, extra: "private" }]) assert.throws(() => parseSessionTier(value)); +}); + +test("role lanes isolated and tool loop changes only after complete matching results", async () => { + const { policy } = fixture(); + let t = await start(policy); + await t.finish(a, { success: true, toolCalls: [{ id: "one" }, { id: "two" }] }); + const delegation = await policy.begin({ lane: "delegation", approved_model: "opencode:b", approved_tier: 2 }, [a, b], user, "r1"); + assert.deepEqual(delegation.candidates, [b]); + const changed = { lane: "main", approved_model: "opencode:b", approved_tier: 2 }; + t = await policy.begin(changed, [b, a], [{ role: "tool", tool_call_id: "one", content: "result" }, ...user], "r1"); + assert.deepEqual(t.candidates, [a]); + await t.finish(a, { success: true }); + t = await policy.begin(changed, [b, a], [{ role: "tool", tool_call_id: "wrong", content: "result" }, ...user], "r1"); + assert.deepEqual(t.candidates, [a]); + await t.finish(a, { success: true }); + t = await policy.begin(changed, [b, a], [{ role: "tool", tool_call_id: "two", content: "result" }, ...user], "r1"); + assert.deepEqual(t.candidates, [b]); +}); + +test("honest fallback keeps approved tier and model; cooldown and grants win", async () => { + const { policy, rows } = fixture(); + let t = await start(policy); + await t.finish(f, { success: true }); + assert.equal([...rows.values()][0].actual_model, "other:a"); + t = await start(policy); + assert.deepEqual(t.candidates, [f, a]); + await t.finish(f, { success: true }); + t = await policy.begin({ lane: "main", approved_model: "opencode:a", approved_tier: 1 }, [a, b, f], user, "r1", { unavailable: c => c === a }); + assert.deepEqual(t.candidates, [f]); + await t.finish(f, { success: true }); + await assert.rejects(policy.begin({ lane: "main", approved_model: "opencode:a", approved_tier: 1 }, [b], user, "r1"), e => e.status === 503); +}); + +test("expiry, revision and credential failure invalidate narrowly; no content retained", async () => { + const { policy, rows } = fixture(); + let t = await start(policy); + await t.finish(a, { success: true, toolCalls: [{ id: "private call", function: { arguments: "private args" } }] }); + assert.doesNotMatch(JSON.stringify([...rows]), /private/); + policy.clock = () => 1900; + t = await start(policy); + assert.deepEqual(t.row.pending_tools, []); + await t.finish(a, { success: true }); + t = await policy.begin({ lane: "main", approved_model: "opencode:b", approved_tier: 2 }, [b], user, "r2"); + assert.deepEqual(t.candidates, [b]); + await t.finish(null, { success: false, credentialFailed: true }); + assert.equal(rows.size, 0); +}); + +test("concurrent or aborted turns cannot create a safe turn or overwrite newer lease", async () => { + const { policy, rows } = fixture(); + const t = await start(policy); + await assert.rejects(start(policy), e => e.code === "session_tier_busy"); + policy.clock = () => 1900; + const newer = await start(policy); + await t.finish(f, { success: true }); + assert.equal([...rows.values()][0].lease, newer.row.lease); + await newer.finish(a, { success: false }); + const changed = await policy.begin({ lane: "main", approved_model: "opencode:b", approved_tier: 2 }, [a, b], user, "r1"); + assert.deepEqual(changed.candidates, [a]); +}); + +test("disabled storage untouched; enabled unavailable storage fails closed", async () => { + const storage = { async get() { throw new Error("private storage failure"); } }; + const disabled = new SessionTierPolicy(storage, { enabled: false }); + assert.equal(await start(disabled), null); + const enabled = new SessionTierPolicy(storage, { enabled: true, ttlSeconds: 1800 }); + await assert.rejects(start(enabled), e => e.code === "session_tier_storage_unavailable" && e.status === 503 && !e.message.includes("private")); + assert.match(readFileSync(new URL(`../intelligence-migrations/${MIGRATION}`, import.meta.url), "utf8"), /CREATE TABLE IF NOT EXISTS session_tiers/); +}); + +const approval = (model = "glm-5.3", lane = "main", tier = 2) => ({ + lane, approved_model: `opencode:${model}`, approved_tier: tier, +}); +const body = (session_tier, messages = user) => ({ model: "roleplay:intelligence", + session_id: "session-tier-test", messages, stream: false, memory: { mode: "off" }, + ...(session_tier ? { routing: { session_tier } } : {}) }); +async function turn(f, payload, headers) { + const response = await handleRoleplayEdgeRequest(roleplayRequest(payload, headers), f.env); + const value = await response.json(); + await f.waitForBackgroundWork(); + return { response, value }; +} + +test("public roleplay tool loop keeps model until matching results and isolates authenticated callers", async () => { + const f = makeRoleplayEnv({ SESSION_TIER_MODE: "sticky", CONTENT_RETENTION_ENABLED: "true" }); + const models = []; + let emitTool = true; + await withGlobalFetch(async (_url, options) => { + const payload = JSON.parse(options.body); models.push(payload.model); + assert.equal(payload.session_tier, undefined); + const response = completionResponse(payload.model, "A tool is required."); + if (!emitTool) return response; + emitTool = false; + const data = await response.json(); + data.choices[0].message.tool_calls = [{ id: "private-tool-id", type: "function", function: { name: "lookup", arguments: "private args" } }]; + data.choices[0].finish_reason = "tool_calls"; + return Response.json(data); + }, async () => { + const headers = { "X-MultiLLM-Retention": "zero" }; + const first = await turn(f, body(approval()), headers); + assert.equal(first.response.status, 200); + assert.equal(first.response.headers.get("X-MultiLLM-Retention"), "zero"); + const second = await turn(f, body(approval("glm-5.3-flash", "main", 1)), headers); + assert.equal(second.response.status, 200); + await turn(f, body(approval("glm-5.3-flash", "main", 1), [{ role: "tool", tool_call_id: "private-tool-id", content: "private result" }, ...user]), headers); + assert.deepEqual(models, ["glm-5.3", "glm-5.3", "glm-5.3-flash"]); + await turn(f, body(approval("glm-5.3-flash", "main", 1)), { ...headers, Authorization: "Bearer janitor-roleplay-key" }); + assert.equal(f.storageBySession.size, 2); + const tierRows = [...f.storageBySession.values()].flatMap(v => [...v.storage.values].filter(([k]) => k.startsWith("session-tier:"))); + assert.doesNotMatch(JSON.stringify(tierRows), /private|session-tier-test|arguments|content/); + assert.doesNotMatch(JSON.stringify([...f.storageBySession.values()].flatMap(v => [...v.storage.values])), /private prompt|private args|private result/); + }); +}); + +test("default and unopted requests keep provider response and add no tier storage or headers", async () => { + for (const mode of [undefined, "", "off", "invalid", "sticky"]) { + const f = makeRoleplayEnv({ SESSION_TIER_MODE: mode }); + await withGlobalFetch(async (_url, options) => completionResponse(JSON.parse(options.body).model), async () => { + const result = await turn(f, body(null)); + assert.equal(result.response.status, 200); + assert.equal(result.value.choices[0].message.content, "In character."); + assert.equal([...result.response.headers.keys()].some(k => k.includes("session-tier")), false); + assert.equal([...f.storageBySession.values()].some(v => [...v.storage.values.keys()].some(k => k.startsWith("session-tier:"))), false); + }); + } +}); + +test("public roleplay storage failures return clear 503 before upstream generation", async () => { + const f = makeRoleplayEnv({ SESSION_TIER_MODE: "sticky" }); + await withGlobalFetch(async (_url, options) => completionResponse(JSON.parse(options.body).model), async () => { + await turn(f, body(null)); + }); + const { storage } = [...f.storageBySession.values()][0]; + const original = storage.get.bind(storage); + storage.get = async key => { + if (key.startsWith("session-tier:")) throw new Error("private unavailable table"); + return original(key); + }; + await withGlobalFetch(() => assert.fail("upstream dispatch"), async () => { + const result = await turn(f, body(approval())); + assert.equal(result.response.status, 503); + assert.equal(result.value.error.code, "session_tier_storage_unavailable"); + assert.doesNotMatch(JSON.stringify(result.value), /private/); + }); +}); + +test("public pinned route remains caller-selected and writes no tier binding", async () => { + const f = makeRoleplayEnv({ SESSION_TIER_MODE: "sticky" }); + await withGlobalFetch(async (_url, options) => { + assert.equal(JSON.parse(options.body).model, "glm-5.3-flash"); + return completionResponse("glm-5.3-flash"); + }, async () => { + const payload = body(approval()); + payload.routing = { ...payload.routing, mode: "pinned", provider: "opencode", model: "glm-5.3-flash" }; + const result = await turn(f, payload); + assert.equal(result.response.status, 200); + assert.equal([...f.storageBySession.values()].some(v => [...v.storage.values.keys()].some(k => k.startsWith("session-tier:"))), false); + }); +}); + +test("stream tool identifiers are observed without changing bytes", async () => { + const { policy } = fixture(); + const t = await start(policy); + const data = 'data: {"choices":[{"delta":{"tool_calls":[{"index":0,"id":"call-"}]}}]}\n\n' + + 'data: {"choices":[{"delta":{"tool_calls":[{"index":0,"id":"one"}]},"finish_reason":"tool_calls"}]}\n\n' + + 'data: [DONE]\n\n'; + assert.equal(await t.observe(new Response(data, { headers: { "content-type": "text/event-stream" } })).text(), data); + await t.finish(a, { success: true, finishReason: "tool_calls" }); + const next = await policy.begin({ lane: "main", approved_model: "opencode:b", approved_tier: 2 }, [a, b], user, "r1"); + assert.deepEqual(next.candidates, [a]); +}); + +test("public roleplay fallback records the actual eligible provider", async () => { + const f = makeRoleplayEnv({ SESSION_TIER_MODE: "sticky", NAVYAI_API_KEY: "fixture-navy-key" }); + const calls = []; + await withGlobalFetch(async (url, options) => { + calls.push(String(url)); + if (String(url).includes("opencode")) return Response.json({ error: { message: "unavailable" } }, { status: 503 }); + return completionResponse(JSON.parse(options.body).model); + }, async () => { + const result = await turn(f, body(approval())); + assert.equal(result.response.status, 200); + const saved = [...f.storageBySession.values()][0].storage.values; + const row = [...saved].find(([key]) => key.startsWith("session-tier:"))[1]; + assert.equal(row.approved_model, "opencode:glm-5.3"); + assert.equal(row.actual_model, "navyai:glm-5.3"); + await turn(f, body(approval())); + assert.equal(calls.filter(url => url.includes("opencode")).length, 1); + }); +}); + +test("public streaming completion persists matching tool markers", async () => { + const f = makeRoleplayEnv({ SESSION_TIER_MODE: "sticky" }); + await withGlobalFetch(async () => new Response( + 'data: {"id":"stream-test","choices":[{"index":0,"delta":{"content":"Using the tool.","tool_calls":[{"index":0,"id":"stream-call"}]},"finish_reason":null}]}\n\n' + + 'data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}\n\n' + + 'data: [DONE]\n\n', { headers: { "content-type": "text/event-stream" } }), async () => { + const response = await handleRoleplayEdgeRequest(roleplayRequest({ ...body(approval()), stream: true }), f.env); + assert.equal(response.status, 200); + assert.match(await response.text(), /\[DONE\]/); + await f.waitForBackgroundWork(); + const row = [...[...f.storageBySession.values()][0].storage.values].find(([key]) => key.startsWith("session-tier:"))[1]; + assert.equal(row.pending_tools.length, 1); + assert.equal(row.lease, ""); + }); +}); + +test("simultaneous same-lane starts acquire exactly one lease", async () => { + const { policy } = fixture(); + const results = await Promise.allSettled(Array.from({ length: 8 }, () => start(policy))); + assert.equal(results.filter(r => r.status === "fulfilled").length, 1); + assert.ok(results.filter(r => r.status === "rejected").every(r => r.reason.code === "session_tier_busy")); +}); + +test("cancelling a queued request cannot release the active request's tier lease", async () => { + const f = makeRoleplayEnv({ SESSION_TIER_MODE: "sticky" }); + let release, entered; + const upstreamStarted = new Promise(resolve => { entered = resolve; }); + const gate = new Promise(resolve => { release = resolve; }); + await withGlobalFetch(async (_url, options) => { + entered(); await gate; + return completionResponse(JSON.parse(options.body).model); + }, async () => { + const first = handleRoleplayEdgeRequest(roleplayRequest(body(approval())), f.env); + await upstreamStarted; + const { instance, storage } = [...f.storageBySession.values()][0]; + const saved = () => [...storage.values].find(([key]) => key.startsWith("session-tier:"))[1]; + const lease = saved().lease; + let queued; + const queuedRequest = new Promise(resolve => { queued = resolve; }); + const acquire = instance.turnQueue.acquire.bind(instance.turnQueue); + instance.turnQueue.acquire = (...args) => { const result = acquire(...args); queued(); return result; }; + const controller = new AbortController(); + const second = handleRoleplayEdgeRequest(new Request(roleplayRequest(body(approval())), { signal: controller.signal }), f.env); + await queuedRequest; + controller.abort(); + assert.equal((await second).status, 499); + assert.equal(saved().lease, lease); + release(); + const response = await first; + assert.equal(response.status, 200); + await response.text(); + await f.waitForBackgroundWork(); + assert.equal(saved().lease, ""); + }); +}); + +test("unknown streamed tool state stays unresolved until expiry or revision", async () => { + const { policy } = fixture(); + let t = await start(policy); + const data = 'data: {"choices":[{"delta":{},"finish_reason":"tool_calls"}]}\n\ndata: [DONE]\n\n'; + await t.observe(new Response(data, { headers: { "content-type": "text/event-stream" } })).text(); + await t.finish(a, { success: true, finishReason: "tool_calls" }); + t = await start(policy); + await t.finish(a, { success: true }); + t = await policy.begin({ lane: "main", approved_model: "opencode:b", approved_tier: 2 }, [a, b], user, "r1"); + assert.deepEqual(t.candidates, [a]); +}); diff --git a/worker/container-env.mjs b/worker/container-env.mjs index b6315525..d57557bc 100644 --- a/worker/container-env.mjs +++ b/worker/container-env.mjs @@ -30,6 +30,8 @@ const DIRECT_ENV_KEYS = [ "DEFERRED_TOOLS_ENABLED", "GENERATION_CACHE_BACKEND", "GENERATION_CACHE_SHARED_ENABLED", + "SESSION_TIER_MODE", + "SESSION_TIER_TTL_SECONDS", "USAGE_HOLD_REVIEW_AFTER_SECONDS", "USAGE_RESERVATIONS_ENABLED", "CONNECTION_PROFILES_DB_PATH", diff --git a/worker/d1-schema.mjs b/worker/d1-schema.mjs index 28a31eed..d39d522d 100644 --- a/worker/d1-schema.mjs +++ b/worker/d1-schema.mjs @@ -10,7 +10,7 @@ export const REQUIRED_D1_TABLES = Object.freeze(["intelligence_policy", "intelli "usage_events", "usage_daily", "usage_batches", "media_jobs", "media_job_items", "shadow_eval_samples", "shadow_eval_results", "shadow_eval_config", "shadow_eval_policy_backups", "prompt_templates", - "config_snapshot_revisions", "config_snapshots", "config_snapshot_applications", "control_revisions", "tool_grants", "generation_cache", "usage_reservation_budgets", "usage_reservations", "usage_reservation_transitions"]); + "config_snapshot_revisions", "config_snapshots", "config_snapshot_applications", "control_revisions", "tool_grants", "generation_cache", "usage_reservation_budgets", "usage_reservations", "usage_reservation_transitions", "session_tiers"]); /** * Readiness for the D1 schema. A deploy that skipped `wrangler d1 migrations apply` reports diff --git a/worker/roleplay/endpoint.mjs b/worker/roleplay/endpoint.mjs index 7c25eadd..f9f42938 100644 --- a/worker/roleplay/endpoint.mjs +++ b/worker/roleplay/endpoint.mjs @@ -7,7 +7,8 @@ import { recoveryTemplate, preserveRecovery, handleRecoverySnapshot, resolveRoleplayRetention, retentionResponse, retentionRequestId, retentionAllowsContent, createRetentionStateRepository, } from "./recovery.mjs"; -import { parseRoutingPolicy, filterRoutingCandidates, parameterReceipt } from "./routing-policy.mjs"; +import { parseRoutingPolicy, filterRoutingCandidates, parameterReceipt, + SessionTierError, SessionTierPolicy, sessionTierSettings, beginRoleplaySessionTier } from "./routing-policy.mjs"; import { isRoleplayPath, @@ -100,6 +101,7 @@ export class RoleplaySession extends DurableObject { ...buildIntelligenceCandidates(env, this.settings), ]; this.stateRepository = createRoleplayStateRepository(ctx.storage); + this.sessionTiers = new SessionTierPolicy(ctx.storage, sessionTierSettings(env)); this.refreshSessionAlarm = createSessionAlarmRefresher( ctx, this.settings.sessionTtlSeconds, @@ -146,26 +148,32 @@ export class RoleplaySession extends DurableObject { async enqueueTurn(request, settings, retention = resolveRoleplayRetention(this.env, request)) { const trace = this.traces.begin(retention); + const tierScope = { turn: null }; try { return await this.turnQueue.run( request, settings, async (turnRequest, queueMs) => { trace.phase("preparing"); trace.metrics({ queueMs }); - const result = await this.handleTurn(turnRequest, settings, queueMs, trace, retention); + const result = await this.handleTurn(turnRequest, settings, queueMs, trace, retention, tierScope); result.response.headers.set("X-Roleplay-Trace-ID", trace.id); result.completion = Promise.resolve(result.completion).then(async (completion) => { if (completion?.persistenceFailed) await trace.finish(false, "persistence_failed"); return completion; }); - if (!result.response.ok) await trace.finish(false, "request_failed"); + if (!result.response.ok) { + await tierScope.turn?.finish(null, { success: false }); + await trace.finish(false, "request_failed"); + } return result; }, (completion) => this.ctx.waitUntil(completion), ); } catch (error) { + try { await tierScope.turn?.finish(null, { success: false }); } + catch (storageError) { error = storageError; } await trace.finish(false, error.code || "turn_failed"); - if (error instanceof RoleplayTurnError) { + if (error instanceof RoleplayTurnError || error instanceof SessionTierError) { return errorResponse(error.message, error.status, error.code); } if (error instanceof RoleplayRequestError) { @@ -184,7 +192,7 @@ export class RoleplaySession extends DurableObject { } } - async handleTurn(request, settings, queueMs = 0, trace = null, retention = resolveRoleplayRetention(this.env, request)) { + async handleTurn(request, settings, queueMs = 0, trace = null, retention = resolveRoleplayRetention(this.env, request), tierScope = { turn: null }) { const stateRepository = createRetentionStateRepository(this.ctx.storage, this.stateRepository, retention, this.configuredCandidates); const scanCounts = [0, 0]; const applyScanHeader = headers => { @@ -212,7 +220,7 @@ export class RoleplaySession extends DurableObject { request.headers.get(INTERNAL_OUTPUT_MODE_HEADER) === "unlimited", }, ); - parsedInitial.routing = parseRoutingPolicy(payload.routing); + parsedInitial.routing = parseRoutingPolicy(payload.routing, { sessionTierEnabled: this.sessionTiers.settings.enabled }); if (parsedInitial.routing.fallback === "none") { settings = { ...settings, refusalFallbackEnabled: false, maxAutoContinuations: 0, maxOutputContractRepairs: 0 }; } @@ -306,6 +314,13 @@ export class RoleplaySession extends DurableObject { if (!candidates.length && autoPreference !== parsed.modelPreference) { candidates = rankFor(parsed.modelPreference); } + const tierTurn = await beginRoleplaySessionTier(this.sessionTiers, parsed, payload, + candidates, state.stats, this.configuredCandidates); + tierScope.turn = tierTurn; + if (tierTurn) { + candidates = tierTurn.candidates; + settings = { ...settings, refusalFallbackEnabled: false, maxAutoContinuations: 0, maxOutputContractRepairs: 0 }; + } if (parsed.routing.fallback === "none") candidates.splice(1); if (!candidates.length) { state = markRoleplayRequest(state, idempotencyKey, "no_provider"); @@ -521,7 +536,7 @@ export class RoleplaySession extends DurableObject { estimatedInputTokens, plan.estimatedTokens + checkpointSavedTokens, ); - const generationCandidates = prepareRoleplayCandidates( + let generationCandidates = prepareRoleplayCandidates( candidates, estimatedInputTokens, parsed.maxTokens, @@ -530,6 +545,7 @@ export class RoleplaySession extends DurableObject { ? { messages: roleplayMessages, estimateTokens } : null, ); + if (tierTurn) generationCandidates = tierTurn.select(generationCandidates); if (!generationCandidates.length) { state = markRoleplayRequest(state, idempotencyKey, "context_too_large"); await stateRepository.save(state); @@ -569,6 +585,8 @@ export class RoleplaySession extends DurableObject { ); state = attempted.state; if (attempted.terminalResponse) { + await tierTurn?.finish(null, { success: false, + credentialFailed: [401, 403].includes(attempted.terminalResponse.status) }); await preserveRecovery(this.ctx.storage, recovery, { success: false, reason: "provider_failed" }, trace?.id, retention); state = markRoleplayRequest(state, idempotencyKey, "provider_failed"); await stateRepository.save(state); @@ -666,7 +684,7 @@ export class RoleplaySession extends DurableObject { responseHeaders.set("Cache-Control", "no-cache, no-transform"); const observed = createObservedStream({ onProgress: (progress) => trace?.progress(progress), - upstreamBody: response.body, + upstreamBody: tierTurn ? tierTurn.observe(response).body : response.body, requestSignal: request.signal, upstreamController: controller, heartbeatMs: settings.streamHeartbeatMs, @@ -702,6 +720,8 @@ export class RoleplaySession extends DurableObject { } = completion; state = continuation.state; const finalCandidate = continuation.candidate; + await tierTurn?.finish(finalCandidate, { success: success && !request.signal.aborted && + ["stop", "tool_calls"].includes(completion.finishReason), finishReason: completion.finishReason }); trace?.selected(finalCandidate, parameterReceipt(parsed, finalCandidate, settings)); logRoleplayStreamCompletion({ candidate: finalCandidate, @@ -827,6 +847,10 @@ export class RoleplaySession extends DurableObject { failureStatus: "completion_failed", }); const finalCandidate = continuation.candidate; + await tierTurn?.finish(finalCandidate, { success: completionResult.success && !request.signal.aborted && + contentType.toLowerCase().includes("application/json"), + finishReason: completionResult.finishReason, + toolCalls: completionResult.payload?.choices?.[0]?.message?.tool_calls ?? [] }); state = applyRoleplayCompletionState({ state, candidate: finalCandidate, diff --git a/worker/roleplay/routing-policy.mjs b/worker/roleplay/routing-policy.mjs index b298dc1c..70265712 100644 --- a/worker/roleplay/routing-policy.mjs +++ b/worker/roleplay/routing-policy.mjs @@ -1,14 +1,17 @@ import { RoleplayRequestError } from "./validation.mjs"; import { applyReasoningPolicy, requestedReasoningEffort } from "./reasoning.mjs"; import { glmModelVariant } from "./model-selection.mjs"; +import { parseSessionTier } from "./session-tier-policy.mjs"; +export { SessionTierError, SessionTierPolicy, sessionTierSettings, beginRoleplaySessionTier } from "./session-tier-policy.mjs"; const MODES = ["provider-priority", "fastest-eligible", "quality", "pinned"]; const PROVIDERS = ["nanogpt", "opencode", "openrouter", "linkapi", "navyai", "cline-pass"]; -export function parseRoutingPolicy(value) { +export function parseRoutingPolicy(value, { sessionTierEnabled = false } = {}) { if (value === undefined) value = {}; if (!value || typeof value !== "object" || Array.isArray(value)) throw new RoleplayRequestError("routing must be an object"); - if (Object.keys(value).some((key) => !["mode", "provider", "model", "billing", "fallback"].includes(key))) { + const allowed = ["mode", "provider", "model", "billing", "fallback", ...(sessionTierEnabled ? ["session_tier"] : [])]; + if (Object.keys(value).some((key) => !allowed.includes(key))) { throw new RoleplayRequestError("Unsupported routing option"); } const result = { mode: "provider-priority", provider: "", model: "", billing: "configured", fallback: "safe", ...value }; @@ -18,6 +21,7 @@ export function parseRoutingPolicy(value) { if (result.provider !== "" && !PROVIDERS.includes(result.provider)) throw new RoleplayRequestError("Invalid routing provider"); if (typeof result.model !== "string" || result.model.length > 200 || /[\s\u0000-\u001f]/.test(result.model)) throw new RoleplayRequestError("Invalid routing model"); if (result.mode === "pinned" && (!result.provider || !result.model)) throw new RoleplayRequestError("Pinned routing requires provider and model"); + if (Object.hasOwn(result, "session_tier")) result.session_tier = parseSessionTier(result.session_tier); return result; } diff --git a/worker/roleplay/session-tier-policy.mjs b/worker/roleplay/session-tier-policy.mjs new file mode 100644 index 00000000..8d5f1f4f --- /dev/null +++ b/worker/roleplay/session-tier-policy.mjs @@ -0,0 +1,186 @@ +import { RoleplayTurnError } from "./turn-runtime.mjs"; + +const LANES = new Set(["main", "delegation", "aux"]); +const MAX_TOOLS = 128; +let warned = false; +export class SessionTierError extends RoleplayTurnError {} +const error = (code, message, status = 503) => new SessionTierError(code, status, message); + +export function sessionTierSettings(env = {}) { + const mode = String(env.SESSION_TIER_MODE || "off").trim() || "off"; + const raw = String(env.SESSION_TIER_TTL_SECONDS || "1800").trim() || "1800"; + const ttlSeconds = Number(raw); + if (!["off", "sticky"].includes(mode) || !/^\d+$/.test(raw) || !Number.isSafeInteger(ttlSeconds) || ttlSeconds < 1 || ttlSeconds > 86400) { + if (!warned) { warned = true; console.warn("Invalid session tier settings; session tiers disabled"); } + return { enabled: false, ttlSeconds: 1800 }; + } + return { enabled: mode === "sticky", ttlSeconds }; +} + +export function parseSessionTier(value) { + if (!value || typeof value !== "object" || Array.isArray(value) || + Object.keys(value).sort().join(",") !== "approved_model,approved_tier,lane" || + !LANES.has(value.lane) || !Number.isInteger(value.approved_tier) || value.approved_tier < 0 || value.approved_tier > 100 || + typeof value.approved_model !== "string" || value.approved_model.length > 256 || + !/^[a-z][a-z0-9-]{0,31}:[A-Za-z0-9][A-Za-z0-9._:/+@-]*$/.test(value.approved_model)) { + throw error("invalid_session_tier", "Session tier requires a lane and explicit model/tier approval", 400); + } + return { ...value }; +} + +async function digest(...parts) { + const bytes = await crypto.subtle.digest("SHA-256", new TextEncoder().encode(JSON.stringify(parts))); + return Array.from(new Uint8Array(bytes), byte => byte.toString(16).padStart(2, "0")).join(""); +} +const modelId = c => `${c.provider}:${c.model}`; +async function toolHash(id) { + if (typeof id !== "string" || !id.trim() || id.length > 200) throw error("invalid_session_tier", "Invalid tool call marker", 400); + return digest("tool", id); +} +async function pendingAfterMessages(prior, messages) { + if (!Array.isArray(messages) || messages.length > 256) throw error("invalid_session_tier", "Invalid session messages", 400); + const pending = new Set(prior); + for (const message of messages) { + if (!message || typeof message !== "object") throw error("invalid_session_tier", "Invalid session message", 400); + if (message.role === "assistant") { + const calls = message.tool_calls || []; + if (!Array.isArray(calls) || calls.length > MAX_TOOLS) throw error("session_tier_tool_limit", "Invalid tool call markers", 400); + for (const call of calls) pending.add(await toolHash(call?.id)); + } else if (message.role === "tool") pending.delete(await toolHash(message.tool_call_id)); + if (pending.size > MAX_TOOLS) throw error("session_tier_tool_limit", "Too many outstanding tool calls", 400); + } + return [...pending].sort(); +} + +export async function sessionTierRevision(candidates) { + // Route metadata only: never fingerprint or retain the configured credentials. + return digest("roleplay-policy", candidates.map(c => [c.provider, c.model, c.credentialId, + c.subscriptionOnly, c.billingMode, c.familyRank, c.modelRank, c.providerRank, + c.priorityRank, c.contextWindow, c.maxOutputTokens])); +} + +export class SessionTierPolicy { + constructor(storage, settings, clock = () => Date.now() / 1000) { + this.storage = storage; + this.settings = settings; + this.clock = clock; + this.tail = Promise.resolve(); + } + + async update(key, operation) { + const previous = this.tail; + let release; + this.tail = new Promise(resolve => { release = resolve; }); + await previous; + try { + const row = await this.storage.get(key); + const [next, result] = await operation(row); + if (next) await this.storage.put(key, next); + else if (row) await this.storage.delete(key); + return result; + } catch (e) { + if (e instanceof RoleplayTurnError) throw e; + throw error("session_tier_storage_unavailable", "Session tier storage is unavailable"); + } finally { release(); } + } + + async begin(raw, candidates, messages, revision, { explicit = false, unavailable = () => false } = {}) { + if (!this.settings.enabled || !raw || explicit) return null; + const requested = parseSessionTier(raw); + // This repository belongs to an authenticated, session-scoped Durable Object. + const key = `session-tier:${await digest("lane", requested.lane)}`; + const policyRevision = await digest("revision", revision); + return this.update(key, async row => { + if (row && row.expires_at <= this.clock()) row = null; + if (row?.lease) throw error("session_tier_busy", "A session lane already has an unresolved request", 409); + if (row && row.policy_revision !== policyRevision) row = null; + const pending = await pendingAfterMessages(row?.pending_tools ?? [], messages); + const safe = !pending.length && messages.at(-1)?.role === "user" && row?.safe_turn; + const change = !row || (safe && (row.approved_model !== requested.approved_model || row.tier !== requested.approved_tier)); + const approved = change ? requested.approved_model : row.approved_model; + const tier = change ? requested.approved_tier : row.tier; + const nativeModel = approved.slice(approved.indexOf(":") + 1); + // Roleplay has no reviewed numeric quality tiers. The approval is a label; + // fallback is restricted to the identical native model on eligible providers. + const allowed = candidates.filter(c => c.model === nativeModel && !unavailable(c)); + if (change && !allowed.some(c => modelId(c) === approved)) throw error("session_tier_ineligible", "Approved model is not eligible"); + if (!allowed.length) return [null, error("session_tier_ineligible", "No eligible route remains for approved model")]; + const next = { lane: requested.lane, tier, approved_model: approved, + actual_model: change ? approved : row.actual_model, policy_revision: policyRevision, + expires_at: this.clock() + this.settings.ttlSeconds, safe_turn: false, + pending_tools: pending, lease: crypto.randomUUID() }; + const ordered = [...allowed].sort((a, b) => Number(modelId(a) !== next.actual_model) - Number(modelId(b) !== next.actual_model)); + return [next, new SessionTierTurn(this, key, next, ordered)]; + }).then(result => { if (result instanceof Error) throw result; return result; }); + } +} + +class SessionTierTurn { + constructor(policy, key, row, candidates) { + this.policy = policy; this.key = key; this.row = row; this.candidates = candidates; + this.observedTools = []; + this.observationInvalid = false; + } + + select(candidates) { + const allowed = new Set(this.candidates.map(modelId)); + return candidates.filter(c => allowed.has(modelId(c))).sort((a, b) => + Number(modelId(a) !== this.row.actual_model) - Number(modelId(b) !== this.row.actual_model)); + } + + observe(response) { + if (!response.body || !response.headers.get("content-type")?.includes("text/event-stream")) return response; + const decoder = new TextDecoder(); + let buffer = ""; + const ids = new Map(); + const transform = new TransformStream({ transform: (chunk, controller) => { + controller.enqueue(chunk); + if (this.observationInvalid) return; + buffer += decoder.decode(chunk, { stream: true }); + if (buffer.length > 65536) { this.observationInvalid = true; buffer = ""; return; } + const lines = buffer.split(/\r?\n/); buffer = lines.pop(); + for (const line of lines) { + if (!line.startsWith("data:") || line.slice(5).trim() === "[DONE]") continue; + try { + const frame = JSON.parse(line.slice(5)); + for (const call of frame.choices?.[0]?.delta?.tool_calls ?? []) { + if (!Number.isInteger(call.index) || call.index < 0 || call.index >= MAX_TOOLS) { this.observationInvalid = true; break; } + if (typeof call.id === "string") ids.set(call.index, (ids.get(call.index) ?? "") + call.id); + if ((ids.get(call.index)?.length ?? 0) > 200) this.observationInvalid = true; + this.observedTools = [...ids.values()].map(id => ({ id })); + } + } catch { this.observationInvalid = true; } + } + }, flush: () => { this.observedTools = [...ids.values()].map(id => ({ id })); buffer = ""; ids.clear(); } }); + return new Response(response.body.pipeThrough(transform), { status: response.status, statusText: response.statusText, headers: response.headers }); + } + + async finish(candidate, { success = false, toolCalls = this.observedTools, finishReason = "", credentialFailed = false } = {}) { + return this.policy.update(this.key, async current => { + if (!current || current.lease !== this.row.lease) return [current, null]; + if (credentialFailed) return [null, null]; + if (candidate) { + if (!this.candidates.some(c => modelId(c) === modelId(candidate))) throw error("session_tier_ineligible", "Actual route is outside approved model"); + current.actual_model = modelId(candidate); + } + if (!Array.isArray(toolCalls) || toolCalls.length > MAX_TOOLS) throw error("session_tier_tool_limit", "Too many tool calls"); + const pending = new Set(current.pending_tools); + for (const call of toolCalls) pending.add(await toolHash(call?.id)); + if (this.observationInvalid || (finishReason === "tool_calls" && !toolCalls.length)) { + pending.add(await digest("unobserved-tool-state")); + } + if (pending.size > MAX_TOOLS) throw error("session_tier_tool_limit", "Too many outstanding tool calls"); + current.pending_tools = [...pending].sort(); + current.safe_turn = Boolean(candidate && success && !this.observationInvalid && (finishReason !== "tool_calls" || toolCalls.length)); + current.lease = ""; + return [current, null]; + }); + } +} + +export async function beginRoleplaySessionTier(policy, parsed, payload, candidates, stats, configured) { + const raw = parsed.routing.session_tier; + if (!raw || !policy.settings.enabled || parsed.routing.mode === "pinned" || parsed.routing.model) return null; + return policy.begin(raw, candidates, payload.messages ?? parsed.messages, + await sessionTierRevision(configured), { unavailable: c => stats[c.key]?.cooldownUntil > Date.now() }); +}