Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 107 additions & 0 deletions docs/session-tiers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Role-scoped sticky session tiers

Session tiers keep an approved route stable across tool calls in the `main`,
`delegation` and `aux` lanes. They are disabled by default:

```text
SESSION_TIER_MODE=off
SESSION_TIER_TTL_SECONDS=1800
```

With the feature disabled, existing routing, response headers and storage are
unchanged. An empty setting uses its default. A malformed mode or TTL disables
the feature and logs one warning without its value. The TTL must be 1–86400 seconds.

Enable sticky routing with `SESSION_TIER_MODE=sticky`. Requests without tier
metadata continue to use ordinary routing. An explicit model or pinned roleplay
route remains caller-selected and never acquires a tier binding.

Managed Chat Completions requests use this metadata alongside the normal body:

```json
{
"model": "auto:intelligence",
"messages": [{"role": "user", "content": "Continue the task."}],
"session_tier": {
"session": "example-session",
"lane": "main",
"approved_model": "openai:reviewed-model",
"approved_tier": 2
}
}
```

The managed dispatcher must bind the session to its authenticated principal,
remove this metadata before request parsing and upstream dispatch, and supply
the current reviewed policy revision. `prepare_session_tier` accepts only already
eligible candidates and an optional credential/cooldown admission predicate.
`select_candidates(..., session_tier=turn)` preserves the existing eligibility
checks and applies the lane's ordering. Finalization records the actual candidate
and tool-call IDs after classified completion. It does not authorize another attempt.

Public roleplay requests carry the approval inside `routing`:

```json
{
"model": "roleplay:intelligence",
"session_id": "example-session",
"messages": [{"role": "user", "content": "Continue."}],
"routing": {
"session_tier": {
"lane": "main",
"approved_model": "opencode:glm-5.3-flash",
"approved_tier": 1
}
}
}
```

Roleplay storage belongs to the existing authenticated session's Durable Object.
The tier is an explicit numeric label, not a model-quality measurement. Roleplay
has no reviewed numeric tier catalogue, so fallback can use only an identical
native model name on another eligible provider. Provider aliases with different
native names do not qualify. General managed candidates must match both the approved
native model and reviewed `quality_tier`. Billing, entitlement, capability, context limits,
credential availability and cooldown always take precedence over stickiness.

A different approved model or tier takes effect only at a user-turn boundary
after successful completion and all matching tool results. Partial or unrelated
results leave the outstanding calls unresolved. A continuation without a new user
turn cannot change the approval. Failures, cancellations and ambiguous results
cannot open a safe boundary. Streaming roleplay observes bounded tool-call IDs
without changing response bytes; incomplete or oversized observations prevent a
safe boundary. Unknown streamed tool state stays unresolved until expiry or revision
invalidation. Roleplay automatic continuation, refusal fallback and output-contract
repair are disabled for an opted-in tier turn; existing pre-output fallback rules
still apply within the constrained candidate list.

The approved route and actual fallback route are separate fields. A fallback is
recorded honestly and preferred on subsequent turns without rewriting the approved
route. Expiry, a policy revision change or confirmed credential failure invalidates
only the affected lane. An in-flight request holds a lease; another request for
that lane receives HTTP 409 `session_tier_busy`. Late completion cannot overwrite
a replacement lease. A lease expires with the lane's TTL, including after a crash.

Bindings retain hashed principal/session identities, lane, route identifiers, a
hashed revision, expiry, a completed-generation marker, hashed outstanding tool
IDs and a lease. They add no prompt, response, tool arguments or result content
retention. Existing zero-retention restrictions and response headers still apply.
Live bindings expire after the configured idle TTL; the local adapter prunes
expired entries during access and the Durable Object reuses its bounded three-lane
storage. This feature does not create a transcript or change existing conversation
memory behavior.

The local Python adapter is process-local and permits at most 10,000 live lanes.
Use the injected SQLite adapter when persistence across restarts is required;
it also limits live rows to 10,000. Apply the additive `session_tiers` migration
before using that adapter. It never creates tables during a request. Missing or
unavailable enabled storage returns HTTP 503 `session_tier_storage_unavailable`.
Capacity exhaustion and unavailable approved routes return HTTP 503; invalid
approval metadata returns HTTP 400. Tool markers are limited to 128 outstanding
calls, with identifiers at most 200 characters. Worker storage needs no D1 table
because roleplay retains its existing Durable Object ownership.

Stickiness does not reserve spend, guarantee a provider's capabilities or metering,
grant paid access, or promote a model autonomously. Every approval still passes
the existing operator policy and budget checks. A remote classifier cannot choose
the approved tier. Storage failure is an error, never a synthetic completion.
14 changes: 14 additions & 0 deletions intelligence-migrations/0021_session_tiers.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
-- Scoped hashes, reviewed route identifiers and bounded turn markers only.
CREATE TABLE IF NOT EXISTS session_tiers (
scope_hash TEXT PRIMARY KEY CHECK(length(scope_hash) = 64),
lane TEXT NOT NULL CHECK(lane IN ('main', 'delegation', 'aux')),
tier INTEGER NOT NULL CHECK(tier >= 0 AND tier <= 100),
approved_model TEXT NOT NULL CHECK(length(approved_model) <= 256),
actual_model TEXT NOT NULL CHECK(length(actual_model) <= 256),
policy_revision TEXT NOT NULL CHECK(length(policy_revision) = 64),
expires_at REAL NOT NULL,
safe_turn INTEGER NOT NULL CHECK(safe_turn IN (0, 1)),
pending_tools TEXT NOT NULL CHECK(json_valid(pending_tools) AND length(pending_tools) <= 9000),
lease TEXT NOT NULL CHECK(length(lease) <= 64)
);
CREATE INDEX IF NOT EXISTS session_tiers_expiry ON session_tiers(expires_at);
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_session_tiers_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
15 changes: 10 additions & 5 deletions services/intelligence_policy.py
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ def expected_reply_ms(candidate, request, now=None):
return first_ms + tokens * 1000 / max(rate, 1)


def select_candidates(policy, request, config):
def select_candidates(policy, request, config, *, session_tier=None):
candidates = []
for priority, candidate in enumerate(policy["candidates"]):
if request.explicit and candidate["model"] != request.payload["model"]:
Expand Down Expand Up @@ -252,28 +252,33 @@ def rank(item):
return (-score, -candidate.get("quality_tier", 0), *speed, priority)
return (priority, -score, *speed)

def apply_session_tier(choices):
if session_tier is not None and not request.explicit:
return session_tier.select(choices)
return choices

ranked = sorted(candidates, key=rank)
if request.explicit or request.profile == "balanced":
return [candidate for _, candidate in ranked]
return apply_session_tier([candidate for _, candidate in ranked])
preference = (
validate_preference(policy["precision_preference"])
if "precision_preference" in policy
else configured_preference()
)
if not preference or len(ranked) < 2:
return [candidate for _, candidate in ranked]
return apply_session_tier([candidate for _, candidate in ranked])
metadata = {
f"{model.provider}:{model.model_id}": model.metadata
for model in ProviderCatalogService.list_models()
}
# Every original ranking dimension precedes precision. A quality tier also
# separates fast-profile ties without altering their existing relative order.
return prefer_precision(
return apply_session_tier(prefer_precision(
ranked,
preference,
lambda item: (*rank(item)[:-1], item[1].get("quality_tier", 0)),
metadata,
)
))


def model_advertisement(policy, config=None):
Expand Down
Loading
Loading