Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions docs/canary-traffic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Canary traffic for auto routes

`CANARY_TRAFFIC_ENABLED` defaults to `false`. An empty value also disables it.
Malformed values disable it and produce one warning without logging the value.
Each route defaults to `{"enabled": false}`. Explicit provider models and routes
without an enabled canary policy keep their existing dispatch and response headers.

An administrator saves the candidate chain and its canary policy together through
the existing authenticated `PUT /admin/auto-routes` endpoint. For example:

```json
{
"route_id": "auto:experiment",
"candidates": ["openai:baseline", "openai:candidate"],
"canary": {
"enabled": true,
"mode": "shadow",
"weights": {"baseline": 90, "candidate": 10},
"salt_revision": "r1",
"approved_candidates": ["openai:candidate"]
}
}
```

With `CANARY_TRAFFIC_ENABLED=true`, shadow mode assigns a proposed cohort and
records its proposed order. It sends exactly the baseline request; it makes no
comparison or additional provider call. Change `mode` to `live` in a reviewed save
to use the proposed order for candidate sessions. Approved candidates move ahead
of the other candidates, preserving the configured order within each group.
Health ordering and per-candidate permission, budget, credential, cooldown and
deadline checks run afterwards and retain precedence. The existing refusal and
stream failover rules still apply. Assignment does not grant retry permission.

Weights must be non-negative integers summing to 100. Approved candidates must
be distinct models already in this route. A nonzero enabled candidate weight
requires an approved candidate. The salt revision is a nonempty identifier of
at most 128 letters, digits, dots, underscores or hyphens, starting with a letter
or digit. Missing optional fields use shadow mode, 100/0 weights, revision `1`
and no approved candidates. Invalid configuration returns HTTP 400 before saving.
Weights never change automatically.

For local storage, include `expected_updated_at` from the route read to reject
a concurrent edit with HTTP 409. Durable saves require `current_revision` when
configuration snapshots or revision sync are enabled; stale revisions return
409. Candidate order, policy and the durable revision commit atomically. Any
durable save failure returns 503 rather than retaining an unconfirmed local copy.
A save omitting `canary` resets the route to its disabled default. To disable an
experiment explicitly, save its chain with `"canary": {"enabled": false}`.
Clearing the global flag disables routing changes without changing saved policy.

Cohorts use HMAC-SHA256 with the existing server-side `JWT_SECRET`. The input is
compact UTF-8 JSON containing the format label `multillm-canary-v1`, the verified
principal, session identifier, route ID and salt revision. The principal is the
compact JSON pair `[tenant_id, id-or-username]` from authenticated request state.
The first four digest bytes, interpreted as an unsigned big-endian integer modulo
100, select baseline below its weight and candidate otherwise. Changing weights,
salt revision, identity or the server secret can reassign sessions.

Session lookup follows prompt cache affinity: `X-OpenCode-Session`, `Session-Id`,
`Thread-Id`, body `session_id`, body `conversation_id`, metadata `session_id`,
metadata `conversation_id`, then the signed authenticated session ID. Without a
session or verified principal the cohort is baseline. A caller-supplied principal
or cohort is ignored. An unavailable assignment secret gives baseline routing and
one warning. A session identifier is an affinity hint, not an authorization grant.

Responses from enabled routes carry the browser-readable
`X-MultiLLM-Canary-Cohort: candidate; mode=shadow` header (or `baseline` and `live`
as appropriate). Cohort identifies an assignment, not which provider won after
the existing filters and fallback. The auto-selected-model header remains authoritative.

The additive `0025_canary_traffic.sql` migration stores only reviewed route policy
and the matching route update timestamp in `canary_traffic`. Apply it before
enabling durable routing. A missing table, failed read or invalid stored policy
falls back to baseline with one warning and does not cause a generation 500.
An older policy never applies to a newly saved candidate chain. Local storage
creates this table only during an explicit canary save; ordinary saves do not.

Request observations contain route, cohort, mode, proposed order and health-ordered
candidate order; candidates rejected during validation are removed from the latter.
Unattempted candidates still need their normal validation if reached. Aggregate
cohort counts retain only route/cohort/mode/count, with at most 800 counters per
process. Counts reset on process restart and measure requests, not unique sessions.
Native Worker dispatch supplies verified identity through `authenticatedPrincipal`
and `sessionIdentifier`, applies `prepareCanary` before eligibility, and observes
the resulting order through its request lifecycle. Forwarded requests use the
Container assignment. Neither runtime stores session plaintext, principal,
HMAC output, prompts, responses or credentials for this feature.

Shadow mode has the baseline request's cost. Live mode can select a more expensive
approved model; existing budgets still apply. Cohort counts do not establish
quality, statistical significance, provider capabilities or exact billing.
6 changes: 6 additions & 0 deletions intelligence-migrations/0025_canary_traffic.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
-- Reviewed route metadata only. Assignments retain no session or principal.
CREATE TABLE IF NOT EXISTS canary_traffic (
route_id TEXT PRIMARY KEY,
route_updated_at TEXT NOT NULL,
configuration TEXT NOT NULL
);
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_context_pages_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_wave4_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
2 changes: 1 addition & 1 deletion route_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
"X-MultiLLM-Auto-Route, X-MultiLLM-Auto-Selected-Model, "
"X-MultiLLM-Auto-Attempts, X-MultiLLM-Auto-Selected-Priority, "
"X-MultiLLM-Auto-Ordering, X-MultiLLM-Auto-Failover-Reasons, "
"X-MultiLLM-Cache, X-MultiLLM-Cache-Backend, X-MultiLLM-Usage-Basis, X-MultiLLM-Provider-Calls, Age, X-MultiLLM-Retention, X-MultiLLM-Roleplay-Recovery, X-MultiLLM-Transport-Failure, X-MultiLLM-Token-Count, "
"X-MultiLLM-Cache, X-MultiLLM-Cache-Backend, X-MultiLLM-Usage-Basis, X-MultiLLM-Provider-Calls, Age, X-MultiLLM-Retention, X-MultiLLM-Canary-Cohort, X-MultiLLM-Roleplay-Recovery, X-MultiLLM-Transport-Failure, X-MultiLLM-Token-Count, "
"X-MultiLLM-Latency-Ms, X-MultiLLM-Estimated-Cost-USD, "
"X-MultiLLM-Cost-Basis, WWW-Authenticate, X-PAYMENT-RESPONSE, X-Poll-After, "
"X-NanoGPT-Advisor-ID, X-NanoGPT-Data-Endpoint, "
Expand Down
25 changes: 24 additions & 1 deletion routes/auto_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from services.model_cooldown import ModelCooldownCapacity, ModelCooldownExhausted
from providers.registry import get_registry
from services import cloudflare_ai
from services import canary_traffic
from services.auto_route_service import AutoRoute, AutoRouteService
from services.media_catalog import (
TRANSPORT_FAILURE_HEADER,
Expand Down Expand Up @@ -129,6 +130,7 @@ def _decorate_response(
route_decision: str | None = None,
ordering: str = "priority",
failures: list[tuple[str, str]] | None = None,
canary: canary_traffic.CanaryAssignment | None = None,
) -> Response:
selected_provider, _ = ModelRegistry.parse_model_id(selected_model)
if route_decision is None:
Expand All @@ -147,6 +149,8 @@ def _decorate_response(
g.multillm_provider = selected_provider
g.multillm_model = selected_model
g.multillm_route_decision = route_decision
if canary is not None:
canary.decorate(response)
return response


Expand All @@ -172,7 +176,11 @@ def dispatch_auto_route(
status_code=404,
)

order = RouteHealth.order(route.id, route.candidates, circuit_state=_circuit_state)
canary = canary_traffic.prepare_request(route, payload)
candidates = canary.dispatch_order if canary is not None else route.candidates
order = RouteHealth.order(route.id, candidates, circuit_state=_circuit_state)
if canary is not None:
canary.observe(order.candidates)
priorities = {candidate: index for index, candidate in enumerate(route.candidates)}
attempts = 0
failures: list[tuple[str, str]] = []
Expand All @@ -195,6 +203,9 @@ def dispatch_auto_route(
type(error).__name__,
)
failures.append((candidate, "skipped"))
if canary is not None:
g.multillm_canary["eligible_order"] = [model for model in g.multillm_canary["eligible_order"]
if model != candidate]
continue

attempts += 1
Expand Down Expand Up @@ -260,6 +271,7 @@ def dispatch_auto_route(
route_decision=route_decision,
ordering=order.mode,
failures=failures,
canary=canary,
)

logger.info("Auto route %s moves past %s (%s)", route.id, candidate, reason)
Expand All @@ -281,6 +293,7 @@ def dispatch_auto_route(
route_decision=last_decision,
ordering=order.mode,
failures=[failure for failure in failures if failure[0] != last_model],
canary=canary,
)
raise APIError(
f"No configured provider is available for auto route: {route.id}",
Expand Down Expand Up @@ -428,6 +441,7 @@ def _admin_payload(app, auth_service_cls) -> dict:
"candidates": candidates,
"updated_at": route.updated_at,
"ordering": settings.mode_for(route.id),
**({"canary": route.canary.as_dict()} if route.canary.enabled else {}),
}
)
return {
Expand All @@ -454,6 +468,9 @@ def register_auto_route_admin_routes(
auth_service_cls,
proxy_service_cls,
) -> None:
# Preserve cohort visibility when existing policy guards raise an API error.
app.after_request(canary_traffic.decorate_current_response)

@app.route("/admin/auto-routes", methods=["GET", "PUT"])
@login_required
def admin_auto_routes():
Expand All @@ -467,10 +484,16 @@ def admin_auto_routes():
status_code=400,
)
try:
canary_options = {"canary": payload["canary"]} if "canary" in payload else {}
if "expected_updated_at" in payload:
canary_options["expected_updated_at"] = payload["expected_updated_at"]
if "current_revision" in payload:
canary_options["current_revision"] = payload["current_revision"]
AutoRouteService.save_route(
payload.get("route_id"),
payload.get("candidates"),
app.config["API_BASE_URLS"],
**canary_options,
)
except ValueError as error:
raise APIError(str(error), status_code=400) from error
Expand Down
Loading
Loading