Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 110 additions & 0 deletions docs/latency-slo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Latency-SLO admission

Latency admission predicts completion risk before provider handoff. It uses measured
time to first visible output and output throughput from completed observations.
It never sends generation probes, writes routing policy, changes an explicit model,
or grants permission to retry a request that may have generated output.

## Configuration

`LATENCY_SLO_MODE=off` is the default. Empty values also mean off. In this mode,
admission, routing, responses, headers and observation storage remain unchanged.
Malformed settings disable admission and produce one warning per setting without
including the value. Both runtimes use the same settings and prediction formula.

To reject predicted misses on an exact managed route:

```sh
LATENCY_SLO_MODE=reject
LATENCY_SLO_MIN_SAMPLES=20
LATENCY_SLO_MAX_MODELS=128
LATENCY_SLO_POLICY_JSON='{"routes":{"/v1/chat/completions":{"deadline_ms":5000,"require_coverage":false}}}'
```

To allow eligible alternatives for an automatic route:

```sh
LATENCY_SLO_MODE=reroute
LATENCY_SLO_POLICY_JSON='{"routes":{"auto:intelligence":{"deadline_ms":5000,"require_coverage":true}},"keys":{"account-42":{"deadline_ms":3000}}}'
```

`routes` matches exact HTTP paths or exact automatic model identifiers. `keys`
matches an authenticated key/account identifier, never a caller-provided header
or API credential. Each entry requires integer `deadline_ms` between 1 and
300000; `require_coverage` is an optional boolean, default false. If several
entries match, the smallest deadline and any coverage requirement win. No match
means ordinary policy. The default policy is `{}`.

Policies reject duplicate keys, unknown fields, invalid identifiers, more than
128 entries per scope, or more than 32768 UTF-8 bytes. The sample minimum is an
integer between 1 and 1000, default 20. The model cap is an integer between 1 and
512, default 128. Invalid numeric settings disable the feature.

## Prediction and admission

Each model retains at most 1000 paired observations from the last 15 minutes.
The oldest inactive models are evicted when the model cap is reached. Partial
measurements, nonpositive throughput, invalid timing, and future samples do not
provide coverage. Measurements are process/isolate-local and are lost on restart.
Stored routing-health averages and reviewed latency estimates do not count as
recent SLO samples. With admission disabled, no new SLO observations are stored.

For each recent pair, predicted milliseconds are
`ttft_ms + requested_output_tokens * 1000 / tokens_per_second`. The prediction is
the empirical p95 using the nearest-rank method, rounded up. If both `max_tokens`
and `max_completion_tokens` are present, the smaller valid limit is used. The
default is 1024 output tokens. Intelligence selection uses its already validated
output limit. Limits must be integers from 1 to 131072; requests outside this
prediction bound are unknown and remain subject to ordinary request validation.
Admission never rewrites the payload to impose a prediction limit.

The decision reports sample count, required sample count, coverage fraction,
latest observation age, window length, requested output, and predicted time.
Successful Flask decisions are request-local; errors include this content-free
report in `error.prediction`. No response header is added.

Sparse or old evidence yields `prediction_unknown`. Ordinary policy applies
unless coverage is required, in which case admission returns HTTP 503 with
`latency_slo_unavailable`. A known miss in reject mode returns HTTP 503 with
`latency_slo_predicted_miss` before provider handoff. Both errors have
`retryable: false` and `Cache-Control: no-store`.

Reroute mode changes only automatic selections, using alternatives already
eligible under the existing authorization, entitlement, privacy, capabilities,
context, billing and approved session-lane policy. Alternatives must have the
same reviewed quality tier as the original selection and a known prediction
within the deadline. Missing tier approval never authorizes rerouting. An
explicit model receives the reject behavior on a known miss; unknown predictions
pass through when coverage is optional. A reroute removes unsafe alternatives
from that selection rather than adding models or retry attempts.

Generation deadlines remain the hard runtime limit. Admission uses the remaining
runtime budget when present and never extends, resets, or cancels that deadline.
A passing prediction can still time out or fail. Predictions are neither billing
facts nor a guarantee of provider latency, and rejected admission creates no
provider usage or fabricated successful completion.

## Runtime interfaces

Flask registers `latency_slo_request_hook` immediately after the generation
deadline hook and before the idempotency claim. Managed chat, messages, Responses
and intelligence requests are covered; provider passthrough is unchanged.
`latency_slo_candidate_policy` is an explicit read-only collaborator returning an
ordered list after eligibility and approved lane selection. An unresolved auto
selection has unknown coverage at this early boundary. Intelligence selection
and automatic route-health ordering enforce the policy again on their actual
candidate lists before dispatch. Existing provider validation still applies.

Native Worker routes call `prepareLatencySLO` after verified authentication and
deadline setup, with canonical provider/model IDs, the verified key identifier,
and already eligible approved candidates. The returned rejection must be handled
before dispatch, and rerouted candidates must remain restricted at dispatch.
`recordLatencyObservation` consumes the existing telemetry finalization event.
Only successful, measured, non-cache observations with positive observed output
and time after first output contribute throughput. Forwarded or passthrough
traffic must not call this native admission interface twice.

Observation windows contain model IDs and numeric timings only. They store no
prompts, completions, credentials, identity, queue size, prices or usage ledgers.
Content retention controls remain independent. No database table or migration is
required.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs tests/test_latency_slo_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
15 changes: 13 additions & 2 deletions services/gateway_extensions.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

AUTHENTICATED_HOOK_ORDER = (
"request_policy_hook", "prompt_injection_request_hook", "spillover_hook", "pii_request_hook",
"responses_state_hook", "generation_deadline_hook", "idempotency_request_hook", "admit",
"responses_state_hook", "generation_deadline_hook", "latency_slo_request_hook", "idempotency_request_hook", "admit",
)


Expand Down Expand Up @@ -93,7 +93,7 @@ def gateway_callbacks(*, csrf=None):
from middleware.rate_limit_headers import register_rate_limit_headers
from services.pii_redaction import register_pii_redaction
return (register_retention, register_injection_decision, register_batch_spillover, register_pii_redaction,
partial(register_hosted_responses, csrf=csrf), register_deadline, register_managed_idempotency,
partial(register_hosted_responses, csrf=csrf), register_deadline, register_latency_slo, register_managed_idempotency,
register_admission, register_cooldown_errors, register_rate_limit_headers)


Expand Down Expand Up @@ -125,6 +125,17 @@ def register_deadline(app):
register_generation_deadline(app, verify_internal=verified_internal_transport, limits_ms=generation_limits_ms)


def latency_slo_candidate_policy(payload):
"""Read-only collaborator returns candidates after eligibility and approved lane selection."""
callback = current_app.extensions.get("latency_slo_candidate_policy")
return callback(payload) if callback is not None else None


def register_latency_slo(app):
from services.latency_slo import register_latency_slo as register
register(app, is_managed=managed_generation_request, candidates=latency_slo_candidate_policy)


def managed_policy_revision(payload):
from services.managed_turn import idempotency_policy_revision
return idempotency_policy_revision(payload)
Expand Down
7 changes: 5 additions & 2 deletions services/intelligence_policy.py
Original file line number Diff line number Diff line change
Expand Up @@ -254,8 +254,11 @@ def rank(item):

def apply_session_tier(choices):
if session_tier is not None and not request.explicit:
return session_tier.select(choices)
return choices
choices = session_tier.select(choices)
from services.latency_slo import selection_candidates
return selection_candidates(choices, route="auto:intelligence",
output_tokens=request.output_tokens, auto=not request.explicit,
lane_selected=session_tier is not None)

ranked = sorted(candidates, key=rank)
if request.explicit or request.profile == "balanced":
Expand Down
Loading
Loading