Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions docs/generation-deadline.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Generation deadlines

Send `X-MultiLLM-Deadline-Ms` to set a relative budget for one generation. The
header accepts an integer from 1 to `GENERATION_DEADLINE_MAX_MS` (300000 by
default). Setup, token counting, admission, allowed retry waits, provider calls,
stream preflight and body reads all consume that same budget. A shorter key or
route limit takes precedence.

Without the header, existing timeouts, retries, response bytes and stream behavior
remain unchanged. An empty `GENERATION_DEADLINE_MAX_MS` uses the default. A
malformed maximum disables deadlines and logs one warning without its value.
A malformed request header returns HTTP 400 `invalid_generation_deadline`.

For example, this request has a ten-second generation budget:

```sh
curl "$GATEWAY_URL/v1/chat/completions" \
-H "Authorization: Bearer $GATEWAY_TOKEN" \
-H 'Content-Type: application/json' \
-H 'X-MultiLLM-Deadline-Ms: 10000' \
-d '{"model":"openai:gpt-4.1","messages":[{"role":"user","content":"Hello"}],"stream":true}'
```

Omit the deadline header to use the existing timeouts:

```sh
curl "$GATEWAY_URL/v1/chat/completions" \
-H "Authorization: Bearer $GATEWAY_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"model":"openai:gpt-4.1","messages":[{"role":"user","content":"Hello"}]}'
```

Before response commitment, expiry returns HTTP 504 with an error whose code is
`generation_deadline_exceeded`. After an SSE stream is committed, expiry emits
an error event in the selected Chat, Responses or Messages protocol and cancels
the upstream. It does not append a stop event or `[DONE]`, or start a fallback.
Binary streams terminate with a transport error because they have no SSE error
envelope. Chat SSE preflight uses the existing 65536-byte/32-event validation
bounds. Other SSE prefixes inspect at most 65536 bytes before commitment;
transport reads can deliver a larger chunk containing that prefix.
Exhausting the prefix bound or reaching EOF before useful output returns a real
HTTP 502 `upstream_stream_invalid` error and does not authorize replay.

The budget uses each host's monotonic clock. Forwarding sends only the rounded-down
remaining milliseconds in `X-MultiLLM-Internal-Deadline-Ms`. The forwarding
boundary strips any caller-supplied internal budget. The receiving Flask boundary
accepts that header only after its fixed transport verifier authenticates the
internal hop; public API authentication alone is not proof of an internal hop.
No monotonic timestamp is transmitted or compared across machines.

The deadline limits waiting and requests local cancellation. It cannot guarantee
that a provider stops computation or billing. Synchronous CPU setup cannot be
preempted; it is checked before further dispatch. Flask transport connects and
reads use bounded timeouts, and the cancellation owner closes a handed-off
response at expiry. A timeout before upstream handoff relies on the transport's
connect/read timeout rather than a separate socket owner.

Cancellation never grants replay permission. A handoff without confirmed final
usage remains ambiguous, even after clean EOF. Finalizers use
`settlement_information` (Flask) or `settlementInformation` (Worker) to retain
unknown usage and reservation information for settlement. Deadlines do not
create a durable reservation store or estimate a successful charge. They store
no prompt or response content and do not change the request's retention policy.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
2 changes: 1 addition & 1 deletion route_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

CORS_ALLOWED_METHODS = "GET, POST, PUT, DELETE, PATCH, OPTIONS"
CORS_DEFAULT_HEADERS = (
"Authorization, Content-Type, Accept, Origin, X-Requested-With, X-MultiLLM-Tool-Repair, X-MultiLLM-Conversion-Report, X-MultiLLM-Retention, "
"Authorization, Content-Type, Accept, Origin, X-Requested-With, X-MultiLLM-Tool-Repair, X-MultiLLM-Conversion-Report, X-MultiLLM-Retention, X-MultiLLM-Deadline-Ms, "
"X-Api-Key, X-Goog-Api-Key, X-MultiLLM-Api-Key, Anthropic-Version, "
"Anthropic-Beta, Anthropic-Dangerous-Direct-Browser-Access, "
"Idempotency-Key, X-MultiLLM-Image-QA, X-MultiLLM-Cascade, OpenAI-Beta, OpenAI-Organization, "
Expand Down
10 changes: 9 additions & 1 deletion routes/auto_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@
from services.resilience_service import ResilienceService
from services.route_health import RouteHealth, ordering_settings
from services.stream_preflight import preflight_chat_stream
from services.generation_deadline import (
GenerationDeadlineExceeded, check_deadline, check_response_deadline,
)

# Payment-required responses mean this candidate cannot serve the request with
# its current credentials. Explicit auto routes may safely try the next
Expand Down Expand Up @@ -161,6 +164,7 @@ def dispatch_auto_route(
Each attempt's outcome and time to response feed route health, and a route set to
health ordering tries its candidates in the order RouteHealth.order returns.
"""
check_deadline()
route = AutoRouteService.get_route(payload.get("model"))
if route is None:
raise APIError(
Expand All @@ -177,10 +181,12 @@ def dispatch_auto_route(
last_priority = 0
last_decision = "auto-failover"
for position, candidate in enumerate(order.candidates):
check_deadline()
priority = priorities[candidate]
try:
validate_candidate(candidate)
except (ModelCooldownExhausted, ModelCooldownCapacity):
check_deadline()
except (ModelCooldownExhausted, ModelCooldownCapacity, GenerationDeadlineExceeded):
raise
except (APIError, ValueError) as error:
logger.info(
Expand All @@ -202,6 +208,7 @@ def dispatch_auto_route(
candidate,
route_decision,
)
check_response_deadline(response)
except AutoRouteCandidateUnavailable as error:
logger.info(
"Skipping auto route candidate %s before generation (%s)",
Expand All @@ -226,6 +233,7 @@ def dispatch_auto_route(
):
preflight = preflight_chat_stream(response)
response = preflight.response
check_response_deadline(response)
preflight_failed = preflight.outcome not in {"skipped", "validated"}
preflight_outcome = preflight.outcome
ok, reason = attempt_outcome(response)
Expand Down
Loading
Loading