Skip to content

feat(routing): add opt-in latency SLO admission - #61

Merged
ALikesToCode merged 1 commit into
integrate/gateway-waves-1-5from
feat/latency-slo
Oct 9, 2026
Merged

ALikesToCode merged 1 commit into
integrate/gateway-waves-1-5from
feat/latency-slo

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds opt-in latency-SLO admission. It predicts whether a request will finish within its deadline and can reject it, or reroute an automatic selection, before any provider handoff. With LATENCY_SLO_MODE unset or off (the default), admission, routing, responses and headers are byte-for-byte unchanged, and no observations are recorded.

  • Settings:
    • LATENCY_SLO_MODE is off, reject or reroute. Empty means off; a malformed value is off with one warning that leaves the value out.
    • LATENCY_SLO_MIN_SAMPLES (default 20, 1–1000) and LATENCY_SLO_MAX_MODELS (default 128, 1–512).
    • LATENCY_SLO_POLICY_JSON (default {}) is validated strictly. It names exact paths or auto model IDs under routes, and authenticated key IDs under keys, each with deadline_ms (1–300000) and an optional require_coverage. When several entries match, the smallest deadline and any coverage requirement win. A request that matches no entry uses ordinary policy.
  • Prediction:
    • It uses only measurements the gateway already records: time to first output and output throughput per model, from the last 15 minutes.
    • Storage is capped at 1,000 observations per model and the configured model count. Observations are process-local and hold only model IDs and numbers.
    • The prediction is the nearest-rank p95 of ttft_ms + output_tokens * 1000 / tokens_per_second. Output tokens come from max_tokens/max_completion_tokens, or 1024 by default.
    • Each decision reports sample count, coverage, observation age and window.
  • Admission:
    • Sparse or old data gives prediction_unknown. That uses ordinary policy, unless the matching entry requires coverage; then the request gets 503 latency_slo_unavailable.
    • A known miss in reject mode returns 503 latency_slo_predicted_miss before any provider call. Both errors are retryable: false, carry Cache-Control: no-store, and include the content-free prediction report.
    • reroute applies only to automatic selections. It keeps already eligible alternatives with the same reviewed quality tier and a known prediction within the deadline. It never adds models or retries.
    • An explicit model is never changed. It gets the reject behaviour on a known miss.
  • Deadline: the generation deadline stays the hard runtime limit. Admission uses its remaining budget and never extends or resets it.
  • No probes, policy writes, tables or migrations.

Details are in docs/latency-slo.md.

Wiring

  • services/gateway_extensions.py registers latency_slo_request_hook by name, right after the generation deadline hook and before the idempotency claim. It also adds a read-only latency_slo_candidate_policy collaborator for auto routes, which returns None until it is supplied.
  • services/intelligence_policy.py: select_candidates applies the policy to its final candidate list after session-tier selection. With the mode off it returns the same list.
  • services/route_health.py records observations (only when enabled) and passes automatic route ordering through order_auto_candidates. That returns the order unchanged unless the request has an active SLO decision.
  • worker/latency-slo.mjs provides the same settings and prediction for native Worker routes (prepareLatencySLO, recordLatencyObservation). It is wired into the native generation path with the Wave 6 integration.

Shared registrations

  • tests/test_latency_slo_worker.mjs in test:worker.
  • worker/latency-slo.mjs in the edge module inventory.
  • LATENCY_SLO_MODE, LATENCY_SLO_MIN_SAMPLES, LATENCY_SLO_MAX_MODELS and LATENCY_SLO_POLICY_JSON in the Container env keys.
  • tests/test_wave4_registrar.py and tests/test_wave5_registrar.py now expect latency_slo_request_hook between the deadline and idempotency hooks.

No D1 tables, Flask or Worker routes, or browser-readable headers.

Tests

  • tests/test_latency_slo.py and tests/test_latency_slo_worker.mjs cover:
    • flag off, byte for byte, with no observations stored;
    • malformed settings warning once without the value;
    • strict policy validation, including duplicate keys and size limits;
    • window bounds, expiry, model eviction and p95 prediction;
    • unknown coverage passing or returning 503, as the policy says;
    • reject mode returning 503 before any provider call;
    • reroute keeping only same-tier eligible alternatives within the deadline;
    • explicit models never being changed;
    • the remaining generation deadline tightening the SLO;
    • the hook's position right after the deadline hook.
  • Full Python suite: 4,481 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,649 passed. npm run test:integration: 7 passed.
  • migrations:check (28 contiguous, additive migrations), test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 6. It is based on integrate/gateway-waves-1-5, the Wave 5 integration branch (#60). The README line comes with the Wave 6 integration PR.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8cd45f4e-43be-449c-8fd3-03916d40c8db

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy 6c67249 Oct 09 2026, 05:06 AM

@ALikesToCode
ALikesToCode merged commit 6514d0c into integrate/gateway-waves-1-5 Oct 9, 2026
4 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the feat/latency-slo branch October 26, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant