Repository navigation
feat(routing): add opt-in bounded hedged auto requests - #65
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
multillm-proxy | 1ef4abf | Oct 09 2026, 05:21 AM |
3c9ba9f
into
integrate/gateway-waves-1-5
Summary
Adds opt-in bounded hedging for managed auto routes. When the first candidate hasn't returned a useful, complete response after a short delay, the gateway starts the second candidate, and the first valid response wins. There are never more than two attempts. With
HEDGED_REQUESTS_ENABLEDunset oroff(the default), auto-route dispatch, headers, accounting and storage are byte-for-byte unchanged:route_policyreturnsNonebefore any other work, and both dispatch hooks do nothing.HEDGED_REQUESTS_ENABLED: empty means off; a malformed value is off with one warning that leaves the value out.HEDGED_REQUESTS_POLICY_JSON(default{}) maps an auto route ID to{"enabled", "delay_ms", "max_duplicates", "idempotent_safe"}.delay_msis 1–1000 (default 150),max_duplicatesmust be exactly 2 (the original plus one), and the flags must be booleans. Any invalid entry disables the whole map with one warning.idempotent_safe: true;POST /v1/chat/completions;Idempotency-Keyheld by managed idempotency;max_tokens/max_completion_tokensandnof 1;tools,tool_choice,functionsorfunction_call, no tool repair and no context paging.delay_msfrom the first submission, only while the shared generation deadline has room, and only if a second admission lease is available at once. Admission is tried once, with no queue or retry.Details are in
docs/hedged-requests.md.Behaviour to know
unpriced_reservation, as the brief requires ("unknown price under cap rejects"). Configure prices for both candidates before enabling a route. If both holds don't fit the caller's caps, the request runs single-call under the normal budget policy.nof 1, and excludes tool repair and context paging. These go beyond the brief.Wiring
routes/auto_routes.py:dispatch_auto_routemakes one call todispatch_hedged_autofor chat auto routes, before the existing candidate loop.Nonemeans "use the loop".services/managed_dispatch.py:execute_managed_attemptreports each attempt's submission and response to its hedge attempt;retry_managed_attemptgrants no transport retry inside a race;isolated_managed_attemptkeeps each attempt's managed-turn finalizers out of the parent turn.Shared registrations
HEDGED_REQUESTS_ENABLEDandHEDGED_REQUESTS_POLICY_JSONin the Container env keys. No Worker modules or tests, D1 tables, routes or browser-readable headers.Tests
tests/test_hedged_requests.pycovers:npm run test:worker: 1,637 passed.npm run test:integration: 7 passed.migrations:check(28 contiguous, additive migrations),test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.CI notes
python/nodejobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.AdmissionCoordinatorDurable Object migration, whichwrangler versions uploadcannot apply. This is expected untilwrangler deployruns on main.This is part of Wave 6. It is based on
integrate/gateway-waves-1-5, the Wave 5 integration branch (#60). The README line comes with the Wave 6 integration PR.