Skip to content

feat(routing): add opt-in bounded hedged auto requests - #65

Merged
ALikesToCode merged 1 commit into
integrate/gateway-waves-1-5from
feat/hedged-requests
Oct 9, 2026
Merged

ALikesToCode merged 1 commit into
integrate/gateway-waves-1-5from
feat/hedged-requests

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds opt-in bounded hedging for managed auto routes. When the first candidate hasn't returned a useful, complete response after a short delay, the gateway starts the second candidate, and the first valid response wins. There are never more than two attempts. With HEDGED_REQUESTS_ENABLED unset or off (the default), auto-route dispatch, headers, accounting and storage are byte-for-byte unchanged: route_policy returns None before any other work, and both dispatch hooks do nothing.

  • Settings:
    • HEDGED_REQUESTS_ENABLED: empty means off; a malformed value is off with one warning that leaves the value out.
    • HEDGED_REQUESTS_POLICY_JSON (default {}) maps an auto route ID to {"enabled", "delay_ms", "max_duplicates", "idempotent_safe"}. delay_ms is 1–1000 (default 150), max_duplicates must be exactly 2 (the original plus one), and the flags must be booleans. Any invalid entry disables the whole map with one warning.
    • No auto-route storage schema change and no migration.
  • Eligible requests (anything else runs the existing single-call loop unchanged):
    • the route policy is enabled with idempotent_safe: true;
    • a non-streaming POST /v1/chat/completions;
    • an Idempotency-Key held by managed idempotency;
    • usage reservations are on and the caller has budget caps;
    • a positive integer max_tokens/max_completion_tokens and n of 1;
    • no tools, tool_choice, functions or function_call, no tool repair and no context paging.
  • Race:
    • Both conservative costs are reserved before the first submission.
    • The second attempt starts only after delay_ms from the first submission, only while the shared generation deadline has room, and only if a second admission lease is available at once. Admission is tried once, with no queue or retry.
    • The winner must be an HTTP 200, complete JSON envelope with useful content in every choice and no tool calls (validation buffers at most 1 MiB).
    • The loser is cancelled through the existing request-cancellation transport, once.
    • A race uses at most two request-local threads, which are joined deterministically.
    • After entering the race there is no third candidate and no new retry.
  • Cost:
    • Every dispatched attempt has its own reservation and accounting row, and measured usage settles both, including the loser.
    • An unknown outcome keeps its hold; cancellation is never treated as proof that the provider didn't bill.
    • A duplicate that never dispatched releases its hold.
  • No new headers, logs, routes or migrations. The existing auto-route headers report the selected model and attempt count.

Details are in docs/hedged-requests.md.

Behaviour to know

  • Unknown prices: an eligible request on an opted-in route whose candidate has no known price returns 503 unpriced_reservation, as the brief requires ("unknown price under cap rejects"). Configure prices for both candidates before enabling a route. If both holds don't fit the caller's caps, the request runs single-call under the normal budget policy.
  • Failover: inside a race, only the first two candidates are used. A definite pre-generation failure of both returns that failure instead of moving on to a third candidate.
  • Builder additions: to bound cost, the builder also requires an explicit output limit and n of 1, and excludes tool repair and context paging. These go beyond the brief.

Wiring

  • routes/auto_routes.py:dispatch_auto_route makes one call to dispatch_hedged_auto for chat auto routes, before the existing candidate loop. None means "use the loop".
  • services/managed_dispatch.py:
    • execute_managed_attempt reports each attempt's submission and response to its hedge attempt;
    • retry_managed_attempt grants no transport retry inside a race;
    • isolated_managed_attempt keeps each attempt's managed-turn finalizers out of the parent turn.

Shared registrations

HEDGED_REQUESTS_ENABLED and HEDGED_REQUESTS_POLICY_JSON in the Container env keys. No Worker modules or tests, D1 tables, routes or browser-readable headers.

Tests

  • tests/test_hedged_requests.py covers:
    • flag off and every ineligible request staying single-call;
    • strict policy validation warning once without the value;
    • unknown-price rejection only for an otherwise eligible, opted-in route;
    • both holds reserved before dispatch, and a fast primary releasing the unused hold;
    • the delay, deadline and admission bounds, including a denied second lease;
    • the first useful response winning and the loser being cancelled and closed once;
    • measured settlement for both attempts, and an unknown loser keeping its hold;
    • no third candidate and no retry after an ambiguous race;
    • caller cancellation stopping both attempts, and thread cleanup.
  • The auto-route, managed, idempotency and cancellation suites (204 tests) pass with the new call sites.
  • Full Python suite: 4,514 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,637 passed. npm run test:integration: 7 passed.
  • migrations:check (28 contiguous, additive migrations), test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 6. It is based on integrate/gateway-waves-1-5, the Wave 5 integration branch (#60). The README line comes with the Wave 6 integration PR.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4fb59eee-cb0b-4ae1-ac54-41b59f069d4b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy 1ef4abf Oct 09 2026, 05:21 AM

@ALikesToCode
ALikesToCode merged commit 3c9ba9f into integrate/gateway-waves-1-5 Oct 9, 2026
4 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the feat/hedged-requests branch October 26, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant