Repository navigation
feat(gateway): add opt-in end-to-end generation deadlines - #41
Conversation
Carry one monotonic budget from X-MultiLLM-Deadline-Ms through setup, admission, retry waits, provider calls, preflight and stream reads. Expiry returns 504 before commitment or a protocol error event after it, cancels the upstream once and settles usage as unknown. Requests without the header are unchanged.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
multillm-proxy | 631c5cb | Oct 09 2026, 12:23 AM |
7f21af1
into
integrate/gateway-waves-1-3
Summary
Adds end-to-end generation deadlines. A caller opts in per request with
X-MultiLLM-Deadline-Ms, an integer from 1 toGENERATION_DEADLINE_MAX_MS(default 300,000). Without the header, existing timeouts, retries, response bytes and stream behaviour are unchanged.generation_deadline_exceeded.[DONE], no fallback;invalid_generation_deadline. A malformed maximum disables deadlines and logs one warning, without the value.X-MultiLLM-Internal-Deadline-Ms, and strip caller-supplied internal headers. The integration PR wires them into Worker-to-Container forwarding.Details are in
docs/generation-deadline.md.Integration
The deadline helpers are no-ops until a deadline is set. The Wave 4 integration PR does the following:
register_generation_deadlinein the Flask hook order, after identity and before admission;generationDeadlineHookto the native Worker lifecycle;Until then, the header has no effect.
services/proxy_service.py(3,580 lines) only gains small call sites.Shared registrations
tests/test_generation_deadline_worker.mjstotest:worker.GENERATION_DEADLINE_MAX_MSto the Container env keys.worker/generation-deadline.mjsto the secret firewall inventory.X-MultiLLM-Deadline-Msto the browser CORS allowed request headers in Flask and the Worker.Tests
tests/test_generation_deadline.pyandtests/test_generation_deadline_worker.mjs: new. They cover:npm run test:worker: 1,365 passed.cf:migrations:check,test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.pythonandnodejobs currently fail at the dependency-audit step on every branch, including main, because of new advisories. The tests themselves don't run there. A separate dependency PR fixes this.This is part of Wave 4. It is based on
integrate/gateway-waves-1-3, and the Wave 4 integration PR adds the README line.