Skip to content

feat(gateway): add opt-in end-to-end generation deadlines - #41

Merged
ALikesToCode merged 1 commit into
integrate/gateway-waves-1-3from
feat/generation-deadlines
Oct 9, 2026
Merged

ALikesToCode merged 1 commit into
integrate/gateway-waves-1-3from
feat/generation-deadlines

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds end-to-end generation deadlines. A caller opts in per request with X-MultiLLM-Deadline-Ms, an integer from 1 to GENERATION_DEADLINE_MAX_MS (default 300,000). Without the header, existing timeouts, retries, response bytes and stream behaviour are unchanged.

  • One monotonic budget covers the whole generation:
    • setup, token counting and admission;
    • allowed retry waits;
    • provider calls;
    • stream preflight and body reads.
  • A shorter key or route limit takes precedence.
  • Expiry before the response is committed returns 504 generation_deadline_exceeded.
  • Expiry after an SSE stream is committed emits a real error event in the selected protocol (Chat, Responses or Messages) and cancels the upstream:
    • no stop event, no [DONE], no fallback;
    • binary streams end with a transport error.
  • A malformed header returns 400 invalid_generation_deadline. A malformed maximum disables deadlines and logs one warning, without the value.
  • The forwarding helpers send only the remaining budget, in the authenticated internal header X-MultiLLM-Internal-Deadline-Ms, and strip caller-supplied internal headers. The integration PR wires them into Worker-to-Container forwarding.
  • An expired deadline's outcome is unknown usage, never zero.

Details are in docs/generation-deadline.md.

Integration

The deadline helpers are no-ops until a deadline is set. The Wave 4 integration PR does the following:

  • registers register_generation_deadline in the Flask hook order, after identity and before admission;
  • adds generationDeadlineHook to the native Worker lifecycle;
  • passes the remaining budget through authenticated forwarding.

Until then, the header has no effect. services/proxy_service.py (3,580 lines) only gains small call sites.

Shared registrations

  • Added tests/test_generation_deadline_worker.mjs to test:worker.
  • Added GENERATION_DEADLINE_MAX_MS to the Container env keys.
  • Added worker/generation-deadline.mjs to the secret firewall inventory.
  • Added X-MultiLLM-Deadline-Ms to the browser CORS allowed request headers in Flask and the Worker.

Tests

  • tests/test_generation_deadline.py and tests/test_generation_deadline_worker.mjs: new. They cover:
    • no header leaving everything unchanged;
    • header bounds;
    • 504 before commit;
    • protocol error events after commit for Chat, Responses and Messages;
    • cancelling the upstream exactly once;
    • retry waits consuming the budget;
    • bounded preflight;
    • stripping of internal headers;
    • unknown-usage settlement.
  • Full Python suite: 3,574 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,365 passed.
  • cf:migrations:check, test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.
  • GitHub's python and node jobs currently fail at the dependency-audit step on every branch, including main, because of new advisories. The tests themselves don't run there. A separate dependency PR fixes this.

This is part of Wave 4. It is based on integrate/gateway-waves-1-3, and the Wave 4 integration PR adds the README line.

Carry one monotonic budget from X-MultiLLM-Deadline-Ms through setup, admission, retry waits, provider calls, preflight and stream reads. Expiry returns 504 before commitment or a protocol error event after it, cancels the upstream once and settles usage as unknown. Requests without the header are unchanged.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: beb8055e-1840-4bc1-9251-cf2e5fdcc3be

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy 631c5cb Oct 09 2026, 12:23 AM

@ALikesToCode
ALikesToCode merged commit 7f21af1 into integrate/gateway-waves-1-3 Oct 9, 2026
4 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the feat/generation-deadlines branch October 26, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant