Skip to content

feat(rate-limit): add opt-in gateway rate-limit headers - #28

Merged
ALikesToCode merged 2 commits into
integrate/gateway-waves-1-2from
feat/rate-limit-headers
Oct 9, 2026
Merged

ALikesToCode merged 2 commits into
integrate/gateway-waves-1-2from
feat/rate-limit-headers

Conversation

@ALikesToCode

@ALikesToCode ALikesToCode commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

Adds gateway rate-limit advisory headers behind RATE_LIMIT_HEADERS_ENABLED, which defaults to false. While the flag is off, response headers, 429 Retry-After values and GET /v1/usage output are unchanged.

  • With the flag on, managed responses carry X-MultiLLM-RateLimit-Limit, -Remaining and -Reset for known gateway counters.
    • The values are captured at admission, so they don't drift while a stream runs.
    • Unknown or unlimited fields are omitted, never sent as 0.
  • A gateway-generated 429 carries the precise time until the counter resets as Retry-After.
  • Provider RateLimit-*, X-RateLimit-* and Retry-After headers are preserved, and raw provider routes get no advisory headers.
  • GET /v1/usage and the admin usage view add a bounded rate_limits snapshot.
  • No header contains principal names, key prefixes, client addresses or provider quota.
  • Native Worker routes enforce no gateway request counters, so their responses carry no advisory headers. Forwarded responses keep the Container's headers.

Details are in docs/rate-limit-headers.md.

Integration

The Flask middleware (middleware/rate_limit_headers.py) is mounted through the Wave 3 Flask registrar in the Wave 3 integration PR. Until then, this branch only changes the admission snapshot and the usage JSON.

Shared registrations

  • Added the three X-MultiLLM-RateLimit-* headers to the Flask and Worker CORS expose lists.
  • Added RATE_LIMIT_HEADERS_ENABLED to the Container env keys.

Tests

  • tests/test_rate_limit_headers.py: new.
  • Full Python suite: 3,209 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,234 passed (before the unused Worker module and its test were removed). cf:migrations:check, test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

This is part of Wave 3. It is based on the Wave 2 integration branch (#25), and the Wave 3 integration PR adds the README line.

Expose admission-time gateway counters as X-MultiLLM-RateLimit-* headers and a bounded usage snapshot behind RATE_LIMIT_HEADERS_ENABLED, default off. Provider headers and raw routes are untouched.
@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2375a26a-2232-4a7f-9038-2dfc850103cb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-08T22:20:48.108724Z 0d886bb New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
multillm-proxy 0d886bb Oct 08 2026, 10:20 PM

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: afcf1e87d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +123 to +124
if response.status_code == 429 and _integer(retry_after, 1) and "Retry-After" not in response.headers:
response.headers["Retry-After"] = str(retry_after)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bind Retry-After to the returned denial

When an internal admission denial is handled and the request continues, this condition can attach its stale retry delay to an unrelated 429. For example, routes/optimized.py deliberately continues after summary_limit is denied when require_target is false; if the subsequent provider response is a 429 without its own Retry-After, g.gateway_rate_limit_retry_after from the rejected summary attempt is emitted here as though the gateway rejected the final request. Track whether the denial response itself is being returned, or clear the marker when a denial is handled.

Useful? React with 👍 / 👎.

if not path.exists():
return {}
try:
with closing(sqlite3.connect(path.as_uri() + "?mode=ro", uri=True)) as connection:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Read rate-limit usage from the configured database

When CONTROL_PLANE_DATABASE_URL is configured, rate-limit admissions use PostgreSQL because RateLimitService._connect() delegates to services/sqlite_store.connect, but this snapshot bypasses that connection and opens the local SQLite path directly. Consequently /v1/usage reports an empty rate_limits map when that file is absent, or stale counters if an old file remains, rather than the quotas actually being enforced in PostgreSQL. Use the configured store for this read or explicitly suppress snapshots for that backend.

Useful? React with 👍 / 👎.

Comment on lines +108 to +109
if has_request_context() and not hasattr(g, "rate_limit_advice"):
g.rate_limit_advice = dict(snapshot)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Replace the snapshot when a later admission check rejects

For the two-phase optimized route, reserve_request_slot() captures an RPM snapshot before the transformed input size is known, while finalize_request_slot() can subsequently reject the request on TPM. Because this guard permanently retains the first snapshot, that gateway-generated token_rate_limit_exceeded 429 is decorated with the earlier RPM limit and potentially nonzero RPM remaining instead of the binding token counter calculated in the denial metadata. Preserve the first snapshot for admitted responses, but replace it when the later check is the rejection actually returned to the client.

Useful? React with 👍 / 👎.

Native Worker routes enforce no gateway request counters, so they have nothing to advertise; forwarded responses already carry the Container's headers.
@ALikesToCode
ALikesToCode merged commit 1c441c3 into integrate/gateway-waves-1-2 Oct 9, 2026
5 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the feat/rate-limit-headers branch October 26, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant