Repository navigation
feat(rate-limit): add opt-in gateway rate-limit headers - #28
Conversation
Expose admission-time gateway counters as X-MultiLLM-RateLimit-* headers and a bounded usage snapshot behind RATE_LIMIT_HEADERS_ENABLED, default off. Provider headers and raw routes are untouched.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
multillm-proxy | 0d886bb | Oct 08 2026, 10:20 PM |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: afcf1e87d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if response.status_code == 429 and _integer(retry_after, 1) and "Retry-After" not in response.headers: | ||
| response.headers["Retry-After"] = str(retry_after) |
There was a problem hiding this comment.
Bind Retry-After to the returned denial
When an internal admission denial is handled and the request continues, this condition can attach its stale retry delay to an unrelated 429. For example, routes/optimized.py deliberately continues after summary_limit is denied when require_target is false; if the subsequent provider response is a 429 without its own Retry-After, g.gateway_rate_limit_retry_after from the rejected summary attempt is emitted here as though the gateway rejected the final request. Track whether the denial response itself is being returned, or clear the marker when a denial is handled.
Useful? React with 👍 / 👎.
| if not path.exists(): | ||
| return {} | ||
| try: | ||
| with closing(sqlite3.connect(path.as_uri() + "?mode=ro", uri=True)) as connection: |
There was a problem hiding this comment.
Read rate-limit usage from the configured database
When CONTROL_PLANE_DATABASE_URL is configured, rate-limit admissions use PostgreSQL because RateLimitService._connect() delegates to services/sqlite_store.connect, but this snapshot bypasses that connection and opens the local SQLite path directly. Consequently /v1/usage reports an empty rate_limits map when that file is absent, or stale counters if an old file remains, rather than the quotas actually being enforced in PostgreSQL. Use the configured store for this read or explicitly suppress snapshots for that backend.
Useful? React with 👍 / 👎.
| if has_request_context() and not hasattr(g, "rate_limit_advice"): | ||
| g.rate_limit_advice = dict(snapshot) |
There was a problem hiding this comment.
Replace the snapshot when a later admission check rejects
For the two-phase optimized route, reserve_request_slot() captures an RPM snapshot before the transformed input size is known, while finalize_request_slot() can subsequently reject the request on TPM. Because this guard permanently retains the first snapshot, that gateway-generated token_rate_limit_exceeded 429 is decorated with the earlier RPM limit and potentially nonzero RPM remaining instead of the binding token counter calculated in the denial metadata. Preserve the first snapshot for admitted responses, but replace it when the later check is the rejection actually returned to the client.
Useful? React with 👍 / 👎.
Native Worker routes enforce no gateway request counters, so they have nothing to advertise; forwarded responses already carry the Container's headers.
1c441c3
into
integrate/gateway-waves-1-2
Summary
Adds gateway rate-limit advisory headers behind
RATE_LIMIT_HEADERS_ENABLED, which defaults to false. While the flag is off, response headers, 429Retry-Aftervalues andGET /v1/usageoutput are unchanged.X-MultiLLM-RateLimit-Limit,-Remainingand-Resetfor known gateway counters.Retry-After.RateLimit-*,X-RateLimit-*andRetry-Afterheaders are preserved, and raw provider routes get no advisory headers.GET /v1/usageand the admin usage view add a boundedrate_limitssnapshot.Details are in
docs/rate-limit-headers.md.Integration
The Flask middleware (
middleware/rate_limit_headers.py) is mounted through the Wave 3 Flask registrar in the Wave 3 integration PR. Until then, this branch only changes the admission snapshot and the usage JSON.Shared registrations
X-MultiLLM-RateLimit-*headers to the Flask and Worker CORS expose lists.RATE_LIMIT_HEADERS_ENABLEDto the Container env keys.Tests
tests/test_rate_limit_headers.py: new.npm run test:worker: 1,234 passed (before the unused Worker module and its test were removed).cf:migrations:check,test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.This is part of Wave 3. It is based on the Wave 2 integration branch (#25), and the Wave 3 integration PR adds the README line.