Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions docs/rate-limit-headers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Rate-limit advisory headers

`RATE_LIMIT_HEADERS_ENABLED` defaults to false. Missing, empty, `false`, `0`,
`no` and `off` leave response headers and usage JSON unchanged. Enable with
`true`, `1`, `yes` or `on`. Values are case insensitive and trimmed. Malformed
values disable advice and log one warning per process without the value.

Managed, authenticated responses with a gateway admission snapshot can expose:

- `X-MultiLLM-RateLimit-Limit`: the selected gateway counter's positive limit.
- `X-MultiLLM-RateLimit-Remaining`: capacity at admission, after reserving the
current request. A known exhausted counter has remaining `0`.
- `X-MultiLLM-RateLimit-Reset`: positive integer seconds until capacity expires,
captured at admission rather than recalculated when the response finishes.

Unknown, unlimited, invalid and unavailable fields are omitted. No header contains
principal names, key prefixes, client addresses, provider credentials or provider
quota information. Provider `RateLimit-*`, `X-RateLimit-*` and `Retry-After`
headers are preserved. Raw provider paths receive no advisory headers. Streaming
bodies are forwarded unchanged; advice does not read or buffer them.

With advice disabled, a successful managed response has no new headers. With
advice enabled, a request admitted against a two-request minute limit can return:

```http
HTTP/1.1 200 OK
X-MultiLLM-RateLimit-Limit: 2
X-MultiLLM-RateLimit-Remaining: 1
X-MultiLLM-RateLimit-Reset: 60
```

An exhausted minute counter, observed 17 seconds after the first admission:

```http
HTTP/1.1 429 Too Many Requests
X-MultiLLM-RateLimit-Limit: 2
X-MultiLLM-RateLimit-Remaining: 0
X-MultiLLM-RateLimit-Reset: 43
Retry-After: 43
```

The ordinary counter is requests per minute. On rejection, the selected counter
follows the existing enforcement order: RPM, TPM, then daily requests. The JSON
error identifies the rejected counter. Daily reset advice uses the rolling
24-hour window, not midnight. A token denial conservatively uses the last token
usage expiry; advice is not a guarantee that the next request will fit. Existing
replay rules remain authoritative: neither these headers nor a 429 authorizes a
retry of a request that may have produced output. Upstream errors and bodies are
not replaced. Only a known gateway rejection fills a missing `Retry-After`.

SQLite snapshots are read under the admission transaction. Request-local copies
survive preprocessing, finalization and subsequent admissions unchanged. The D1
ledger supplies its current gateway counts, including cached shared figures;
precise expiry is unavailable and `Reset` is omitted. D1 consistency remains
bounded by its existing synchronization behavior; this feature does not make it
a globally atomic limiter. When exact expiry is unavailable on a gateway denial,
`Retry-After` uses the applicable full rolling window as conservative advice.

When enabled, `GET /v1/usage` adds `rate_limits`, a mapping of at most 32 observed
providers to bounded counter snapshots for the authenticated caller. Selected
principal responses from `/usage/data` include the same shape, respecting existing
administrator authorization. Usage reads are current snapshots, not admissions:
no request is reserved or charged. SQLite reads use a read-only transaction;
missing storage or tables returns an empty mapping. D1 usage snapshots are empty
because its existing public interface lacks a bounded provider inventory; there
is no additional D1 synchronization or remote request. These are gateway limits,
not monetary budgets or provider free quotas.

No new table, durable state, remote call or content retention is introduced.
Copies live only for the current request. Enabled SQLite requests add one bounded
expiry query under the existing lock; usage reads have a 32-provider bound.
Disabled traffic does not query expiry or usage storage. This feature adds no
provider token cost and does not change admission, reservation or retry policy.

## Where the headers apply

Flask adds the headers to managed `/v1/` responses; raw provider routes are
excluded. Counters are never read from request headers or payloads. Requests the
Worker forwards to the Container keep the Container's headers unchanged. Native
Worker routes (codex-easy, linkapi, opencode direct) enforce no gateway request
counters, so their responses carry no advisory headers. Set
`RATE_LIMIT_HEADERS_ENABLED` on the Container.
206 changes: 206 additions & 0 deletions middleware/rate_limit_headers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
"""Gateway-only admission advice; never infer provider quotas or replay permission."""

import logging
import math
import os
import sqlite3
from contextlib import closing
from datetime import datetime, timedelta, timezone
from typing import Mapping

from flask import g, has_request_context, request

logger = logging.getLogger(__name__)
HEADER_PREFIX = "X-MultiLLM-RateLimit-"
MAX_COUNTER = 2**53 - 1
MAX_USAGE_PROVIDERS = 32
_warned_invalid = False


def enabled() -> bool:
global _warned_invalid
value = os.environ.get("RATE_LIMIT_HEADERS_ENABLED", "").strip().lower()
if value in {"1", "true", "yes", "on"}:
return True
if value not in {"", "0", "false", "no", "off"} and not _warned_invalid:
_warned_invalid = True
logger.warning("Invalid RATE_LIMIT_HEADERS_ENABLED; advisory headers disabled")
return False


def _integer(value, minimum=0):
return type(value) is int and minimum <= value <= MAX_COUNTER


def bounded_snapshot(snapshot) -> dict:
if not isinstance(snapshot, Mapping) or not _integer(snapshot.get("limit"), 1):
return {}
result = {"limit": snapshot["limit"]}
remaining = snapshot.get("remaining")
if _integer(remaining) and remaining <= result["limit"]:
result["remaining"] = remaining
reset = snapshot.get("reset")
if _integer(reset, 1) and reset <= 86400:
result["reset"] = reset
return result


def snapshot_headers(snapshot) -> dict:
return {HEADER_PREFIX + name.title(): str(value)
for name, value in bounded_snapshot(snapshot).items()}


def _reset_seconds(connection, metadata, window, now, *, offset=0, latest=False):
"""Read expiry from the same SQLite transaction as the admission counters."""
query = (
"SELECT created_at FROM request_usage "
"WHERE identity = ? AND provider = ? AND created_at >= ? AND id != ? "
"ORDER BY created_at DESC LIMIT 1 OFFSET ?"
) if latest else (
"SELECT created_at FROM request_usage "
"WHERE identity = ? AND provider = ? AND created_at >= ? AND id != ? "
"ORDER BY created_at ASC LIMIT 1 OFFSET ?"
)
row = connection.execute(query, (
metadata["identity"], metadata["provider"], (now - timedelta(seconds=window)).isoformat(),
metadata.get("reservation_id", -1), offset,
)).fetchone()
if row is None:
return None
admitted = datetime.fromisoformat(row["created_at"])
return min(window, max(1, math.ceil((admitted + timedelta(seconds=window) - now).total_seconds())))


def _counter_snapshot(metadata, minute_count, minute_tokens, daily_count, token_limited, *, consume=False):
rpm, daily = metadata["rpm_limit"], metadata["daily_request_limit"]
denied = minute_count >= rpm or token_limited or daily_count >= daily
limit, count, window, kind = rpm, minute_count, 60, "requests"
if minute_count < rpm and token_limited:
limit, count, kind = metadata["tpm_limit"], minute_tokens, "tokens"
elif minute_count < rpm and not token_limited and daily_count >= daily:
limit, count, window = daily, daily_count, 86400
remaining = max(0, limit - count - int(consume and not denied))
return bounded_snapshot({"limit": limit, "remaining": remaining}), count, window, kind, denied


def capture_admission(metadata, minute_count, minute_tokens, daily_count,
token_limited, now, *, connection=None):
"""Small service hook, called under its existing transaction before check-and-admit.

The first snapshot stays request-local even if transformation or fallback later
rechecks capacity. D1 counts are known to its ledger; precise expiry is not.
"""
if not enabled():
return None
snapshot, count, window, kind, denied = _counter_snapshot(
metadata, minute_count, minute_tokens, daily_count, token_limited, consume=True)
reset = None
if snapshot and connection is not None:
offset = max(0, count - snapshot["limit"]) if denied and kind != "tokens" else 0
reset = _reset_seconds(connection, metadata, window, now, offset=offset,
latest=denied and kind == "tokens")
if reset is None and not denied:
reset = window # The about-to-be-admitted request starts this window.
if reset is not None:
snapshot["reset"] = reset
if snapshot:
metadata["rate_limit_advice"] = dict(snapshot)
if has_request_context() and not hasattr(g, "rate_limit_advice"):
g.rate_limit_advice = dict(snapshot)
Comment on lines +108 to +109

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Replace the snapshot when a later admission check rejects

For the two-phase optimized route, reserve_request_slot() captures an RPM snapshot before the transformed input size is known, while finalize_request_slot() can subsequently reject the request on TPM. Because this guard permanently retains the first snapshot, that gateway-generated token_rate_limit_exceeded 429 is decorated with the earlier RPM limit and potentially nonzero RPM remaining instead of the binding token counter calculated in the denial metadata. Preserve the first snapshot for admitted responses, but replace it when the later check is the rejection actually returned to the client.

Useful? React with 👍 / 👎.

retry_after = (reset or window) if denied else None
if denied and has_request_context():
g.gateway_rate_limit_retry_after = retry_after
return retry_after


def apply_rate_limit_headers(response, *, managed=True):
if not enabled() or not managed or not getattr(g, "authenticated_user", None):
return response
snapshot = getattr(g, "rate_limit_advice", None)
for name, value in snapshot_headers(snapshot).items():
response.headers[name] = value
retry_after = getattr(g, "gateway_rate_limit_retry_after", None)
if response.status_code == 429 and _integer(retry_after, 1) and "Retry-After" not in response.headers:
response.headers["Retry-After"] = str(retry_after)
Comment on lines +123 to +124

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bind Retry-After to the returned denial

When an internal admission denial is handled and the request continues, this condition can attach its stale retry delay to an unrelated 429. For example, routes/optimized.py deliberately continues after summary_limit is denied when require_target is false; if the subsequent provider response is a 429 without its own Retry-After, g.gateway_rate_limit_retry_after from the rejected summary attempt is emitted here as though the gateway rejected the final request. Track whether the denial response itself is being returned, or clear the marker when a denial is handled.

Useful? React with 👍 / 👎.

return response


def register_rate_limit_headers(app, *, is_managed=None):
"""W22 registrar hook; the coordinator may inject its managed-route predicate."""
if is_managed is None:
is_managed = lambda: request.path.startswith("/v1/")

@app.after_request
def decorate(response):
if not enabled():
return response
return apply_rate_limit_headers(response, managed=is_managed())


def _utcnow():
return datetime.now(timezone.utc)


def usage_snapshot(user, remote_addr=None, *, service=None, now=None) -> dict:
"""Read at most 32 observed provider counters for one authorized principal.

D1 has no bounded public provider/expiry inventory; omit unavailable fields
rather than read internal ledger state or start a new synchronization.
"""
if not enabled() or os.environ.get("RATE_LIMIT_ENABLED", "true").lower() in {"0", "false", "no"}:
return {}
if service is None:
from services.rate_limit_service import RateLimitService
service = RateLimitService
from services import rate_limit_d1
if rate_limit_d1.active():
return {}
now = now or _utcnow()
identity, _ = service._identity_for_user(user, remote_addr)
path = service._get_storage_path().resolve()
if not path.exists():
return {}
try:
with closing(sqlite3.connect(path.as_uri() + "?mode=ro", uri=True)) as connection:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Read rate-limit usage from the configured database

When CONTROL_PLANE_DATABASE_URL is configured, rate-limit admissions use PostgreSQL because RateLimitService._connect() delegates to services/sqlite_store.connect, but this snapshot bypasses that connection and opens the local SQLite path directly. Consequently /v1/usage reports an empty rate_limits map when that file is absent, or stale counters if an old file remains, rather than the quotas actually being enforced in PostgreSQL. Use the configured store for this read or explicitly suppress snapshots for that backend.

Useful? React with 👍 / 👎.

connection.row_factory = sqlite3.Row
connection.execute("BEGIN")
providers = connection.execute(
"SELECT DISTINCT provider FROM request_usage WHERE identity = ? AND created_at >= ? "
"ORDER BY provider LIMIT ?", (identity, (now - timedelta(days=1)).isoformat(), MAX_USAGE_PROVIDERS),
).fetchall()
result = {}
for row in providers:
snapshot = _usage_provider_snapshot(connection, service, identity, row["provider"], now)
if snapshot:
result[row["provider"]] = snapshot
return result
except (sqlite3.Error, ValueError):
logger.warning("Gateway rate-limit usage snapshot unavailable")
return {}


def _usage_provider_snapshot(connection, service, identity, provider, now):
counts = connection.execute(
"SELECT SUM(CASE WHEN created_at >= ? THEN 1 ELSE 0 END) AS requests, "
"SUM(CASE WHEN created_at >= ? THEN estimated_tokens ELSE 0 END) AS tokens, "
"COUNT(*) AS daily FROM request_usage WHERE identity = ? AND provider = ? AND created_at >= ?",
((now - timedelta(seconds=60)).isoformat(), (now - timedelta(seconds=60)).isoformat(),
identity, provider, (now - timedelta(days=1)).isoformat()),
).fetchone()
metadata = {
"identity": identity, "provider": provider,
"rpm_limit": service._provider_limit(provider, "RATE_LIMIT_RPM", service.RATE_LIMITS.get(
provider, service.RATE_LIMITS["default"])["requests"]),
"tpm_limit": service._provider_limit(provider, "RATE_LIMIT_TPM", 200000),
"daily_request_limit": service._provider_limit(provider, "DAILY_REQUEST_LIMIT", 10000),
}
tokens = counts["tokens"] or 0
snapshot, count, window, kind, denied = _counter_snapshot(
metadata, counts["requests"] or 0, tokens, counts["daily"], tokens >= metadata["tpm_limit"])
if snapshot:
reset = _reset_seconds(connection, metadata, window, now,
offset=max(0, count - snapshot["limit"]) if denied and kind != "tokens" else 0,
latest=denied and kind == "tokens")
if reset is not None:
snapshot["reset"] = reset
return snapshot
2 changes: 1 addition & 1 deletion route_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
+ ", ".join([*CLIENT_HEADER_NAMES.values(), *OPENCODE_CLIENT_HEADER_NAMES.values()])
)
CORS_EXPOSE_HEADERS = (
"Retry-After, X-Request-ID, X-MultiLLM-Optimization, X-MultiLLM-Tool-Repair, X-MultiLLM-Image-QA, X-MultiLLM-Cascade, X-MultiLLM-Conversion-Fidelity, X-MultiLLM-Conversion-Fields, "
"Retry-After, X-Request-ID, X-MultiLLM-RateLimit-Limit, X-MultiLLM-RateLimit-Remaining, X-MultiLLM-RateLimit-Reset, X-MultiLLM-Optimization, X-MultiLLM-Tool-Repair, X-MultiLLM-Image-QA, X-MultiLLM-Cascade, X-MultiLLM-Conversion-Fidelity, X-MultiLLM-Conversion-Fields, "
"X-MultiLLM-Optimization-Mode, X-MultiLLM-Estimated-Input-Before, "
"X-MultiLLM-Estimated-Input-After, X-MultiLLM-Image-Prompts-Compacted, "
"X-MultiLLM-Messages-Summarized, X-MultiLLM-Optimization-Target-Met, "
Expand Down
5 changes: 5 additions & 0 deletions routes/usage.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from flask import g, jsonify, render_template, request

from error_handlers import APIError
from middleware.rate_limit_headers import enabled as rate_limit_headers_enabled, usage_snapshot
from request_validation import json_object_body
from route_helpers import api_authenticate_only, login_required
from services import key_controls, usage_ledger, usage_store
Expand Down Expand Up @@ -73,6 +74,8 @@ def api_usage():
principal = str(user.get("username") or user.get("id"))
controls = key_controls.public(user)
return jsonify({
**({"rate_limits": usage_snapshot(user, request.remote_addr)}
if rate_limit_headers_enabled() else {}),
"object": "usage",
"principal": principal,
"key_prefix": user.get("api_key_prefix"),
Expand Down Expand Up @@ -105,6 +108,8 @@ def usage_data():
current_user.get("username"))
payload["budget"] = BudgetService.status(record or {"username": principal})
payload["controls"] = key_controls.public(record or {})
if rate_limit_headers_enabled():
payload["rate_limits"] = usage_snapshot(record or {"username": principal}, request.remote_addr)
if is_admin:
payload["ledger"] = usage_ledger.LEDGER.stats()
return jsonify(payload)
Expand Down
Loading
Loading