Repository navigation
feat(credentials): add opt-in model-scoped credential cooldown - #26
Conversation
MODEL_COOLDOWN_ENABLED (default off) scopes a throttled or exhausted credential to the affected model or an explicitly configured quota bucket, so other models keep using the key. Only a genuine credential-wide authentication rejection cools every model. When every eligible key is cooling, the request gets a real 429 with a bounded Retry-After. Calls that pass no model keep the existing provider-wide cooldowns, even with the flag on.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
multillm-proxy | 08a4094 | Oct 08 2026, 09:42 PM |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 08a4094e66
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| config = settings() | ||
| if config.enabled and (model or quota_bucket): |
There was a problem hiding this comment.
Pass request scope into production pool calls
With MODEL_COOLDOWN_ENABLED=true, this branch still never executes for repository production traffic because all callers retain the new arguments' None defaults: AuthService.get_api_key calls CredentialPool.select(provider), ProxyService.make_request calls record_headers without scope, and the NanoGPT proxy, unified, and intelligence paths likewise omit model from selection and recording. Consequently a production 429 continues using the legacy provider-wide cooldown rather than the advertised model-scoped state; pass the resolved upstream model or configured quota bucket through both sides of each request.
Useful? React with 👍 / 👎.
| def cooldown_error_response(error: ModelCooldownExhausted | ModelCooldownCapacity): | ||
| """Emit the cooldown status and bounded Retry-After advice.""" | ||
| response = jsonify(error.to_dict()) | ||
| response.status_code = error.status_code | ||
| response.headers["Retry-After"] = str(error.retry_after) |
There was a problem hiding this comment.
Register the cooldown error adapter
When a scoped pool raises ModelCooldownExhausted or ModelCooldownCapacity, this response builder is never registered by the application: app.py only invokes init_error_handlers, whose generic APIError handler omits Retry-After. Thus clients receive the status but lose the only bounded wait guidance and may retry immediately; register this adapter for both exception classes during application initialization.
Useful? React with 👍 / 👎.
daa32fa
into
integrate/gateway-waves-1-2
Summary
Adds model-scoped credential cooldown behind
MODEL_COOLDOWN_ENABLED, which defaults to false. While the flag is off, key selection and cooldowns are unchanged.MODEL_COOLDOWN_MAX_SECONDS(default 3600) caps any cooldown.Retry-Afterinstead of being dispatched anyway.Details are in
docs/model-cooldown.md.Integration
CredentialPoolandNanoGPTKeyPoolaccept the model, quota bucket and outcome. The request paths that call them (auth_service,intelligence_transport,proxy_service,routes/proxy.pyandroutes/provider_credentials.py) start passing the model in the Wave 3 integration PR. Until then, every path keeps the provider-wide behaviour.Shared registrations
MODEL_COOLDOWN_ENABLEDandMODEL_COOLDOWN_MAX_SECONDSto the Container env keys.Tests
tests/test_model_cooldown.py: new. It covers the flag-on calls with no model, which keep the 429 and 401 cooldowns.npm run test:worker: 1,227 passed.cf:migrations:check,test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.This is part of Wave 3. It is based on the Wave 2 integration branch (#25), and the Wave 3 integration PR adds the README line.