Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 127 additions & 0 deletions docs/model-cooldown.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Model-scoped credential cooldown

`MODEL_COOLDOWN_ENABLED` defaults to `false`. Disabled pools keep their existing
selection, cooldown, validation and response behavior, including the legacy
Codex Everywhere fallback to the first key when every key is resting.

With the flag enabled, model scoping applies only where the request path supplies
a model or an explicit quota bucket. Paths without either keep the same
provider-wide cooldown bookkeeping, selection and exceptions as the disabled
path, including 429 and 401 handling.

Enable with:

```text
MODEL_COOLDOWN_ENABLED=true
MODEL_COOLDOWN_MAX_SECONDS=3600
```

Disable with `MODEL_COOLDOWN_ENABLED=false`, or clear it to an empty value.
An empty maximum uses 3600 seconds. Whitespace is ignored. Malformed settings
log once per setting name, without the supplied value, and disable this feature.
The maximum must be an integer from 1 through 2147483647.

## Evidence and scope

The pool methods accept keyword-only `model`, `quota_bucket`, `outcome`,
`credential_wide_auth` and `retry_after_seconds` inputs where applicable.
Selection and observation must receive the same provider, model and quota bucket.
Use the actual upstream model, without changing an explicit model selection.
A quota bucket is an explicit provider configuration shared by its member models;
never infer it from a response body, an arbitrary caller field or a model prefix.
Model names and quota bucket names occupy distinct namespaces.

Use `UpstreamOutcome` evidence. A `throttled` credential outcome cools only
that credential and model, or its explicitly configured quota bucket.
A classified quota-exhaustion observation can also use `throttled`; a bare 402
is a classified caller error and cannot establish exhaustion. Caller errors, transport
failures, cancellation and provider failures create no cooldown.

A classified credential rejection is global only with genuine auth evidence.
Pool adapters treat a classified 401 as credential-wide unless
`credential_wide_auth=False` is supplied. A 403 needs an explicit
`credential_wide_auth=True`; endpoint/model permission denial is not enough.
A supplied classification overrides the status classification, including for
cancellation. Neither health evidence nor cooldown grants replay permission.

Trusted normalized reset advice is passed as finite nonnegative delay seconds,
relative to the observation's monotonic time. The store does not parse raw
headers or provider bodies. Invalid advice uses the existing pool's fallback:
60 seconds for throttling, 300 seconds for Codex Everywhere auth, and the
configured NanoGPT validation TTL for auth. The configured maximum caps each
observation; positive fractional delays round up only in `Retry-After`.
At least one second is retained for zero delay to prevent immediate redispatch.
Success clears only its model/bucket observation. It does not clear another
model, an auth rejection, a legacy credential-wide rest or a capacity guard.
Unscoped recording uses the legacy provider-wide rules and leaves model state
unchanged. Genuine authentication rejection on a scoped path also rests the
credential in the legacy map, using the same capped deadline, so unscoped
selection observes it.

## Selection and errors

Scoped selection excludes keys resting in either the model store or the legacy
provider-wide map. Available keys retain configured order. NanoGPT retains its existing active-key
preference, single-configured-key probe bypass, validation TTL and request-count
rules. Filtering several configured keys down to one does not skip validation.
During scoped selection, catalog probes carry no model quota evidence: only
genuine auth rejection can cool every model from that probe. Unscoped validation
keeps its legacy rejection rules. Raw and unified NanoGPT pools remain isolated.

`CredentialPool.available` returns the eligible list. `CredentialPool.select`,
`NanoGPTKeyPool.select_key` and `select_available_key` with model/bucket context raise
`ModelCooldownExhausted` when all configured eligible keys are cooling. There is
no secret dispatch, additional wait, retry or synthetic completion. An empty
configuration retains the existing no-key result. The error is a real 429 with
`error: model_cooldown`; the named Flask adapter emits bounded integer
`Retry-After` advice to the earliest eligible key's release. A credential's
release waits for both its auth and model/bucket observations to expire.

Register `cooldown_error_response` for `ModelCooldownExhausted` and
`ModelCooldownCapacity` in the Flask application. Preserve these
errors through route wrappers instead of translating them to 500/503 or a
candidate fallback. Without the adapter, the existing APIError handler does not
include `Retry-After`.

## Storage and operational limits

Model-scoped state is process-local, protected by a lock and limited to 10000 live
observations across both pool implementations. Credential IDs are SHA-256 HMACs
with a random process-local salt and provider/pool namespace. Scope labels are
also HMACs. New cooldown state stores only digests and monotonic expiry times;
it never serializes or logs keys, prompts, completions or model names.
The existing pools still retain configured/active keys for dispatch.

Expired observations are pruned on reads and writes. A process restart discards
observations and changes HMAC identities. This feature does not provide
multi-process, Container or distributed consistency. Configured credentials and
existing raw/unified pool precedence remain authoritative.

At capacity, live observations are never evicted. A scalar overflow deadline
covers the omitted observation's capped TTL. During that interval new admission
fails closed with 503 and bounded `Retry-After`; known all-cooling pools still
produce 429. This capacity failure is not a credential-wide cooldown. It may
briefly block unrelated traffic; that is the cost of preserving live cooldowns
within a hard memory bound. Observation itself cannot interrupt an already
produced response. There are no extra provider calls or charges from cooldown
tracking. Existing permitted validation probes retain their existing cost.

## Request integration

Pass the resolved upstream model and configured quota bucket to both selection
and recording. Forward classified outcomes, genuine credential-wide auth
evidence and normalized trusted reset advice to recording. For `available`
callers, an empty eligible result needs the same 429 admission decision using
`model_cooldown.select` with the original keys and legacy rest deadlines. An
empty configuration remains distinct. Preserve cooldown exceptions through
route wrappers rather than treating them as generic candidate failures.

Native Worker roleplay maintains independent health state. Worker-forwarded
Flask traffic uses these pools. Forward `MODEL_COOLDOWN_ENABLED` and
`MODEL_COOLDOWN_MAX_SECONDS` through the Container environment allowlist.
Expose `Retry-After` through Flask and Worker CORS for browser clients.
No migration, table, new production route or Worker route/module is required.

Tests cover pool selection and recording plus a registered Flask error adapter
with fake dispatch. Live provider quota semantics, deployment, billing and
distributed consistency require separate operational verification.
126 changes: 119 additions & 7 deletions services/credential_pool.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

from providers.codex_everywhere import CODEX_EVERYWHERE_POOLS
from services.auth_primitives import usable_credential
from services.model_cooldown import model_cooldown, settings
from services.upstream_outcome import UpstreamOutcome, classify_upstream_outcome

REFUSED_REST_SECONDS = 300
RATE_LIMITED_REST_SECONDS = 60
Expand Down Expand Up @@ -64,26 +66,109 @@ def keys(provider: str, environ: Mapping[str, str] | None = None) -> list[str]:

@classmethod
def available(
cls, provider: str, keys: list[str] | None = None, *, now: float | None = None
cls,
provider: str,
keys: list[str] | None = None,
*,
now: float | None = None,
model: str | None = None,
quota_bucket: str | None = None,
) -> list[str]:
"""Configured keys that are not resting, in preference order."""
keys = cls.keys(provider) if keys is None else keys
config = settings()
if config.enabled and (model or quota_bucket):
Comment on lines +79 to +80

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Pass request scope into production pool calls

With MODEL_COOLDOWN_ENABLED=true, this branch still never executes for repository production traffic because all callers retain the new arguments' None defaults: AuthService.get_api_key calls CredentialPool.select(provider), ProxyService.make_request calls record_headers without scope, and the NanoGPT proxy, unified, and intelligence paths likewise omit model from selection and recording. Consequently a production 429 continues using the legacy provider-wide cooldown rather than the advertised model-scoped state; pass the resolved upstream model or configured quota bucket through both sides of each request.

Useful? React with 👍 / 👎.

with cls._lock:
legacy = {key: cls._resting.get((provider, key), 0) for key in keys}
return model_cooldown.available(
provider,
keys,
model=model,
quota_bucket=quota_bucket,
max_seconds=config.max_seconds,
now=now,
legacy_rest_until=legacy,
)
current = time.monotonic() if now is None else now
with cls._lock:
return [key for key in keys if cls._resting.get((provider, key), 0) <= current]
return [
key for key in keys if cls._resting.get((provider, key), 0) <= current
]

@classmethod
def select(cls, provider: str, *, now: float | None = None) -> str | None:
def select(
cls,
provider: str,
*,
now: float | None = None,
model: str | None = None,
quota_bucket: str | None = None,
) -> str | None:
"""The key to use now; the first configured one when every key is resting."""
keys = cls.keys(provider)
config = settings()
if config.enabled and (model or quota_bucket):
with cls._lock:
legacy = {key: cls._resting.get((provider, key), 0) for key in keys}
return model_cooldown.select(
provider,
keys,
model=model,
quota_bucket=quota_bucket,
max_seconds=config.max_seconds,
now=now,
legacy_rest_until=legacy,
)
return next(iter(cls.available(provider, keys, now=now) or keys), None)

@classmethod
def record(
cls, provider: str, key: str | None, status: int, *, now: float | None = None
cls,
provider: str,
key: str | None,
status: int,
*,
now: float | None = None,
model: str | None = None,
quota_bucket: str | None = None,
outcome: UpstreamOutcome | None = None,
credential_wide_auth: bool | None = None,
retry_after_seconds: float | None = None,
) -> None:
if not key or provider not in POOLED_KEY_ENVS:
return
config = settings()
if config.enabled and (model or quota_bucket):
classified = outcome or classify_upstream_outcome(status)
wide_auth = (
status == 401 if credential_wide_auth is None else credential_wide_auth
)
until = model_cooldown.record(
provider,
key,
classified,
model=model,
quota_bucket=quota_bucket,
now=now,
credential_wide_auth=wide_auth,
retry_after_seconds=retry_after_seconds,
max_seconds=config.max_seconds,
fallback_seconds=(
REFUSED_REST_SECONDS
if classified.credential_health == "rejected"
else RATE_LIMITED_REST_SECONDS
),
)
if (
classified.credential_health == "rejected"
and wide_auth
and until is not None
):
with cls._lock:
cls._resting[(provider, key)] = max(
cls._resting.get((provider, key), 0), until
)
return
current = time.monotonic() if now is None else now
with cls._lock:
if status in REFUSED_STATUSES:
Expand All @@ -94,15 +179,42 @@ def record(
cls._resting.pop((provider, key), None)

@classmethod
def record_headers(cls, provider: str, headers: Mapping[str, str], status: int) -> None:
def record_headers(
cls,
provider: str,
headers: Mapping[str, str],
status: int,
*,
model: str | None = None,
quota_bucket: str | None = None,
outcome: UpstreamOutcome | None = None,
credential_wide_auth: bool | None = None,
retry_after_seconds: float | None = None,
now: float | None = None,
) -> None:
"""Record the result for the bearer key a request carried."""
if provider not in POOLED_KEY_ENVS:
return
authorization = next(
(value for name, value in headers.items() if name.lower() == "authorization"), ""
(
value
for name, value in headers.items()
if name.lower() == "authorization"
),
"",
)
if isinstance(authorization, str) and authorization.startswith("Bearer "):
cls.record(provider, authorization[len("Bearer "):].strip(), status)
cls.record(
provider,
authorization[len("Bearer ") :].strip(),
status,
model=model,
quota_bucket=quota_bucket,
outcome=outcome,
credential_wide_auth=credential_wide_auth,
retry_after_seconds=retry_after_seconds,
now=now,
)

@classmethod
def reset(cls) -> None:
Expand Down
Loading
Loading