Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 93 additions & 0 deletions docs/pool-reset-scheduling.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Reset-aware account pool scheduling

Reset scheduling is disabled by default. It refines account choice inside the
provider pool and priority tier already selected by the gateway. It does not
change provider order, model choice, authorization, request bodies or replay
permission.

```sh
POOL_RESET_SCHEDULING_ENABLED=false
POOL_OBSERVATION_TTL_SECONDS=60
```

To enable local reset scheduling:

```sh
POOL_RESET_SCHEDULING_ENABLED=true
POOL_OBSERVATION_TTL_SECONDS=60
```

An empty flag means disabled. An empty TTL means 60 seconds. The TTL accepts
integers from 5 through 3600 seconds. A malformed setting disables scheduling
and emits one warning per setting per process, without its value.

## Account choice

The scheduler first applies the existing credential and model cooldowns. Fresh
quota windows with zero remaining quota also exclude an account. Among accounts
with fresh usable observations, the earliest relevant reset wins, with the
configured account order breaking ties. Accounts with no fresh usable evidence
retain their existing positions; only positions with fresh evidence reorder.
For example, two observed accounts with resets in 100 and 20 seconds exchange
order, while an unobserved account between them stays in its position.

Account observations apply across models. Model observations apply only to the
same model; an explicitly supplied quota bucket shares observations only within
that bucket. Raw and unified NanoGPT pools have distinct identities. Existing
prompt cache affinity may prefer an account only after this eligibility filter.

Contradictory quota totals, negative or nonfinite values, conflicting absolute
and relative reset timestamps, missing reset/quota pairs, expired windows and
stale observations do not change legacy eligibility or ordering. Remaining quota
can be derived from consistent limit and used values. Multiple relevant windows
are checked independently: any fresh exhausted window excludes the account.
An expired reset stops supplying advice; it does not assert that quota renewed.

When every configured account is cooling or freshly exhausted, selection raises
the existing model cooldown HTTP 429 error before dispatch. Its error body stays
the same. `Retry-After` uses the soonest fresh reset, rounded up and bounded to
1–3600 seconds, also respecting a smaller configured model cooldown maximum.
Without fresh reset evidence it uses the existing cooldown advice. Advice never
authorizes another request or clears a cooldown. The gateway's cooldown error
adapter must be registered to expose `Retry-After` on HTTP responses.

## Observation sources and cost

The scheduler consumes normalized windows only from authorized usage snapshots
already fetched by the gateway and from metadata already supplied with classified
request results. Cached snapshots do not renew observation freshness. NanoGPT's
subscription usage endpoint is excluded as a scheduling source. No observations
come from browser sessions, subscriptions, rewards or check-ins.

`record_pool_usage(provider, credential, windows, observed_at=..., model=...,
quota_bucket=...)` consumes already received normalized windows. Pool result
methods also accept `usage_windows`; a classified quota throttle with existing
`retry_after_seconds` supplies exhausted quota and a relative reset. Callers must
pass trusted response metadata with the actual credential and dispatch scope.
Unscoped metadata is account-wide; no model scope is inferred from an unrelated
request. The scheduler itself performs no network, storage or refresh calls.

With reset scheduling enabled, NanoGPT's existing catalog validation probes
keep their cadence. The scheduler adds no probes of its own; it filters freshly
exhausted keys and orders eligible keys before the existing validation logic
runs. A validated active key stays in use until its existing re-check unless a
fresh observation shows it exhausted. The selection method that does not probe
continues to use local cooldown and quota evidence without claiming validation.

## Retention and bounds

State is process-local and disappears on restart. There is no distributed
account-state guarantee and no persistent table. At most 10,000 account/scope
observations are retained, with at most 32 quota windows per observation.
Expired observations are removed before the oldest received observation is
evicted for capacity. Newer observations supersede older observations for the
same identity and scope; delayed older updates cannot replace fresh evidence.

Only salted HMAC credential identities are stored, never keys or key prefixes.
Neither observations nor requests are logged. Freshness is checked with wall
and monotonic clocks. Future timestamp skew is tolerated up to five seconds,
without extending the TTL; larger skew and backward clock movement invalidate
observations. Reset timestamps are accepted up to one year after observation.
TTL expiration, missing evidence and capacity eviction restore existing account
selection; these observations are scheduling advice, not a quota accounting
authority. They cannot certify provider metering or live reset behavior.
54 changes: 53 additions & 1 deletion services/credential_pool.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@

from providers.codex_everywhere import CODEX_EVERYWHERE_POOLS
from services.auth_primitives import usable_credential
from services.model_cooldown import model_cooldown, settings
from services.model_cooldown import ModelCooldownExhausted, model_cooldown, settings
from services import pool_reset_schedule as reset_schedule
from services.upstream_outcome import UpstreamOutcome, classify_upstream_outcome

REFUSED_REST_SECONDS = 300
Expand Down Expand Up @@ -77,6 +78,11 @@ def available(
) -> list[str]:
"""Configured keys that are not resting, in preference order."""
keys = cls.keys(provider) if keys is None else keys
if reset_schedule.settings().enabled:
return cls._scheduled_available(
provider, keys, now=now, model=model, quota_bucket=quota_bucket,
require_eligible=require_eligible,
)
config = settings()
if config.enabled and (model or quota_bucket):
with cls._lock:
Expand All @@ -102,6 +108,37 @@ def available(
key for key in keys if cls._resting.get((provider, key), 0) <= current
]

@classmethod
def _scheduled_available(
cls, provider: str, keys: list[str], *, now: float | None = None,
model: str | None = None, quota_bucket: str | None = None,
require_eligible: bool = False,
) -> list[str]:
current = time.monotonic() if now is None else now
config = settings()
with cls._lock:
legacy = {key: cls._resting.get((provider, key), 0) for key in keys}
eligible = [key for key in keys if legacy[key] <= current]
fallback = min((max(1, until - current) for until in legacy.values()), default=60)
if config.enabled and (model or quota_bucket):
eligible = model_cooldown.available(
provider, keys, model=model, quota_bucket=quota_bucket,
max_seconds=config.max_seconds, now=current, legacy_rest_until=legacy,
)
if not eligible and require_eligible:
try:
model_cooldown.select(
provider, keys, model=model, quota_bucket=quota_bucket,
max_seconds=config.max_seconds, now=current, legacy_rest_until=legacy,
)
except ModelCooldownExhausted as error:
fallback = error.retry_after
return reset_schedule.pool_reset_schedule.admit(
provider, keys, eligible, model=model, quota_bucket=quota_bucket,
require_eligible=require_eligible, fallback=fallback,
max_seconds=config.max_seconds if config.enabled else 3600,
)

@classmethod
def select(
cls,
Expand All @@ -113,6 +150,13 @@ def select(
) -> str | None:
"""The key to use now; the first configured one when every key is resting."""
keys = cls.keys(provider)
if reset_schedule.settings().enabled:
available = cls._scheduled_available(
provider, keys, now=now, model=model, quota_bucket=quota_bucket,
require_eligible=True,
)
from services.prompt_cache_affinity import prefer_credential
return prefer_credential(provider, model, available) or next(iter(available), None)
config = settings()
if config.enabled and (model or quota_bucket):
with cls._lock:
Expand Down Expand Up @@ -148,9 +192,15 @@ def record(
outcome: UpstreamOutcome | None = None,
credential_wide_auth: bool | None = None,
retry_after_seconds: float | None = None,
usage_windows: list[Mapping] | None = None,
) -> None:
if not key or provider not in POOLED_KEY_ENVS:
return
reset_schedule.record_result_observation(
provider, key, status, model=model, quota_bucket=quota_bucket,
outcome=outcome, retry_after_seconds=retry_after_seconds,
usage_windows=usage_windows,
)
config = settings()
if config.enabled and (model or quota_bucket):
classified = outcome or classify_upstream_outcome(status)
Expand Down Expand Up @@ -204,6 +254,7 @@ def record_headers(
outcome: UpstreamOutcome | None = None,
credential_wide_auth: bool | None = None,
retry_after_seconds: float | None = None,
usage_windows: list[Mapping] | None = None,
now: float | None = None,
) -> None:
"""Record the result for the bearer key a request carried."""
Expand All @@ -227,6 +278,7 @@ def record_headers(
outcome=outcome,
credential_wide_auth=credential_wide_auth,
retry_after_seconds=retry_after_seconds,
usage_windows=usage_windows,
now=now,
)

Expand Down
58 changes: 57 additions & 1 deletion services/nanogpt_key_pool.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@
from typing import ClassVar

from services.auth_primitives import is_placeholder_credential
from services.model_cooldown import model_cooldown, settings
from services.model_cooldown import ModelCooldownExhausted, model_cooldown, settings
from services import pool_reset_schedule as reset_schedule
from services.upstream_outcome import UpstreamOutcome, classify_upstream_outcome

_DIRECT_KEY_NAMES = ("NANOGPT_API_KEY", "NANO_GPT_KEY")
Expand Down Expand Up @@ -118,6 +119,8 @@ def select_available_key(
"""Select an uncooled credential without claiming a successful probe."""
configured = list(dict.fromkeys(key.strip() for key in keys if key.strip()))
now = time.monotonic() if now is None else now
if reset_schedule.settings().enabled:
return cls._scheduled_key(configured, model=model, quota_bucket=quota_bucket, now=now)
config = settings()
scoped = config.enabled and bool(model or quota_bucket)
with cls._lock:
Expand All @@ -138,6 +141,48 @@ def select_available_key(
(key for key in ordered if cls._rejected_until.get(key, 0) <= now), None
)

@classmethod
def _scheduled_key(
cls, configured: list[str], *, model: str | None = None,
quota_bucket: str | None = None, now: float,
) -> str | None:
with cls._lock:
cls._prune(configured, now)
ordered = [cls._active_key] if cls._active_key in configured else []
ordered.extend(key for key in configured if key not in ordered)
return next(iter(cls._scheduled_candidates(
configured, ordered, model=model, quota_bucket=quota_bucket, now=now,
)), None)

@classmethod
def _scheduled_candidates(
cls, configured: list[str], ordered: list[str], *,
model: str | None = None, quota_bucket: str | None = None, now: float,
) -> list[str]:
config = settings()
eligible = [key for key in ordered if cls._rejected_until.get(key, 0) <= now]
fallback = min((max(1, cls._rejected_until.get(key, now + 60) - now) for key in configured), default=60)
if config.enabled and (model or quota_bucket):
eligible = model_cooldown.available(
cls._model_cooldown_provider, ordered, model=model,
quota_bucket=quota_bucket, max_seconds=config.max_seconds,
now=now, legacy_rest_until=cls._rejected_until,
)
if not eligible:
try:
model_cooldown.select(
cls._model_cooldown_provider, ordered, model=model,
quota_bucket=quota_bucket, max_seconds=config.max_seconds,
now=now, legacy_rest_until=cls._rejected_until,
)
except ModelCooldownExhausted as error:
fallback = error.retry_after
return reset_schedule.pool_reset_schedule.admit(
cls._model_cooldown_provider, configured, eligible, model=model,
quota_bucket=quota_bucket, require_eligible=True, fallback=fallback,
max_seconds=config.max_seconds if config.enabled else 3600,
)

@classmethod
def select_key(
cls,
Expand All @@ -161,6 +206,11 @@ def select_key(
single_configured_key = len(configured) == 1
with cls._lock:
cls._prune(configured, checked_at)
if reset_schedule.settings().enabled:
configured = cls._scheduled_candidates(
configured, configured, model=model,
quota_bucket=quota_bucket, now=checked_at,
)
if scoped:
eligible = model_cooldown.available(
cls._model_cooldown_provider,
Expand Down Expand Up @@ -308,10 +358,16 @@ def record_result(
outcome: UpstreamOutcome | None = None,
credential_wide_auth: bool | None = None,
retry_after_seconds: float | None = None,
usage_windows: list[Mapping] | None = None,
) -> None:
"""Record one upstream request and invalidate definite key failures."""
if not key:
return
reset_schedule.record_result_observation(
cls._model_cooldown_provider, key, status, model=model,
quota_bucket=quota_bucket, outcome=outcome,
retry_after_seconds=retry_after_seconds, usage_windows=usage_windows,
)
if settings().enabled and (model or quota_bucket):
cls._record_model_result(
key,
Expand Down
Loading
Loading