Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions docs/learned-cooldown.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Learned provider cooldowns

Cooldown tuning uses natural throttle and recovery outcomes already recorded by
the credential pools. It sends no probes or extra provider requests. It adjusts
only a cooldown duration; route order, credential preference and permission to
replay a request remain controlled by their existing policies.

## Configuration

The default is disabled:

```sh
LEARNED_COOLDOWN_MODE=off
LEARNED_COOLDOWN_POLICY_JSON='{}'
```

Empty values use these defaults. A malformed mode or policy disables tuning and
logs one warning without the supplied value. Modes are `off`, `shadow` and
`apply`. Policies contain only a `buckets` array, with at most 128 entries and
32 KiB of JSON. Each entry requires exactly `provider`, `model` and
`quota_bucket`. Provider must be a nonempty string; model and quota bucket are
nonempty strings or `null`, and at least one must be nonnull. Names match
exactly, including null values. Wildcards, duplicate fields, duplicate scopes,
control characters and unknown fields are rejected.

To observe a specific model without changing its existing cooldown:

```sh
LEARNED_COOLDOWN_MODE=shadow
LEARNED_COOLDOWN_POLICY_JSON='{"buckets":[{"provider":"ce-gpt-pro","model":"gpt-5","quota_bucket":null}]}'
```

Shadow logs the suggested seconds, opaque bucket digest, confidence and step
count. It preserves existing cooldowns and selection. Change the mode to
`apply` to use a suggested duration after sufficient consistent evidence.
For a named quota bucket, specify its exact name and the model recorded by the
transport. A quota-only observation uses `model:null`. A configured name learns
nothing until a pool receives natural outcomes with that exact scope.

`MODEL_COOLDOWN_ENABLED` still controls the existing model-scoped admission
policy. Tuning does not enable it. With model cooldowns disabled, the pool keeps
its legacy credential-wide rest semantics while learning separate scope records.

## Evidence and bounds

The interval starts at 1–3,600 seconds. A later natural throttle in the same
credential/model/quota bucket raises its lower bound from the failed wait.
A genuine successful request after a throttle lowers its upper bound from the
elapsed wait. Success without a preceding throttle provides no recovery sample.
Authentication rejection, caller errors, cancellation, transport errors and
catalog validation probes provide no learning evidence.

Three consistent directional observations allow a midpoint trial. Each state
allows at most 12 trial steps. Contradictory observations or nonincreasing
timestamps widen the interval and remove confidence. Apply uses the existing
cooldown until confidence returns. The existing model cooldown cap also bounds
an applied trial. Trusted Retry-After or reset seconds supplied
by the transport are a hard minimum, even when longer than 3,600 seconds. Such
hints can prevent any shortening. Invalid or nonfinite hints are ignored. No
header is parsed or rewritten by the learner.

Each credential, provider, model and quota bucket has separate evidence. Model
and quota names are never stored. Credential-keyed HMAC digests provide stable
opaque identities across processes; credentials and prefixes are never stored
in learned state. Rotating a credential creates a new identity.

## Storage, errors and retention

Standalone Flask uses process-local state, bounded to 1,024 entries. It resets
on process restart and is not a distributed quota authority. State expires
seven days after creation; observations do not extend its lifetime.

With `INTELLIGENCE_STORAGE_BACKEND=d1`, learned state uses the private
`/v1/state/learned-cooldown` storage domain, backed by the additive
`0031_learned_cooldown.sql` migration. Its fixed operations are `get` and `put`;
puts use revision comparison and a transactional capacity check. Storage is
bounded to 1,024 rows. Reads ignore expired state; writes reclaim expired rows.
Physical deletion therefore depends on subsequent writes rather than a timer.

A missing table or unavailable D1 store returns a sanitized JSON 503 from the
private storage operation. The pool falls back to its existing cooldown with
one warning and continues serving traffic. A stale revision, invalid stored
state or full capacity also prevents applying a suggestion. No public route or
new response header is introduced. Default-off mode performs no learned-state
storage operations or reports.

Natural traffic can be sparse, concurrent or affected by changing provider
limits. The inferred interval is a suggestion, not proof of a provider's quota
or successful recovery. Shadow observation and D1 writes add storage work;
they add no provider usage, usage refreshes or paid generation calls. Existing
transport accounting and retry safety remain authoritative.
20 changes: 20 additions & 0 deletions intelligence-migrations/0031_learned_cooldown.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
-- Opaque identities and bounded, expiring cooldown evidence.
CREATE TABLE IF NOT EXISTS learned_cooldown (
bucket_digest TEXT PRIMARY KEY,
credential_digest TEXT NOT NULL,
lower_seconds INTEGER NOT NULL CHECK (lower_seconds BETWEEN 1 AND 3600),
upper_seconds INTEGER NOT NULL CHECK (upper_seconds BETWEEN lower_seconds AND 3600),
trial_seconds INTEGER NOT NULL CHECK (trial_seconds BETWEEN 1 AND 3600),
samples INTEGER NOT NULL CHECK (samples BETWEEN 0 AND 2147483647),
consistent INTEGER NOT NULL CHECK (consistent BETWEEN 0 AND 3),
steps INTEGER NOT NULL CHECK (steps BETWEEN 0 AND 12),
confident INTEGER NOT NULL CHECK (confident IN (0, 1)),
last_kind TEXT NOT NULL,
last_throttle REAL,
last_observed REAL NOT NULL,
floor_until REAL NOT NULL,
created_at REAL NOT NULL,
expires_at REAL NOT NULL,
revision INTEGER NOT NULL CHECK (revision BETWEEN 1 AND 2147483647)
);
CREATE INDEX IF NOT EXISTS learned_cooldown_expiry ON learned_cooldown (expires_at);
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs tests/test_stream_cost_breaker_worker.mjs tests/test_realtime_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs tests/test_learned_cooldown_worker.mjs tests/test_stream_cost_breaker_worker.mjs tests/test_realtime_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
9 changes: 8 additions & 1 deletion services/credential_pool.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
from services.auth_primitives import usable_credential
from services.model_cooldown import ModelCooldownExhausted, model_cooldown, settings
from services import pool_reset_schedule as reset_schedule
from services import learned_cooldown as learned
from services.upstream_outcome import UpstreamOutcome, classify_upstream_outcome

REFUSED_REST_SECONDS = 300
Expand Down Expand Up @@ -234,11 +235,17 @@ def record(
)
return
current = time.monotonic() if now is None else now
delay = learned.adjust_cooldown(
provider, key, outcome or classify_upstream_outcome(status),
model=model, quota_bucket=quota_bucket, now=now,
current_seconds=RATE_LIMITED_REST_SECONDS,
retry_after_seconds=retry_after_seconds,
)
with cls._lock:
if status in REFUSED_STATUSES:
cls._resting[(provider, key)] = current + REFUSED_REST_SECONDS
elif status in RATE_LIMITED_STATUSES:
cls._resting[(provider, key)] = current + RATE_LIMITED_REST_SECONDS
cls._resting[(provider, key)] = current + delay
elif 200 <= status < 300:
cls._resting.pop((provider, key), None)

Expand Down
Loading
Loading