Repository navigation
feat(pools): add opt-in learned provider cooldowns - #67
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
multillm-proxy | ad04d06 | Oct 09 2026, 05:53 AM |
Summary
Adds opt-in learned provider cooldowns. The credential pools already rest a key after a throttle; this feature learns a better rest length from natural throttle and recovery outcomes, per credential, provider, model and quota bucket. It sends no probes or extra provider requests. It changes only the rest length, never route order, credential preference or replay permission. With
LEARNED_COOLDOWN_MODEunset oroff(the default), pool selection, cooldowns and storage are byte-for-byte unchanged, and the learner is never called.LEARNED_COOLDOWN_MODEisoff,shadoworapply. Empty means off; a malformed mode or policy is off with one warning that leaves the value out.LEARNED_COOLDOWN_POLICY_JSON(default{}) holds only abucketsarray, at most 128 entries and 32 KiB. Each entry names an exactprovider,modelandquota_bucket, where model or quota bucket may benullbut not both. Wildcards, duplicates and unknown fields are rejected.Retry-Afteror reset hint is a hard minimum, and the existing model cooldown cap still applies.INTELLIGENCE_STORAGE_BACKEND=d1, state uses D1 tablelearned_cooldown(migration0031) through a private storage domain withget/put, revision checks and a 1,024-row cap.Details are in
docs/learned-cooldown.md.Wiring
services/credential_pool.py,services/nanogpt_key_pool.pyandservices/model_cooldown.pyeach get one call tolearned_cooldown.adjust_cooldown. It returns the existing duration unchanged when the mode is off, the bucket isn't configured, the outcome isn't a throttle, or confidence is insufficient.worker/learned-cooldown-d1.mjsis the D1 storage handler. It is connected to the private control-state dispatcher, and the Flask store gets itslearned-cooldownendpoint, with the Wave 6 integration. Until then a D1 deployment falls back to the existing cooldowns, as above.Review fix
The NanoGPT pool call originally passed
max(1, rejected_cooldown_seconds)as the existing duration. With the mode off, a configured cooldown of 0 would have become 1 second. It now passes the configured value through unchanged; the learner clamps internally when it is on.test_off_passes_nanogpt_rejection_cooldown_throughcovers this and fails without the fix.Shared registrations
tests/test_learned_cooldown_worker.mjsintest:worker.worker/learned-cooldown-d1.mjsin the edge module inventory.LEARNED_COOLDOWN_MODEandLEARNED_COOLDOWN_POLICY_JSONin the Container env keys.learned_cooldowninREQUIRED_D1_TABLES.No Flask or public Worker routes, and no browser-readable headers.
Stacked on #66
Migration numbers must be contiguous in each PR, so this branch merges
feat/stream-cost-breaker(#66, migration 0030), which mergesfeat/realtime(#62, migration 0029). Merge those first. The usage receipt PR stacks on this one.Tests
tests/test_learned_cooldown.pyandtests/test_learned_cooldown_worker.mjscover:npm run test:worker: 1,675 passed.npm run test:integration: 7 passed.cf:migrations:checkreports 31 contiguous, additive migrations.test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.Apply the migration before enabling D1 storage:
npx wrangler d1 migrations apply multillm-intelligence --remote.CI notes
python/nodejobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.AdmissionCoordinatorDurable Object migration, whichwrangler versions uploadcannot apply. This is expected untilwrangler deployruns on main.This is part of Wave 6. The README line comes with the Wave 6 integration PR.