Skip to content

feat(pools): add opt-in learned provider cooldowns - #67

Merged
ALikesToCode merged 4 commits into
feat/stream-cost-breakerfrom
feat/learned-cooldown
Oct 9, 2026
Merged

ALikesToCode merged 4 commits into
feat/stream-cost-breakerfrom
feat/learned-cooldown

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds opt-in learned provider cooldowns. The credential pools already rest a key after a throttle; this feature learns a better rest length from natural throttle and recovery outcomes, per credential, provider, model and quota bucket. It sends no probes or extra provider requests. It changes only the rest length, never route order, credential preference or replay permission. With LEARNED_COOLDOWN_MODE unset or off (the default), pool selection, cooldowns and storage are byte-for-byte unchanged, and the learner is never called.

  • Settings:
    • LEARNED_COOLDOWN_MODE is off, shadow or apply. Empty means off; a malformed mode or policy is off with one warning that leaves the value out.
    • LEARNED_COOLDOWN_POLICY_JSON (default {}) holds only a buckets array, at most 128 entries and 32 KiB. Each entry names an exact provider, model and quota_bucket, where model or quota bucket may be null but not both. Wildcards, duplicates and unknown fields are rejected.
  • Learning:
    • The interval lives in 1–3,600 seconds.
    • A later throttle in the same bucket raises the lower bound. A genuine success after a throttle lowers the upper bound.
    • Three consistent observations allow a midpoint trial, with at most 12 trial steps.
    • Contradictory or out-of-order observations widen the interval and remove confidence.
    • Authentication rejections, caller errors, cancellations, transport errors and catalog probes teach nothing.
    • State expires seven days after creation.
  • Shadow logs the suggested seconds, an opaque bucket digest, confidence and step count, and changes nothing.
  • Apply uses a confident suggestion for throttles only. A trusted Retry-After or reset hint is a hard minimum, and the existing model cooldown cap still applies.
  • Privacy: buckets are identified by credential-keyed HMAC digests. Keys, key prefixes, model names and quota names are never stored.
  • Storage:
    • Standalone Flask keeps up to 1,024 entries in process memory.
    • With INTELLIGENCE_STORAGE_BACKEND=d1, state uses D1 table learned_cooldown (migration 0031) through a private storage domain with get/put, revision checks and a 1,024-row cap.
    • A missing table or unavailable store falls back to the existing cooldown with one warning. It never blocks traffic.

Details are in docs/learned-cooldown.md.

Wiring

  • services/credential_pool.py, services/nanogpt_key_pool.py and services/model_cooldown.py each get one call to learned_cooldown.adjust_cooldown. It returns the existing duration unchanged when the mode is off, the bucket isn't configured, the outcome isn't a throttle, or confidence is insufficient.
  • worker/learned-cooldown-d1.mjs is the D1 storage handler. It is connected to the private control-state dispatcher, and the Flask store gets its learned-cooldown endpoint, with the Wave 6 integration. Until then a D1 deployment falls back to the existing cooldowns, as above.

Review fix

The NanoGPT pool call originally passed max(1, rejected_cooldown_seconds) as the existing duration. With the mode off, a configured cooldown of 0 would have become 1 second. It now passes the configured value through unchanged; the learner clamps internally when it is on. test_off_passes_nanogpt_rejection_cooldown_through covers this and fails without the fix.

Shared registrations

  • tests/test_learned_cooldown_worker.mjs in test:worker.
  • worker/learned-cooldown-d1.mjs in the edge module inventory.
  • LEARNED_COOLDOWN_MODE and LEARNED_COOLDOWN_POLICY_JSON in the Container env keys.
  • learned_cooldown in REQUIRED_D1_TABLES.

No Flask or public Worker routes, and no browser-readable headers.

Stacked on #66

Migration numbers must be contiguous in each PR, so this branch merges feat/stream-cost-breaker (#66, migration 0030), which merges feat/realtime (#62, migration 0029). Merge those first. The usage receipt PR stacks on this one.

Tests

  • tests/test_learned_cooldown.py and tests/test_learned_cooldown_worker.mjs cover:
    • mode off, empty or malformed preserving state, cooldowns and order with no storage calls;
    • binary trial after three recoveries, and later throttles raising the lower bound;
    • contradictions and out-of-order noise widening the interval and disabling apply;
    • trusted hints as hard floors, and untrusted hints ignored;
    • the 12-step limit and the seven-day expiry;
    • authentication, caller and provider failures teaching nothing;
    • shadow logging the suggestion without changing cooldowns or logging keys;
    • all three pool record paths changing only the duration;
    • isolation between models, quota buckets, credentials and providers;
    • strict policy validation warning once without the value;
    • D1 revision races, the row cap, expiry and the missing-table 503.
  • Full Python suite: 4,532 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,675 passed. npm run test:integration: 7 passed.
  • cf:migrations:check reports 31 contiguous, additive migrations. test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

Apply the migration before enabling D1 storage: npx wrangler d1 migrations apply multillm-intelligence --remote.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 6. The README line comes with the Wave 6 integration PR.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4e133035-f555-4366-9fd6-cf736c074e0e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy ad04d06 Oct 09 2026, 05:53 AM

@ALikesToCode
ALikesToCode merged commit 63aa627 into feat/stream-cost-breaker Oct 9, 2026
4 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant