Repository navigation
feat(routing): add opt-in outcome-bound prompt cache affinity - #40
Conversation
Prefer the credential that actually completed a reusable prefix on authenticated managed automatic routes behind PROMPT_CACHE_AFFINITY_ENABLED (default off). Bindings come only from classified successful attempts, are hash-only and bounded by TTL and LRU, and never override cooldown, grants, budgets or fallback eligibility.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
multillm-proxy | 22d205c | Oct 09 2026, 12:23 AM |
07e45d9
into
integrate/gateway-waves-1-3
Summary
Adds outcome-bound prompt cache affinity behind
PROMPT_CACHE_AFFINITY_ENABLED, which defaults to false. While it is off, request and response bytes, headers, credential order and retry behaviour are unchanged.PROMPT_CACHE_AFFINITY_TTL_SECONDS(default 900, from 1 to 86,400), with an LRU limit. Bindings are isolated by principal, session, role lane, route, opening prefix, tool schema and policy revision.Details are in
docs/prompt-cache-affinity.md.Integration
The Wave 4 integration PR connects two remaining hooks:
AffinityScope.order_candidatesfor a tier-aware scheduler, and the intelligence transport's completion callback. Until then, affinity applies to the managed unified route and its automatic aliases only.routes/unified.pyis 999 lines; the edits are small call sites.Shared registrations
PROMPT_CACHE_AFFINITY_ENABLEDandPROMPT_CACHE_AFFINITY_TTL_SECONDSto the Container env keys.Tests
tests/test_prompt_cache_affinity.py: new. It covers:npm run test:worker: 1,349 passed.cf:migrations:check,test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.pythonandnodejobs currently fail at the dependency-audit step on every branch, including main, because of new advisories. The tests themselves don't run there. A separate dependency PR fixes this.This is part of Wave 4. It is based on
integrate/gateway-waves-1-3, and the Wave 4 integration PR adds the README line.