Repository navigation
feat(cache): add opt-in shared exact generation cache - #43
Conversation
Share complete, deterministic non-streaming responses across Flask Containers and native Worker routes behind GENERATION_CACHE_SHARED_ENABLED with GENERATION_CACHE_BACKEND=d1-r2 (default memory, unchanged). Entries are isolated by principal and resolved policy, stored as D1 metadata (migration 0019) with bounded bodies in the existing R2 bucket, and zero-content retention bypasses storage.
# Conflicts: # package.json # worker/container-env.mjs # worker/d1-schema.mjs
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
multillm-proxy | c7fa539 | Oct 09 2026, 12:33 AM |
Summary
Adds an opt-in shared exact generation cache. Complete, deterministic, non-streaming Chat Completions responses can be shared between Flask Containers and native Worker routes. With the defaults (
GENERATION_CACHE_BACKEND=memory,GENERATION_CACHE_SHARED_ENABLED=false), the existing process-memory cache is unchanged.GENERATION_CACHE_BACKEND=d1-r2;GENERATION_CACHE_SHARED_ENABLED=true;X-MultiLLM-Cache: onon each request.temperature: 0or an integer seed), single-choice, non-streaming, with no runnable tools.X-MultiLLM-Cache: hit|missandX-MultiLLM-Cache-Backend: d1-r2.Age,X-MultiLLM-Usage-Basis: cache-servedandX-MultiLLM-Provider-Calls: 0. These are added to the browser-exposed CORS headers.generation_cache(migration0019).multillm_mediaR2 bucket undergeneration-cache/v1/.POST /v1/state/generation-cache(fixedget,putandpruneoperations). It is unavailable while the feature is off.Details are in
docs/shared-generation-cache.md.Stacking
This PR is based on
feat/deferred-tools(#38), which owns migration0018, socf:migrations:checksees contiguous numbering. Review only the top commit,feat(cache): add opt-in shared exact generation cache; the merge commit brings in #38.Deployment
Apply the migration before enabling the flag. A push to main does not apply D1 migrations:
Scheduled cleanup of expired entries (
cleanupGenerationCache, or the privatepruneoperation) is not installed yet. The Wave 4 integration PR schedules it.Shared registrations
tests/test_shared_generation_cache_worker.mjstotest:worker.GENERATION_CACHE_BACKENDandGENERATION_CACHE_SHARED_ENABLEDto the Container env keys.generation_cacheto the D1 readiness tables.worker/exact-generation-cache.mjsandworker/generation-cache-d1.mjsto the secret firewall inventory.X-MultiLLM-Cache-Backend,X-MultiLLM-Usage-Basis,X-MultiLLM-Provider-CallsandAgeto the exposed CORS headers in Flask and the Worker.Tests
tests/test_shared_generation_cache.pyandtests/test_shared_generation_cache_worker.mjs.npm run test:worker: 1,386 passed.npm run test:integration: 7 passed.cf:migrations:checkreports 19 contiguous, additive migrations.test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.CI notes
python/nodejobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.AdmissionCoordinatorDurable Object migration, whichwrangler versions uploadcannot apply. This is expected untilwrangler deployruns on main.This is part of Wave 4. The README line comes with the Wave 4 integration PR.