Skip to content

feat(cache): add opt-in shared exact generation cache - #43

Merged
ALikesToCode merged 2 commits into
feat/deferred-toolsfrom
feat/shared-generation-cache
Oct 9, 2026
Merged

ALikesToCode merged 2 commits into
feat/deferred-toolsfrom
feat/shared-generation-cache

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds an opt-in shared exact generation cache. Complete, deterministic, non-streaming Chat Completions responses can be shared between Flask Containers and native Worker routes. With the defaults (GENERATION_CACHE_BACKEND=memory, GENERATION_CACHE_SHARED_ENABLED=false), the existing process-memory cache is unchanged.

  • Enabling it needs three things:
    • GENERATION_CACHE_BACKEND=d1-r2;
    • GENERATION_CACHE_SHARED_ENABLED=true;
    • X-MultiLLM-Cache: on on each request.
  • Order of checks: authentication, key scopes, model permissions and retention policy run before lookup.
  • Cache identity: the canonical request, the principal and key digest, the route, the model, the effective server policy and the workflow version.
    • Native identity also includes the upstream endpoint, organization/project headers and configuration revisions.
    • Requests that are ambiguous or carry their own credentials bypass the cache.
  • What can be stored:
    • The request must be deterministic (temperature: 0 or an integer seed), single-choice, non-streaming, with no runnable tools.
    • Only HTTP 200 JSON responses with a complete finish reason are stored.
    • Errors, partial, filtered, malformed and oversized results are never stored.
    • There is no semantic matching.
  • Response headers:
    • Eligible responses carry X-MultiLLM-Cache: hit|miss and X-MultiLLM-Cache-Backend: d1-r2.
    • Hits add Age, X-MultiLLM-Usage-Basis: cache-served and X-MultiLLM-Provider-Calls: 0. These are added to the browser-exposed CORS headers.
    • Accounting records a hit as a replay with zero new provider cost.
  • Storage:
    • Metadata lives in the new D1 table generation_cache (migration 0019).
    • Bodies go in the existing multillm_media R2 bucket under generation-cache/v1/.
    • Limits: 512 entries and 16 MiB per principal, 1 MiB per body, 300-second TTL.
    • A storage failure falls through to normal dispatch.
  • Retention: zero-content retention bypasses the cache before any content is read or written.
  • Private endpoint: Flask reaches storage through POST /v1/state/generation-cache (fixed get, put and prune operations). It is unavailable while the feature is off.

Details are in docs/shared-generation-cache.md.

Stacking

This PR is based on feat/deferred-tools (#38), which owns migration 0018, so cf:migrations:check sees contiguous numbering. Review only the top commit, feat(cache): add opt-in shared exact generation cache; the merge commit brings in #38.

Deployment

Apply the migration before enabling the flag. A push to main does not apply D1 migrations:

npx wrangler d1 migrations apply multillm-intelligence --remote

Scheduled cleanup of expired entries (cleanupGenerationCache, or the private prune operation) is not installed yet. The Wave 4 integration PR schedules it.

Shared registrations

  • Added tests/test_shared_generation_cache_worker.mjs to test:worker.
  • Added GENERATION_CACHE_BACKEND and GENERATION_CACHE_SHARED_ENABLED to the Container env keys.
  • Added generation_cache to the D1 readiness tables.
  • Added worker/exact-generation-cache.mjs and worker/generation-cache-d1.mjs to the secret firewall inventory.
  • Added X-MultiLLM-Cache-Backend, X-MultiLLM-Usage-Basis, X-MultiLLM-Provider-Calls and Age to the exposed CORS headers in Flask and the Worker.

Tests

  • New tests: tests/test_shared_generation_cache.py and tests/test_shared_generation_cache_worker.mjs.
  • Full Python suite: 3,593 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,386 passed. npm run test:integration: 7 passed.
  • cf:migrations:check reports 19 contiguous, additive migrations.
  • test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 4. The README line comes with the Wave 4 integration PR.

Share complete, deterministic non-streaming responses across Flask Containers and native Worker routes behind GENERATION_CACHE_SHARED_ENABLED with GENERATION_CACHE_BACKEND=d1-r2 (default memory, unchanged). Entries are isolated by principal and resolved policy, stored as D1 metadata (migration 0019) with bounded bodies in the existing R2 bucket, and zero-content retention bypasses storage.
# Conflicts:
#	package.json
#	worker/container-env.mjs
#	worker/d1-schema.mjs
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 63a0ad2e-3f6f-44af-a873-ad444a00c461

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy c7fa539 Oct 09 2026, 12:33 AM

@ALikesToCode
ALikesToCode merged commit 352c847 into feat/deferred-tools Oct 9, 2026
4 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant