Skip to content

feat(cache): add opt-in scoped semantic generation cache - #56

Merged
ALikesToCode merged 2 commits into
feat/canary-trafficfrom
feat/semantic-cache
Oct 9, 2026
Merged

ALikesToCode merged 2 commits into
feat/canary-trafficfrom
feat/semantic-cache

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds an opt-in scoped semantic generation cache. With SEMANTIC_CACHE_ENABLED off and SEMANTIC_CACHE_POLICY_JSON={} (the defaults), and for any route or key that isn't opted in, the exact cache and every request are unchanged.

  • Opt-in:
    • The policy names routes and/or authenticated account IDs, plus an explicit provider:model embedding model that the gateway already has configured and priced.
    • Only allow_tools: false and allow_streams: false are supported. Setting either to true is invalid and turns the feature off.
    • Eligible traffic is limited to direct, explicitly selected Chat Completions models: no tools, streams or multiple choices. Auto and cascade routes bypass the cache.
  • Matching:
    • Everything except the final user text must match exactly: principal and key, provider, model, route, system prompt and earlier turns, every other field, sampling, seed, schemas, and the grant, secret-policy, retention and policy revisions.
    • The final user text alone is embedded and needs cosine similarity of at least 0.98.
    • A hit is refused unless numbers, dates, month and weekday names, number words, negations and quoted strings all match in order.
    • Volatile terms such as today, latest, weather and stock bypass the cache.
  • Spending:
    • Embedding input is capped at 1,024 estimated tokens; longer inputs bypass the cache and are never truncated.
    • Lookup is skipped if the embedding price is unknown or one lookup could exceed $0.001.
    • Every embedding is charged through usage accounting, including misses. Embedding permissions and budget admission apply.
    • No warmup generations.
  • Hits:
    • A hit replays the stored JSON bytes with X-MultiLLM-Cache: semantic-hit, X-MultiLLM-Cache-Backend: semantic-d1-r2, X-MultiLLM-Usage-Basis: cache-served, X-MultiLLM-Provider-Calls: 0 and Age.
    • Internally it first carries the existing hit marker, so accounting, reservations and idempotency settle exactly as for an exact-cache hit.
    • no-store, no-cache, max-age and X-MultiLLM-Cache: refresh are honored.
  • Storage:
    • Metadata goes in semantic_generation_cache (migration 0026): scoped digests, a bounded vector, expiry, an integrity hash and a body pointer.
    • Bodies go in the existing multillm_media bucket under semantic-cache/.
    • Limits: a 300 s TTL, 256 entries per principal (oldest evicted) and 1 MiB per body, with an exact scan and no Vectorize.
    • Only responses passing the existing cache_response_allowed classification are stored.
  • Failures:
    • Zero-content retention bypasses embedding, lookup and storage.
    • A missing schema while enabled returns JSON 503 semantic_cache_schema_missing before any embedding or generation.
    • Other cache or embedding errors fall through to normal dispatch once.

Details are in docs/semantic-generation-cache.md.

Shared registrations

  • tests/test_semantic_generation_cache_worker.mjs in test:worker.
  • worker/semantic-cache-d1.mjs and worker/semantic-generation-cache.mjs in the edge module inventory.
  • SEMANTIC_CACHE_ENABLED and SEMANTIC_CACHE_POLICY_JSON in the Container env keys.
  • semantic_generation_cache in REQUIRED_D1_TABLES.
  • The migration was renumbered from 0028 to 0026 to follow the Wave 5 stack.

Stacked on #54

Migration numbers must be contiguous in each PR, so this branch stacks on feat/canary-traffic (#54, migration 0025), which stacks on feat/context-pages (#50, migration 0024). Merge those first.

This branch starts from the glued Wave 4 base (cd4ac5f), so integrate/gateway-waves-1-4 was merged into feat/context-pages, then down the stack. That keeps this PR's diff to the semantic cache. #50 and #54 were re-tested after that merge.

Left for the Wave 5 integration

  • Pass the private semantic-cache state domain to handleSemanticCache in worker/control-state-d1.mjs, gated by the flag, with a 2 MiB limit. Until then, the Flask cache can't reach storage in a D1 deployment, and lookups fall through to normal dispatch.
  • Native Worker routes: call prepareSemanticCache in nativeGenerationFetch with the embedding transport, grant, budget and accounting collaborators.
  • Add cleanupSemanticCache to the scheduled maintenance.

Tests

  • tests/test_semantic_generation_cache.py and tests/test_semantic_generation_cache_worker.mjs cover:
    • defaults, empty and invalid configuration, and tools or streams turning the feature off;
    • every partition field except the last user text;
    • fact mismatches and similarity below the threshold;
    • ineligible requests (tools, streams, n > 1, retention, no-store, unknown price, scopes that aren't opted in);
    • the 503 for a missing schema before any provider call;
    • embedding grants and budget admission before dispatch;
    • nothing stored for cancelled, unclassified, incomplete or failed outcomes;
    • an embedding timeout aborting its transport;
    • scoped parallel writes within capacity;
    • integrity checks and oldest-first eviction.
  • Full Python suite: 4,051 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,524 passed on a solo run (two timing-sensitive tests failed while three CI runs shared the machine). npm run test:integration: 7 passed.
  • migrations:check (26 contiguous, additive migrations), test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.

Apply the migration before enabling the flag: npx wrangler d1 migrations apply multillm-intelligence --remote.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 5. The README line comes with the Wave 5 integration PR.

Behind SEMANTIC_CACHE_ENABLED and an explicit route or key opt-in in SEMANTIC_CACHE_POLICY_JSON, managed nonstream chat requests without tools can reuse a complete answer whose last user message has cosine similarity of at least 0.98. Every other part of the request, the model, sampling settings and policy revisions must match exactly, and a hit is refused when numbers, dates, negations or quoted text differ. Embeddings use an explicitly configured, priced model capped at 1,024 tokens and $0.001 per lookup. Entries are scoped to the principal, kept for 300 seconds, 256 per principal and 1 MiB each, in D1 (migration 0026) and the existing media bucket. Hits report semantic-hit provenance with no provider call; zero-content retention bypasses the cache.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4259374d-d397-46c5-91e4-df407306e4a2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy ddffeba Oct 09 2026, 02:35 AM

@ALikesToCode
ALikesToCode merged commit 969f920 into feat/canary-traffic Oct 9, 2026
4 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant