Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
134 changes: 134 additions & 0 deletions docs/semantic-generation-cache.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# Scoped semantic generation cache

Semantic reuse is off by default. `SEMANTIC_CACHE_ENABLED` defaults to `false`, and
`SEMANTIC_CACHE_POLICY_JSON` defaults to `{}`, which opts in no routes or keys.
Empty environment values mean these defaults. Invalid configuration disables the
feature and emits one warning without the supplied value. The exact response cache
keeps its existing opt-in and behavior when semantic reuse is disabled or unscoped.

To disable semantic reuse:

```text
SEMANTIC_CACHE_ENABLED=false
SEMANTIC_CACHE_POLICY_JSON={}
```

To opt in the authenticated Chat Completions route:

```text
SEMANTIC_CACHE_ENABLED=true
SEMANTIC_CACHE_POLICY_JSON={"routes":["/v1/chat/completions"],"embedding_model":"openai:text-embedding-3-small","revision":"1","model_revision":"1","allow_tools":false,"allow_streams":false}
MODEL_PRICING_USD_PER_MILLION={"openai:text-embedding-3-small":{"input":0.02,"output":0}}
```

Pricing values are operator-supplied USD per million tokens, not a price catalog.
Preserve the other model entries in an existing pricing configuration. The embedding
model must be an explicit `provider:model` already configured in the gateway. No new
provider credentials, origins or bindings are introduced. Use `keys:["account-id"]`
to opt in an authenticated account ID, or its username when it has no ID. These are
server-authenticated identities, not raw API keys. Route and key opt-ins are combined
with OR. Worker routes use their full route name, such as
`/codex-easy/v1/chat/completions`, rather than the upstream path.

Only `allow_tools:false` and `allow_streams:false` are supported. Either option set to
`true` is invalid configuration and disables semantic reuse. Tool definitions, tool
results, function calls, streams and requests for more than one choice are ineligible.
Only direct, explicitly selected Chat Completions models are supported; automatic and
cascade routes bypass semantic reuse. The last message must be a nonempty user text.
Dynamic state, provider-selection headers and ambiguous upstream credentials bypass
reuse. Existing exact caching remains independent for these requests.

## Compatibility and fact checks

Reuse requires exact matches for the authenticated principal and presented key,
provider, model, route, system prompt, earlier turns, every request field apart from
the final user text, sampling parameters, seed, response/output schema and policy
revisions. Grants, secret policy and retention participate in the compatibility
identity. Organization, project and protocol headers also partition responses.
Change `revision` after changing a compatibility policy, and `model_revision` when
the configured embedding model changes without changing its name.

The final user text alone is embedded. Cosine similarity must be at least 0.98.
A matching vector is insufficient: the two texts must have the same ordered numbers,
dates, English month/weekday names and number words, negation words and exact quoted strings. Their protected facts are stored only
as a digest. Changing a number, a date, `not`, `never`, a negative contraction, or quoted
text prevents replay. Recognized volatile terms such as `today`, `now`, `current`,
`latest`, `weather` and `stock` bypass semantic reuse. These conservative lexical checks
cannot prove factual equivalence; opt in only routes whose answers tolerate paraphrases.

## Spending and provenance

Embedding input is limited to 1,024 estimated tokens (4,096 UTF-8 bytes). Oversized
inputs bypass lookup without truncating the user's question. A lookup is skipped if
embedding pricing is unknown, or if the conservative 1,024-token exposure including
a flat request price exceeds $0.001. Every dispatched embedding is separately recorded
through usage accounting, including misses and failed embeddings. Provider token counts
are used when present; otherwise accounting uses the conservative estimate. Embedding
permissions and budget admission apply before dispatch. No warmup generations run.

A hit replays the complete stored JSON bytes with:

```text
X-MultiLLM-Cache: semantic-hit
X-MultiLLM-Cache-Backend: semantic-d1-r2
X-MultiLLM-Usage-Basis: cache-served
X-MultiLLM-Provider-Calls: 0
Age: <seconds>
```

The generation ledger records zero generation cost and cache provenance. The separate
embedding charge still applies. Stored provider token usage remains in the original
JSON response; the provenance headers explain that it describes the stored generation.
`Cache-Control:no-store` bypasses lookup and storage; `no-cache` and the existing
`X-MultiLLM-Cache:refresh` force a generation. `max-age` can further restrict replay age.

## Storage and failures

Apply the additive `0026_semantic_cache.sql` migration before enabling the feature.
D1 holds `semantic_generation_cache`, with scoped digests, a bounded vector, embedding
revision, expiry, integrity hash and body pointer. It scans at most 256 scoped entries
exactly; Vectorize is not used. The private transport returns the four best compatible
candidates to bound response size even for large vectors. A corrupt body among these
candidates can reduce reuse without changing normal dispatch. Complete successful single-choice JSON responses of at
most 1 MiB are stored in the existing `multillm_media` R2 bucket under `semantic-cache/`.
The exact cache's separate capacity policy does not provide semantic oldest-entry
replacement, so semantic bodies use their own prefix.

Entries expire after 300 seconds. D1 transactions evict the oldest entries to keep at
most 256 per principal, across all partitions. Replaced or evicted bodies stay immutable
until expiry so concurrent readers remain safe. Bounded maintenance removes expired
metadata and R2 bodies without touching another prefix. Missing, corrupt, foreign or
expired bodies never replay. Request cancellation and policy changes prevent reuse
or storing an in-flight result. Responses with errors, truncation, tool calls or
`Cache-Control:no-store` are not stored.

Zero-content retention bypasses semantic embedding, lookup and storage entirely.
Retention controls must be enabled for the gateway to honor a zero-retention request.
A missing enabled schema returns JSON HTTP 503 with `error:semantic_cache_schema_missing`
before embedding or generation. Ordinary cache or embedding failures fall through to
normal dispatch once; they never fabricate a success or grant a generation retry.

## Runtime interfaces

The Flask Chat Completions cache decorator handles semantic lookup and storage. Its
fixed private transport uses `/v1/state/semantic-cache` on `intelligence.internal`.
The private Worker state dispatcher must pass validated operations to
`handleSemanticCache`. This private handler is not a public route.

Native Worker integrations use `prepareSemanticCache` after authentication, retention
and policy admission. Call `lookup` before generation and return any resulting response,
including a schema error. Call `store` only after classified successful finalization;
its required `canStore` callback must return `true` only for complete, classified success.
Missing classification prevents storage. A hit uses
`semanticCacheServedEvent` instead of provider generation telemetry. Supply named
`embed`, `embeddingAllowed`, `reserveEmbedding` and `accountEmbedding` collaborators
from the existing configured embedding transport, grant checks, budget reservations
and usage ledger. The embedding transport must honor its abort signal and must not
introduce retries or caller-selected origins. `accountEmbedding` settles the reservation
according to `submission_outcome` (`before-dispatch`, `response` or `unknown`);
unknown submissions retain their hold. All four collaborators are required; a missing
collaborator bypasses lookup. `reserveEmbedding` returns `false` to deny admission,
or a reservation (including `null` for an unbudgeted principal). Embedding and private
storage operations have two-second deadlines in the Worker.
Use `cleanupSemanticCache` in scheduled maintenance. The Worker module does not mount
public routes or automatically select a provider transport.
23 changes: 23 additions & 0 deletions intelligence-migrations/0026_semantic_cache.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
-- Scoped semantic metadata; response content stays in the existing media bucket.
CREATE TABLE IF NOT EXISTS semantic_generation_cache (
principal_hash TEXT NOT NULL,
entry_id TEXT NOT NULL,
partition_hash TEXT NOT NULL,
model_revision TEXT NOT NULL,
vector TEXT NOT NULL,
guard_hash TEXT NOT NULL,
created_at REAL NOT NULL,
expires_at REAL NOT NULL,
body_pointer TEXT NOT NULL,
body_bytes INTEGER NOT NULL CHECK (body_bytes BETWEEN 1 AND 1048576),
body_hash TEXT NOT NULL,
metadata TEXT NOT NULL,
PRIMARY KEY (principal_hash, entry_id),
CHECK (expires_at > created_at AND expires_at <= created_at + 300)
);
CREATE INDEX IF NOT EXISTS semantic_generation_cache_partition
ON semantic_generation_cache (principal_hash, partition_hash, model_revision, expires_at);
CREATE INDEX IF NOT EXISTS semantic_generation_cache_oldest
ON semantic_generation_cache (principal_hash, created_at, entry_id);
CREATE INDEX IF NOT EXISTS semantic_generation_cache_expiry
ON semantic_generation_cache (expires_at);
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_wave4_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
42 changes: 40 additions & 2 deletions routes/chat_cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
from flask import Response, g, request

from route_helpers import request_api_key
from services import cache_policy
from services import cache_policy, semantic_generation_cache as semantic
from services.retention_policy import request_policy
from services.cache_service import ResponseCache, shared_response_cache
from services import shared_generation_cache as shared_cache
Expand Down Expand Up @@ -183,7 +183,7 @@ def cached_chat_completion(view):
"""Wrap an authenticated Chat Completions view; apply it below the auth decorator."""

@wraps(view)
def wrapper(*args, **kwargs):
def exact_wrapper(*args, **kwargs):
retention = request_policy()
mode, max_age = request_mode()
settings = _settings()
Expand Down Expand Up @@ -229,6 +229,44 @@ def wrapper(*args, **kwargs):
_store_if_complete(response, key, settings, shared=shared, identity=identity)
return _shared_headers(response, "miss") if use_shared else _mark(response, "miss")

@wraps(view)
def wrapper(*args, **kwargs):
if not semantic.settings().enabled:
return exact_wrapper(*args, **kwargs)
payload = request.get_json(silent=True)
try:
prepared = semantic.prepare_request(payload, _principal())
except semantic.SemanticCacheSchemaMissing:
return Response(json.dumps({"error": "semantic_cache_schema_missing",
"message": "The semantic generation cache schema is missing. Apply its migration before enabling it."}),
status=503, content_type="application/json")
except Exception:
return exact_wrapper(*args, **kwargs)
if prepared is None:
return exact_wrapper(*args, **kwargs)
mode, max_age = request_mode()
if max_age is None:
match = _MAX_AGE.search(request.headers.get("Cache-Control", "").lower())
max_age = int(match.group(1)) if match else None
if mode != "refresh" and "no-cache" not in request.headers.get("Cache-Control", "").lower():
try:
entry = prepared.lookup()
except semantic.SemanticCacheSchemaMissing:
return Response(json.dumps({"error": "semantic_cache_schema_missing",
"message": "The semantic generation cache schema is missing. Apply its migration before enabling it."}),
status=503, content_type="application/json")
if entry is not None and (max_age is None or entry[2] <= max_age):
return semantic.semantic_hit(entry)
response = exact_wrapper(*args, **kwargs)
if isinstance(response, Response):
prepared.save(response, {
"content_type": "application/json",
"headers": {name: response.headers[name] for name in _STORED_HEADERS if name in response.headers},
"provider": getattr(g, "multillm_provider", None),
"model": getattr(g, "multillm_model", None),
})
return response

return wrapper


Expand Down
12 changes: 12 additions & 0 deletions services/cache_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,3 +157,15 @@ def shared_response_cache():
store = SharedGenerationCache()
current_app.extensions["shared_generation_cache"] = store
return store


def semantic_response_cache():
"""The fixed private semantic store, overridable by an application collaborator."""
from flask import current_app
from services.semantic_generation_cache import SemanticGenerationCache

store = current_app.extensions.get("semantic_generation_cache")
if store is None:
store = SemanticGenerationCache()
current_app.extensions["semantic_generation_cache"] = store
return store
Loading
Loading