Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions docs/zero-content-retention.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Zero-content retention

`CONTENT_RETENTION_ENABLED` defaults to `false`. While off, the retention header
and policy JSON are ignored. Cache, sampling, roleplay, memo and handoff behavior
stays unchanged. An empty flag means off; an empty policy means `{}`.
Malformed settings disable these controls and log a warning once per setting,
without logging its value. Invalid configuration therefore does not provide a
retention guarantee: validate settings before enabling the feature.

When enabled, `CONTENT_RETENTION_POLICY_JSON` accepts this shape:

```json
{
"default": "inherit",
"keys": { "verified-key-id-or-sha256": "zero" },
"routes": { "/v1/chat/completions": "zero" }
}
```

Each value is `inherit` or `zero`; route matching is exact. Key selectors are
server-authenticated account IDs (username when no ID exists) or SHA-256 digests
of the authenticated presented key. Use a key digest for individual keys within
one account. Never put raw keys in the policy. The tightest default, key, route
and header wins. `X-MultiLLM-Retention: zero` tightens the policy;
`X-MultiLLM-Retention: inherit` and unknown header values cannot loosen it.

For example, with the flag off, `X-MultiLLM-Cache: on` still caches an eligible
completion even if the caller sends `X-MultiLLM-Retention: zero`. With the flag
on and policy `{}`, that same retention header bypasses both cache reads and
writes, and bypasses shadow sample construction before a stream collector or
background queue can capture content. With a configured zero key or route,
sending `inherit` still bypasses retention.

Zero roleplay turns use fresh request-local conversation state. Callers must
supply the conversation history needed for each turn. They cannot recover a
durable conversation, import a branch or use operator memory under that policy.
Turn responses expose `X-MultiLLM-Retention: zero` and
`X-MultiLLM-Roleplay-Recovery: unavailable`. Recovery returns HTTP 409 with
`recovery_unavailable`; memory operations return HTTP 409 with
`retention_forbidden`. Errors and completion bytes otherwise keep their existing
meaning. No replacement completion or new retry permission is introduced.

Only numeric roleplay counters, bounded numeric model observations and hashed
request/model identities are written to a separate counter record. Content-free
traces exclude parameter receipts, retain at most 30 turns, and write separately
from existing traces. Zero turns neither populate the conversation cache nor
refresh the existing session deletion alarm. Old content, checkpoints, recovery
records and alarms are left in place; an already scheduled alarm can still run
under its original lifecycle. The feature does not scan or delete stored data.

Knowledge memo lookup, embeddings, bundle construction and background writes are
disabled for zero requests. A derived handoff save returns HTTP 409
`retention_forbidden` before parsing or SQL mutation. It returns no saved ID.
Explicit source-corpus ingestion remains separately authorized and is unaffected.

The gateway must still hold and send content while serving a request, including
streaming and existing turn scheduling. Usage, cost, security and health counters
continue; provider calls can incur their normal cost. This is a prospective
gateway storage policy, not a provider-side retention agreement, secure memory
erasure, deletion API or prohibition on caller-authorized corpus storage. It does
not change provider prompt-cache settings or cover media storage and future
state/batch features; those features must adopt the same policy explicitly.

## Where the policy applies

- Flask resolves the policy after authentication and before cache and dispatch
hooks, and exposes it as `g.multillm_retention_policy`. The chat response cache
and shadow sampling check it before reading, writing or queueing content.
- The Worker resolves the policy only from the verified key identity and the
public request path. It ignores identity in caller payloads and caller-set
`X-MultiLLM-Retention-Key-ID`, `X-MultiLLM-Retention-Key-Hash` and
`X-MultiLLM-Retention-Route` headers, and sets those headers itself for
roleplay turns, operator recovery and memory requests.
- Knowledge retrieval uses a non-storing cache for zero requests. Handoff saves
are rejected before content is serialized into a Durable Object call.
- Set `CONTENT_RETENTION_ENABLED` and `CONTENT_RETENTION_POLICY_JSON` on both the
Worker and the Container. No routes, D1 tables or migrations are added.
- Browser clients can send `X-MultiLLM-Retention` and read
`X-MultiLLM-Retention` and `X-MultiLLM-Roleplay-Recovery`.

Any feature that stores content must check the same policy before it builds a
record, queues content or persists it.

Tests use fake providers and storage. They do not verify a deployment or
provider-side retention.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_zero_content_retention_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
4 changes: 2 additions & 2 deletions route_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

CORS_ALLOWED_METHODS = "GET, POST, PUT, DELETE, PATCH, OPTIONS"
CORS_DEFAULT_HEADERS = (
"Authorization, Content-Type, Accept, Origin, X-Requested-With, X-MultiLLM-Tool-Repair, X-MultiLLM-Conversion-Report, "
"Authorization, Content-Type, Accept, Origin, X-Requested-With, X-MultiLLM-Tool-Repair, X-MultiLLM-Conversion-Report, X-MultiLLM-Retention, "
"X-Api-Key, X-Goog-Api-Key, X-MultiLLM-Api-Key, Anthropic-Version, "
"Anthropic-Beta, Anthropic-Dangerous-Direct-Browser-Access, "
"Idempotency-Key, X-MultiLLM-Image-QA, X-MultiLLM-Cascade, OpenAI-Beta, OpenAI-Organization, "
Expand All @@ -50,7 +50,7 @@
"X-MultiLLM-Auto-Route, X-MultiLLM-Auto-Selected-Model, "
"X-MultiLLM-Auto-Attempts, X-MultiLLM-Auto-Selected-Priority, "
"X-MultiLLM-Auto-Ordering, X-MultiLLM-Auto-Failover-Reasons, "
"X-MultiLLM-Cache, X-MultiLLM-Transport-Failure, X-MultiLLM-Token-Count, "
"X-MultiLLM-Cache, X-MultiLLM-Retention, X-MultiLLM-Roleplay-Recovery, X-MultiLLM-Transport-Failure, X-MultiLLM-Token-Count, "
"X-MultiLLM-Latency-Ms, X-MultiLLM-Estimated-Cost-USD, "
"X-MultiLLM-Cost-Basis, WWW-Authenticate, X-PAYMENT-RESPONSE, X-Poll-After, "
"X-NanoGPT-Advisor-ID, X-NanoGPT-Data-Endpoint, "
Expand Down
4 changes: 4 additions & 0 deletions routes/chat_cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@

from route_helpers import request_api_key
from services import cache_policy
from services.retention_policy import request_policy
from services.cache_service import ResponseCache

CACHE_HEADER = "X-MultiLLM-Cache"
Expand Down Expand Up @@ -166,10 +167,13 @@ def cached_chat_completion(view):

@wraps(view)
def wrapper(*args, **kwargs):
retention = request_policy()
mode, max_age = request_mode()
settings = _settings()
if mode is None or request.method != "POST":
return view(*args, **kwargs)
if not retention.allows_content:
return _mark(view(*args, **kwargs), "bypass")
payload = request.get_json(silent=True)
principal = _principal()
if not settings["enabled"] or principal is None or not request_is_cacheable(payload):
Expand Down
94 changes: 94 additions & 0 deletions services/retention_policy.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
"""Immutable, prospective retention decisions resolved after authentication."""
from __future__ import annotations

import hashlib
import json
import logging
import os
from dataclasses import dataclass
from functools import lru_cache
from typing import Mapping

from flask import g, has_request_context, request

HEADER = "X-MultiLLM-Retention"
_MODES = {"inherit", "zero"}
logger = logging.getLogger(__name__)


@dataclass(frozen=True)
class RetentionPolicy:
mode: str = "inherit"
enabled: bool = False
revision: str = "legacy"

@property
def allows_content(self) -> bool:
return not self.enabled or self.mode != "zero"


@lru_cache(maxsize=2)
def _warn_once(setting: str) -> None:
logger.warning("Invalid %s; content retention controls disabled", setting)


def _rules(value: object) -> bool:
if not isinstance(value, dict):
return False
return all(isinstance(key, str) and bool(key) and isinstance(mode, str) and mode in _MODES
for key, mode in value.items())


def resolve_policy(env: Mapping[str, str] | None = None, *, key_id: str = "",
route: str = "", header: str = "", key_hash: str = "") -> RetentionPolicy:
"""Tightest of default, authenticated key, exact route and caller opt-out."""
env = os.environ if env is None else env
flag = str(env.get("CONTENT_RETENTION_ENABLED", "")).strip().lower()
if flag in {"", "0", "false", "no", "off"}:
return RetentionPolicy()
if flag not in {"1", "true", "yes", "on"}:
_warn_once("CONTENT_RETENTION_ENABLED")
return RetentionPolicy()
raw = env.get("CONTENT_RETENTION_POLICY_JSON", "").strip() or "{}"
try:
config = json.loads(raw)
if (not isinstance(config, dict) or set(config) - {"default", "keys", "routes"}
or not isinstance(config.get("default", "inherit"), str)
or config.get("default", "inherit") not in _MODES
or not _rules(config.get("keys", {})) or not _rules(config.get("routes", {}))):
raise ValueError("Invalid retention configuration")
except (ValueError, TypeError):
_warn_once("CONTENT_RETENTION_POLICY_JSON")
return RetentionPolicy()
modes = [config.get("default", "inherit"), config.get("routes", {}).get(route),
config.get("keys", {}).get(key_id), config.get("keys", {}).get(key_hash),
str(header).strip().lower()]
revision = hashlib.sha256(json.dumps(config, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
return RetentionPolicy("zero" if "zero" in modes else "inherit", True, revision)


def request_policy() -> RetentionPolicy:
"""Snapshot once per request; stream callbacks must capture the returned value."""
if not has_request_context():
return RetentionPolicy()
captured = getattr(g, "multillm_content_retention", None)
if isinstance(captured, RetentionPolicy):
return captured
user = getattr(g, "authenticated_user", None) or {}
from route_helpers import request_api_key
key = request_api_key() if user else ""
key_hash = hashlib.sha256(key.encode()).hexdigest() if key else ""
policy = resolve_policy(key_id=str(user.get("id") or user.get("username") or ""),
key_hash=key_hash, route=request.path, header=request.headers.get(HEADER, ""))
g.multillm_content_retention = policy
if policy.enabled:
g.multillm_retention_policy = {"mode": policy.mode, "revision": policy.revision}
Comment on lines +84 to +85

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep inherited retention compatible with the v1 cache

If CONTENT_RETENTION_ENABLED=true is combined with RESPONSE_CACHE_POLICY_REVISION=v1, even an effective inherit policy disables every otherwise eligible cache request. This assignment replaces the legacy snapshot with {mode: "inherit", ...}, while services/cache_policy._policy_snapshot accepts only mode === "legacy" and therefore returns None; the wrapper consequently bypasses both reads and writes. Extend the isolated-cache policy contract to represent enabled inherited retention rather than treating it as unavailable.

Useful? React with 👍 / 👎.

return policy


def init_retention_policy(app) -> None:
"""Coordinator hook: call after authentication and before managed dispatch."""
@app.before_request
def capture_retention():
if getattr(g, "authenticated_user", None):
request_policy()
21 changes: 17 additions & 4 deletions services/shadow_eval_sampling.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

from services.shadow_eval_contract import ANSWER_BYTES, REQUEST_BYTES, REQUEST_FIELDS, clean_usage, eligible, encoded, make_sample
from services.shadow_eval_store import ShadowEvalStore
from services.retention_policy import RetentionPolicy, request_policy

logger = logging.getLogger(__name__)
_QUEUE = queue.Queue(maxsize=32)
Expand Down Expand Up @@ -38,16 +39,16 @@ class CaptureOversize(ValueError):
pass


def submit(sample):
def submit(sample, *, retention_policy=None):
global _WORKER
if sample is None:
if sample is None or retention_policy is not None and not retention_policy.allows_content:
return
with _LOCK:
if _WORKER is None or not _WORKER.is_alive():
_WORKER = threading.Thread(target=_persist, name="shadow-samples", daemon=True)
_WORKER.start()
try:
_QUEUE.put_nowait(sample)
_QUEUE.put_nowait((sample, retention_policy) if retention_policy is not None else sample)
count("sampled")
except queue.Full:
count("skipped_queue_full")
Expand All @@ -57,6 +58,10 @@ def _persist():
while True:
sample = _QUEUE.get()
try:
if isinstance(sample, tuple) and isinstance(sample[1], RetentionPolicy):
sample, retention = sample
if not retention.allows_content:
continue
ShadowEvalStore.put(sample)
except Exception as error:
count("skipped_error")
Expand Down Expand Up @@ -201,6 +206,9 @@ def sample_success(response, payload=None):
if (response.status_code >= 400 or request.method != "POST" or getattr(g, "shadow_eval_internal", False)
or getattr(g, "gateway_subrequest", False)):
return response
retention = request_policy()
if not retention.allows_content:
return response
payload = payload if payload is not None else request.get_json(silent=True)
selected = _sampling_decision(payload)
if selected is None:
Expand All @@ -211,9 +219,14 @@ def sample_success(response, payload=None):
model = response.headers.get("X-MultiLLM-Auto-Selected-Model") or getattr(g, "multillm_model", None)

def capture(answer, selected, usage, finish_reason=None):
if not retention.allows_content:
return
sample = make_sample(payload, user, route, answer, selected,
(time.monotonic() - started) * 1000, usage, finish_reason=finish_reason, on_skip=count)
submit(sample)
if retention.enabled:
submit(sample, retention_policy=retention)
else:
submit(sample)

if response.is_streamed:
if response.mimetype == "text/event-stream":
Expand Down
1 change: 1 addition & 0 deletions tests/fixtures/secret_firewall_coverage.json
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,7 @@
"worker/opencode/reasoning-response.mjs",
"worker/prometheus.mjs",
"worker/prompt-templates-d1.mjs",
"worker/retention-policy.mjs",
"worker/roleplay/candidate-context.mjs",
"worker/roleplay/capacity.mjs",
"worker/roleplay/checkpoint.mjs",
Expand Down
Loading
Loading