Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions docs/context-pages.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Reversible context paging

Context paging replaces complete older conversation exchanges with retrieval handles.
It preserves the exact JSON body of each stored exchange, including whitespace and
Unicode in message content, tool-call IDs, arguments, and tool results. It does not
summarize history or invent a tool call. System and developer messages, incomplete
exchanges, explicitly protected indices, and the latest user exchange remain in full.

Paging is off by default. Leave `CONTEXT_PAGING_ENABLED` unset, empty, or `false` to
keep existing traffic and storage unchanged. Invalid flag values disable paging and
produce one warning containing only the setting name. Raw forwarding is unchanged.

## Enable for a managed client

Apply the context-pages migration before setting `CONTEXT_PAGING_ENABLED=true`.
The deployment must expose the authenticated context retrieval route and private
storage operation, and supply verified principal, session, current policy revision,
current chat/tool grants, and retention decisions to the managed dispatch path.
The same existing `multillm_media` R2 bucket holds bodies under `context-pages/`;
`INTELLIGENCE_DB.context_pages` holds metadata. There are no additional bindings.
The existing media signing secret (or Flask signing secret) signs page handles.

A managed client explicitly declares retrieval capability:

```json
{
"model": "provider:explicit-model",
"capabilities": ["multillm_context_retrieve"],
"messages": [
{"role": "user", "content": "Older complete exchange"},
{"role": "assistant", "content": "Older answer"},
{"role": "user", "content": "Current request"}
]
}
```

Only exchanges that need paging to fit the candidate's input window are stored.
The upstream messages contain a handle marker instead of the old group, plus the
`multillm_context_retrieve` function schema. Page metadata returned to the client is
`{page_id, sha256, expires_at, tool_schema}`; expiry is a Unix timestamp. The managed
response exposes this as `context_pages`. No body or base64 blob is attached to the
upstream generation request. The explicit model and output reservation are preserved.

Retrieve only when needed by calling `multillm_context_retrieve({"page_id":"cp_..."})`
or making an authenticated `GET /v1/context/pages/cp_...`. The caller's current grants
and policy revision must still authorize access; a handle alone does not grant access.
The route returns `page_id`, `sha256`, `expires_at`, `messages`, and `body_base64`.
`body_base64` decodes to the exact UTF-8 bytes that were hashed and stored. Responses
use `Cache-Control: private, no-store`. Retrieving does not dispatch a provider call;
the client chooses whether to include restored content in a later generation.

With the flag off, the same request keeps its original history and does not write
pages. A client without the capability also keeps its existing behavior. Zero-content
retention disables paging before any storage call and denies retrieval of existing
pages for that request. Changing policy does not delete earlier retained objects.

## Bounds and errors

- Each page holds at most 64 KiB; active pages hold at most 1 MiB per principal/session.
The D1 limit check and all metadata inserts run in one statement, so competing
writers cannot exceed the session allowance. Candidate plans reuse identical pages
within one request.
- Pages expire after 3,600 seconds. Every retrieval checks expiry, ownership, session,
current revision, the HMAC, byte length, and SHA-256 before exposing content.
- HTTP 413 `context_window_exceeded` means protected content and handles cannot fit
the candidate window after reserving output and safety tokens. Nothing is silently
truncated. Oversized pages/session content also return 413.
- HTTP 404 covers disabled retrieval, absent, expired, foreign, or stale-revision
handles. HTTP 403 covers missing current grants or zero retention.
- HTTP 503 JSON reports unavailable D1 tables, R2 bindings, signing configuration,
authority, or storage, and corrupted bodies. Generation must stop before dispatch.
Failed storage is never permission to truncate, summarize, or replay a generation.

Token estimates use serialized UTF-8 byte counts, not a provider tokenizer. Retrieval
can increase a later request's input cost; paging does not authorize a model or budget
change. Configure an R2 lifecycle rule for `context-pages/` according to retention
policy: expiry denies access immediately, but physical storage cleanup follows that
rule. D1 metadata may be retained for operations; expired metadata does not count
against the active session allowance. No prompt or retrieved body is logged.
13 changes: 13 additions & 0 deletions intelligence-migrations/0024_context_pages.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
-- Page bodies live only in the context-pages/ prefix of multillm_media.
CREATE TABLE IF NOT EXISTS context_pages (
page_id TEXT PRIMARY KEY,
principal TEXT NOT NULL,
session_hash TEXT NOT NULL,
revision TEXT NOT NULL,
sha256 TEXT NOT NULL,
byte_length INTEGER NOT NULL CHECK (byte_length > 0 AND byte_length <= 65536),
expires_at INTEGER NOT NULL,
r2_key TEXT NOT NULL UNIQUE
);
CREATE INDEX IF NOT EXISTS context_pages_session_expiry
ON context_pages (principal, session_hash, expires_at);
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_context_pages_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
34 changes: 34 additions & 0 deletions routes/context_pages.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
"""Registration boundary for authenticated context retrieval."""
from __future__ import annotations

from flask import jsonify

from services.context_pages import ContextPageError, ContextPageService, paging_enabled


def register_context_page_routes(app, *, service: ContextPageService | None = None, authorize=None) -> None:
"""authorize resolves current principal, session, revision, chat grant and retention.

The callback must authenticate the request; callers never select authority through
URL/query fields. An absent authority or storage collaborator fails closed.
"""
service = service or ContextPageService()

@app.route("/v1/context/pages/<page_id>", methods=["GET"])
def retrieve_context_page(page_id):
try:
if not paging_enabled():
raise ContextPageError("context_page_not_found", 404)
if authorize is None:
raise ContextPageError("context_paging_authority_unavailable")
authority = authorize()
if not isinstance(authority, dict) or not {"scope", "retention_policy", "granted"} <= set(authority):
raise ContextPageError("context_paging_authority_unavailable")
result = service.retrieve(page_id=page_id, scope=authority["scope"],
retention_policy=authority["retention_policy"], granted=authority["granted"] is True)
response = jsonify(result)
except ContextPageError as error:
response = jsonify({"error": error.code, "message": "Context page retrieval could not be completed."})
response.status_code = error.status
response.headers["Cache-Control"] = "private, no-store"
return response
20 changes: 20 additions & 0 deletions services/context_optimizer.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from typing import Any

from services.context_analysis_cache import ContextAnalysisCache, ImagePromptSpan
from services.context_pages import PagingRequest

EARLIER_IMAGE_PROMPT_PLACEHOLDER = (
"[Earlier image-generation prompt omitted by MultiLLM context optimizer; "
Expand Down Expand Up @@ -127,6 +128,7 @@ class ContextOptimizationResult:
reasons: tuple[str, ...]
analysis_cache_hits: int
analysis_cache_misses: int
pages: tuple[dict[str, Any], ...] = ()


def estimate_payload_tokens(payload: Mapping[str, Any]) -> int:
Expand Down Expand Up @@ -478,6 +480,7 @@ def optimize_chat_payload(
default_target_tokens: int,
summary_digest: Mapping[str, list[str]] | None = None,
defer_required_target: bool = False,
context_paging: PagingRequest | None = None,
) -> ContextOptimizationResult:
if not isinstance(payload, Mapping):
raise ContextOptimizationError("Request body must be a JSON object")
Expand All @@ -491,6 +494,23 @@ def optimize_chat_payload(
)
optimized_payload = copy.deepcopy(dict(payload))
optimized_payload.pop("optimization", None)
if context_paging is not None and context_paging.eligible():
before = estimate_payload_tokens(optimized_payload)
paged = context_paging.service.page_payload(
optimized_payload, scope=context_paging.scope,
capabilities=context_paging.capabilities,
target_input_tokens=options.target_input_tokens,
retention_policy=context_paging.retention_policy, managed=context_paging.managed,
protected_indices=options.preserve_message_indices,
)
after = estimate_payload_tokens(paged.payload)
return ContextOptimizationResult(
payload=dict(paged.payload), options=options, status="paged" if paged.pages else "unchanged",
estimated_input_before=before, estimated_input_after=after,
image_prompts_compacted=0, messages_summarized=0, summary_source_messages=(),
needs_summary=False, target_met=after <= options.target_input_tokens,
reasons=(), analysis_cache_hits=0, analysis_cache_misses=0, pages=paged.pages,
)
messages = optimized_payload["messages"]
before_tokens = estimate_payload_tokens(optimized_payload)
triggered = before_tokens > options.trigger_input_tokens
Expand Down
Loading
Loading