Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions app.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
is_api_request_path,
login_required,
)
from routes.config_snapshots import register_config_snapshot_routes
from routes.core import register_core_routes
from routes.dashboard_security import register_dashboard_security_routes
from routes.intelligence_media import register_intelligence_media_routes
Expand Down Expand Up @@ -139,6 +140,7 @@ def create_app() -> Flask:
register_core_routes(app)
register_dashboard_security_routes(app)
register_workbench_routes(app)
register_config_snapshot_routes(app)
register_shadow_eval_routes(app, csrf, AuthService, MetricsService, ProxyService)
register_knowledge_routes(app, csrf)
register_gateway_mcp_routes(app, csrf)
Expand Down
103 changes: 103 additions & 0 deletions docs/config-snapshots.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Control configuration snapshots

`CONFIG_SNAPSHOTS_ENABLED` defaults to false. All snapshot paths return 404
while disabled, and existing auto-route saves keep their current payload,
storage and response behavior. No provider calls or generation are involved.

The public administration paths run in Flask for both direct administration
and Worker-forwarded Container traffic. They use the existing session login,
administrator check and CSRF protection. They are not gateway-key API routes.
The existing private `auto_routes` adapter stores configuration in D1; there is
no new public Worker dispatch or private outbound endpoint.

Before enabling, apply `intelligence-migrations/0015_config_snapshots.sql`
(`npx wrangler d1 migrations apply multillm-intelligence --remote`), and configure
`INTELLIGENCE_STORAGE_BACKEND=d1` for Flask. Enable the flag on both Worker
and Container. Enabled snapshots without durable D1 or any required table
fail closed with JSON HTTP 503. Enabled ordinary D1 route saves also fail
closed if the snapshot migration is missing. Local SQLite/PostgreSQL snapshot
storage is outside this initial private-D1 implementation boundary.

## Review and apply

`GET /admin/config/snapshots` returns `current_revision`, snapshot metadata,
and the latest 100 application audit records. It never includes stored
configuration documents or account objects. The only supported domain is
`auto_routes`; its revision begins at zero when the feature is introduced.

Create a review with `POST /admin/config/snapshots`:

```json
{"domain":"auto_routes","base_revision":0,"configuration":{"routes":[{"route_id":"auto:review","candidates":["openai:model-a","nanogpt:model-b"]}]}}
```

HTTP 201 returns the immutable snapshot metadata and ID. Creation requires
that `base_revision` still matches; it does not activate any route.
Configuration is a patch containing only the listed route IDs and their
ordered candidate model IDs. Omitted routes and seeded defaults remain active.
No deletion or wholesale replacement is performed.

`GET /admin/config/snapshots/<id>/diff` returns only route IDs and the approved
before/after model ID lists, plus the base and current revisions. Flask displays
the effective seeded defaults when there is no durable override. Diff pages
contain at most 20 routes; use `?offset=<next_offset>` until `next_offset` is
null. A review spanning pages must be restarted if `current_revision` changes.
An unknown snapshot returns 404.

Apply with `POST /admin/config/snapshots/<id>/apply`:

```json
{"current_revision":0,"confirm":true}
```

Both literal `confirm=true` and the current integer revision are required.
A mismatch, stale review, repeat application, or patch that would exceed the
existing 200-row durable route read bound returns 409. The snapshot base
must equal the submitted revision. The transactional D1 batch compares the
revision and exact durable route state, writes the reviewed routes, increments
the revision, and appends an audit record with an opaque administrator digest.
A route write failure rolls back the revision, audit and route updates together.
Ordinary enabled D1 route saves advance the same revision. A fingerprint also
rejects a review if routes were saved while the flag was temporarily disabled.

A successful apply clears this Container’s route cache. Other Containers can
retain their last-good route copy for the existing 30-second cache interval;
an outage preserves their last-good routing. There is no cluster-wide immediate
cache invalidation. Seeded candidate orders retired by the existing auto-route
service are rejected at creation, rather than claiming an apply that the route
service would subsequently replace with current defaults.

To roll back, create a new snapshot containing the previous candidate order
at the current revision, review its diff and explicitly apply it. Historical
snapshots and application records are retained; rollback adds a revision.

## Bounds, secrets and errors

There are at most 100 snapshots per domain. Each configuration is bounded to
256 KiB, 200 routes and 16 distinct candidates per route; model IDs are bounded
to 256 characters. The existing private transport also bounds its complete
request envelope to 256 KiB, so configurations at that boundary require room
for metadata. Oversized requests return 413. Count exhaustion or a concurrent
revision change at creation returns 409. No automatic archival or deletion is
performed and there is no delete endpoint: once a domain has 100 snapshots,
creation returns 409 until unapplied snapshots are removed from D1 (applied
ones are referenced by their audit records). Application records are append-only and grow with
successful applies; list responses are bounded to the latest 100 records.

Only the exact route configuration schema is accepted. Provider keys, gateway
keys, headers, prompts, connections, environment fields and arbitrary metadata
are rejected, not redacted into storage. Known credential-shaped identifiers
are also rejected. Snapshots contain configuration, a durable-state digest and
opaque administrator attribution only; no runtime account or environment
objects are read into snapshot documents.

Malformed configuration returns 400, non-admin sessions return 403, and
unauthenticated sessions follow the existing login flow. Every snapshot response
uses `Cache-Control: no-store`. Storage failures return JSON 503 with
`config_snapshot_storage_unavailable`; they do not claim a completed apply.
Private submissions are never retried. If a response is lost after submission,
check the current revision and audit records before making a new review.

Snapshot operations consume D1 reads/writes and bounded private transport
capacity. They make no upstream generation calls, but D1 usage can have storage
and request costs.
26 changes: 26 additions & 0 deletions intelligence-migrations/0015_config_snapshots.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
-- Reviewed configuration only. No keys, headers, connection profiles or request content.
CREATE TABLE IF NOT EXISTS config_snapshot_revisions (
domain TEXT PRIMARY KEY CHECK(domain = 'auto_routes'),
revision INTEGER NOT NULL CHECK(revision >= 0 AND revision <= 9007199254740991)
);
CREATE TABLE IF NOT EXISTS config_snapshots (
id TEXT PRIMARY KEY,
domain TEXT NOT NULL CHECK(domain = 'auto_routes'),
base_revision INTEGER NOT NULL CHECK(base_revision >= 0),
configuration TEXT NOT NULL CHECK(json_valid(configuration)),
base_fingerprint TEXT NOT NULL CHECK(length(base_fingerprint) = 64),
created_at TEXT NOT NULL,
created_by TEXT NOT NULL,
size_bytes INTEGER NOT NULL CHECK(size_bytes > 0 AND size_bytes <= 262144)
);
CREATE INDEX IF NOT EXISTS config_snapshots_domain_created ON config_snapshots(domain, created_at, id);
CREATE TABLE IF NOT EXISTS config_snapshot_applications (
id TEXT PRIMARY KEY,
domain TEXT NOT NULL CHECK(domain = 'auto_routes'),
snapshot_id TEXT NOT NULL REFERENCES config_snapshots(id),
base_revision INTEGER NOT NULL,
revision INTEGER NOT NULL,
applied_at TEXT NOT NULL,
applied_by TEXT NOT NULL,
UNIQUE(domain, revision)
);
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prompt_templates_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
58 changes: 58 additions & 0 deletions routes/config_snapshots.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
"""Administrator-only snapshot review; existing session and CSRF protections apply."""
from functools import wraps

from flask import jsonify, request

from error_handlers import APIError

from route_helpers import login_required
from routes.core import require_admin_dashboard_user
from services import config_snapshots


def json_errors(view):
@wraps(view)
def wrapped(*args, **kwargs):
try:
return view(*args, **kwargs)
except APIError as error:
return jsonify(error.to_dict()), error.status_code
return wrapped


def register_config_snapshot_routes(app):
def disabled_config_snapshots():
if (request.path == "/admin/config/snapshots" or request.path.startswith("/admin/config/snapshots/")) and not config_snapshots.enabled():
return jsonify({"error": "not_found"}), 404

# Feature-off requests must be 404 even before session and CSRF checks.
app.before_request_funcs.setdefault(None, []).insert(0, disabled_config_snapshots)

@app.after_request
def private_config_snapshots(response):
if request.path == "/admin/config/snapshots" or request.path.startswith("/admin/config/snapshots/"):
response.headers["Cache-Control"] = "no-store"
return response

@app.route("/admin/config/snapshots", methods=["GET", "POST"])
@json_errors
@login_required
def admin_config_snapshots():
user = require_admin_dashboard_user()
if request.method == "GET":
return jsonify(config_snapshots.call("snapshot_list"))
return jsonify(config_snapshots.create(request.get_json(silent=True), user, app.config["API_BASE_URLS"])), 201

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enforce the snapshot byte limit before JSON parsing

The create handler reads and parses the entire request before installing any snapshot-specific body limit, so it inherits the application's 32 MiB default rather than the advertised 256 KiB bound. An authenticated request near that limit is materialized by get_json, then serialized and encoded again by validate_configuration before it can return 413, which can consume several times the request size per worker; set request.max_content_length in the early snapshot gate, as the prompt-template routes do, so oversized snapshots are rejected before parsing.

Useful? React with 👍 / 👎.


@app.get("/admin/config/snapshots/<identifier>/diff")
@json_errors
@login_required
def admin_config_snapshot_diff(identifier):
require_admin_dashboard_user()
return jsonify(config_snapshots.diff(identifier, request.args.get("offset", "0")))

@app.post("/admin/config/snapshots/<identifier>/apply")
@json_errors
@login_required
def admin_config_snapshot_apply(identifier):
user = require_admin_dashboard_user()
return jsonify(config_snapshots.apply(identifier, request.get_json(silent=True), user))
21 changes: 21 additions & 0 deletions services/auto_route_d1.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,24 @@ def save_route(route_id, candidates, updated_at):
def reset_cache():
with _lock:
_cache["routes"], _cache["expires"] = None, 0.0


def snapshot_request(operation, **values):
"""Snapshot operations fail closed, without the routing read cache or a retry."""
try:
if not using_d1():
raise ValueError("Durable snapshots require D1")
result = intelligence_d1_store.request_private_intelligence(
{"operation": operation, **values}, endpoint="auto_routes")
if (not isinstance(result, dict) or type(result.get("version")) is not int
or result["version"] != 1 or "error" in result):
raise ValueError("Invalid snapshot storage response")
return result
except intelligence_d1_store.PrivateIntelligenceError as error:
if error.status in {400, 404, 409, 413}:
raise APIError("Snapshot request was rejected", error.status,
{"error": error.code}) from None
except Exception:
pass
raise APIError("Snapshot storage is unavailable; apply was not confirmed", 503,
{"error": "config_snapshot_storage_unavailable"}) from None
Loading
Loading