Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions app.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
from services.cache_service import CacheService
from services.image_relay_catalog import ImageRelayCatalogRefresh
from services.provider_catalog_refresh import ProviderCatalogAutoRefresh
from services.gateway_extensions import register_gateway_extensions
from services.metrics_service import MetricsService
from services.proxy_service import ProxyService
from services import usage_ledger
Expand Down Expand Up @@ -146,6 +147,7 @@ def create_app() -> Flask:
register_gateway_mcp_routes(app, csrf)
register_documentation_routes(app, AuthService, ProxyService)
register_usage_routes(app, csrf)
register_gateway_extensions(app)
# Restore the dashboard's recent requests from the durable ledger in the background.
usage_ledger.start(MetricsService.get_instance())

Expand Down
116 changes: 116 additions & 0 deletions docs/config-revision-sync.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# Revision-aware live configuration sync

Configuration revisions are committed counters, not a push consistency guarantee. The
feature is off by default. Disabled startup does not register a poller or a freshness
guard, and existing reads, saves and upstream catalog refresh timing stay unchanged.
No disabled operation reads or writes `control_revisions`.

```text
CONFIG_REVISION_SYNC_ENABLED=false
```

Apply migration `0017_control_revisions.sql` through the separately approved migration
process, after the config snapshot and usage bucket migrations (`0015` and `0016`). A
code deploy does not apply D1 migrations. Enable on the Worker and its Containers:

```text
CONFIG_REVISION_SYNC_ENABLED=true
CONFIG_SYNC_TTL_SECONDS=30
CONFIG_SECURITY_TTL_SECONDS=5
INTELLIGENCE_STORAGE_BACKEND=d1
AUTH_STORAGE_BACKEND=d1
```

Flask sync requires Container D1 control and account storage. Standalone SQL/SQLite
mode and unsupported storage combinations disable the feature and log one warning,
without changing existing authentication. This does not fall back from an unavailable
D1 authority: configured D1 with a missing migration or an outage still fails closed.
Storage selection is a startup setting; restart Containers after changing it.

Empty values mean defaults. Malformed settings disable sync and log the variable name
once, without its value. Ordinary TTL accepts 1–3600 seconds. Security TTL accepts
1–5 seconds; a longer value cannot extend the security safety bound.

The existing control-state scheduler polls revisions with an interval between 80% and
100% of the TTL. Each consumer permits one poll at a time and failures keep the same
bounded schedule. A revision check costs one private D1 call per due group. Changed
domains additionally load their stored copy and verify the revision again. Revisions
that decrease, failed loads, and writes racing a load cannot certify freshness. A copy
loads at startup and thereafter only when a newer committed revision is observed.
Scheduler delay and authority latency can cause a temporary fail-closed response.

`GET /admin/config/revisions` requires a dashboard administrator when enabled and is
404 when disabled. It returns only domain names, revision numbers, age, stale flags,
security classification and the time until the next poll. It contains no keys, model
lists, prompts, responses or provider payloads. Ordinary auto routes and provider
catalogs retain the last installed copy during an outage and report `stale: true`.
Before the first route load, seeded routes remain available. New processes cannot
recover an ordinary last-good copy while its authority is unavailable.

API and MCP dispatch fails with HTTP 503 and `config_security_stale` while security
metadata has never been verified, a newer security revision cannot be installed, or
the last verification is older than the security TTL. Admin recovery and status remain
accessible; preflight does not dispatch. Enabled revision operations with a missing
table return a content-free JSON 503. There is no automatic policy change, inference
retry, or success response substituted for a failure.

## Committed domains

`auto_routes` retains the config snapshots counter in `config_snapshot_revisions`; it
has no second row in `control_revisions`. The private auto-route wrapper reuses atomic
snapshot saves for normal writes without enabling snapshot APIs. Snapshot apply
retains CAS and bumps exactly once. Model overrides and provider catalog writes update
their domain counter in the same D1 transaction as their data.

Account upserts and deletes atomically increment both `key_controls` and `model_grants`
with the account mutation and its audit record. Upserts replace the full account,
covering revocation, rotation, scopes, model allowlists, budgets, expiry and address
ranges; both counters advance conservatively even for an unchanged replacement or a
missing-account delete. Usage-only touches and refused account writes do not advance
security revisions. With the feature off, existing writes keep their original SQL and
responses, including compatibility reads for older account schemas.

Flask installs strict account refreshers for both security domains by default. Each
reads all account pages from the private D1 authority using the full control schema,
validates the records, clears the verified-key memo, and installs the account and grant
copy. Invalid records or incomplete schemas raise; there is no fallback to missing
controls or cached grants. Authentication uses this installed copy, serialized with
refreshes, so a revoked, deleted or rotated key is rejected after the successful poll
without waiting for the existing 60-second memo TTL. Model grants and scopes come from
the same installed records. Budgets, model allowlists, expiry and address restrictions
remain enforced by their existing request policies. Environment bootstrap admin and
integration-key authentication retain their existing authorities; integration keys are
verified against their private authority on each request.

Both domains currently load the shared account table separately. Each refresh costs
one private call per page of up to 200 accounts plus the group's revision confirmation;
new revisions clear all verified-key memos in that process. Account copies remain in
process memory only. The revision table holds one counter and timestamp per domain,
with no event history or request content. Existing account audit, snapshot history and
catalog retention policies remain unchanged.

## Runtime interfaces

`register_gateway_extensions` mounts the Flask guard and status route. Other middleware
can be supplied as explicit callbacks in policy order; there are no dynamic imports or
plugin discovery. `configure_sync` supplies the model-override and both account/grant
installers automatically. An explicit `security_refreshers` override must load from
its committed authority or raise; an absent installer remains fail closed.

Private auto-route dispatch can call
`handleRevisionedAutoRoutes(request, env, {handleAutoRoutesRequest, boundedBody})` using
its static handler and bounded body reader. Revision persistence has no dependency on
the account or auto-route handler. `commitRevision(db, ["key_controls", "model_grants"],
statements)` accepts trusted fixed D1 statements, never SQL from a request. Its optional
expected revision permits one concurrent CAS winner; a failed CAS rolls back all data,
audit and revision statements.

Native Worker dispatch can supply strict installers to `RevisionConsumer`, invoke
`tick` through its background lifecycle, and call `requireFreshSecurity` before dispatch.
The guard returns a JSON 503 response or null when freshness permits normal dispatch.
Read callbacks return an exact domain-to-integer mapping; installer callbacks throw on
authority failure. The native consumer has no default account/grant installer: a missing
callback is its fail-closed fallback, equivalent to Flask's `missing_security_refresh`
for an explicitly absent installer. Native status can use the content-free `status`
method. Polling does not guarantee immediate cross-region visibility or live deployment
health.
6 changes: 6 additions & 0 deletions intelligence-migrations/0017_control_revisions.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
-- Content-free counters. auto_routes retains its W13 counter for snapshot CAS.
CREATE TABLE IF NOT EXISTS control_revisions (
domain TEXT PRIMARY KEY CHECK(domain IN ('model_overrides', 'provider_catalog', 'key_controls', 'model_grants')),
revision INTEGER NOT NULL CONSTRAINT revision_cas_confirmed CHECK(revision >= 0 AND revision <= 9007199254740991),
updated_at TEXT NOT NULL
);
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_prompt_cache_cost_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_prompt_cache_cost_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand All @@ -18,7 +18,7 @@
"cf:dry-run": "wrangler deploy --dry-run",
"cf:tail": "wrangler tail",
"test:knowledge": "node --test tests/test_knowledge_*.mjs",
"test:integration": "node --test tests/test_d1_integration.mjs",
"test:integration": "node --test tests/test_d1_integration.mjs tests/test_config_revision_sync_integration.mjs",
"cf:knowledge:dry-run": "wrangler deploy --config wrangler.knowledge.jsonc --dry-run",
"cf:knowledge:deploy": "wrangler deploy --config wrangler.knowledge.jsonc",
"cf:migrations:check": "node scripts/verify_d1_migrations.mjs --check",
Expand Down
4 changes: 4 additions & 0 deletions services/auth_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -983,6 +983,10 @@ def verify_api_key(cls, api_key: Optional[str], remote_addr: Optional[str] = Non
if api_key.startswith(KEY_NAMESPACE):
return verify_integration_key(api_key)

revisioned = key_controls.verify_revisioned_key(cls, api_key, remote_addr)
if revisioned is not key_controls.LEGACY_AUTH:
return revisioned

remembered = cls._remembered_username(api_key)
user = cls._users.get(remembered) if remembered else None
if remembered and user and not user.get("revoked_at"):
Expand Down
6 changes: 4 additions & 2 deletions services/auto_route_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from typing import Iterable, Mapping

from providers.registry import get_registry
from services import auto_route_d1
from services import auto_route_d1, config_revision_sync
from services.model_registry import ModelRegistry
from services.sqlite_store import connect, storage_path

Expand Down Expand Up @@ -404,7 +404,9 @@ def _rows_to_routes(cls, rows: Iterable[sqlite3.Row]) -> list[AutoRoute]:
@classmethod
def _durable_routes(cls) -> list[AutoRoute]:
"""Seeded defaults overlaid by the routes operators saved in D1."""
stored = auto_route_d1.stored_routes()
stored = config_revision_sync.route_copy()
if stored is None:
stored = auto_route_d1.stored_routes()
routes = {route_id: AutoRoute(route_id, candidates, "") for route_id, candidates in DEFAULT_AUTO_ROUTES.items()}
for route_id, (candidates, updated_at) in stored.items():
# A route still holding a retired default follows the current default.
Expand Down
Loading
Loading