Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 96 additions & 0 deletions docs/gateway-alerts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# Spend, pool and health alerts

Alerts report gateway spend estimates, unknown-price coverage, open provider circuits,
exhausted credential pools and recent provider health failures. They never start a
generation or a health probe. Paid health probes require their own explicit action.

Alerts are disabled by default. Empty environment values use the defaults:

```text
GATEWAY_ALERTS_ENABLED=false
GATEWAY_ALERT_WEBHOOK_ALLOWLIST=[]
```

Enable the feature and allow an exact HTTPS origin before configuring delivery:

```text
GATEWAY_ALERTS_ENABLED=true
GATEWAY_ALERT_WEBHOOK_ALLOWLIST=["https://hooks.example"]
```

The allowlist accepts at most 20 public DNS origins on port 443. IP literals, local
hostnames, credentials, query strings, fragments and redirects are rejected. A
destination can have a path, but it must belong to an allowlisted origin. Addresses
are stored as private configuration and are omitted from status and alert payloads.
Use an operator-controlled origin with public DNS; this origin policy does not pin DNS
answers or certify the destination's downstream behavior. Remove the origin or clear
the allowlist to stop delivery without changing generation traffic.

`GET /admin/alerts` requires an administrator's existing dashboard session. It returns
the current revision, normalized rules, whether a webhook is configured, and up to
100 recent event states. All responses have `Cache-Control: no-store`. POST retains
the dashboard's CSRF protection; supply the existing CSRF token with the request.
Use the revision from GET to replace the configuration:

```json
{
"revision": 0,
"destination": "https://hooks.example/operator-hook",
"rules": [
{"kind": "spend", "period": "day", "budget_usd": 25},
{"kind": "unknown_price", "period": "day"},
{"kind": "provider_circuit", "provider": "openai"},
{"kind": "pool_exhaustion", "provider": "nanogpt"},
{"kind": "provider_health", "provider": "openai", "failures": 3}
]
}
```

Spend rules use an operator-supplied global gateway budget in USD, with default
thresholds of 85% and 100%. `day` and `month` use UTC ledger periods. At most 20 rules
and four distinct spend thresholds per rule are accepted. Set `thresholds` explicitly
to change the spend percentages. An unknown-price rule defaults to alerting whenever
any recorded request lacks a price; `threshold_percent` can raise that minimum.
Provider names must belong to the reviewed provider catalog. An empty rule list stops
new observations. Replacing configuration fails waiting events from the old revision.

Spend payloads label their basis as `gateway_cost_estimate`; they are estimates from
recorded usage and configured model prices, never actual provider invoices. Coverage
payloads label `unknown_price_coverage`; an unpriced request is unknown, not free.
Provider alerts label `observed_health`. Health failure counts expire after one hour.
No prompt, response, tool content, raw credential, username or recipient appears in
an alert payload, status response or delivery diagnostic. Numeric observations can
still reveal aggregate traffic and spend; protect the webhook's access and retention.

The durable store deduplicates each event type, rule, threshold, period and revision
for 15 minutes. A sustained condition can create a new event after that interval.
Delivery is limited to ten events per scheduled run and three attempts per event,
with a hard three-second timeout per attempt. Retries are due after 60 seconds and
only run on the next scheduled invocation. Delivery claims prevent concurrent
runners from sending the same event simultaneously. Timeouts and process termination
can leave ambiguous delivery: consumers should deduplicate the stable event identity
and rule/revision/occurrence fields. A successful delivery means the destination
returned HTTP 2xx; it does not prove an operator read the notification.

States distinguish waiting, delivered and failed events; `attempts`,
`last_attempt_at` and a fixed `error_code` expose failures without response bodies.
Each event stores at most three attempt timestamps. History is logically retained
for seven days, physically pruned in batches of 100 during scheduled runs, and capped
at 2,000 dedupe records. A full queue cannot accept a new distinct event until pruning
makes room. Delivery JSON and configuration requests are bounded to 8 KiB; status
responses are bounded to 64 KiB.

Disabled or malformed settings return HTTP 404. Invalid rules and destinations return
400, revision conflicts 409, oversized configuration 413, and unavailable durable
state 503 with `gateway_alert_storage_unavailable`. The feature requires the D1
migration and private state dispatcher. An enabled flag with absent tables fails
closed. Configuration and observation do not deliver webhooks synchronously.

The Worker scheduler calls `runAlertDelivery` with a fixed aggregate collector and
redirect-rejecting transport. The collector receives normalized rules and returns
only `{rule_id, value, basis, window}` observations; a provider window is `current`.
The private dispatcher calls `handleAlertState` only after internal authentication.
The Flask registrar mounts the admin route and exposes `gateway_alert_observer`;
free checks observe existing results through the same service. Generation requests
are never replayed or delayed to send alerts. No live webhook or paid metering check
is needed to run the deterministic tests.
25 changes: 25 additions & 0 deletions intelligence-migrations/0022_gateway_alerts.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
-- Numeric alert evidence; webhook addresses are private configuration only.
CREATE TABLE IF NOT EXISTS gateway_alert_rules (
id INTEGER PRIMARY KEY CHECK(id = 1),
revision INTEGER NOT NULL CHECK(revision > 0 AND revision < 9007199254740991),
configuration TEXT NOT NULL CHECK(length(configuration) <= 8192),
updated_at REAL NOT NULL
);
CREATE TABLE IF NOT EXISTS gateway_alert_events (
dedupe_key TEXT PRIMARY KEY,
event_id TEXT NOT NULL,
rule_id TEXT NOT NULL,
revision INTEGER NOT NULL,
payload TEXT NOT NULL CHECK(length(payload) <= 8192),
created_at REAL NOT NULL,
state TEXT NOT NULL CHECK(state IN ('pending', 'delivered', 'failed')),
attempts INTEGER NOT NULL DEFAULT 0 CHECK(attempts BETWEEN 0 AND 3),
attempt_times TEXT NOT NULL DEFAULT '[]' CHECK(length(attempt_times) <= 128),
last_attempt_at REAL,
next_attempt_at REAL NOT NULL,
claim_token TEXT,
lease_until REAL NOT NULL DEFAULT 0,
error_code TEXT
);
CREATE INDEX IF NOT EXISTS gateway_alert_events_due ON gateway_alert_events(state, next_attempt_at, lease_until);
CREATE INDEX IF NOT EXISTS gateway_alert_events_retention ON gateway_alert_events(created_at);
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_session_tiers_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
39 changes: 39 additions & 0 deletions routes/alerts.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
"""Administrator alert controls retain existing session and CSRF protection."""
from flask import jsonify, request

from services.auth_service import AuthService
from services import gateway_alerts


def register_alert_routes(app):
def disabled_alerts():
if request.path == "/admin/alerts" and not gateway_alerts.settings()[0]:
return jsonify({"error": {"code": "not_found"}}), 404
if request.path == "/admin/alerts" and not AuthService.is_authenticated():
return jsonify({"error": {"code": "authentication_required"}}), 401
return None

app.before_request_funcs.setdefault(None, []).insert(0, disabled_alerts)

@app.after_request
def private_alerts(response):
if request.path == "/admin/alerts":
response.headers["Cache-Control"] = "no-store"
return response

@app.route("/admin/alerts", methods=["GET", "POST"])
def admin_alerts():
if not AuthService.is_authenticated():
return jsonify({"error": {"code": "authentication_required"}}), 401
if not (AuthService.get_current_user() or {}).get("is_admin"):
return jsonify({"error": {"code": "admin_required"}}), 403
try:
if request.method == "GET":
result = gateway_alerts.call("get")
else:
if request.content_length is not None and request.content_length > gateway_alerts.MAX_BYTES:
raise gateway_alerts.AlertError("gateway_alert_too_large", 413)
result = gateway_alerts.configure(request.get_json(silent=True), set(app.config["API_BASE_URLS"]))
return jsonify(result)
except gateway_alerts.AlertError as error:
return jsonify({"error": {"code": error.code}}), error.status
Loading
Loading