Repository navigation
feat(alerts): add opt-in spend, pool and provider health alerts - #46
Conversation
Evaluate operator alert rules over content-free aggregates (spend estimate, unknown-price coverage, open circuits, exhausted pools, consecutive provider failures) behind GATEWAY_ALERTS_ENABLED (default off). Rules and delivery events live in D1 (migration 0022). The Worker's scheduled run delivers deduplicated webhooks to allowlisted HTTPS origins (GATEWAY_ALERT_WEBHOOK_ALLOWLIST), at most three attempts per event. Administrators manage rules at /admin/alerts, which returns 404 while the flag is off.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 37552414 | Triggered | Generic Password | f1e4188 | worker/alert-delivery.mjs | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
multillm-proxy | 6487bac | Oct 09 2026, 12:42 AM |
Summary
Adds opt-in spend, pool and provider health alerts behind
GATEWAY_ALERTS_ENABLED, which defaults to false. With the flag off,/admin/alertsreturns 404, nothing is observed or stored, and existing routes, responses and storage are unchanged.spend: an operator USD budget per UTC day or month, with default thresholds of 85% and 100%.unknown_price: the share of recorded requests that have no price.provider_circuit: an open circuit.pool_exhaustion: no eligible credential is left in the pool.provider_health: consecutive failures within one hour.gateway_cost_estimate,unknown_price_coverageorobserved_health. A spend alert is an estimate from recorded usage and configured prices, not a provider invoice.GET/POST /admin/alertsrequires an administrator dashboard session and keeps the existing CSRF protection.Cache-Control: no-store.GATEWAY_ALERT_WEBHOOK_ALLOWLISTtakes up to 20 exact public HTTPS origins on port 443. IP literals, local hostnames, credentials and redirects are rejected.gateway_alert_rulesandgateway_alert_events(migration0022). With the flag on and the tables absent, the feature fails closed with 503.Details are in
docs/gateway-alerts.md.Glue left for the Wave 4 integration PR
runAlertDeliveryfrom the Worker's scheduled handler, with the fixed aggregate collector and the redirect-rejecting transport.handleAlertState) behind internal authentication, and add thealertsprivate endpoint mapping inservices/intelligence_d1_store.py.The Flask registrar already mounts the admin route and exposes
gateway_alert_observer, and the free health checks call the observer.Stacking
This PR is based on
feat/session-tiers(#45), so migrations0018–0022stay contiguous. Review only the top commit,feat(alerts): add opt-in spend, pool and provider health alerts; the merge commit brings in the lower layers.Deployment
Apply the migration before merging to main.
/readychecks the tables, and a push to main does not apply D1 migrations:Shared registrations
tests/test_gateway_alerts_worker.mjstotest:worker.GATEWAY_ALERTS_ENABLEDandGATEWAY_ALERT_WEBHOOK_ALLOWLISTto the Container env keys.worker/alert-delivery.mjsto the secret firewall module inventory, and/admin/alertsto the Flask route inventory. The route only talks to the private state endpoint and dispatches to no provider.Tests
tests/test_gateway_alerts.pyandtests/test_gateway_alerts_worker.mjs. They cover:npm run test:worker: 1,434 passed.npm run test:integration: 7 passed.cf:migrations:checkreports 22 contiguous, additive migrations.test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.CI notes
python/nodejobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.AdmissionCoordinatorDurable Object migration, whichwrangler versions uploadcannot apply. This is expected untilwrangler deployruns on main.This is part of Wave 4.