Repository navigation
Integrate Wave 3 gateway features - #39
Conversation
MODEL_COOLDOWN_ENABLED (default off) scopes a throttled or exhausted credential to the affected model or an explicitly configured quota bucket, so other models keep using the key. Only a genuine credential-wide authentication rejection cools every model. When every eligible key is cooling, the request gets a real 429 with a bounded Retry-After. Calls that pass no model keep the existing provider-wide cooldowns, even with the flag on.
UPSTREAM_RETRY_AFTER_ADVICE_ENABLED (default off) parses Retry-After and documented reset headers into bounded local advice. An allowed managed retry then waits for the longer of the existing backoff and the advice, and is skipped when the wait would pass the request deadline. Parsing never grants a retry, and upstream headers are never rewritten. The managed retry scheduling moves into services/managed_dispatch.py and behaves exactly as before while the flag is off.
Expose admission-time gateway counters as X-MultiLLM-RateLimit-* headers and a bounded usage snapshot behind RATE_LIMIT_HEADERS_ENABLED, default off. Provider headers and raw routes are untouched.
Add a Durable Object admission authority with per-principal and per-model-group concurrency limits, renewable leases and Flask/Worker clients behind ADMISSION_ENABLED, default off. The class is inert while disabled.
Record authenticated native Worker generations in the D1 usage ledger and expose isolate-local Prometheus counters behind NATIVE_EDGE_METRICS_ENABLED, default off. Adds a static lifecycle hook chain for native dispatch.
Give Flask streaming iterators and the Worker roleplay transport a single cancellation owner, so a disconnect, read error or completion releases the upstream exactly once and is recorded as an ambiguous outcome with unknown usage. A failed SSE socket read now ends the stream instead of re-reading the partly consumed response.
…e/gateway-waves-1-3
…ate/gateway-waves-1-3
…tegrate/gateway-waves-1-3
…grate/gateway-waves-1-3 # Conflicts: # package.json
…integrate/gateway-waves-1-3 # Conflicts: # package.json # tests/fixtures/secret_firewall_coverage.json # tests/helpers/load_cloudflare_worker.mjs
…integrate/gateway-waves-1-3 # Conflicts: # package.json
Add schema digests to /mcp tools/list on Flask and the edge Worker and reject a mismatched X-MultiLLM-MCP-Contract pin with 409 before dispatch, behind MCP_CONTRACT_DIGESTS_ENABLED, default off. The catalogue script gains --baseline and --diff.
Resolve an immutable inherit/zero retention policy per default, verified key, route and request header behind CONTENT_RETENTION_ENABLED, default off. Zero requests bypass the chat cache and shadow sampling, keep roleplay state request-local and skip Knowledge memo and handoff writes.
…tegrate/gateway-waves-1-3 # Conflicts: # package.json
…o integrate/gateway-waves-1-3 # Conflicts: # package.json # tests/fixtures/secret_firewall_coverage.json # worker/cors-policy.mjs
Record ordinary input, cache-read, cache-write and output tokens and costs separately behind PROMPT_CACHE_USAGE_BUCKETS_ENABLED, default off, with cache_read/cache_write prices and migration 0016 adding nullable usage_events columns.
…egrate/gateway-waves-1-3 # Conflicts: # package.json # tests/fixtures/secret_firewall_coverage.json
Native Worker routes enforce no gateway request counters, so they have nothing to advertise; forwarded responses already carry the Container's headers.
…tegrate/gateway-waves-1-3 # Conflicts: # package.json # tests/fixtures/secret_firewall_coverage.json
Commit a revision counter with every model override, catalog, key-control and grant write, poll revisions through the control-state scheduler and reload only on newer revisions, behind CONFIG_REVISION_SYNC_ENABLED, default off. Security config fails closed when it cannot be verified within its TTL. Adds the static Flask gateway registrar and migration 0017.
…integrate/gateway-waves-1-3 # Conflicts: # package.json
Validate non-streaming managed results against a caller-supplied JSON schema behind OUTPUT_SCHEMA_VALIDATION_ENABLED (default off). Invalid results return 502 output_schema_violation with bounded, content-free paths; no regeneration or repair.
Carry reviewed per-protocol fields outside the Chat pivot behind PROTOCOL_EXTRAS_ENABLED (default off), restore them only for providers that admit each field, and report retained versus re-emitted fields without values.
Add multillm.tools.discover on both MCP endpoints behind DEFERRED_TOOLS_ENABLED (default off): grant-filtered local ranking with complete schemas, signed short-lived cursors, and grant and argument rechecks before every tool call. Grants live in the new tool_grants table (migration 0018) with compatibility rows for current tools.
Register the Flask post-authentication hooks in policy order (retention, admission, dispatch, rate-limit headers), share the admission identity and model group across runtimes, run each native Worker collaborator under its own flag, own hidden upstream streams before iteration, record buckets in the D1 ledger, carry trusted retention through roleplay and Knowledge transport, and mount the revisioned auto-route handler and native config polling.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| 🔵 In progress View logs |
multillm-proxy | bf939e0 | Oct 09 2026, 10:41 AM |
…o feat/deferred-tools # Conflicts: # package.json
… integrate/gateway-waves-1-5 # Conflicts: # package.json
…grate/gateway-waves-1-5 # Conflicts: # package.json
On a slow runner the 20 ms deadline expired before the request reached the fetcher, so the hold was released instead of kept as unknown. The fetcher never settles, so a 1000 ms deadline still fires after handoff.
The test checks admission uses the 1000 ms routing deadline rather than the 5000 ms policy one; a 3000 ms bound still tells them apart.
…ails An intermittent CI failure returned 502 with no log, so the assertion now reports the provider call count and the response body.
feat(gateway): wire the Wave 7 enterprise features together
feat(gateway): integrate the Wave 6 gateway features
feat(gateway): integrate the Wave 5 gateway features
feat(gateway): integrate the Wave 4 gateway features
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 37552414 | Triggered | Generic Password | c8787fc | worker/alert-delivery.mjs | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
6ed5e64
into
integrate/gateway-waves-1-2
Summary
This branch merges all ten Wave 3 feature branches on top of the Wave 2 integration branch (#25) and wires them together. It is based on
integrate/gateway-waves-1-2, so this diff shows only Wave 3.MODEL_COOLDOWN_ENABLEDUPSTREAM_RETRY_AFTER_ADVICE_ENABLEDRATE_LIMIT_HEADERS_ENABLED(Container)ADMISSION_ENABLED,ADMISSION_LIMITS_JSONNATIVE_EDGE_METRICS_ENABLEDMCP_CONTRACT_DIGESTS_ENABLEDCONTENT_RETENTION_ENABLEDPROMPT_CACHE_USAGE_BUCKETS_ENABLED, migration 0016CONFIG_REVISION_SYNC_ENABLED, migration 0017Integration changes
Flask
services.gateway_extensions.after_authentication). The single registrar inapp.pyregisters the collaborators in this fixed order:before_requestguard that runs before authentication, because authentication reads key controls);/v1/responses only.ModelCooldownExhausted,ModelCooldownCapacity) return their own 429 withRetry-Afterthrough route wrappers, with no translation to 500/503 and no fallback to another candidate.PROVIDER_QUOTA_BUCKETSapp config). Normalized Retry-After advice feeds cooldown recording.Worker
worker/gateway-extensions.mjsruns each native collaborator under its own flag. Previously, every collaborator was skipped unless the metrics flag was on./v1/auto-routesdispatch uses the revisioned handler. Revision polling runs from scheduled work and native request background work, through the newworker/native-config-sync.mjswith strict native installers.usage_buckets_unavailable, and a retry with the same batch ID does not duplicate it.X-MultiLLM-Retentionheader. They set the verified internal retention headers from the authenticated principal and strip caller copies. Knowledge retrieval and handoffs receive the trusted policy.Shared admission identity
Both runtimes use the same principal: lowercase hex SHA-256 of
multillm-admission:v1:<username>, from the authenticated account username. The bootstrap admin key resolves toADMIN_USERNAME, which defaults toadmin. For example,admingivesa5d35b6466ec3a0d0858b4eb73370c1703f8ccdb68c6586f9d41d1cd0075ddec.The model group is set as follows:
<provider>:<model>;/v1/routes: the requested model ID;default.The
principallimit therefore covers all of an account's keys.docs/admission-leases.mdis updated.Behaviour changes with all flags off
tests/test_cloudflare_worker.mjsasserts the new contract: the reader rejects withAbortError, and the upstream is fetched once in off mode.Registrations
tests/test_wave3_hub_worker.mjsandtests/test_wave3_transport_worker.mjstotest:worker.worker/native-config-sync.mjsto the secret firewall inventory.refactor(rate-limit): drop the unused Worker advice module). The headers apply to Flask managed responses only, and native Worker routes carry none.Before merging to main
Two D1 migrations are included:
0016_usage_buckets.sqland0017_control_revisions.sql. A push to main does not apply them, and/readyreportsd1_schema_missinguntil they exist. Apply them first:Tests
tests/test_wave3_integration.py(Flask, through the real routes),tests/test_wave3_hub_worker.mjs,tests/test_wave3_transport_worker.mjsandtests/test_knowledge_retention_transport.py. They cover:npm run test:worker: 1,349 passed.npm run test:integration: 7 passed.cf:migrations:checkreports 17 contiguous, additive migrations.test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.Wave 4 feature PRs (#36, #37, #38) target this branch.