Skip to content

Integrate Wave 3 gateway features - #39

Merged
ALikesToCode merged 149 commits into
integrate/gateway-waves-1-2from
integrate/gateway-waves-1-3
Oct 9, 2026
Merged

ALikesToCode merged 149 commits into
integrate/gateway-waves-1-2from
integrate/gateway-waves-1-3

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

This branch merges all ten Wave 3 feature branches on top of the Wave 2 integration branch (#25) and wires them together. It is based on integrate/gateway-waves-1-2, so this diff shows only Wave 3.

PR Feature Setting
#26 Model-scoped credential cooldown MODEL_COOLDOWN_ENABLED
#27 Upstream Retry-After advice UPSTREAM_RETRY_AFTER_ADVICE_ENABLED
#28 Rate-limit headers RATE_LIMIT_HEADERS_ENABLED (Container)
#29 Shared concurrency admission ADMISSION_ENABLED, ADMISSION_LIMITS_JSON
#30 Native edge request metrics NATIVE_EDGE_METRICS_ENABLED
#31 Disconnect-aware cancellation always on (transport fix)
#32 MCP contract digests MCP_CONTRACT_DIGESTS_ENABLED
#33 Zero-content retention CONTENT_RETENTION_ENABLED
#34 Prompt cache usage and cost PROMPT_CACHE_USAGE_BUCKETS_ENABLED, migration 0016
#35 Revision-aware configuration sync CONFIG_REVISION_SYNC_ENABLED, migration 0017

Integration changes

Flask

  • Authentication runs one named post-authentication hook (services.gateway_extensions.after_authentication). The single registrar in app.py registers the collaborators in this fixed order:
    1. configuration security freshness (a before_request guard that runs before authentication, because authentication reads key controls);
    2. retention policy;
    3. admission, for forwarded and Flask-served generations, with the generation's real deadline;
    4. dispatch;
    5. rate-limit headers, on managed /v1/ responses only.
  • Cooldown errors (ModelCooldownExhausted, ModelCooldownCapacity) return their own 429 with Retry-After through route wrappers, with no translation to 500/503 and no fallback to another candidate.
  • Credential selection and recording share the resolved upstream model and the trusted quota bucket (PROVIDER_QUOTA_BUCKETS app config). Normalized Retry-After advice feeds cooldown recording.
  • Hidden upstream streams get a cancellation owner before the lazy Flask generator is built. Losing an admission lease cancels that owner.
  • Knowledge requests from the Container now carry the request's retention snapshot. The Container rejects a zero-retention handoff save with 409 before secret scanning or transport.
  • With the bucket flag on, SQLite mode records prompt cache bucket columns end to end.

Worker

  • worker/gateway-extensions.mjs runs each native collaborator under its own flag. Previously, every collaborator was skipped unless the metrics flag was on.
  • Order for native direct generations (codex-easy, linkapi, opencode): authentication → security freshness → retention → admission → dispatch with the request's abort signal → metrics and cancellation outcome. The admission lease is released exactly once.
  • The private /v1/auto-routes dispatch uses the revisioned handler. Revision polling runs from scheduled work and native request background work, through the new worker/native-config-sync.mjs with strict native installers.
  • The usage ledger writes prompt cache buckets inside the existing batch transaction. If the extended write fails, the base row is kept and the request returns 503 usage_buckets_unavailable, and a retry with the same batch ID does not duplicate it.
  • The roleplay edges carry the caller's X-MultiLLM-Retention header. They set the verified internal retention headers from the authenticated principal and strip caller copies. Knowledge retrieval and handoffs receive the trusted policy.
  • Forwarded requests acquire admission only in Flask.

Shared admission identity

Both runtimes use the same principal: lowercase hex SHA-256 of multillm-admission:v1:<username>, from the authenticated account username. The bootstrap admin key resolves to ADMIN_USERNAME, which defaults to admin. For example, admin gives a5d35b6466ec3a0d0858b4eb73370c1703f8ccdb68c6586f9d41d1cd0075ddec.

The model group is set as follows:

  • provider-prefixed routes: <provider>:<model>;
  • managed /v1/ routes: the requested model ID;
  • otherwise: the provider name, or default.

The principal limit therefore covers all of an account's keys. docs/admission-leases.md is updated.

Behaviour changes with all flags off

  • The Worker now cancels the upstream fetch when the client aborts a gated Codex Everywhere stream. Previously it kept reading. tests/test_cloudflare_worker.mjs asserts the new contract: the reader rejects with AbortError, and the upstream is fetched once in off mode.
  • Flask accounting records a managed stream cancelled after handoff as status 499 with unknown usage, instead of its original status. The budget reservation is released as it is for any request without measured usage. Durable holds for ambiguous usage are part of a later wave.

Registrations

  • Added tests/test_wave3_hub_worker.mjs and tests/test_wave3_transport_worker.mjs to test:worker.
  • Added worker/native-config-sync.mjs to the secret firewall inventory.
  • Added README lines for the ten features.
  • The rate-limit headers branch dropped its unused Worker module (refactor(rate-limit): drop the unused Worker advice module). The headers apply to Flask managed responses only, and native Worker routes carry none.

Before merging to main

Two D1 migrations are included: 0016_usage_buckets.sql and 0017_control_revisions.sql. A push to main does not apply them, and /ready reports d1_schema_missing until they exist. Apply them first:

npx wrangler d1 migrations apply multillm-intelligence --remote

Tests

  • New: tests/test_wave3_integration.py (Flask, through the real routes), tests/test_wave3_hub_worker.mjs, tests/test_wave3_transport_worker.mjs and tests/test_knowledge_retention_transport.py. They cover:
    • all flags off, byte for byte;
    • each flag alone, through its documented contract;
    • all flags on, in hook order;
    • the admission identity test vector.
  • Full Python suite: 3,549 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,349 passed.
  • npm run test:integration: 7 passed.
  • cf:migrations:check reports 17 contiguous, additive migrations. test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

Wave 4 feature PRs (#36, #37, #38) target this branch.

MODEL_COOLDOWN_ENABLED (default off) scopes a throttled or exhausted credential to the affected model or an explicitly configured quota bucket, so other models keep using the key. Only a genuine credential-wide authentication rejection cools every model. When every eligible key is cooling, the request gets a real 429 with a bounded Retry-After. Calls that pass no model keep the existing provider-wide cooldowns, even with the flag on.
UPSTREAM_RETRY_AFTER_ADVICE_ENABLED (default off) parses Retry-After and documented reset headers into bounded local advice. An allowed managed retry then waits for the longer of the existing backoff and the advice, and is skipped when the wait would pass the request deadline. Parsing never grants a retry, and upstream headers are never rewritten. The managed retry scheduling moves into services/managed_dispatch.py and behaves exactly as before while the flag is off.
Expose admission-time gateway counters as X-MultiLLM-RateLimit-* headers and a bounded usage snapshot behind RATE_LIMIT_HEADERS_ENABLED, default off. Provider headers and raw routes are untouched.
Add a Durable Object admission authority with per-principal and per-model-group concurrency limits, renewable leases and Flask/Worker clients behind ADMISSION_ENABLED, default off. The class is inert while disabled.
Record authenticated native Worker generations in the D1 usage ledger and expose isolate-local Prometheus counters behind NATIVE_EDGE_METRICS_ENABLED, default off. Adds a static lifecycle hook chain for native dispatch.
Give Flask streaming iterators and the Worker roleplay transport a single cancellation owner, so a disconnect, read error or completion releases the upstream exactly once and is recorded as an ambiguous outcome with unknown usage. A failed SSE socket read now ends the stream instead of re-reading the partly consumed response.
…grate/gateway-waves-1-3

# Conflicts:
#	package.json
…integrate/gateway-waves-1-3

# Conflicts:
#	package.json
#	tests/fixtures/secret_firewall_coverage.json
#	tests/helpers/load_cloudflare_worker.mjs
…integrate/gateway-waves-1-3

# Conflicts:
#	package.json
Add schema digests to /mcp tools/list on Flask and the edge Worker and reject a mismatched X-MultiLLM-MCP-Contract pin with 409 before dispatch, behind MCP_CONTRACT_DIGESTS_ENABLED, default off. The catalogue script gains --baseline and --diff.
Resolve an immutable inherit/zero retention policy per default, verified key, route and request header behind CONTENT_RETENTION_ENABLED, default off. Zero requests bypass the chat cache and shadow sampling, keep roleplay state request-local and skip Knowledge memo and handoff writes.
…tegrate/gateway-waves-1-3

# Conflicts:
#	package.json
…o integrate/gateway-waves-1-3

# Conflicts:
#	package.json
#	tests/fixtures/secret_firewall_coverage.json
#	worker/cors-policy.mjs
Record ordinary input, cache-read, cache-write and output tokens and costs separately behind PROMPT_CACHE_USAGE_BUCKETS_ENABLED, default off, with cache_read/cache_write prices and migration 0016 adding nullable usage_events columns.
…egrate/gateway-waves-1-3

# Conflicts:
#	package.json
#	tests/fixtures/secret_firewall_coverage.json
Native Worker routes enforce no gateway request counters, so they have nothing to advertise; forwarded responses already carry the Container's headers.
…tegrate/gateway-waves-1-3

# Conflicts:
#	package.json
#	tests/fixtures/secret_firewall_coverage.json
Commit a revision counter with every model override, catalog, key-control and grant write, poll revisions through the control-state scheduler and reload only on newer revisions, behind CONFIG_REVISION_SYNC_ENABLED, default off. Security config fails closed when it cannot be verified within its TTL. Adds the static Flask gateway registrar and migration 0017.
…integrate/gateway-waves-1-3

# Conflicts:
#	package.json
Validate non-streaming managed results against a caller-supplied JSON schema behind OUTPUT_SCHEMA_VALIDATION_ENABLED (default off). Invalid results return 502 output_schema_violation with bounded, content-free paths; no regeneration or repair.
Carry reviewed per-protocol fields outside the Chat pivot behind PROTOCOL_EXTRAS_ENABLED (default off), restore them only for providers that admit each field, and report retained versus re-emitted fields without values.
Add multillm.tools.discover on both MCP endpoints behind DEFERRED_TOOLS_ENABLED (default off): grant-filtered local ranking with complete schemas, signed short-lived cursors, and grant and argument rechecks before every tool call. Grants live in the new tool_grants table (migration 0018) with compatibility rows for current tools.
Register the Flask post-authentication hooks in policy order (retention, admission, dispatch, rate-limit headers), share the admission identity and model group across runtimes, run each native Worker collaborator under its own flag, own hidden upstream streams before iteration, record buckets in the D1 ledger, carry trusted retention through roleplay and Knowledge transport, and mount the revisioned auto-route handler and native config polling.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 81988c0c-efb6-4e4a-bb70-fc59d912461d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
🔵 In progress
View logs
multillm-proxy bf939e0 Oct 09 2026, 10:41 AM

…o feat/deferred-tools

# Conflicts:
#	package.json
… integrate/gateway-waves-1-5

# Conflicts:
#	package.json
…grate/gateway-waves-1-5

# Conflicts:
#	package.json
On a slow runner the 20 ms deadline expired before the request reached the
fetcher, so the hold was released instead of kept as unknown. The fetcher
never settles, so a 1000 ms deadline still fires after handoff.
The test checks admission uses the 1000 ms routing deadline rather than the
5000 ms policy one; a 3000 ms bound still tells them apart.
…ails

An intermittent CI failure returned 502 with no log, so the assertion now
reports the provider call count and the response body.
feat(gateway): wire the Wave 7 enterprise features together
feat(gateway): integrate the Wave 6 gateway features
feat(gateway): integrate the Wave 5 gateway features
feat(gateway): integrate the Wave 4 gateway features
@gitguardian

gitguardian Bot commented Oct 9, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
37552414 Triggered Generic Password c8787fc worker/alert-delivery.mjs View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@ALikesToCode
ALikesToCode merged commit 6ed5e64 into integrate/gateway-waves-1-2 Oct 9, 2026
5 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the integrate/gateway-waves-1-3 branch October 26, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant