Skip to content

feat(gateway): integrate the Wave 5 gateway features - #60

Merged
ALikesToCode merged 87 commits into
integrate/gateway-waves-1-4from
integrate/gateway-waves-1-5
Oct 9, 2026
Merged

ALikesToCode merged 87 commits into
integrate/gateway-waves-1-4from
integrate/gateway-waves-1-5

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

This branch merges all eleven Wave 5 feature branches on top of the Wave 4 integration branch (#53) and wires them together. It is based on integrate/gateway-waves-1-4, so this diff shows only Wave 5.

PR Feature Setting
#59 Hosted Responses state HOSTED_RESPONSES_ENABLED, migration 0028
#57 Gateway batches and explicit spillover GATEWAY_BATCHES_ENABLED, BATCH_SPILLOVER_ENABLED, migration 0027
#51 Observability exports OBSERVABILITY_EXPORTERS_JSON
#50 Context pages CONTEXT_PAGING_ENABLED, migration 0024
#56 Semantic generation cache SEMANTIC_CACHE_ENABLED, SEMANTIC_CACHE_POLICY_JSON, migration 0026
#54 Canary traffic CANARY_TRAFFIC_ENABLED, migration 0025
#49 Bandit recommendations BANDIT_MODE
#48 Local inference providers OLLAMA_BASE_URL and four others
#58 PII redaction PII_REDACTION_ENABLED, PII_REDACTION_POLICY_JSON
#52 Prompt injection heuristics PROMPT_INJECTION_MODE, PROMPT_INJECTION_THRESHOLD
#55 Reset-aware pool scheduling POOL_RESET_SCHEDULING_ENABLED

The five migration PRs form one stack (#50 → #54 → #56 → #57 → #59) so each keeps its numbering contiguous. All eleven are merged here.

Integration changes

Flask hook order

register_gateway_extensions now runs the post-authentication hooks in one named order. Registrars no longer insert themselves by list position.

  1. Configuration security freshness (existing).
  2. Retention policy (existing).
  3. Prompt injection: a block returns 422 here, before any storage, batch, cache, claim, admission or provider call.
  4. Batch spillover: an explicit X-MultiLLM-Priority: batch + Prefer: respond-async request becomes a real one-item batch (202), under the caller's normal retention.
    • It runs before PII redaction. PII's request-local zero retention would otherwise turn every redacted spillover into retention_conflict.
    • The item later runs through the real managed route, where every enabled hook, PII included, runs again.
  5. PII redaction: replaces detected values and sets request-local zero retention.
  6. Hosted Responses state. It runs after PII, so a redacted request asking for hosted state gets 400 retention_conflict instead of storing content.
  7. Generation deadline (existing).
  8. Idempotency claim or replay (existing).
  9. Admission (existing).
  10. Dispatch through the managed pipeline.
  11. Rate-limit headers (existing).

app.py also registers the batch and context-page routes. The private context-page store is reached at /v1/managed-state/context-pages (2 MiB bound).

Managed dispatch

  • Context paging runs on the selected candidate before any lossy optimization. The capabilities field is removed before the provider call.
  • multillm_context_retrieve is an MCP tool, listed only while paging is on. It isn't in the default tool grants, so with tool grants on it needs an explicit grant.
  • The exact cache is checked before the semantic cache.
    • Paged requests, requests flagged in injection log mode and canary cohorts bypass semantic reuse.
    • PII-redacted requests bypass every cache through their zero retention.
  • Canary cohort, mode and route are added to request metrics. Session and principal are not.
  • Pool reset scheduling reads rate-limit windows from response headers the gateway already receives, only when its flag is on. No probes are added.
  • routes/unified.py is down to 990 lines.

Worker

  • Native direct generations:

    • prompt injection before admission;
    • semantic cache lookup after the exact cache;
    • the observability export hook, flushed with ctx.waitUntil;
    • cohort metadata in native telemetry.

    Native direct routes can't resolve auto routes, so canary selection stays in the managed path.

  • Private domains:

    • semantic-cache in the control-state dispatcher, which returns 404 while its flag is off and has a 2 MiB limit;
    • context-pages in the managed-state dispatcher;
    • canary policy saves in the D1 auto-route handler.
  • Roleplay:

    • context paging before compaction;
    • the authenticated PII key scope in transport settings;
    • redacted turns are neither cached nor replayable, and their responses carry no-store.

    Both scope values travel in internal headers on the Durable Object hop only. The edge builds that request from an allowlist of client headers, so a caller can't supply them.

  • Roleplay page retrieval: GET /v1/context/pages/<id>?roleplay_session_id=… is served in the Worker with the roleplay token's own session authority.

  • Scheduled maintenance: adds bounded semantic cache cleanup. With every flag off, the scheduled handler registers no extra work.

Coordinator fixes

  • Batch items skipped the response finalizers. A batch item ran the managed view directly, so after_request handlers never ran for it. Those handlers cover admission release, idempotency, the hosted Responses commit and PII restoration. The item now goes through app.process_response, as the route itself would. tests/test_gateway_batches.py covers this, and the test fails without the fix.
  • Batch items and context paging: page handles belong to the caller's API key, which stored batch work never holds. With paging on, an item that requests it now fails with 400 context_paging_unsupported before dispatch, at no cost. Before, it failed later with an authority error that was counted as an unknown outcome. Both docs say so.

Known limits

  • Context pages on failure: pages are written before dispatch, so a failed generation leaves its pages behind. They are owner- and session-scoped, expire after an hour and are never written under zero retention.
  • Paging and the prompt-size guard: paging runs after the existing prompt-size guard, so it can't yet rescue a prompt the guard rejects.
  • Paging and streams: streaming responses carry no context_pages metadata; complete JSON responses do.
  • Spillover budget: spillover, like POST /v1/batches, doesn't check the caller's account budget at creation. Each item checks key controls, budget and admission before dispatch.
  • Native semantic lookup: cross-provider embeddings bypass native semantic lookup, because the Worker has no authorized embedding transport for them.
  • Large files: cloudflare-worker.mjs (2,170 lines), routes/proxy.py (1,019) and services/proxy_service.py (3,581) received only small call sites.

Before merging to main

Five D1 migrations are included: 0024_context_pages.sql through 0028_responses_state.sql. A push to main does not apply them. Apply them first:

npx wrangler d1 migrations apply multillm-intelligence --remote

Tests

  • New tests:
    • tests/test_wave5_registrar.py: hook order, routes and storage plumbing;
    • tests/test_wave5_managed.py: the managed pipeline through the registered routes;
    • tests/test_wave5_worker.mjs: native order, private domains, roleplay and scheduled work.
  • They cover:
    • all flags off, byte for byte;
    • each flag alone;
    • the combined cases:
      • an injection block creates no batch, claim, lease or provider call;
      • PII-redacted requests are never cached, replayed or stored as hosted state;
      • a semantic hit makes no provider call;
      • canary shadow mode makes no extra call;
      • pool observations make no probe.
  • Full Python suite: 4,456 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,637 passed. npm run test:integration: 7 passed.
  • cf:migrations:check reports 28 contiguous, additive migrations. test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

Behind BANDIT_MODE (off, shadow or recommendation; default off), build task/model cells from stored evaluation results and nonce-protected administrator feedback, gate them on sample count, decay and the evaluation noise floor, and propose at most a one-position move in an automatic route's eligible order. Neither mode calls a provider or writes route policy; a proposal is applied only through the reviewed configuration snapshot apply with its revision check. Unknown cost or quality never ranks as favourable.
Behind CONTEXT_PAGING_ENABLED (default off), a managed client that declares the multillm_context_retrieve capability can have complete older exchanges replaced by signed retrieval handles instead of being summarized or truncated. Pages keep the exact stored JSON, never split a tool call from its results, and are scoped to the principal, session and policy revision. Bodies go in the existing R2 bucket and metadata in D1 (migration 0024), with 64 KiB per page, 1 MiB per session and a one-hour TTL. Zero-content retention disables paging, and a request whose protected content still does not fit returns 413.
Register ollama, vllm, lmstudio, llamacpp and sglang as OpenAI-compatible providers only when their base URL variable is set (all unset by default). URLs are validated to an explicit /v1 endpoint, plain HTTP is limited to loopback and private addresses, and an optional provider key is used when configured; an explicitly configured local server may be called without one. No port scanning, discovery or downloads; catalog entries carry declared capabilities with provenance.
Send content-free request observations to explicitly configured Langfuse and Helicone collectors when OBSERVABILITY_EXPORTERS_JSON lists them. Delivery uses a bounded in-memory queue with retries limited to transport failures, 429 and 5xx, and counts acknowledgements per exporter. Credentials come from variables ending in _EXPORT_CREDENTIAL. With the setting unset, nothing is sent and OTLP export is unchanged.
Score managed requests against bounded prompt injection and jailbreak rules when PROMPT_INJECTION_MODE is log or block. Block mode refuses with 422 before dispatch; both modes report the decision in X-MultiLLM-Injection-Action and a content-free log line. Key and route policies can only tighten the operator policy, and request bodies cannot change it. With the mode unset, requests are not inspected.
Behind CANARY_TRAFFIC_ENABLED, an auto route can carry a reviewed canary policy with fixed baseline and candidate weights. Sessions are assigned to a sticky cohort by an HMAC of the authenticated principal, session, route and salt revision; requests without a session stay on the baseline. Shadow mode records the proposed order without changing dispatch, and live mode puts approved candidates first for the candidate cohort while every existing filter still applies. Responses on opted-in routes carry X-MultiLLM-Canary-Cohort. Canary policies are stored in a new canary_traffic table (migration 0025).
Behind POOL_RESET_SCHEDULING_ENABLED, account pools use fresh reset observations from usage snapshots and response metadata the gateway already receives. Within the existing pool and priority tier, freshly exhausted accounts are skipped and the earliest fresh reset goes first; stale, missing or contradictory observations keep the existing order. When every account is cooling or exhausted, the existing 429 carries a Retry-After from the soonest fresh reset instead of sending another request. NanoGPT key validation keeps its existing cadence, and the scheduler adds no probes. Observations live in process for POOL_OBSERVATION_TTL_SECONDS and store only HMAC credential identities.
Behind SEMANTIC_CACHE_ENABLED and an explicit route or key opt-in in SEMANTIC_CACHE_POLICY_JSON, managed nonstream chat requests without tools can reuse a complete answer whose last user message has cosine similarity of at least 0.98. Every other part of the request, the model, sampling settings and policy revisions must match exactly, and a hit is refused when numbers, dates, negations or quoted text differ. Embeddings use an explicitly configured, priced model capped at 1,024 tokens and $0.001 per lookup. Entries are scoped to the principal, kept for 300 seconds, 256 per principal and 1 MiB each, in D1 (migration 0026) and the existing media bucket. Hits report semantic-hit provenance with no provider call; zero-content retention bypasses the cache.
OpenAI-compatible /v1/files and /v1/batches backed by D1 and R2, executed item by item through the managed chat route by the scheduled handler. Explicit spillover turns a chat request with X-MultiLLM-Priority: batch and Prefer: respond-async into a one-item batch. Both flags default to off.
Opted-in routes and key scopes replace detected emails, phones and cards with request-local HMAC placeholders before dispatch and restore them in JSON and SSE responses. Maps live only in request memory, redacted requests bypass caches and idempotency replay, and both settings default to off.
With HOSTED_RESPONSES_ENABLED on, a managed /v1/responses request with gateway_state and store keeps the completed response under a gateway-issued ID. previous_response_id continues from it in order, and GET and DELETE act only on the caller's own IDs. State lives in D1 and R2 with a 24-hour TTL, and only classified complete successes are stored.
Registers the post-authentication hooks in one named order (retention, injection, spillover, PII, hosted responses, deadline, idempotency, admission), mounts the batch and context-page routes, and connects context paging, semantic caching, canary metrics, pool usage windows, native injection, observability flushing, roleplay paging and PII scope, private storage domains and semantic cache cleanup. Batch items now run the route's response finalizers and refuse context paging before dispatch. Adds the README lines for the Wave 5 features.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 725f2b9e-7b8f-4dbc-948f-0c0d6a751466

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
🔵 In progress
View logs
multillm-proxy 3fee165 Oct 09 2026, 10:40 AM

…o feat/observability-exports

# Conflicts:
#	package.json
…o feat/prompt-injection

# Conflicts:
#	package.json
… integrate/gateway-waves-1-5

# Conflicts:
#	package.json
…grate/gateway-waves-1-5

# Conflicts:
#	package.json
feat(gateway): wire the Wave 7 enterprise features together
feat(gateway): integrate the Wave 6 gateway features
@ALikesToCode
ALikesToCode merged commit efb3012 into integrate/gateway-waves-1-4 Oct 9, 2026
6 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the integrate/gateway-waves-1-5 branch October 26, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant