Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
6c67249
feat(routing): add opt-in latency SLO admission
ALikesToCode Oct 9, 2026
a4b2231
feat(pools): add opt-in learned provider cooldowns
ALikesToCode Oct 9, 2026
c3712d1
feat(realtime): add opt-in Worker Realtime WebSocket transport
ALikesToCode Oct 9, 2026
4af6c28
feat(security): add opt-in context-leak canary tokens
ALikesToCode Oct 9, 2026
5581f44
feat(usage): add opt-in signed usage receipts
ALikesToCode Oct 9, 2026
13231aa
feat(enterprise): add optional enterprise integration contracts
ALikesToCode Oct 9, 2026
1ef4abf
feat(routing): add opt-in bounded hedged auto requests
ALikesToCode Oct 9, 2026
ffb93c6
feat(streaming): add opt-in per-key stream cost breaker
ALikesToCode Oct 9, 2026
f0ceaf3
Merge branch 'feat/realtime' into feat/stream-cost-breaker
ALikesToCode Oct 9, 2026
46a1c40
Merge branch 'feat/stream-cost-breaker' into feat/learned-cooldown
ALikesToCode Oct 9, 2026
5402b2d
Merge branch 'feat/learned-cooldown' into feat/usage-receipts
ALikesToCode Oct 9, 2026
c1c3646
Merge branch 'feat/usage-receipts' into integrate/gateway-waves-1-6
ALikesToCode Oct 9, 2026
3c9ba9f
Merge branch 'feat/hedged-requests' into integrate/gateway-waves-1-6
ALikesToCode Oct 9, 2026
6514d0c
Merge branch 'feat/latency-slo' into integrate/gateway-waves-1-6
ALikesToCode Oct 9, 2026
12f83c9
Merge branch 'feat/context-canary' into integrate/gateway-waves-1-6
ALikesToCode Oct 9, 2026
b2f2fd7
Merge branch 'feat/enterprise-contracts' into integrate/gateway-waves…
ALikesToCode Oct 9, 2026
82270c6
test(d1): skip the stream cost column with its accounts table
ALikesToCode Oct 9, 2026
7f65c83
Merge branch 'feat/stream-cost-breaker' into feat/learned-cooldown
ALikesToCode Oct 9, 2026
fa431c3
Merge branch 'feat/learned-cooldown' into feat/usage-receipts
ALikesToCode Oct 9, 2026
4e5e3d0
Merge branch 'feat/usage-receipts' into integrate/gateway-waves-1-6
ALikesToCode Oct 9, 2026
470a092
test(d1): skip the stream cost column in the unmigrated integration s…
ALikesToCode Oct 9, 2026
ad04d06
Merge branch 'feat/stream-cost-breaker' into feat/learned-cooldown
ALikesToCode Oct 9, 2026
ec0f357
Merge branch 'feat/learned-cooldown' into feat/usage-receipts
ALikesToCode Oct 9, 2026
314c2fe
Merge branch 'feat/usage-receipts' into integrate/gateway-waves-1-6
ALikesToCode Oct 9, 2026
db0968d
feat(gateway): wire the Wave 6 gateway features together
ALikesToCode Oct 9, 2026
3d427ee
feat(auth): add opt-in SAML federation through a trusted broker
ALikesToCode Oct 9, 2026
5953994
feat(billing): add opt-in Stripe top-ups credited only from verified …
ALikesToCode Oct 9, 2026
248addd
feat(provisioning): add opt-in SCIM 2.0 user and group provisioning
ALikesToCode Oct 9, 2026
4d362a7
feat(credits): add an opt-in gateway credits ledger
ALikesToCode Oct 9, 2026
649f8f9
feat(governance): add opt-in hierarchical grants, budgets and attribu…
ALikesToCode Oct 9, 2026
048e2ff
feat(tenants): add opt-in organisations, teams and verified workspaces
ALikesToCode Oct 9, 2026
389f1c6
Merge branch 'feat/tenant-hierarchy' into feat/tenant-governance
ALikesToCode Oct 9, 2026
5083e2f
Merge branch 'feat/tenant-governance' into feat/saml-federation
ALikesToCode Oct 9, 2026
20f68a1
Merge branch 'feat/saml-federation' into feat/scim-provisioning
ALikesToCode Oct 9, 2026
7538aec
Merge branch 'feat/scim-provisioning' into feat/credits-ledger
ALikesToCode Oct 9, 2026
81629b4
Merge branch 'feat/credits-ledger' into feat/payment-billing
ALikesToCode Oct 9, 2026
0cfdf92
test(d1): start the integration Worker with the production nodejs_com…
ALikesToCode Oct 9, 2026
764de2a
Merge branch 'feat/payment-billing' into integrate/gateway-waves-1-7
ALikesToCode Oct 9, 2026
7559c87
Merge branch 'feat/tenant-hierarchy' into integrate/gateway-waves-1-7
ALikesToCode Oct 9, 2026
de9a471
Merge branch 'feat/tenant-hierarchy' into feat/tenant-governance
ALikesToCode Oct 9, 2026
0c9d36f
Merge branch 'feat/tenant-governance' into feat/saml-federation
ALikesToCode Oct 9, 2026
606bba8
Merge branch 'feat/saml-federation' into feat/scim-provisioning
ALikesToCode Oct 9, 2026
3a2a109
Merge branch 'feat/scim-provisioning' into feat/credits-ledger
ALikesToCode Oct 9, 2026
cc933b4
Merge branch 'feat/credits-ledger' into feat/payment-billing
ALikesToCode Oct 9, 2026
cee6ade
Merge branch 'feat/payment-billing' into integrate/gateway-waves-1-7
ALikesToCode Oct 9, 2026
83e9193
feat(gateway): wire the Wave 7 enterprise features together
ALikesToCode Oct 9, 2026
df64359
Merge branch 'integrate/gateway-waves-1-6' into integrate/gateway-wav…
ALikesToCode Oct 9, 2026
b4ec747
Merge branch 'integrate/gateway-waves-1-5' into integrate/gateway-wav…
ALikesToCode Oct 9, 2026
e83bec4
Merge branch 'integrate/gateway-waves-1-6' into integrate/gateway-wav…
ALikesToCode Oct 9, 2026
e7e2166
Merge branch 'integrate/gateway-waves-1-5' into integrate/gateway-wav…
ALikesToCode Oct 9, 2026
4510992
fix(ci): run the credits parity check with the CI interpreter
ALikesToCode Oct 9, 2026
27c5f1d
Merge branch 'integrate/gateway-waves-1-6' into integrate/gateway-wav…
ALikesToCode Oct 9, 2026
44519f4
Merge branch 'integrate/gateway-waves-1-5' into integrate/gateway-wav…
ALikesToCode Oct 9, 2026
546cd41
Merge pull request #76 from ALikesToCode/integrate/gateway-waves-1-7
ALikesToCode Oct 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,20 @@ store data name the D1 migration to apply before enabling them.
- [PII redaction](docs/pii-rehydration.md): `PII_REDACTION_ENABLED` with `PII_REDACTION_POLICY_JSON` replaces detected emails, phone numbers and card numbers with request-local placeholders before dispatch and restores them in JSON and streamed responses.
- [Prompt injection heuristics](docs/prompt-injection.md): `PROMPT_INJECTION_MODE=log` or `block` scores managed requests for injection and jailbreak patterns before dispatch.
- [Reset-aware pool scheduling](docs/pool-reset-scheduling.md): `POOL_RESET_SCHEDULING_ENABLED` skips pooled accounts whose quota is freshly exhausted and prefers the earliest reset, using only data the gateway already receives.
- [Latency SLO admission](docs/latency-slo.md): `LATENCY_SLO_MODE=reject` or `reroute` with `LATENCY_SLO_POLICY_JSON` predicts deadline risk from recent model measurements and can reject requests or reroute eligible automatic selections before dispatch.
- [Hedged auto requests](docs/hedged-requests.md): `HEDGED_REQUESTS_ENABLED` with `HEDGED_REQUESTS_POLICY_JSON` races at most two candidates on auto routes explicitly marked idempotent-safe, and accounts for every attempt.
- [Realtime WebSocket transport](docs/realtime.md): `REALTIME_ENABLED` with `REALTIME_PROVIDERS_JSON` bridges authenticated OpenAI-compatible Realtime sockets directly in the Worker, under a per-session cost cap (migration 0029).
- [Stream cost breaker](docs/stream-cost-breaker.md): `STREAM_COST_BREAKER_ENABLED` lets a key carry `max_stream_cost_usd`, and stops a stream with the protocol's error when its observed or conservatively estimated cost would pass the cap (migration 0030).
- [Learned provider cooldowns](docs/learned-cooldown.md): `LEARNED_COOLDOWN_MODE=shadow` or `apply` with `LEARNED_COOLDOWN_POLICY_JSON` narrows a bounded cooldown interval from natural throttle and recovery observations, without probes (migration 0031).
- [Context-leak canary tokens](docs/context-canary.md): `CONTEXT_CANARY_MODE=log` or `block` with `CONTEXT_CANARY_POLICY_JSON` detects opted-in disclosure of a request-specific marker. It does not claim complete leak detection.
- [Signed usage receipts](docs/usage-receipts.md): `USAGE_RECEIPTS_ENABLED` with a reviewed Ed25519 key writes a hash-chained receipt per settled usage event that its owner can fetch and verify, keeping unknown cost explicit (migration 0032).
- [Enterprise integration contracts](docs/enterprise-contracts.md): typed identity, tenancy, quota and credit boundaries for later organisation, SSO and payment modules. `ENTERPRISE_PREVIEW_ENABLED` shows a read-only administrator preview; nothing else activates.
- [Organisations and teams](docs/tenant-hierarchy.md): `ORGANISATIONS_ENABLED` adds optional organisation and team membership with a workspace identity the server verifies, and keeps each workspace's caches, idempotency, hosted state and usage separate (migration 0033).
- [Tenant governance](docs/tenant-governance.md): `TENANT_GOVERNANCE_ENABLED` intersects organisation, team and key grants, reserves organisation and team budgets together with the key's, and attributes usage to the verified workspace (migration 0034).
- [SAML federation](docs/saml-federation.md): `SAML_ENABLED` with `SAML_BROKER_URL`, `SAML_TRUST_CONFIG_JSON` and `SAML_CALLBACK_URL` signs gateway accounts in to the dashboard through an explicitly trusted SAML broker, only for subjects an administrator has linked to an account (migration 0035).
- [SCIM provisioning](docs/scim-provisioning.md): `SCIM_ENABLED` with `SCIM_TRUST_CONFIG_JSON` provisions scoped users and organisation teams through SCIM 2.0; it needs D1 storage and security revision sync (migration 0036).
- [Gateway credits](docs/credits-ledger.md): `CREDITS_ENABLED` keeps an append-only ledger of credits, holds and settlements, separate from usage estimates. `CREDITS_ENFORCEMENT=funded` or `all` makes requests reserve and settle credits (migration 0037).
- [Top-ups](docs/payment-billing.md): `PAYMENTS_ENABLED` with `PAYMENT_PROCESSOR_CONFIG_JSON` and `PAYMENT_WEBHOOK_KEY_REF` creates Stripe Checkout top-ups and credits explicit USD payments only from verified, idempotent webhooks (migration 0038).

## Setup

Expand Down
9 changes: 8 additions & 1 deletion app.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
login_required,
)
from routes.config_snapshots import register_config_snapshot_routes
from routes.enterprise_preview import register_enterprise_preview_routes
from routes.tenants import register_tenant_routes
from routes.core import register_core_routes
from routes.dashboard_security import register_dashboard_security_routes
from routes.intelligence_media import register_intelligence_media_routes
Expand All @@ -34,13 +36,14 @@
from routes.unified import register_unified_routes
from routes.media import register_media_routes
from routes.usage import register_usage_routes
from routes.usage_receipts import register_usage_receipt_routes
from routes.workbench import register_workbench_routes
from security_config import load_max_content_length, validate_runtime_secrets
from services.auth_service import AuthService
from services.cache_service import CacheService
from services.image_relay_catalog import ImageRelayCatalogRefresh
from services.provider_catalog_refresh import ProviderCatalogAutoRefresh
from services.gateway_extensions import gateway_callbacks, register_gateway_extensions
from services.gateway_extensions import gateway_callbacks, register_gateway_extensions, register_enterprise_features
from services.metrics_service import MetricsService
from services.proxy_service import ProxyService
from services import usage_ledger
Expand Down Expand Up @@ -144,11 +147,15 @@ def create_app() -> Flask:
register_dashboard_security_routes(app)
register_workbench_routes(app)
register_config_snapshot_routes(app)
register_enterprise_preview_routes(app)
register_tenant_routes(app, csrf=csrf)
register_enterprise_features(app, csrf)
register_shadow_eval_routes(app, csrf, AuthService, MetricsService, ProxyService)
register_knowledge_routes(app, csrf)
register_gateway_mcp_routes(app, csrf)
register_documentation_routes(app, AuthService, ProxyService)
register_usage_routes(app, csrf)
register_usage_receipt_routes(app, csrf)
register_gateway_extensions(app, callbacks=gateway_callbacks(csrf=csrf))
register_gateway_batch_routes(app)
from functools import partial
Expand Down
11 changes: 7 additions & 4 deletions cloudflare-worker.mjs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { nativeGenerationFetch, withForwardedCorrelation, withNativeMetrics, nativeCacheHeader, nativeGenerationSetup, generationErrorResponse, forwardedGenerationHeaders, runScheduledMaintenance, scheduledMaintenanceEnabled, handleRoleplayContextPageRequest } from "./worker/gateway-extensions.mjs";
import { nativeGenerationFetch, withForwardedCorrelation, withNativeMetrics, nativeCacheHeader, nativeGenerationSetup, generationErrorResponse, forwardedGenerationHeaders, runScheduledMaintenance, scheduledMaintenanceEnabled, handleRoleplayContextPageRequest, handleRealtimeRequest } from "./worker/gateway-extensions.mjs";
import { batchesEnabled, runScheduledBatches } from "./worker/batch-jobs.mjs";
import { tickNativeRevisionSync } from "./worker/native-config-sync.mjs";
import { firewallFetch } from "./worker/secret-firewall.mjs";
Expand Down Expand Up @@ -1500,7 +1500,7 @@ async function handleDirectOpencodeRequest(request, env, requestUrl, ctx) {
...(bodyAllowed && request.body ? { duplex: "half" } : {}),
});
const upstreamResponse = await nativeGenerationFetch(upstreamRequest, env, ctx, {
deadlineHook: generationSetup.hook, route: requestUrl.pathname, cacheRequest: request, retentionHeader: request.headers.get("X-MultiLLM-Retention"), principal: { id: env.ADMIN_USERNAME || "admin" },
authenticated: true, keyId: env.ADMIN_USERNAME || "admin", deadlineHook: generationSetup.hook, route: requestUrl.pathname, cacheRequest: request, retentionHeader: request.headers.get("X-MultiLLM-Retention"), principal: { id: env.ADMIN_USERNAME || "admin" },
provider: requestUrl.pathname.split("/")[1],
}, async (upstreamRequest, env, authority) => await firewallFetch(upstreamRequest, env, authority));

Expand Down Expand Up @@ -1636,7 +1636,7 @@ async function handleDirectLinkApiRequest(request, env, requestUrl, ctx) {
...(bodyAllowed && normalizedRequest.body ? { duplex: "half" } : {}),
});
const upstreamResponse = await nativeGenerationFetch(upstreamRequest, env, ctx, {
deadlineHook: generationSetup.hook, route: requestUrl.pathname, cacheRequest: request, retentionHeader: request.headers.get("X-MultiLLM-Retention"), principal: { id: env.ADMIN_USERNAME || "admin" },
authenticated: true, keyId: env.ADMIN_USERNAME || "admin", deadlineHook: generationSetup.hook, route: requestUrl.pathname, cacheRequest: request, retentionHeader: request.headers.get("X-MultiLLM-Retention"), principal: { id: env.ADMIN_USERNAME || "admin" },
provider: requestUrl.pathname.split("/")[1],
}, async (upstreamRequest, env, authority) => await firewallFetch(upstreamRequest, env, authority));

Expand Down Expand Up @@ -1695,7 +1695,7 @@ async function handleDirectCodexEasyRequest(request, env, requestUrl, ctx) {
...(bodyAllowed && normalizedRequest.body ? { duplex: "half" } : {}),
});
const upstreamResponse = await nativeGenerationFetch(upstreamRequest, env, ctx, {
deadlineHook: generationSetup.hook, route: requestUrl.pathname, cacheRequest: request, retentionHeader: request.headers.get("X-MultiLLM-Retention"), principal: { id: env.ADMIN_USERNAME || "admin" },
authenticated: true, keyId: env.ADMIN_USERNAME || "admin", deadlineHook: generationSetup.hook, route: requestUrl.pathname, cacheRequest: request, retentionHeader: request.headers.get("X-MultiLLM-Retention"), principal: { id: env.ADMIN_USERNAME || "admin" },
provider: requestUrl.pathname.split("/")[1],
}, async (upstreamRequest, env, authority) => await firewallFetch(upstreamRequest, env, authority));

Expand Down Expand Up @@ -1891,6 +1891,9 @@ export default {
return applyCorsHeaders(request, buildFallbackHealthResponse(), env);
}

const realtimeResponse = await handleRealtimeRequest(request, env, ctx);
if (realtimeResponse) return realtimeResponse.status === 101 ? realtimeResponse : applyCorsHeaders(request, realtimeResponse, env);

if (requestUrl.pathname === "/status.prometheus" || requestUrl.pathname === "/v1/metrics/prometheus") {
if (!prometheusEnabled(env.PROMETHEUS_ENABLED)) {
return new Response("Not found", { status: 404, headers: { "Cache-Control": "no-store" } });
Expand Down
56 changes: 56 additions & 0 deletions docs/context-canary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Context-leak canary tokens

Context canaries detect disclosure of a request-specific marker in opted-in managed generation. They do not detect every context leak. An absent marker is not proof that private instructions or other context stayed private.

## Configuration

`CONTEXT_CANARY_MODE` accepts `off`, `log`, or `block`. The default and an empty value mean `off`. A malformed mode disables the feature and produces one warning that names the setting without its value.

`CONTEXT_CANARY_POLICY_JSON` defaults to `{}`. It accepts only `routes` and `keys`, each an array of exact scope names. Each array is limited to 512 nonempty strings of at most 256 characters, without control characters. Configuration is limited to 65,536 UTF-8 bytes. Unknown fields, invalid JSON, and invalid array values disable the feature with one warning. An empty value means `{}`. Enabling the mode alone does not opt in any requests.

Disabled example:

```text
CONTEXT_CANARY_MODE=off
CONTEXT_CANARY_POLICY_JSON={}
```

Opt in one managed route for removal and detection logging:

```text
CONTEXT_CANARY_MODE=log
CONTEXT_CANARY_POLICY_JSON={"routes":["/v1/roleplay/chat/completions"]}
```

Opt in an authenticated key scope for blocking:

```text
CONTEXT_CANARY_MODE=block
CONTEXT_CANARY_POLICY_JSON={"keys":["key-7"]}
```

Key scopes are identifiers established by authentication, never API credentials or caller-supplied policy overrides. Roleplay uses its canonical `/v1/roleplay/chat/completions` scope for compatible aliases. Raw provider passthrough does not insert or inspect canaries. With mode off or no applicable opt-in, the original payload and response objects pass through unchanged, and no canary trace or log record is created.

## Request and response handling

An opted-in managed request receives one clearly labelled gateway system annotation. Its 32-character hexadecimal marker is an HMAC-SHA256 result derived from a fresh random 32-byte secret. No persistent canary secret or remote classifier is used. The annotation is placed in a system message for chat, in the system field for Messages, or in instructions for Responses. Existing instructions remain present. The marker belongs only to that request; a marker from another request is not recognised.

`log` removes detected markers and continues delivering the provider response. `block` returns HTTP 502 with `error.code=context_canary_leak` when detection occurs before the response is committed. After streaming begins, the client receives the protocol's SSE error event without a fabricated completion or `[DONE]` event. Messages translation preserves the error message and may omit the internal code. The upstream is cancelled and generation is classified as unsuccessful. Detection never grants a retry or replay permission.

Earlier emitted text cannot be withdrawn. Some safe text may already have reached the client before detection. The canary does not prevent the disclosure of unrelated context or encoded, altered, or partial markers.

JSON strings are decoded before inspection, including Unicode escapes. SSE JSON data is assembled across reads and UTF-8 boundaries. Content, reasoning, text and tool argument fragments use independent text lanes with semantic choice and tool indices. A scanner retains only a possible marker prefix, at most 31 characters per lane. Unmatched prefixes are delivered at termination. Complete SSE frames are limited to 65,536 bytes, with at most 32 text lanes, structural depth 32, and 32,768 JSON nodes per frame. Nonstream response bodies are limited to 16 MiB. Unsupported, malformed, invalid UTF-8, or oversized opted-in responses fail with `context_canary_scan_failed`. These checks do not alter disabled traffic.

## Costs, retention and lifecycle

Cancellation does not prove that the provider stopped work or avoided billing. Blocked, interrupted or client-cancelled generation remains ambiguous. The finalizer marks the captured accounting context as ambiguous rather than releasing a cost hold. Measured provider usage and the existing accounting authority remain responsible for settlement. A stream error does not settle unknown spend as zero.

Canary event records contain only the marker digest, the rule `context_canary_leak`, and the trace ID. They contain no marker, annotation, prompt or response content. Each request records at most one detection. Roleplay's trace journal validates these fields and preserves the same restriction under zero-content retention. Request annotations are excluded from retained conversation and recovery inputs; output is inspected before completion persistence. Request-local parser state is released on completion, abort and client cancellation.

## Managed integration interfaces

Flask registers `context_canary_request_hook` after hosted Responses state and before the generation deadline. It decides opt-in from authenticated key scopes and explicitly managed routes without changing the caller's request body. A pre-dispatch rejection clears that scope. The managed turn calls `services.context_canary.prepare_request(payload, env, route=..., key_scope=..., protocol=...)` on the provider dispatch payload after request identity and admission, and preserves its request-local context. Caller payloads remain the source for fingerprints, hosted state and context pages. Supply `raw=True` for passthrough.

Response and stream finalization calls `services.context_canary.finalize_response(response, context, cancel=..., accounting=...)` before cache, replay, receipt and accounting finalization. Capture the cancellation collaborator and accounting context while the Flask request context is active; lazy generators do not consult Flask globals. Opted-in requests bypass exact, shared and semantic cross-request caches. A keyed replay of the caller's own completed, already scanned response remains allowed. The caller retains ownership of measured usage and reservations.

The Worker equivalents are `prepareRequest` and `finalizeResponse`. The response finalizer accepts an upstream abort controller, request signal, cancellation callback and captured accounting object. Native roleplay prepares its context before generation, annotates only dispatch payloads, scans initial and continuation responses before observing completion, and translates a late inspection failure into the client protocol error. Native direct generations inspect responses before metrics and settlement and bypass cross-request caches when opted in. Requests forwarded to Flask use Flask's finalizer once.
Loading
Loading