Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion cloudflare-worker.mjs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { nativeGenerationFetch, withForwardedCorrelation, withNativeMetrics, nativeCacheHeader, nativeGenerationSetup, generationErrorResponse, forwardedGenerationHeaders, runScheduledMaintenance, scheduledMaintenanceEnabled, handleRoleplayContextPageRequest } from "./worker/gateway-extensions.mjs";
import { nativeGenerationFetch, withForwardedCorrelation, withNativeMetrics, nativeCacheHeader, nativeGenerationSetup, generationErrorResponse, forwardedGenerationHeaders, runScheduledMaintenance, scheduledMaintenanceEnabled, handleRoleplayContextPageRequest, handleRealtimeRequest } from "./worker/gateway-extensions.mjs";
import { batchesEnabled, runScheduledBatches } from "./worker/batch-jobs.mjs";
import { tickNativeRevisionSync } from "./worker/native-config-sync.mjs";
import { firewallFetch } from "./worker/secret-firewall.mjs";
Expand Down Expand Up @@ -1891,6 +1891,9 @@ export default {
return applyCorsHeaders(request, buildFallbackHealthResponse(), env);
}

const realtimeResponse = await handleRealtimeRequest(request, env, ctx);
if (realtimeResponse) return realtimeResponse.status === 101 ? realtimeResponse : applyCorsHeaders(request, realtimeResponse, env);

if (requestUrl.pathname === "/status.prometheus" || requestUrl.pathname === "/v1/metrics/prometheus") {
if (!prometheusEnabled(env.PROMETHEUS_ENABLED)) {
return new Response("Not found", { status: 404, headers: { "Cache-Control": "no-store" } });
Expand Down
91 changes: 91 additions & 0 deletions docs/realtime.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Worker Realtime WebSocket transport

Realtime bridges authenticated OpenAI-compatible sockets directly in the Worker.
It does not start the Container. `REALTIME_ENABLED` is off by default; an unset,
empty, false or malformed value preserves existing route handling. A malformed
provider mapping disables Realtime and warns once without printing its value.

Enable it with `REALTIME_ENABLED=true` and an exact model mapping:

```json
{"openai:realtime-model":{"url":"wss://approved.example/v1/realtime?model=realtime-model"}}
```

Set this JSON as `REALTIME_PROVIDERS_JSON`. Every URL must use WSS, a public DNS
hostname and port 443. The mapping supplies the complete upstream URL, including
any provider model query. Client URLs, query fields, hosts and credentials never
select an upstream. Redirects are refused. OpenAI, OpenCode, LinkAPI, Codex Easy,
NanoGPT, OpenRouter, Groq and xAI use their existing Worker provider key bindings
with Bearer authentication. Configure only providers whose socket contract uses
that authentication. No caller header reaches upstream except the exact
`OpenAI-Beta: realtime=v1` version header. Upstream headers are not returned.

Connect with `GET /v1/realtime?model=openai:realtime-model`, `Upgrade: websocket`
and `Authorization: Bearer <gateway key>` or `X-MultiLLM-API-Key`. The bootstrap
gateway key, verifiable D1 dashboard keys and D1 integration keys are accepted.
Dashboard model grants, expiry, revocation and IP controls apply. Both `chat` and
`audio` scopes are required. Keys stored only in the Container cannot open sockets.
Each socket is authorized at admission; revoking a key stops later admissions,
including tickets. Already-open sockets expire within 15 minutes.

With Realtime enabled, invalid upgrades return JSON 426, absent or invalid keys
return 401 and insufficient scopes or model grants return 403. Unsupported models
return 403. Missing storage, migrations, price evidence, monetary ceilings or
admission authority return JSON 503 before dialing. A full session or budget limit
returns 429. Upstream handshake failure returns 502; no automatic retry occurs.
With Realtime disabled, `/v1/realtime` keeps the existing Container handling.

## Cost admission

`REALTIME_SESSION_CAP_USD` must be positive. `REALTIME_PRICING_JSON` maps each
exact model to prices per million tokens and a provider-enforced session ceiling:

```json
{"openai:realtime-model":{"input_text":1,"output_text":2,"input_audio":3,"output_audio":4,"provider_session_limit_usd":0.10}}
```

These numbers are illustrative, not a provider price quotation. The provider
ceiling must be enforced by an independently verified upstream account or session
contract and must be no greater than the gateway cap. Ordinary usage events arrive
after generation and cannot enforce a hard monetary ceiling themselves. Do not
configure an estimated cost as `provider_session_limit_usd`. If the upstream has
no enforceable ceiling, leave it unset: admission refuses the session. Operators
must verify that the ceiling includes every billed activity, including transcription.
The gateway cannot discover or certify a provider's billing contract from a socket.

The entire ceiling is reserved through durable usage reservations before dialing.
Key daily/monthly budgets apply; otherwise set a positive
`REALTIME_DAILY_BUDGET_USD`. Existing unknown holds still reduce available budget.
Known final audio/text usage uses the supplied prices. Cached input requires
`cached_text` and `cached_audio` prices plus the provider's modality breakdown.
Partial, inconsistent, unpriced, interrupted or in-flight usage retains an unknown
hold instead of inventing a zero bill. Unpriced transcription remains unknown.
The usage ledger stores measured aggregate tokens and nullable cost. Realtime
session rows also store modality counts, opaque key identity, lease and timestamps.
Storage failures retain reservation evidence for operator reconciliation.

## Lifetime and tickets

At most two simultaneous sockets per gateway key are admitted atomically in D1.
Enable the existing admission authority with a nonzero principal or model-group
limit and its existing binding. Its renewable lease is held until either side
closes. Sessions have a 900-second TTL, 30-second idle limit and 1 MiB frame limit.
Both text and binary frames retain their bytes. Policy violations close 1008.
Upstream error events are forwarded verbatim before closing both sides with 1011,
unless an event exposes the provider key, in which case it is withheld and the
session closes 1008. Transport failures close 1011 without fabricated events.

Optional `POST /v1/realtime/client_secrets` accepts an authenticated JSON body
`{"model":"openai:realtime-model"}`. It requires the existing `JWT_SECRET` binding
and returns `{value,expires_at}`: a gateway-scoped signed ticket, never a provider
secret. Present `value` as the Bearer token for the socket. Tickets last 60 seconds,
are restricted to that model and current credential, and are consumed atomically
once. At most 16 outstanding tickets per key may exist. A ticket cannot issue
another ticket. No credentials are accepted in URL parameters or subprotocols.

Apply migration `0029_realtime_sessions.sql` and the existing usage/admission
migrations before enabling. No prompt, transcript, audio or event content is
persisted or logged. Socket metadata and unknown holds remain until the operator's
approved retention/reconciliation process handles them. The implementation does
not claim live provider protocol, billing or WebSocket compatibility from local
fake-socket checks.
14 changes: 14 additions & 0 deletions intelligence-migrations/0029_realtime_sessions.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
CREATE TABLE IF NOT EXISTS realtime_tickets (
nonce TEXT PRIMARY KEY, principal_hash TEXT NOT NULL, owner TEXT NOT NULL,
credential_kind TEXT NOT NULL, credential_fingerprint TEXT NOT NULL,
model TEXT NOT NULL, expires_at INTEGER NOT NULL, consumed_at INTEGER
);
CREATE INDEX IF NOT EXISTS realtime_ticket_expiry ON realtime_tickets(expires_at);
CREATE TABLE IF NOT EXISTS realtime_sessions (
id TEXT PRIMARY KEY, principal_hash TEXT NOT NULL, owner TEXT NOT NULL, model TEXT NOT NULL,
lease_id TEXT, created_at INTEGER NOT NULL, expires_at INTEGER NOT NULL, lease_until INTEGER NOT NULL,
state TEXT NOT NULL CHECK(state IN ('admitted','active','settled','unknown','released')),
input_text_tokens INTEGER, input_audio_tokens INTEGER, output_text_tokens INTEGER, output_audio_tokens INTEGER,
cost_usd REAL, closed_at INTEGER
);
CREATE INDEX IF NOT EXISTS realtime_session_scope ON realtime_sessions(principal_hash,state,lease_until);
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_responses_state_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs tests/test_observability_exports_worker.mjs tests/test_prompt_injection_worker.mjs tests/test_pii_rehydration_worker.mjs tests/test_wave5_worker.mjs tests/test_realtime_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
3 changes: 3 additions & 0 deletions tests/fixtures/secret_firewall_coverage.json
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,9 @@
"worker/prometheus.mjs",
"worker/prompt-injection-detection.mjs",
"worker/prompt-templates-d1.mjs",
"worker/realtime-metering.mjs",
"worker/realtime-session.mjs",
"worker/realtime.mjs",
"worker/request-telemetry.mjs",
"worker/reservations-d1.mjs",
"worker/responses-state-d1.mjs",
Expand Down
Loading
Loading