Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions cloudflare-worker.mjs
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { nativeGenerationFetch, withForwardedCorrelation, withNativeMetrics, nativeCacheHeader, nativeGenerationSetup, generationErrorResponse, forwardedGenerationHeaders, runScheduledMaintenance, scheduledMaintenanceEnabled } from "./worker/gateway-extensions.mjs";
import { batchesEnabled, runScheduledBatches } from "./worker/batch-jobs.mjs";
import { tickNativeRevisionSync } from "./worker/native-config-sync.mjs";
import { firewallFetch } from "./worker/secret-firewall.mjs";
import { Container, ContainerProxy, getContainer } from "@cloudflare/containers";
Expand Down Expand Up @@ -1837,6 +1838,9 @@ export default {
tickNativeRevisionSync(env, ctx);
if (scheduledMaintenanceEnabled(env)) ctx.waitUntil(runScheduledMaintenance(env));
const container = getContainer(env.MULTILLM_PROXY_CONTAINER, "primary");
if (batchesEnabled(env)) ctx.waitUntil(runScheduledBatches(env, container).catch(() => {
console.warn(JSON.stringify({ event: "gateway_batch_schedule_failed" }));
}));
ctx.waitUntil(runScheduledShadowEval(env, container));
ctx.waitUntil(runScheduledHealth(controller, env, container).catch((error) => {
logStructuredError("scheduled_health_failed", error);
Expand Down
52 changes: 52 additions & 0 deletions docs/gateway-batches.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Gateway batches

The Batch API stores owned JSONL files and executes each item through the gateway's managed Chat Completions or Responses route. Both `GATEWAY_BATCHES_ENABLED` and `BATCH_SPILLOVER_ENABLED` default to `false`. Empty settings are off; malformed settings are off with one warning that omits the value. Disabled batch routes return 404 and normal generations continue unchanged.

Apply the gateway batch D1 migration before enabling `GATEWAY_BATCHES_ENABLED` in the Worker and Container. Storage uses `INTELLIGENCE_DB` and the existing `multillm_media` R2 binding. No extra binding or cron schedule is required. An enabled endpoint with missing tables or storage returns JSON 503 `gateway_batches_unavailable` before provider submission.

## Upload and create

Each UTF-8 JSONL line contains a unique `custom_id`, `method: "POST"`, a supported `url`, and a nonstreaming request `body` with a model. A file can contain at most 1,000 items and 10 MiB. Invalid input returns 400 `invalid_file` with the first bad line number. Other file purposes return 400. Requests must use a concrete priced model or a route whose every candidate has a known price; custom routing definitions are rejected.

```json
{"custom_id":"question-1","method":"POST","url":"/v1/chat/completions","body":{"model":"mimo:mimo-v2.5","messages":[{"role":"user","content":"Explain tides briefly."}],"max_tokens":128}}
```

Upload with an existing gateway credential in your client environment:

```sh
curl "$GATEWAY_URL/v1/files" -H "Authorization: Bearer $GATEWAY_KEY" \
-F purpose=batch -F [email protected]
```

Create a batch using the returned file ID:

```json
{"input_file_id":"file_RETURNED_ID","endpoint":"/v1/chat/completions","completion_window":"24h","metadata":{"multillm_budget_usd":"0.50"}}
```

Send this JSON to `POST /v1/batches`. All items must match `endpoint`, which accepts `/v1/chat/completions` or `/v1/responses`. The budget is required: a positive decimal string of at most 100000 USD with at most ten fractional digits. Metadata accepts at most 16 string values, with keys up to 64 and values up to 512 characters. Unknown prices fail creation with 400 `unknown_price`. Model permissions apply at creation and execution.

Explicit output limits must be integers from 1 to 262144 tokens; completion count `n`, when present, is from 1 to 16. Admission uses input JSON byte count and the largest output bound, multiplied by `n`, rounded upwards in units of 0.0000000001 USD. Without an output limit, admission conservatively reserves 262144 output tokens. Set an explicit output limit to keep the reservation useful. This is a gateway spend bound, not a provider batch discount or a guarantee of provider billing accuracy.

## Read, cancel and delete

`GET /v1/files` and `GET /v1/batches` return list objects with `data`, `first_id`, `last_id`, and `has_more`. They accept `limit` from 1 to 100 and an `after` ID. File metadata is available at `GET /v1/files/<id>`; bytes at `GET /v1/files/<id>/content`; deletion at `DELETE /v1/files/<id>`. Every operation is restricted to the authenticated owner, including administrators. Another owner's ID returns 404. An input file used by an active batch cannot be deleted (409 `file_in_use`). Existing media routes remain unchanged.

`GET /v1/batches/<id>` returns the batch object, timestamps, request counts, and output/error file IDs. `POST /v1/batches/<id>/cancel` stops new work; already dispatched items finish and remain recorded. Cancellation moves through `cancelling` to `cancelled`. Normal execution moves through `validating`, `in_progress`, `finalizing`, and `completed`. Expiry stops unfinished work 24 hours after creation and ends as `expired`. Individual failures appear in the error file; a completed batch can contain failed items.

Output and error files contain JSONL records keyed by `custom_id`, with `response` and `error` fields. They use the same owned file APIs. Individual results are limited to 1 MiB; aggregate output and error content are bounded at 16 MiB each. Oversized results become errors and are never retried. API files remain stored until their owner deletes them; the 24-hour completion window is not file retention expiry. Intermediate result objects and assembly checkpoints persist independently under `batches/`; there is no automatic TTL or batch deletion endpoint.

## Execution and recovery

The existing Worker scheduler claims at most two items per invocation using an atomic D1 lease and spend hold. Each item has a 30-second deadline; the invocation has a 65-second work budget. Remaining items continue on the next invocation. Completion time depends on the configured cron frequency and the two-items-per-run limit. A single shard processing 1,000 items needs at least 500 invocations.

The scheduler sends the Container an expiring, signed owner capability and an item lease. It does not store or replay the owner's API credential. The Container consumes the durable lease once, loads canonical content from batch storage, reloads the owner's current account, and checks key rotation, expiry, original client-IP restrictions, scopes, model permissions, rate limits, budget, and retention. It invokes the real managed route with its dispatch, conversion, validation, cache, accounting and cancellation machinery. A missing or revoked account cannot execute. The private item route rejects unsigned calls.

Measured spend replaces each known hold. Missing usage, an uncertain provider result, lost transport, or an expired dispatched lease produces `outcome_unknown` and preserves the hold. That item is never dispatched again. Result-storage failures also retain the hold conservatively. Once measured plus held spend reaches the batch budget, further items fail `budget_exhausted`; an item that would exceed the remaining budget is also refused before dispatch. A pricing increase beyond its reserved estimate fails before dispatch. Finalization uses durable checkpoints so output assembly can continue after a scheduler interruption.

Zero-content retention rejects upload, batch creation and spillover with 400 `retention_conflict`. Current retention is checked again before executing stored work. Content lives only under the R2 `batches/` prefix; D1 stores ownership, opaque signing capability, control identifiers, status, spend, leases and content pointers. Scheduler warnings omit prompts, completions, credentials and transport details.

## Explicit asynchronous submission

With both flags enabled, a managed `POST /v1/chat/completions` containing both `X-MultiLLM-Priority: batch` and `Prefer: respond-async`, plus `metadata.multillm_budget_usd`, creates a real one-item batch and returns HTTP 202. Its `Location` header points to `/v1/batches/<id>`. Without both headers, or with spillover disabled, the request follows the existing synchronous path. Submission is explicit; overload does not queue a request automatically.
29 changes: 29 additions & 0 deletions intelligence-migrations/0027_gateway_batches.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
CREATE TABLE IF NOT EXISTS gateway_batch_files (
id TEXT PRIMARY KEY, owner TEXT NOT NULL, filename TEXT NOT NULL,
purpose TEXT NOT NULL CHECK (purpose IN ('batch', 'batch_output')),
bytes INTEGER NOT NULL, r2_key TEXT NOT NULL, created_at INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS gateway_batch_files_owner ON gateway_batch_files(owner, created_at, id);
CREATE TABLE IF NOT EXISTS gateway_batches (
id TEXT PRIMARY KEY, owner TEXT NOT NULL, input_file_id TEXT NOT NULL,
endpoint TEXT NOT NULL, status TEXT NOT NULL, metadata TEXT NOT NULL,
principal TEXT NOT NULL, client_ip TEXT NOT NULL, key_hash TEXT NOT NULL, key_prefix TEXT NOT NULL,
budget_units INTEGER NOT NULL CHECK (budget_units > 0), created_at INTEGER NOT NULL, expires_at INTEGER NOT NULL,
in_progress_at INTEGER, finalizing_at INTEGER, completed_at INTEGER,
cancelling_at INTEGER, cancelled_at INTEGER, expired_at INTEGER, failed_at INTEGER,
output_file_id TEXT, error_file_id TEXT, terminal_status TEXT,
finalize_cursor INTEGER NOT NULL DEFAULT 0, checkpoint TEXT,
lease_token TEXT, lease_until INTEGER
);
CREATE INDEX IF NOT EXISTS gateway_batches_owner ON gateway_batches(owner, created_at, id);
CREATE INDEX IF NOT EXISTS gateway_batches_active ON gateway_batches(status, expires_at);
CREATE TABLE IF NOT EXISTS gateway_batch_items (
batch_id TEXT NOT NULL REFERENCES gateway_batches(id), idx INTEGER NOT NULL,
owner TEXT NOT NULL, custom_id TEXT NOT NULL, status TEXT NOT NULL DEFAULT 'queued',
estimate_units INTEGER NOT NULL CHECK (estimate_units >= 0),
held_units INTEGER NOT NULL DEFAULT 0 CHECK (held_units >= 0),
cost_units INTEGER NOT NULL DEFAULT 0 CHECK (cost_units >= 0),
result_key TEXT, error_code TEXT, lease_token TEXT, lease_until INTEGER,
PRIMARY KEY (batch_id, idx), UNIQUE (batch_id, custom_id)
);
CREATE INDEX IF NOT EXISTS gateway_batch_items_claim ON gateway_batch_items(status, batch_id, idx);
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_gateway_batches_worker.mjs tests/test_semantic_generation_cache_worker.mjs tests/test_canary_traffic_worker.mjs tests/test_context_pages_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
4 changes: 2 additions & 2 deletions route_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,15 @@
"Authorization, Content-Type, Accept, Origin, X-Requested-With, X-MultiLLM-Tool-Repair, X-MultiLLM-Conversion-Report, X-MultiLLM-Retention, X-MultiLLM-Deadline-Ms, "
"X-Api-Key, X-Goog-Api-Key, X-MultiLLM-Api-Key, Anthropic-Version, "
"Anthropic-Beta, Anthropic-Dangerous-Direct-Browser-Access, "
"Idempotency-Key, X-MultiLLM-Image-QA, X-MultiLLM-Cascade, OpenAI-Beta, OpenAI-Organization, "
"Idempotency-Key, X-MultiLLM-Priority, Prefer, X-MultiLLM-Image-QA, X-MultiLLM-Cascade, OpenAI-Beta, OpenAI-Organization, "
"OpenAI-Project, Moderation, Moderation-Model, Redaction, X-App-Name, "
"X-Billing-Mode, X-BYOK-Provider, X-Client-Request-ID, X-Encryption-Key, "
"X-Encryption-Passphrase, X-Fal-Object-Lifecycle-Preference, X-PAYMENT, "
"X-Prompt-Caching-Cut-After, X-Provider, X-Team-ID, X-Use-BYOK, x-x402, "
+ ", ".join([*CLIENT_HEADER_NAMES.values(), *OPENCODE_CLIENT_HEADER_NAMES.values()])
)
CORS_EXPOSE_HEADERS = (
"Retry-After, X-Request-ID, X-MultiLLM-RateLimit-Limit, X-MultiLLM-RateLimit-Remaining, X-MultiLLM-RateLimit-Reset, X-MultiLLM-Idempotency, X-MultiLLM-Optimization, X-MultiLLM-Tool-Repair, X-MultiLLM-Image-QA, X-MultiLLM-Cascade, X-MultiLLM-Conversion-Fidelity, X-MultiLLM-Conversion-Fields, "
"Retry-After, Location, X-Request-ID, X-MultiLLM-RateLimit-Limit, X-MultiLLM-RateLimit-Remaining, X-MultiLLM-RateLimit-Reset, X-MultiLLM-Idempotency, X-MultiLLM-Optimization, X-MultiLLM-Tool-Repair, X-MultiLLM-Image-QA, X-MultiLLM-Cascade, X-MultiLLM-Conversion-Fidelity, X-MultiLLM-Conversion-Fields, "
"X-MultiLLM-Optimization-Mode, X-MultiLLM-Estimated-Input-Before, "
"X-MultiLLM-Estimated-Input-After, X-MultiLLM-Image-Prompts-Compacted, "
"X-MultiLLM-Messages-Summarized, X-MultiLLM-Optimization-Target-Met, "
Expand Down
Loading
Loading