Skip to content

feat(batches): add an opt-in gateway Batch API with explicit spillover - #57

Merged
ALikesToCode merged 2 commits into
feat/semantic-cachefrom
feat/gateway-batches
Oct 9, 2026
Merged

ALikesToCode merged 2 commits into
feat/semantic-cachefrom
feat/gateway-batches

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds an opt-in, OpenAI-compatible Batch API with owned files and explicit asynchronous spillover. With GATEWAY_BATCHES_ENABLED and BATCH_SPILLOVER_ENABLED off (the defaults), the new routes return 404, and existing routes, the scheduled handler and storage are unchanged.

  • Files:
    • /v1/files (list and upload), /v1/files/<id> (metadata and delete) and /v1/files/<id>/content.
    • JSONL input: at most 1,000 items and 10 MiB. Each line needs a unique custom_id, method: "POST", a supported url and a non-streaming body.
    • Invalid input returns 400 invalid_file with the first bad line number.
    • Every operation is limited to the authenticated owner, administrators included. Another owner's ID returns 404. A file used by an active batch can't be deleted (409 file_in_use).
  • Batches:
    • /v1/batches (list and create), /v1/batches/<id> and /v1/batches/<id>/cancel. Supported endpoints are /v1/chat/completions and /v1/responses.
    • A budget is required in metadata.multillm_budget_usd.
    • Every model must have a known price, or creation fails with 400 unknown_price.
    • Admission reserves the largest output bound (262,144 tokens when no limit is set).
    • Model permissions apply at creation and again at execution.
    • States: validating → in_progress → finalizing → completed. Batches can also end cancelling → cancelled, or expired after 24 hours.
  • Execution:
    • The existing scheduled handler claims at most two items per run, with an atomic D1 lease and spend hold. Each item has a 30 s deadline.
    • The Container receives an expiring, signed owner capability and an item lease, never the owner's API key.
    • Before running an item, it consumes the lease once and reloads the owner's current account. It then checks key rotation and expiry, client IP restrictions, scopes, model permissions, rate limits, budget and retention.
    • Items run through the real managed route.
    • The private item route (/internal/gateway/batch-item) rejects unsigned calls, and the Worker already blocks public /internal paths.
  • Spend:
    • Measured usage replaces each hold.
    • An uncertain outcome (missing usage, lost transport, an expired dispatched lease) is recorded as outcome_unknown. It keeps its hold and is never dispatched again.
    • Once measured plus held spend reaches the budget, remaining items fail with budget_exhausted.
  • Spillover: with both flags on, a managed POST /v1/chat/completions that carries both X-MultiLLM-Priority: batch and Prefer: respond-async, plus a budget, creates a real one-item batch. It returns 202 with Location: /v1/batches/<id>. Nothing is queued automatically, and the gateway never returns a fake completion.
  • Retention: zero-content retention rejects upload, batch creation and spillover with 400 retention_conflict. Content lives only under the R2 batches/ prefix. D1 stores ownership, status, spend, leases and pointers.

Details are in docs/gateway-batches.md.

Shared registrations

  • /v1/gateway-batches is mounted on the private intelligence.internal outbound handler, next to /v1/reservations. The handler identity in outboundByHost is unchanged.
  • The scheduled handler runs runScheduledBatches only while GATEWAY_BATCHES_ENABLED is on.
  • tests/test_gateway_batches_worker.mjs in test:worker, plus a test loader mapping for worker/batch-jobs.mjs.
  • worker/batch-files.mjs and worker/batch-jobs.mjs in the edge module inventory. The seven new Flask routes are in the route inventory.
  • GATEWAY_BATCHES_ENABLED and BATCH_SPILLOVER_ENABLED in the Container env keys.
  • gateway_batch_files, gateway_batches and gateway_batch_items in REQUIRED_D1_TABLES.
  • CORS:
    • X-MultiLLM-Priority and Prefer are allowed request headers;
    • Location is exposed in both lists.

Stacked on #56

Migration numbers must be contiguous in each PR, so this branch stacks on feat/semantic-cache (#56, migration 0026) and adds 0027_gateway_batches.sql. Merge #50, #54 and #56 first.

Left for the Wave 5 integration

  • Call register_gateway_batch_routes(app) in app.py after register_gateway_extensions. The registrar inserts the spillover hook right after retention. Until then, the Flask routes aren't registered.
  • README line.

Tests

  • tests/test_gateway_batches.py and tests/test_gateway_batches_worker.mjs cover:
    • flags off and malformed values never touching storage or a provider;
    • file validation limits and the first bad line;
    • foreign-owner privacy and input deletion protection;
    • budget parsing, token bounds and unknown prices;
    • concurrent schedulers claiming each item once with atomic holds;
    • a capability starting an item only once;
    • expired or IP-restricted owners, permissions and budget checked before any provider call;
    • cancellation races, 24-hour expiry and resumable finalization checkpoints;
    • lost usage, lost results and the 30 s deadline keeping an outcome_unknown hold without redispatch;
    • explicit spillover through the registered routes;
    • zero-retention conflicts;
    • an additive, repeatable migration.
  • Full Python suite: 4,093 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,545 passed. npm run test:integration: 7 passed.
  • migrations:check (27 contiguous, additive migrations), test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

Apply the migration before enabling the flag: npx wrangler d1 migrations apply multillm-intelligence --remote.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 5. The README line comes with the Wave 5 integration PR.

OpenAI-compatible /v1/files and /v1/batches backed by D1 and R2, executed item by item through the managed chat route by the scheduled handler. Explicit spillover turns a chat request with X-MultiLLM-Priority: batch and Prefer: respond-async into a one-item batch. Both flags default to off.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d8c9a346-9328-423b-a749-c56df9981936

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy ab9445f Oct 09 2026, 02:50 AM

@ALikesToCode
ALikesToCode merged commit 86572f2 into feat/semantic-cache Oct 9, 2026
4 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant