Repository navigation
feat(batches): add an opt-in gateway Batch API with explicit spillover - #57
Conversation
OpenAI-compatible /v1/files and /v1/batches backed by D1 and R2, executed item by item through the managed chat route by the scheduled handler. Explicit spillover turns a chat request with X-MultiLLM-Priority: batch and Prefer: respond-async into a one-item batch. Both flags default to off.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
multillm-proxy | ab9445f | Oct 09 2026, 02:50 AM |
Summary
Adds an opt-in, OpenAI-compatible Batch API with owned files and explicit asynchronous spillover. With
GATEWAY_BATCHES_ENABLEDandBATCH_SPILLOVER_ENABLEDoff (the defaults), the new routes return 404, and existing routes, the scheduled handler and storage are unchanged./v1/files(list and upload),/v1/files/<id>(metadata and delete) and/v1/files/<id>/content.custom_id,method: "POST", a supportedurland a non-streamingbody.invalid_filewith the first bad line number.file_in_use)./v1/batches(list and create),/v1/batches/<id>and/v1/batches/<id>/cancel. Supported endpoints are/v1/chat/completionsand/v1/responses.metadata.multillm_budget_usd.unknown_price.validating→in_progress→finalizing→completed. Batches can also endcancelling→cancelled, orexpiredafter 24 hours./internal/gateway/batch-item) rejects unsigned calls, and the Worker already blocks public/internalpaths.outcome_unknown. It keeps its hold and is never dispatched again.budget_exhausted.POST /v1/chat/completionsthat carries bothX-MultiLLM-Priority: batchandPrefer: respond-async, plus a budget, creates a real one-item batch. It returns 202 withLocation: /v1/batches/<id>. Nothing is queued automatically, and the gateway never returns a fake completion.retention_conflict. Content lives only under the R2batches/prefix. D1 stores ownership, status, spend, leases and pointers.Details are in
docs/gateway-batches.md.Shared registrations
/v1/gateway-batchesis mounted on the privateintelligence.internaloutbound handler, next to/v1/reservations. The handler identity inoutboundByHostis unchanged.runScheduledBatchesonly whileGATEWAY_BATCHES_ENABLEDis on.tests/test_gateway_batches_worker.mjsintest:worker, plus a test loader mapping forworker/batch-jobs.mjs.worker/batch-files.mjsandworker/batch-jobs.mjsin the edge module inventory. The seven new Flask routes are in the route inventory.GATEWAY_BATCHES_ENABLEDandBATCH_SPILLOVER_ENABLEDin the Container env keys.gateway_batch_files,gateway_batchesandgateway_batch_itemsinREQUIRED_D1_TABLES.X-MultiLLM-PriorityandPreferare allowed request headers;Locationis exposed in both lists.Stacked on #56
Migration numbers must be contiguous in each PR, so this branch stacks on
feat/semantic-cache(#56, migration 0026) and adds0027_gateway_batches.sql. Merge #50, #54 and #56 first.Left for the Wave 5 integration
register_gateway_batch_routes(app)inapp.pyafterregister_gateway_extensions. The registrar inserts the spillover hook right after retention. Until then, the Flask routes aren't registered.Tests
tests/test_gateway_batches.pyandtests/test_gateway_batches_worker.mjscover:outcome_unknownhold without redispatch;npm run test:worker: 1,545 passed.npm run test:integration: 7 passed.migrations:check(27 contiguous, additive migrations),test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.Apply the migration before enabling the flag:
npx wrangler d1 migrations apply multillm-intelligence --remote.CI notes
python/nodejobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.AdmissionCoordinatorDurable Object migration, whichwrangler versions uploadcannot apply. This is expected untilwrangler deployruns on main.This is part of Wave 5. The README line comes with the Wave 5 integration PR.