Repository navigation
feat(budget): add opt-in durable ambiguous usage holds - #44
Conversation
Track each budgeted request through reserve, dispatched, settled or unknown, and reconciled states behind USAGE_RESERVATIONS_ENABLED (default off). An unknown completion keeps a conservative estimated hold instead of a zero charge or a timed refund; USAGE_HOLD_REVIEW_AFTER_SECONDS marks old holds for review, and admins reconcile with measured evidence or an audited adjustment. Reservations are stored in D1 (migration 0020).
# Conflicts: # package.json # worker/container-env.mjs # worker/d1-schema.mjs
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
multillm-proxy | 4194733 | Oct 09 2026, 12:38 AM |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 41947335e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if current["state"] == "reserved": | ||
| cls.mark_dispatched(reservation) | ||
| current = store.get(reservation) |
There was a problem hiding this comment.
Release reservations that never reached dispatch
When a route rejects locally after request_accounting.begin()—for example, payload validation returns 400 before any provider call—the reservation is still reserved, but this branch marks it dispatched and then transitions it to unknown. Because unknown holds never expire, ordinary client errors permanently consume the key's budget and require administrator reconciliation; a still-reserved completion should instead settle at zero because the dispatch hook was never reached.
Useful? React with 👍 / 👎.
| if reservation_store.enabled(): | ||
| if before_dispatch: | ||
| store = reservation_store.get_store() | ||
| row = store.get(reservation) | ||
| store.transition(reservation, row["revision"], "settled", cost_usd=0, basis="released", | ||
| settlement_id=reservation, transition_id=uuid.uuid4().hex) | ||
| return |
There was a problem hiding this comment.
Preserve pre-dispatch releases for outer accounting
With reservations enabled, settle(reservation) now silently returns unless before_dispatch=True, but services/accounted_dispatch.py:20 still uses the old call when cascades, image quality, and quality-checked media replace an aggregate request with subrequests. Those flows therefore retain the unused outer hold while creating additional holds for each subrequest, eventually exhausting the user's budget even after successful settlement.
Useful? React with 👍 / 👎.
| if reservation_store.enabled(): | ||
| return cls._reserve_durable(user, estimate_usd, now) |
There was a problem hiding this comment.
Reject unpriced gateway-owned subrequests
The durable path only rejects an unpriced request when it receives None, but services/accounted_dispatch.py:43 continues to call estimate_cost(), which converts a missing model price to 0.0. With reservations enabled, an unpriced cascade, image-quality, or shadow-evaluation subrequest therefore receives a zero-dollar reservation and bypasses budget enforcement instead of returning unpriced_reservation as other request paths do.
Useful? React with 👍 / 👎.
| store.transition(identity,0,'dispatched',transition_id='${id(2)}',now=now) | ||
| store.transition(identity,1,'unknown',transition_id='${id(3)}',input_tokens=0,now=now) | ||
| print(json.dumps([store.get(identity),store.summary('alice',now)]))`; | ||
| const python = spawnSync("/home/mysterious/storage/github/MultiLLM-Proxy/.venv/bin/python", ["-I", "-c", code], { |
There was a problem hiding this comment.
Invoke Python through a portable interpreter path
This newly registered worker test hard-codes an author-specific virtualenv path. On any CI runner or checkout not located under /home/mysterious/storage/github/MultiLLM-Proxy, spawnSync cannot start the process, returns a null status, and makes npm run test:worker fail before comparing the contracts; use the repository/environment Python executable instead.
Useful? React with 👍 / 👎.
| export class ReservationError extends Error { | ||
| constructor(code = "usage_reservations_unavailable", status = 503) { super(code); this.code = code; this.status = status; } | ||
| response() { return reply({ error: { code: this.code, message: "The usage reservation operation could not be completed." } }, this.status); } |
There was a problem hiding this comment.
Convert reservation failures into native gateway responses
When this lifecycle is registered with nativeGenerationFetch, admission or handoff failures throw ReservationError, but the gateway catch path only converts AdmissionError instances into responses and rethrows every other error. A missing reservation table, unavailable transport, or budget denial will therefore escape as an unhandled worker error instead of the promised bounded JSON 503 or 429 response, even though ReservationError.response() exists.
Useful? React with 👍 / 👎.
| if reservation_store.enabled(): | ||
| durable = cls._durable_totals(user, now) |
There was a problem hiding this comment.
Skip durable status reads for keys without budgets
When the global flag is enabled, BudgetService.status() selects the reservation authority for every principal, even if both monetary limits are unset. Consequently an unbudgeted key's /v1/usage response gains reservation fields and becomes dependent on the new storage—reporting unavailable totals when that storage is down—despite the stated guarantee that keys without monetary budgets remain unchanged.
Useful? React with 👍 / 👎.
| const result = await call({ version: 1, operation: "reserve", id: crypto.randomUUID().replaceAll("-", ""), ...values }); | ||
| if (result.error) throw new ReservationError(result.error.code, result.error.code === "budget_exceeded" ? 429 : 503); | ||
| reservation = result.reservation; |
There was a problem hiding this comment.
Reject malformed reservation replies before dispatch
If the injected call transport resolves to a non-error object with a missing or malformed reservation field—for example, an adapter accidentally returns an undecoded Response or a truncated contract object—admission assigns undefined and returns successfully. before_dispatch() then becomes a no-op, so the provider request proceeds without any durable hold; validate the version and reservation shape and fail closed as the Python D1 client does.
Useful? React with 👍 / 👎.
59e6e45
into
feat/shared-generation-cache
Summary
Adds opt-in durable usage holds for dollar-budgeted keys behind
USAGE_RESERVATIONS_ENABLED, which defaults to false. With the flag off, accounting, responses, headers and storage are unchanged. Requests without a monetary budget are unchanged even with the flag on.reserved→dispatched, then tosettled(measured provider cost) or tounknown(completion or usage uncertain).budget_exceeded.unpriced_reservation.USAGE_HOLD_REVIEW_AFTER_SECONDS(default three days) marks an unresolved holdneeds_review. It never releases money./v1/usage:budget.reservationscounters.in_flight_usdincludes unresolved holds.USAGE_DB_PATH.usage_reservation_budgets,usage_reservationsandusage_reservation_transitions(migration0020) through the privatehttp://intelligence.internal/v1/reservationsendpoint.usage_reservations_unavailablebefore provider submission.Details, including the operator steps before first enabling it, are in
docs/usage-reservations.md.Glue left for the Wave 4 integration PR
request_accounting.mark_dispatchedimmediately before provider submission. The existing cancellation observer also records the handoff.createReservationLifecycle./v1/reservationsendpoint, and adds the README line. There is no public reconciliation endpoint yet.Stacking
This PR is based on
feat/shared-generation-cache(#43), so migrations0018–0020stay contiguous. Review only the top commit,feat(budget): add opt-in durable ambiguous usage holds; the merge commit brings in #43.Deployment
Apply the migration before enabling the flag. A push to main does not apply D1 migrations:
Shared registrations
tests/test_usage_reservations_worker.mjstotest:worker.USAGE_RESERVATIONS_ENABLEDandUSAGE_HOLD_REVIEW_AFTER_SECONDSto the Container env keys.worker/reservations-d1.mjsto the secret firewall inventory.Tests
tests/test_usage_reservations.pyandtests/test_usage_reservations_worker.mjs.npm run test:worker: 1,399 passed.npm run test:integration: 7 passed.cf:migrations:checkreports 20 contiguous, additive migrations.test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.CI notes
python/nodejobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.AdmissionCoordinatorDurable Object migration, whichwrangler versions uploadcannot apply. This is expected untilwrangler deployruns on main.This is part of Wave 4.