Repository navigation
feat(usage): add opt-in prompt cache usage and cost buckets - #34
Conversation
Record ordinary input, cache-read, cache-write and output tokens and costs separately behind PROMPT_CACHE_USAGE_BUCKETS_ENABLED, default off, with cache_read/cache_write prices and migration 0016 adding nullable usage_events columns.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
multillm-proxy | 2bef0d4 | Oct 08 2026, 10:14 PM |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2bef0d40eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "request_id": context.request_id, | ||
| } | ||
| if prompt_cache_cost.enabled() and context.kind in {"chat", "responses", "proxy", "embeddings"}: | ||
| breakdown = CostService.price_buckets(selected or requested, usage, requests=context.units) |
There was a problem hiding this comment.
Preserve estimates when usage is absent
When buckets are enabled and a successful provider response contains no usage object, the estimated observation created above is discarded and price_buckets receives None. It therefore overwrites the legacy estimated cost with None, and _record does not perform its partial-usage fallback because usage is None; the request's reservation is then released without recording any spend. Providers that omit usage can consequently make repeated requests without consuming configured budgets, so price the request from the estimated observation as the disabled path did.
Useful? React with 👍 / 👎.
| metadata = _metadata(os.environ.get("PROMPT_CACHE_PRICE_METADATA_JSON", "").strip()) or {} | ||
| supplement = metadata.get(model, {}) |
There was a problem hiding this comment.
Apply wildcard lookup to cache-price metadata
When an operator uses the supported wildcard pricing pattern, such as MODEL_PRICING_USD_PER_MILLION={"openai:*":...} with cache prices under the same openai:* key in metadata, _entry finds the explicit price but this exact-model lookup returns no supplement. Any positive cached-token count then leaves the total cost unknown despite the configured cache price. Normalize the metadata keys and resolve them with the same exact/provider/global wildcard precedence as the main pricing table.
Useful? React with 👍 / 👎.
| details = usage.get("prompt_tokens_details", usage.get("input_tokens_details")) | ||
| details = details if isinstance(details, dict) else {} | ||
| raw = count(typed.input_tokens) | ||
| read = count(usage.get("cache_read_input_tokens")) if anthropic else count(details.get("cached_tokens")) | ||
| write = count(usage.get("cache_creation_input_tokens")) if anthropic else count(details.get("cache_write_tokens", 0 if raw is not None or details else None)) |
There was a problem hiding this comment.
Preserve cache-count uncertainty through Responses translation
When protocol translation runs before accounting, responses_usage_to_chat omits prompt details for an explicitly reported cached_tokens: 0, while chat_usage_to_responses emits cached_tokens: 0 when the source omitted cache details. This parser consequently turns a measured zero into unknown in one direction and an unknown count into measured zero in the other, making the ordinary-input bucket and total cost depend on the route's translation direction rather than provider data. Preserve the null-versus-zero distinction through the Responses usage translators when buckets are enabled.
Useful? React with 👍 / 👎.
75a8a53
into
integrate/gateway-waves-1-2
Summary
Splits prompt-cache usage and cost into separate buckets behind
PROMPT_CACHE_USAGE_BUCKETS_ENABLED, which defaults to false. While the flag is off, usage rows, SQL, costs and the Anthropic/OpenAI usage translations are unchanged, and migration 0016 is not needed.bucket_basis(measured, estimated or unknown) and abucket_source.MODEL_PRICING_USD_PER_MILLIONacceptscache_readandcache_writeprices.PROMPT_CACHE_PRICE_METADATA_JSONcan fill missing bucket prices for an explicitly priced model.0016_usage_buckets.sql.worker/usage-buckets-d1.mjsadds the Worker serializer and the extended insert for the D1 ledger.Details are in
docs/prompt-cache-cost.md.Deployment
Apply
0016_usage_buckets.sqlbefore enabling the flag. A push to main does not apply D1 migrations:The migration only adds nullable columns to
usage_events.Integration
The Wave 3 integration PR wires the serializer into the Worker ledger's
recordandrecentoperations. Until then, leave the flag off in D1 mode.Shared registrations
tests/test_prompt_cache_cost_worker.mjstotest:worker.PROMPT_CACHE_USAGE_BUCKETS_ENABLEDandPROMPT_CACHE_PRICE_METADATA_JSONto the Container env keys.worker/usage-buckets-d1.mjsto the secret firewall inventory.test_d1_schema_worker,test_control_users_worker,test_d1_integration) now skip 0016 along with 0007, because 0016 alters 0007's table.Tests
tests/test_prompt_cache_cost.pyandtests/test_prompt_cache_cost_worker.mjs: new.npm run test:worker: 1,231 passed.npm run test:integration: 6 passed, run with the virtualenvpython3first on PATH (with the systempython3it fails the same way on the base branch, because Flask is missing).cf:migrations:checkreports 16 contiguous, additive migrations.test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.This is part of Wave 3. It is based on the Wave 2 integration branch (#25), and the Wave 3 integration PR adds the README line.