Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions docs/prompt-cache-cost.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Prompt cache usage and price buckets

`PROMPT_CACHE_USAGE_BUCKETS_ENABLED` defaults to false. Empty values use the default.
Malformed flags or metadata disable buckets and produce one warning per setting without its value.
Disabled deployments use the existing rows, SQL, costs and protocol usage translations; migration 0016 is unnecessary.

Enable with `PROMPT_CACHE_USAGE_BUCKETS_ENABLED=true` after applying `0016_usage_buckets.sql`.
`MODEL_PRICING_USD_PER_MILLION` keeps input/output/request prices and accepts explicit
`cache_read` and `cache_write` prices per million tokens. For example:

```json
{"openai:example":{"input":2,"output":8,"cache_read":0.5,"cache_write":3}}
```

`PROMPT_CACHE_PRICE_METADATA_JSON` defaults to `{}`. It may supply missing bucket prices
for an explicitly priced model, using the same model keys and price names. Explicit
prices win, including zero and invalid prices. Metadata alone cannot price a model.
No provider-wide discount factors are assumed.

OpenAI prompt/input counts include cache tokens. Anthropic input counts exclude reads
and writes. Only these explicit usage conventions are normalized; unknown cache reads
remain null. An OpenAI adapter has no cache-write category unless `cache_write_tokens`
is explicitly reported. Ordinary input is null when the inclusive total cannot safely
be split. Negative, boolean, string and counts above 2^53-1 remain unknown.
SSE observations merge valid components across events with a 64 KiB line bound.

Rows add nullable ordinary-input/cache-read/cache-write counts alongside the existing
output count, four costs in microUSD, `bucket_basis` (measured/estimated/unknown) and
`bucket_source` (openai/anthropic/request_estimate/unknown). No missing count becomes
zero. A measured zero costs zero without a price. Unknown positive-count prices leave
the total null while known component costs remain visible. Decimal arithmetic is bounded
at 10^12 microUSD per component and total (one million USD). Costs may contain fractional
microUSD; totals are computed before conversion to floats. Estimates are not invoices.
Gateway response-cache hits have no provider charge.

SQLite adds nullable columns locally only when enabled and keeps old rows readable.
The control-plane PostgreSQL adapter also adds nullable columns when enabled; it has
not been validated against a live PostgreSQL database. D1 needs migration 0016 before
enabling the feature. The migration adds nullable columns to the existing `usage_events`
table and never replaces it.

In D1 mode the Worker's private ledger writes the bucket columns in the same batch
transaction as the base row, with the same batch guard and daily rollups. If the
extended write fails, the base row is still saved and the call returns HTTP 503 JSON
`usage_buckets_unavailable` with `cache-control: no-store`. A retry with the same batch
ID does not duplicate the base row. This is a private storage error, not a reason to
replay generation. Set `PROMPT_CACHE_USAGE_BUCKETS_ENABLED` on both the Worker and the
Container. Native Worker generations record base usage only, without cache buckets.

Buckets follow the existing usage-event retention and contain no prompt, response or key
content. Daily rollups retain their existing total-cost schema; per-bucket history is in
raw events. No billing, retention policy, live-provider metering or migration deployment
is performed here.
10 changes: 10 additions & 0 deletions intelligence-migrations/0016_usage_buckets.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
-- Nullable prompt cache buckets on the 0007 usage ledger; output_tokens already exists.
ALTER TABLE usage_events ADD COLUMN ordinary_input_tokens INTEGER;
ALTER TABLE usage_events ADD COLUMN cache_read_input_tokens INTEGER;
ALTER TABLE usage_events ADD COLUMN cache_write_input_tokens INTEGER;
ALTER TABLE usage_events ADD COLUMN ordinary_input_cost_microusd REAL;
ALTER TABLE usage_events ADD COLUMN cache_read_input_cost_microusd REAL;
ALTER TABLE usage_events ADD COLUMN cache_write_input_cost_microusd REAL;
ALTER TABLE usage_events ADD COLUMN output_cost_microusd REAL;
ALTER TABLE usage_events ADD COLUMN bucket_basis TEXT;
ALTER TABLE usage_events ADD COLUMN bucket_source TEXT;
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_prompt_cache_cost_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
7 changes: 7 additions & 0 deletions services/cost_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,3 +131,10 @@ def estimate(
+ Decimal(safe_output_tokens) * prices["output"]
) / Decimal(1_000_000) + Decimal(cls._token_count(requests)) * prices.get("request", Decimal(0))
return float(total.quantize(Decimal("0.0000000001")))

@classmethod
def price_buckets(cls, model_id, usage, *, requests: int = 1) -> dict:
"""Return nullable bucket costs without changing legacy estimate semantics."""
from services.prompt_cache_cost import price_buckets

return price_buckets(model_id, usage, requests=requests)
Loading
Loading