Repository navigation
feat(costs): what a run cost, and a budget that stops it (P5) - #53
Conversation
An eval suite is the one workload where a config change costs real money nobody
notices until the invoice: a thousand rows x three repetitions x an LLM judge is
three thousand paid calls that exist purely to grade, and on a provider
dashboard they are indistinguishable from production traffic. Repeated sampling
(P1) multiplies that bill by the repetition count, so shipping precision without
cost control hands people a way to spend 3x more without telling them.
- Costs\PriceBook: config-declared rates per million tokens, longest-prefix
match after stripping a vendor/ prefix. No rates ship by default -- a stale
price baked into a package is a wrong number with a straight face. A
half-declared rate is ignored rather than half applied.
- Costs\CostLedger / RunCost / ModelCost: reported (provider billed us),
derived (computed from rates), unpriced (neither -- counted in tokens, absent
from the money, and named). isComplete()/cost.complete is the machine-readable
form of "this total is a floor, not a figure": a report that quietly says
$0.00 for a self-hosted model gets believed and quoted.
- Costs\RunBudget + --budget-usd: halts at the row that crossed the line, keeps
the rows already scored, and does not invoke the pipeline again --
SerialBatch::runEach gained a shouldStop callback checked between samples, so
the stop also prevents the next paid call.
- A halted run always exits non-zero, even with zero failures. Incomplete data
that exits zero is the worst outcome a gate can produce: the rows that would
have failed are exactly the ones that never ran.
- One budget spans repetitions and covers --outputs, where grading is the bill.
A metric that threw after calling a provider is still charged.
- Costs\Events\EvalRunCosted: the seam. Every run announces its cost under
eval:{dataset}; a FinOps package subscribes and neither side depends on the
other.
- ProviderUsageDetails now extracts the top-level model, the field that turns
token counts into money.
Tests 1066 -> 1108. composer validate --strict, phpunit, PHPStan level 6 and
Pint all green.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EZ4u82zdoa8naYm75kppMC
Deploying laravel-patent-box-tracker with
|
| Latest commit: |
a34377b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://0932f0ed.laravel-patent-box-tracker.pages.dev |
| Branch Preview URL: | https://feat-p5-finops.laravel-patent-box-tracker.pages.dev |
Deploying eval-harness with
|
| Latest commit: |
a34377b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://2d8e64c1.eval-harness.pages.dev |
| Branch Preview URL: | https://feat-p5-finops.eval-harness.pages.dev |
# Conflicts: # docs/LESSON.md
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 468d0b98c3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // A halted run is incomplete data, and incomplete data that exits zero | ||
| // is the worst outcome a gate can produce: the rows that would have | ||
| // failed are exactly the ones that never ran. | ||
| $exitCode = $report->totalFailures() === 0 && ! $report->wasHalted() ? self::SUCCESS : self::FAILURE; |
There was a problem hiding this comment.
When --budget-usd halts a run with no captured metric failures, this line sets a failure exit code but execution still continues into compareAgainstReference(), whose promotion guard checks only gate failures and totalFailures(). Therefore --promote-baseline --out=... can promote a partial report as the new baseline, causing future comparisons to use incomplete data; skip comparison/promotion after a halt or include wasHalted() in the promotion predicate.
Useful? React with 👍 / 👎.
| budget: $budget, | ||
| actualOutputs: $this->lazyParallelBatch()->run( |
There was a problem hiding this comment.
With --batch=lazy-parallel, the budget is not consulted until LazyParallelBatch::run() has returned every SUT output, so every queued pipeline invocation is dispatched and completed before scoring can halt on the first over-budget row. This contradicts the new budget guarantee that the next pipeline call is prevented and can still incur the full pipeline bill; lazy execution needs a budget-aware dispatch/scoring window or the option must be rejected or explicitly limited in this mode.
Useful? React with 👍 / 👎.
| if ($candidate === '' || ! str_starts_with($needle, $candidate)) { | ||
| continue; |
There was a problem hiding this comment.
A raw str_starts_with() treats unrelated model families as dated variants: for example, a declared gpt-4 rate also prices gpt-4o and gpt-4.1, and a declared claude-3 rate prices any future identifier beginning with those bytes. In these configurations the report derives an incorrect amount and may halt a budget at the wrong time; accept an exact match or a recognized version separator after the declared name rather than any textual prefix.
Useful? React with 👍 / 👎.
| // A metric that threw after calling a provider still spent the | ||
| // money. Charging only for successes would let a run that | ||
| // fails every judge call look free. | ||
| $budget?->record($failure->details); |
There was a problem hiding this comment.
When eval-harness.runtime.raise_exceptions is enabled and a MetricException occurs after a provider response was received—for example, an LLM judge returns billed usage but malformed score JSON—the earlier rethrow bypasses this newly added failure charge and also prevents finish() from dispatching EvalRunCosted. Thus the exact failed provider calls this feature intends to account for disappear in strict CI runs; capture the metric's usage before rethrowing and ensure aborted runs publish the incurred cost.
Useful? React with 👍 / 👎.
| // Neither tokens nor money: a latency-only detail block is not a call. | ||
| if ($promptTokens === 0 && $completionTokens === 0 && $reported === null) { | ||
| return; |
There was a problem hiding this comment.
For a provider or custom metric that reports total_tokens without separate prompt/completion counts and without cost_usd, ProviderUsageDetails accepts the usage but this guard drops it because only the split counters are examined. The resulting report claims zero calls and remains complete instead of naming the call as unpriced, undermining both cost accuracy and budget completeness; use total_tokens when detecting observable calls and retain such calls as unpriced when split-token pricing is impossible.
Useful? React with 👍 / 👎.
Fifth item of the roadmap. Stacked on #52 (P4) → #51 (P3a) → #50 (P2).
Why this belongs next to P1, not later
An eval suite is the one workload where a config change costs real money nobody notices until the invoice: a thousand rows × three repetitions × an LLM judge is three thousand paid calls that exist purely to grade, and on a provider dashboard they are indistinguishable from production traffic — same key, same model, same endpoint. So "how much are we spending on quality?" is normally unanswerable.
P1 shipped repeated sampling, which multiplies that bill by the repetition count. Shipping precision without cost control hands people a way to spend 3× more without telling them.
Reported, derived, unpriced — and the third one is the point
A cost report that quietly prints
$0.00for a self-hosted or newly-released model gets believed, budgeted against, and quoted in a meeting. So an unpriced call is named in the report — "This total is a floor, not a figure" — andRunCost::isComplete()/cost.completeis the machine-readable form for anything that gates on the number.PriceBookmatches by longest prefix after stripping avendor/prefix, sogpt-4o-minicoversgpt-4o-mini-2024-07-18(providers echo the dated form) andopenai/gpt-4o-mini, while an explicitly declared dated name still wins. A half-declared rate is ignored rather than half applied — billing one side at zero is the same failure as guessing.No rates ship by default. Provider prices change monthly; a rate baked into a package is a wrong number delivered with a straight face in somebody else's cost report.
A budget that halts, and the half that matters more
The run stops at the row that crossed the line, keeps every row already scored, and does not invoke the pipeline again —
SerialBatch::runEachgained ashouldStopcallback checked between samples, so the stop also prevents the next paid SUT call. (Before scoring would waste an answer already paid for; only after the next call would waste the call the budget existed to prevent.)A halted run always exits non-zero, even with zero failures. This is the half that matters more than the halting: incomplete data that exits zero is the worst outcome a CI gate can produce, because the rows that would have failed are disproportionately the ones that never ran. Green must mean "everything ran and everything passed", never "we ran out of money before we got to the bad rows". Same failure family as the empty-join bug review caught in #50.
Also: one budget spans repetitions (a halt in pass two ends the run rather than starting pass three with a fresh wallet);
--outputsis budgeted too, where the grading bill is the bill; and a metric that threw after calling a provider is still charged, or a run that fails every judge call looks free.The seam
Every run dispatches
EvalRunCosted— dataset,costCenter(eval:{dataset}, configurable), the fullRunCostincludingisComplete(), rows, executions, andhalted.padosoft/laravel-ai-finopssubscribes to exactly this shape; so can a home-grown ledger or a Grafana exporter. Neither package depends on the other, which is the whole reason this is an event rather than an integration. A halted run still fires it — that is precisely the run a FinOps listener most wants to hear about.Contract changes (all additive)
eval-harness.report.v1gainscostandbudgetblocksProviderUsageDetailsalso extracts the top-levelmodel— the field that turns token counts into moneyEvalReportgains?RunCost $cost/?BudgetOutcome $budgettrailing constructor params andwasHalted()run()/runBatch()/scoreOutputs()gain a trailing?float $budgetUsdSerialBatch::runEach()gains a trailing?callable $shouldStopeval-harness.costs(emptymodelsby default)Verification
composer validate --strict,phpunit, PHPStan level 6, Pint — all green locally.docs-site/docs/guides/cost-and-budgets.md+ nav; README feature bullet; config block carrying the reasoning inline;docs/PROGRESS.md+docs/LESSON.md.🤖 Generated with Claude Code
https://claude.ai/code/session_01EZ4u82zdoa8naYm75kppMC
Generated by Claude Code