Repository navigation
feat: land P3a, P4, P5 and P6 on main - #55
Merged
Merged
Conversation
… (P3a) Every metric in this package scored a string. That is the right unit for a RAG pipeline and the wrong one for an agent: the final answer is the one part a broken agent can still get right by accident. An agent that replies "your order ships Tuesday" without ever calling the order lookup has guessed — it will be correct often enough to look healthy, every text metric here scores that guess 1.0, and the day it is wrong it will be wrong with total confidence. Adds a Trajectory DTO (tool calls with arguments, order, steps, finish reason, approvals), a recorder the system under test writes into, and seven metrics that read it: tool-called, tool-not-called, tool-called-with, tool-call-order, steps-below, no-pending-approvals, approval-gated. Three decisions: - The DTO is SDK-agnostic. The obvious implementation accepts whatever the agent SDK returns, which is what the closest comparable tool does, and it means those assertions exist only inside that one SDK. This one is populated from laravel/ai, a custom orchestrator, MCP, laravel-flow saga steps, or a JSON file recorded last week — the adapter changes per runtime, the assertions do not. The laravel/ai adapter lives in a satellite package so this one stays free of an SDK dependency, which is design rationale #1 of the harness. - The Metric contract is untouched. TrajectoryMetric extends Metric and the engine hands a trajectory only to that family; widening the contract for one family would have changed every metric in the package and every one a host has written. - A missing trajectory raises a captured MetricException rather than scoring 0 or 1. Scoring 0 blames the agent for the harness's missing wiring; scoring 1 lets a whole dataset go green because nobody plugged the recorder in. Arguments match as a subset and order as a subsequence, so adding a trace id or an extra tool does not fail an eval that was about something else. Saved outputs carry an optional trajectory block, which makes the whole family usable offline with no agent runtime at all. approval-gated answers a question no text metric can: did the agent ask before it acted? Where an agent can spend money or change a record, an unapproved action is a compliance finding, not a UX detail. Tests: 1018 (was 981), 2792 assertions. PHPStan level 6 clean. Co-Authored-By: Claude <[email protected]>
…s acted on A red CI job says an evaluation failed. The JSON report that would say what to do about it is a thousand lines nobody opens, so the loop breaks in the same place every time. - Brief\RunBriefing: failing rows worst-first (worst execution only, so repetitions do not eat the budget) with input, expected, actual output, metric scores, the judge's own reason, trajectory summary and metrics that threw. - Brief\MetricGlossary: one line of semantics per built-in metric. An assertion describes itself; a metric has meaning. "retrieval-mrr: 0.31" becomes "the first relevant document came back around position 3", which points at the retriever rather than the prompt. Unknown names get no description, not a guess. - Cohort diagnosis when one tag carries most of the failures, and adversarial rows named as safety findings with category, severity and framework. - The prompt-injection guard: the document quotes model output verbatim into something with repository access, so it declares its quoted blocks untrusted data before the first fence, and a value containing a fence gets a longer one. - --dataset= supplies the question and golden answer, joined by content hash first and id second. The report records what the pipeline said, never what it was asked, and copying corpora into a committed artifact is not a trade worth making. Without it the briefing says so; it never invents a question. - --format=github as a collapsed PR comment, --format=json as eval-harness.brief.v1 carrying both the structure and the markdown. - Truncation is declared with counts; an artifact that is not a report is refused rather than briefed as "every row passed". Tests 1031 -> 1066. composer validate --strict, phpunit, PHPStan level 6 and Pint all green. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EZ4u82zdoa8naYm75kppMC
An eval suite is the one workload where a config change costs real money nobody
notices until the invoice: a thousand rows x three repetitions x an LLM judge is
three thousand paid calls that exist purely to grade, and on a provider
dashboard they are indistinguishable from production traffic. Repeated sampling
(P1) multiplies that bill by the repetition count, so shipping precision without
cost control hands people a way to spend 3x more without telling them.
- Costs\PriceBook: config-declared rates per million tokens, longest-prefix
match after stripping a vendor/ prefix. No rates ship by default -- a stale
price baked into a package is a wrong number with a straight face. A
half-declared rate is ignored rather than half applied.
- Costs\CostLedger / RunCost / ModelCost: reported (provider billed us),
derived (computed from rates), unpriced (neither -- counted in tokens, absent
from the money, and named). isComplete()/cost.complete is the machine-readable
form of "this total is a floor, not a figure": a report that quietly says
$0.00 for a self-hosted model gets believed and quoted.
- Costs\RunBudget + --budget-usd: halts at the row that crossed the line, keeps
the rows already scored, and does not invoke the pipeline again --
SerialBatch::runEach gained a shouldStop callback checked between samples, so
the stop also prevents the next paid call.
- A halted run always exits non-zero, even with zero failures. Incomplete data
that exits zero is the worst outcome a gate can produce: the rows that would
have failed are exactly the ones that never ran.
- One budget spans repetitions and covers --outputs, where grading is the bill.
A metric that threw after calling a provider is still charged.
- Costs\Events\EvalRunCosted: the seam. Every run announces its cost under
eval:{dataset}; a FinOps package subscribes and neither side depends on the
other.
- ProviderUsageDetails now extracts the top-level model, the field that turns
token counts into money.
Tests 1066 -> 1108. composer validate --strict, phpunit, PHPStan level 6 and
Pint all green.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EZ4u82zdoa8naYm75kppMC
# Conflicts: # docs/LESSON.md
Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EZ4u82zdoa8naYm75kppMC
# Conflicts: # docs/LESSON.md
A golden dataset written at a desk encodes what its authors already thought to worry about. The questions that actually break a system arrive at 11pm from a real user, and online monitoring was already scoring them -- nothing turned them into tests. - The interaction has to be kept first: eval_harness_online_scores stored a score, not the text. Additive migration adds nullable input/expected_output/ actual_output/redactor/redacted_at, filled only when retention is enabled. Keeping production text is a different decision, with a different legal basis, from keeping a number -- so it is a different switch, and it is off. - Online\Redactor: one method, text in, text out. This package ships none. A redactor trustworthy with production text is a package of its own, and an eval harness with its own half-good PII regex is a package quietly promising a compliance property it cannot keep. - Online\InteractionRetention: require_redactor defaults on, so retention with nothing bound raises rather than storing raw text -- the failure that actually happens is "enabled in a hurry, redactor next sprint, six months of customer questions in a table". A throwing redactor drops the interaction, and the exception deliberately does not carry the text it failed on. Nested input is walked at any depth, string keys included. - Redaction happens at the boundary, in the job, never written raw and cleaned up later: later is after the backup, the replica and the log shipper each took a copy. - eval-harness:promote-online: failures by default, --all/--max-score/--since/ --limit (which keeps the worst-scoring rows), --merge writing back in place so the diff is pure additions. - Dedup by RowHash, the same content hash the regression gate joins on: a nightly promotion is idempotent, a hand-written row covering the case is not duplicated, and a promoted row keeps its history from the moment it lands. Row ids derive from that hash, not the database id, so promoting from a second environment does not renumber a committed dataset. - Unredacted rows are skipped unless --allow-unredacted, and the skip is announced. A corrupt --merge target stops the command rather than replacing a curated dataset with this run's promotions. Tests 1111 -> 1136. composer validate --strict, phpunit, PHPStan level 6 and Pint all green. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EZ4u82zdoa8naYm75kppMC
feat(online): production failures become regression tests, safely (P6)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge-train tip. #50 (P2) is already on main; #51, #52, #53 and #54 each merged into their stacked base, so this brings the remaining four onto main in one go.
TrajectoryDTO + seven tool-call metrics — score how an answer was produced, not just the answereval-harness:brief— a failed run as a document a person reads in a minute and a coding agent acts on, with the prompt-injection guard on its own output--budget-usdthat halts, and theEvalRunCostedFinOps seameval-harness:promote-online— production failures become regression tests, behind a required PII redactorEach was reviewed and merged individually; nothing new is introduced here.
Verification on the tip
composer validate --strict,phpunit(1136 tests, 3078 assertions), PHPStan level 6, Pint — all green locally.🤖 Generated with Claude Code
https://claude.ai/code/session_01EZ4u82zdoa8naYm75kppMC
Generated by Claude Code