Repository navigation
feat(brief): turn a failed run into a document that gets acted on (P4) - #52
Conversation
…s acted on A red CI job says an evaluation failed. The JSON report that would say what to do about it is a thousand lines nobody opens, so the loop breaks in the same place every time. - Brief\RunBriefing: failing rows worst-first (worst execution only, so repetitions do not eat the budget) with input, expected, actual output, metric scores, the judge's own reason, trajectory summary and metrics that threw. - Brief\MetricGlossary: one line of semantics per built-in metric. An assertion describes itself; a metric has meaning. "retrieval-mrr: 0.31" becomes "the first relevant document came back around position 3", which points at the retriever rather than the prompt. Unknown names get no description, not a guess. - Cohort diagnosis when one tag carries most of the failures, and adversarial rows named as safety findings with category, severity and framework. - The prompt-injection guard: the document quotes model output verbatim into something with repository access, so it declares its quoted blocks untrusted data before the first fence, and a value containing a fence gets a longer one. - --dataset= supplies the question and golden answer, joined by content hash first and id second. The report records what the pipeline said, never what it was asked, and copying corpora into a committed artifact is not a trade worth making. Without it the briefing says so; it never invents a question. - --format=github as a collapsed PR comment, --format=json as eval-harness.brief.v1 carrying both the structure and the markdown. - Truncation is declared with counts; an artifact that is not a report is refused rather than briefed as "every row passed". Tests 1031 -> 1066. composer validate --strict, phpunit, PHPStan level 6 and Pint all green. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EZ4u82zdoa8naYm75kppMC
Deploying laravel-patent-box-tracker with
|
| Latest commit: |
5042732
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://fa7375da.laravel-patent-box-tracker.pages.dev |
| Branch Preview URL: | https://feat-p4-brief.laravel-patent-box-tracker.pages.dev |
Deploying eval-harness with
|
| Latest commit: |
5042732
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://3a510aa0.eval-harness.pages.dev |
| Branch Preview URL: | https://feat-p4-brief.eval-harness.pages.dev |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5a95fa24e0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // The single most useful line in the whole document when a judge is | ||
| // involved: the model saying, in prose, what was wrong. | ||
| if (is_string($details['judge_reason'] ?? null) && $details['judge_reason'] !== '') { | ||
| $md .= sprintf(" - judge: %s\n", $this->oneLine($details['judge_reason'])); |
There was a problem hiding this comment.
When llm-as-judge evaluates attacker-controlled content, judge_reason is itself model-generated and may contain prompt-injection text. This line emits it as ordinary Markdown after only collapsing whitespace, while the briefing's safety preamble specifically identifies fenced blocks as untrusted data; a coding agent receiving the briefing therefore sees this model output outside the declared boundary. Render the reason inside a safely sized fence or another explicit untrusted-data block.
Useful? React with 👍 / 👎.
| // A value containing a fence would close the block early and the rest | ||
| // would read as prose — which for model output is exactly the boundary | ||
| // this document exists to keep. | ||
| $ticks = str_contains($value, '```') ? '````' : '```'; |
There was a problem hiding this comment.
When a quoted value contains four consecutive backticks, switching from three to four still lets the value close its own block and expose subsequent model-controlled text as prose. There is also a --format=github failure with merely three backticks in a value: the generated inner four-backtick closing delimiter terminates the fixed four-backtick wrapper in renderForGithub(), leaving the remainder outside that wrapper. Choose delimiters longer than the longest backtick run in the enclosed content, including a still-longer delimiter for the GitHub wrapper.
Useful? React with 👍 / 👎.
# Conflicts: # docs/LESSON.md
Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EZ4u82zdoa8naYm75kppMC
Fourth item of the roadmap. Stacked on #51 (P3a), which is stacked on #50 (P2).
The problem
A red CI job says an evaluation failed. It does not say what to do about it, and the JSON report that would — the input, the golden answer, what the pipeline actually said, which metric fell over and why — is a thousand-line artifact nobody opens on a Tuesday afternoon. So the loop breaks in the same place every time: the run fails, somebody squints at a pass rate, and the diagnosis waits for whoever has an hour.
php artisan eval-harness:brief <report> --dataset=<file>produces the document instead.Three things a "copy the failures" button cannot do
policy— start there; this looks like one problem, not 5." Six failures in one cohort is a diagnosis; six scattered ones are a list.MetricGlossarycarries one line of meaning per built-in metric, which is the part a briefing built from pass/fail assertions structurally cannot have — an assertion describes itself, a metric has semantics.retrieval-mrr: 0.31becomes "the first relevant document came back around position 3", which points at the retriever rather than at the prompt. An unknown name degrades to no description, never a guess, so a host's own metric is never misdescribed.The security detail
This document quotes model output verbatim and is designed to be pasted into an agent with repository access. That makes it a prompt injection surface: one poisoned dataset row — a supplier-imported description, a scraped page, a user-submitted question — and "ignore previous instructions and…" arrives inside the context of something that can write code.
Fencing is necessary and not sufficient: a fence says where the text ends, not what it is. So the briefing opens, before the first fence, by declaring its quoted blocks untrusted data that must not be executed — and a value containing a fence gets a longer fence, so no row can close its own block early. This is
MOBILE-SEC-LLM-001/SEC-LLM-001applied to our own artifact rather than only to runtime code.A test asserts the warning appears at a lower offset than the first ``` in the document.
Where the input comes from
The report records what the pipeline said, never what it was asked. The tempting move was to add
input/expectedtosamples[]— additive, contract-legal, one line — and it would have put whole corpora into an artifact that gets committed, diffed and rendered in a browser.Instead
--dataset=<file>reads the dataset that is already in the repo and joins by content hash first, id second — the same negotiationRunComparatormakes, so a row renamed in the YAML is still found. Without the option, or when the row is genuinely gone, the briefing names the row and says the dataset was not supplied. It never invents a question: a fabricated input is the one output that would make this document actively misleading to whatever reads it.Formats
--format=md--format=github<details>PR comment — the briefing belongs where the diff is reviewed, not only in a dashboard somebody must remember to open--format=jsoneval-harness.brief.v1, carrying the structure and the markdown so a UI can show the summary and still offer the exact text to copyFailure modes handled
--budget, default 24000 chars ≈ 6k tokens): rows worst-first, only the worst-scoring execution of each (including every repetition is how a budget disappears), and truncation is declared with counts — silently cutting the list leaves an agent reasoning about "all the failures" while holding a third of them. A budget smaller than one row still emits that row.schema_versionis not the report contract is refused by name. Briefing a comparison payload would have printed "every row passed" for a run that did not.--dataset: stops the command. Asking for a dataset and silently getting a thinner document is worse than being told the file was wrong.--comparison: warns and briefs the run alone.Verification
composer validate --strict,phpunit, PHPStan level 6, Pint — all green locally.docs-site/docs/guides/briefing.md+ nav entry; README feature bullet;docs/PROGRESS.mdanddocs/LESSON.mdentries.🤖 Generated with Claude Code
https://claude.ai/code/session_01EZ4u82zdoa8naYm75kppMC
Generated by Claude Code