Repository navigation
docs(readme): a quickstart whose three steps actually run in the order printed - #57
Conversation
…r printed Three Codex threads on #56, all real. Copy-pasting the thirty-second version into a fresh install produced three different errors, none of them visible to someone reading the page. - The registration snippet had no home and no system-under-test. `eval-harness:run` starts its own Artisan process, so a bare `$eval->dataset(...)->register()` reaches "Dataset is not registered", and without a callable or SampleRunner under `eval-harness.sut` it reaches "No system-under-test bound". Shown as a registrar, with the binding, matching the docs-site quickstart. - `--compare=baseline --max-regressions=0` was the first command run. On an installation with no baseline, resolveReferenceReport() warns and skips the comparison — so the gate the surrounding paragraph is selling cannot fire, and the run passes. There are now two runs: the first records the baseline with --promote-baseline, the second is gated against it. - `brief report.json` read a file the preceding run never wrote: without --json --out the report goes to stdout as Markdown. The gated run now writes run-2.json and comparison.json on the reports disk, which is where BriefCommand resolves a relative path. The illustrated output was wrong in four smaller ways while I was in there: the `Macro-F1 …, pass rate …` headline belongs to RunBriefing, not to the run; the comparison line prints a trailing `%d compared`; the gate message is "N confident row(s) regressed against <reference> (allowed: N)"; and the counts claimed two improved rows in a one-sample dataset. Read off the format strings and moved the headline down to the briefing step where it is emitted. Added a note that llm-as-judge needs a provider and rouge-l does not, so the loop is runnable with no API key. Two LESSON entries. Unit 1136, Architecture 3, Pint and composer validate --strict clean. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EZ4u82zdoa8naYm75kppMC
Deploying laravel-patent-box-tracker with
|
| Latest commit: |
8425ea8
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://86318259.laravel-patent-box-tracker.pages.dev |
| Branch Preview URL: | https://claude-readme-quickstart-run.laravel-patent-box-tracker.pages.dev |
Deploying eval-harness with
|
| Latest commit: |
8425ea8
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://5671884e.eval-harness.pages.dev |
| Branch Preview URL: | https://claude-readme-quickstart-run.eval-harness.pages.dev |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8425ea8be2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ->loadFromYaml(database_path('evals/rag.factuality.yaml')) | ||
| ->withMetrics(['exact-match', 'llm-as-judge']) | ||
| ->register(); | ||
| // app/Console/EvalRegistrar.php |
There was a problem hiding this comment.
When this block is copied to the advertised app/Console/EvalRegistrar.php, it begins without <?php, declares no App\Console namespace, and imports neither Container nor EvalEngine. Composer therefore cannot resolve App\Console\EvalRegistrar, so both run commands fail before registering the dataset. Include the normal PHP header, namespace, and imports as the fuller quickstart does.
Useful? React with 👍 / 👎.
| php artisan eval-harness:brief run-2.json \ | ||
| --dataset=database/evals/rag.factuality.yaml \ | ||
| --comparison=comparison.json | claude -p |
There was a problem hiding this comment.
With the default configuration, the preceding relative outputs are stored as eval-harness/reports/run-2.json and eval-harness/reports/comparison.json, not as cwd files or disk-root run-2.json. BriefCommand::readPayload() checks the supplied disk path verbatim and does not prepend the configured prefix, so this command exits with Report [run-2.json] could not be read as JSON. Pass the prefixed paths here, or write the artifacts as literal paths with --raw-path.
Useful? React with 👍 / 👎.
| 1 regressed (1 beyond this run's 4.7-point detectable difference), 0 improved, 0 added, 0 removed, 1 compared. | ||
| refund-window 100% → 33% score -0.4200 beyond noise, newly failing | ||
| Gate failed: 1 confident regression. | ||
| Gate failed: 1 confident row regressed against the baseline [eval-harness/reports/run-1.json] (allowed: 0) |
There was a problem hiding this comment.
For this one-row dataset with repetitions: 3, a current pass rate of 33% has within-row variance about 0.222, giving the implemented per-row detectable difference about 75.4 points, not 4.7. Neither the displayed 66.7-point pass-rate drop nor the 0.4200 score drop exceeds that resolution, so the row is not confident and --confident-only --max-regressions=0 counts zero regressions rather than printing the shown gate failure. Increase repetitions, omit --confident-only, or adjust the example output.
Useful? React with 👍 / 👎.
Sottotask
ID: follow-up to #56 (merged). Three Codex review threads left unresolved on that PR, all three real.
Plan ref: none — this closes review feedback on the README opening.
Summary
Copy-pasting "The thirty-second version" into a fresh install produced three different errors, none of them visible to someone reading the page. The steps now run in the order printed, from nothing.
Changes
File modificati:
README.md,docs/LESSON.md. No migration, no test changes (docs-only).The three findings
The registration snippet had no home and no system-under-test.
eval-harness:runstarts its own Artisan process, so a bare$eval->dataset(...)->register()reaches "Dataset is not registered", and without a callable orSampleRunnerundereval-harness.sutit reaches "No system-under-test bound" (Console/Concerns/ResolvesSystemUnderTest.php:14). Shown as a registrar with the binding, matching the docs-site quickstart.--compare=baseline --max-regressions=0was the first command run. On an installation with no baseline,resolveReferenceReport()warns and returnsnull(Console/Concerns/ComparesRuns.php:42) — the comparison is skipped and the run passes, so the gate the surrounding paragraph is selling cannot fire. There are now two runs: the first records the baseline with--promote-baseline, the second is gated against it.brief report.jsonread a file the preceding run never wrote. Without--json --out, the report goes to stdout as Markdown, andBriefCommandfails with "Report [report.json] could not be read as JSON". The gated run now writesrun-2.jsonandcomparison.json, and relative paths resolve on the reports disk for both commands.While I was in there
The illustrated console output was wrong in four smaller ways, none flagged:
Macro-F1 …, pass rate …attributed to the runRunBriefing::headlineNumbers()— moved down to the briefing step0 removed.%d comparedGate failed: 1 confident regression.N confident row(s) regressed against <reference> (allowed: N)2 improvedAlso added a note that
exact-matchscores offline whilellm-as-judgeneeds a judge provider, withrouge-las the no-API-key swap — so the whole loop is runnable on a fresh install.Two
docs/LESSON.mdentries: a quickstart is a claim that gets executed, and illustrated console output drifts because nobody diffs a fenced block against the sprintf that produces it.Test gate
{"tool":"pint","result":"passed"})composer validate --strictvalidArchitecture impact
None. Documentation only; no package contract, report schema or API surface changed. The commands and options shown were verified against
EvalCommand,BaselineCommandandBriefCommandsignatures rather than invented.Security impact
None. No new PII path, no ACL change, no credential in the examples.
Risk
Low. Docs-only. The risk it removes is larger than the one it adds: the previous quickstart failed three times for a first-time reader, and the third failure was silent (a gate that skipped rather than fired).
Rollback plan
Revert the merge commit. Nothing else depends on this text.
Generated by Claude Code