Repository navigation
fix(cli): validate before anything is sent, put status on stderr, exit 1 when nothing was measured - #83
Merged
Conversation
… exit 1 when nothing was measured Options (#41): --limit, --workers, --max-cases and --boot must be positive, checked by the parser, and limit is tested with "is not None" rather than truthiness, so --limit 0 no longer runs every row and a negative limit no longer slices from the end. --boot 0 used to fail with a traceback after the run and the artifact, when the calls were already paid for. An option the chosen adapter does not take (--revision on typesafe_wire), a missing API key, and a --report path that is a directory are now each one line, before anything is sent. Streams and exit code (#42): status lines and errors go to stderr, so stdout carries only the report when no --report is given and `run > report.md` captures just the report. A run where no case produced a prediction writes its artifact and report, then exits 1, saying the shared reason when there is one. Report (#15): with no figures and one shared failure reason, the report prints that reason instead of pointing at the artifact; with several, it says how many and leaves them to the artifact. README says what goes to which stream and what the exit codes mean. Fixes #41. Fixes #42. Fixes #15. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #41. Fixes #42. Fixes #15.
#41: options that misbehaved or crashed.
--limit,--workers,--max-casesand--bootmust now be positive, checked by the parser. The limit is tested withis not Nonerather than truthiness, so:--limit 0no longer runs every row;--boot 0used to fail with a traceback after the run and the artifact, when the calls were already paid for. It is now refused at parse time.--revisionontypesafe_wire);--reportpath that is a directory.#42: the exit code and the streams.
--reportis given, andplumbline run ... > report.mdcaptures just the report.#15: the shared failure reason. With no figures and one shared reason (usually a missing extra or key), the report now prints that reason instead of pointing at the artifact. With several different reasons, it says how many and leaves them to the artifact.
Checked
mainand pass now:--limit 0,--limit -1and--boot 0are refused, with no traceback and no results directory created;--revisionontypesafe_wireand a missingTYPESAFE_API_KEYare one line each;--reportis refused before the run;result.output, which holds both streams.ruff,mypy --strict, the fullpytestsuite, and the site build all pass.🤖 Generated with Claude Code