Skip to content

fix(cli): validate before anything is sent, put status on stderr, exit 1 when nothing was measured - #83

Merged
TMHSDigital merged 1 commit into
mainfrom
fix/cli-validation-and-exit-codes
Sep 23, 2026
Merged

TMHSDigital merged 1 commit into
mainfrom
fix/cli-validation-and-exit-codes

Conversation

@TMHSDigital

Copy link
Copy Markdown
Owner

Fixes #41. Fixes #42. Fixes #15.

#41: options that misbehaved or crashed.

  • --limit, --workers, --max-cases and --boot must now be positive, checked by the parser. The limit is tested with is not None rather than truthiness, so:
    • --limit 0 no longer runs every row;
    • a negative limit no longer slices from the end.
  • --boot 0 used to fail with a traceback after the run and the artifact, when the calls were already paid for. It is now refused at parse time.
  • Each of these is now one line, before anything is sent:
    • an option the chosen adapter doesn't take (such as --revision on typesafe_wire);
    • a missing API key, with the variable to set;
    • a --report path that is a directory.

#42: the exit code and the streams.

  • Status lines and errors go to stderr, so stdout carries only the report when no --report is given, and plumbline run ... > report.md captures just the report.
  • A run where no case produced a prediction writes its artifact and report, then exits 1. When every failure shares one reason, it says that reason.
  • README documents which stream carries what, and the exit codes.

#15: the shared failure reason. With no figures and one shared reason (usually a missing extra or key), the report now prints that reason instead of pointing at the artifact. With several different reasons, it says how many and leaves them to the artifact.

Checked

  • New tests failed on main and pass now:
    • --limit 0, --limit -1 and --boot 0 are refused, with no traceback and no results directory created;
    • --revision on typesafe_wire and a missing TYPESAFE_API_KEY are one line each;
    • a directory --report is refused before the run;
    • status goes to stderr and the report alone to stdout;
    • an all-failing run exits 1, still writes the artifact, and names the reason in stderr and in the report;
    • the report handles both the one-reason and many-reasons cases.
  • Existing CLI tests that read status lines now read result.output, which holds both streams.
  • ruff, mypy --strict, the full pytest suite, and the site build all pass.

🤖 Generated with Claude Code

… exit 1 when nothing was measured

Options (#41): --limit, --workers, --max-cases and --boot must be
positive, checked by the parser, and limit is tested with "is not None"
rather than truthiness, so --limit 0 no longer runs every row and a
negative limit no longer slices from the end. --boot 0 used to fail with
a traceback after the run and the artifact, when the calls were already
paid for. An option the chosen adapter does not take (--revision on
typesafe_wire), a missing API key, and a --report path that is a
directory are now each one line, before anything is sent.

Streams and exit code (#42): status lines and errors go to stderr, so
stdout carries only the report when no --report is given and
`run > report.md` captures just the report. A run where no case produced
a prediction writes its artifact and report, then exits 1, saying the
shared reason when there is one.

Report (#15): with no figures and one shared failure reason, the report
prints that reason instead of pointing at the artifact; with several, it
says how many and leaves them to the artifact.

README says what goes to which stream and what the exit codes mean.

Fixes #41. Fixes #42. Fixes #15.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@TMHSDigital
TMHSDigital merged commit a65c55d into main Sep 23, 2026
17 checks passed
@TMHSDigital
TMHSDigital deleted the fix/cli-validation-and-exit-codes branch September 23, 2026 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant