Skip to content

feat(cli): --dry-run, --base-url, --timeout, and the load report in the artifact - #94

Merged
TMHSDigital merged 1 commit into
mainfrom
feat/cli-dry-run-endpoint
Sep 24, 2026
Merged

TMHSDigital merged 1 commit into
mainfrom
feat/cli-dry-run-endpoint

Conversation

@TMHSDigital

Copy link
Copy Markdown
Owner

Fixes #58.

--dry-run. execute.plan() prices the run and applies the guard without sending anything, and execute.run() now starts from it too. So the dry run refuses exactly what the run would refuse (a case limit, an unpriced model under --max-cost-usd, an estimate over the limit), with the same messages. When the run would go ahead, it prints the plan to stdout and exits 0 without writing a file:

dry run: nothing was sent and nothing was written.

- 105 cases for the mock adapter, model `mock-1`.
- Endpoint: the adapter's default.
- Timeout: the adapter's default.
- Probability semantics: calibrated_claim.
- Cost: not estimated: `mock-1` is not in the pricing table. Pass --pricing with an entry for it to cost the run in advance.
- Guard: no cost limit, no case limit; the run would start.

With a priced model, the cost line names the estimate, the pricing entry and the date it was read.

--base-url and --timeout. These are passed to adapters that accept them: typesafe_wire and generative. The endpoint was already recorded in the artifact and part of the cache key. The timeout is now recorded too, as timeout_seconds. --timeout must be above 0.

--semantics. It now overrides the declared semantics for any adapter that accepts it, not only the mock. The artifact records semantics_set_by: operator, and the report adds a line saying the operator set it.

Refused options. An option an adapter doesn't accept is now refused by name before the adapter is built, using the new registry.accepts(), which reads the adapter's signature. For example, the generative adapter does not take --semantics. Before, the message listed every option that had been passed.

The load report in the artifact. The new LoadSummary holds the load's counts, refusals, notes, and the rows excluded by question type, without the cases. RunResult.load stores it. LoadReport.statement() now goes through LoadSummary.statement(), so both print the same sentence.

plumbline report passes the stored load when every artifact covers one dataset and they all stored the same load. Otherwise it leaves the section out rather than print one run's counts over another's figures. Artifacts written before this change still load, just without the section.

Docs. README, CONTRIBUTING and .env.example said a new endpoint was "a config entry", which had no mechanism behind it. They now say --base-url. README also suggests --dry-run before a paid run.

Tests. The new tests/test_cli_run_options.py covers:

  • a dry run makes no adapter call and writes no artifact
  • the priced estimate names its entry and date
  • the dry run refuses what the run would refuse
  • the endpoint, timeout and semantics reach a typesafe_wire dry run, and the dummy key never appears in the output
  • a rejected option is named exactly, for three adapter and option pairs
  • --timeout 0 is refused
  • a rebuilt report's Dataset section matches the one written at run time
  • an artifact without a load still reports
  • the load round-trips through the artifact
  • the timeout is recorded

docs/example-report.md is unchanged, since its command passes none of the new options. The full gate, the site build and every site check, and the prose checks all pass.

🤖 Generated with Claude Code

…he artifact

A dry run loads and checks the dataset, builds the adapter, and applies
the cost guard through the same plan the run starts from, then prints the
case count, endpoint, timeout, semantics, and estimate, and sends nothing.

--base-url and --timeout reach an adapter that takes them, and the timeout
is recorded beside the endpoint. --semantics now overrides any adapter
that takes it, and the report says the operator set it. An option an
adapter does not take is refused by name, read from its signature.

An artifact stores a LoadSummary, so plumbline report rebuilds the Dataset
section it used to drop.

Fixes #58.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@TMHSDigital
TMHSDigital merged commit 0c2811b into main Sep 24, 2026
23 checks passed
@TMHSDigital
TMHSDigital deleted the feat/cli-dry-run-endpoint branch September 24, 2026 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CLI: --dry-run, --base-url, --timeout, and the load report in the artifact

1 participant