Skip to content

docs: document the dataset format, fix the vendor command, and name the PyPI collision - #84

Merged
TMHSDigital merged 1 commit into
mainfrom
docs/install-and-dataset-format
Sep 23, 2026
Merged

TMHSDigital merged 1 commit into
mainfrom
docs/install-and-dataset-format

Conversation

@TMHSDigital

Copy link
Copy Markdown
Owner

Fixes #29. Fixes #60. Fixes #61.

#61: the dataset format was undocumented. New page, docs/datasets.md ("Your own data"), hosted on the site under Using plumbline. It gives:

  • the fields: id, text, labels, gold_label, and the optional question_type and label_descriptions;
  • what each question type means, and an example row of each kind;
  • what the loader refuses and why;
  • the commands a user needs next:
    • a hosted run with --limit and a cost cap;
    • a local checkpoint with its pinned 40-hex revision;
    • the cascade's two costs;
    • plumbline report over existing artifacts.

A test loads the page's example rows through the loader, so the documented format can't drift from the real one. The page is honest that label_descriptions are loaded but not yet sent anywhere (#39).

#29: the README's hosted-vendor command sent nothing. It passed --max-cases 40 against the fixture's 105 scoreable rows, and that flag refuses rather than truncates. It now uses --limit 40, and --max-cases says in its help that it's a guard, not a truncation.

#60: the PyPI name collision. The README said pip install plumbline "will not work". It works, and installs an unrelated project.

  • The README now says so and gives pip commands that install this repository.
  • The local_logits missing-extra error says uv sync --extra local (and the pip form), instead of pointing at the PyPI package.
  • PLAN notes the name is taken.

Checked

  • The page's example rows load: 5 rows, none refused, and the score row held back from scoring.
  • ruff, mypy --strict, and the full pytest suite pass.
  • On the built site (12 pages), check_site_links, check_search (97 entries) and smoke_site pass, and the em-dash and -- checks are clean.

🤖 Generated with Claude Code

…he PyPI collision

#61: nothing documented the JSONL format a user writes their own data in.
docs/datasets.md gives the fields (id, text, labels, gold_label, and the
optional question_type and label_descriptions), what each question type
means, an example row of each kind, what the loader refuses and why, and
the commands a user needs next: a hosted run with a limit and a cap, a
local checkpoint with its pinned 40-hex revision, the cascade's two
costs, and plumbline report over existing artifacts. It is hosted on the
site under Using plumbline, and a test loads its example rows through the
loader so the documented format cannot drift from the real one.

#29: the README's hosted-vendor command passed --max-cases 40 against the
fixture's 105 scoreable rows, which refuses rather than truncates, so the
first real command a user copied sent nothing. It uses --limit 40, and
--max-cases now says in its help that it is a guard, not a truncation.

#60: the README said pip install plumbline "will not work". It works, and
installs an unrelated project with the same name. The README says so and
gives pip commands that install this repository, the local_logits
missing-extra error says uv sync --extra local (and the pip form) instead
of pointing at the PyPI package, and PLAN notes the name is taken.

Fixes #29. Fixes #60. Fixes #61.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@TMHSDigital
TMHSDigital merged commit b0e14f1 into main Sep 23, 2026
17 checks passed
@TMHSDigital
TMHSDigital deleted the docs/install-and-dataset-format branch September 23, 2026 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant