Repository navigation
docs: document the dataset format, fix the vendor command, and name the PyPI collision - #84
Merged
Merged
Conversation
…he PyPI collision #61: nothing documented the JSONL format a user writes their own data in. docs/datasets.md gives the fields (id, text, labels, gold_label, and the optional question_type and label_descriptions), what each question type means, an example row of each kind, what the loader refuses and why, and the commands a user needs next: a hosted run with a limit and a cap, a local checkpoint with its pinned 40-hex revision, the cascade's two costs, and plumbline report over existing artifacts. It is hosted on the site under Using plumbline, and a test loads its example rows through the loader so the documented format cannot drift from the real one. #29: the README's hosted-vendor command passed --max-cases 40 against the fixture's 105 scoreable rows, which refuses rather than truncates, so the first real command a user copied sent nothing. It uses --limit 40, and --max-cases now says in its help that it is a guard, not a truncation. #60: the README said pip install plumbline "will not work". It works, and installs an unrelated project with the same name. The README says so and gives pip commands that install this repository, the local_logits missing-extra error says uv sync --extra local (and the pip form) instead of pointing at the PyPI package, and PLAN notes the name is taken. Fixes #29. Fixes #60. Fixes #61. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #29. Fixes #60. Fixes #61.
#61: the dataset format was undocumented. New page,
docs/datasets.md("Your own data"), hosted on the site under Using plumbline. It gives:id,text,labels,gold_label, and the optionalquestion_typeandlabel_descriptions;--limitand a cost cap;plumbline reportover existing artifacts.A test loads the page's example rows through the loader, so the documented format can't drift from the real one. The page is honest that
label_descriptionsare loaded but not yet sent anywhere (#39).#29: the README's hosted-vendor command sent nothing. It passed
--max-cases 40against the fixture's 105 scoreable rows, and that flag refuses rather than truncates. It now uses--limit 40, and--max-casessays in its help that it's a guard, not a truncation.#60: the PyPI name collision. The README said
pip install plumbline"will not work". It works, and installs an unrelated project.local_logitsmissing-extra error saysuv sync --extra local(and the pip form), instead of pointing at the PyPI package.Checked
scorerow held back from scoring.ruff,mypy --strict, and the fullpytestsuite pass.check_site_links,check_search(97 entries) andsmoke_sitepass, and the em-dash and--checks are clean.🤖 Generated with Claude Code