datasets.md

Your own data

plumbline measures a model on your labeled rows, so this is the file format you write them in, and the commands you run next. Put your file anywhere; the repository's datasets/private/ directory is gitignored for exactly this, so a clone never commits it.

The format

One JSON object per line (JSONL), UTF-8, with or without a byte order mark. Blank lines are skipped. Every other line is one case.

field	required	what it is
id	yes	A non-empty string, unique in the file. It names the row in the artifact.
text	yes	A non-empty string: what the model is shown.
labels	yes	A list of at least two distinct strings: the options it chooses between.
gold_label	yes	The correct option. It must be one of labels.
question_type	no	choice (the default), noul, or score. See below.
label_descriptions	no	An object mapping an option to a sentence describing it. typesafe_wire sends them as the choice's criteria; the other adapters do not use them, and the report says so when the rows carried any.

question_type says what the row actually asks, and plumbline asks it that way:

• choice: pick one of the options. Most classification rows are this.
• noul: a yes/no question. The options must be a yes/no pair (yes/no or true/false), because which option is the yes is a fact about your data, not something to guess. A transport that has a native yes/no question asks it as one, and gets back one probability rather than a distribution.
• score: an ordinal level. Its labels are its levels, the integers 0 to K minus 1 as strings, and a row whose options are not is refused. It is read by rank, by three figures of its own, and never by the choice figures, which would score wrong by one level and wrong by three the same. A level's label_descriptions entry is its rubric, which a score transport sends.

An example

{"id": "t-1", "text": "My card was charged twice for one order.", "labels": ["billing", "shipping", "returns", "other"], "gold_label": "billing"}
{"id": "t-2", "text": "The parcel says delivered but it is not here.", "labels": ["billing", "shipping", "returns", "other"], "gold_label": "shipping"}
{"id": "t-3", "text": "Is this order eligible for a refund?", "labels": ["yes", "no"], "gold_label": "yes", "question_type": "noul"}
{"id": "t-4", "text": "How urgent is this complaint, 0 to 3?", "labels": ["0", "1", "2", "3"], "gold_label": "2", "question_type": "score"}
{"id": "t-5", "text": "Where is my refund?", "labels": ["billing", "returns"], "gold_label": "returns", "label_descriptions": {"billing": "charges and invoices", "returns": "sending an item back, and its refund"}}

What is refused, and why

A row that cannot be scored honestly is refused with its line number, and the rest of the file still runs. --strict refuses the whole run instead.

• A missing required field, an empty id or text, or labels that is not a list of at least two distinct, non-empty strings. A label written as null, true or a number is refused rather than turned into the text "None", "True" or "1".
• A label_descriptions entry for something that is not one of the row's options, or a description that is not a non-empty string.
• A gold_label that is not one of the row's own labels. Scoring it would mark every model wrong on that row, which reads as a model failure and is a data error.
• A question_type other than the three above. An unknown type is not assumed to be a choice.
• An id already used earlier in the file.
• A line that is not a JSON object.

A file with no rows at all, a path that does not exist, or a file that is not UTF-8 is an error rather than an empty run; the last names the line with the byte that is not.

Running it

The mock needs nothing and spends nothing, so start there to check the file:

uv run plumbline run my-data.jsonl --adapter mock --results results --report results/report.md

A hosted vendor needs its key in the environment and, for a cost column, a pricing table you filled in (see the README). --limit runs only the first N cases; --max-cases and --max-cost-usd refuse the run instead of truncating it, so they are guards rather than ways to run less:

uv run plumbline run my-data.jsonl --adapter typesafe_wire --model jev-latest \
    --limit 40 --max-cost-usd 0.50 --pricing my-pricing.json \
    --results results --report results/report.md

A local checkpoint needs the local extra (uv sync --extra local), a model id, and the commit it is pinned to, as 40 hex characters. A branch name or a tag can move, and everything measured would then be attributed to other weights:

uv run plumbline run my-data.jsonl --adapter local_logits \
    --model Qwen/Qwen2.5-0.5B-Instruct --revision 7ae557604adf67be50417f59c2c2f167def9a775 \
    --results results --report results/report.md

The cascade needs two numbers only you know

Where to set a threshold depends on what one escalation to the expensive path costs you and what one wrong answer costs you. plumbline never defaults either, because a made-up default would decide the threshold. Pass both, in USD:

uv run plumbline run my-data.jsonl --adapter mock --escalation-cost 0.02 --error-cost 1.00 \
    --results results --report results/report.md

The threshold is chosen on half of the scored rows and judged on the other half, so a threshold needs at least 400 scored rows; below that the report says so instead of printing one.

Reports from runs that already happened

Every run writes an artifact to --results. plumbline report renders one document from one or more of them, so a finished run is never repeated to get a report, or to try different cascade costs:

uv run plumbline report results/*.json --out results/report.md \
    --escalation-cost 0.02 --error-cost 1.00