Blocked on #3. The local arm has never run outside the test suite, and
everything in the v0.2 milestone assumes it works. Do #3 first.
The problem
plumbline sends one request per case. Every cost and latency figure it reports
is therefore an upper bound on what batched use would cost, and the README says
so under Limitations.
This is the largest measurement gap in v0.1, because the gap is not small.
Vendor documentation describes packing many questions against one shared state
in a single call, and the shared state is the expensive part of a request on a
long-context decision API. A dataset whose rows share context could be
materially cheaper and faster batched, and plumbline currently cannot say by
how much.
Worse, the gap is not uniform. It depends on how much state the rows share, so
a single "batching is about N times cheaper" number would itself be
misleading. It is a property of the dataset, which is exactly the kind of thing
this tool exists to measure on the user's own data rather than assume.
What done looks like
- Batching is a measurement dimension, not just an optimisation: a run can
report cost and latency both batched and unbatched over the same rows, so
the saving is measured on the user's data rather than quoted from a vendor
page.
- The batch composition is recorded in the artifact. Which cases shared a
request changes the result, so it belongs beside the prompt hash.
- Per-case latency under batching is reported honestly. A batched request has
one wall-clock time covering many cases, and dividing it by the case count is
a different quantity from an unbatched per-case latency. Do not put them in
the same column.
- Cases in one batch must not leak into each other's scoring. If a batched
answer cannot be attributed to its case with certainty, refuse the batch
rather than guessing.
- The README's Limitations entry about conservative cost and latency is
replaced with what was measured.
Open design question
Whether batch composition should be automatic (group by shared state) or
caller-specified. Automatic grouping is friendlier and makes the result depend
on a heuristic nobody chose. Leaning caller-specified with a documented helper,
on the same reasoning that escalation cost and error cost are never defaulted.
The problem
plumbline sends one request per case. Every cost and latency figure it reports
is therefore an upper bound on what batched use would cost, and the README says
so under Limitations.
This is the largest measurement gap in v0.1, because the gap is not small.
Vendor documentation describes packing many questions against one shared state
in a single call, and the shared state is the expensive part of a request on a
long-context decision API. A dataset whose rows share context could be
materially cheaper and faster batched, and plumbline currently cannot say by
how much.
Worse, the gap is not uniform. It depends on how much state the rows share, so
a single "batching is about N times cheaper" number would itself be
misleading. It is a property of the dataset, which is exactly the kind of thing
this tool exists to measure on the user's own data rather than assume.
What done looks like
report cost and latency both batched and unbatched over the same rows, so
the saving is measured on the user's data rather than quoted from a vendor
page.
request changes the result, so it belongs beside the prompt hash.
one wall-clock time covering many cases, and dividing it by the case count is
a different quantity from an unbatched per-case latency. Do not put them in
the same column.
answer cannot be attributed to its case with certainty, refuse the batch
rather than guessing.
replaced with what was measured.
Open design question
Whether batch composition should be automatic (group by shared state) or
caller-specified. Automatic grouping is friendlier and makes the result depend
on a heuristic nobody chose. Leaning caller-specified with a documented helper,
on the same reasoning that escalation cost and error cost are never defaulted.