This is a recorded finding, not a defect. plumbline already does the
right thing. This documents why, so nobody "simplifies" it back.
What was measured
On a live run of 40 choice rows against jev-1.13.0, one row
(hard-opus-c-temporal_numeric-04) returned two options tied for the maximum
at identical probability, 0.23 each.
The vendor selected one of them. An argmax over the returned distribution
picked the other, because which one an argmax returns is decided by dictionary
iteration order and nothing about the answer.
The gold label was the option the vendor did not select. So on that row,
two implementations disagreed about whether the system was right, and neither
logged anything unusual. One in 40 rows.
Why ties are ordinary here
Probabilities arrive on a two-decimal grid over three to six options. At that
resolution a tie for the maximum is an expected event, not a freak one. See the
quantization issue for the mechanism.
The rule
Read the vendor's selected label. Never recompute it.
A distribution is evidence about the selection; it is not the selection.
plumbline reads answer.choice and looks up probabilities[answer.choice] for
the calibratable column, which is correct even on a tie. The adapter docstring
and METHODOLOGY both state the rule, and METHODOLOGY frames it as guidance for
anyone writing an adapter rather than as a note about one vendor.
Note that the value is always a maximum: zero of 40 rows had prob_selected
differing from max(distribution.values()). It is the key that can differ.
What remains open
- Should a tie be recorded in the artifact? Currently a tied row looks like
any other. A reader auditing a close result cannot tell that a row was
decided by the vendor's tie-break rather than by a margin. A flag would cost
nothing and would make the audit possible.
- Should the report surface the tie count? If a meaningful fraction of rows
were decided by tie-break, that is a fact about the measurement's precision
and arguably belongs beside the accuracy figure.
- Is the vendor's tie-break deterministic? Unknown. If the same row can
resolve differently between calls, cached and live results can disagree for a
reason that is nobody's fault, and the cache documentation should say so.
What was measured
On a live run of 40 choice rows against
jev-1.13.0, one row(
hard-opus-c-temporal_numeric-04) returned two options tied for the maximumat identical probability, 0.23 each.
The vendor selected one of them. An argmax over the returned distribution
picked the other, because which one an argmax returns is decided by dictionary
iteration order and nothing about the answer.
The gold label was the option the vendor did not select. So on that row,
two implementations disagreed about whether the system was right, and neither
logged anything unusual. One in 40 rows.
Why ties are ordinary here
Probabilities arrive on a two-decimal grid over three to six options. At that
resolution a tie for the maximum is an expected event, not a freak one. See the
quantization issue for the mechanism.
The rule
Read the vendor's selected label. Never recompute it.
A distribution is evidence about the selection; it is not the selection.
plumbline reads
answer.choiceand looks upprobabilities[answer.choice]forthe calibratable column, which is correct even on a tie. The adapter docstring
and METHODOLOGY both state the rule, and METHODOLOGY frames it as guidance for
anyone writing an adapter rather than as a note about one vendor.
Note that the value is always a maximum: zero of 40 rows had
prob_selecteddiffering from
max(distribution.values()). It is the key that can differ.What remains open
any other. A reader auditing a close result cannot tell that a row was
decided by the vendor's tie-break rather than by a margin. A flag would cost
nothing and would make the audit possible.
were decided by tie-break, that is a fact about the measurement's precision
and arguably belongs beside the accuracy figure.
resolve differently between calls, cached and live results can disagree for a
reason that is nobody's fault, and the cache documentation should say so.