Skip to content

Finding: vendor ties mean the reported choice can differ from argmax, so adapters must read the choice #10

Description

@TMHSDigital

This is a recorded finding, not a defect. plumbline already does the
right thing. This documents why, so nobody "simplifies" it back.

What was measured

On a live run of 40 choice rows against jev-1.13.0, one row
(hard-opus-c-temporal_numeric-04) returned two options tied for the maximum
at identical probability
, 0.23 each.

The vendor selected one of them. An argmax over the returned distribution
picked the other, because which one an argmax returns is decided by dictionary
iteration order and nothing about the answer.

The gold label was the option the vendor did not select. So on that row,
two implementations disagreed about whether the system was right, and neither
logged anything unusual. One in 40 rows.

Why ties are ordinary here

Probabilities arrive on a two-decimal grid over three to six options. At that
resolution a tie for the maximum is an expected event, not a freak one. See the
quantization issue for the mechanism.

The rule

Read the vendor's selected label. Never recompute it.

A distribution is evidence about the selection; it is not the selection.
plumbline reads answer.choice and looks up probabilities[answer.choice] for
the calibratable column, which is correct even on a tie. The adapter docstring
and METHODOLOGY both state the rule, and METHODOLOGY frames it as guidance for
anyone writing an adapter rather than as a note about one vendor.

Note that the value is always a maximum: zero of 40 rows had prob_selected
differing from max(distribution.values()). It is the key that can differ.

What remains open

  • Should a tie be recorded in the artifact? Currently a tied row looks like
    any other. A reader auditing a close result cannot tell that a row was
    decided by the vendor's tie-break rather than by a margin. A flag would cost
    nothing and would make the audit possible.
  • Should the report surface the tie count? If a meaningful fraction of rows
    were decided by tie-break, that is a fact about the measurement's precision
    and arguably belongs beside the accuracy figure.
  • Is the vendor's tie-break deterministic? Unknown. If the same row can
    resolve differently between calls, cached and live results can disagree for a
    reason that is nobody's fault, and the cache documentation should say so.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    adapterAdapter transports and vendor configurationmethodologyHow a number is computed and what it does or does not meanopen questionA measured finding with something still unresolved. Not a defect.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions