feat: record a tie for the top option and count ties in the report - #97
Merged
Merged
Conversation
On a two-decimal grid two options sharing the top probability is ordinary, and a row decided by the vendor's tie-break looked like any other. Prediction.tied_for_top names the tied options, each artifact record carries them, and the report says beside accuracy how many rows tied and on how many the gold label was a tied option the vendor did not choose. Part of #10: whether the vendor's tie-break is deterministic needs live calls, and the flag is what lets that be checked. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #10. This covers the two open questions that can be settled without live calls.
Should a tie be recorded in the artifact? Yes.
Prediction.tied_for_topreturns the options sharing the highest probability when two or more do, and otherwise returns(), including when there is no distribution. Each artifact record now carriestied_for_top. It's derived from the distribution already stored, and it's written for auditors, not read back.Should the report surface the tie count? Yes, beside accuracy, and only when something tied:
The selected label is still always the vendor's pick, never a recomputed argmax, as METHODOLOGY requires. METHODOLOGY's "Read the vendor's selected label" section now describes the flag and the report line, and says the tie-break's determinism is unknown.
Still open on #10: whether the vendor's tie-break is deterministic. That needs repeated live calls on a tied row, and the new flag is what makes those rows findable.
Tests: the property on tied, untied and single-option distributions; the flag in a written artifact; the count and the against-gold count in the report; no line when nothing tied.
docs/example-report.mdis unchanged, since the mock produces no ties. The gate, the site checks and the prose checks all pass.🤖 Generated with Claude Code