Measure yes/no answers on the probability that they are right - #5
Merged
Merged
Conversation
A noul record stores its confidence as the distance from a coin flip, |p - 0.5| * 2, and every calibration measure read that value as the probability of being right, which is max(p, 1 - p). A perfectly calibrated yes/no question reported ECE 0.23 and "underconfidence", and the pooled ECE carried the error into the verdict. DecisionRecord.stated_probability returns 0.5 + confidence / 2 for noul and confidence otherwise, and calibration_point, the threshold sweep, the labeling queue and recalibration read it. Stored records are unchanged. The synthetic tests came first and failed before the change: a calibrated yes/no question reports ECE near zero, an inflated one is flagged overconfident, and a yes/no answer is measured on the same scale as a choice answer. The schema test asserted the old, wrong calibration point (0.8 for P(yes) = 0.9); it now asserts 0.9. Compatibility: noul thresholds move to the same scale, so a stored noul line of 0.83 corresponds to about 0.92. Signed-off-by: rlaope <[email protected]>
The verdict, the plan output and the pooled ECE move with the fix. The pandas comparison quoted figures no current run reproduced; it now quotes the intent question, 0.091 against 0.107 under equal-width bins, reproducible with --bins-equal-width. The noul scale and its threshold compatibility note are in the README, llms.txt and the CHANGELOG. Signed-off-by: rlaope <[email protected]>
…the queue A version-2 baseline holds noul figures on the old scale and showed a false ECE improvement of about 0.19 on identical data, so snapshots move to schema_version 3 and a version-2 file is refused with the reason and the command that re-saves it. Two tests pin the probability scale in the threshold sweep and the labeling queue; both fail when either module reads the stored distance instead. Signed-off-by: rlaope <[email protected]>
…s figures A bare confidence for a noul row is read as the distance from a coin flip, so the README and llms.txt point to a probabilities field instead. The CHANGELOG no longer mixes two demo configurations in one sentence, and the pandas paragraph names its two ECE figures in the order of its clauses. Signed-off-by: rlaope <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
noul(yes/no) records store confidence as|p − 0.5| × 2, and every calibration measure read that stored value as the probability of being right. A perfectly calibrated yes/no question reported ECE ≈ 0.21–0.22 and "underconfidence". In the demo, the error reached the verdict as "Band 0.03-0.44", a range where no probability of being right can fall.DecisionRecord.stated_probabilityreturns0.5 + confidence/2fornoul. Calibration, the threshold sweep, the labeling queue and recalibration all read it. Stored records are unchanged.noulthresholds and recalibration maps move to the same probability scale. A storednoulline of 0.83 is about 0.92 now. Baseline snapshots go toschema_version3, and version-2 files are refused with a message to re-save them.Test plan
mainand pass here: calibrated yes/no gives ECE near zero, inflated yes/no is flagged as overconfident, and yes/no is measured on the same scale as choice.uv run pytest,ruff,mypyclean.