docs(methodology): the no-distribution penalty is a bias, and more rows show it plainer - #99
Merged
Merged
Conversation
…ws show it plainer scripts/distribution_penalty.py repeats METHODOLOGY's one-size comparison from 500 to 20,000 rows. The top-line form's leftover miscalibration stays about 0.18 on an underconfident model and 0.02 on an overconfident one at every size while the floor falls, so its ratio to the floor grows from 2.9 to 17 times. The multiclass form inverts the injected temperature exactly, landing where the same mock does with no skew at all. Part of #8: whether a monotone fit such as isotonic regression does better on a scalar-only column is still open. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #8. It answers "How much worse is it, as a function of n?"
scripts/distribution_penalty.pyrepeats METHODOLOGY's comparison on the same mock (accuracy 0.75, four options, a pure temperature injected into the log probabilities) at 500 to 20,000 rows, averaged over 5 seeds. Each run is recalibrated twice on the same rows, once with the distributions and once with the top line only:The answer to the issue's question: the penalty persists, so the issue's own framing makes it "a property of the transport". The top-line form's leftover ECE is flat in absolute terms at every size, about 0.18 for T = 2 and 0.02 for T = 0.5. Only its ratio to the falling floor grows. On the overconfident case, the penalty is hidden at a few hundred rows and obvious by a few thousand.
A check on the multiclass column: it gives identical figures for both skews, and I reran it with no skew injected (T = 1) and got the same figures again: 1.05x at 500 and 1.43x at 20,000. So the fit inverts the injection exactly, and the drift above 1.0 is the mock's own baseline calibration, not something the fit left behind.
Already done before this PR: "Should the report say this at the point of use?" The report already prints a Distribution caveat, and the recalibration section's note, for any arm that returned a probability without a distribution.
Still open on #8: whether a monotone fit such as isotonic regression does better on a top-line-only column. That's a new correction method, so it needs its own design.
Only METHODOLOGY, CHANGELOG and the new script change. The gate, the site checks and the prose checks all pass.
🤖 Generated with Claude Code