Leaderboard: sampling error and McNemar significance groups - #60
Conversation
The board reports a resolve rate per entry and nothing about how precisely it
is measured, so adjacent rows read as ordered even when the data does not order
them. This adds two columns computed from per-instance results the site already
carries:
± SE binomial standard error of that entry's own rate, in percentage
points, over the instances it was scored on. Stated as the precision
of one number, not as the error of a difference: entries share an
instance set, so comparing two of them is a paired question.
Tie that paired question, answered. Exact two-sided McNemar at alpha =
0.05; the highest entry that publishes per-instance results anchors
group 1 and every entry the test cannot separate from it joins the
group. Membership is a property of the comparison with the anchor,
not of an entry alone.
Nothing is stored: build.py annotates the loaded leaderboards, so
leaderboards.json stays the record of what was submitted and these stay a
function of it. Entries without per-instance results get a dash rather than a
blank, and sort last in both directions, so "not published" cannot read as a
small error or a top group. The columns appear when the group anchor is the
entry at the top of the view, which today means the bash-only view.
On the current data this puts the top six bash-only entries (76.8 down to 74.2)
in one group. leaderboard_statistics.py --selftest covers the test, the SE, the
anchor rule and the empty cases.
Both numbers condition on the single run each entry reports. Nothing here bounds how much a score would move on a rerun of the same system, so neither column is the uncertainty of the ranking. Stated in the note a reader sees and in the module docstring, next to the two claims that were already explicit (SE is not the error of a difference; the group is a paired result against a named anchor at alpha = 0.05 with no multiplicity correction). Raised by @KeilerHirsch in SWE-bench/SWE-bench#621: marginal uncertainty, paired comparison resolution and repeatability are three different things and the wording should not let one be read as another.
…he board Both columns are computed from per-instance outcomes while the board's own number comes from elsewhere, so the two can be checked against each other. They now are: an entry is annotated only when the rate implied by its per-instance results agrees with its published rate to within one instance (100/n percentage points). One instance is the finest step the published rate can move in, so a smaller gap is rounding or a denominator that differs by one, and a larger one means the two artifacts disagree about at least one instance. This is not hypothetical. On the current data two submissions publish per-instance files that imply 0.00 % and 10.20 % against published rates of 69.60 % and 52.80 %. Before this commit the first of them was rendered with the most precise-looking cell on the page, an SE of +/- 0.00, because a rate of exactly zero makes sqrt(p(1-p)/n) vanish; the second was placed in a significance group computed from a 10.20 % outcome set. Both now show a dash whose tooltip says which of the two published numbers disagrees with which, and the build prints them by name so the gap is not only visible on the page. The anchor and the size of the top group are unchanged on every board. Three further cases where a formula returns a number that is not a measurement: - an empty sample raises instead of returning an SE of 0.0; - a boundary rate (every instance resolved, or none) withholds the SE, since the zero is a property of the estimator rather than of the entry, while the entry still takes part in the paired comparisons, which remain valid; - two entries that share no instance are no longer given a p-value of 1.0 and put in the same group, which reported absence of paired evidence as absence of difference. mcnemar_exact returns None there and the entry anchors its own group. Selftest covers each, including both sides of the one-instance threshold, and each guard was checked by mutation: loosening the gate by 2x or 10x, tightening it 10x, returning 0.0 for an empty sample, treating disjoint sets as a tie, and publishing the boundary zero all fail the selftest.
|
Pushed Two submissions publish per-instance files that contradict their own board number. Both columns are computed from per-instance outcomes, while the board's rate comes from elsewhere, so the two can be checked against each other. They now are:
Before this commit my column rendered the first of those as The rule now: an entry is annotated only when the rate implied by its per-instance results agrees with the published rate to within one instance ( The result the PR is about does not move. The anchor is Three further cases where a formula returns a number that is not a measurement, all fixed in the same commit:
Selftest covers each case including both sides of the one-instance threshold, and I checked the guards by mutation rather than trusting them: loosening the gate 2x or 10x, tightening it 10x, returning The two inconsistent per-instance files are worth a look on their own, since anything downstream that reads |
|
Reviewing this from the measurement side: the new consistency gate is exactly the right direction. Refusing to manufacture precision when the aggregate and per-instance artifacts contradict each other is much better than choosing one silently. I have one substantive terminology/model objection before I'd call the SE column clean. The board is a fixed set of instances. On those exact instances, 396/500 is an observed descriptive rate, not itself a sampling estimate with uncertainty. So I would make the inference target explicit in the column definition:
That keeps the three uncertainty objects from being collapsed into one "precision" concept. The contradictory per-instance files are a very good independent data-quality finding and I agree they deserve a separate upstream issue regardless of whether this PR lands. |
The resolve rate on a board is an exact descriptive fact: every entry is scored on the same fixed set of instances, so 396 of 500 is 79.20 % with nothing left to estimate. sqrt(p(1-p)/n) is a standard error only under a declared target of inference, and the column never declared one. The note and the column tooltip now separate four objects that were running together: the observed proportion on a fixed set (exact, descriptive), the binomial SE (model-based, generalising to comparable tasks under an exchangeability assumption), the paired McNemar comparison on shared instances, and repeatability, which a single run cannot observe. The exchangeability assumption is stated rather than implied, and stated against these item sets in particular: they are curated and human-filtered rather than randomly drawn, which makes the assumption questionable, not merely unstated. Wording only. No change to leaderboard_statistics.py, and the computed output is unchanged: Verified still reports 38 of 180 entries with per-instance results and 6 in group 1 anchored on Claude 4.5 Opus.
|
Pushed The note and the column tooltip now keep four objects apart instead of letting them run together:
One thing beyond your list, because leaving it out would have made the column look better than it The per-instance data problem is now a separate upstream issue, as you suggested: |
|
Yes — this resolves my objection. The important improvement is not the formula but that the measurement object and the inference claim are now kept separate. Calling 396/500 an exact descriptive fact on the fixed set, while making the binomial SE conditional on an explicit wider-population model, is the distinction I wanted. Explicitly stating that exchangeability is questionable on a curated/human-filtered benchmark makes the note stronger, not weaker. And splitting the contradictory per-instance artifacts into experiments#467 was the right move. That's an independent data-integrity issue, not something this statistics PR should quietly absorb. From the measurement side, I'm happy with this wording. Maintainers' court now. :) |
2 participants
Adds the two columns proposed in SWE-bench/SWE-bench#621, computed from per-instance results this repo already ships. No new data collection, no change to
data/leaderboards.json.What the columns say
± SEsqrt(p(1-p)/n)in percentage points. It is the precision of a single number, not the error of a difference. Two entries are scored on the same instances, so whether they differ is a paired question, and overlapping SEs are not the answer to it.TieNo multiplicity correction is applied to the comparisons against an anchor. A correction makes separation harder and groups larger, so the group sizes here are a lower bound on how much of a board is indistinguishable.
What it shows on the current data
The bash-only view, which is where the site lands by default:
Six entries spanning 2.6 points are one group: the ordering among them is not something these 500 instances decide. Twelve groups over the 40 entries with per-instance results. The 297-instance Multilingual entry is why
nis read per entry rather than assumed per board.Implementation
data/leaderboard_statistics.py— stdlib only, no new dependencies. Exact McNemar viamath.comb, binomial SE, the grouping rule, and--selftest.build.py— annotates the loaded leaderboards at build time.data/leaderboards.jsonis untouched, so it stays the record of what was submitted and these numbers stay a function of it, recomputed whenever a submission lands.js/mainResults.js,css/leaderboard.css— two columns, a note under the table stating both definitions and the anchor by name, and widths.Where it deliberately shows nothing
An entry that publishes no per-instance results gets an em dash, not a blank, and sorts last whichever way the arrow points, so "not published" cannot read as the smallest error or the top group. The columns appear only when the group anchor is the entry at the top of the view; otherwise the number would be anchored on a row far down the page.
That rule is why the full Verified board does not get the columns, and the reason is worth stating plainly: 140 of its 180 entries publish no per-instance results, and the six highest-scoring entries are among them. The board's headline number cannot currently be compared with its runner-up using data this site has. Ingesting the
resolvedlists fromSWE-bench/experimentswould light the columns up board-wide; I am happy to do that in a follow-up PR if you want it.Checks
python data/leaderboard_statistics.py --selftestcovers: identical entries are never separated; 40 planted one-directional flips are detected (so the first check is not passing for lack of power); 200 balanced discordant pairs are not called a difference; SE matchessqrt(p(1-p)/n)and is zero at a perfect score; only shared instances are compared; entries without per-instance results get no numbers and do not shift anyone else's group; the anchor is the highest measurable entry and is reported with a count of how many higher entries publish nothing. The p-values were also cross-checked againstscipy.stats.binomon the live top pair (both 0.583132 for 29/24 discordant); scipy is not a dependency of the site and is not used at build time.Verified in the browser in light and dark mode, with the bash-only preset on and off, and with both new columns sorted in both directions.
Note on scope. I dropped the separate
ncolumn from the format in the issue: within a board it is nearly constant, so it sits in the SE cell's tooltip and in the JSON rather than taking table width. Say the word and I will make it a column. cc @KeilerHirsch, who offered to review the measurement language.