Skip to content

Add contest scoring: OLS ports, mean-imputation, and ranking (Rules §5) - #3

Merged
bradfordlynch merged 2 commits into
mainfrom
contest-scoring
Aug 1, 2026
Merged

bradfordlynch merged 2 commits into
mainfrom
contest-scoring

Conversation

@bradfordlynch

Copy link
Copy Markdown
Contributor

The Official Rules incorporate src/examples/scoring.py as the controlling scoring methodology, so it must carry the full contest calculation, not just the percentile transform.

  • Verbatim float-for-float ports of the production OLS: OLSFit, ols_fit (univariate surprise-only benchmark), ols_fit2 (two-regressor model via the centered 2×2 normal equations). Same guards as production: None on too-few points, zero variance, or a singular system — and deliberately no univariate fallback.
  • Rules §5 imputation: impute_predictions computes the submission-specific mean over its timely, valid predictions on the common Scored Events set before filling, then imputes it into every missed event. No coverage ⇒ None (mean undefined, not prize-eligible).
  • score_submission: both metric families for one submission — the no-fill fit on its own predicted events and the _imputed family over the full common sample, with the imputed surprise-only benchmark fit once on the full common sample (submission-invariant). Emits n_obs (pre-imputation coverage, the tie-break), all seven metrics per family, imputed_mean, imputed_event_count.
  • rank_submissions: contest ordering — metric (descending; ascending for the MSE family), then higher pre-imputation coverage, then submission id (display-only; residual exact ties share the combined prize per the Rules — no random tie-break). Undefined metric ⇒ unranked, dropped.
  • 20 new tests: exact-value OLS recovery, degenerate-fit guards, mean-before-fill arithmetic, full-coverage family equality, benchmark invariance across submissions, tie-break ordering.

Note for merging: §5 pins a frozen version of this file — the post-merge commit hash on main is what should be stamped into the Rules doc before the Aug 9 registration deadline.

The Official Rules incorporate this file as the controlling scoring
methodology, so it must carry the full contest calculation, not just the
percentile transform. Adds verbatim float-for-float ports of the
production OLS (ols_fit / ols_fit2, degenerate-fit guards included), the
submission-specific mean imputation over the common Scored Events set
(mean computed before filling; no coverage => ineligible), score_submission
emitting both the no-fill and `_imputed` metric families, and
rank_submissions implementing the contest order (metric, then higher
pre-imputation coverage, then submission id; residual exact ties share the
combined prize — no random tie-break).
Code review on the PR: the new code was wrapped at 88 columns and failed
the CI ruff-format gate (repo formats to 100). The reformat is
AST-neutral — an AST-normalized comparison and a 400-trial randomized
differential against the production scorer confirmed float-for-float
parity is unaffected — but the ported text is no longer byte-identical
to production, so the module docstring now claims an exact semantic
port rather than a verbatim one.
@bradfordlynch
bradfordlynch merged commit 5da45b3 into main Aug 1, 2026
4 checks passed
@bradfordlynch
bradfordlynch deleted the contest-scoring branch August 3, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant