Repository navigation
Add contest scoring: OLS ports, mean-imputation, and ranking (Rules §5) - #3
Merged
Merged
Conversation
The Official Rules incorporate this file as the controlling scoring methodology, so it must carry the full contest calculation, not just the percentile transform. Adds verbatim float-for-float ports of the production OLS (ols_fit / ols_fit2, degenerate-fit guards included), the submission-specific mean imputation over the common Scored Events set (mean computed before filling; no coverage => ineligible), score_submission emitting both the no-fill and `_imputed` metric families, and rank_submissions implementing the contest order (metric, then higher pre-imputation coverage, then submission id; residual exact ties share the combined prize — no random tie-break).
Code review on the PR: the new code was wrapped at 88 columns and failed the CI ruff-format gate (repo formats to 100). The reformat is AST-neutral — an AST-normalized comparison and a 400-trial randomized differential against the production scorer confirmed float-for-float parity is unaffected — but the ported text is no longer byte-identical to production, so the module docstring now claims an exact semantic port rather than a verbatim one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Official Rules incorporate
src/examples/scoring.pyas the controlling scoring methodology, so it must carry the full contest calculation, not just the percentile transform.OLSFit,ols_fit(univariate surprise-only benchmark),ols_fit2(two-regressor model via the centered 2×2 normal equations). Same guards as production:Noneon too-few points, zero variance, or a singular system — and deliberately no univariate fallback.impute_predictionscomputes the submission-specific mean over its timely, valid predictions on the common Scored Events set before filling, then imputes it into every missed event. No coverage ⇒None(mean undefined, not prize-eligible).score_submission: both metric families for one submission — the no-fill fit on its own predicted events and the_imputedfamily over the full common sample, with the imputed surprise-only benchmark fit once on the full common sample (submission-invariant). Emitsn_obs(pre-imputation coverage, the tie-break), all seven metrics per family,imputed_mean,imputed_event_count.rank_submissions: contest ordering — metric (descending; ascending for the MSE family), then higher pre-imputation coverage, then submission id (display-only; residual exact ties share the combined prize per the Rules — no random tie-break). Undefined metric ⇒ unranked, dropped.Note for merging: §5 pins a frozen version of this file — the post-merge commit hash on
mainis what should be stamped into the Rules doc before the Aug 9 registration deadline.