fix(data): seed load_dataset's random sampling with random_state - #593
Open
breimanntools wants to merge 1 commit into
Open
breimanntools wants to merge 1 commit into
breimanntools wants to merge 1 commit into
Conversation
`aa.load_dataset(random=True)` had no seed, so the first line of most workflows drew a different sample on every call and `options['random_state']` never reached it: a benchmark, tutorial or bug report written with `random=True` could not be reproduced by its own author. `load_dataset` now takes `random_state` (appended after `verbose`, so every positional call keeps working), resolved through `ut.check_random_state` like AAclust, CPP and dPULearn, which also gives it the documented `options['random_state']` override. The per-class draws share one `numpy.random.RandomState` instance, so the classes continue one stream instead of repeating it, and the legacy generator's permanent stream guarantee makes a seed reproduce the same frame in another process or environment. Nothing else changes. `random=False` keeps the deterministic head-of-class selection, and an unseeded `random=True` keeps drawing from numpy's global state: over all 14 bundled datasets (83 loading variants x 5 `PYTHONHASHSEED` values, 415 comparisons) the sha256 of the returned frame is identical to a pristine export of the parent commit, and with the global numpy state pinned the unseeded random path matches byte for byte too. An unseeded `random=True` deliberately stays silent: no other stochastic entry point in the package warns on `random_state=None`, `random=True` is an explicit opt-in to randomness, and a warning here would fire in the example notebook and every docs build for a documented default. Closes #582 Co-Authored-By: Claude Fable 5.1 <[email protected]>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #593 +/- ##
=======================================
Coverage 95.39% 95.39%
=======================================
Files 222 222
Lines 23387 23393 +6
Branches 4073 4074 +1
=======================================
+ Hits 22309 22315 +6
Misses 631 631
Partials 447 447
🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
aa.load_dataset(random=True)was not reproducible: norandom_state, andaa.options["random_state"]did not reach it. Three successive calls returned three different protein sets. Sinceload_datasetis the first line of most workflows, that nondeterminism propagated into everything downstream.Appended after
verbose(the same positionAAclust.__init__uses), so every existing positional call keeps working.No second mechanism: the Validate block gained one line,
random_state = ut.check_random_state(random_state=random_state)— the same helperAAclust,CPP,dPULearn,TreeModelandShapModeluse — sooptions["random_state"]overrides an explicit argument exactly as documented elsewhere. Both directions are asserted.The seed reaches the draw through one shared legacy
RandomState, passed to both class draws: sharing one instance means the second class continues the stream rather than repeating it, andRandomStatecarries a cross-version stream guarantee, which is what makes "identical across processes" true rather than best-effort.random=Falseis byte-identical — measured, not assumedA pristine package was exported from the parent commit with
git archive, then 83 digests (14 bundled datasets + Overview, 5–7 variants each: sha256 overto_csv, plus shape, plus per-column dtypes) were compared against the branch underPYTHONHASHSEED0, 1, 2, 3, 42: 415/415 identical, 0 differences. The unseededrandom=Truepath is unchanged too, checked with the global numpy state pinned.No warning for unseeded
random=TrueDeliberate: no other stochastic entry point in the package warns on
random_state=None,random=Trueis an explicit opt-in rather than a soft failure, and a warning would fire in the package's own notebooks and every docs build. The docstring carries the guidance instead.Verification
18 new tests (26 collected) including a cross-process check via subprocess, a hypothesis sweep over seeds, the options-override precedence, and
random=Falseunaffected by either. 768 tests pass. pyright: 4 pre-existing errors on this file, the same 4 on master — 0 new. Notebook re-executed with bothrandomandrandom_stateby name.It uncovered a bigger problem — #588
While proving determinism, the same measurement showed
load_datasetis not deterministic even withrandom=False: 30 of 83 digests change withPYTHONHASHSEED, because the class blocks are ordered by asetof string labels. Andnon_canonical_aa='gap'builds a regex character class from a set, so with-in play it can form a range —[U-X]silently replaces canonical V and W with gaps,[X-U]raisesbad character range.Both are filed as #588 (prio:1) rather than fixed here: the ordering fix changes committed
AA_*notebook outputs, which is a deliberate output change needing its own re-execution pass.Refs #582.
🤖 Generated with Claude Code