Repository navigation
test(parser): count-bound the differential-parse sample floor - #2750
Merged
Merged
Conversation
The nightly sets CsCheck_Time=15, which turns HotspotModule.Sample(..., iter: 1500) into a 15 s run. The generator is slow, so on a busy hosted runner the in-subset floor (samples.Count > 50) became a wall-clock floor: nightly runs 44458291834 (41 samples) and 43676725192 (48) went red with no parser mismatch. refs #2458 (cell 6890413350). Change (CPythonDifferentialParseTests only): - Collection is an explicit draw loop: candidate i is drawn from its own PCG(1, SplitMix64(seed, i)), drawn in parallel batches and walked in index order, so the result is fixed by the seed whatever the thread count. It stops at targetCount in-subset samples or maxCandidates draws. CsCheck_Time no longer reaches it (Gen.Generate, not Sample). - The seed is random per run, printed, and SHARPY_DIFFPARSE_SEED replays it (checked: seed 7919 replays 150/303 exactly). - New floor: in-subset / candidates >= 0.35, plus the unchanged samples > 50, plus dual-parsed > 50 after the mismatch asserts. - The samples/candidates/ratio/seed line prints before the floors. - Depth: targetCount = 150 x scale, maxCandidates = 1500 x scale, scale = PropertyIterations.Injection / 100 clamped to [1, 6]; the nightly's SHARPY_PROPERTY_ITER=600 gives scale 6 (900 samples). Measured (4 cores, python 3.15.2), scale 1, 30 seeds (7919 x 1..30): ratio min 0.495, median 0.553, max 0.649, s.d. 0.032; 231-303 candidates; dual-parsed >= 149; 20-42 s per collection. r = 0.35. Scale 6 (CsCheck_Time=15, SHARPY_PROPERTY_ITER=600): 900 samples from 1554 candidates, ratio 0.579, 181 s collection, 3 m 1 s test. Guard (rule 12): - Regression control, original code @ 6d6c5c9 with CsCheck_Time=1: red, "Too few in-subset samples generated (8)". New code with CsCheck_Time=1: green (see the PR for the run). - Mutation: SubsetFilter.IsInSharedSubset additionally returned false for every in-subset module whose unparsed source's FNV-1a hash was not 0 mod 10 (rejects ~90%). Seed 7919: red, "In-subset ratio 0.071 (106/1500) is below 0.35". The old floor alone (> 50) would have passed with 106. Restored from a cp copy (cmp clean): green, 150/303, ratio 0.495. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01LGMKZXE9K8LiDXK2rj4doP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CPythonDifferentialParseTests.SharedSubset_SharpyAndCPythonAgreenow collects its samples by count instead of by time.CsCheck_Timeno longer shrinks the sample. The floor now checks what its message means: the in-subset ratio. Test-only; one file. Driven on the owner's word (2026-11-19: "drive it").The defect
The nightly sets
CsCheck_Time=15. That turnsHotspotModule.Sample(..., iter: 1500)into a 15-second run. The generator is slow, so on a busy hosted runner the floorsamples.Count > 50became a wall-clock floor:mainline@ 36987bc): 41 samples.mainline@ 7ea270f): 48 samples.Neither run reported a parser mismatch.
The change
PCG(1, SplitMix64(seed, i))withGen.Generate. Batches are drawn in parallel and walked in index order, so the seed alone fixes the result, whatever the thread count. Collection stops attargetCountin-subset samples ormaxCandidatesdraws, whichever comes first.SHARPY_DIFFPARSE_SEEDreplays it.inSubset / candidates ≥ 0.35(new).samples > 50(unchanged).dual-parsed > 50(new), checked after the mismatch assertions.targetCount = 150 × scaleandmaxCandidates = 1500 × scale.scale = PropertyIterations.Injection / 100, clamped to [1, 6].SHARPY_PROPERTY_ITER=600gives scale 6.Measurements (worker, 4 cores, python 3.15.2)
rResults @ 33aef23:
CPythonDifferentialParseTests: 23/0/0 unset, atCsCheck_Time=15withSHARPY_PROPERTY_ITER=600, and atCsCheck_Time=1.Properties.Parser: 61/0/0.Mutation record (rule 12)
CsCheck_Time=1is red ("Too few … (8)"). The new code is green.IsInSharedSubsetalso rejects in-subset modules whose source's FNV-1a hash is not 0 mod 10 (about 90%).> 50floor would have passed.cpcopy: green, 150/303, ratio 0.495.refs #2458 (cell 6890413350)
Gate: .NET 10 run 44474619503 success, Python Build Tools 44474619608 success, Commit Subjects 44474619463 success @ 33aef23 (measured)
Independent review @ 33aef23: merge; 0 applied, 3 declined (reasons below)
The reviewer ran
SharedSubset_SharpyAndCPythonAgreesix times on the built head with CPython 3.15.2, and it passed every time. Seed 7919 replayed identically at 4 cores and at 1 core. Declined, all marked optional by the reviewer:SHARPY_DIFFPARSE_SEEDfalls back to a random seed without a warning. The printed line shows the seed actually used, so a mistyped replay is visible.> 50can no longer fail. The> 50check stays as a backstop.https://claude.ai/code/session_01LGMKZXE9K8LiDXK2rj4doP
Generated by Claude Code