Skip to content

test(parser): count-bound the differential-parse sample floor - #2750

Merged
antonsynd merged 1 commit into
devfrom
claude/diffparse-sample-floor
Nov 19, 2026
Merged

antonsynd merged 1 commit into
devfrom
claude/diffparse-sample-floor

Conversation

@antonsynd

@antonsynd antonsynd commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Summary

CPythonDifferentialParseTests.SharedSubset_SharpyAndCPythonAgree now collects its samples by count instead of by time. CsCheck_Time no longer shrinks the sample. The floor now checks what its message means: the in-subset ratio. Test-only; one file. Driven on the owner's word (2026-11-19: "drive it").

The defect

The nightly sets CsCheck_Time=15. That turns HotspotModule.Sample(..., iter: 1500) into a 15-second run. The generator is slow, so on a busy hosted runner the floor samples.Count > 50 became a wall-clock floor:

  • Nightly run 44458291834 (2026-11-19, mainline @ 36987bc): 41 samples.
  • Nightly run 43676725192 (2026-11-15, mainline @ 7ea270f): 48 samples.

Neither run reported a parser mismatch.

The change

  • Collection: an explicit draw loop. Candidate i is drawn from its own PCG(1, SplitMix64(seed, i)) with Gen.Generate. Batches are drawn in parallel and walked in index order, so the seed alone fixes the result, whatever the thread count. Collection stops at targetCount in-subset samples or maxCandidates draws, whichever comes first.
  • Seed: random per run and printed; SHARPY_DIFFPARSE_SEED replays it.
  • Floors:
    • inSubset / candidates ≥ 0.35 (new).
    • samples > 50 (unchanged).
    • dual-parsed > 50 (new), checked after the mismatch assertions.
  • Output: the samples/candidates/ratio/seed line is printed before the floors.
  • Nightly depth:
    • targetCount = 150 × scale and maxCandidates = 1500 × scale.
    • scale = PropertyIterations.Injection / 100, clamped to [1, 6].
    • The nightly's SHARPY_PROPERTY_ITER=600 gives scale 6.

Measurements (worker, 4 cores, python 3.15.2)

Result
Scale 1, 30 seeds ratio min 0.495 / median 0.553 / max 0.649, s.d. 0.032; 231–303 candidates; dual-parsed ≥ 149; 20–42 s each
Chosen r 0.35 (0.145 below the measured min, about 4.5 s.d.)
Scale 6 (nightly settings) 900 samples from 1554 candidates, ratio 0.579, 3 m 1 s
Unset about 30 s (was about 2.5 min)

Results @ 33aef23:

  • Build: succeeded, 0 warnings.
  • CPythonDifferentialParseTests: 23/0/0 unset, at CsCheck_Time=15 with SHARPY_PROPERTY_ITER=600, and at CsCheck_Time=1.
  • Properties.Parser: 61/0/0.
  • Whitespace gate: clean.

Mutation record (rule 12)

  • Control: the original code with CsCheck_Time=1 is red ("Too few … (8)"). The new code is green.
  • Mutation:
    • Change: IsInSharedSubset also rejects in-subset modules whose source's FNV-1a hash is not 0 mod 10 (about 90%).
    • Result, seed 7919: red, "In-subset ratio 0.071 (106/1500) is below 0.35". The old > 50 floor would have passed.
  • Restore: restored from the cp copy: green, 150/303, ratio 0.495.

refs #2458 (cell 6890413350)

Gate: .NET 10 run 44474619503 success, Python Build Tools 44474619608 success, Commit Subjects 44474619463 success @ 33aef23 (measured)

Independent review @ 33aef23: merge; 0 applied, 3 declined (reasons below)

The reviewer ran SharedSubset_SharpyAndCPythonAgree six times on the built head with CPython 3.15.2, and it passed every time. Seed 7919 replayed identically at 4 cores and at 1 core. Declined, all marked optional by the reviewer:

  • A malformed SHARPY_DIFFPARSE_SEED falls back to a random seed without a warning. The printed line shows the seed actually used, so a mistyped replay is visible.
  • The seed is printed after collection, so a crash inside the generator would leave no seed. The old code printed none at all, so this is not a regression.
  • Cosmetic: the random seed never sets bit 63, and once the ratio floor passes, > 50 can no longer fail. The > 50 check stays as a backstop.
sharpy-issues:
  closes: []
  files: []

https://claude.ai/code/session_01LGMKZXE9K8LiDXK2rj4doP


Generated by Claude Code

The nightly sets CsCheck_Time=15, which turns HotspotModule.Sample(...,
iter: 1500) into a 15 s run. The generator is slow, so on a busy hosted
runner the in-subset floor (samples.Count > 50) became a wall-clock
floor: nightly runs 44458291834 (41 samples) and 43676725192 (48) went
red with no parser mismatch. refs #2458 (cell 6890413350).

Change (CPythonDifferentialParseTests only):
- Collection is an explicit draw loop: candidate i is drawn from its own
  PCG(1, SplitMix64(seed, i)), drawn in parallel batches and walked in
  index order, so the result is fixed by the seed whatever the thread
  count. It stops at targetCount in-subset samples or maxCandidates
  draws. CsCheck_Time no longer reaches it (Gen.Generate, not Sample).
- The seed is random per run, printed, and SHARPY_DIFFPARSE_SEED
  replays it (checked: seed 7919 replays 150/303 exactly).
- New floor: in-subset / candidates >= 0.35, plus the unchanged
  samples > 50, plus dual-parsed > 50 after the mismatch asserts.
- The samples/candidates/ratio/seed line prints before the floors.
- Depth: targetCount = 150 x scale, maxCandidates = 1500 x scale,
  scale = PropertyIterations.Injection / 100 clamped to [1, 6]; the
  nightly's SHARPY_PROPERTY_ITER=600 gives scale 6 (900 samples).

Measured (4 cores, python 3.15.2), scale 1, 30 seeds (7919 x 1..30):
ratio min 0.495, median 0.553, max 0.649, s.d. 0.032; 231-303
candidates; dual-parsed >= 149; 20-42 s per collection. r = 0.35.
Scale 6 (CsCheck_Time=15, SHARPY_PROPERTY_ITER=600): 900 samples from
1554 candidates, ratio 0.579, 181 s collection, 3 m 1 s test.

Guard (rule 12):
- Regression control, original code @ 6d6c5c9 with CsCheck_Time=1:
  red, "Too few in-subset samples generated (8)". New code with
  CsCheck_Time=1: green (see the PR for the run).
- Mutation: SubsetFilter.IsInSharedSubset additionally returned false
  for every in-subset module whose unparsed source's FNV-1a hash was
  not 0 mod 10 (rejects ~90%). Seed 7919: red, "In-subset ratio 0.071
  (106/1500) is below 0.35". The old floor alone (> 50) would have
  passed with 106. Restored from a cp copy (cmp clean): green, 150/303,
  ratio 0.495.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LGMKZXE9K8LiDXK2rj4doP
@antonsynd
antonsynd merged commit a2876b2 into dev Nov 19, 2026
35 of 36 checks passed
@antonsynd
antonsynd deleted the claude/diffparse-sample-floor branch November 19, 2026 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants