Skip to content

Add the stability report between runs (S8) - #12

Merged
mgilardi merged 2 commits into
mainfrom
s8-stability-report
Nov 15, 2026
Merged

mgilardi merged 2 commits into
mainfrom
s8-stability-report

Conversation

@mgilardi

Copy link
Copy Markdown
Contributor

Closes K408. Replaces #11, rebased on main without force-pushing.

compareRuns / assertStable in onefamily-analysis: labels matched by overlap; stop when more than 5% of people placed in both runs change line at two levels; deeper levels, groups, newly placed and no longer placed reported. 'Too few' (under 50) is reported, not a stop, like S7's side check.

Tested: 282 tests, typecheck, leak check. Real exercise outside the repo: original run vs OneFamily's on the first finished kit, 1.9% changed, stable. Not wired into the app (one run per kit so far).

🤖 Generated with Claude Code

mgilardi and others added 2 commits November 15, 2026 06:20
A new run of a kit should leave most people on the line they were on.
compareRuns matches branch labels by overlap, so renumbering isn't
counted as change, and assertStable stops when more than 5% of the people
placed in both runs change line (side and branch). Deeper levels and
groups are reported without stopping, since third-level splits are often
borderline.

Tested on made-up runs and a planted family. On the first finished kit's
real data, outside the repo, the original run against OneFamily's came out
stable at 1.9%. The app keeps one run per kit, so nothing calls it yet.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@mgilardi
mgilardi merged commit b41a1a0 into main Nov 15, 2026
2 checks passed
@mgilardi
mgilardi deleted the s8-stability-report branch November 15, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant