Skip to content

docs(pbt): PBT effectiveness study — sensitivity vs realized yield - #74

Merged
cordialApple merged 3 commits into
mainfrom
docs/pbt-effectiveness-study
Jul 29, 2026
Merged

cordialApple merged 3 commits into
mainfrom
docs/pbt-effectiveness-study

Conversation

@cordialApple

Copy link
Copy Markdown
Owner

Retrospective on what property-based testing bought Scroll, on TreapOrderIndex. Lives in analysis/pbt/ + a new section in the repo README.

Two questions, two answers

  • Sensitivity (could PBT catch faults?): ~100% by 10 cases, and earned — ablations show worst-class detection drop to 80% when the oracle or generator is weakened.
  • Realized yield (did it fix real wrong code?): ~1, and unrecoverable from git. Squash-merge + an in-loop agent fixing defects before commit erased the broken intermediate states. That loss is the finding, not a hedge.

What is here

  • harness/ mutation sweep (20 mutants, 5 classes, budgets 1–3000 × 10 seeds) + ablation, in-process esbuild + dynamic import vs a pristine array/Fenwick oracle.
  • config/ PBT-only Stryker + vitest. results/ raw JSON. data/ star-schema CSVs.
  • viz/ seaborn charts + composite dashboard (the README figure). powerbi/ version-controlled PBIP over the same schema.
  • Root README.md retrospective section; package.json gains Stryker dev deps.

No app source touched.

🤖 Generated with Claude Code

@cordialApple
cordialApple enabled auto-merge (squash) July 29, 2026 05:03
@cordialApple
cordialApple merged commit 7a83c19 into main Jul 29, 2026
3 checks passed
@cordialApple
cordialApple deleted the docs/pbt-effectiveness-study branch July 29, 2026 05:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant