docs(protocols): add P11, the benchmark protocol

[breimanntools]

protocols/protocol11_benchmark.ipynb — a reproducible evaluation protocol across the three prediction levels, assembled entirely from what already ships: load_dataset, SequenceFeature, CPP, aa.bind_groups, AAPred.eval(cv=, baseline=), comp_per_protein_ap.

Zero new public symbols, as intended: #25 is the protocol, #488 is the runner that would later automate it, and the runner is not built here.

The four pinned decisions

• Datasets — AA_CASPASE3 (residue), DOM_GSEC (domain), SEQ_AMYLO (protein): the trio the rest of the docs already teach on.
• Split — StratifiedGroupKFold(5, shuffle=True, random_state=42) bound via aa.bind_groups on the protein accession, scored out-of-fold. A benchmark that ignored grouping would teach the very leak this package just fixed.
• Metric — ROC-AUC at all three levels against the aac / dpc composition baselines through identical folds, plus per-protein AP at the residue level.
• Seed — 42 throughout, random=False pinned.

Recorded baselines (seed 42): residue cpp 0.95 / aac 0.85 / dpc 0.87 · domain cpp 0.93 / 0.74 / 0.62 · protein cpp 0.83 / aac 0.91 / dpc 0.90.

The tolerance is derived, not declared: a ten-seed sweep measured a widest across-seed σ of 0.014, so 3σ rounds to 0.05 absolute ROC-AUC. Reproducibility is asserted in the notebook — rerun max absolute difference 0.0.

Three findings it reports honestly

1. load_dataset(random=True) is unseeded, so the protocol pins random=False. Now fixed in fix(data): seed load_dataset's random sampling with random_state #593 / filed as load_dataset is not deterministic: set-ordered class blocks, and a regex character class that can gap canonical residues #588.
2. random=False alone does not give a usable residue benchmark: entries read CASPASE3_<protein>_pos<index> and the deterministic head-of-class selection puts all 30 negatives in one protein at n=30. The protocol therefore defines its own seeded selection rule — and defining that rule is the work the issue asked for.
3. The grouping gap on this data is small (MCC 0.618 ungrouped → 0.576 grouped) and it says so rather than overselling. The concept figure shows the leak structurally instead: df_folds_ reveals the ungrouped split putting all 20 proteins in both halves of every fold versus 16-vs-4 grouped.

Verification

pytest --nbmake protocols/ → 11 passed in 175s (the whole set stayed green). Docs gate: 0 error · 167 critical, baseline held. 341 api_tests. The captured Sphinx log shows no warning mentioning the new pages.

Two things for you

• The scores are post-selection optimistic: CPP features are selected once on the full set before the folds are cut. Kept deliberately — it freezes the feature set so a rerun compares methods rather than reshuffled features — and stated plainly in When to use it and Common mistakes. Running nested selection instead is a real design change and a much slower notebook.
• "All ten protocols" is now stale in notebook_examples.yml, CONTRIBUTING.rst + its docs copy, and .claude/rules/workflow.md. Two are CONFIRM-FIRST, so none were touched; they want one approval together.

Refs #25.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.39%. Comparing base (3c11c90) to head (4e53566).
⚠️ Report is 10 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #596   +/-   ##
=======================================
  Coverage   95.39%   95.39%           
=======================================
  Files         222      222           
  Lines       23387    23387           
  Branches     4073     4073           
=======================================
  Hits        22309    22309           
  Misses        631      631           
  Partials      447      447           

see 2 files with indirect coverage changes

Components	Coverage Δ
cpp_core	95.97% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.