Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/guides/protocol_style_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ CPP signature is P1; the exploratory no-label first look is P2:
5 engineer features 6 compositional vs positional
7 select & reduce features 8 classifier
9 interpretability 10 validate ("can I trust this?")
11 benchmark ("is this number comparable?")
```

This is a **living catalog**: append protocols as the package grows.
Expand Down
Binary file added docs/source/_static/img/thumbs/protocol11.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 7 additions & 0 deletions docs/source/index/evaluation/eval_feature_selection.rst
Original file line number Diff line number Diff line change
Expand Up @@ -101,3 +101,10 @@ and does not license:
For the canonical short definitions of ``selection_scope`` and the four regimes, see the
project glossary. For the mechanism itself, see :func:`~aaanalysis.pipe.find_features` and
its ``selection_scope`` parameter.

Naming the regime fixes what a score *means*, but not whether two scores may be compared.
That needs the dataset, the split, the metric and the seed pinned as well, which is the
subject of :doc:`P11: Benchmark protocol </generated/protocol11_benchmark>`. It runs the
first regime above deliberately (the feature set is selected once, then frozen) so that a
rerun measures the method rather than a reshuffled feature set, and it records the
resulting scores together with the seed spread that separates noise from a regression.
6 changes: 5 additions & 1 deletion docs/source/protocols.rst
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ protocol; click it to open that protocol.
<a href="generated/protocol8_prediction.html"><img src="_static/img/thumbs/protocol8.png" alt="P8: Prediction"><div class="cap">P8: Prediction</div></a>
<a href="generated/protocol9_interpretability.html"><img src="_static/img/thumbs/protocol9.png" alt="P9: Interpretability"><div class="cap">P9: Interpretability</div></a>
<a href="generated/protocol10_validation.html"><img src="_static/img/thumbs/protocol10.png" alt="P10: Validation"><div class="cap">P10: Validation</div></a>
<a href="generated/protocol11_benchmark.html"><img src="_static/img/thumbs/protocol11.png" alt="P11: Benchmark protocol"><div class="cap">P11: Benchmark protocol</div></a>
</div>

AAanalysis turns a biological *question* into an
Expand All @@ -61,7 +62,9 @@ distinguish them), and the rest of the pipeline helps you sample fairly, enginee
features, select what matters, predict, explain, and check that the signal is
real. The catalog follows that data flow, opening with the CPP signature, then an
exploratory no-label first look, and on through sampling, feature engineering,
selection, modelling, explanation, and validation.
selection, modelling, explanation, and validation. It closes with the **benchmark
protocol**, which pins the dataset, split, metric and seed a score has to come from
before two scores can be compared at all.

.. toctree::
:maxdepth: 1
Expand All @@ -77,3 +80,4 @@ selection, modelling, explanation, and validation.
generated/protocol8_prediction
generated/protocol9_interpretability
generated/protocol10_validation
generated/protocol11_benchmark
Loading
Loading