diff --git a/.github/workflows/deploy-docs.yml b/.github/workflows/deploy-docs.yml index 9570fde..9d3c276 100644 --- a/.github/workflows/deploy-docs.yml +++ b/.github/workflows/deploy-docs.yml @@ -44,15 +44,36 @@ jobs: - name: Install docs dependencies run: pip install -r requirements-docs.txt - - name: Install GenAIRR from PyPI (mkdocstrings needs to import it) + - name: Set up Rust + uses: dtolnay/rust-toolchain@stable + + - name: Cache cargo build + uses: actions/cache@v4 + with: + path: | + ~/.cargo/registry + ~/.cargo/git + engine_rs/target + key: ${{ runner.os }}-docs-cargo-${{ hashFiles('engine_rs/Cargo.lock') }} + restore-keys: ${{ runner.os }}-docs-cargo- + + - name: Install GenAIRR from this checkout (mkdocstrings needs to import it) # The reference pages use `:::` mkdocstrings directives to render # signatures + docstrings from real classes (Experiment, - # SimulationResult, ReferenceCartridgeBuilder, etc.). For those - # to resolve, GenAIRR must be importable in the build environment. - # Installing from PyPI is fast (precompiled wheels) and means the - # rendered API reference matches the latest released version, - # which is what published docs should reflect. - run: pip install GenAIRR + # SimulationResult, ReferenceCartridgeBuilder, etc.). For those to + # resolve, GenAIRR must be importable in the build environment. + # + # This used to install from PyPI, on the reasoning that published + # docs should reflect the published release. That inverts the + # dependency: `site_docs/` is versioned with the source, so a page + # documenting anything newer than the last release cannot build. It + # broke on `GenAIRR._reference.artifact.Reference` and the deploy + # then failed on every merge to master from 2026-10-21 until the + # 2.4.0 release work, which is why the live site served + # pre-release text for a fortnight. Building the checkout keeps the + # rendered reference and the prose describing it in step by + # construction. + run: pip install . - name: Build site with mkdocs build --strict run: make docs-build diff --git a/CHANGELOG.md b/CHANGELOG.md index 470e2f4..f5c45e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -148,6 +148,28 @@ have. ### Added +- **Published germline sources, and dossiers anyone can verify offline.** The + new public module `GenAIRR.reference_sources` locks one *named* published + germline release and keeps the evidence. `OgrdbSource` acquires an explicit + OGRDB germline-set release -- an integer release, never "latest" -- and pins + its exact bytes in a lock; `fetch` is the only operation in GenAIRR that uses + the network, and `from_files` and `read` cover the offline and + behind-a-proxy cases. `source.build(recipe)` projects an explicit recipe into + a `ReferenceBuild`: a runtime `Reference` you pass straight to + `Experiment.on(...)`, plus the lock, the recipe, the build report and a + manifest recording the GenAIRR version that produced it. + `build.write_dossier(directory)` writes the whole thing as a portable + dossier, and `ReferenceBuild.read_dossier(directory)` checks one with no + network and no trust in its author: it regenerates every derived file from + the locked bytes and the recipe and compares, so a single tampered byte comes + back as `ReferenceSourceError(kind='digest_mismatch', path=...)` naming the + member. Failures on this surface all carry a stable `kind` and a `path`. + One build ships inside the wheel -- + `ReferenceBuild.bundled("human_igh_ogrdb_10_observed_v1")` -- so the workflow + is available with no network at all; it also reports IMGT allele names, where + `"human_igh"` reports ASC cluster names. See + [Published germline sources](https://mutejester.github.io/GenAIRR/guides/published-references.html). + - **Two clonal models beside the deprecated star.** `Experiment.clonal_lineage(...)` grows a real BCR genealogy per clone: a generation-synchronous birth-death process with per-division S5F diff --git a/MIGRATING.md b/MIGRATING.md index ec78bd1..3dece6a 100644 --- a/MIGRATING.md +++ b/MIGRATING.md @@ -190,6 +190,28 @@ Two further counter changes to know about: A genotype run with novel alleles must be validated against `compiled.refdata`. +## 7. What you may now want to use + +Everything above is about breakage. These are the 2.4.0 additions most +likely to change how you work, in case the migration is the only part of +the release you read: + +- **Pin a published germline release, and ship the evidence.** + `GenAIRR.reference_sources` locks one named OGRDB release and writes a + dossier a reviewer can verify offline, with no network and no trust in + you. `ReferenceBuild.bundled("human_igh_ogrdb_10_observed_v1")` needs + no network at all. See + [Published germline sources](https://mutejester.github.io/GenAIRR/guides/published-references.html). +- **Real clonal genealogies.** `clonal_lineage()` replaces the star + topology with a birth-death process and hands back the ground-truth + tree per clone; `clonal_repertoire()` covers TCR and flat-BCR + abundance. Both are the intended replacements for `expand_clones()` + (§3). +- **Per-individual diploid genotypes**, with population priors, cohorts + and a genotype truth sidecar for scoring inference. +- The full list is the `### Added` section of + [CHANGELOG.md](CHANGELOG.md). + --- # Migrating from GenAIRR 1.x to 2.0 diff --git a/mkdocs.yml b/mkdocs.yml index 3b5ebdb..a26c8cb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -137,33 +137,33 @@ plugins: # NOTE: no 'learn.md' redirect — learn.md is a real nav page # ("Choose your path"); a redirect here would clobber the built # learn.html with a stub and make the page unreachable. - "lesson-1.md": "getting-started/lesson-1-vdj-recombination-from-bases-up.md" - "lesson-2.md": "getting-started/lesson-2-the-pipeline-scrubber.md" - "lesson-3.md": "getting-started/lesson-3-s5f-mutation-isnt-uniform.md" - "lesson-4.md": "getting-started/lesson-4-sequencing-artifacts-dialed-in.md" - "lesson-5.md": "getting-started/lesson-5-the-ground-truth-payoff.md" + "lesson-1.md": "getting-started/index.md" + "lesson-2.md": "getting-started/index.md" + "lesson-3.md": "getting-started/index.md" + "lesson-4.md": "getting-started/index.md" + "lesson-5.md": "getting-started/index.md" "concepts.md": "concepts/index.md" - "concept-pipeline.md": "concepts/pipeline.md" - "concept-persistent-ir.md": "concepts/persistent-ir.md" - "concept-contracts.md": "concepts/contracts.md" + "concept-pipeline.md": "concepts/index.md" + "concept-persistent-ir.md": "concepts/index.md" + "concept-contracts.md": "concepts/index.md" "concept-airr-record.md": "concepts/airr-record.md" - "concept-live-call.md": "concepts/live-call.md" + "concept-live-call.md": "concepts/index.md" "guides.md": "guides/index.md" - "guide-build-config.md": "guides/build-config.md" + "guide-build-config.md": "guides/build-reference-cartridge.md" "guide-productive.md": "guides/productive.md" "guide-clonal-families.md": "guides/clonal-families.md" - "guide-v-usage.md": "guides/v-usage.md" - "guide-tune-corruption.md": "guides/tune-corruption.md" - "guide-export.md": "guides/export.md" - "guide-streaming.md": "guides/streaming.md" - "guide-with-metadata.md": "guides/with-metadata.md" - "guide-audit-realism.md": "validation/audit-realism.md" - "guide-benchmark-aligner.md": "validation/benchmark-aligner.md" - "guide-compare-shm.md": "validation/compare-shm.md" - "guide-replay.md": "validation/replay.md" - "guide-reproduce.md": "validation/reproduce.md" - "guide-smoke-test.md": "validation/smoke-test.md" - "guide-trace-introspection.md": "validation/trace-introspection.md" + "guide-v-usage.md": "guides/recombination-junction.md" + "guide-tune-corruption.md": "guides/corruption-sequencing.md" + "guide-export.md": "getting-started/export-results.md" + "guide-streaming.md": "reference/index.md" + "guide-with-metadata.md": "getting-started/export-results.md" + "guide-audit-realism.md": "validation/index.md" + "guide-benchmark-aligner.md": "validation/index.md" + "guide-compare-shm.md": "validation/index.md" + "guide-replay.md": "guides/trace-replay.md" + "guide-reproduce.md": "guides/trace-replay.md" + "guide-smoke-test.md": "validation/index.md" + "guide-trace-introspection.md": "guides/trace-replay.md" "reference.md": "reference/index.md" - mkdocstrings: handlers: @@ -214,6 +214,7 @@ nav: - Corruption + sequencing artefacts: guides/corruption-sequencing.md - Paired-end reads and FASTQ: guides/paired-end-fastq.md - Portable local references: guides/portable-references.md + - Published germline sources: guides/published-references.md - Reference Cartridge Authoring: - cartridges/index.md - Reference evidence and quarantine: cartridges/reference-evidence.md diff --git a/site_docs/architecture/adding-a-pass.md b/site_docs/architecture/adding-a-pass.md deleted file mode 100644 index 31164c0..0000000 --- a/site_docs/architecture/adding-a-pass.md +++ /dev/null @@ -1,8 +0,0 @@ -# Adding a pass - - -_Linked from `docs/adding_a_pass.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/architecture/audit-index.md b/site_docs/architecture/audit-index.md deleted file mode 100644 index d1fd67e..0000000 --- a/site_docs/architecture/audit-index.md +++ /dev/null @@ -1,8 +0,0 @@ -# Audit + design doc index - - -_Auto-generated catalogue of every `docs/*audit.md` and `docs/*design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/architecture/engine.md b/site_docs/architecture/engine.md deleted file mode 100644 index 466783d..0000000 --- a/site_docs/architecture/engine.md +++ /dev/null @@ -1,8 +0,0 @@ -# Engine architecture - - -_Linked from `docs/engine_architecture.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/architecture/validation-matrix.md b/site_docs/architecture/validation-matrix.md deleted file mode 100644 index 5fd94a2..0000000 --- a/site_docs/architecture/validation-matrix.md +++ /dev/null @@ -1,8 +0,0 @@ -# Validation matrix - - -_Linked from `docs/validation_matrix.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Validation & Debugging](https://mutejester.github.io/GenAIRR/validation/index.html). diff --git a/site_docs/cartridges/build-from-fasta.md b/site_docs/cartridges/build-from-fasta.md deleted file mode 100644 index 540ff12..0000000 --- a/site_docs/cartridges/build-from-fasta.md +++ /dev/null @@ -1,11 +0,0 @@ -# Build a cartridge from FASTA - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Build a reference cartridge](https://mutejester.github.io/GenAIRR/guides/build-reference-cartridge.html). - -New authoring also preserves source-header declarations and carries optional -sequence-bound landmark, coding-phase and coding-boundary evidence. Missing -species and unresolved anchors remain explicit. See -[Reference evidence and catalogue quarantine](reference-evidence.md) for the -schema, source requirements and bundled audit status. diff --git a/site_docs/cartridges/empirical-models.md b/site_docs/cartridges/empirical-models.md deleted file mode 100644 index 3a74fec..0000000 --- a/site_docs/cartridges/empirical-models.md +++ /dev/null @@ -1,5 +0,0 @@ -# Empirical models - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/allele-usage.md b/site_docs/cartridges/estimators/allele-usage.md deleted file mode 100644 index 3241463..0000000 --- a/site_docs/cartridges/estimators/allele-usage.md +++ /dev/null @@ -1,8 +0,0 @@ -# Allele usage estimator - - -_Rewritten from `docs/allele_usage_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/index.md b/site_docs/cartridges/estimators/index.md deleted file mode 100644 index bf40b13..0000000 --- a/site_docs/cartridges/estimators/index.md +++ /dev/null @@ -1,8 +0,0 @@ -# Estimators overview - - -_Hub for the 5 cartridge estimators._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/np-base-model.md b/site_docs/cartridges/estimators/np-base-model.md deleted file mode 100644 index f2a4c20..0000000 --- a/site_docs/cartridges/estimators/np-base-model.md +++ /dev/null @@ -1,8 +0,0 @@ -# NP base model estimator - - -_Rewritten from `docs/np_base_model_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/np-length.md b/site_docs/cartridges/estimators/np-length.md deleted file mode 100644 index bc9ba2e..0000000 --- a/site_docs/cartridges/estimators/np-length.md +++ /dev/null @@ -1,8 +0,0 @@ -# NP length estimator - - -_Rewritten from `docs/np_length_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/p-nucleotide-length.md b/site_docs/cartridges/estimators/p-nucleotide-length.md deleted file mode 100644 index 31d6018..0000000 --- a/site_docs/cartridges/estimators/p-nucleotide-length.md +++ /dev/null @@ -1,8 +0,0 @@ -# P-nucleotide length estimator - - -_Rewritten from `docs/p_nucleotide_length_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/trim-distributions.md b/site_docs/cartridges/estimators/trim-distributions.md deleted file mode 100644 index 3d74c62..0000000 --- a/site_docs/cartridges/estimators/trim-distributions.md +++ /dev/null @@ -1,8 +0,0 @@ -# Trim distributions estimator - - -_Rewritten from `docs/trim_distribution_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/rules-and-identity.md b/site_docs/cartridges/rules-and-identity.md deleted file mode 100644 index 6d4eff8..0000000 --- a/site_docs/cartridges/rules-and-identity.md +++ /dev/null @@ -1,5 +0,0 @@ -# Rules + identity - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Reference Cartridge (concept)](https://mutejester.github.io/GenAIRR/concepts/reference-cartridge.html). diff --git a/site_docs/cartridges/v-subregion-annotations.md b/site_docs/cartridges/v-subregion-annotations.md deleted file mode 100644 index 693def9..0000000 --- a/site_docs/cartridges/v-subregion-annotations.md +++ /dev/null @@ -1,8 +0,0 @@ -# V-subregion annotations - - -_Rewritten from `docs/v_region_substructure_audit.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Targeted SHM rates](https://mutejester.github.io/GenAIRR/guides/shm-targeting.html). diff --git a/site_docs/concepts/contracts.md b/site_docs/concepts/contracts.md deleted file mode 100644 index 471a5e5..0000000 --- a/site_docs/concepts/contracts.md +++ /dev/null @@ -1,5 +0,0 @@ -# Contracts - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Sample only productive sequences](https://mutejester.github.io/GenAIRR/guides/productive.html). diff --git a/site_docs/concepts/live-call.md b/site_docs/concepts/live-call.md deleted file mode 100644 index 84bff3f..0000000 --- a/site_docs/concepts/live-call.md +++ /dev/null @@ -1,5 +0,0 @@ -# Live Calls - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/concepts/persistent-ir.md b/site_docs/concepts/persistent-ir.md deleted file mode 100644 index 722e778..0000000 --- a/site_docs/concepts/persistent-ir.md +++ /dev/null @@ -1,5 +0,0 @@ -# Persistent IR - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/concepts/pipeline.md b/site_docs/concepts/pipeline.md deleted file mode 100644 index 0c0adf8..0000000 --- a/site_docs/concepts/pipeline.md +++ /dev/null @@ -1,5 +0,0 @@ -# The Simulation Pipeline - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [The Experiment builder](https://mutejester.github.io/GenAIRR/guides/experiment-builder.html). diff --git a/site_docs/concepts/reference-cartridge.md b/site_docs/concepts/reference-cartridge.md index 31305a8..ee1c536 100644 --- a/site_docs/concepts/reference-cartridge.md +++ b/site_docs/concepts/reference-cartridge.md @@ -175,8 +175,13 @@ ship typed defaults that users override per-experiment. !!! warning "Which bundled cartridges carry *real* data-derived distributions" Only the human **IGH**, **IGK**, **IGL**, and **TCRB** cartridges ship with empirical distributions fitted from **real repertoire - data** — trim lengths, NP-region lengths, the NP base / Markov - model, and allele usage. + data** — trim lengths, NP-region lengths, and the NP base / Markov + model. + + Allele usage is **not** among them. No bundled cartridge ships an + `allele_usage` plane, so allele draws are uniform over the + catalogue until you weight them yourself — see + [Controlling the allele universe](../guides/recombination-junction.md#controlling-the-allele-universe). Every **other** bundled species/locus cartridge carries a real germline **allele catalogue** (from IMGT), but its data-derived @@ -417,10 +422,10 @@ legacy fields are sitting on the cartridge unused. For the original design audit and the engine-side `RefDataConfig` bridge, see -[`docs/reference_cartridge.md`](https://github.com/MuteJester/GenAIRR/blob/master/docs/reference_cartridge.md) +[`audit-docs/reference_cartridge.md`](https://github.com/MuteJester/GenAIRR/blob/master/audit-docs/reference_cartridge.md) and -[`docs/reference_cartridge_authoring_audit.md`](https://github.com/MuteJester/GenAIRR/blob/master/docs/reference_cartridge_authoring_audit.md). +[`audit-docs/reference_cartridge_authoring_audit.md`](https://github.com/MuteJester/GenAIRR/blob/master/audit-docs/reference_cartridge_authoring_audit.md). For per-estimator audits (allele usage, trims, NP length, NP base, -P-nucleotide length), see the matching design docs in `docs/`. For +P-nucleotide length), see the matching design docs in `audit-docs/`. For the full engine-wide validation matrix, see -[`docs/validation_matrix.md`](https://github.com/MuteJester/GenAIRR/blob/master/docs/validation_matrix.md). +[`audit-docs/validation_matrix.md`](https://github.com/MuteJester/GenAIRR/blob/master/audit-docs/validation_matrix.md). diff --git a/site_docs/getting-started/export-results.md b/site_docs/getting-started/export-results.md index 95469b4..4304399 100644 --- a/site_docs/getting-started/export-results.md +++ b/site_docs/getting-started/export-results.md @@ -57,7 +57,7 @@ Writes one record per sequence with the `sequence_id` plus `v_call` and `j_call` in the header: ```text ->seq0 IGHVF10-G38*04 IGHJ2*01 +>seq0|v_call=IGHVF6-G23*03|j_call=IGHJ1*01 gaggtgcagctggtg... ``` @@ -65,14 +65,42 @@ Pass `prefix="myseq"` to change the per-record ID prefix. ### Blinded handoff - strip truth columns -FASTA is the natural format when the next stage is an aligner -that should NOT see the ground-truth columns - e.g. an aligner -benchmark or a tool comparison where leaking `v_call` / -`truth_v_call` would defeat the point. `to_fasta` already -emits only `sequence_id` + the assembled sequence, so it's -implicitly blinded. +When the next stage is a tool you are scoring, it must not see the +generating calls. The exporters differ in what they carry: -For a blinded TSV, two options: +| Exporter | Carries truth? | What ends up in the header / row | +|---|---|---| +| `to_paired_fastq(r1, r2)` | No | `@seq0/1` and `@seq0/2` - `sequence_id` only | +| `to_fasta(path)` | **Yes** | `>seq0|v_call=...|j_call=...` | +| `to_fastq(path)` | **Yes** | `@seq0|v_call=...|j_call=...` | +| `to_tsv` / `to_csv` / `to_dataframe` | **Yes** | `v_call`, `d_call`, `j_call`, every counter, plus `truth_*` under `expose_provenance=True` | + +`to_fasta` and `to_fastq` annotate their headers on purpose, which is +convenient when you are the consumer and wrong when the consumer is a +tool under test. For a blinded panel, write the records yourself - +`sequence_id` is all the downstream join needs: + +```python +import GenAIRR as ga + +result = ( + ga.Experiment.on("human_igh") + .recombine() + .productive_only() + .run_records(n=10_000, seed=42) +) + +# Blinded FASTA - sequence_id and bases, nothing else. +with open("panel_blinded.fasta", "w") as fh: + for rec in result: + fh.write(f">{rec['sequence_id']}\n{rec['sequence'].upper()}\n") +``` + +`.upper()` is deliberate: the corruption passes lowercase the bases +they touch, and some aligners read case as a soft-mask. Normalise on +the way out and keep the original `sequence` in your truth table. + +For the truth side, two options: ```python # Option A - write a regular TSV, then drop truth columns @@ -95,10 +123,70 @@ result = ( result.to_tsv("panel_blinded.tsv") ``` -Option B is the default - `truth_*` columns are only emitted -when you pass `expose_provenance=True` to `run_records(...)`. -Option A is the recipe for *post hoc* blinding when the same -batch needs to be shared in both forms. +Option B is the default - `truth_*` columns are only emitted when you +pass `expose_provenance=True` to `run_records(...)`. Option A is the +recipe for *post hoc* blinding when the same run has to produce both +the panel and the answer key. + +Either way, note what Option B does not do: it blinds the `truth_*` +aliases, not the record. `v_call`, `d_call` and `j_call` are always the +generating alleles. A panel for a tool you are scoring should be FASTA +or FASTQ written as above, not a TSV with columns removed one at a +time. + +### Keeping the truth table beside the panel + +The usual shape is two files joined on `sequence_id`: + +```python +result = ( + ga.Experiment.on("human_igh") + .recombine() + .productive_only() + .mutate(model="s5f", rate=0.03) + .run_records(n=10_000, seed=42, expose_provenance=True) +) + +# 1. the blinded input the tool sees +with open("panel.fasta", "w") as fh: + for rec in result: + fh.write(f">{rec['sequence_id']}\n{rec['sequence'].upper()}\n") + +# 2. the answer key you keep +result.to_tsv("panel_truth.tsv") +``` + +Run the tool on `panel.fasta`, then join its output back on +`sequence_id` - see +[How `sequence_id` joins everything](#how-sequence_id-joins-everything). +Two things to settle before you score: + +- **Allele naming.** On `"human_igh"` the truth calls are ASC cluster + names (`IGHVF6-G23*03`), while an aligner against an IMGT reference + emits IMGT names (`IGHV4-31*10`). Translate one side using + `cfg.asc_tables["V"]`, or simulate on an IMGT-named cartridge + instead: + + ```python + from GenAIRR.data import HUMAN_IGH_IMGT + + result = ( + ga.Experiment.on(HUMAN_IGH_IMGT) + .curate_refdata("functional_anchors_only") + .recombine() + .run_records(n=10_000, seed=42) + ) + ``` + +- **Which truth column.** `v_call` is the allele the engine + *generated*. `recalled_v_call` is what GenAIRR's own walker infers + from the mutated sequence, and it may be a comma-separated tie. + Score against `v_call` to ask "did the tool recover the generating + allele"; against `recalled_v_call` to ask "did it agree with an + independent caller on the same evidence". The two diverge exactly + where mutation has made the generating allele unrecoverable, which + is the interesting part of the benchmark. See + [Truth and evidence-based re-calls](first-airr-record.md#truth-and-evidence-based-re-calls). ## FASTQ diff --git a/site_docs/getting-started/lesson-1-vdj-recombination-from-bases-up.md b/site_docs/getting-started/lesson-1-vdj-recombination-from-bases-up.md deleted file mode 100644 index 4512eaa..0000000 --- a/site_docs/getting-started/lesson-1-vdj-recombination-from-bases-up.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 1 - V(D)J recombination, from bases up - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Quick start](https://mutejester.github.io/GenAIRR/getting-started/quick-start.html). diff --git a/site_docs/getting-started/lesson-2-the-pipeline-scrubber.md b/site_docs/getting-started/lesson-2-the-pipeline-scrubber.md deleted file mode 100644 index b91b95a..0000000 --- a/site_docs/getting-started/lesson-2-the-pipeline-scrubber.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 2 - The pipeline scrubber - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [The Experiment builder](https://mutejester.github.io/GenAIRR/guides/experiment-builder.html). diff --git a/site_docs/getting-started/lesson-3-s5f-mutation-isnt-uniform.md b/site_docs/getting-started/lesson-3-s5f-mutation-isnt-uniform.md deleted file mode 100644 index d66812d..0000000 --- a/site_docs/getting-started/lesson-3-s5f-mutation-isnt-uniform.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 3 - S5F: mutation isn't uniform - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Targeted SHM rates](https://mutejester.github.io/GenAIRR/guides/shm-targeting.html). diff --git a/site_docs/getting-started/lesson-4-sequencing-artifacts-dialed-in.md b/site_docs/getting-started/lesson-4-sequencing-artifacts-dialed-in.md deleted file mode 100644 index cc549ad..0000000 --- a/site_docs/getting-started/lesson-4-sequencing-artifacts-dialed-in.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 4 - Sequencing artifacts, dialed in - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Corruption + sequencing artefacts](https://mutejester.github.io/GenAIRR/guides/corruption-sequencing.html). diff --git a/site_docs/getting-started/lesson-5-the-ground-truth-payoff.md b/site_docs/getting-started/lesson-5-the-ground-truth-payoff.md deleted file mode 100644 index b8c15f8..0000000 --- a/site_docs/getting-started/lesson-5-the-ground-truth-payoff.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 5 - The ground-truth payoff - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Your first AIRR record](https://mutejester.github.io/GenAIRR/getting-started/first-airr-record.html). diff --git a/site_docs/guides/biology-map.md b/site_docs/guides/biology-map.md index f0d34c3..8939582 100644 --- a/site_docs/guides/biology-map.md +++ b/site_docs/guides/biology-map.md @@ -27,6 +27,7 @@ better starting point. | **D inversion** | `.invert_d(prob=...)` | Recombination editing | `d_inverted` | [D inversion + receptor revision](recombination-editing.md) | | **Receptor revision** | `.receptor_revision(prob=...)` | Recombination editing | `receptor_revision_applied`, `original_v_call`, post-revision `v_call` | [D inversion + receptor revision](recombination-editing.md) | | **Productivity constraint** | `.productive_only()` | Constraint-aware sampling | `productive=True` (by construction) | [Experiment builder](experiment-builder.md#how-productive_only-works) | +| **Allele usage** | `.recombine(v_allele_weights=..., d_allele_weights=..., j_allele_weights=...)` (experiment) + `cfg.reference_models.allele_usage` (cartridge) | Recombination | Which alleles `v_call`/`d_call`/`j_call` draw, and how often. Default is uniform over the catalogue: no bundled cartridge ships a usage plane | [Recombination + junction biology](recombination-junction.md#controlling-the-allele-universe) | | **Allele restriction** | `.restrict_alleles(v=..., d=..., j=...)` | Constraint-aware sampling | Sampled `v_call`/`d_call`/`j_call` restricted to the listed alleles | [Recombination + junction biology](recombination-junction.md) | | **Somatic hypermutation (SHM)** | `.mutate(model="s5f", rate=...)` or `.mutate(model="uniform", count=...)` | Biology - descendant phase | `n_mutations`, `mutation_rate`, `n_v_mutations`, `n_d_mutations`, `n_j_mutations`, `n_np_mutations` | [SHM and mutation targeting](shm-targeting.md) | | **Targeted SHM** | `.mutate(..., segment_rates=..., v_subregion_rates=...)` | Biology - descendant phase | Same plus the six V-subregion counters (`n_fwr1_mutations` … `n_v_unannotated_mutations`) | [SHM and mutation targeting](shm-targeting.md) | diff --git a/site_docs/guides/build-config.md b/site_docs/guides/build-config.md deleted file mode 100644 index 01575f5..0000000 --- a/site_docs/guides/build-config.md +++ /dev/null @@ -1,5 +0,0 @@ -# Build a config for your locus or species - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Build a reference cartridge](https://mutejester.github.io/GenAIRR/guides/build-reference-cartridge.html) or [Reference Cartridge Authoring](https://mutejester.github.io/GenAIRR/cartridges/index.html). diff --git a/site_docs/guides/export.md b/site_docs/guides/export.md deleted file mode 100644 index c0b321e..0000000 --- a/site_docs/guides/export.md +++ /dev/null @@ -1,5 +0,0 @@ -# Export to AIRR TSV, FASTA, or FASTQ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Export the results](https://mutejester.github.io/GenAIRR/getting-started/export-results.html). diff --git a/site_docs/guides/published-references.md b/site_docs/guides/published-references.md new file mode 100644 index 0000000..fdf44c8 --- /dev/null +++ b/site_docs/guides/published-references.md @@ -0,0 +1,199 @@ +# Published germline sources and dossiers + +
When a result has to be defensible a year from now, "we +used the current IMGT release" is not a reference. This page locks one +named published germline release, builds a runtime reference +from it, and writes a dossier that anyone can verify offline without +trusting you, the network, or the registry.
+ +Three names do all of it, from `GenAIRR.reference_sources`: + +| Name | What it is for | +|---|---| +| `OgrdbSource` | Acquire and pin one explicit OGRDB germline-set release | +| `ReferenceBuild` | A runtime `Reference` plus the complete record of how it was made | +| `ReferenceSourceError` | Every failure here, with a stable `kind` and a `path` | + +This is a different thing from +[Portable local references](portable-references.md). There you build a +reference from *your own* FASTA files. Here you pin a *published* +release and keep the evidence that you did. + +## Start from the shipped build + +One build ships inside the wheel, so the whole workflow is available +with no network at all: + +```python +import GenAIRR as ga +from GenAIRR.reference_sources import ReferenceBuild + +build = ReferenceBuild.bundled("human_igh_ogrdb_10_observed_v1") + +result = ( + ga.Experiment.on(build.reference) + .recombine() + .run_records(n=1000, seed=0) +) +result.records[0]["v_call"] # 'IGHV4-30-2*06' - IMGT names +``` + +Ask for a name that does not exist and the error lists the ones that +do, so the catalogue is always one call away: + +```python +ReferenceBuild.bundled("nope") +# ReferenceSourceError(kind='missing_asset'): +# no bundled reference build is named exactly +# 'human_igh_ogrdb_10_observed_v1' +``` + +!!! note "This is also the simplest route to IMGT allele names" + The bundled build reports IMGT names (`IGHV4-30-2*06`), where + `"human_igh"` reports ASC cluster names (`IGHVF6-G23*03`). If you + are scoring a tool against an IMGT reference, that matters - see + [Blinded handoff](../getting-started/export-results.md#blinded-handoff-strip-truth-columns). + +## What a build carries + +A `ReferenceBuild` is the reference *and* its provenance. Four +read-only parts: + +| Member | Holds | +|---|---| +| `build.reference` | The runtime `Reference` you pass to `Experiment.on(...)` | +| `build.lock` | The pinned release: provider, `set_id`, every asset and its digest | +| `build.recipe` | The explicit projection - selection, decisions, defaults profile | +| `build.report` | What came out: counts, capabilities, claims, comparison, inventory | +| `build.manifest` | Member versions, hashes, and the generator that wrote it | + +The manifest records the GenAIRR version that produced the build: + +```python +build.manifest.builder +# Generator(name='GenAIRR', version='2.4.0') +``` + +That is deliberate. A reference built by a different version is a +different artifact even when the germline bytes are identical, because +the projection rules may have moved. + +## Write a dossier, verify it offline + +`write_dossier` takes a **directory**, not an archive: + +```python +build.write_dossier("igh_dossier/") +``` + +It writes the named members plus the content-addressed blobs they +refer to: + +```text +source.lock.json recipe.json report.json +reference.genairr-ref.json manifest.json annotations.json +v.fasta d.fasta j.fasta +