diff --git a/.github/workflows/deploy-docs.yml b/.github/workflows/deploy-docs.yml index 9570fde..9d3c276 100644 --- a/.github/workflows/deploy-docs.yml +++ b/.github/workflows/deploy-docs.yml @@ -44,15 +44,36 @@ jobs: - name: Install docs dependencies run: pip install -r requirements-docs.txt - - name: Install GenAIRR from PyPI (mkdocstrings needs to import it) + - name: Set up Rust + uses: dtolnay/rust-toolchain@stable + + - name: Cache cargo build + uses: actions/cache@v4 + with: + path: | + ~/.cargo/registry + ~/.cargo/git + engine_rs/target + key: ${{ runner.os }}-docs-cargo-${{ hashFiles('engine_rs/Cargo.lock') }} + restore-keys: ${{ runner.os }}-docs-cargo- + + - name: Install GenAIRR from this checkout (mkdocstrings needs to import it) # The reference pages use `:::` mkdocstrings directives to render # signatures + docstrings from real classes (Experiment, - # SimulationResult, ReferenceCartridgeBuilder, etc.). For those - # to resolve, GenAIRR must be importable in the build environment. - # Installing from PyPI is fast (precompiled wheels) and means the - # rendered API reference matches the latest released version, - # which is what published docs should reflect. - run: pip install GenAIRR + # SimulationResult, ReferenceCartridgeBuilder, etc.). For those to + # resolve, GenAIRR must be importable in the build environment. + # + # This used to install from PyPI, on the reasoning that published + # docs should reflect the published release. That inverts the + # dependency: `site_docs/` is versioned with the source, so a page + # documenting anything newer than the last release cannot build. It + # broke on `GenAIRR._reference.artifact.Reference` and the deploy + # then failed on every merge to master from 2026-10-21 until the + # 2.4.0 release work, which is why the live site served + # pre-release text for a fortnight. Building the checkout keeps the + # rendered reference and the prose describing it in step by + # construction. + run: pip install . - name: Build site with mkdocs build --strict run: make docs-build diff --git a/CHANGELOG.md b/CHANGELOG.md index 470e2f4..f5c45e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -148,6 +148,28 @@ have. ### Added +- **Published germline sources, and dossiers anyone can verify offline.** The + new public module `GenAIRR.reference_sources` locks one *named* published + germline release and keeps the evidence. `OgrdbSource` acquires an explicit + OGRDB germline-set release -- an integer release, never "latest" -- and pins + its exact bytes in a lock; `fetch` is the only operation in GenAIRR that uses + the network, and `from_files` and `read` cover the offline and + behind-a-proxy cases. `source.build(recipe)` projects an explicit recipe into + a `ReferenceBuild`: a runtime `Reference` you pass straight to + `Experiment.on(...)`, plus the lock, the recipe, the build report and a + manifest recording the GenAIRR version that produced it. + `build.write_dossier(directory)` writes the whole thing as a portable + dossier, and `ReferenceBuild.read_dossier(directory)` checks one with no + network and no trust in its author: it regenerates every derived file from + the locked bytes and the recipe and compares, so a single tampered byte comes + back as `ReferenceSourceError(kind='digest_mismatch', path=...)` naming the + member. Failures on this surface all carry a stable `kind` and a `path`. + One build ships inside the wheel -- + `ReferenceBuild.bundled("human_igh_ogrdb_10_observed_v1")` -- so the workflow + is available with no network at all; it also reports IMGT allele names, where + `"human_igh"` reports ASC cluster names. See + [Published germline sources](https://mutejester.github.io/GenAIRR/guides/published-references.html). + - **Two clonal models beside the deprecated star.** `Experiment.clonal_lineage(...)` grows a real BCR genealogy per clone: a generation-synchronous birth-death process with per-division S5F diff --git a/MIGRATING.md b/MIGRATING.md index ec78bd1..3dece6a 100644 --- a/MIGRATING.md +++ b/MIGRATING.md @@ -190,6 +190,28 @@ Two further counter changes to know about: A genotype run with novel alleles must be validated against `compiled.refdata`. +## 7. What you may now want to use + +Everything above is about breakage. These are the 2.4.0 additions most +likely to change how you work, in case the migration is the only part of +the release you read: + +- **Pin a published germline release, and ship the evidence.** + `GenAIRR.reference_sources` locks one named OGRDB release and writes a + dossier a reviewer can verify offline, with no network and no trust in + you. `ReferenceBuild.bundled("human_igh_ogrdb_10_observed_v1")` needs + no network at all. See + [Published germline sources](https://mutejester.github.io/GenAIRR/guides/published-references.html). +- **Real clonal genealogies.** `clonal_lineage()` replaces the star + topology with a birth-death process and hands back the ground-truth + tree per clone; `clonal_repertoire()` covers TCR and flat-BCR + abundance. Both are the intended replacements for `expand_clones()` + (§3). +- **Per-individual diploid genotypes**, with population priors, cohorts + and a genotype truth sidecar for scoring inference. +- The full list is the `### Added` section of + [CHANGELOG.md](CHANGELOG.md). + --- # Migrating from GenAIRR 1.x to 2.0 diff --git a/mkdocs.yml b/mkdocs.yml index 3b5ebdb..a26c8cb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -137,33 +137,33 @@ plugins: # NOTE: no 'learn.md' redirect — learn.md is a real nav page # ("Choose your path"); a redirect here would clobber the built # learn.html with a stub and make the page unreachable. - "lesson-1.md": "getting-started/lesson-1-vdj-recombination-from-bases-up.md" - "lesson-2.md": "getting-started/lesson-2-the-pipeline-scrubber.md" - "lesson-3.md": "getting-started/lesson-3-s5f-mutation-isnt-uniform.md" - "lesson-4.md": "getting-started/lesson-4-sequencing-artifacts-dialed-in.md" - "lesson-5.md": "getting-started/lesson-5-the-ground-truth-payoff.md" + "lesson-1.md": "getting-started/index.md" + "lesson-2.md": "getting-started/index.md" + "lesson-3.md": "getting-started/index.md" + "lesson-4.md": "getting-started/index.md" + "lesson-5.md": "getting-started/index.md" "concepts.md": "concepts/index.md" - "concept-pipeline.md": "concepts/pipeline.md" - "concept-persistent-ir.md": "concepts/persistent-ir.md" - "concept-contracts.md": "concepts/contracts.md" + "concept-pipeline.md": "concepts/index.md" + "concept-persistent-ir.md": "concepts/index.md" + "concept-contracts.md": "concepts/index.md" "concept-airr-record.md": "concepts/airr-record.md" - "concept-live-call.md": "concepts/live-call.md" + "concept-live-call.md": "concepts/index.md" "guides.md": "guides/index.md" - "guide-build-config.md": "guides/build-config.md" + "guide-build-config.md": "guides/build-reference-cartridge.md" "guide-productive.md": "guides/productive.md" "guide-clonal-families.md": "guides/clonal-families.md" - "guide-v-usage.md": "guides/v-usage.md" - "guide-tune-corruption.md": "guides/tune-corruption.md" - "guide-export.md": "guides/export.md" - "guide-streaming.md": "guides/streaming.md" - "guide-with-metadata.md": "guides/with-metadata.md" - "guide-audit-realism.md": "validation/audit-realism.md" - "guide-benchmark-aligner.md": "validation/benchmark-aligner.md" - "guide-compare-shm.md": "validation/compare-shm.md" - "guide-replay.md": "validation/replay.md" - "guide-reproduce.md": "validation/reproduce.md" - "guide-smoke-test.md": "validation/smoke-test.md" - "guide-trace-introspection.md": "validation/trace-introspection.md" + "guide-v-usage.md": "guides/recombination-junction.md" + "guide-tune-corruption.md": "guides/corruption-sequencing.md" + "guide-export.md": "getting-started/export-results.md" + "guide-streaming.md": "reference/index.md" + "guide-with-metadata.md": "getting-started/export-results.md" + "guide-audit-realism.md": "validation/index.md" + "guide-benchmark-aligner.md": "validation/index.md" + "guide-compare-shm.md": "validation/index.md" + "guide-replay.md": "guides/trace-replay.md" + "guide-reproduce.md": "guides/trace-replay.md" + "guide-smoke-test.md": "validation/index.md" + "guide-trace-introspection.md": "guides/trace-replay.md" "reference.md": "reference/index.md" - mkdocstrings: handlers: @@ -214,6 +214,7 @@ nav: - Corruption + sequencing artefacts: guides/corruption-sequencing.md - Paired-end reads and FASTQ: guides/paired-end-fastq.md - Portable local references: guides/portable-references.md + - Published germline sources: guides/published-references.md - Reference Cartridge Authoring: - cartridges/index.md - Reference evidence and quarantine: cartridges/reference-evidence.md diff --git a/site_docs/architecture/adding-a-pass.md b/site_docs/architecture/adding-a-pass.md deleted file mode 100644 index 31164c0..0000000 --- a/site_docs/architecture/adding-a-pass.md +++ /dev/null @@ -1,8 +0,0 @@ -# Adding a pass - - -_Linked from `docs/adding_a_pass.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/architecture/audit-index.md b/site_docs/architecture/audit-index.md deleted file mode 100644 index d1fd67e..0000000 --- a/site_docs/architecture/audit-index.md +++ /dev/null @@ -1,8 +0,0 @@ -# Audit + design doc index - - -_Auto-generated catalogue of every `docs/*audit.md` and `docs/*design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/architecture/engine.md b/site_docs/architecture/engine.md deleted file mode 100644 index 466783d..0000000 --- a/site_docs/architecture/engine.md +++ /dev/null @@ -1,8 +0,0 @@ -# Engine architecture - - -_Linked from `docs/engine_architecture.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/architecture/validation-matrix.md b/site_docs/architecture/validation-matrix.md deleted file mode 100644 index 5fd94a2..0000000 --- a/site_docs/architecture/validation-matrix.md +++ /dev/null @@ -1,8 +0,0 @@ -# Validation matrix - - -_Linked from `docs/validation_matrix.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Validation & Debugging](https://mutejester.github.io/GenAIRR/validation/index.html). diff --git a/site_docs/cartridges/build-from-fasta.md b/site_docs/cartridges/build-from-fasta.md deleted file mode 100644 index 540ff12..0000000 --- a/site_docs/cartridges/build-from-fasta.md +++ /dev/null @@ -1,11 +0,0 @@ -# Build a cartridge from FASTA - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Build a reference cartridge](https://mutejester.github.io/GenAIRR/guides/build-reference-cartridge.html). - -New authoring also preserves source-header declarations and carries optional -sequence-bound landmark, coding-phase and coding-boundary evidence. Missing -species and unresolved anchors remain explicit. See -[Reference evidence and catalogue quarantine](reference-evidence.md) for the -schema, source requirements and bundled audit status. diff --git a/site_docs/cartridges/empirical-models.md b/site_docs/cartridges/empirical-models.md deleted file mode 100644 index 3a74fec..0000000 --- a/site_docs/cartridges/empirical-models.md +++ /dev/null @@ -1,5 +0,0 @@ -# Empirical models - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/allele-usage.md b/site_docs/cartridges/estimators/allele-usage.md deleted file mode 100644 index 3241463..0000000 --- a/site_docs/cartridges/estimators/allele-usage.md +++ /dev/null @@ -1,8 +0,0 @@ -# Allele usage estimator - - -_Rewritten from `docs/allele_usage_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/index.md b/site_docs/cartridges/estimators/index.md deleted file mode 100644 index bf40b13..0000000 --- a/site_docs/cartridges/estimators/index.md +++ /dev/null @@ -1,8 +0,0 @@ -# Estimators overview - - -_Hub for the 5 cartridge estimators._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/np-base-model.md b/site_docs/cartridges/estimators/np-base-model.md deleted file mode 100644 index f2a4c20..0000000 --- a/site_docs/cartridges/estimators/np-base-model.md +++ /dev/null @@ -1,8 +0,0 @@ -# NP base model estimator - - -_Rewritten from `docs/np_base_model_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/np-length.md b/site_docs/cartridges/estimators/np-length.md deleted file mode 100644 index bc9ba2e..0000000 --- a/site_docs/cartridges/estimators/np-length.md +++ /dev/null @@ -1,8 +0,0 @@ -# NP length estimator - - -_Rewritten from `docs/np_length_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/p-nucleotide-length.md b/site_docs/cartridges/estimators/p-nucleotide-length.md deleted file mode 100644 index 31d6018..0000000 --- a/site_docs/cartridges/estimators/p-nucleotide-length.md +++ /dev/null @@ -1,8 +0,0 @@ -# P-nucleotide length estimator - - -_Rewritten from `docs/p_nucleotide_length_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/trim-distributions.md b/site_docs/cartridges/estimators/trim-distributions.md deleted file mode 100644 index 3d74c62..0000000 --- a/site_docs/cartridges/estimators/trim-distributions.md +++ /dev/null @@ -1,8 +0,0 @@ -# Trim distributions estimator - - -_Rewritten from `docs/trim_distribution_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/rules-and-identity.md b/site_docs/cartridges/rules-and-identity.md deleted file mode 100644 index 6d4eff8..0000000 --- a/site_docs/cartridges/rules-and-identity.md +++ /dev/null @@ -1,5 +0,0 @@ -# Rules + identity - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Reference Cartridge (concept)](https://mutejester.github.io/GenAIRR/concepts/reference-cartridge.html). diff --git a/site_docs/cartridges/v-subregion-annotations.md b/site_docs/cartridges/v-subregion-annotations.md deleted file mode 100644 index 693def9..0000000 --- a/site_docs/cartridges/v-subregion-annotations.md +++ /dev/null @@ -1,8 +0,0 @@ -# V-subregion annotations - - -_Rewritten from `docs/v_region_substructure_audit.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Targeted SHM rates](https://mutejester.github.io/GenAIRR/guides/shm-targeting.html). diff --git a/site_docs/concepts/contracts.md b/site_docs/concepts/contracts.md deleted file mode 100644 index 471a5e5..0000000 --- a/site_docs/concepts/contracts.md +++ /dev/null @@ -1,5 +0,0 @@ -# Contracts - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Sample only productive sequences](https://mutejester.github.io/GenAIRR/guides/productive.html). diff --git a/site_docs/concepts/live-call.md b/site_docs/concepts/live-call.md deleted file mode 100644 index 84bff3f..0000000 --- a/site_docs/concepts/live-call.md +++ /dev/null @@ -1,5 +0,0 @@ -# Live Calls - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/concepts/persistent-ir.md b/site_docs/concepts/persistent-ir.md deleted file mode 100644 index 722e778..0000000 --- a/site_docs/concepts/persistent-ir.md +++ /dev/null @@ -1,5 +0,0 @@ -# Persistent IR - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/concepts/pipeline.md b/site_docs/concepts/pipeline.md deleted file mode 100644 index 0c0adf8..0000000 --- a/site_docs/concepts/pipeline.md +++ /dev/null @@ -1,5 +0,0 @@ -# The Simulation Pipeline - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [The Experiment builder](https://mutejester.github.io/GenAIRR/guides/experiment-builder.html). diff --git a/site_docs/concepts/reference-cartridge.md b/site_docs/concepts/reference-cartridge.md index 31305a8..ee1c536 100644 --- a/site_docs/concepts/reference-cartridge.md +++ b/site_docs/concepts/reference-cartridge.md @@ -175,8 +175,13 @@ ship typed defaults that users override per-experiment. !!! warning "Which bundled cartridges carry *real* data-derived distributions" Only the human **IGH**, **IGK**, **IGL**, and **TCRB** cartridges ship with empirical distributions fitted from **real repertoire - data** — trim lengths, NP-region lengths, the NP base / Markov - model, and allele usage. + data** — trim lengths, NP-region lengths, and the NP base / Markov + model. + + Allele usage is **not** among them. No bundled cartridge ships an + `allele_usage` plane, so allele draws are uniform over the + catalogue until you weight them yourself — see + [Controlling the allele universe](../guides/recombination-junction.md#controlling-the-allele-universe). Every **other** bundled species/locus cartridge carries a real germline **allele catalogue** (from IMGT), but its data-derived @@ -417,10 +422,10 @@ legacy fields are sitting on the cartridge unused. For the original design audit and the engine-side `RefDataConfig` bridge, see -[`docs/reference_cartridge.md`](https://github.com/MuteJester/GenAIRR/blob/master/docs/reference_cartridge.md) +[`audit-docs/reference_cartridge.md`](https://github.com/MuteJester/GenAIRR/blob/master/audit-docs/reference_cartridge.md) and -[`docs/reference_cartridge_authoring_audit.md`](https://github.com/MuteJester/GenAIRR/blob/master/docs/reference_cartridge_authoring_audit.md). +[`audit-docs/reference_cartridge_authoring_audit.md`](https://github.com/MuteJester/GenAIRR/blob/master/audit-docs/reference_cartridge_authoring_audit.md). For per-estimator audits (allele usage, trims, NP length, NP base, -P-nucleotide length), see the matching design docs in `docs/`. For +P-nucleotide length), see the matching design docs in `audit-docs/`. For the full engine-wide validation matrix, see -[`docs/validation_matrix.md`](https://github.com/MuteJester/GenAIRR/blob/master/docs/validation_matrix.md). +[`audit-docs/validation_matrix.md`](https://github.com/MuteJester/GenAIRR/blob/master/audit-docs/validation_matrix.md). diff --git a/site_docs/getting-started/export-results.md b/site_docs/getting-started/export-results.md index 95469b4..4304399 100644 --- a/site_docs/getting-started/export-results.md +++ b/site_docs/getting-started/export-results.md @@ -57,7 +57,7 @@ Writes one record per sequence with the `sequence_id` plus `v_call` and `j_call` in the header: ```text ->seq0 IGHVF10-G38*04 IGHJ2*01 +>seq0|v_call=IGHVF6-G23*03|j_call=IGHJ1*01 gaggtgcagctggtg... ``` @@ -65,14 +65,42 @@ Pass `prefix="myseq"` to change the per-record ID prefix. ### Blinded handoff - strip truth columns -FASTA is the natural format when the next stage is an aligner -that should NOT see the ground-truth columns - e.g. an aligner -benchmark or a tool comparison where leaking `v_call` / -`truth_v_call` would defeat the point. `to_fasta` already -emits only `sequence_id` + the assembled sequence, so it's -implicitly blinded. +When the next stage is a tool you are scoring, it must not see the +generating calls. The exporters differ in what they carry: -For a blinded TSV, two options: +| Exporter | Carries truth? | What ends up in the header / row | +|---|---|---| +| `to_paired_fastq(r1, r2)` | No | `@seq0/1` and `@seq0/2` - `sequence_id` only | +| `to_fasta(path)` | **Yes** | `>seq0|v_call=...|j_call=...` | +| `to_fastq(path)` | **Yes** | `@seq0|v_call=...|j_call=...` | +| `to_tsv` / `to_csv` / `to_dataframe` | **Yes** | `v_call`, `d_call`, `j_call`, every counter, plus `truth_*` under `expose_provenance=True` | + +`to_fasta` and `to_fastq` annotate their headers on purpose, which is +convenient when you are the consumer and wrong when the consumer is a +tool under test. For a blinded panel, write the records yourself - +`sequence_id` is all the downstream join needs: + +```python +import GenAIRR as ga + +result = ( + ga.Experiment.on("human_igh") + .recombine() + .productive_only() + .run_records(n=10_000, seed=42) +) + +# Blinded FASTA - sequence_id and bases, nothing else. +with open("panel_blinded.fasta", "w") as fh: + for rec in result: + fh.write(f">{rec['sequence_id']}\n{rec['sequence'].upper()}\n") +``` + +`.upper()` is deliberate: the corruption passes lowercase the bases +they touch, and some aligners read case as a soft-mask. Normalise on +the way out and keep the original `sequence` in your truth table. + +For the truth side, two options: ```python # Option A - write a regular TSV, then drop truth columns @@ -95,10 +123,70 @@ result = ( result.to_tsv("panel_blinded.tsv") ``` -Option B is the default - `truth_*` columns are only emitted -when you pass `expose_provenance=True` to `run_records(...)`. -Option A is the recipe for *post hoc* blinding when the same -batch needs to be shared in both forms. +Option B is the default - `truth_*` columns are only emitted when you +pass `expose_provenance=True` to `run_records(...)`. Option A is the +recipe for *post hoc* blinding when the same run has to produce both +the panel and the answer key. + +Either way, note what Option B does not do: it blinds the `truth_*` +aliases, not the record. `v_call`, `d_call` and `j_call` are always the +generating alleles. A panel for a tool you are scoring should be FASTA +or FASTQ written as above, not a TSV with columns removed one at a +time. + +### Keeping the truth table beside the panel + +The usual shape is two files joined on `sequence_id`: + +```python +result = ( + ga.Experiment.on("human_igh") + .recombine() + .productive_only() + .mutate(model="s5f", rate=0.03) + .run_records(n=10_000, seed=42, expose_provenance=True) +) + +# 1. the blinded input the tool sees +with open("panel.fasta", "w") as fh: + for rec in result: + fh.write(f">{rec['sequence_id']}\n{rec['sequence'].upper()}\n") + +# 2. the answer key you keep +result.to_tsv("panel_truth.tsv") +``` + +Run the tool on `panel.fasta`, then join its output back on +`sequence_id` - see +[How `sequence_id` joins everything](#how-sequence_id-joins-everything). +Two things to settle before you score: + +- **Allele naming.** On `"human_igh"` the truth calls are ASC cluster + names (`IGHVF6-G23*03`), while an aligner against an IMGT reference + emits IMGT names (`IGHV4-31*10`). Translate one side using + `cfg.asc_tables["V"]`, or simulate on an IMGT-named cartridge + instead: + + ```python + from GenAIRR.data import HUMAN_IGH_IMGT + + result = ( + ga.Experiment.on(HUMAN_IGH_IMGT) + .curate_refdata("functional_anchors_only") + .recombine() + .run_records(n=10_000, seed=42) + ) + ``` + +- **Which truth column.** `v_call` is the allele the engine + *generated*. `recalled_v_call` is what GenAIRR's own walker infers + from the mutated sequence, and it may be a comma-separated tie. + Score against `v_call` to ask "did the tool recover the generating + allele"; against `recalled_v_call` to ask "did it agree with an + independent caller on the same evidence". The two diverge exactly + where mutation has made the generating allele unrecoverable, which + is the interesting part of the benchmark. See + [Truth and evidence-based re-calls](first-airr-record.md#truth-and-evidence-based-re-calls). ## FASTQ diff --git a/site_docs/getting-started/lesson-1-vdj-recombination-from-bases-up.md b/site_docs/getting-started/lesson-1-vdj-recombination-from-bases-up.md deleted file mode 100644 index 4512eaa..0000000 --- a/site_docs/getting-started/lesson-1-vdj-recombination-from-bases-up.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 1 - V(D)J recombination, from bases up - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Quick start](https://mutejester.github.io/GenAIRR/getting-started/quick-start.html). diff --git a/site_docs/getting-started/lesson-2-the-pipeline-scrubber.md b/site_docs/getting-started/lesson-2-the-pipeline-scrubber.md deleted file mode 100644 index b91b95a..0000000 --- a/site_docs/getting-started/lesson-2-the-pipeline-scrubber.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 2 - The pipeline scrubber - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [The Experiment builder](https://mutejester.github.io/GenAIRR/guides/experiment-builder.html). diff --git a/site_docs/getting-started/lesson-3-s5f-mutation-isnt-uniform.md b/site_docs/getting-started/lesson-3-s5f-mutation-isnt-uniform.md deleted file mode 100644 index d66812d..0000000 --- a/site_docs/getting-started/lesson-3-s5f-mutation-isnt-uniform.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 3 - S5F: mutation isn't uniform - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Targeted SHM rates](https://mutejester.github.io/GenAIRR/guides/shm-targeting.html). diff --git a/site_docs/getting-started/lesson-4-sequencing-artifacts-dialed-in.md b/site_docs/getting-started/lesson-4-sequencing-artifacts-dialed-in.md deleted file mode 100644 index cc549ad..0000000 --- a/site_docs/getting-started/lesson-4-sequencing-artifacts-dialed-in.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 4 - Sequencing artifacts, dialed in - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Corruption + sequencing artefacts](https://mutejester.github.io/GenAIRR/guides/corruption-sequencing.html). diff --git a/site_docs/getting-started/lesson-5-the-ground-truth-payoff.md b/site_docs/getting-started/lesson-5-the-ground-truth-payoff.md deleted file mode 100644 index b8c15f8..0000000 --- a/site_docs/getting-started/lesson-5-the-ground-truth-payoff.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 5 - The ground-truth payoff - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Your first AIRR record](https://mutejester.github.io/GenAIRR/getting-started/first-airr-record.html). diff --git a/site_docs/guides/biology-map.md b/site_docs/guides/biology-map.md index f0d34c3..8939582 100644 --- a/site_docs/guides/biology-map.md +++ b/site_docs/guides/biology-map.md @@ -27,6 +27,7 @@ better starting point. | **D inversion** | `.invert_d(prob=...)` | Recombination editing | `d_inverted` | [D inversion + receptor revision](recombination-editing.md) | | **Receptor revision** | `.receptor_revision(prob=...)` | Recombination editing | `receptor_revision_applied`, `original_v_call`, post-revision `v_call` | [D inversion + receptor revision](recombination-editing.md) | | **Productivity constraint** | `.productive_only()` | Constraint-aware sampling | `productive=True` (by construction) | [Experiment builder](experiment-builder.md#how-productive_only-works) | +| **Allele usage** | `.recombine(v_allele_weights=..., d_allele_weights=..., j_allele_weights=...)` (experiment) + `cfg.reference_models.allele_usage` (cartridge) | Recombination | Which alleles `v_call`/`d_call`/`j_call` draw, and how often. Default is uniform over the catalogue: no bundled cartridge ships a usage plane | [Recombination + junction biology](recombination-junction.md#controlling-the-allele-universe) | | **Allele restriction** | `.restrict_alleles(v=..., d=..., j=...)` | Constraint-aware sampling | Sampled `v_call`/`d_call`/`j_call` restricted to the listed alleles | [Recombination + junction biology](recombination-junction.md) | | **Somatic hypermutation (SHM)** | `.mutate(model="s5f", rate=...)` or `.mutate(model="uniform", count=...)` | Biology - descendant phase | `n_mutations`, `mutation_rate`, `n_v_mutations`, `n_d_mutations`, `n_j_mutations`, `n_np_mutations` | [SHM and mutation targeting](shm-targeting.md) | | **Targeted SHM** | `.mutate(..., segment_rates=..., v_subregion_rates=...)` | Biology - descendant phase | Same plus the six V-subregion counters (`n_fwr1_mutations` … `n_v_unannotated_mutations`) | [SHM and mutation targeting](shm-targeting.md) | diff --git a/site_docs/guides/build-config.md b/site_docs/guides/build-config.md deleted file mode 100644 index 01575f5..0000000 --- a/site_docs/guides/build-config.md +++ /dev/null @@ -1,5 +0,0 @@ -# Build a config for your locus or species - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Build a reference cartridge](https://mutejester.github.io/GenAIRR/guides/build-reference-cartridge.html) or [Reference Cartridge Authoring](https://mutejester.github.io/GenAIRR/cartridges/index.html). diff --git a/site_docs/guides/export.md b/site_docs/guides/export.md deleted file mode 100644 index c0b321e..0000000 --- a/site_docs/guides/export.md +++ /dev/null @@ -1,5 +0,0 @@ -# Export to AIRR TSV, FASTA, or FASTQ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Export the results](https://mutejester.github.io/GenAIRR/getting-started/export-results.html). diff --git a/site_docs/guides/published-references.md b/site_docs/guides/published-references.md new file mode 100644 index 0000000..fdf44c8 --- /dev/null +++ b/site_docs/guides/published-references.md @@ -0,0 +1,199 @@ +# Published germline sources and dossiers + +

When a result has to be defensible a year from now, "we +used the current IMGT release" is not a reference. This page locks one +named published germline release, builds a runtime reference +from it, and writes a dossier that anyone can verify offline without +trusting you, the network, or the registry.

+ +Three names do all of it, from `GenAIRR.reference_sources`: + +| Name | What it is for | +|---|---| +| `OgrdbSource` | Acquire and pin one explicit OGRDB germline-set release | +| `ReferenceBuild` | A runtime `Reference` plus the complete record of how it was made | +| `ReferenceSourceError` | Every failure here, with a stable `kind` and a `path` | + +This is a different thing from +[Portable local references](portable-references.md). There you build a +reference from *your own* FASTA files. Here you pin a *published* +release and keep the evidence that you did. + +## Start from the shipped build + +One build ships inside the wheel, so the whole workflow is available +with no network at all: + +```python +import GenAIRR as ga +from GenAIRR.reference_sources import ReferenceBuild + +build = ReferenceBuild.bundled("human_igh_ogrdb_10_observed_v1") + +result = ( + ga.Experiment.on(build.reference) + .recombine() + .run_records(n=1000, seed=0) +) +result.records[0]["v_call"] # 'IGHV4-30-2*06' - IMGT names +``` + +Ask for a name that does not exist and the error lists the ones that +do, so the catalogue is always one call away: + +```python +ReferenceBuild.bundled("nope") +# ReferenceSourceError(kind='missing_asset'): +# no bundled reference build is named exactly +# 'human_igh_ogrdb_10_observed_v1' +``` + +!!! note "This is also the simplest route to IMGT allele names" + The bundled build reports IMGT names (`IGHV4-30-2*06`), where + `"human_igh"` reports ASC cluster names (`IGHVF6-G23*03`). If you + are scoring a tool against an IMGT reference, that matters - see + [Blinded handoff](../getting-started/export-results.md#blinded-handoff-strip-truth-columns). + +## What a build carries + +A `ReferenceBuild` is the reference *and* its provenance. Four +read-only parts: + +| Member | Holds | +|---|---| +| `build.reference` | The runtime `Reference` you pass to `Experiment.on(...)` | +| `build.lock` | The pinned release: provider, `set_id`, every asset and its digest | +| `build.recipe` | The explicit projection - selection, decisions, defaults profile | +| `build.report` | What came out: counts, capabilities, claims, comparison, inventory | +| `build.manifest` | Member versions, hashes, and the generator that wrote it | + +The manifest records the GenAIRR version that produced the build: + +```python +build.manifest.builder +# Generator(name='GenAIRR', version='2.4.0') +``` + +That is deliberate. A reference built by a different version is a +different artifact even when the germline bytes are identical, because +the projection rules may have moved. + +## Write a dossier, verify it offline + +`write_dossier` takes a **directory**, not an archive: + +```python +build.write_dossier("igh_dossier/") +``` + +It writes the named members plus the content-addressed blobs they +refer to: + +```text +source.lock.json recipe.json report.json +reference.genairr-ref.json manifest.json annotations.json +v.fasta d.fasta j.fasta + ... +``` + +Anyone can check it with no network and no trust in you: + +```python +from GenAIRR.reference_sources import ReferenceBuild + +back = ReferenceBuild.read_dossier("igh_dossier/") +back.manifest.hashes == build.manifest.hashes # True +``` + +`read_dossier` does not merely read the stored hashes. It regenerates +every derived file from the locked bytes and the recipe and compares +the result, so a tampered member is caught and located: + +```python +# after changing one byte of reference.genairr-ref.json +ReferenceBuild.read_dossier("igh_dossier/") +# ReferenceSourceError(kind='digest_mismatch', +# path='/reference.genairr-ref.json') +``` + +This is the property worth having in a methods section: the dossier is +checkable by a reviewer who has only the directory. + +## Pin a release yourself + +`OgrdbSource.fetch` is the **only** operation in GenAIRR that uses the +network. It takes one explicit release - never "latest" - and writes a +lock that pins the exact bytes: + +```python +from GenAIRR.reference_sources import OgrdbSource + +source = OgrdbSource.fetch( + set_id="Homo sapiens IGH", + release=10, # an int, never "latest" + profile="ogrdb_api_v2_human_ig_observed_v1", + cache="blobs/", # content-addressed blob directory +) +source.write_lock("igh.lock.json") +``` + +`release` is an explicit positive integer; there is no "latest". A +`profile` names the interpretation rules applied to the release, and +`cache` is the content-addressed directory the downloaded bytes land +in. By default all four exports are acquired (`airr`, `airr_ex`, +`gapped`, `gapped_ex`) - a build compares both record exports, and V +junction anchors are accepted only when bound to the gapped ones. + +Pass `expected_lock=` to reacquire a release and *require* its pinned +bytes: anything that changed under the same release is reported as +drift, and nothing is stored or replaced. + +!!! warning "The download ignores proxy settings" + `fetch` uses the standard-library HTTPS client, which does not + honour `HTTPS_PROXY` and friends. Behind a proxy, download the + exports with your own tools and record them with + `OgrdbSource.from_files(...)`. + +Three ways to get a source, depending on what you have: + +| Call | Use when | +|---|---| +| `OgrdbSource.fetch(...)` | You are online and pinning a release for the first time | +| `OgrdbSource.from_files(...)` | You already hold the release's exports locally | +| `OgrdbSource.read(lock, cache=...)` | You have a lock and a populated cache - fully offline | + +Then `inspect()` to see what the release holds, and `build(recipe)` to +project it into a `ReferenceBuild`: + +```python +source = OgrdbSource.read("igh.lock.json", cache="blobs/") +source.inspect() # what is in the release +build = source.build(recipe) # -> ReferenceBuild +``` + +`read` verifies every blob the lock needs against `cache` and never +touches the network; a missing or corrupt blob is an error naming the +asset and its pinned digest. + +The recipe is explicit by design: which alleles are selected, which +decisions were taken, which defaults profile applies. Nothing about a +build is implied by the time you ran it. + +## Errors + +Everything on this surface raises `ReferenceSourceError` with two +fields worth branching on - a stable `kind` and a `path` locating the +problem: + +```python +from GenAIRR.reference_sources import ReferenceSourceError + +try: + build = ReferenceBuild.read_dossier(path) +except ReferenceSourceError as exc: + print(exc.kind) # 'digest_mismatch', 'missing_asset', ... + print(exc.path) # '/reference.genairr-ref.json' +``` + +`kind` is part of the contract and safe to branch on; the message is +for people. diff --git a/site_docs/guides/recombination-junction.md b/site_docs/guides/recombination-junction.md index f0d1a60..acc693d 100644 --- a/site_docs/guides/recombination-junction.md +++ b/site_docs/guides/recombination-junction.md @@ -199,9 +199,14 @@ which equal the standard generating calls. ### Controlling the allele universe The cartridge defines which alleles exist - bundled cartridges -ship 100+ V alleles per locus, and the recombination pass -samples uniformly across them by default. Two surfaces narrow -the universe: +ship 100+ V alleles per locus. Three surfaces decide which of +them a run actually draws, and they do not compose: + +| Surface | Scope | Effect | +|---|---|---| +| `.restrict_alleles(v=[...])` | per experiment | Hard subset. Only the listed alleles can be drawn, uniformly among them. | +| `.recombine(v_allele_weights={...})` | per experiment | Relative sampling weights over the **whole** catalogue. | +| `cfg.reference_models.allele_usage` | per cartridge | An empirical usage plane carried by the reference itself. | ```python # Per-experiment subset: @@ -220,6 +225,62 @@ allele-usage weights are an empirical bias from the cartridge. See [Reference cartridge](../concepts/reference-cartridge.md) for the full surface. +!!! warning "`v_allele_weights` is relative to every allele, not just the ones you list" + An allele you do not mention keeps its **default weight of 1.0** - + it is not excluded and it is not set to zero. So handing the + shares straight from a usage table does close to the opposite of + what it reads like. Measured on `"human_igh"` (198 V alleles) over + 2,000 records: + + ```python + weights = {"IGHVF6-G23*01": 0.30, "IGHVF3-G10*01": 0.20} + ga.Experiment.on("human_igh").recombine(v_allele_weights=weights) + # -> 6 of 2000 records (0.300%) land on those two alleles, not 50%. + ``` + + At weight 0.2-0.3 against 196 alleles at 1.0, the two you asked + for become *rarer* than the ones you never mentioned. And `0.0` + raises `ValueError` ("must be a finite positive number"), so + "exclude this allele" has no spelling on this surface - use + `.restrict_alleles(...)` for exclusion. + +To reproduce a usage distribution, give weights **full support**: one +entry per allele in the catalogue, with a negligible weight for the +tail you are not modelling. `.restrict_alleles(...)` cannot help here - +passing both raises `ValueError: pass either allowed_ids OR weights, +not both`. + +```python +import GenAIRR as ga +from GenAIRR.data import HUMAN_IGH_OGRDB + +target = {"IGHVF6-G23*01": 0.5, "IGHVF3-G10*01": 0.3, "IGHVF10-G38*04": 0.2} +catalogue = sorted( + a.name for group in HUMAN_IGH_OGRDB.v_alleles.values() for a in group +) +weights = {name: target.get(name, 1e-9) for name in catalogue} + +result = ( + ga.Experiment.on("human_igh") + .recombine(v_allele_weights=weights) + .run_records(n=2000, seed=1) +) +# observed: 0.501 / 0.299 / 0.200, and zero records outside the three. +``` + +The same applies to `d_allele_weights` and `j_allele_weights`. + +!!! note "The bundled cartridges do not ship an allele-usage plane" + `cartridge_manifest()["models"]["allele_usage"]["available"]` is + `False` on `HUMAN_IGH_OGRDB`, `HUMAN_IGK_OGRDB`, `HUMAN_IGL_OGRDB` + and `HUMAN_TCRB_IMGT`. A legacy `gene_use_dict` is present on the + cartridge but is not read by the recombination pass + (`legacy_fallback: False`). So the default V usage is **uniform + over the whole catalogue** - 198 alleles on human IGH. If your + benchmark is sensitive to gene usage, set it explicitly with the + recipe above, or fit a usage plane from your own data with + [Estimate models from data](estimate-cartridge-models.md). + ## Trims The recombination pass models exonuclease trimming at each diff --git a/site_docs/guides/streaming.md b/site_docs/guides/streaming.md deleted file mode 100644 index d851b8f..0000000 --- a/site_docs/guides/streaming.md +++ /dev/null @@ -1,5 +0,0 @@ -# Stream millions of sequences without OOM - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [The Experiment builder](https://mutejester.github.io/GenAIRR/guides/experiment-builder.html). diff --git a/site_docs/guides/tune-corruption.md b/site_docs/guides/tune-corruption.md deleted file mode 100644 index 056ea9a..0000000 --- a/site_docs/guides/tune-corruption.md +++ /dev/null @@ -1,5 +0,0 @@ -# Tune corruption rates to match your sequencer - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Corruption + sequencing artefacts](https://mutejester.github.io/GenAIRR/guides/corruption-sequencing.html). diff --git a/site_docs/guides/v-usage.md b/site_docs/guides/v-usage.md deleted file mode 100644 index abff3d1..0000000 --- a/site_docs/guides/v-usage.md +++ /dev/null @@ -1,5 +0,0 @@ -# Match an empirical V-gene usage distribution - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/guides/with-metadata.md b/site_docs/guides/with-metadata.md deleted file mode 100644 index e361d3b..0000000 --- a/site_docs/guides/with-metadata.md +++ /dev/null @@ -1,5 +0,0 @@ -# Attach sample metadata to records - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [The Experiment builder](https://mutejester.github.io/GenAIRR/guides/experiment-builder.html). diff --git a/site_docs/validation/audit-realism.md b/site_docs/validation/audit-realism.md deleted file mode 100644 index 3446cba..0000000 --- a/site_docs/validation/audit-realism.md +++ /dev/null @@ -1,5 +0,0 @@ -# Audit a simulation for biological realism - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Validation & Debugging](https://mutejester.github.io/GenAIRR/validation/index.html). diff --git a/site_docs/validation/benchmark-aligner.md b/site_docs/validation/benchmark-aligner.md deleted file mode 100644 index b7528cb..0000000 --- a/site_docs/validation/benchmark-aligner.md +++ /dev/null @@ -1,5 +0,0 @@ -# Benchmark an aligner against truth - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Validation & Debugging](https://mutejester.github.io/GenAIRR/validation/index.html). diff --git a/site_docs/validation/compare-shm.md b/site_docs/validation/compare-shm.md deleted file mode 100644 index d399045..0000000 --- a/site_docs/validation/compare-shm.md +++ /dev/null @@ -1,5 +0,0 @@ -# Compare two SHM models - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Targeted SHM rates](https://mutejester.github.io/GenAIRR/guides/shm-targeting.html). diff --git a/site_docs/validation/replay.md b/site_docs/validation/replay.md deleted file mode 100644 index 0280a56..0000000 --- a/site_docs/validation/replay.md +++ /dev/null @@ -1,5 +0,0 @@ -# Replay a simulation with one knob changed - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Trace, replay, reproducibility](https://mutejester.github.io/GenAIRR/guides/trace-replay.html). diff --git a/site_docs/validation/reproduce.md b/site_docs/validation/reproduce.md deleted file mode 100644 index c55aa5a..0000000 --- a/site_docs/validation/reproduce.md +++ /dev/null @@ -1,5 +0,0 @@ -# Reproduce a published dataset - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Trace, replay, reproducibility](https://mutejester.github.io/GenAIRR/guides/trace-replay.html). diff --git a/site_docs/validation/smoke-test.md b/site_docs/validation/smoke-test.md deleted file mode 100644 index 43f6065..0000000 --- a/site_docs/validation/smoke-test.md +++ /dev/null @@ -1,5 +0,0 @@ -# Smoke-test a config change - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Installation](https://mutejester.github.io/GenAIRR/getting-started/installation.html). diff --git a/site_docs/validation/trace-introspection.md b/site_docs/validation/trace-introspection.md deleted file mode 100644 index a1d16f0..0000000 --- a/site_docs/validation/trace-introspection.md +++ /dev/null @@ -1,5 +0,0 @@ -# Introspect the trace - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Trace, replay, reproducibility](https://mutejester.github.io/GenAIRR/guides/trace-replay.html). diff --git a/site_docs/validation/two-layer-model.md b/site_docs/validation/two-layer-model.md deleted file mode 100644 index 28da66f..0000000 --- a/site_docs/validation/two-layer-model.md +++ /dev/null @@ -1,8 +0,0 @@ -# Two-layer integrity model - - -_Rewritten from `docs/validation_matrix.md` §1.0._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Validation & Debugging](https://mutejester.github.io/GenAIRR/validation/index.html). diff --git a/tests/acceptance/test_docs_redirect_targets_are_real_pages.py b/tests/acceptance/test_docs_redirect_targets_are_real_pages.py new file mode 100644 index 0000000..5a6a4b5 --- /dev/null +++ b/tests/acceptance/test_docs_redirect_targets_are_real_pages.py @@ -0,0 +1,88 @@ +"""Every preserved URL must land on a page that says something. + +`mkdocs.yml`'s `redirect_maps` keeps the pre-MkDocs URLs resolvable. A +redirect is only worth having if its destination answers the question the +old URL answered: a reader who follows a bookmark or a citation into a +placeholder is worse off than one who gets a 404, because the 404 at least +tells them to search. + +This ran green while 22 of the 28 redirects pointed at pages whose entire +body was an admonition saying the page had not been written. It did not +catch that, because nothing checked it. These two tests do. + +They deliberately do not depend on the contributor-only migration plan the +sibling `test_docs_mkdocs_migration_contract` module parses, which is +absent from every clone and skips that whole module in CI. +""" + +from __future__ import annotations + +import re + +import pytest + +from tests.support.paths import repo_root + +_MKDOCS_YML = repo_root() / "mkdocs.yml" +_SITE_DOCS = repo_root() / "site_docs" + +# Phrases a placeholder page uses to admit it is one. Matched +# case-insensitively against the whole page body. +_PLACEHOLDER_MARKERS = ( + "has not been written yet", + "this page is a placeholder", + "migration placeholder", + "content lands in phase", +) + +pytestmark = pytest.mark.pin + + +def _redirect_map() -> dict[str, str]: + """`{fake source .md path: destination .md path}` from `redirect_maps`.""" + text = _MKDOCS_YML.read_text(encoding="utf-8") + pairs = re.findall(r'^\s*"([^"]+\.md)":\s*"([^"]+\.md)"\s*$', text, re.M) + assert pairs, "no redirect_maps entries found in mkdocs.yml" + return dict(pairs) + + +def test_pin_present_every_redirect_target_is_a_real_file() -> None: + """A redirect whose destination does not exist fails the build late. + + `mkdocs-redirects` emits the redirect file regardless, so the breakage + shows up as a 404 for a reader rather than as an error for us. + """ + missing = sorted( + f"{source} -> {target}" + for source, target in _redirect_map().items() + if not (_SITE_DOCS / target).is_file() + ) + assert not missing, ( + "redirect destinations that do not exist under site_docs/:\n " + + "\n ".join(missing) + ) + + +def test_pin_present_no_redirect_lands_on_a_placeholder() -> None: + """The preserved URL must answer the question the old URL answered. + + If a page has not been written, point its redirect at the nearest page + that has been, and delete the placeholder. Keeping the placeholder as a + destination spends the URL on an apology. + """ + landing_on_placeholder = [] + for source, target in sorted(_redirect_map().items()): + page = _SITE_DOCS / target + if not page.is_file(): + continue # the test above owns this failure + body = page.read_text(encoding="utf-8").lower() + for marker in _PLACEHOLDER_MARKERS: + if marker in body: + landing_on_placeholder.append(f"{source} -> {target} ({marker!r})") + break + assert not landing_on_placeholder, ( + "redirects pointing at placeholder pages:\n " + + "\n ".join(landing_on_placeholder) + + "\n\nRepoint each at the nearest page that has content, and delete " + "the placeholder." + ) diff --git a/tools/inventory-declarations.json b/tools/inventory-declarations.json index 5126d74..38e046b 100644 --- a/tools/inventory-declarations.json +++ b/tools/inventory-declarations.json @@ -1,8 +1,5 @@ { - "revised": { - "MANIFEST_PIN": "the packaged reference dossier was regenerated for the 2.4.0 release, so its manifest hash moved from 9ab6ba63 to 101abcebe. The manifest records the GenAIRR version that generated the dossier, which is why a version bump alone retargets this pin; tools/build_published_reference.py prints the new value and names every pin site, and test_the_packaged_dossier_was_generated_by_the_version_this_checkout_declares fails if the two disagree. The same value is pinned once more in src/GenAIRR/_reference_sources/bundled.py's REGISTRY and was updated there in the same change. Nothing about the reference build itself changed: the runtime execution, catalogue and models hashes and the source lock all came back byte-identical, which is the evidence that only the recorded builder version moved.", - "ARCHIVE_PIN": "the raw sha256 of the regenerated archive, 52df85b3 to d8d82abc. The archive holds the manifest, so it changes with the version for the same reason MANIFEST_PIN does -- the comment above this constant already said so before this change. The archive is byte-identical in length (20110025 bytes both times), which is consistent with a change confined to the recorded version string rather than to any packaged member." - }, + "revised": {}, "replaced": {}, "allow": {} }