From e3a5658c6cdc689ab96b3dd238e11fee20e123c3 Mon Sep 17 00:00:00 2001 From: thomas Date: Fri, 6 Nov 2026 09:50:02 +0200 Subject: [PATCH 1/7] docs: the blinded handoff now blinds The export page claimed `to_fasta` "emits only `sequence_id` + the assembled sequence, so it's implicitly blinded". It does not. Both `to_fasta` and `to_fastq` annotate every header with the generating V and J calls: >seq0|v_call=IGHVF6-G23*03|j_call=IGHJ1*01 So a reader who followed this page handed the answer to the tool they were scoring, inside the input of their own benchmark. The page already documented that header correctly fifteen lines earlier, under FASTA, which made the claim a self-contradiction rather than a stale fact -- and the wrong half was the one a benchmarking reader acts on, because FASTA is the format an aligner wants. Replaced the claim with a table of what each exporter carries, and a blinded-FASTA writer that emits `sequence_id` and bases only. The `.upper()` in it is deliberate: the corruption passes lowercase the bases they touch and some aligners read case as a soft-mask. Two things a scorer has to settle are now stated where they are needed rather than left to be discovered: that the default human IGH cartridge reports ASC cluster names while an IMGT-referenced aligner reports IMGT names, with both escape hatches; and that `v_call` and `recalled_v_call` answer different questions, diverging exactly where mutation has made the generating allele unrecoverable. The illustrative header under FASTA was also stale -- it showed space-separated calls with no `key=` prefixes. It now shows what the exporter writes for this page's own example parameters. --- site_docs/getting-started/export-results.md | 112 +++++++++++++++++--- 1 file changed, 100 insertions(+), 12 deletions(-) diff --git a/site_docs/getting-started/export-results.md b/site_docs/getting-started/export-results.md index 95469b46..4304399b 100644 --- a/site_docs/getting-started/export-results.md +++ b/site_docs/getting-started/export-results.md @@ -57,7 +57,7 @@ Writes one record per sequence with the `sequence_id` plus `v_call` and `j_call` in the header: ```text ->seq0 IGHVF10-G38*04 IGHJ2*01 +>seq0|v_call=IGHVF6-G23*03|j_call=IGHJ1*01 gaggtgcagctggtg... ``` @@ -65,14 +65,42 @@ Pass `prefix="myseq"` to change the per-record ID prefix. ### Blinded handoff - strip truth columns -FASTA is the natural format when the next stage is an aligner -that should NOT see the ground-truth columns - e.g. an aligner -benchmark or a tool comparison where leaking `v_call` / -`truth_v_call` would defeat the point. `to_fasta` already -emits only `sequence_id` + the assembled sequence, so it's -implicitly blinded. +When the next stage is a tool you are scoring, it must not see the +generating calls. The exporters differ in what they carry: -For a blinded TSV, two options: +| Exporter | Carries truth? | What ends up in the header / row | +|---|---|---| +| `to_paired_fastq(r1, r2)` | No | `@seq0/1` and `@seq0/2` - `sequence_id` only | +| `to_fasta(path)` | **Yes** | `>seq0|v_call=...|j_call=...` | +| `to_fastq(path)` | **Yes** | `@seq0|v_call=...|j_call=...` | +| `to_tsv` / `to_csv` / `to_dataframe` | **Yes** | `v_call`, `d_call`, `j_call`, every counter, plus `truth_*` under `expose_provenance=True` | + +`to_fasta` and `to_fastq` annotate their headers on purpose, which is +convenient when you are the consumer and wrong when the consumer is a +tool under test. For a blinded panel, write the records yourself - +`sequence_id` is all the downstream join needs: + +```python +import GenAIRR as ga + +result = ( + ga.Experiment.on("human_igh") + .recombine() + .productive_only() + .run_records(n=10_000, seed=42) +) + +# Blinded FASTA - sequence_id and bases, nothing else. +with open("panel_blinded.fasta", "w") as fh: + for rec in result: + fh.write(f">{rec['sequence_id']}\n{rec['sequence'].upper()}\n") +``` + +`.upper()` is deliberate: the corruption passes lowercase the bases +they touch, and some aligners read case as a soft-mask. Normalise on +the way out and keep the original `sequence` in your truth table. + +For the truth side, two options: ```python # Option A - write a regular TSV, then drop truth columns @@ -95,10 +123,70 @@ result = ( result.to_tsv("panel_blinded.tsv") ``` -Option B is the default - `truth_*` columns are only emitted -when you pass `expose_provenance=True` to `run_records(...)`. -Option A is the recipe for *post hoc* blinding when the same -batch needs to be shared in both forms. +Option B is the default - `truth_*` columns are only emitted when you +pass `expose_provenance=True` to `run_records(...)`. Option A is the +recipe for *post hoc* blinding when the same run has to produce both +the panel and the answer key. + +Either way, note what Option B does not do: it blinds the `truth_*` +aliases, not the record. `v_call`, `d_call` and `j_call` are always the +generating alleles. A panel for a tool you are scoring should be FASTA +or FASTQ written as above, not a TSV with columns removed one at a +time. + +### Keeping the truth table beside the panel + +The usual shape is two files joined on `sequence_id`: + +```python +result = ( + ga.Experiment.on("human_igh") + .recombine() + .productive_only() + .mutate(model="s5f", rate=0.03) + .run_records(n=10_000, seed=42, expose_provenance=True) +) + +# 1. the blinded input the tool sees +with open("panel.fasta", "w") as fh: + for rec in result: + fh.write(f">{rec['sequence_id']}\n{rec['sequence'].upper()}\n") + +# 2. the answer key you keep +result.to_tsv("panel_truth.tsv") +``` + +Run the tool on `panel.fasta`, then join its output back on +`sequence_id` - see +[How `sequence_id` joins everything](#how-sequence_id-joins-everything). +Two things to settle before you score: + +- **Allele naming.** On `"human_igh"` the truth calls are ASC cluster + names (`IGHVF6-G23*03`), while an aligner against an IMGT reference + emits IMGT names (`IGHV4-31*10`). Translate one side using + `cfg.asc_tables["V"]`, or simulate on an IMGT-named cartridge + instead: + + ```python + from GenAIRR.data import HUMAN_IGH_IMGT + + result = ( + ga.Experiment.on(HUMAN_IGH_IMGT) + .curate_refdata("functional_anchors_only") + .recombine() + .run_records(n=10_000, seed=42) + ) + ``` + +- **Which truth column.** `v_call` is the allele the engine + *generated*. `recalled_v_call` is what GenAIRR's own walker infers + from the mutated sequence, and it may be a comma-separated tie. + Score against `v_call` to ask "did the tool recover the generating + allele"; against `recalled_v_call` to ask "did it agree with an + independent caller on the same evidence". The two diverge exactly + where mutation has made the generating allele unrecoverable, which + is the interesting part of the benchmark. See + [Truth and evidence-based re-calls](first-airr-record.md#truth-and-evidence-based-re-calls). ## FASTQ From bae0a4b1273669cd03666dd5e3def56a78854500 Mon Sep 17 00:00:00 2001 From: thomas Date: Fri, 6 Nov 2026 09:52:17 +0200 Subject: [PATCH 2/7] docs: allele usage is a documented surface with a documented trap `index.md` advertises controllable V-gene usage and no page of the site mentioned the knob that controls it. `v_allele_weights`, `d_allele_weights` and `j_allele_weights` appeared nowhere outside one paragraph of the README about the cartridge estimator, and `biology-map.md` -- which claims to map every biological mechanism -- had no row for allele usage at all. The knob also reads like the opposite of what it does. An allele you do not list keeps its default weight of 1.0, so passing the shares from a real usage table makes the alleles you asked for rarer than the ones you never mentioned. Measured on human IGH over 2,000 records, asking for 0.30 and 0.20 puts 6 records -- 0.300% -- on those two alleles instead of half of them. `0.0` is rejected as not finite-positive, so exclusion has no spelling here either, and `restrict_alleles` cannot be combined with weights: that raises `ValueError`. So the section now gives the three surfaces, states that they do not compose, and carries a full-support recipe that reproduces a target distribution exactly -- 0.501/0.299/0.200 against 0.5/0.3/0.2, with no records outside the three. While measuring it, the default turned out to be worth stating too. `allele_usage.available` is False on all four human cartridges the concepts page named as carrying data-fitted distributions, and the legacy `gene_use_dict` they do carry is not read by the recombination pass. Default V usage is therefore uniform over 198 alleles on human IGH, which is the wrong prior for any usage-sensitive benchmark and the opposite of what that page promised. Corrected there, and cross-linked. The same page also pointed at three `docs/*.md` files that have not existed under that name since the audit documents were renamed, so three links on the published site were dead. --- site_docs/concepts/reference-cartridge.md | 17 ++++-- site_docs/guides/biology-map.md | 1 + site_docs/guides/recombination-junction.md | 67 +++++++++++++++++++++- 3 files changed, 76 insertions(+), 9 deletions(-) diff --git a/site_docs/concepts/reference-cartridge.md b/site_docs/concepts/reference-cartridge.md index 31305a8b..ee1c536d 100644 --- a/site_docs/concepts/reference-cartridge.md +++ b/site_docs/concepts/reference-cartridge.md @@ -175,8 +175,13 @@ ship typed defaults that users override per-experiment. !!! warning "Which bundled cartridges carry *real* data-derived distributions" Only the human **IGH**, **IGK**, **IGL**, and **TCRB** cartridges ship with empirical distributions fitted from **real repertoire - data** — trim lengths, NP-region lengths, the NP base / Markov - model, and allele usage. + data** — trim lengths, NP-region lengths, and the NP base / Markov + model. + + Allele usage is **not** among them. No bundled cartridge ships an + `allele_usage` plane, so allele draws are uniform over the + catalogue until you weight them yourself — see + [Controlling the allele universe](../guides/recombination-junction.md#controlling-the-allele-universe). Every **other** bundled species/locus cartridge carries a real germline **allele catalogue** (from IMGT), but its data-derived @@ -417,10 +422,10 @@ legacy fields are sitting on the cartridge unused. For the original design audit and the engine-side `RefDataConfig` bridge, see -[`docs/reference_cartridge.md`](https://github.com/MuteJester/GenAIRR/blob/master/docs/reference_cartridge.md) +[`audit-docs/reference_cartridge.md`](https://github.com/MuteJester/GenAIRR/blob/master/audit-docs/reference_cartridge.md) and -[`docs/reference_cartridge_authoring_audit.md`](https://github.com/MuteJester/GenAIRR/blob/master/docs/reference_cartridge_authoring_audit.md). +[`audit-docs/reference_cartridge_authoring_audit.md`](https://github.com/MuteJester/GenAIRR/blob/master/audit-docs/reference_cartridge_authoring_audit.md). For per-estimator audits (allele usage, trims, NP length, NP base, -P-nucleotide length), see the matching design docs in `docs/`. For +P-nucleotide length), see the matching design docs in `audit-docs/`. For the full engine-wide validation matrix, see -[`docs/validation_matrix.md`](https://github.com/MuteJester/GenAIRR/blob/master/docs/validation_matrix.md). +[`audit-docs/validation_matrix.md`](https://github.com/MuteJester/GenAIRR/blob/master/audit-docs/validation_matrix.md). diff --git a/site_docs/guides/biology-map.md b/site_docs/guides/biology-map.md index f0d34c3b..8939582e 100644 --- a/site_docs/guides/biology-map.md +++ b/site_docs/guides/biology-map.md @@ -27,6 +27,7 @@ better starting point. | **D inversion** | `.invert_d(prob=...)` | Recombination editing | `d_inverted` | [D inversion + receptor revision](recombination-editing.md) | | **Receptor revision** | `.receptor_revision(prob=...)` | Recombination editing | `receptor_revision_applied`, `original_v_call`, post-revision `v_call` | [D inversion + receptor revision](recombination-editing.md) | | **Productivity constraint** | `.productive_only()` | Constraint-aware sampling | `productive=True` (by construction) | [Experiment builder](experiment-builder.md#how-productive_only-works) | +| **Allele usage** | `.recombine(v_allele_weights=..., d_allele_weights=..., j_allele_weights=...)` (experiment) + `cfg.reference_models.allele_usage` (cartridge) | Recombination | Which alleles `v_call`/`d_call`/`j_call` draw, and how often. Default is uniform over the catalogue: no bundled cartridge ships a usage plane | [Recombination + junction biology](recombination-junction.md#controlling-the-allele-universe) | | **Allele restriction** | `.restrict_alleles(v=..., d=..., j=...)` | Constraint-aware sampling | Sampled `v_call`/`d_call`/`j_call` restricted to the listed alleles | [Recombination + junction biology](recombination-junction.md) | | **Somatic hypermutation (SHM)** | `.mutate(model="s5f", rate=...)` or `.mutate(model="uniform", count=...)` | Biology - descendant phase | `n_mutations`, `mutation_rate`, `n_v_mutations`, `n_d_mutations`, `n_j_mutations`, `n_np_mutations` | [SHM and mutation targeting](shm-targeting.md) | | **Targeted SHM** | `.mutate(..., segment_rates=..., v_subregion_rates=...)` | Biology - descendant phase | Same plus the six V-subregion counters (`n_fwr1_mutations` … `n_v_unannotated_mutations`) | [SHM and mutation targeting](shm-targeting.md) | diff --git a/site_docs/guides/recombination-junction.md b/site_docs/guides/recombination-junction.md index f0d1a606..acc693d2 100644 --- a/site_docs/guides/recombination-junction.md +++ b/site_docs/guides/recombination-junction.md @@ -199,9 +199,14 @@ which equal the standard generating calls. ### Controlling the allele universe The cartridge defines which alleles exist - bundled cartridges -ship 100+ V alleles per locus, and the recombination pass -samples uniformly across them by default. Two surfaces narrow -the universe: +ship 100+ V alleles per locus. Three surfaces decide which of +them a run actually draws, and they do not compose: + +| Surface | Scope | Effect | +|---|---|---| +| `.restrict_alleles(v=[...])` | per experiment | Hard subset. Only the listed alleles can be drawn, uniformly among them. | +| `.recombine(v_allele_weights={...})` | per experiment | Relative sampling weights over the **whole** catalogue. | +| `cfg.reference_models.allele_usage` | per cartridge | An empirical usage plane carried by the reference itself. | ```python # Per-experiment subset: @@ -220,6 +225,62 @@ allele-usage weights are an empirical bias from the cartridge. See [Reference cartridge](../concepts/reference-cartridge.md) for the full surface. +!!! warning "`v_allele_weights` is relative to every allele, not just the ones you list" + An allele you do not mention keeps its **default weight of 1.0** - + it is not excluded and it is not set to zero. So handing the + shares straight from a usage table does close to the opposite of + what it reads like. Measured on `"human_igh"` (198 V alleles) over + 2,000 records: + + ```python + weights = {"IGHVF6-G23*01": 0.30, "IGHVF3-G10*01": 0.20} + ga.Experiment.on("human_igh").recombine(v_allele_weights=weights) + # -> 6 of 2000 records (0.300%) land on those two alleles, not 50%. + ``` + + At weight 0.2-0.3 against 196 alleles at 1.0, the two you asked + for become *rarer* than the ones you never mentioned. And `0.0` + raises `ValueError` ("must be a finite positive number"), so + "exclude this allele" has no spelling on this surface - use + `.restrict_alleles(...)` for exclusion. + +To reproduce a usage distribution, give weights **full support**: one +entry per allele in the catalogue, with a negligible weight for the +tail you are not modelling. `.restrict_alleles(...)` cannot help here - +passing both raises `ValueError: pass either allowed_ids OR weights, +not both`. + +```python +import GenAIRR as ga +from GenAIRR.data import HUMAN_IGH_OGRDB + +target = {"IGHVF6-G23*01": 0.5, "IGHVF3-G10*01": 0.3, "IGHVF10-G38*04": 0.2} +catalogue = sorted( + a.name for group in HUMAN_IGH_OGRDB.v_alleles.values() for a in group +) +weights = {name: target.get(name, 1e-9) for name in catalogue} + +result = ( + ga.Experiment.on("human_igh") + .recombine(v_allele_weights=weights) + .run_records(n=2000, seed=1) +) +# observed: 0.501 / 0.299 / 0.200, and zero records outside the three. +``` + +The same applies to `d_allele_weights` and `j_allele_weights`. + +!!! note "The bundled cartridges do not ship an allele-usage plane" + `cartridge_manifest()["models"]["allele_usage"]["available"]` is + `False` on `HUMAN_IGH_OGRDB`, `HUMAN_IGK_OGRDB`, `HUMAN_IGL_OGRDB` + and `HUMAN_TCRB_IMGT`. A legacy `gene_use_dict` is present on the + cartridge but is not read by the recombination pass + (`legacy_fallback: False`). So the default V usage is **uniform + over the whole catalogue** - 198 alleles on human IGH. If your + benchmark is sensitive to gene usage, set it explicitly with the + recipe above, or fit a usage plane from your own data with + [Estimate models from data](estimate-cartridge-models.md). + ## Trims The recombination pass models exonuclease trimming at each From 37a7848ff7cb54035d729dd67abe427b368efd8d Mon Sep 17 00:00:00 2001 From: thomas Date: Fri, 6 Nov 2026 09:55:40 +0200 Subject: [PATCH 3/7] docs: published germline sources, which shipped undocumented `GenAIRR.reference_sources` is a public module with its own `__all__`, added after the 2.3.1 tag, and it had zero mentions across the docs site, the changelog, the migration guide and the README. The release commit for 2.4.0 is itself about the packaged dossier. The only shipped build id was discoverable solely by asking for one that does not exist and reading the error. So the capability it exists for -- pinning a named published germline release and handing a reviewer a dossier they can verify with no network and no trust in its author -- was unreachable for anyone who had not read the source. New guide, with every claim executed against the shipped build: `bundled()` runs under `Experiment.on(...)` and reports IMGT names, which also makes it the shortest answer to the ASC-naming problem; `write_dossier` takes a directory, not an archive, and emits the nine named members plus their content-addressed blobs; `read_dossier` regenerates every derived file from the locked bytes and the recipe rather than re-reading stored hashes, so one altered byte comes back as `kind='digest_mismatch'` with the member's path. The `fetch` and `read` examples are written against the real signatures rather than the obvious guesses: `release` is an int and there is no "latest", `profile` and `cache` are required keyword arguments, and the standard-library HTTPS client ignores proxy settings, which is why `from_files` exists. Also added the changelog entry this should have had, and a section to the migration guide saying what is new and worth adopting -- it was entirely about breakage, so a reader following it to the end learned nothing about what they had just gained. --- CHANGELOG.md | 22 +++ MIGRATING.md | 22 +++ mkdocs.yml | 1 + site_docs/guides/published-references.md | 199 +++++++++++++++++++++++ 4 files changed, 244 insertions(+) create mode 100644 site_docs/guides/published-references.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 470e2f41..f5c45e79 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -148,6 +148,28 @@ have. ### Added +- **Published germline sources, and dossiers anyone can verify offline.** The + new public module `GenAIRR.reference_sources` locks one *named* published + germline release and keeps the evidence. `OgrdbSource` acquires an explicit + OGRDB germline-set release -- an integer release, never "latest" -- and pins + its exact bytes in a lock; `fetch` is the only operation in GenAIRR that uses + the network, and `from_files` and `read` cover the offline and + behind-a-proxy cases. `source.build(recipe)` projects an explicit recipe into + a `ReferenceBuild`: a runtime `Reference` you pass straight to + `Experiment.on(...)`, plus the lock, the recipe, the build report and a + manifest recording the GenAIRR version that produced it. + `build.write_dossier(directory)` writes the whole thing as a portable + dossier, and `ReferenceBuild.read_dossier(directory)` checks one with no + network and no trust in its author: it regenerates every derived file from + the locked bytes and the recipe and compares, so a single tampered byte comes + back as `ReferenceSourceError(kind='digest_mismatch', path=...)` naming the + member. Failures on this surface all carry a stable `kind` and a `path`. + One build ships inside the wheel -- + `ReferenceBuild.bundled("human_igh_ogrdb_10_observed_v1")` -- so the workflow + is available with no network at all; it also reports IMGT allele names, where + `"human_igh"` reports ASC cluster names. See + [Published germline sources](https://mutejester.github.io/GenAIRR/guides/published-references.html). + - **Two clonal models beside the deprecated star.** `Experiment.clonal_lineage(...)` grows a real BCR genealogy per clone: a generation-synchronous birth-death process with per-division S5F diff --git a/MIGRATING.md b/MIGRATING.md index ec78bd14..3dece6af 100644 --- a/MIGRATING.md +++ b/MIGRATING.md @@ -190,6 +190,28 @@ Two further counter changes to know about: A genotype run with novel alleles must be validated against `compiled.refdata`. +## 7. What you may now want to use + +Everything above is about breakage. These are the 2.4.0 additions most +likely to change how you work, in case the migration is the only part of +the release you read: + +- **Pin a published germline release, and ship the evidence.** + `GenAIRR.reference_sources` locks one named OGRDB release and writes a + dossier a reviewer can verify offline, with no network and no trust in + you. `ReferenceBuild.bundled("human_igh_ogrdb_10_observed_v1")` needs + no network at all. See + [Published germline sources](https://mutejester.github.io/GenAIRR/guides/published-references.html). +- **Real clonal genealogies.** `clonal_lineage()` replaces the star + topology with a birth-death process and hands back the ground-truth + tree per clone; `clonal_repertoire()` covers TCR and flat-BCR + abundance. Both are the intended replacements for `expand_clones()` + (§3). +- **Per-individual diploid genotypes**, with population priors, cohorts + and a genotype truth sidecar for scoring inference. +- The full list is the `### Added` section of + [CHANGELOG.md](CHANGELOG.md). + --- # Migrating from GenAIRR 1.x to 2.0 diff --git a/mkdocs.yml b/mkdocs.yml index 3b5ebdbf..93c77f9c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -214,6 +214,7 @@ nav: - Corruption + sequencing artefacts: guides/corruption-sequencing.md - Paired-end reads and FASTQ: guides/paired-end-fastq.md - Portable local references: guides/portable-references.md + - Published germline sources: guides/published-references.md - Reference Cartridge Authoring: - cartridges/index.md - Reference evidence and quarantine: cartridges/reference-evidence.md diff --git a/site_docs/guides/published-references.md b/site_docs/guides/published-references.md new file mode 100644 index 00000000..fdf44c8d --- /dev/null +++ b/site_docs/guides/published-references.md @@ -0,0 +1,199 @@ +# Published germline sources and dossiers + +

When a result has to be defensible a year from now, "we +used the current IMGT release" is not a reference. This page locks one +named published germline release, builds a runtime reference +from it, and writes a dossier that anyone can verify offline without +trusting you, the network, or the registry.

+ +Three names do all of it, from `GenAIRR.reference_sources`: + +| Name | What it is for | +|---|---| +| `OgrdbSource` | Acquire and pin one explicit OGRDB germline-set release | +| `ReferenceBuild` | A runtime `Reference` plus the complete record of how it was made | +| `ReferenceSourceError` | Every failure here, with a stable `kind` and a `path` | + +This is a different thing from +[Portable local references](portable-references.md). There you build a +reference from *your own* FASTA files. Here you pin a *published* +release and keep the evidence that you did. + +## Start from the shipped build + +One build ships inside the wheel, so the whole workflow is available +with no network at all: + +```python +import GenAIRR as ga +from GenAIRR.reference_sources import ReferenceBuild + +build = ReferenceBuild.bundled("human_igh_ogrdb_10_observed_v1") + +result = ( + ga.Experiment.on(build.reference) + .recombine() + .run_records(n=1000, seed=0) +) +result.records[0]["v_call"] # 'IGHV4-30-2*06' - IMGT names +``` + +Ask for a name that does not exist and the error lists the ones that +do, so the catalogue is always one call away: + +```python +ReferenceBuild.bundled("nope") +# ReferenceSourceError(kind='missing_asset'): +# no bundled reference build is named exactly +# 'human_igh_ogrdb_10_observed_v1' +``` + +!!! note "This is also the simplest route to IMGT allele names" + The bundled build reports IMGT names (`IGHV4-30-2*06`), where + `"human_igh"` reports ASC cluster names (`IGHVF6-G23*03`). If you + are scoring a tool against an IMGT reference, that matters - see + [Blinded handoff](../getting-started/export-results.md#blinded-handoff-strip-truth-columns). + +## What a build carries + +A `ReferenceBuild` is the reference *and* its provenance. Four +read-only parts: + +| Member | Holds | +|---|---| +| `build.reference` | The runtime `Reference` you pass to `Experiment.on(...)` | +| `build.lock` | The pinned release: provider, `set_id`, every asset and its digest | +| `build.recipe` | The explicit projection - selection, decisions, defaults profile | +| `build.report` | What came out: counts, capabilities, claims, comparison, inventory | +| `build.manifest` | Member versions, hashes, and the generator that wrote it | + +The manifest records the GenAIRR version that produced the build: + +```python +build.manifest.builder +# Generator(name='GenAIRR', version='2.4.0') +``` + +That is deliberate. A reference built by a different version is a +different artifact even when the germline bytes are identical, because +the projection rules may have moved. + +## Write a dossier, verify it offline + +`write_dossier` takes a **directory**, not an archive: + +```python +build.write_dossier("igh_dossier/") +``` + +It writes the named members plus the content-addressed blobs they +refer to: + +```text +source.lock.json recipe.json report.json +reference.genairr-ref.json manifest.json annotations.json +v.fasta d.fasta j.fasta + ... +``` + +Anyone can check it with no network and no trust in you: + +```python +from GenAIRR.reference_sources import ReferenceBuild + +back = ReferenceBuild.read_dossier("igh_dossier/") +back.manifest.hashes == build.manifest.hashes # True +``` + +`read_dossier` does not merely read the stored hashes. It regenerates +every derived file from the locked bytes and the recipe and compares +the result, so a tampered member is caught and located: + +```python +# after changing one byte of reference.genairr-ref.json +ReferenceBuild.read_dossier("igh_dossier/") +# ReferenceSourceError(kind='digest_mismatch', +# path='/reference.genairr-ref.json') +``` + +This is the property worth having in a methods section: the dossier is +checkable by a reviewer who has only the directory. + +## Pin a release yourself + +`OgrdbSource.fetch` is the **only** operation in GenAIRR that uses the +network. It takes one explicit release - never "latest" - and writes a +lock that pins the exact bytes: + +```python +from GenAIRR.reference_sources import OgrdbSource + +source = OgrdbSource.fetch( + set_id="Homo sapiens IGH", + release=10, # an int, never "latest" + profile="ogrdb_api_v2_human_ig_observed_v1", + cache="blobs/", # content-addressed blob directory +) +source.write_lock("igh.lock.json") +``` + +`release` is an explicit positive integer; there is no "latest". A +`profile` names the interpretation rules applied to the release, and +`cache` is the content-addressed directory the downloaded bytes land +in. By default all four exports are acquired (`airr`, `airr_ex`, +`gapped`, `gapped_ex`) - a build compares both record exports, and V +junction anchors are accepted only when bound to the gapped ones. + +Pass `expected_lock=` to reacquire a release and *require* its pinned +bytes: anything that changed under the same release is reported as +drift, and nothing is stored or replaced. + +!!! warning "The download ignores proxy settings" + `fetch` uses the standard-library HTTPS client, which does not + honour `HTTPS_PROXY` and friends. Behind a proxy, download the + exports with your own tools and record them with + `OgrdbSource.from_files(...)`. + +Three ways to get a source, depending on what you have: + +| Call | Use when | +|---|---| +| `OgrdbSource.fetch(...)` | You are online and pinning a release for the first time | +| `OgrdbSource.from_files(...)` | You already hold the release's exports locally | +| `OgrdbSource.read(lock, cache=...)` | You have a lock and a populated cache - fully offline | + +Then `inspect()` to see what the release holds, and `build(recipe)` to +project it into a `ReferenceBuild`: + +```python +source = OgrdbSource.read("igh.lock.json", cache="blobs/") +source.inspect() # what is in the release +build = source.build(recipe) # -> ReferenceBuild +``` + +`read` verifies every blob the lock needs against `cache` and never +touches the network; a missing or corrupt blob is an error naming the +asset and its pinned digest. + +The recipe is explicit by design: which alleles are selected, which +decisions were taken, which defaults profile applies. Nothing about a +build is implied by the time you ran it. + +## Errors + +Everything on this surface raises `ReferenceSourceError` with two +fields worth branching on - a stable `kind` and a `path` locating the +problem: + +```python +from GenAIRR.reference_sources import ReferenceSourceError + +try: + build = ReferenceBuild.read_dossier(path) +except ReferenceSourceError as exc: + print(exc.kind) # 'digest_mismatch', 'missing_asset', ... + print(exc.path) # '/reference.genairr-ref.json' +``` + +`kind` is part of the contract and safe to branch on; the message is +for people. From 170526f5ac28a79ea402a1456497c68e383bf743 Mon Sep 17 00:00:00 2001 From: thomas Date: Fri, 6 Nov 2026 09:58:50 +0200 Subject: [PATCH 4/7] docs: no preserved URL lands on an apology any more 22 of the 28 `redirect_maps` entries pointed at pages whose entire body was an admonition saying the page had not been written. The live site is worse still: it serves an older placeholder text that names `docs/docs_mkdocs_migration_plan.md`, a contributor-only document no reader can open, and then tells them to go look at the old site -- which is the page they just came from, because the old URL now redirects here. So a bookmark or a citation to, say, `/guide-benchmark-aligner.html` used to serve a 1,059-word recipe and now serves 33 words of apology. That is worse than a 404, which at least tells a reader to search. Each of the 22 now points at the nearest page that has content: the lessons and the four concept placeholders at their section indexes, the three trace/replay guides at `guides/trace-replay.md`, the four remaining validation recipes at `validation/index.md`, and the rest at the specific page that absorbed their subject -- `guide-v-usage` at the allele-universe section written earlier in this branch. All 37 placeholder pages are deleted, which also takes them out of the search index, where five of them were the only title match for their own obvious query. Two new pins enforce it, because nothing did: every redirect target must exist, and no redirect target may carry a placeholder marker. Verified by reintroducing a marker and watching the second one name all four affected redirects. They deliberately do not read the contributor-only migration plan that the sibling contract module parses -- that file is absent from every clone, which skips all 26 of its pins in CI, so the one existing check on this contract has never run where it matters. What this does not do is bring the content back. 16,251 words of the original prose are still in `website/`, one written page behind each of these URLs. Porting it is a separate piece of work; it needs modernising, not copying. --- mkdocs.yml | 44 +++++----- site_docs/architecture/adding-a-pass.md | 8 -- site_docs/architecture/audit-index.md | 8 -- site_docs/architecture/engine.md | 8 -- site_docs/architecture/validation-matrix.md | 8 -- site_docs/cartridges/build-from-fasta.md | 11 --- site_docs/cartridges/empirical-models.md | 5 -- .../cartridges/estimators/allele-usage.md | 8 -- site_docs/cartridges/estimators/index.md | 8 -- .../cartridges/estimators/np-base-model.md | 8 -- site_docs/cartridges/estimators/np-length.md | 8 -- .../estimators/p-nucleotide-length.md | 8 -- .../estimators/trim-distributions.md | 8 -- site_docs/cartridges/rules-and-identity.md | 5 -- .../cartridges/v-subregion-annotations.md | 8 -- site_docs/concepts/contracts.md | 5 -- site_docs/concepts/live-call.md | 5 -- site_docs/concepts/persistent-ir.md | 5 -- site_docs/concepts/pipeline.md | 5 -- ...esson-1-vdj-recombination-from-bases-up.md | 5 -- .../lesson-2-the-pipeline-scrubber.md | 5 -- .../lesson-3-s5f-mutation-isnt-uniform.md | 5 -- ...lesson-4-sequencing-artifacts-dialed-in.md | 5 -- .../lesson-5-the-ground-truth-payoff.md | 5 -- site_docs/guides/build-config.md | 5 -- site_docs/guides/export.md | 5 -- site_docs/guides/streaming.md | 5 -- site_docs/guides/tune-corruption.md | 5 -- site_docs/guides/v-usage.md | 5 -- site_docs/guides/with-metadata.md | 5 -- site_docs/validation/audit-realism.md | 5 -- site_docs/validation/benchmark-aligner.md | 5 -- site_docs/validation/compare-shm.md | 5 -- site_docs/validation/replay.md | 5 -- site_docs/validation/reproduce.md | 5 -- site_docs/validation/smoke-test.md | 5 -- site_docs/validation/trace-introspection.md | 5 -- site_docs/validation/two-layer-model.md | 8 -- ...st_docs_redirect_targets_are_real_pages.py | 88 +++++++++++++++++++ 39 files changed, 110 insertions(+), 249 deletions(-) delete mode 100644 site_docs/architecture/adding-a-pass.md delete mode 100644 site_docs/architecture/audit-index.md delete mode 100644 site_docs/architecture/engine.md delete mode 100644 site_docs/architecture/validation-matrix.md delete mode 100644 site_docs/cartridges/build-from-fasta.md delete mode 100644 site_docs/cartridges/empirical-models.md delete mode 100644 site_docs/cartridges/estimators/allele-usage.md delete mode 100644 site_docs/cartridges/estimators/index.md delete mode 100644 site_docs/cartridges/estimators/np-base-model.md delete mode 100644 site_docs/cartridges/estimators/np-length.md delete mode 100644 site_docs/cartridges/estimators/p-nucleotide-length.md delete mode 100644 site_docs/cartridges/estimators/trim-distributions.md delete mode 100644 site_docs/cartridges/rules-and-identity.md delete mode 100644 site_docs/cartridges/v-subregion-annotations.md delete mode 100644 site_docs/concepts/contracts.md delete mode 100644 site_docs/concepts/live-call.md delete mode 100644 site_docs/concepts/persistent-ir.md delete mode 100644 site_docs/concepts/pipeline.md delete mode 100644 site_docs/getting-started/lesson-1-vdj-recombination-from-bases-up.md delete mode 100644 site_docs/getting-started/lesson-2-the-pipeline-scrubber.md delete mode 100644 site_docs/getting-started/lesson-3-s5f-mutation-isnt-uniform.md delete mode 100644 site_docs/getting-started/lesson-4-sequencing-artifacts-dialed-in.md delete mode 100644 site_docs/getting-started/lesson-5-the-ground-truth-payoff.md delete mode 100644 site_docs/guides/build-config.md delete mode 100644 site_docs/guides/export.md delete mode 100644 site_docs/guides/streaming.md delete mode 100644 site_docs/guides/tune-corruption.md delete mode 100644 site_docs/guides/v-usage.md delete mode 100644 site_docs/guides/with-metadata.md delete mode 100644 site_docs/validation/audit-realism.md delete mode 100644 site_docs/validation/benchmark-aligner.md delete mode 100644 site_docs/validation/compare-shm.md delete mode 100644 site_docs/validation/replay.md delete mode 100644 site_docs/validation/reproduce.md delete mode 100644 site_docs/validation/smoke-test.md delete mode 100644 site_docs/validation/trace-introspection.md delete mode 100644 site_docs/validation/two-layer-model.md create mode 100644 tests/acceptance/test_docs_redirect_targets_are_real_pages.py diff --git a/mkdocs.yml b/mkdocs.yml index 93c77f9c..a26c8cb0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -137,33 +137,33 @@ plugins: # NOTE: no 'learn.md' redirect — learn.md is a real nav page # ("Choose your path"); a redirect here would clobber the built # learn.html with a stub and make the page unreachable. - "lesson-1.md": "getting-started/lesson-1-vdj-recombination-from-bases-up.md" - "lesson-2.md": "getting-started/lesson-2-the-pipeline-scrubber.md" - "lesson-3.md": "getting-started/lesson-3-s5f-mutation-isnt-uniform.md" - "lesson-4.md": "getting-started/lesson-4-sequencing-artifacts-dialed-in.md" - "lesson-5.md": "getting-started/lesson-5-the-ground-truth-payoff.md" + "lesson-1.md": "getting-started/index.md" + "lesson-2.md": "getting-started/index.md" + "lesson-3.md": "getting-started/index.md" + "lesson-4.md": "getting-started/index.md" + "lesson-5.md": "getting-started/index.md" "concepts.md": "concepts/index.md" - "concept-pipeline.md": "concepts/pipeline.md" - "concept-persistent-ir.md": "concepts/persistent-ir.md" - "concept-contracts.md": "concepts/contracts.md" + "concept-pipeline.md": "concepts/index.md" + "concept-persistent-ir.md": "concepts/index.md" + "concept-contracts.md": "concepts/index.md" "concept-airr-record.md": "concepts/airr-record.md" - "concept-live-call.md": "concepts/live-call.md" + "concept-live-call.md": "concepts/index.md" "guides.md": "guides/index.md" - "guide-build-config.md": "guides/build-config.md" + "guide-build-config.md": "guides/build-reference-cartridge.md" "guide-productive.md": "guides/productive.md" "guide-clonal-families.md": "guides/clonal-families.md" - "guide-v-usage.md": "guides/v-usage.md" - "guide-tune-corruption.md": "guides/tune-corruption.md" - "guide-export.md": "guides/export.md" - "guide-streaming.md": "guides/streaming.md" - "guide-with-metadata.md": "guides/with-metadata.md" - "guide-audit-realism.md": "validation/audit-realism.md" - "guide-benchmark-aligner.md": "validation/benchmark-aligner.md" - "guide-compare-shm.md": "validation/compare-shm.md" - "guide-replay.md": "validation/replay.md" - "guide-reproduce.md": "validation/reproduce.md" - "guide-smoke-test.md": "validation/smoke-test.md" - "guide-trace-introspection.md": "validation/trace-introspection.md" + "guide-v-usage.md": "guides/recombination-junction.md" + "guide-tune-corruption.md": "guides/corruption-sequencing.md" + "guide-export.md": "getting-started/export-results.md" + "guide-streaming.md": "reference/index.md" + "guide-with-metadata.md": "getting-started/export-results.md" + "guide-audit-realism.md": "validation/index.md" + "guide-benchmark-aligner.md": "validation/index.md" + "guide-compare-shm.md": "validation/index.md" + "guide-replay.md": "guides/trace-replay.md" + "guide-reproduce.md": "guides/trace-replay.md" + "guide-smoke-test.md": "validation/index.md" + "guide-trace-introspection.md": "guides/trace-replay.md" "reference.md": "reference/index.md" - mkdocstrings: handlers: diff --git a/site_docs/architecture/adding-a-pass.md b/site_docs/architecture/adding-a-pass.md deleted file mode 100644 index 31164c0f..00000000 --- a/site_docs/architecture/adding-a-pass.md +++ /dev/null @@ -1,8 +0,0 @@ -# Adding a pass - - -_Linked from `docs/adding_a_pass.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/architecture/audit-index.md b/site_docs/architecture/audit-index.md deleted file mode 100644 index d1fd67e4..00000000 --- a/site_docs/architecture/audit-index.md +++ /dev/null @@ -1,8 +0,0 @@ -# Audit + design doc index - - -_Auto-generated catalogue of every `docs/*audit.md` and `docs/*design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/architecture/engine.md b/site_docs/architecture/engine.md deleted file mode 100644 index 466783d6..00000000 --- a/site_docs/architecture/engine.md +++ /dev/null @@ -1,8 +0,0 @@ -# Engine architecture - - -_Linked from `docs/engine_architecture.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/architecture/validation-matrix.md b/site_docs/architecture/validation-matrix.md deleted file mode 100644 index 5fd94a2d..00000000 --- a/site_docs/architecture/validation-matrix.md +++ /dev/null @@ -1,8 +0,0 @@ -# Validation matrix - - -_Linked from `docs/validation_matrix.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Validation & Debugging](https://mutejester.github.io/GenAIRR/validation/index.html). diff --git a/site_docs/cartridges/build-from-fasta.md b/site_docs/cartridges/build-from-fasta.md deleted file mode 100644 index 540ff12a..00000000 --- a/site_docs/cartridges/build-from-fasta.md +++ /dev/null @@ -1,11 +0,0 @@ -# Build a cartridge from FASTA - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Build a reference cartridge](https://mutejester.github.io/GenAIRR/guides/build-reference-cartridge.html). - -New authoring also preserves source-header declarations and carries optional -sequence-bound landmark, coding-phase and coding-boundary evidence. Missing -species and unresolved anchors remain explicit. See -[Reference evidence and catalogue quarantine](reference-evidence.md) for the -schema, source requirements and bundled audit status. diff --git a/site_docs/cartridges/empirical-models.md b/site_docs/cartridges/empirical-models.md deleted file mode 100644 index 3a74fecb..00000000 --- a/site_docs/cartridges/empirical-models.md +++ /dev/null @@ -1,5 +0,0 @@ -# Empirical models - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/allele-usage.md b/site_docs/cartridges/estimators/allele-usage.md deleted file mode 100644 index 32414633..00000000 --- a/site_docs/cartridges/estimators/allele-usage.md +++ /dev/null @@ -1,8 +0,0 @@ -# Allele usage estimator - - -_Rewritten from `docs/allele_usage_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/index.md b/site_docs/cartridges/estimators/index.md deleted file mode 100644 index bf40b133..00000000 --- a/site_docs/cartridges/estimators/index.md +++ /dev/null @@ -1,8 +0,0 @@ -# Estimators overview - - -_Hub for the 5 cartridge estimators._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/np-base-model.md b/site_docs/cartridges/estimators/np-base-model.md deleted file mode 100644 index f2a4c203..00000000 --- a/site_docs/cartridges/estimators/np-base-model.md +++ /dev/null @@ -1,8 +0,0 @@ -# NP base model estimator - - -_Rewritten from `docs/np_base_model_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/np-length.md b/site_docs/cartridges/estimators/np-length.md deleted file mode 100644 index bc9ba2ef..00000000 --- a/site_docs/cartridges/estimators/np-length.md +++ /dev/null @@ -1,8 +0,0 @@ -# NP length estimator - - -_Rewritten from `docs/np_length_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/p-nucleotide-length.md b/site_docs/cartridges/estimators/p-nucleotide-length.md deleted file mode 100644 index 31d6018b..00000000 --- a/site_docs/cartridges/estimators/p-nucleotide-length.md +++ /dev/null @@ -1,8 +0,0 @@ -# P-nucleotide length estimator - - -_Rewritten from `docs/p_nucleotide_length_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/estimators/trim-distributions.md b/site_docs/cartridges/estimators/trim-distributions.md deleted file mode 100644 index 3d74c62d..00000000 --- a/site_docs/cartridges/estimators/trim-distributions.md +++ /dev/null @@ -1,8 +0,0 @@ -# Trim distributions estimator - - -_Rewritten from `docs/trim_distribution_estimation_design.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/cartridges/rules-and-identity.md b/site_docs/cartridges/rules-and-identity.md deleted file mode 100644 index 6d4eff82..00000000 --- a/site_docs/cartridges/rules-and-identity.md +++ /dev/null @@ -1,5 +0,0 @@ -# Rules + identity - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Reference Cartridge (concept)](https://mutejester.github.io/GenAIRR/concepts/reference-cartridge.html). diff --git a/site_docs/cartridges/v-subregion-annotations.md b/site_docs/cartridges/v-subregion-annotations.md deleted file mode 100644 index 693def9d..00000000 --- a/site_docs/cartridges/v-subregion-annotations.md +++ /dev/null @@ -1,8 +0,0 @@ -# V-subregion annotations - - -_Rewritten from `docs/v_region_substructure_audit.md`._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Targeted SHM rates](https://mutejester.github.io/GenAIRR/guides/shm-targeting.html). diff --git a/site_docs/concepts/contracts.md b/site_docs/concepts/contracts.md deleted file mode 100644 index 471a5e56..00000000 --- a/site_docs/concepts/contracts.md +++ /dev/null @@ -1,5 +0,0 @@ -# Contracts - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Sample only productive sequences](https://mutejester.github.io/GenAIRR/guides/productive.html). diff --git a/site_docs/concepts/live-call.md b/site_docs/concepts/live-call.md deleted file mode 100644 index 84bff3f2..00000000 --- a/site_docs/concepts/live-call.md +++ /dev/null @@ -1,5 +0,0 @@ -# Live Calls - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/concepts/persistent-ir.md b/site_docs/concepts/persistent-ir.md deleted file mode 100644 index 722e7784..00000000 --- a/site_docs/concepts/persistent-ir.md +++ /dev/null @@ -1,5 +0,0 @@ -# Persistent IR - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Architecture](https://mutejester.github.io/GenAIRR/architecture/index.html). diff --git a/site_docs/concepts/pipeline.md b/site_docs/concepts/pipeline.md deleted file mode 100644 index 0c0adf82..00000000 --- a/site_docs/concepts/pipeline.md +++ /dev/null @@ -1,5 +0,0 @@ -# The Simulation Pipeline - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [The Experiment builder](https://mutejester.github.io/GenAIRR/guides/experiment-builder.html). diff --git a/site_docs/getting-started/lesson-1-vdj-recombination-from-bases-up.md b/site_docs/getting-started/lesson-1-vdj-recombination-from-bases-up.md deleted file mode 100644 index 4512eaad..00000000 --- a/site_docs/getting-started/lesson-1-vdj-recombination-from-bases-up.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 1 - V(D)J recombination, from bases up - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Quick start](https://mutejester.github.io/GenAIRR/getting-started/quick-start.html). diff --git a/site_docs/getting-started/lesson-2-the-pipeline-scrubber.md b/site_docs/getting-started/lesson-2-the-pipeline-scrubber.md deleted file mode 100644 index b91b95a2..00000000 --- a/site_docs/getting-started/lesson-2-the-pipeline-scrubber.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 2 - The pipeline scrubber - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [The Experiment builder](https://mutejester.github.io/GenAIRR/guides/experiment-builder.html). diff --git a/site_docs/getting-started/lesson-3-s5f-mutation-isnt-uniform.md b/site_docs/getting-started/lesson-3-s5f-mutation-isnt-uniform.md deleted file mode 100644 index d66812db..00000000 --- a/site_docs/getting-started/lesson-3-s5f-mutation-isnt-uniform.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 3 - S5F: mutation isn't uniform - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Targeted SHM rates](https://mutejester.github.io/GenAIRR/guides/shm-targeting.html). diff --git a/site_docs/getting-started/lesson-4-sequencing-artifacts-dialed-in.md b/site_docs/getting-started/lesson-4-sequencing-artifacts-dialed-in.md deleted file mode 100644 index cc549adf..00000000 --- a/site_docs/getting-started/lesson-4-sequencing-artifacts-dialed-in.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 4 - Sequencing artifacts, dialed in - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Corruption + sequencing artefacts](https://mutejester.github.io/GenAIRR/guides/corruption-sequencing.html). diff --git a/site_docs/getting-started/lesson-5-the-ground-truth-payoff.md b/site_docs/getting-started/lesson-5-the-ground-truth-payoff.md deleted file mode 100644 index b8c15f8c..00000000 --- a/site_docs/getting-started/lesson-5-the-ground-truth-payoff.md +++ /dev/null @@ -1,5 +0,0 @@ -# Lesson 5 - The ground-truth payoff - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Your first AIRR record](https://mutejester.github.io/GenAIRR/getting-started/first-airr-record.html). diff --git a/site_docs/guides/build-config.md b/site_docs/guides/build-config.md deleted file mode 100644 index 01575f57..00000000 --- a/site_docs/guides/build-config.md +++ /dev/null @@ -1,5 +0,0 @@ -# Build a config for your locus or species - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Build a reference cartridge](https://mutejester.github.io/GenAIRR/guides/build-reference-cartridge.html) or [Reference Cartridge Authoring](https://mutejester.github.io/GenAIRR/cartridges/index.html). diff --git a/site_docs/guides/export.md b/site_docs/guides/export.md deleted file mode 100644 index c0b321ed..00000000 --- a/site_docs/guides/export.md +++ /dev/null @@ -1,5 +0,0 @@ -# Export to AIRR TSV, FASTA, or FASTQ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Export the results](https://mutejester.github.io/GenAIRR/getting-started/export-results.html). diff --git a/site_docs/guides/streaming.md b/site_docs/guides/streaming.md deleted file mode 100644 index d851b8ff..00000000 --- a/site_docs/guides/streaming.md +++ /dev/null @@ -1,5 +0,0 @@ -# Stream millions of sequences without OOM - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [The Experiment builder](https://mutejester.github.io/GenAIRR/guides/experiment-builder.html). diff --git a/site_docs/guides/tune-corruption.md b/site_docs/guides/tune-corruption.md deleted file mode 100644 index 056ea9a2..00000000 --- a/site_docs/guides/tune-corruption.md +++ /dev/null @@ -1,5 +0,0 @@ -# Tune corruption rates to match your sequencer - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Corruption + sequencing artefacts](https://mutejester.github.io/GenAIRR/guides/corruption-sequencing.html). diff --git a/site_docs/guides/v-usage.md b/site_docs/guides/v-usage.md deleted file mode 100644 index abff3d10..00000000 --- a/site_docs/guides/v-usage.md +++ /dev/null @@ -1,5 +0,0 @@ -# Match an empirical V-gene usage distribution - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Estimate models from data](https://mutejester.github.io/GenAIRR/guides/estimate-cartridge-models.html). diff --git a/site_docs/guides/with-metadata.md b/site_docs/guides/with-metadata.md deleted file mode 100644 index e361d3b4..00000000 --- a/site_docs/guides/with-metadata.md +++ /dev/null @@ -1,5 +0,0 @@ -# Attach sample metadata to records - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [The Experiment builder](https://mutejester.github.io/GenAIRR/guides/experiment-builder.html). diff --git a/site_docs/validation/audit-realism.md b/site_docs/validation/audit-realism.md deleted file mode 100644 index 3446cbaa..00000000 --- a/site_docs/validation/audit-realism.md +++ /dev/null @@ -1,5 +0,0 @@ -# Audit a simulation for biological realism - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Validation & Debugging](https://mutejester.github.io/GenAIRR/validation/index.html). diff --git a/site_docs/validation/benchmark-aligner.md b/site_docs/validation/benchmark-aligner.md deleted file mode 100644 index b7528cbe..00000000 --- a/site_docs/validation/benchmark-aligner.md +++ /dev/null @@ -1,5 +0,0 @@ -# Benchmark an aligner against truth - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Validation & Debugging](https://mutejester.github.io/GenAIRR/validation/index.html). diff --git a/site_docs/validation/compare-shm.md b/site_docs/validation/compare-shm.md deleted file mode 100644 index d3990451..00000000 --- a/site_docs/validation/compare-shm.md +++ /dev/null @@ -1,5 +0,0 @@ -# Compare two SHM models - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Targeted SHM rates](https://mutejester.github.io/GenAIRR/guides/shm-targeting.html). diff --git a/site_docs/validation/replay.md b/site_docs/validation/replay.md deleted file mode 100644 index 0280a561..00000000 --- a/site_docs/validation/replay.md +++ /dev/null @@ -1,5 +0,0 @@ -# Replay a simulation with one knob changed - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Trace, replay, reproducibility](https://mutejester.github.io/GenAIRR/guides/trace-replay.html). diff --git a/site_docs/validation/reproduce.md b/site_docs/validation/reproduce.md deleted file mode 100644 index c55aa5a2..00000000 --- a/site_docs/validation/reproduce.md +++ /dev/null @@ -1,5 +0,0 @@ -# Reproduce a published dataset - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Trace, replay, reproducibility](https://mutejester.github.io/GenAIRR/guides/trace-replay.html). diff --git a/site_docs/validation/smoke-test.md b/site_docs/validation/smoke-test.md deleted file mode 100644 index 43f60659..00000000 --- a/site_docs/validation/smoke-test.md +++ /dev/null @@ -1,5 +0,0 @@ -# Smoke-test a config change - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Installation](https://mutejester.github.io/GenAIRR/getting-started/installation.html). diff --git a/site_docs/validation/trace-introspection.md b/site_docs/validation/trace-introspection.md deleted file mode 100644 index a1d16f06..00000000 --- a/site_docs/validation/trace-introspection.md +++ /dev/null @@ -1,5 +0,0 @@ -# Introspect the trace - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Trace, replay, reproducibility](https://mutejester.github.io/GenAIRR/guides/trace-replay.html). diff --git a/site_docs/validation/two-layer-model.md b/site_docs/validation/two-layer-model.md deleted file mode 100644 index 28da66fc..00000000 --- a/site_docs/validation/two-layer-model.md +++ /dev/null @@ -1,8 +0,0 @@ -# Two-layer integrity model - - -_Rewritten from `docs/validation_matrix.md` §1.0._ - -!!! info "This page has not been written yet" - There is no content here yet. What you are looking for is most - likely on [Validation & Debugging](https://mutejester.github.io/GenAIRR/validation/index.html). diff --git a/tests/acceptance/test_docs_redirect_targets_are_real_pages.py b/tests/acceptance/test_docs_redirect_targets_are_real_pages.py new file mode 100644 index 00000000..8b043ee1 --- /dev/null +++ b/tests/acceptance/test_docs_redirect_targets_are_real_pages.py @@ -0,0 +1,88 @@ +"""Every preserved URL must land on a page that says something. + +`mkdocs.yml`'s `redirect_maps` keeps the pre-MkDocs URLs resolvable. A +redirect is only worth having if its destination answers the question the +old URL answered: a reader who follows a bookmark or a citation into a +placeholder is worse off than one who gets a 404, because the 404 at least +tells them to search. + +This ran green while 22 of the 28 redirects pointed at pages whose entire +body was an admonition saying the page had not been written. It did not +catch that, because nothing checked it. These two tests do. + +They deliberately do not depend on the contributor-only migration plan the +sibling `test_docs_mkdocs_migration_contract` module parses, which is +absent from every clone and skips that whole module in CI. +""" + +from __future__ import annotations + +import pathlib +import re + +import pytest + +_REPO_ROOT = pathlib.Path(__file__).resolve().parents[2] +_MKDOCS_YML = _REPO_ROOT / "mkdocs.yml" +_SITE_DOCS = _REPO_ROOT / "site_docs" + +# Phrases a placeholder page uses to admit it is one. Matched +# case-insensitively against the whole page body. +_PLACEHOLDER_MARKERS = ( + "has not been written yet", + "this page is a placeholder", + "migration placeholder", + "content lands in phase", +) + +pytestmark = pytest.mark.pin + + +def _redirect_map() -> dict[str, str]: + """`{fake source .md path: destination .md path}` from `redirect_maps`.""" + text = _MKDOCS_YML.read_text(encoding="utf-8") + pairs = re.findall(r'^\s*"([^"]+\.md)":\s*"([^"]+\.md)"\s*$', text, re.M) + assert pairs, "no redirect_maps entries found in mkdocs.yml" + return dict(pairs) + + +def test_pin_present_every_redirect_target_is_a_real_file() -> None: + """A redirect whose destination does not exist fails the build late. + + `mkdocs-redirects` emits the redirect file regardless, so the breakage + shows up as a 404 for a reader rather than as an error for us. + """ + missing = sorted( + f"{source} -> {target}" + for source, target in _redirect_map().items() + if not (_SITE_DOCS / target).is_file() + ) + assert not missing, ( + "redirect destinations that do not exist under site_docs/:\n " + + "\n ".join(missing) + ) + + +def test_pin_present_no_redirect_lands_on_a_placeholder() -> None: + """The preserved URL must answer the question the old URL answered. + + If a page has not been written, point its redirect at the nearest page + that has been, and delete the placeholder. Keeping the placeholder as a + destination spends the URL on an apology. + """ + landing_on_placeholder = [] + for source, target in sorted(_redirect_map().items()): + page = _SITE_DOCS / target + if not page.is_file(): + continue # the test above owns this failure + body = page.read_text(encoding="utf-8").lower() + for marker in _PLACEHOLDER_MARKERS: + if marker in body: + landing_on_placeholder.append(f"{source} -> {target} ({marker!r})") + break + assert not landing_on_placeholder, ( + "redirects pointing at placeholder pages:\n " + + "\n ".join(landing_on_placeholder) + + "\n\nRepoint each at the nearest page that has content, and delete " + "the placeholder." + ) From 9233392675cc354ff118c52fe29446cc843c859c Mon Sep 17 00:00:00 2001 From: thomas Date: Fri, 6 Nov 2026 09:59:41 +0200 Subject: [PATCH 5/7] ci: the docs build from the checkout, not from the last release The docs deploy has failed on every merge to master since 2026-10-21 -- 15 consecutive runs. `mkdocstrings` could not collect `GenAIRR._reference.artifact.Reference`, because the workflow installed GenAIRR from PyPI and that class does not exist in 2.3.1. That is the dependency the wrong way round. `site_docs/` is versioned with the source, so a page documenting anything added since the last release cannot build, and the site silently keeps serving whatever the last green run produced. It had been serving a fortnight-old build -- which is why the published placeholder pages still named a contributor-only planning document, long after the repo stopped mentioning it. Building the checkout keeps the rendered reference and the prose around it in step by construction, at the cost of a Rust build in the docs job; the cargo cache and toolchain action are the same ones `test.yml` already uses. This is the configuration the local `mkdocs build --strict` has been passing under all along. The deploy job itself was never the problem: it last succeeded on 2026-10-21, so GitHub Pages is configured correctly and resumes once the build does. --- .github/workflows/deploy-docs.yml | 35 ++++++++++++++++++++++++------- 1 file changed, 28 insertions(+), 7 deletions(-) diff --git a/.github/workflows/deploy-docs.yml b/.github/workflows/deploy-docs.yml index 9570fde6..9d3c2764 100644 --- a/.github/workflows/deploy-docs.yml +++ b/.github/workflows/deploy-docs.yml @@ -44,15 +44,36 @@ jobs: - name: Install docs dependencies run: pip install -r requirements-docs.txt - - name: Install GenAIRR from PyPI (mkdocstrings needs to import it) + - name: Set up Rust + uses: dtolnay/rust-toolchain@stable + + - name: Cache cargo build + uses: actions/cache@v4 + with: + path: | + ~/.cargo/registry + ~/.cargo/git + engine_rs/target + key: ${{ runner.os }}-docs-cargo-${{ hashFiles('engine_rs/Cargo.lock') }} + restore-keys: ${{ runner.os }}-docs-cargo- + + - name: Install GenAIRR from this checkout (mkdocstrings needs to import it) # The reference pages use `:::` mkdocstrings directives to render # signatures + docstrings from real classes (Experiment, - # SimulationResult, ReferenceCartridgeBuilder, etc.). For those - # to resolve, GenAIRR must be importable in the build environment. - # Installing from PyPI is fast (precompiled wheels) and means the - # rendered API reference matches the latest released version, - # which is what published docs should reflect. - run: pip install GenAIRR + # SimulationResult, ReferenceCartridgeBuilder, etc.). For those to + # resolve, GenAIRR must be importable in the build environment. + # + # This used to install from PyPI, on the reasoning that published + # docs should reflect the published release. That inverts the + # dependency: `site_docs/` is versioned with the source, so a page + # documenting anything newer than the last release cannot build. It + # broke on `GenAIRR._reference.artifact.Reference` and the deploy + # then failed on every merge to master from 2026-10-21 until the + # 2.4.0 release work, which is why the live site served + # pre-release text for a fortnight. Building the checkout keeps the + # rendered reference and the prose describing it in step by + # construction. + run: pip install . - name: Build site with mkdocs build --strict run: make docs-build From 6865d10dec691653efdf2a9c78bfe133f080d2cf Mon Sep 17 00:00:00 2001 From: thomas Date: Fri, 6 Nov 2026 10:00:33 +0200 Subject: [PATCH 6/7] chore: the release declarations are spent, so the gate asks for them gone Both entries explained the 2.4.0 dossier regeneration, which is now in the baseline. A declaration left behind after its change merges excuses the next edit to the same name, so the gate reports them as explaining nothing on this run. --- tools/inventory-declarations.json | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/tools/inventory-declarations.json b/tools/inventory-declarations.json index 5126d74c..38e046b3 100644 --- a/tools/inventory-declarations.json +++ b/tools/inventory-declarations.json @@ -1,8 +1,5 @@ { - "revised": { - "MANIFEST_PIN": "the packaged reference dossier was regenerated for the 2.4.0 release, so its manifest hash moved from 9ab6ba63 to 101abcebe. The manifest records the GenAIRR version that generated the dossier, which is why a version bump alone retargets this pin; tools/build_published_reference.py prints the new value and names every pin site, and test_the_packaged_dossier_was_generated_by_the_version_this_checkout_declares fails if the two disagree. The same value is pinned once more in src/GenAIRR/_reference_sources/bundled.py's REGISTRY and was updated there in the same change. Nothing about the reference build itself changed: the runtime execution, catalogue and models hashes and the source lock all came back byte-identical, which is the evidence that only the recorded builder version moved.", - "ARCHIVE_PIN": "the raw sha256 of the regenerated archive, 52df85b3 to d8d82abc. The archive holds the manifest, so it changes with the version for the same reason MANIFEST_PIN does -- the comment above this constant already said so before this change. The archive is byte-identical in length (20110025 bytes both times), which is consistent with a change confined to the recorded version string rather than to any packaged member." - }, + "revised": {}, "replaced": {}, "allow": {} } From 19504bc881552f4bf7c08f1254510e857ecd853b Mon Sep 17 00:00:00 2001 From: thomas Date: Fri, 6 Nov 2026 10:27:51 +0200 Subject: [PATCH 7/7] test: the redirect pins use the fixed path anchor like every other module `test_no_module_builds_a_path_from_file_except_the_recorded_ones` fails on a module that derives the repo root from `__file__` without a recorded reason, and this one did. `tests.support.paths.repo_root()` is anchored to a fixed location and is what the rest of the suite uses, so the fix is to follow the convention rather than record an exception for a test that needs nothing special. --- .../test_docs_redirect_targets_are_real_pages.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/acceptance/test_docs_redirect_targets_are_real_pages.py b/tests/acceptance/test_docs_redirect_targets_are_real_pages.py index 8b043ee1..5a6a4b59 100644 --- a/tests/acceptance/test_docs_redirect_targets_are_real_pages.py +++ b/tests/acceptance/test_docs_redirect_targets_are_real_pages.py @@ -17,14 +17,14 @@ from __future__ import annotations -import pathlib import re import pytest -_REPO_ROOT = pathlib.Path(__file__).resolve().parents[2] -_MKDOCS_YML = _REPO_ROOT / "mkdocs.yml" -_SITE_DOCS = _REPO_ROOT / "site_docs" +from tests.support.paths import repo_root + +_MKDOCS_YML = repo_root() / "mkdocs.yml" +_SITE_DOCS = repo_root() / "site_docs" # Phrases a placeholder page uses to admit it is one. Matched # case-insensitively against the whole page body.