v0.8.1 results: tables, manifests, site data and the v081 release entry

CHANGELOG.md

@@ -8,7 +8,7 @@ release completed.
 
 ## [0.8.1] — TBD — `v0.8.1`
 
-> DRAFT release prep (do not merge until the release is cut). The entries below are those of the merged pull requests; the definitive run (`--preset v0.8.1`, long `GENOME_TIMEOUT` for the nine rust genomes of the allow list) and its counts, and the date, are still to be filled in. v0.8.1 is a re-run of the definitive v0.8.0 run with: the #228 fallback after withheld hits (preset `v0.8.1`), the *S. pombe* silent-pair rule (#254; the #252 tie-break is reverted), report-only `resolution_flags`, the cross-family `cassette` indication with its shared-flank measurements, `two_idiomorphs` for Basidiomycota MAT and redPR, the rust allow list (#165) and the *Hyphopichia* MTLA1 record (#148; 11 of 389 sampled Serinales genomes drop a tier, tracked under #149). Not in v0.8.1: #255, the MATam family, the Ogataea candidate records.
+> DRAFT release prep (do not merge until the release is cut). The entries below are those of the merged pull requests; the definitive run and its counts are in "The definitive run" below; the release date is still to be filled in. v0.8.1 is a re-run of the definitive v0.8.0 run with: the #228 fallback after withheld hits (preset `v0.8.1`), the *S. pombe* silent-pair rule (#254; the #252 tie-break is reverted), report-only `resolution_flags`, the cross-family `cassette` indication with its shared-flank measurements, `two_idiomorphs` for Basidiomycota MAT and redPR, the rust allow list (#165) and the *Hyphopichia* MTLA1 record (#148; 11 of 389 sampled Serinales genomes drop a tier, tracked under #149). Not in v0.8.1: #255, the MATam family, the Ogataea candidate records.
 
 ### Changed
 - *S. pombe* silent cassettes (#254; curator ruling 2026-11-05): a merged `mat1` call whose genes hold a P cluster and an M cluster 5-20 kb apart on one contig (the silent `mat2`-P / `mat3`-M pair, about 11 kb apart) is split into a `mat2` P call and a `mat3` M call, each with a `cassette` block (role `silent_pair`, the partner family and the distance). The #252 tie-break by core identity is reverted (it relabelled a lone P cassette `mat2`, probably wrong: the records are cassette-only so identity cannot tell mat1 from mat2); a lone cassette elsewhere stays `mat1`; a pair farther apart (35 kb in GCA_006386235.1) stays one call marked `arrangement_ambiguous`. It applies only where the species-scoped `mat2` and `mat3` families are in the genome's lineage (*S. pombe*), as part of `--cross-family-policy on` (`detect/silent_pair.py`). It changes *S. pombe* locus counts and labels.
@@ -50,6 +50,24 @@ release completed.
 - Browser tool: a Reset button (reloads the page, which frees the interpreter and the mounted database of earlier runs; choose the genome again) and a Copy log button (the log with the page address, browser and time, for a bug report). The log is cleared and its timer restarts at each run.
 - `matpredict detect --phylum auto` (and `auto-weak`): ask [GmG](https://github.com/stajichlab/GmG) which phylum the genome is in, then search that phylum's curated families (the order or family taxid GmG gives is tried first, so the lineage routing applies, and the genetic code is read from it). A weak call, a genome GmG says is not a fungus, no call, or a phylum with no curated family is NOT searched and the report says why (`parameters.phylum_guess`); `auto-weak` also accepts a weak call; `--exhaustive` still forces a search. An explicit `--phylum NAME` or `--taxid` is unchanged and wins. `--gmg-index FILE|NAME|latest|URL` picks the GmG reference index (downloaded once to `~/.cache/matpredict/gmg`, checked against GmG's published sha256, pinned by default, with a warning when a newer one exists). Needs the `gmg` binary or GmG's Python bindings; needs GmG with `routing_hint` for the full rule (an older GmG is read with the same rule). Design: `docs/notes/2026-11-02_gmg-phylum-routing.md`.
 
+### The definitive run
+The v0.8.1 data come from commit `50bc950` (main after the merge of #272; `src`, `db` and `scripts` as released), run with `--preset v0.8.1` (the v0.8 policies plus `--fallback-after-withheld-policy on`) and, for the nine rust genomes of the allow list, `GENOME_TIMEOUT=14400`. The lists are those of the v0.8.0 run: 19,415 Ascomycota, 3,275 Basidiomycota and 974 Mucoromycotina genomes. Two Ascomycota genomes (GCA_016165855.1, GCA_023522825.1) hit the 3,600 s limit on an older node and were re-run alone on an EPYC node with the same code and preset (3,063 s and 1,685 s); the tables take those reports. Folders: `results/2026-11-05_v081_{ascomycota,basidiomycota,mucoromycotina}`.
+
+| Campaign | Genomes listed | Called | Uncalled | No report (suppressed) | Loci | Loci high / medium / low |
+|---|---|---|---|---|---|---|
+| Ascomycota | 19,415 | 17,618 | 1,725 | 72 | 20,843 | 14,623 / 6,120 / 100 |
+| Basidiomycota | 3,275 | 2,995 | 268 | 12 | 5,993 | 3,822 / 2,161 / 10 |
+| Mucoromycotina | 974 | 868 | 105 | 1 | 914 | 689 / 212 / 13 |
+
+Against v0.8.0 (23,570 genomes with a report in both runs): 23,214 are identical in calls, confidence, coordinates and gene models and 356 differ; the nine allowed rust genomes have no v0.8.0 report. No call is lost, no idiomorph changes and the Mucoromycotina campaign is unchanged. Every difference was traced with control runs on the frozen code (`results/2026-11-06_attrib_v081` on the HPCC), and nothing is unexplained:
+- *S. pombe* silent pair (#254, #274): 37 Ascomycota genomes (loci 57 to 97). The merged `mat1` M call becomes a `mat2` P and a `mat3` M call (40 pairs); 7 `mat1` M and 10 lone `mat1` P calls stay. No other family, idiomorph (outside the split) or confidence changes.
+- *Hyphopichia* `MTLA1` record (#148, #273): with the record removed from the database (v0.8.1 code and preset), all 302 affected genomes return exactly to their v0.8.0 report. 190 genomes change a call field: 4 are new calls (*Metschnikowia*, *Cephaloascus*, *Saccharomycopsis*, *Dujonia*), 160 have a changed locus span (mostly Serinales `MTL`, also a few Pichiales and Phaffomycetales `MATsc` genomes searched with the same query set), 36 loci rise and 14 drop a tier (the #149 floor and cross-hit guard: 7 `distant_fragment`, 1 `polished_model_lost`, 6 without a listed mechanism). 112 more genomes keep the same calls and differ only in which reference record supplies a gene model. *Cephaloascus* GCA_030575015.1 is called `MTL` A (medium) through the record; without it the #228 fallback would call `MATsc` MATa instead.
+- Fallback after withheld hits (#228): 17 genomes are called that were not before (7 Ascomycota: *Naumovozyma*, *Ogataea*, *Babjeviella*, and one each of *Terfezia*, *Monosporascus*, *Biscogniauxia*, *Ascodesmis*; 10 Basidiomycota, 15 loci: *Tremellales* `PR` and `HD`, one *Puccinia striiformis* `PR` and one *Meruliopsis* `bLocus`). In each, the report's `fallback_search` names the withheld routed family; running the Ascomycota ones with `--preset v0.8` brings back the v0.8.0 report. The fallback searched 1,147 Ascomycota and 69 Basidiomycota genomes and found a call in these 17.
+- Rust allow list (#165, #269): the nine genomes are searched; eight are called (`rustHD`, 14 loci) and one (GCA_001443065.1) is uncalled. The 12 remaining suppressed Basidiomycota genomes have no report.
+- Report-only changes with no effect on a call: `resolution_flags` is set on 11,857 Ascomycota, 3,489 Basidiomycota and 236 Mucoromycotina loci (#257); `two_idiomorphs` is reported for 9 Basidiomycota genomes (#271); the cross-family `cassette` indication with shared-flank measurements (#264) changes no call or family label in any campaign.
+
+The determinism check (60 Ascomycota genomes drawn with seed 20261106, 58 of them with a report; run twice on this commit with `--preset v0.8.1`) gave no differences in family, idiomorph, confidence, contig, start or end between the two runs or against the campaign; the order of the gene names in `genes_found` differed in one genome between the campaign and the re-runs, which agree with each other (`results/2026-11-06_determinism_v081` on the HPCC).
+
 ## [0.8.0] — 2026-10-31 — `v0.8.0`
 
 (Absorbs the planned v0.7.1 site release: the labels and reads pages, the yeast routing and family work (#188, #189) and the browser examples. Unlike v0.7.0, `src` and `db` change: the new `--cross-family-policy` and `--preset v0.8` (off unless asked), a wider MATog scope and a new MATpk family change which families search Ogataea and *P. kudriavzevii* genomes. The site's data and counts are those of the v0.8.0 run (below), made on commit `a2933f7` with `--preset v0.8`.)

README.md

@@ -10,7 +10,7 @@ idiomorph or mating type it is (for example Plus/Minus, MAT1-1/MAT1-2, HD/PR
 alleles). Every reference record traces to a publication or a validated
 deposit, and every call reports the evidence behind it.
 
-Current release: [`v0.8.0`](https://github.com/stajichlab/MATPredict/releases/tag/v0.8.0) (detection code, database and the data, calls and counts shown on the site are those of the v0.8.0 run).
+Current release: [`v0.8.1`](https://github.com/stajichlab/MATPredict/releases/tag/v0.8.1) (detection code, database and the data, calls and counts shown on the site are those of the v0.8.1 run).
 Changes: [CHANGELOG.md](CHANGELOG.md).
 
 ## Contents
@@ -505,28 +505,28 @@ pixi run -e test pytest -q
 ## Validation and known limits
 
 Measured results (details in [analysis/INDEX.md](analysis/INDEX.md)). The NCBI public genomes and held-out rows are detection (call) rates from
-the v0.8.0 definitive run, not accuracy: no truth set exists for the NCBI public genomes. Subphylum rates count the classes of each subphylum;
-the 7 rust genomes skipped for annotation reasons (#165) are counted as not called. The ground-truth, typing and hold-out rows are
-older measurements and were not repeated for v0.8.0. How each campaign was judged is in
+the v0.8.1 definitive run, not accuracy: no truth set exists for the NCBI public genomes. Subphylum rates count the classes of each subphylum;
+the 12 Basidiomycota genomes suppressed by the shared list (no report) are counted as not called; the nine rust genomes of the allow list (#165) are searched. The ground-truth, typing and hold-out rows are
+older measurements and were not repeated for v0.8.1. How each campaign was judged is in
 [results/2026-10_cleanrun_preparation/](results/2026-10_cleanrun_preparation/).
 
 | Test | Result |
 |---|---|
 | Zygomycete ground-truth set (23 genomes) | 23/23 loci and idiomorphs |
-| Held-out LCG Mucoromycotina genomes (never used in curation) | 549/621 called (v0.8.0 run) |
-| Held-out Jena Mucoromycotina genomes | 61/64 called (v0.8.0 run) |
+| Held-out LCG Mucoromycotina genomes (never used in curation) | 549/621 called (v0.8.1 run) |
+| Held-out Jena Mucoromycotina genomes | 61/64 called (v0.8.1 run) |
 | Idiomorph typing validation | 0/540 typed wrong |
 | Pezizomycotina hold-out, order radius | 7/7 |
-| Ascomycota, 19,415 NCBI public genomes (v0.8.0 run) | 17,607 called (90.7%) |
-| Basidiomycota, 3,275 NCBI public genomes (v0.8.0 run) | 2,977 called (90.9%): Agaricomycotina 92.2%, Ustilaginomycotina 97.7%, Pucciniomycotina 79.5% |
-| Mucoromycotina, 289 NCBI public genomes (v0.8.0 run) | 258 called (89.3%) |
+| Ascomycota, 19,415 NCBI public genomes (v0.8.1 run) | 17,618 called (90.7%) |
+| Basidiomycota, 3,275 NCBI public genomes (v0.8.1 run) | 2,995 called (91.5%): Agaricomycotina 92.6%, Ustilaginomycotina 97.7%, Pucciniomycotina 86.3% |
+| Mucoromycotina, 289 NCBI public genomes (v0.8.1 run) | 258 called (89.3%) |
 
 Known limits (open items in [analysis/open-questions.md](analysis/open-questions.md)):
 
 - Only three phyla have curated families.
 - A genome far from every reference often fails the evidence bar, so an
   uncalled genome is weak evidence that the genome has no MAT locus. The full
-  Ascomycota run (19,415 genomes, v0.8.0) leaves 1,736 uncalled and 72 not run.
+  Ascomycota run (19,415 genomes, v0.8.1) leaves 1,725 uncalled and 72 not run.
 - Fragmented assemblies can split a locus or lose its flanking genes.
 - In Mucoromycota, *Absidia* flanking genes are often off-scaffold, and the
   gene order of the Lichtheimiaceae is not yet known.

results/2026-11-05_v081_ascomycota/MANIFEST.txt

@@ -0,0 +1,12 @@
+v0.8.1 definitive run, ascomycota campaign
+commit: 50bc950d8baabebf1be6f40d537e237345d1c3f3 (main after the merge of #272 = version 0.8.1; src/db/scripts as released)
+worktree: /bigdata/stajichlab/jstajich/projects/MATPredict/.claude/worktrees/run-50bc950
+DETECT_ARGS: --preset v0.8.1  (v0.8 policies + fallback-after-withheld on)
+allow list: db/allow.txt (nine rust genomes searched; large chunk GENOME_TIMEOUT=14400, epyc 8:00:00)
+pixi.lock sha256: 675a15123d22857564dbb020592db9075cd60c8bb82f9cbb53a077abb31f4ac9
+compute: throttled, 3 chains x 16 CPUs = 48 CPUs in flight (submit_throttled.py); jobs: results/2026-11-05_v081_submit/jobs.tsv; submitted 2026-11-05
+rerun: GCA_016165855.1_IBUN_P_ulei_V1 and GCA_023522825.1_DampeI2T2 hit GENOME_TIMEOUT=3600 on the AMD Opteron node c01 (wave_2/small_000, wave_7/small_006: empty folders left there). Rerun 2026-11-06 with GENOME_TIMEOUT=14400, same frozen code and --preset v0.8.1, on the EPYC node r11 (job 31779458): rerun_timeout_epyc/runs/<genome>, 3063 s and 1685 s. A first rerun on c01 (job 31779037, rerun_timeout/) was cancelled after 2 h (empty folders). Cause of the slowness: tblastn -seg no on long low-complexity reference proteins (issue 276).
+db content: nine rust genomes searched via db/allow.txt; Hyphopichia burtonii MTLA1 record 717740_makgeolli_MTL_A (#273) in the database
+tables: scripts/build_cleanrun_tables.py with --baseline the v0.8.0 reports_all.tar.zst (baseline columns compare with v0.8.0); reports_all.tar.zst holds the non-empty reports and wall_seconds
+determinism: 60 Ascomycota genomes (seed 20261106; 58 with a report) run twice on this commit with --preset v0.8.1: 0 differences in family, idiomorph, confidence, contig, start or end between the runs or against this campaign (results/2026-11-06_determinism_v081 on HPCC)
+attribution vs v0.8.0: control runs on the frozen code in results/2026-11-06_attrib_v081 on HPCC (database without the Hyphopichia record; --preset v0.8): 0 unexplained differences

results/2026-11-05_v081_ascomycota/by_class.tsv

@@ -0,0 +1,31 @@
+class	genomes	reports	called	pct_called	routing	wall_median_s	wall_p90_s	wall_max_s	loci	n_label_unsupported	tier_dropped	tier_dropped_noted	margin_fallback	vote_collapse	polished_model_lost	distant_fragment	high_supported	high_tie_break	high_grey	outside_scope_high_supported	outside_scope_high_tie_break	outside_scope_high_grey	defective_loci
+Sordariomycetes	6016	6012	5628	93.6	lineage:5976; not_searched:36	146	234	1685	5754	114	0	0	0	0	0	0	0	0	0	4362	1	1	2203
+Eurotiomycetes	3220	3214	3104	96.6	lineage:3210; not_searched:4	139	210	627	3302	10	0	0	0	0	0	0	0	0	0	2058	0	0	1507
+Pichiomycetes	2967	2966	2827	95.3	lineage:2652; phylum_fallback:220; direct:91; not_searched:3	31	53	124	3279	31	12	6	0	0	1	5	0	0	0	0	0	0	155
+Dothideomycetes	2731	2730	2587	94.8	lineage:2724; not_searched:6	160	248	3063	2705	326	0	0	0	0	0	0	302	11	0	1855	11	7	552
+Saccharomycetes	2646	2598	2211	85.1	lineage:2246; phylum_fallback:337; direct:14; not_searched:1	23	49	105	4430	191	2	2	0	0	0	2	0	0	0	0	0	0	300
+Lecanoromycetes	479	478	413	86.4	lineage:476; not_searched:2	128	219	377	434	9	0	0	0	0	0	0	0	0	0	260	0	0	179
+Leotiomycetes	435	430	399	92.8	lineage:430	144	246	634	414	16	0	0	0	0	0	0	0	0	0	275	1	0	213
+Dipodascomycetes	398	397	157	39.5	phylum_fallback:225; lineage:168; not_searched:4	34	66	144	159	3	0	0	0	0	0	0	0	0	0	0	0	0	10
+Pezizomycetes	172	172	139	80.8	lineage:169; direct:3	59	141	233	139	2	0	0	0	0	0	0	0	0	0	1	0	0	51
+Schizosaccharomycetes	68	67	66	98.5	direct:59; lineage:8	18	30	36	136	0	0	0	0	0	0	0	0	0	0	0	0	0	7
+Orbiliomycetes	67	67	4	6.0	phylum_fallback:67	130	214	254	4	0	0	0	0	0	0	0	0	0	0	1	0	0	0
+Lipomycetes	59	59	3	5.1	phylum_fallback:59	37	88	124	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0
+NA	52	48	29	60.4	phylum_fallback:38; lineage:10	121	274	349	32	1	0	0	0	0	0	0	0	0	0	16	0	0	6
+Taphrinomycetes	22	22	5	22.7	lineage:22	13	26	35	5	0	0	0	0	0	0	0	0	0	0	0	0	0	0
+Trigonopsidomycetes	16	16	0	0.0	phylum_fallback:16	25	66	73	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
+Arthoniomycetes	13	13	8	61.5	lineage:11; not_searched:2	118	205	224	8	1	0	0	0	0	0	0	0	0	0	7	0	0	3
+Geoglossomycetes	12	12	6	50.0	lineage:12	139	244	280	6	0	0	0	0	0	0	0	0	0	0	6	0	0	2
+Pneumocystomycetes	11	11	9	81.8	lineage:11	17	26	26	9	2	0	0	0	0	0	0	0	0	0	0	0	0	0
+Lichinomycetes	10	10	7	70.0	lineage:10	127	173	206	8	1	0	0	0	0	0	0	0	0	0	4	0	0	4
+Candelariomycetes	4	4	4	100.0	lineage:4	77	130	130	4	0	0	0	0	0	0	0	0	0	0	1	0	0	2
+Sporopachydermiomycetes	4	4	4	100.0	phylum_fallback:4	30	60	60	4	2	0	0	0	0	0	0	0	0	0	0	0	0	0
+Alloascoideomycetes	3	3	3	100.0	phylum_fallback:3	28	49	49	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0
+Coniocybomycetes	2	2	2	100.0	lineage:2	147	163	163	2	0	0	0	0	0	0	0	0	0	0	0	0	0	0
+Saitoellomycetes	2	2	0	0.0	lineage:2	17	21	21	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
+Laboulbeniomycetes	1	1	0	0.0	lineage:1	89	89	89	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
+Neolectomycetes	1	1	0	0.0	lineage:1	15	15	15	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
+Novakomycetes	1	1	0	0.0	lineage:1	16	16	16	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
+Sareomycetes	1	1	1	100.0	lineage:1	67	67	67	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0
+Xylobotryomycetes	1	1	1	100.0	lineage:1	54	54	54	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0
+Xylonomycetes	1	1	1	100.0	lineage:1	50	50	50	1	0	0	0	0	0	0	0	0	0	0	1	0	0	1