fix(prepare): don't count reads absent from a pre-filtered POD5 as failures

[jayhesselberth]

Summary

• The bug. A BAM covering the full reference set paired with a POD5 deliberately pre-filtered to a subset of those reads — escpod bam-filter, this project's own pipeline Filter stage — had every absent read counted as a per-read failure on both backends. Once the (by-design) mismatch passed MAX_FAILED_READ_FRACTION (50%), which any selective filter reaches, leech data prepare raised and produced no output at all, although every read that was in the POD5 extracted correctly.
• The fix. Reads the POD5 does not carry are now their own expected-exclusion bucket, sitting beside reads_without_motif rather than inside failed_reads: BatchOutcome.n_reads_missing_from_pod5, stats["reads_missing_from_pod5"], and a figure in the "Read yield" log line. The Python fallback's per-read warning for this became a debug — on a pre-filtered POD5 it fires for most of the BAM, and the run-level count is the number worth reading.
• The failed fraction is measured over the reads actually attempted (total_reads - missing) in both numerator and denominator. Leaving absent reads in the denominator is the mirror bug: 100 reads present with 60 of them failing shows as 6% of a 1000-read BAM and never trips.
• Rust API change. training.rs drops a read whose id is not in the signal map to zero chunks, so from Python a pre-filtered batch and a batch the pipeline silently broke on are indistinguishable — which is exactly what _iter_rust_batches' "submitted reads, no chunks back" heuristic (rust: return struct-of-arrays per batch and per-read outcome counts instead of a list of dicts #267/rust: replace the local chunk pipeline with escapepod_signal::chunk #258) keys off. leech_core.extract_training_chunks therefore returns (chunks, n_missing_from_pod5) instead of a bare chunk list (_process_and_convert_training counts submitted ids absent from the signal map), and the heuristic now fires on n_submitted - n_missing. A leech_core predating the change is caught with a "rebuild the extension" error rather than a cryptic unpack failure.
• Issue prepare: a run that loses every batch exits 0; a backend divergence hides in one config corner; torch is imported before the pool forks #265's zero-tolerance policy is preserved. A read that was found in the POD5 and still produced nothing is still a failure; an outright-failed batch still raises; only a batch whose submitted reads were all absent escapes the heuristic. A wholly mismatched BAM/POD5 pair (data prepare: a directory passed to --pod5 yields 0 chunks and exit 0, with a misleading diagnostic #166) is still caught one level up — it extracts zero chunks, which handle_prepare raises on, and that message now names the absent-read count so the mistake doesn't read as a motif problem.
• Root CLAUDE.md's "Failing loud on a broken prepare run (issue prepare: a run that loses every batch exits 0; a backend divergence hides in one config corner; torch is imported before the pool forks #265)" section documents the new category, the attempted-reads denominator, and the Rust return-shape change; changelog.d/325.fixed.md added.

Closes #325.

Test plan

Rust extension rebuilt before any Python test that exercises it, and every command below run under srun -p rna rather than on the login node.

• srun ... --chdir .../rust -- uv run maturin develop --release (i.e. rust/build.sh) — clean build of the new (chunks, n_missing_from_pod5) signature
• cargo fmt --check — clean
• cargo clippy --all-targets -- -D warnings — clean
• cargo test --no-default-features --features test-utils — 16 passed
• uv run pytest tests/test_parallel_prep.py tests/test_prepare_dispatch.py tests/test_backend_parity.py -v — 128 passed
• uv run pytest (full suite) — 1931 passed, 45 skipped
• uv run ruff format --check . / uv run ruff check . — clean
• uv run ty check src/leech/ — clean

New regression tests, on both backends:

• TestPreFilteredPod5 (tests/test_parallel_prep.py) writes a real POD5 holding 2 of the fixture's 20 reads (via escapepod.Writer), then runs the full BAM against it:
  • Python worker: n_failed == 0, n_missing == 18 (90% of the batch), chunks bit-identical to the unfiltered run for the two present reads
  • Rust worker: n_failed == 0, n_missing == n_submitted - 2, chunks only for the present reads
  • prepare_training_data_parallel end to end on Rust (--backend rust, chunk_size=2, so most batches hold no present read at all) and on Python (real mp.Pool, in a fresh subprocess per the repo's fork/rayon rule): both succeed, failed_reads == 0, absent reads > 50% of the BAM. Reference-anchored, deliberately — under basecall only 9 of 20 reads submit, so the Rust path could never reach the threshold from this fixture
• Genuine failures still caught: a real per-read exception behind a filtered POD5 lands in failed_reads (2) and not in the absent bucket (18); at the dispatcher, ([], 0, 3, 1) (one absent, two found and yielding nothing) still counts 2 failed, while ([], 0, 3, 3) counts none; at the run level, 8 failures behind 10 absent reads of 20 still raises, and the attempted-reads denominator is pinned by a case that passes on the old denominator and fails on the new one

🤖 Generated with Claude Code