Skip to content

Add mafutils info and record block aggregates in the index header - #5

Merged
gwct merged 1 commit into
mainfrom
info-command
Sep 22, 2026
Merged

gwct merged 1 commit into
mainfrom
info-command

Conversation

@gwct

@gwct gwct commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

There was no cheap way to ask what a MAF contains. stats reads the whole file and validate only answers whether the index is trustworthy.

mafutils info prints size, compression, index freshness, block/scaffold counts and a species list. On a 42.70 GB file it takes 1.3 s (0.7 s with --sample-blocks 0), because mafutils index now records the counts in the index header and info normally reads nothing but that one line.

  • New header fields (blocks, scaffolds, ref_bases, aln_cols, seq_lines, max_seqs), appended after hash=. Written identically to both index files, since validate compares the two headers for exact equality. format stays at 2.

  • Three tiers, never scanning the MAF for counts: header only; else stream the index (~52 MB/s) with a warning naming its size and recommending a rebuild; else report what needs no index and point at mafutils index.

  • Species are sampled from the first 1,000 blocks and labelled possibly incomplete. They aren't in the index, and collecting them during indexing measured 2.06× slower. stats remains the exhaustive source.

  • Compatibility: index rows are byte-identical (verified across all 8,094,203 rows of a real index); v0.6.0 code reads a new-format index cleanly (fetch/stats/gc/validate all pass, validate VERIFIED); new code reads old indexes via the fallback. No rebuild required, it only makes info instant. Indexing cost measured at ~1.3% (4.71 s → 4.77 s on a 1.5 GB slice). A full 42.7 GB rebuild produced aggregates matching an independent awk pass exactly. 148 tests passing.

🤖 Generated with Claude Code

@gwct
gwct merged commit ffc5e1b into main Sep 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant