examples

Example input data for Virtool workflow tests, published as a data-only container image at ghcr.io/virtool/examples.

This repository holds workflow-agnostic inputs — subtractions as FASTA, reads as FASTQ, references as index files. It deliberately does not hold per-workflow fixture trees. Tests build what they need from these inputs (Bowtie2 indexes, alignments) and cache the results themselves.

No fixture bytes are committed here

The payload only grows, so git is the wrong store for it. Bytes live in a content-addressed blob store in ghcr, keyed by sha256, and manifest.json describes what the image contains. The address of a blob is the hash of its contents, so there is no way to ask for a fixture and silently receive different bytes.

.gitignore blocks the data by directory and by extension. If you find yourself fighting it, you are about to commit something that should be ingested instead.

Layout

Inside the image, everything lives under /data:

/data/subtractions/arabidopsis_thaliana.fa.gz
/data/subtractions/fragaria_vesca.fa.gz
/data/subtractions/prunus_persica.fa.gz
/data/subtractions/solanum_lycopersicum.fa.gz

Locally, data/ mirrors that tree and is where you put a fixture before ingesting it. context/ is the assembled build context and is disposable.

Consuming the image

In a test stage only, never in a stage a production image is built from:

WORKDIR /app
COPY --from=ghcr.io/virtool/examples:1.0.0 /data/subtractions ./example/subtractions

Pin a full three-part version. Never latest — a fixture change must be a reviewable diff in the consuming repository, not a silent retag.

Versioning

• Bump the minor version when fixtures are added.
• Bump the major version when an existing fixture's bytes change or a path moves. A fixture whose bytes change invalidates every recorded expectation that depends on it, so it is a breaking change by definition.
• Within a major version, a file at a given path always has the same sha256.
• The image digest is published in the release notes, so a consumer that wants immutability can pin examples@sha256:….

manifest.json carries a version field and the release workflow refuses to publish unless it matches the git tag. That binding is what stops a rebuild of an old tag against a newer manifest from quietly publishing different content under a reference consumers have pinned.

Old releases stay rebuildable indefinitely. Blobs are addressed by their own hash, so replacing a fixture writes a new blob rather than overwriting one -- git checkout v1.0.0 && scripts/fetch.sh && docker build reproduces the original image years later.

Working on it

Requires mise (pins oras and shellcheck) and Docker.

Add a fixture:

mise exec -- scripts/ingest.sh data/subtractions/new_host.fa.gz subtractions/new_host.fa.gz
# then write the provenance block the script left blank
mise exec -- scripts/push.sh subtractions/new_host.fa.gz

Replace an existing fixture with different bytes and ingest.sh clears its provenance and note rather than carrying them over, printing the old values so anything still true can be written back deliberately. Provenance that survives a byte change reads as reviewed when nobody reviewed it, which is the one failure mode an audit artefact cannot have.

Build and check the image:

mise exec -- scripts/fetch.sh          # pull + verify every blob into context/
docker build -t ghcr.io/virtool/examples:1.0.0 .
mise exec -- scripts/verify.sh ghcr.io/virtool/examples:1.0.0

fetch.sh skips blobs already present and correct, so reruns are cheap. verify.sh checks every file in the image against the manifest by sha256 and fails on anything undeclared or missing — the truncated-versus-full index swap that motivated this repository is exactly the kind of error that reads correct by eye. It works by building a throwaway image that COPY --froms /data, so it exercises the real consumption contract at the same time.

Reading from the store is anonymous; only push.sh needs credentials, and it needs a token with write:packages:

echo "$TOKEN" | oras login ghcr.io -u <user> --password-stdin

Minimizing a genome

A unit test does not need 120 Mbp of Arabidopsis; it needs something that indexes in under a second and still behaves like a genome. scripts/minimize.sh cuts a full-size FASTA down to a test-sized one:

mise exec -- scripts/minimize.sh --report /tmp/report.json \
    data/subtractions/arabidopsis_thaliana.fa.gz \
    data/subtractions/arabidopsis_thaliana.min.fa.gz

36 MB becomes 60 kB, and bowtie2-build goes from minutes to 0.45 s.

It keeps every record and shortens each one rather than dropping records. A subtraction's record count and its per-record names are visible to Virtool and to every SAM header downstream, so changing them changes what the tests are testing; how much of each record there is, is not. Each record contributes one contiguous window, because every splice between two kept regions is a junction that exists in no genome and reads spanning it map to nothing.

The window is the first one whose ambiguous fraction is at or under --max-n. Chromosome starts are telomeric and N-heavy, which is exactly what a naive head -c would hand you — two of the five Arabidopsis chromosomes have their window pushed past the start for this reason.

Nothing is random, and gzip output is written with -n, so the same source and options always produce the same bytes. A minimized genome is a new fixture, not a copy of one: ingest it like any other, and write its provenance from the report, which records the source hash, the options, and the offset and length taken from each record — enough to reproduce the file from the full genome years from now.

The report also states the GC content of each window against the GC of the record it came from, over unambiguous bases. A short window of a chromosome is under no obligation to resemble that chromosome, and GC is the one property of a subtraction that Virtool stores; drift above five points is called out on stdout rather than left to be discovered.

Why the defaults are what they are

The defaults — 2 Mbp, organelles intact — are set for a Virtool subtraction, because that is the only thing this is used for. They are not the obvious choice, and the reason is worth keeping.

Host reads are not spread evenly over a host genome, but a proportional cut spends its budget as though they were. Measured against unmapped_otus.fq, the read set nuvs puts through its subtraction step:

subtraction	size of genome	reads aligned
committed host_N fake index	704 bp	2 (0.00%)
proportional 200 kb	0.17%	23 (0.02%)
organelles whole, 524 kb	0.44%	19.43%
the defaults: 2 Mbp, whole	1.67%	45.53%
full genome	100%	46.20%

The defaults recover 98.5% of the host signal from 1.67% of the sequence: 600 kB gzipped, and bowtie2-build finishes in 1 s against 52 s and a 178 MB index for the full genome.

The reason the gap is so wide is that the signal is concentrated in three places. Of the alignments in that 2 Mbp run, 25,719 land on AtChr2, 17,775 on the chloroplast and 1,381 on the mitochondrion; AtChr1, AtChr3 and AtChr5 between them take five reads. The top of AtChr2 carries the 45S rDNA, and the organelles are high-copy in any plant library. Proportional sharing hands four of the five chromosomes 80% of the budget to hold reads that are not there.

Note what the default does to AtChr2 specifically: at 200 kb the --max-n rule pushes its window past the N-heavy start to offset 32,922, stepping over the rDNA that carries a quarter of the host reads. At 2 Mbp the window starts at 0 and takes it. Skipping ambiguity is right in general and cost everything here, which is the argument for checking the alignment rate of a new subtraction fixture rather than assuming it.

What --whole does on a scaffold assembly

--whole is all or nothing, and the script drops it if the records it would keep intact come to more than half the budget. It reports doing so:

note: --whole 400000 would keep 98 records intact, 18339476 bases in all, which is
      more than half of the 2000000 base budget (1000000). Ignoring it: every record
      is being cut proportionally.

A chromosome-scale assembly has a handful of records under 400 kb and they are the organelles, which is the case the default is for. A scaffold-level assembly has hundreds: strawberry has 98 scaffolds under 400 kb totalling 18 MB, nine times the budget, and peach has 189 totalling 3.1 MB.

Keeping only some of them would be worse than keeping none, because length does not separate an organelle from a short scaffold. Peach's two plastids are 158,356 and 157,790 bp, and 181 of its junk scaffolds are smaller still — any rule that kept the smallest records first would fill the budget with junk and drop the plastids. So the four subtractions come out at roughly 2 Mbp each: Arabidopsis and tomato with the rule applied, peach and strawberry with it declined and every record cut proportionally.

All records survive either way. --whole changes how much of each one is left, not whether it is there, so a minimized subtraction still carries the plastid records Virtool expects.

Releasing

1. Bump .version in manifest.json.
2. Commit, then tag v<version> and push the tag.

.github/workflows/release.yaml asserts the tag matches the manifest, fetches and verifies every blob, builds, verifies the image against the manifest, and only then pushes X.Y.Z, X.Y, X and latest. Verification happens before any push because a published tag is immutable in practice — consumers pin it, so a bad release cannot be taken back.

CI on pull requests is cheap and does not build: it validates the manifest, lints the scripts, and confirms every blob named in the manifest already resolves in the store, which catches a manifest entry committed without running push.sh.

The manifest

manifest.json is the authority on what the image contains and the artefact that answers "what is our subtraction fixture" in an audit. Each entry records path, size, sha256, and provenance.

Two fields are worth explaining:

content records the sha256 and size of the decompressed stream. A gzip header carries an mtime, so recompressing identical data produces a different file hash — this is not hypothetical, nuvs and pathoscope shipped the same read set under two different hashes for exactly this reason. The content hash gives each fixture an identity that survives recompression.

provenance.upstream_verified distinguishes a claim that has been checked from one that has not. All four subtractions are currently false: they are Virtool exports from 2019-12-04, identified from their deflines but not yet reproduced from their upstream sources. prunus_persica.fa.gz never will be — two of its records were assembled inside Virtool and carry Virtool sequence IDs, so it does not correspond to any single upstream download.

Recording an unverified claim as unverified is the point. A manifest that asserted a TAIR URL nobody had checked would be worse than one that admits what it does not know.