Hybrid pangenome - #33
Merged
Merged
Conversation
Combine the human pangenome with sample short-read fastq and aligned long-read data for variant calling: k-mer counting across both read sets, personalized diploid graph construction with vg, graph updates with PGHapUpdateAlgo using the long-read alignments and LongReadSV calls, re-alignment of extracted short reads to the personalized pangenome with liftover, short-read dedup and metrics, and PangenomeSV + DNAscope calling with LR readgroup attributes. The dedup and metrics job builders move from SentieonPangenome to BasePangenome so both pangenome pipelines share them; the emitted dnascope-pangenome commands are unchanged. Co-Authored-By: Claude Fable 5 <[email protected]>
…ngenome Aligned short-read BAM/CRAM input via `--sr_aln`, mutually exclusive with fastq input. Aligned input is assumed deduplicated: no dedup or metrics jobs run, read extraction and k-mer counting share a single pass over the input (driver ReadWriter | pgutil extract | kmc, with the long-read FASTA streams concatenated), and the lifted alignment is written directly to the output. The calling stages rewrite the input readgroups with `--replace_rg` to set LR:0. Unaligned long-read (uBAM/uCRAM) input via `--lr_align_input` and `--lr_input_ref`. Each input is realigned with minimap2 using the new `minimap2_lr.model` bundle member; k-mer counting reads the original input files in parallel with the realignment. Also fix `cmd_samtools_fastq_minimap2` and `cmd_samtools_fastq_bwa` to decode the input file with `input_ref` rather than the alignment target reference, matching the flag's documented purpose. This changes behavior for dnascope-longread and dnascope-hybrid runs where the two references differ. Co-Authored-By: Claude Fable 5 <[email protected]>
Replace `sort -k1,1 -k2,2n` with `bedtools sort -faidx` in the LongReadSV BED generation so the regions sort correctly for references with an unusual contig order. Co-Authored-By: Claude Fable 5 <[email protected]>
Add `--pangenome_contig_prefix` (default GRCh38#0#) and pass it to PGHapUpdateAlgo, pgutil lift, and PangenomeSV so a non-default pangenome reference name uses a consistent prefix throughout. Co-Authored-By: Claude Fable 5 <[email protected]>
Reject BAM/CRAM inputs with no @rg header lines (previously the LR attribute was silently never applied, or build_dag crashed), require readgroup IDs to be unique across all inputs, and report malformed @rg header lines with the offending file instead of a traceback. check_kmc_patch now parses the k-mer count from the KMC output and fails on a zero count, catching KMC builds that read nothing from piped stdin while still exiting successfully. Co-Authored-By: Claude Fable 5 <[email protected]>
In Sentieon 202503.04 the HybridStage1 `--hap_bam` output is unsorted and `-` sends it to stdout. Send the hap BAM to stdout and pipe it to `sentieon util sort`, and move the algo's fastq output to a named fifo, `stage1_hap.fq`, read by bwa in the first-stage alignment. The haplotype job runs with zero scheduler threads so it always executes concurrently with the first-stage job reading the fifo. Requires sentieon driver 202503.04 or later. Co-Authored-By: Claude Fable 5 <[email protected]>
The two-DAG main() override performed its tmpdir cleanup inline after the last check_execution, so a failing run skipped it and leaked the directory. Wrap the body in try/finally, mirroring BasePipeline.main. Found by qualification (DEFECT_sentieon_pangenome_tmpdir_leak). Co-Authored-By: Claude Fable 5 <[email protected]>
Restore the exit-status-only probe; the piped-KMC defect is being addressed in the KMC build itself. Co-Authored-By: Claude Fable 5 <[email protected]>
Replace the hardcoded sentieon-version matrix value with
${{ vars.SENTIEON_VERSION }} in all three smoke jobs, matching the
pattern already used by the Docker workflow. Each "Install sentieon"
step now fails with an explicit error when the variable is unset,
instead of a confusing tar failure on a 404 from S3.
Co-Authored-By: Claude Opus 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a hybrid short and long-read pangenome pipeline.