Repository navigation
Drop file records with empty sequences and report skipped rows - #54
Conversation
A directly imported set now keeps only records annotated on every mapped chain. Records with a chain that has no sequence or could not be numbered are left out of the dataset and still counted in the import statistics. The prerun id check now also counts, in the same file read, rows without a sequence on a mapped chain and verbatim repeated rows. The settings panel shows each as its own info message with example ids, and waits for both answers before clearing the validating message. The id-collision verdict is retained while a sequence edit re-runs the check.
- Exclude rows missing a mapped sequence from the collision check, and keep the complete row when an id also has an incomplete one. The collision key now includes the sequence columns. - Report a header-only file as zero rows instead of leaving the panel waiting. - Parse the prerun report TSVs with quoted fields, so ids holding a tab, newline or quote are named correctly. - Stop emitting the region annotation status column; the counts stay in the import statistics. - Label the synthetic abundance "Number of Variants" with unit "variants"; the column name is unchanged. - Say "No sequence" without "for at least one chain" when only one chain is mapped.
A, C, G, T and N are amino-acid letters too, so a DNA column passed the amino-acid profile check and was offered as a chain; every record then failed region annotation. The column profile now reports nucleotide-alphabet columns and the chain slot shows a warning when mapped to one.
77b27e2 to
fa0a8f3
Compare
| pt.col(mapping.identity).count().alias(COUNT_COLUMN) | ||
| ). | ||
| filter(pt.col(COUNT_COLUMN).gt(pt.lit(1))). | ||
| select(pt.col(mapping.identity).alias(specs.labelColumn)) |
There was a problem hiding this comment.
Rows with an empty id get past this check, because count() skips nulls and the no-id group counts 0. The import then merges them all into one NA record without any warning. Suggest handling a missing id the way this PR handles a missing sequence: treat an empty or whitespace-only id as missing here, filter those rows out in import-bare-set.tpl.tengo instead of fillNull("NA"), and report the count in the "Rows without a sequence" banner.
| if (!column || !(nucleotideColumns ?? []).includes(column)) return ""; | ||
| return ( | ||
| `"${column}" holds only A, C, G, T or N, so it looks like a nucleotide sequence. ` + | ||
| `Chains are numbered as amino-acid sequences: records from this column fail region ` + |
There was a problem hiding this comment.
The message says "A, C, G, T or N", but NT_RE (column-profile/src/main.py:43) also matches U. Add U here, or drop it from the regex.
| ) | ||
| print(f"Wrote {len(rows)} records to {args.out_tsv}") | ||
| dropped = seen - len(rows) | ||
| print(f"Wrote {len(rows)} records to {args.out_tsv}; left out {dropped} not annotated on every chain") |
There was a problem hiding this comment.
The dropped-record count goes only to the log; the stats are per chain and overlap, so nothing shows how many records were imported versus dropped. Suggest writing seen and dropped to the stats as per-record counts, like AIRR's "Rows Read" and its non-overlapping discard counters.
There was a problem hiding this comment.
Agree, changed the list of columns in stats to "Rows read" + discarded by reasons
…ws like AIRR - Treat an empty or whitespace-only id as missing: filter those rows in the import instead of merging them into one "NA" record, and report them in the row banner. - Drop records missing a sequence in the import pass, so annotation never numbers a chain of a row that is discarded anyway. - Stats: emit AIRR's Rows Read per sample with non-overlapping discard counters (no id, repeated id, no sequence, failed to annotate); remove the per-chain Annotated / Not applicable / Failed counts, keep "Numbered as the other chain" counted over imported records only. - Banner wording: "Empty sequence", rows without an id counted, not named. - Nucleotide column warning lists U, which NT_RE matches. - Group the row-report alerts with a tighter gap.
A directly imported set now keeps only records annotated on every mapped chain. Records with a chain that has no sequence or could not be numbered are left out of the dataset and still counted in the import statistics.
The prerun id check now also counts, in the same file read, rows without a sequence on a mapped chain and verbatim repeated rows. The settings panel shows each as its own info message with example ids, and waits for both answers before clearing the validating message. The id-collision verdict is retained while a sequence edit re-runs the check.
The PR appears safe to merge; no new actionable issue was established.
Summary
The PR filters directly imported records to those annotated on every mapped chain, reports incomplete and repeated rows before the run, and keeps annotation outcomes in import statistics rather than a dataset status column.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart LR A[Selected file and column mapping] --> B[Prerun collision check and row report] A --> C[Import record TSV] C --> D[Annotate mapped chains] D --> E[Retained records and regions] D --> F[Per-chain import statistics] E --> G[Dataset columns and abundance]Reviews (2) · Last reviewed commit: "Fix row report edge cases, drop status c..."