Skip to content

Drop file records with empty sequences and report skipped rows - #54

Merged
erohinaelena merged 6 commits into
mainfrom
feat/bare-set-drop-failed-records
Oct 14, 2026
Merged

erohinaelena merged 6 commits into
mainfrom
feat/bare-set-drop-failed-records

Conversation

@erohinaelena

@erohinaelena erohinaelena commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

A directly imported set now keeps only records annotated on every mapped chain. Records with a chain that has no sequence or could not be numbered are left out of the dataset and still counted in the import statistics.

The prerun id check now also counts, in the same file read, rows without a sequence on a mapped chain and verbatim repeated rows. The settings panel shows each as its own info message with example ids, and waits for both answers before clearing the validating message. The id-collision verdict is retained while a sequence edit re-runs the check.

RetriggerConfidence Score: 5/5

The PR appears safe to merge; no new actionable issue was established.

Summary

The PR filters directly imported records to those annotated on every mapped chain, reports incomplete and repeated rows before the run, and keeps annotation outcomes in import statistics rather than a dataset status column.

  • BareSetMapping: The selected identity and chain-sequence columns; those selections now jointly key the collision check and row report.
  • RowReport: Counts and example identifiers for rows missing a mapped sequence or collapsed as repeats; the model now parses and exposes it.
  • Annotation status: The per-chain outcome of region annotation; it now determines whether a record is retained and remains available through import statistics.
  • Synthetic abundance: A value of one per retained record; its displayed label and unit now describe variants rather than UMIs.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Selected file and column mapping] --> B[Prerun collision check and row report]
  A --> C[Import record TSV]
  C --> D[Annotate mapped chains]
  D --> E[Retained records and regions]
  D --> F[Per-chain import statistics]
  E --> G[Dataset columns and abundance]
Loading

Reviews (2) · Last reviewed commit: "Fix row report edge cases, drop status c..."

Comment thread model/src/index.ts Resolved
@erohinaelena

Copy link
Copy Markdown
Contributor Author

@greptileai

A directly imported set now keeps only records annotated on every
mapped chain. Records with a chain that has no sequence or could not be
numbered are left out of the dataset and still counted in the import
statistics.

The prerun id check now also counts, in the same file read, rows
without a sequence on a mapped chain and verbatim repeated rows. The
settings panel shows each as its own info message with example ids,
and waits for both answers before clearing the validating message.
The id-collision verdict is retained while a sequence edit re-runs the
check.
- Exclude rows missing a mapped sequence from the collision check, and
  keep the complete row when an id also has an incomplete one. The
  collision key now includes the sequence columns.
- Report a header-only file as zero rows instead of leaving the panel
  waiting.
- Parse the prerun report TSVs with quoted fields, so ids holding a
  tab, newline or quote are named correctly.
- Stop emitting the region annotation status column; the counts stay
  in the import statistics.
- Label the synthetic abundance "Number of Variants" with unit
  "variants"; the column name is unchanged.
- Say "No sequence" without "for at least one chain" when only one
  chain is mapped.
A, C, G, T and N are amino-acid letters too, so a DNA column passed the
amino-acid profile check and was offered as a chain; every record then
failed region annotation. The column profile now reports nucleotide-alphabet
columns and the chain slot shows a warning when mapped to one.
@erohinaelena
erohinaelena force-pushed the feat/bare-set-drop-failed-records branch from 77b27e2 to fa0a8f3 Compare October 1, 2026 17:29
pt.col(mapping.identity).count().alias(COUNT_COLUMN)
).
filter(pt.col(COUNT_COLUMN).gt(pt.lit(1))).
select(pt.col(mapping.identity).alias(specs.labelColumn))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rows with an empty id get past this check, because count() skips nulls and the no-id group counts 0. The import then merges them all into one NA record without any warning. Suggest handling a missing id the way this PR handles a missing sequence: treat an empty or whitespace-only id as missing here, filter those rows out in import-bare-set.tpl.tengo instead of fillNull("NA"), and report the count in the "Rows without a sequence" banner.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Comment thread ui/src/pages/messages.ts
if (!column || !(nucleotideColumns ?? []).includes(column)) return "";
return (
`"${column}" holds only A, C, G, T or N, so it looks like a nucleotide sequence. ` +
`Chains are numbered as amino-acid sequences: records from this column fail region ` +

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The message says "A, C, G, T or N", but NT_RE (column-profile/src/main.py:43) also matches U. Add U here, or drop it from the regex.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Comment thread software/region-annotation/src/main.py Outdated
)
print(f"Wrote {len(rows)} records to {args.out_tsv}")
dropped = seen - len(rows)
print(f"Wrote {len(rows)} records to {args.out_tsv}; left out {dropped} not annotated on every chain")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dropped-record count goes only to the log; the stats are per chain and overlap, so nothing shows how many records were imported versus dropped. Suggest writing seen and dropped to the stats as per-record counts, like AIRR's "Rows Read" and its non-overlapping discard counters.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, changed the list of columns in stats to "Rows read" + discarded by reasons

…ws like AIRR

- Treat an empty or whitespace-only id as missing: filter those rows in the import
  instead of merging them into one "NA" record, and report them in the row banner.
- Drop records missing a sequence in the import pass, so annotation never numbers a
  chain of a row that is discarded anyway.
- Stats: emit AIRR's Rows Read per sample with non-overlapping discard counters
  (no id, repeated id, no sequence, failed to annotate); remove the per-chain
  Annotated / Not applicable / Failed counts, keep "Numbered as the other chain"
  counted over imported records only.
- Banner wording: "Empty sequence", rows without an id counted, not named.
- Nucleotide column warning lists U, which NT_RE matches.
- Group the row-report alerts with a tighter gap.
@erohinaelena
erohinaelena added this pull request to the merge queue Oct 14, 2026
Merged via the queue into main with commit 6f8b35a Oct 14, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants