Skip to content

Too many edges for number of nodes #327

Closed

Description

@rbdavid

We're seeing failures of the Convergence Ratio analysis runs with the below error. After some digging, the fatal issue is that, by the time convergence ratio is calculated, the blast parquet file contains self-alignments still, resulting in convergence ratio values greater than 1 (which should not be possible).

This continues the work I initiated a couple months ago in #258

The flow needs to be:

  • Condense sequences. (ALL_BY_ALL workflow)
  • Run all-by-all BLAST calcs. (ALL_BY_ALL workflow)
  • Run blastreduce to take the top triangle of the all-by-all BLAST results. Keep self-alignments here. (ALL_BY_ALL workflow)
  • Restore from the condensed sequence set to the full sequence set. Remove self-alignments here. (ALL_BY_ALL workflow)
    • a DuckDB call happens that outputs condensed.out.
    • a restore_condensed_sequences.py call happens that outputs 1.out. This is likely where the self-alignments need to be removed.
    • a transcode_restored_blast.py call happens that outputs the 1.out.parquet file.
  • Calculate convergence ratio. (REPORTING workflow)

Activity

  1. self-assigned this
    on May 18, 2026
  2. added theissue type on May 18, 2026
  3. rbdavid commented on May 27, 2026

    @rbdavid
    ContributorAuthor

    I believe this is fixed with #329 now merged but testing with John's use case in mind needs to happen still.

    Note, in #329, the SQL query to remove self-alignments was implemented in the restore_condensed_sequences.py script (restore_condensed proccess block). In this script, a CLI argument also controls whether intra- CD-HIT cluster edges are created or not. It should be emphasized that intra-cluster edges are the most important edges (since the sequences in a CD-HIT cluster are all 100% sequence identical) in the network. If we don't include them, our degree centrality and convergence ratio metrics as well as the network topology, when visualized in cytoscape, will be incorrect. Omission of these edges may or may not have an effect on the final scientific hypotheses being developed from the SSN.

    Currently, the use of the "old method" (omitting intra-cluster edges) is turned on by default. This should be changed once the code is in a better state to fix the bug.

  4. locked and limited conversation to collaborators on Jul 17, 2026
  5. converted this issue into a discussion #407 on Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions