Skip to content

Read the store once for the pseudobulk tree; preload when memory allows - #24

Merged
YPARK merged 1 commit into
mainfrom
read-once-auto-preload
Oct 2, 2026
Merged

YPARK merged 1 commit into
mainfrom
read-once-auto-preload

Conversation

@YPARK

@YPARK YPARK commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Tree step reads the store once

With the pseudobulk tree on, the reassignment read the whole matrix and then the tree read every top node's cells again, one parallel column read per node. A node's cells are spread over the entire store, so each zarr chunk was decoded once per node, and the concurrent reads thrashed a slow disk.

The matrix is now read once, in column order. The reassignment and the tree share it; each node's block is cut from it in memory (select_columns), which replaces the equivalent test fixture.

Automatic preload

  • Loaders preload columns unless LEGUME_AUTO_PRELOAD is 0/false/off.
  • The preload budget defaults to half the memory available when the process first asks, shared by every preload of the process. A fixed 8 GiB remains the fallback when the system does not report memory.
  • LEGUME_PRELOAD_BUDGET_BYTES still overrides the budget; data over budget stays on the streaming path as before.
  • New dependency: sysinfo 0.38 (system feature only; 0.39 needs rustc 1.95).

Other

Cargo.lock is no longer tracked.

The tree step read every top node's cells with a separate parallel
column read after the reassignment had already read the whole matrix.
A node's cells are spread over the entire store, so each zarr chunk was
decoded once per node, and the concurrent reads thrashed a slow disk.
The matrix is now read once, in column order; the reassignment and the
tree share it, and each node's block is cut from it in memory.

Loaders now preload columns unless LEGUME_AUTO_PRELOAD is 0/false/off.
The preload budget defaults to half the memory available when the
process first asks, shared by all preloads of the process, instead of a
fixed 8 GiB (still the fallback when the system does not report it).
LEGUME_PRELOAD_BUDGET_BYTES still overrides it; data over budget stays
on the streaming path as before.

Cargo.lock is no longer tracked.
@YPARK
YPARK merged commit d20d622 into main Oct 2, 2026
5 checks passed
@YPARK
YPARK deleted the read-once-auto-preload branch October 2, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant