Read the store once for the pseudobulk tree; preload when memory allows - #24
Merged
Merged
Conversation
The tree step read every top node's cells with a separate parallel column read after the reassignment had already read the whole matrix. A node's cells are spread over the entire store, so each zarr chunk was decoded once per node, and the concurrent reads thrashed a slow disk. The matrix is now read once, in column order; the reassignment and the tree share it, and each node's block is cut from it in memory. Loaders now preload columns unless LEGUME_AUTO_PRELOAD is 0/false/off. The preload budget defaults to half the memory available when the process first asks, shared by all preloads of the process, instead of a fixed 8 GiB (still the fallback when the system does not report it). LEGUME_PRELOAD_BUDGET_BYTES still overrides it; data over budget stays on the streaming path as before. Cargo.lock is no longer tracked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tree step reads the store once
With the pseudobulk tree on, the reassignment read the whole matrix and then the tree read every top node's cells again, one parallel column read per node. A node's cells are spread over the entire store, so each zarr chunk was decoded once per node, and the concurrent reads thrashed a slow disk.
The matrix is now read once, in column order. The reassignment and the tree share it; each node's block is cut from it in memory (
select_columns), which replaces the equivalent test fixture.Automatic preload
LEGUME_AUTO_PRELOADis0/false/off.LEGUME_PRELOAD_BUDGET_BYTESstill overrides the budget; data over budget stays on the streaming path as before.sysinfo0.38 (system feature only; 0.39 needs rustc 1.95).Other
Cargo.lockis no longer tracked.