Repository navigation
Add blosc2.set_prefetch(): read-ahead for on-disk arrays - #751
Merged
Merged
Conversation
NDArray.get_slice_numpy() decompressed with the GIL held, so threads reading separate arrays, or separate handles of one file, ran one at a time; on NFS this included the wait for the server. Release the GIL around b2nd_get_slice_cbuffer(). The GIL was also what kept two threads from using one super-chunk at the same time. Replace the per-NDArray read_lock with one reentrant lock per super-chunk, shared by the SChunk wrapper, the NDArray, its views and vlmeta, and hold it in every entry point into C. Keep the GIL when a Python postfilter, codec or filter could run during the read, and on wasm32. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SwMZj7Wi7Sc8HtQpELyNXB
Hold the operands' handle locks while the miniexpr prefilter of a lazy expression reads their super-chunks from C. Count the reads that run without the GIL, so that exit and the registration of Python codecs and filters wait for them, and free handle locks that other threads held at a fork in the child. Lock the metalayer counters too. Skip the new tests on wasm32, shorten them, retry the GIL check on busy machines, and add tests for lazy expressions and fork. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SwMZj7Wi7Sc8HtQpELyNXB
Offset and size of a chunk in the file of a super-chunk stored as a contiguous frame on disk, from the frame header and the chunk index, without reading the chunk (C-Blosc2's blosc2_schunk_get_chunk_range()). Callers that know which chunks they will need can read them, or ask the operating system to read them ahead, themselves. The size is 0 for chunks stored as special values; a RuntimeError for chunks that do not exist and super-chunks without a frame. Co-Authored-By: Claude Opus 5.5 <[email protected]>
blosc2.set_prefetch(nthreads, max_ahead=0, piece_size=0) turns on C-Blosc2's read-ahead (blosc2_set_prefetch(); BLOSC_PREFETCH=<nthreads> in the environment does the same at import): slices of arrays and super-chunks on disk (a[...], NDArray.slice(), orthogonal selections, SChunk slices) read the chunks they need, and that are not in the page cache, with a few background threads in large pieces, and decompress each chunk as soon as it is in. Of chunks that a[...] or slice() covers only in part, just the blocks they need are read. Off by default; a no-op on Windows. The docstring says which settings to use where (network file systems, local disks, many reader processes); blosc2.open() points to it for files on network storage, and a new optimization tip (with its benchmark script) shows the effect on cold reads: 1.7x for 40 cold patch reads on a local NVMe disk, 3-5x for 3D patches on NFS. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Pyodide exposes os.fork as a stub that raises ENOSYS. Keep the read-ahead correctness tests enabled on WASM, but run the fork lifecycle test only where native fork is supported. Co-Authored-By: Codex <[email protected]>
FabianIsensee
force-pushed
the
read-ahead
branch
from
November 4, 2026 13:30
889c222 to
5def38a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi Francesc,
this is the Python side of Blosc/c-blosc2#842:
blosc2.set_prefetch(nthreads, max_ahead=0, piece_size=0), orBLOSC_PREFETCH=<nthreads>in the environment. Slices of arrays and super-chunks on disk (a[...],NDArray.slice(), orthogonal selections,SChunkslices) then read the chunks they need, and that are not in the page cache, with a few background threads in large pieces. Off by default; a no-op on Windows.Depends on:
SChunk.get_chunk_range()): this branch is stacked on it, and through it on Release the GIL in NDArray slice reads #743. Until those are merged, this PR also shows their commits. The new commits are "Add blosc2.set_prefetch(): read-ahead for on-disk arrays" and "Restrict the prefetch fork test to native platforms".It is independent of #746-#749 and can be merged in any order with them; the code is placed so that they do not conflict.
What is in it:
blosc2.open()for files on network storagebench/optim_tips/tip_16_prefetch.pyand its plot: 1.7x for 40 cold 128^3 sub-volume reads of a synthetic 4x256^3 float32 array on a local NVMe disk; 2-9x on NFS for real 3D images (numbers in #842)Tests:
tests/test_prefetch.py(slices, partly covered chunks, settings, fork).pytest tests: 9471 passed, both with read-ahead off and withBLOSC_PREFETCH=4; 2 failed (theb2viewhttp tests, as on main). The doctest and ruff pass.AI disclosure: I used Claude Code (Anthropic) for the investigation, the code, the tests and a first draft of this description, and Claude Code opened this PR on my behalf. I reviewed and tested everything.
Best,
Fabian
🤖 Generated with Claude Code