Repository navigation
Add SChunk.get_chunk_range() - #750
Open
FabianIsensee wants to merge 3 commits into
Open
FabianIsensee wants to merge 3 commits into
FabianIsensee wants to merge 3 commits into
Conversation
This was referenced Nov 3, 2026
NDArray.get_slice_numpy() decompressed with the GIL held, so threads reading separate arrays, or separate handles of one file, ran one at a time; on NFS this included the wait for the server. Release the GIL around b2nd_get_slice_cbuffer(). The GIL was also what kept two threads from using one super-chunk at the same time. Replace the per-NDArray read_lock with one reentrant lock per super-chunk, shared by the SChunk wrapper, the NDArray, its views and vlmeta, and hold it in every entry point into C. Keep the GIL when a Python postfilter, codec or filter could run during the read, and on wasm32. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SwMZj7Wi7Sc8HtQpELyNXB
Hold the operands' handle locks while the miniexpr prefilter of a lazy expression reads their super-chunks from C. Count the reads that run without the GIL, so that exit and the registration of Python codecs and filters wait for them, and free handle locks that other threads held at a fork in the child. Lock the metalayer counters too. Skip the new tests on wasm32, shorten them, retry the GIL check on busy machines, and add tests for lazy expressions and fork. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SwMZj7Wi7Sc8HtQpELyNXB
Offset and size of a chunk in the file of a super-chunk stored as a contiguous frame on disk, from the frame header and the chunk index, without reading the chunk (C-Blosc2's blosc2_schunk_get_chunk_range()). Callers that know which chunks they will need can read them, or ask the operating system to read them ahead, themselves. The size is 0 for chunks stored as special values; a RuntimeError for chunks that do not exist and super-chunks without a frame. Co-Authored-By: Claude Opus 5.5 <[email protected]>
FabianIsensee
force-pushed
the
chunk-ranges
branch
from
November 4, 2026 13:30
89ca129 to
0a77d14
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi Francesc,
this adds
SChunk.get_chunk_range(nchunk) -> (offset, nbytes), the Python side of Blosc/c-blosc2#841. Callers can use it to read, or prefetch, the chunks they need themselves.Depends on:
blosc2_schunk_get_chunk_range(). It is in c-blosc2 3.5.0, which main now bundles, so this branch no longer needs a TEMP commit.It is independent of #746-#749 and can be merged in any order with them.
Tests:
tests/test_chunk_range.py.pytest tests: 9459 passed, 2 failed (theb2viewhttp tests that need aiohttp, as on main).AI disclosure: I used Claude Code (Anthropic) for the code, the tests and a first draft of this description, and Claude Code opened this PR on my behalf. I reviewed and tested everything.
Best,
Fabian
🤖 Generated with Claude Code