[FabianIsensee]

Hi Francesc,

Depends on: b2nd_get_slice_cbuffer_into() from the c-blosc2 PRs Blosc/c-blosc2#844 and Blosc/c-blosc2#845, both merged. The last commit bundles c-blosc2 main e3ff64f2 until the next c-blosc2 release and raises BLOSC2_MIN_VERSION to 3.5.1, since a system c-blosc2 3.5.0 lacks the function. Rebased onto main after #743 and the other merges of today.

arr[key] always returns a new array. Code that needs the data in an existing array pays for a temporary, its page faults and one more copy: the interior of a padded patch (CNN training on images smaller than the patch), one item of a batch, a reused or pinned buffer for GPU transfers. NumPy has out=, h5py has Dataset.read_direct(), zarr has get_basic_selection(..., out=); python-blosc2 only has the undocumented get_slice_numpy(), which needs a C-contiguous buffer of exactly the slice's shape.

NDArray.read_into(out, key=None, casting="same_kind") decompresses arr[key] into the NumPy array out:

• For basic keys (integers, slices with step 1, Ellipsis), out of the array's dtype, and out whose last axis is contiguous and whose other strides are whole multiples of the next one, the data goes straight into out through b2nd_get_slice_cbuffer_into(). That covers C-contiguous arrays and every box of one, also with steps in the outer axes. Nothing outside out is touched. The GIL is released meanwhile, as in the other slice reads.
• Any other layout (Fortran order, transposed, steps along the last axis), key (steps, integer or boolean arrays) or dtype (casting as in numpy.copyto) goes through arr[key] and one copy, so any writeable NumPy array works.
• out must have the selection's shape (no broadcasting). It returns out.

It is documented in the NDArray reference next to __getitem__, with examples.

Benchmarks

Ryzen 9 7950X, warm, on disk, median of 15: a (4, 132, 162, 133) float32 case into the interior of a (4, 160, 192, 160) patch (fresh per call), against patch[interior] = a[...]: 16.1 -> 12.2 ms with 1 thread, 16.1 -> 11.8 ms with 8. Into one item of a batch or into a reused buffer it is not faster than batch[j] = a[key] (0.85-1.0x with 1 thread, 1.0x with 8): there the temporary is cheap, as malloc reuses it and it stays in cache; the gain is in avoiding it where it is not. Per call it costs no more than a[key]: an 8x8x8 crop into a reused buffer takes 12.4 us, against 13.0 us for buf[...] = a[key]. With nnU-Net's data loader on BraTS (cases smaller than the patch), a prototype cut the time per sample by 22% and its CPU time by 23%.

Tests

tests/ndarray/test_read_into.py: in memory, contiguous and sparse frames on disk, and mmap; six dtypes and eleven keys into contiguous out (checked to bypass __getitem__); interiors of padded arrays, batch items, channels and strided views with nothing outside out changed; Fortran-order, transposed, last-axis-step and negative-stride out; other keys (steps, index arrays also as blosc2 arrays, masks, fields) and dtypes, casting; empty and 0-d selections; errors; shape changes made by another handle (SWMR), also with an out of the old shape; threads with one handle each; and 240 random keys into random views of larger arrays against NumPy. Full suite after the rebase: 10634 passed; the 2 b2view HTTP tests fail without aiohttp, as on main.

AI disclosure: I used Claude Code (Anthropic) for the code, the tests and a first draft of this description, and Claude Code opened this PR on my behalf. I reviewed and tested everything.

Best,
Fabian

🤖 Generated with Claude Code

[FrancescAlted]

This has been merged already in c-blosc2 repo. Feel free to update this at your convenience.

[FabianIsensee]

Thanks! Done: the pin now points to c-blosc2 main e3ff64f2 (#844, #845 and the short-rows copy), and BLOSC2_MIN_VERSION is 3.5.1, because a system c-blosc2 3.5.0 lacks b2nd_get_slice_cbuffer_into() (main reports 3.5.1.dev, which pkg-config accepts). Once the next c-blosc2 release is out, the pin can become its tag.

[FrancescAlted]

LGTM. Thanks @FabianIsensee !