[FabianIsensee]

Hi there,

we use blosc2 in nnU-Net to read preprocessed data during training (the write side is in Blosc/c-blosc2#821). Our data loader reads random crops from .b2nd files with threads, and we noticed that throughput barely goes up with more threads, while processes scale fine. I looked into this together with Claude Opus 5.5. The reason is that NDArray.get_slice_numpy() (behind every arr[...]) calls b2nd_get_slice_cbuffer() with the GIL held, so threads decompress one after the other, even when each thread has its own array. On NFS it is worse because the GIL is also held while waiting for the server.

Fix

Just wrapping the call in with nogil: does not work: right now the GIL is the only thing that keeps two threads from using the same super-chunk at the same time. When I tried that I got RuntimeErrors (read + partial write on the same array), errors from arr.schunk chunk access and segfaults with Python postfilters. So:

• every super-chunk gets one reentrant lock (_handle_lock), shared by the SChunk, the NDArray, its views and vlmeta. It replaces the read_lock from Fix two NDArray thread-safety bugs blocking free-threaded support (#555) #713
• all methods that call into C hold that lock (small decorator, @_handle_locked). concat and the miniexpr prefilter of lazy expressions also take the locks of their operands, always in the same order
• get_slice_numpy releases the GIL while holding the lock. It keeps the GIL if a Python callback could run (postfilter, codecs/filters registered from Python) and on wasm32
• registering a codec/filter waits for running GIL-free reads, atexit waits up to 1s for them before blosc2_destroy(), and a forked child resets locks that were held by other threads

Threads sharing one array handle are still serialized, as before. Separate handles now read in parallel.

Benchmarks

Random 512x512x3 uint8 crops (one crop per chunk, LZ4HC), one handle per thread, MB/s, main -> this PR.

Local disk, page cache warm (Ryzen 9 7950X, 16 cores):

• 1 thread: 679 -> 802
• 4 threads: 1312 -> 3116
• 16 threads: 1625 -> 9974 (16 processes: 11513)
• random 128³ patches from a float32 image + uint8 segmentation (mmap), 16 threads: 1784 -> 8067 (processes: 7798)

NFS, cold (client cache dropped so every read goes to the server; 2x Xeon Gold 6330):

• 1 thread: 140 -> 156
• 16 threads: 179 -> 1824 (processes: 1975)
• 32 threads: 207 -> 2508 (processes: 3743)

We only tested NFS, but nothing in the fix is NFS specific. With cold data the gain comes from threads overlapping their waits for the storage instead of holding the GIL while waiting. So I would expect the same for other network filesystems (SMB, Lustre, GPFS, sshfs, ...) and slow disks. How much it helps probably depends on how many requests the client can have in flight (our NFS mount uses nconnect=16).

The lock costs +0.15 to +0.4 µs per method call (e.g. schunk.get_chunk 0.10 -> 0.25 µs) and blosc2.open() goes from 45 to 48 µs. Reading and writing actual data and lazy expressions are unchanged within noise.

Gotchas

• two handles of the same file in one process now behave like two processes: a reader racing a writer on the other handle can get the transient read errors described in the "Sharing containers across processes" guide (0.2-0.4% of reads in a hammer test, never wrong data). Before, the GIL hid this. locking=True fixes it
• Python callbacks (postfilters, prefilters, UDFs) that read other arrays take those arrays' locks. Two arrays whose callbacks read each other, used from two threads, can deadlock. The read_lock from Fix two NDArray thread-safety bugs blocking free-threaded support (#555) #713 had the same issue
• only get_slice_numpy releases the GIL. Other read paths (decompress_chunk, orthogonal/sparse selections, ...) are locked but keep the GIL. Easy to extend if you want that
• shape, nchunks & co are not locked and can be stale for a moment if the file is changed through another handle
• c-blosc2's own mutexes are not reset after a fork, so a fork in the exact moment a reader holds one can still hang the child

Tests

• new tests/ndarray/test_concurrent_access.py (12 tests, ~5s, skipped on wasm32): one handle with a reader plus a writer / chunk access / vlmeta / dparams / resize in another thread, Python postfilters, lazy expressions, concat in both orders, fork while another thread holds a lock. Most of them fail or crash with a plain with nogil:
• full test suite passes, except the 2 b2view HTTP tests that also fail on main (missing optional module)

AI disclosure: I used Claude Code (Anthropic) for the investigation, the code, the tests and a first draft of this description.

Best,
Fabian

[FabianIsensee]

Heads-up: #744 contains this PR's commits unchanged and builds on its per-handle lock (the new fast paths take _handle_lock the same way, and the C code that reads other arrays uses HandleLocks). So merging this one first makes #744 easier to review; otherwise it is superseded by #744.

[FrancescAlted]

Please, rebase.

[FabianIsensee]

done :-)