[FabianIsensee]

Hi Francesc,

Depends on: Blosc/c-blosc2#850 (blosc2_schunk_update_buffer()/blosc2_schunk_insert_buffer()). The two TEMP commits bundle that branch until a c-blosc2 release has it; then they become a version pin (and BLOSC2_MIN_VERSION goes up if 3.5.1 ships without it).

Please check before merging: with set_releasegil(True), two handles of one file written without locking=True can now corrupt it. Details in the section below; if you'd rather keep the old behaviour, there is an alternative.

SChunk.update_data(), append_data(), insert_data(), SChunk(data=...) and set_slice() compress every chunk into a new malloc(nbytes + 32), shrink it, and hand it to c-blosc2. blosc2.asarray() above 16 MB, lazy-expression compute(), _copy_chunkwise() and linalg go through update_data(). With glibc each of those buffers is a fresh mmap(), faulted in while the compression threads write into it.

In 4.14.1 this was mostly hidden after reads. The read path then freed a chunk-sized scratch per chunk, which raised glibc's mmap threshold, so later compressions used warm heap memory. The slice reads no longer use that scratch (c-blosc2 1aae9481), so on main, compressing to a file right after reads is up to 1.4x slower than in 4.14.1 at 28 threads. In a fresh process main is no slower than 4.14.1.

With this PR these five methods call c-blosc2's blosc2_schunk_{append,insert,update}_buffer() through one helper, _store_buffer(). Frames are then compressed into a buffer that the thread reuses and are written without a copy. The copy argument of update_data()/insert_data() has no effect any more and is documented as such: the compressed chunk never leaves c-blosc2.

set_releasegil(True) and several writers on one file: please decide

With set_releasegil(True) the GIL used to be released only around the compression, and the frame write ran with the GIL held. Now it is released around the whole call, so the frame write runs without the GIL too (it runs no Python).

The consequence: two handles of the same file, opened without locking=True and written from two threads at the same time, can now corrupt the file. In a review we ran two threads appending 40 chunks each through two such handles, 8 times per build: main kept all 81 chunks every time, because the GIL happened to serialise the writes; this branch failed every run, and in 3 of the 8 the file could no longer be opened. With locking=True, both are fine. The sharing guide already says that this case is not supported: "Two writers mutating the same container without locking is not supported and can corrupt it." That is why I left it like this.

If you would rather keep the old behaviour, the alternative is to hold the GIL for the whole call when writing to an on-disk frame opened without locking. The price: threads that write different files with set_releasegil(True) would no longer compress in parallel. Tell me which you prefer and I will change it.

Other changes in behaviour

• Buffers larger than BLOSC2_MAX_BUFFERSIZE raise ValueError; before, their size was silently cut to 32 bits.
• The input buffer is released on every error path. set_slice() with too little data, and update_data() with an invalid argument, used to leak it.
• The RuntimeError messages of these methods are reworded.

Benchmarks

Ryzen 9 7950X, 28 threads, float32 array of shape (64, 512, 512) (67 MB, default ZSTD 5 + SHUFFLE, 8 chunks of 8 MiB). Each configuration ran in a fresh process; the numbers are the median of 15 calls over 3 alternated rounds. "After reads" means the process read an array before.

	4.14.1	main	this PR
append_data() x 8, on disk	34.8 ms	28.0 ms	14.6 ms
same, after reads	20.5 ms	29.2 ms	15.2 ms
SChunk(data=...), on disk	35.3 ms	26.1 ms	14.3 ms
SChunk(data=...), contiguous in memory	29.5 ms	26.3 ms	13.5 ms
blosc2.asarray(), on disk	35.7 ms	29.6 ms	23.9 ms
same, after reads	28.2 ms	30.0 ms	22.8 ms
blosc2.asarray(), contiguous in memory	29.4 ms	28.2 ms	22.6 ms
(a + 1).compute(urlpath=...), after reads	31.7 ms	26.0 ms	23.5 ms

schunk[:] = data within the super-chunk goes through c-blosc2's blosc2_schunk_set_slice_buffer(), which the C PR changes as well: 11.3 -> 9.0 ms on disk, 10.1 -> 7.3 ms contiguous in memory.

Plain in-memory arrays (the default) are unchanged: the super-chunk keeps each chunk, so it gets its own allocation as before. In 4.14.1 they compress faster after reads only when glibc happens to keep its heap: identical runs gave either 6.4 or 15 ms for the 8 appends. Copying the chunk out of a reused buffer into an exact-size allocation would make that reliable for append_data(), but it made blosc2.asarray() and compute() in memory 12-44% slower, so it is not in this PR.

Tests

tests/test_schunk_data_writes.py (27 tests), on the 4 storages:

• appends, inserts in the middle and at the end, and three rounds of updates of every chunk, with the GIL held and released;
• a prefilter that must see the right chunk number on append/insert/update (GIL held only: Python prefilters crash with set_releasegil(True) on main; a fix for that comes in a separate PR);
• SChunk(data=...) plus a set_slice() that grows the last chunk and appends, with the GIL held and released;
• failed updates and inserts changing nothing;
• an insert at the new end of a file that another handle appended to, with and without locking;
• 4 Python threads writing frames in memory and on disk at the same time (GIL released; skipped on wasm32, which has no threads).

Full suite: 10661 passed; the 2 b2view HTTP tests fail without aiohttp, as on main. Fork CI: Linux x86 (Python 3.12, 3.14, NumPy 1.26), Linux aarch64, Windows and WASM pass; macOS did not get a runner on the fork for the last push (it passed on the first one).

AI disclosure: I used Claude Code (Anthropic) for the code, the tests and a first draft of this description, and Claude Code opened this PR on my behalf. I reviewed and tested everything.

Best,
Fabian

🤖 Generated with Claude Code

[FrancescAlted]

LGTM.

[FrancescAlted]

My agent struggled a bit when merging this one. Please give the final merged version another look. Thanks!

[FabianIsensee]

Thanks for merging! I checked the squashed commit against my branch. The conflicts with #766 are resolved as intended: _store_buffer() decides on the GIL with _can_compress_without_gil(), and errors go through _raise_callback_error(). Nothing else changed outside the conflicts, the TEMP pins are gone, and the bundled C-Blosc2 (fa29efc0) has Blosc/c-blosc2#850. One resolution dropped a try/finally in SChunk(data=...), so a ValueError from _store_buffer() leaves the input buffer exported; #777 restores it and also runs the prefilter test with the GIL released, now that #766 allows it.