[FabianIsensee]

Hi Francesc,

two fixes for Python callbacks: prefilters, fillers, postfilters, and codecs and filters registered from Python. Two commits on top of main (d3bca9e), independent of my other PRs. PyPI 4.14.1 behaves the same as main.

1. Segfault with set_releasegil(True). With blosc2.set_releasegil(True), every write path compresses inside with nogil:, also when a Python callback runs during the compression. The callback then creates NumPy arrays and calls Python without holding the GIL, and the interpreter crashes. This affects SChunk(data=), append_data(), insert_data(), update_data(), a growing set_slice() (schunk[a:b] = x past the end), compress2(), decompress2() and vlcompress() with a prefilter, a filler, or a Python codec or filter. It also affects to_cframe() of a sparse SChunk with a postfilter, because serializing it decompresses the chunks.

Fix: a compression releases the GIL only if none of the prefilter, codec and filters of its context is a Python one. The ids registered from Python are kept in two C arrays for that, so the check needs no Python lookups, and writes next to a Python codec they do not use still release the GIL. decompress2() cannot know the codec of a chunk beforehand, so it keeps the GIL while any codec or filter is registered from Python, as the slice reads already do (_can_release_gil()). Two small wrappers, _compress_ctx() and _decompress_ctx(), make the decision for all call sites, and _check_cparams()/_check_dparams() now use the same predicates. Unlike the slice reads, these calls are not waited for in register_codec()/register_filter(): a codec registered while such a call runs can only meet it if the call started with an id that nothing was registered under, and such a call is wrong anyway. The set_releasegil() docstring now says when the GIL is kept.

2. Exceptions turn into SystemError. An exception raised in a callback stays set while C-Blosc2 returns an error code. The wrapper then raises its own RuntimeError on top of it, and the user gets SystemError: <class 'RuntimeError'> returned a result with an exception set instead of their exception. This also happens with the GIL held. The second commit raises the pending exception right after the failing C call. Most sites now go through _check_rc(), which does that first; the error paths that free memory first call _raise_callback_error() themselves. The NDArray slice reads used to retry a failed read with the exception still set, a growing set_slice() did not check its first C call, and concat() released its locks first; all three stop now. C-Blosc2 stops at the first failing callback, so no Python code runs with the exception set. The tests check that each failing callback runs exactly once.

Performance: measured against main with interleaved runs (4 KiB and 256 KiB chunks, 1 and 4 threads, set_releasegil on and off, a Python codec registered or not): all cases are within ±2% of main, including multi-threaded writes next to a registered Python codec. The one exception is by design: decompress2() from several threads while a Python codec or filter is registered keeps the GIL, as slice reads do, and is ~3x slower in that case than on main (where it would crash if a chunk used that codec).

Tests: new tests/test_releasegil_callbacks.py, 113 cases:

• Every write path with a prefilter, a filler, a Python codec and a Python filter, with the GIL held and released. The results must be the same. On main, all 16 GIL-released cases segfault (each run in its own process); the GIL-held ones pass.
• Each callback raising KeyError on writes and on reads, for SChunks and NDArrays: the KeyError must reach the caller, the callback must have run once, and the data must be unchanged. compress2(), vlcompress(), decompress2() and decompress() with a Python codec or filter must raise the callback's own RuntimeError("Cannot apply user codec without an SChunk"). Without the second commit, 80 of the 113 cases fail with SystemError.
• test_write_releases_the_gil_next_to_python_codecs checks, like test_get_slice_releases_the_gil, that a write keeps running without the GIL while an unrelated Python codec and filter are registered.
• pytest tests: 10747 passed. The 2 failures are the b2view http tests that need aiohttp, as on main.
• Fork CI: green on Linux x86 (Python 3.12, 3.14), Linux ARM, Windows, NumPy 1.26 and WASM (macOS was not scheduled).
• ruff passes.

Not fixed here, found on the way (I can open issues):

• decompress2() of a chunk made with a Python codec or filter segfaults with more than one thread (its default), with or without the GIL, and so does decompress() after set_nthreads(n > 1): the decoder runs on C-Blosc2's worker threads. SChunks refuse that in _check_dparams(), but these functions cannot see the chunk's codec.
• blosc2_vlmeta_get() decompresses the vlmetalayers with the SChunk's dparams, postfilter included. A Python postfilter then runs on the vlmeta bytes as if they were items of its dtype, which overruns the buffer (heap corruption, also when just reading schunk.vlmeta[...]).
• to_cframe() of an SChunk with a postfilter stores the postfiltered data for every storage except an in-memory contiguous one (schunk_copy_skeleton() forces the filters when copying).
• Every vlmeta read leaks the decompressed content (get_vlmeta() never frees it).

AI disclosure: the analysis, fix and tests were done with Claude Code; I reviewed them.

Best,
Fabian

🤖 Generated with Claude Code

[FrancescAlted]

LGTM. And yes, please open tickets for discovered issues.

[FabianIsensee]

Thanks Francesc! The issues listed in the description are all in PRs now:

• decompress2()/decompress() with more than one thread on a chunk that uses a Python codec or filter: Decide threads and the GIL per chunk for Python codecs and filters #769
• blosc2_vlmeta_get() running the postfilter on the vlmeta content: Keep prefilters, postfilters and chunk cparams away from vlmetalayer content c-blosc2#857 (merged) and Refuse buffers of another typesize in Python prefilters and postfilters #773 (refuses buffers whose typesize does not match the callback's dtype)
• to_cframe() storing postfiltered data: Serialize super-chunks that have filters as stored, whatever the storage c-blosc2#858
• the leak in every vlmeta read: Free the content of vlmeta reads; fix metalayer reads of missing names #771

Other follow-ups to this PR: #772 (crash at exit while daemon threads still run without the GIL), #774 (C prefilters that could need the GIL) and #775 (span reads of chunks with a Python codec or filter).

The BLOSC_NTHREADS case mentioned in #769 (SChunks with a Python codec still run it on worker threads) is being worked on; a PR will follow.

I ran into a number of smaller bugs along the way. I'll collect them in one issue once the current PRs are through, so they don't get lost.