[FabianIsensee]

Hi,

this is the python-blosc2 companion to Blosc/c-blosc2#821. As described there, we use blosc2 in nnU-Net and ran into very slow writes on our network drives. I investigated this together with Claude Opus 5.5. With the c-blosc2 fix in place, most of the remaining cost of arr[...] = x on NFS comes from python-blosc2's index bookkeeping, which this PR addresses.

Problem

NDArray.__setitem__ calls indexing.mark_indexes_stale() after every write, also for arrays that have no index. For persistent arrays this loads the index store, which does a Path.resolve() plus a stat() for every cached persistent path. On NFS a stat() of a file with buffered writes first pushes those writes to the server. Right now nobody notices because c-blosc2 flushes on every chunk update anyway, but with Blosc/c-blosc2#821 these stats become the main cost of a setitem.

Fix

mark_indexes_stale() returns early if the array is persistent, no store cached in the process has indexes and the array's vlmeta has no blosc2_indexes entry. In all other cases the existing code runs unchanged. resize() benefits too since it calls the same function.

Benchmarks

setitem, one 786 KB chunk per call, clevel 0, on NFS with the c-blosc2 fix: ~8 ms -> 1.0 ms per chunk. Without the c-blosc2 fix there is no measurable difference.

This one is probably mostly NFS-specific (it's the flush-before-stat that hurts), although on other network filesystems a stat() can still cost a round trip to the server. Not tested.

Gotchas

• the early return only applies as long as no array in the process has indexes. After that _purge_stale_persistent_caches() still stats every cached path on each write, so the cost grows with the number of arrays a process has touched. Might be worth a separate look

Tests

• test_setitem_without_indexes_skips_index_store: setitem on an unindexed persistent array must not load the index store (fails on main)
• test_setitem_marks_on_disk_index_stale_with_cold_caches: with all in-process caches cleared, setitem still marks an index that is only known from the file's vlmeta as stale
• full test suite passes, except the 2 b2view HTTP tests that also fail on main (missing optional module)

AI disclosure: I used Claude Code (Anthropic) for the investigation, the code, the tests and a first draft of this description.

Best,
Fabian

[FabianIsensee]

Heads-up: I just opened #744 (python-blosc2 performance and bug fixes, companion of Blosc/c-blosc2#833). It contains this PR's commits unchanged, since the work there was done on top of it. You can merge this one first (then #744's diff gets smaller) or close it when merging #744. Whatever is easier for you.

[FrancescAlted]

Please, rebase.

[FabianIsensee]

done :-)