Introduce more flexible storeChunk() syntax, use to add ADIOS2 memory selection

[franzpoeschel]

We currently don't support memory selections yet, e.g. when you have a 2D memory buffer but want to write only an arbitrary block from it. That block is then non-contiguous, but ADIOS2 has so-called memory selections to support this.

We currently don't have an API that would be able to expose this (except in Python where the native Python buffer protocol is powerful enough to express this; we currently throw an error when detecting this strides in selection are inefficient, not implemented!). Since the loadChunk()/storeChunk() API is somewhat convoluted by now anyway, I didn't want to add even further overloads.

New API Design

The new API is centered around RecordComponent::prepareLoadStore(), which returns a configuration object that can be chained with various methods before executing the actual load/store operation.

Methods return an auxiliary::DeferredComputation<T> object that encapsulates the operation without immediately executing it. The computation is only performed when get() (or operator()()) is called on the returned object:

// Returns DeferredComputation<std::shared_ptr<int>>
auto result = E_y.prepareLoadStore()
    .offset({0, 5})
    .extent({1, 5})
    .load<int>();

// Execution happens here when get() is called
auto data = result.get();

Configuration Methods

The configuration chain supports:

• offset(Offset) - Set the offset within the dataset (optional)
• extent(Extent) - Set the extent within the dataset (optional)
• memorySelection(MemorySelection) - Set memory selection for non-contiguous buffers (only available after withSharedPtr(), withRawPtr(), or withContiguousContainer())
• unsafeNoAutomaticFlush(): The returned DeferredComputation object will be a no-op object. Buffers will still be returned, but they might not yet be written/read until manually flushing.

Buffer Specification Methods

If not specifying a buffer, the following allocating load/store operations are available:

• storeSpan<T>() - Returns DynamicMemoryView<T> (a span-like view)
• load<T>() - Returns DeferredComputation<std::shared_ptr<T>>
• loadVariant() - Returns DeferredComputation<variant> for type-erased loading
• load<T>(policy) - Returns std::shared_ptr<T>>

Alternatively, you may specify a buffer using one of:

• withSharedPtr(std::shared_ptr<T>) - For shared pointer managed buffers
• withUniquePtr(UniquePtrWithLambda<T>) - For unique pointer with custom deleter
• withRawPtr(T*) - For raw pointer buffers
• withContiguousContainer(Container&) - For contiguous containers (automatically deduces size if extent is not set)

The following operations are only available after specifying a buffer. Template parameters are no longer needed as the type is given through the buffer type:

• store() / load() - Returns DeferredComputation<void>

(Load operations are only available after specifying a writable buffer, i.e. raw/shared pointer or contiguous container of non-const type)

ADIOS2 Memory Selection Support

This PR adds support for ADIOS2 memory selections, allowing writing from non-contiguous memory blocks:

std::shared_ptr<int> data;
E_y.prepareLoadStore()
    .offset({0, 5})
    .extent({1, 5})
    .withSharedPtr(data)
    .memorySelection({{1, 1}, {5, 5}})  // Write block from within the buffer
    .store();

Note: Memory selection support depends on ADIOS2 version. For ADIOS2 versions that cannot reset memory selections (PR ornladios/ADIOS2#4169), a warning is displayed. The API gracefully handles both cases.

Migration from Old API

The old loadChunk() and storeChunk() methods are now fully implemented using the new API internally, providing full backward compatibility while benefiting from the new implementation. Existing code continues to work without changes.

Examples of Old to New Migration

Old:

std::shared_ptr<int[]> data;
E_y.loadChunk(data, {0, 5}, {1, 5});

New:

auto result = E_y.prepareLoadStore()
    .offset({0, 5})
    .extent({1, 5})
    .withSharedPtr(data)
    .load<int>();
// Later: result.get();

Old:

std::vector<int> data(100);
E_y.storeChunk(data, {0, 5});

New:

E_y.prepareLoadStore()
    .offset({0, 5})
    .extent({1, 5})
    .withContiguousContainer(data)
    .store();

Design Considerations & Open Questions

Memory Selection Limitations

• HDF5 backend throws an error when memory selection is specified (not supported by HDF5)
• ADIOS2 requires version with memory selection reset support for clean reuse of variables
• Memory selection only available after buffer specification (not on bare prepareLoadStore())

Files Changed

• include/openPMD/LoadStoreChunk.hpp - New header with the main API
• include/openPMD/LoadStoreChunk.tpp - Template implementations
• include/openPMD/auxiliary/Future.hpp - New DeferredComputation wrapper
• include/openPMD/Dataset.hpp - Added MemorySelection struct
• include/openPMD/RecordComponent.hpp - Integration of new API
• src/LoadStoreChunk.cpp - Non-template implementations
• src/auxiliary/Future.cpp - DeferredComputation implementation
• All IO backends (ADIOS2, HDF5, etc.) - Memory selection support

TODO

• What about Python? The buffer protocol in Python may already support this; consider keeping Python API unchanged for now
• Decide on deferred vs. immediate API style
• Consider introducing .noop_future() option to disable future return types for cases where you don't need to track completion
• Further binary size optimization

Related

• ADIOS2 PR ornladios/ADIOS2#4169: Memory selection reset support

TODO:

• look at hpx future for inspiration

[ax3l]

Codex:

[P2] Apply the deleter to its argument after resetting a converted pointer

Capturing original_ptr makes the deleter ignore the pointer currently owned by the resulting UniquePtrWithLambda. This class publicly inherits std::unique_ptr, so auto q = std::move(p).static_cast_<void>(); q.reset(replacement); q.reset(); is valid usage: the first reset should delete the original allocation and the second should delete the replacement. A tracking-deleter reproducer instead receives the original address twice. With a normal deleting deleter this becomes a double free, while the replacement leaks. Please retain deletion of the supplied pointer, with the necessary type/const conversion, instead of binding deletion permanently to the original address.

[franzpoeschel]

The only safe thing to do here is to detect this condition and throw (the alternative would be to assume that the new pointer has the same provenance and can hence be reinterpret_casted back to the original type for applying the original deleter…).
Note that .swap() remains safe as that also swaps deleters.

[ax3l]

Claude:

Source-compatibility regression: the std::shared_ptr<T[]> overloads of loadChunk() / storeChunk() were removed. Calls without explicit template arguments still work (T deduces to double[]), but calls with explicit template arguments no longer compile:

std::shared_ptr<double[]> p(new double[10]);
rc.loadChunk<double>(p, {0}, {10});  // dev: OK, PR: no matching function
rc.storeChunk<double>(p, {0}, {10}); // dev: OK, PR: no matching function

(Verified with GCC 13 against the dev and PR headers.) Suggest keeping them as thin, header-only forwarding overloads. No extra explicit instantiations are needed, partial ordering still selects them for the implicit call, and inside the forwarder loadChunk<T[]> SFINAE-drops the T[][] candidate. With this and the storeChunk counterpart below, explicit, implicit and const[] calls compile and link again:

Suggested change

	template <typename T>
	void loadChunk(std::shared_ptr<T> data, Offset offset, Extent extent);
	template <typename T>
	void loadChunk(std::shared_ptr<T> data, Offset offset, Extent extent);
	/** Load a chunk of data into pre-allocated memory, array version.
	*
	* Kept for source compatibility with explicit template arguments,
	* e.g. loadChunk<double>(std::shared_ptr<double[]>, ...).
	*/
	template <typename T>
	void loadChunk(std::shared_ptr<T[]> data, Offset offset, Extent extent)
	{
	loadChunk<T[]>(std::move(data), std::move(offset), std::move(extent));
	}

[franzpoeschel]

Will not fix. These overloads were only needed because the array types required special handling previously, and don't need special handling any longer with this PR. Assume the following snippet:

std::shared_ptr<double[]> p(new double[10]);
rc.loadChunk(p, {0}, {10});

We previously needed a special overload to handle this, i.e. this would resolve to:

std::shared_ptr<double[]> p(new double[10]);
rc.loadChunk<double>(p, {0}, {10});

This is no longer needed, the base template works for this use case:

std::shared_ptr<double[]> p(new double[10]);
rc.loadChunk<double[]>(p, {0}, {10}); // note the different type inside <>

Claude recognized this changed template resolution as an API break (which it technically is, if specifying template arguments explicitly, the old usage will fail.) Template arguments never needed to be explicitly specified here.

[ax3l]

Claude:

Same source-compatibility regression for storeChunk<double>(std::shared_ptr<double[]>, ...) (see the loadChunk comment above):

Suggested change

	template <typename T>
	void storeChunk(std::shared_ptr<T> data, Offset offset, Extent extent);
	template <typename T>
	void storeChunk(std::shared_ptr<T> data, Offset offset, Extent extent);
	/** Store a chunk of data from a chunk of memory, array version.
	*
	* Kept for source compatibility with explicit template arguments,
	* e.g. storeChunk<double>(std::shared_ptr<double[]>, ...).
	*/
	template <typename T>
	void storeChunk(std::shared_ptr<T[]> data, Offset offset, Extent extent)
	{
	storeChunk<T[]>(std::move(data), std::move(offset), std::move(extent));
	}

[franzpoeschel]

see #1620 (comment)