NativeBuffer<TItem, TTracker>.Grow doubles the current length in a checked context and does not
saturate, so once a buffer passes half of the addressable maximum its next grow throws
OverflowException — however small the requested increase, and even though the requested size still
fits comfortably.
// src/Apache.Arrow/Memory/NativeBuffer.cs
// Exponential growth (2x) to amortise repeated grows
// TODO: There might be a size that's big enough to work for this case but not too big to overflow.
// We could use that instead of blindly doubling.
int newCount = Math.Max(newElementCount, checked(Length * 2));
The existing TODO on those lines describes exactly this.
For a NativeBuffer<byte, …> that means a hard ceiling near 1 GiB: at Length above
int.MaxValue / 2, checked(Length * 2) overflows, so a buffer cannot grow to 1.2 GiB even though
int.MaxValue bytes are addressable and Memory<byte> could hold them.
Impact
Any caller that grows a buffer incrementally past that point fails, and fails misleadingly: the
exception is arithmetic overflow, which reads as an internal error rather than as a size limit the
caller was approaching. There is no way for a caller to avoid it — asking for a smaller increment
does not help, because the overflow is in the doubling and not in the request.
I hit this decoding a large Parquet BYTE_ARRAY column, where the buffer grows once per page. A column
with roughly 1.5 GiB of string data — comfortably inside what a StringArray can address — failed
with OverflowException from inside the allocator rather than reading successfully.
Expected
Growth saturates at the largest addressable element count instead of overflowing, so a request that
fits succeeds. A request that genuinely cannot be addressed should still fail, as it does today, at
the byte-size calculation.
Note on the neighbouring limit
This is separate from the 2 GiB ceiling on ArrowBuffer itself (ReadOnlyMemory<byte>, int Length). Fixing this does not raise that ceiling; it just stops buffers failing at half of it.
NativeBuffer<TItem, TTracker>.Growdoubles the current length in acheckedcontext and does notsaturate, so once a buffer passes half of the addressable maximum its next grow throws
OverflowException— however small the requested increase, and even though the requested size stillfits comfortably.
The existing
TODOon those lines describes exactly this.For a
NativeBuffer<byte, …>that means a hard ceiling near 1 GiB: atLengthaboveint.MaxValue / 2,checked(Length * 2)overflows, so a buffer cannot grow to 1.2 GiB even thoughint.MaxValuebytes are addressable andMemory<byte>could hold them.Impact
Any caller that grows a buffer incrementally past that point fails, and fails misleadingly: the
exception is arithmetic overflow, which reads as an internal error rather than as a size limit the
caller was approaching. There is no way for a caller to avoid it — asking for a smaller increment
does not help, because the overflow is in the doubling and not in the request.
I hit this decoding a large Parquet BYTE_ARRAY column, where the buffer grows once per page. A column
with roughly 1.5 GiB of string data — comfortably inside what a
StringArraycan address — failedwith
OverflowExceptionfrom inside the allocator rather than reading successfully.Expected
Growth saturates at the largest addressable element count instead of overflowing, so a request that
fits succeeds. A request that genuinely cannot be addressed should still fail, as it does today, at
the byte-size calculation.
Note on the neighbouring limit
This is separate from the 2 GiB ceiling on
ArrowBufferitself (ReadOnlyMemory<byte>,int Length). Fixing this does not raise that ceiling; it just stops buffers failing at half of it.