Repository navigation
compression: size the gzip output buffer once and release its slack - #291
Merged
Merged
Conversation
GzipProvider::compress_inner allocated its output at the input length and then, on the first compress_vec call, reserved the buffer's own capacity again, doubling it to about twice the uncompressed input. That Vec became the frame's Bytes with the capacity intact, so every queued gzip response frame pinned roughly twice its uncompressed size until HTTP/2 wrote it; the zstd path sized its buffer once from compress_bound and pinned about one times. The buffer is now sized once from zlib-rs's conservative deflate bound, about 1.14 times the input plus the 18-byte gzip container, which holds at every level flate2 exposes; the tighter figure classic zlib quotes for its default parameters does not, since the fast level this provider defaults to grows incompressible input by a few percent. The loop only reserves if that bound was somehow exceeded, and with reserve_exact so it cannot double back. Both the gzip and zstd paths hand the buffer over through compressed_output, which shrinks it when more than half is unused. A queued compressible frame now pins at most about twice its compressed size and an incompressible one about its input size. Three tests observe the retained capacity through the existing backing_capacity helper: 1 MiB of zeros through gzip and zstd, and 1 MiB of LCG noise through gzip at levels 0, 1, 6 and 9 against the exact initial sizing. Signed-off-by: Iain McGinniss <[email protected]>
iainmcgin
marked this pull request as ready for review
September 11, 2026 22:24
Signed-off-by: Iain McGinniss <[email protected]>
Signed-off-by: Iain McGinniss <[email protected]>
rpb-ant
approved these changes
Sep 22, 2026
connyay
pushed a commit
to connyay/connect-rust
that referenced
this pull request
Nov 2, 2026
…nking it (connectrpc#323) connectrpc#291 shrinks a compressed frame's working buffer with `shrink_to_fit` when it is more than half empty. For a large input that buffer is above glibc's mmap threshold, so the shrink is an `mremap`, and freeing the few-KB chunk it leaves never raises glibc's dynamic threshold. Every later compression of a large input then maps a fresh buffer, and flate2's Rust backends zero-fill its whole spare capacity, faulting in every page. On a c7i.metal-24xl (turbo off, pinned), gzip of a 1 MiB compressible input: | | per compress | page faults per call | |---|---:|---:| | v0.9.0 | 553 µs | 0 | | `main` (connectrpc#291, `shrink_to_fit`) | 957 µs | 297 | | this PR (copy out) | 497 µs | 0 | This copies the compressed bytes into a buffer of their own and drops the working buffer, in the helper gzip and zstd share, which raises the threshold (up to glibc's 32 MiB cap) so later calls reuse heap memory. A queued frame now retains exactly its compressed size; the retention tests assert that. They would also pass with `shrink_to_fit`, since the allocator effect is not testable in-process. Signed-off-by: Iain McGinniss <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every queued compressed response frame keeps its encoder's whole output buffer alive: the gzip and zstd paths size that
Vecto the input length or a little over (compress_boundfor zstd), and it becomes the frame'sByteswith the capacity intact, so a compressible frame pins about its uncompressed size until HTTP/2 writes it. The gzip sizing on main (#300) is also too small for incompressible input at this provider's default fast level, which grows such input by a few percent: a 1.1 MB frame of noise ends up in a 2.1 MB buffer after the reserve doubles it.Both paths now hand the buffer over through
compressed_output, which gives back the unused capacity when the compressed frame is under half of it, and gzip sizes its buffer from zlib-rs's conservative deflate bound (about 1.14× the input). A queued compressible frame now pins at most about twice its compressed size; an incompressible one, about its input size.