feat: Add request body compression with optional brotli by mixalturek · Pull Request #927 · apify/apify-client-python

mixalturek · 2026-07-08T09:37:16Z

https://github.com/apify/apify-core/pull/28971 added support for brotli compression to BE.
Pros: higher compression, cons: more CPU intensive.
The brotli dependencies are defined as optional, one has to explicitly enable it and choose one. If none is available, the code falls back to gzip.
JS client doesn't compress requests that are too small, this Python client compresses just everything. No change done, I only noticed and stating it.

Closes: #942

Related PRs

codecov · 2026-07-08T09:40:08Z

Codecov Report

❌ Patch coverage is 97.11934% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.59%. Comparing base (7f61a10) to head (c810cfa).
⚠️ Report is 36 commits behind head on master.

Files with missing lines	Patch %	Lines
src/apify_client/_utils/http.py	86.36%	3 Missing ⚠️
src/apify_client/_utils/encoding.py	95.00%	2 Missing ⚠️
src/apify_client/types.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #927      +/-   ##
==========================================
+ Coverage   94.54%   94.59%   +0.04%     
==========================================
  Files          48       58      +10     
  Lines        5100     5239     +139     
==========================================
+ Hits         4822     4956     +134     
- Misses        278      283       +5

Flag	Coverage Δ
integration	`92.40% <80.65%> (-0.52%)`	⬇️
unit	`83.73% <97.11%> (+0.36%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

- apify/apify-core#28971 added support for brotli compression to BE. - Pros: higher compression, cons: more CPU intensive. - The brotli dependencies are defined as optional, one has to explicitly enable it and choose one. If none is available, the code falls back to gzip. - JS client doesn't compress requests that are too small, this Python client compresses just everything. No change done, I only noticed and stating it.

vdusek · 2026-07-08T13:37:11Z

This has a significant impact. @Pijukatel, please check it as well.

- Remove `brotlicffi`, only CPython runtime is supported. - Add a test with bytearray as an input. This would break only in `brotlicffi`, which is no longer in the dependencies. `brotli` dependency handles it correctly and doesn't throw. - Add `brotli` to dev dependencies and use it directly in tests.

- apify/apify-client-python#927 (comment) - The default was 11 (max). Now it uses quality 6 - roughly 2–4× faster with only a modest increase in compressed size.

mixalturek · 2026-07-08T15:37:45Z

@vdusek Nearly all comments are fixed now, thanks again for your great review.

One more thing: I forgot to mention that the JS client skips compression if the payload size is below 1024 B. Do we want to apply it to the Python client as well?

- apify/apify-client-python#927 (comment) - The default was 11 (max). Now it uses quality 6 - roughly 2–4× faster with only a modest increase in compressed size.

vdusek · 2026-07-09T07:01:44Z

One more thing: I forgot to mention that the JS client skips compression if the payload size is below 1024 B. Do we want to apply it to the Python client as well?

Thanks for noticing. I opened an issue for that: #934. Let's resolve it in a separate PR so it is documented in the changelog.

Pijukatel

Not sure about the non-default compression quality , but it is not blocking for me.

vdusek

I removed brotli from the dev dependency group. Sorry, my previous comment wasn't entirely correct. We run uv sync --all-extras everywhere in CI, so there's no need to include it there. The initial mypy/ty override ignores confused me, but they were probably added based on your local setup, where you didn't have the brotli extra installed.

Other than that, one more thing: the documentation 🙂.

Please extend the installation section in the docs here: https://docs.apify.com/api/client/python/docs#installation. You can use a similar section to the one you added to the README.

Also, please create a new concept page for compression. Something like: explaining HTTP compression, how to switch between compression methods, and the pros and cons of each approach.

- apify/apify-core#28971 added support for brotli compression to BE. - Pros: higher compression, cons: more CPU intensive. - The code could be running on too old Node.js, so brotli compression is applied only if possible; the code otherwise falls back to gzip. Too small payloads and unsupported types are still not compressed - no change. ### Related PRs - apify/apify-core#28971 - apify/apify-docs#2750 - #962 - apify/apify-client-python#927 - apify/apify-sdk-python#1031

…dules

vdusek

I did a few additional improvements. @Pijukatel do you want to re-check it?

### TLDR - Ports the `http_compressors` design of the Python client (apify/apify-client-python#927) to the JS client. - Closes: #1064 ### Description Request compression was hard-wired in `maybeCompressValue()`: brotli at quality 6, a silent fallback to gzip, and no way to pick the algorithm, tune the quality, or plug in a custom compressor. The client now has an abstract `HttpCompressor` (`contentEncoding` plus an async `compress()`), a `GzipHttpCompressor` (quality 1 to 9, default 9) and a `BrotliHttpCompressor` (quality 0 to 11, default 6), all exported from the package. `ApifyClient` takes a `compression` option: `'brotli'` (the default), `'gzip'`, or an `HttpCompressor` instance. The option is validated by shape, so an instance from a second copy of the package passes too. `HttpClient` holds the resolved compressor and the compress interceptor uses it. The rules for which bodies get compressed are unchanged: at least 1 KiB, a compressible content type, no caller-set `Content-Encoding`, Node.js only. Two deliberate differences from Python. Brotli stays the default, since Node ships it in `node:zlib` and the dependency reason for Python's gzip default doesn't apply. Streams are never compressed, where Python reads file-likes into memory. `setRecord()` still has no `contentEncoding` option, so the docs route pre-compressed bodies through `client.httpClient.call()`. Docs: a new `HTTP compression` concepts page adapted from the Python one, a section in the v3 upgrade guide, and the regenerated api-extractor report. ### Breaking change The silent fallback from #990 is gone. The configured compressor is the only one the client runs, and its error fails the request, as in Python. A runtime whose `node:zlib` lacks brotli used to get a gzipped or uncompressed body and now needs `compression: 'gzip'`. A client constructed without the option sends the same requests as before. *✍️ Drafted by Claude Code*

mixalturek force-pushed the brotli-compression branch 2 times, most recently from 49bf689 to aabd87c Compare July 8, 2026 10:05

mixalturek force-pushed the brotli-compression branch from aabd87c to f26780a Compare July 8, 2026 10:25

This was referenced Jul 8, 2026

feat: Compress requests using brotli algo apify/apify-client-js#962

Merged

feat: Document support for brotli compression apify/apify-docs#2750

Merged

mixalturek requested review from nmanerikar and vdusek July 8, 2026 11:04

mixalturek mentioned this pull request Jul 8, 2026

feat: Compress requests using brotli algo apify/apify-sdk-python#1031

Closed

vdusek requested changes Jul 8, 2026

View reviewed changes

vdusek requested a review from Pijukatel July 8, 2026 13:36

mixalturek added 3 commits July 8, 2026 17:10

feat: Refactoring of tests

0dd99a9

feat: Use brotli quality=6; lower but faster

3fffa62

vdusek mentioned this pull request Jul 9, 2026

Skip request-body compression for small payloads #934

Closed

Pijukatel approved these changes Jul 9, 2026

View reviewed changes

Comment thread README.md Resolved

rm brotli from dev deps

37f5a47

vdusek reviewed Jul 9, 2026

View reviewed changes

vdusek changed the title ~~feat: Compress requests using brotli algo~~ feat: Add request body compression with optional brotli Jul 9, 2026

feat: Documentation for the compression

7617ed0

mixalturek requested a review from szaganek as a code owner July 9, 2026 11:31

feat: Formatting

28f34e2

vdusek added 4 commits July 14, 2026 19:47

refactor: Split _utils module into a _utils subpackage

a0650b6

refactor: Delegate compressor quality validation to the underlying mo…

f7f6a34

…dules

docs: Polish the compression concept guide

9fc37df

test: Add compressor tests and parametrize charge tests by encoding

0cf66e3

github-actions Bot added this to the 145th sprint - Core Services team milestone Jul 14, 2026

github-actions Bot added the t-core-services Issues with this label are in the ownership of the core services team. label Jul 14, 2026

vdusek added 6 commits July 14, 2026 20:50

refactor: Unify compression naming and extract compressor resolution

0a76634

test: Align compressor tests with renamed param and terminology

29f5315

docs: Rename compression page to HTTP compression and trim install docs

1feee71

build: Format brotli optional-dependency on a single line

4674110

feat: Validate compressor quality range and fail fast

ff4f21e

test: Use pytest.param with explicit ids in compression tests

a90bb02

vdusek added t-tooling Issues with this label are in the ownership of the tooling team. adhoc Ad-hoc unplanned task added during the sprint. labels Jul 14, 2026

vdusek approved these changes Jul 14, 2026

View reviewed changes

szaganek approved these changes Jul 15, 2026

View reviewed changes

Comment thread docs/02_concepts/13_http_compression.mdx Resolved

Comment thread docs/02_concepts/13_http_compression.mdx Resolved

Comment thread docs/02_concepts/13_http_compression.mdx Resolved

vdusek added 5 commits July 15, 2026 09:50

docs: Apply review suggestions to HTTP compression page

72c4e74

refactor: Validate compression eagerly at client construction

dfbaca0

refactor: Align try_import with crawlee-python

6596cef

style: Keep future annotations import in try_import

273c09f

test: Add missing future annotations import to unit tests

c810cfa

vdusek merged commit a8a393b into master Jul 15, 2026
30 checks passed

vdusek deleted the brotli-compression branch July 15, 2026 08:21

vdusek mentioned this pull request Jul 15, 2026

Adopt apify-client v3.1 and enable brotli compression by default apify/apify-sdk-python#1045

Closed

Pijukatel mentioned this pull request Jul 18, 2026

Update to enable extras brotli compression conda-forge/apify-client-feedstock#3

Closed

raethlo added the validated Issues that are resolved and their solutions fulfill the acceptance criteria. label Jul 27, 2026

This was referenced Sep 14, 2026

Make request-body compression configurable, mirroring the Python client apify/apify-client-js#1064

Closed

feat!: make request-body compression configurable apify/apify-client-js#1065

Merged

Conversation

mixalturek commented Jul 8, 2026 • edited by vdusek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related PRs

Uh oh!

codecov Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vdusek commented Jul 8, 2026

Uh oh!

mixalturek commented Jul 8, 2026

Uh oh!

vdusek commented Jul 9, 2026

Uh oh!

Pijukatel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mixalturek commented Jul 8, 2026 •

edited by vdusek

Loading

codecov Bot commented Jul 8, 2026 •

edited

Loading