feat: add zstd request body compression by apify-factory · Pull Request #1218 · apify/apify-client-python

apify-factory · 2026-11-10T09:40:51Z

Reference change

Ported change

HttpCompressionAlgorithm (apify_client.types) now accepts 'zstd', and resolve_compressor maps it to the new compressor. The compression argument of ApifyClient / ApifyClientAsync and of ActorServerClient.request therefore accepts 'zstd'.
New public apify_client.http_compressors.ZstdHttpCompressor (http_compressors/_zstd.py), modeled on the gzip and brotli compressors. It reports content_encoding = 'zstd' and takes a keyword-only quality from 1 to 19, defaulting to 3. Levels 20–22 are rejected because their window exceeds the 8 MB that RFC 9659 requires HTTP decoders to support. The body-selection rules (minimum size, already-compressed content types, caller-set Content-Encoding) are the same as for the other algorithms.
Python-specific differences from the reference:
- Source of zstd. Python 3.14+ uses the standard library's compression.zstd. Older versions use the official backports.zstd package through a new optional zstd extra (pip install "apify-client[zstd]"), following the pattern of the brotli extra.
- Missing zstd. If the extra is missing, or a 3.14+ Python was built without zstd, import apify_client still works. Only accessing ZstdHttpCompressor or passing compression='zstd' raises a clear ImportError.
- Supporting changes. try_import now treats a missing parent package of a dotted dependency name as the dependency being missing, and accepts extra_name=None for dependencies no extra can install. [tool.ty.analysis] allowed-unresolved-imports lists backports.zstd, because ty checks against Python 3.11 even in 3.14+ environments, where the package is not installed.
- Invalid quality. An invalid quality raises ValueError, like the existing Python compressors, rather than the JS ArgumentValidationError.

…endencies

codecov · 2026-11-10T09:43:03Z

Codecov Report

❌ Patch coverage is 81.25000% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.66%. Comparing base (eb8ded4) to head (fd96b88).
⚠️ Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
src/apify_client/http_compressors/_zstd.py	75.00%	7 Missing ⚠️
src/apify_client/http_compressors/__init__.py	90.90%	1 Missing ⚠️
src/apify_client/types.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1218      +/-   ##
==========================================
- Coverage   95.76%   95.66%   -0.10%     
==========================================
  Files          63       64       +1     
  Lines        6676     6717      +41     
==========================================
+ Hits         6393     6426      +33     
- Misses        283      291       +8

Flag	Coverage Δ
integration	`90.63% <58.33%> (-0.04%)`	⬇️
unit	`91.00% <81.25%> (-0.07%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

vdusek

Could you add a self-review / staff review to the porting process?

vdusek

Note: compressor_case in tests/unit/test_http_clients.py (outside the diff) still covers only gzip and brotli. Consider adding a zstd case: pytest.param((ZstdHttpCompressor(), 'zstd', zstd.decompress), id='zstd').

✍️ Drafted by Claude Code

vdusek · 2026-11-11T08:23:03Z

+else:
+    # Not `from backports import zstd`: when another `backports.*` package is installed but this one is not, that
+    # raises a plain `ImportError` instead of the `ModuleNotFoundError` the optional-extra handling expects.
+    import backports.zstd as zstd  # noqa: PLR0402


Note: If backports.zstd is installed but its C extension fails to load, this raises a plain ImportError. try_import catches only ModuleNotFoundError, so import apify_client fails for every user, including gzip users. The 3.14+ branch above handles this case. Should this branch handle it the same way, or is it fine because brotli behaves the same?

✍️ Drafted by Claude Code

Good catch, handled in b114508 and db6ed4e. On Python < 3.14, a plain ImportError from backports.zstd is now re-raised as ModuleNotFoundError(name='backports.zstd'), so import apify_client keeps working and only ZstdHttpCompressor raises. The error says the package is installed but cannot be loaded, and suggests pip install --force-reinstall backports.zstd or gzip/brotli. try_import no longer appends the "install the extra" hint to an error raised from another one, since installing the extra again doesn't help in that case. Both cases have unit tests (test_zstd_import_hook_raises_clear_error_when_backport_fails_to_load, test_try_import_keeps_the_message_of_a_dependency_that_fails_to_load). brotli is left as it is.

vdusek · 2026-11-11T08:23:04Z

+if sys.version_info >= (3, 14):
+    from compression import zstd
+else:
+    import backports.zstd as zstd  # noqa: PLR0402 - the same import form as `http_compressors/_zstd.py`


Nit: The reason for this import form in _zstd.py is to get a ModuleNotFoundError that try_import can catch. Tests don't go through try_import, so the plain form works and the noqa can go.

Suggested change

import backports.zstd as zstd # noqa: PLR0402 - the same import form as `http_compressors/_zstd.py`

from backports import zstd

✍️ Drafted by Claude Code

I tried this in 345e092 but reverted it in 7964ffa. Type checking runs with python-version = "3.11", so ty checks this branch even on 3.14/3.15, where backports.zstd isn't installed. allowed-unresolved-imports = ["backports.zstd"] covers import backports.zstd. from backports import zstd resolves the module backports, so it fails with Cannot resolve imported module 'backports' and would break the 3.14 and 3.15 type_check jobs. Allowing all of backports would also work, but it's broader. I kept the dotted form and made the noqa reason say why.

vdusek · 2026-11-11T08:23:04Z

+if sys.version_info >= (3, 14):
+    from compression import zstd
+else:
+    import backports.zstd as zstd  # noqa: PLR0402 - the same import form as `http_compressors/_zstd.py`


Nit: The reason for this import form in _zstd.py is to get a ModuleNotFoundError that try_import can catch. Tests don't go through try_import, so the plain form works and the noqa can go.

Suggested change

import backports.zstd as zstd # noqa: PLR0402 - the same import form as `http_compressors/_zstd.py`

from backports import zstd

✍️ Drafted by Claude Code

Same as in test_http_compressors.py: I kept import backports.zstd as zstd, because from backports import zstd fails ty on 3.14+ (it resolves backports, which the allowed-unresolved-imports entry doesn't cover). The noqa now gives the reason.

vdusek · 2026-11-11T08:23:04Z

    ```

+    To compress with `zstd`, pass `compression='zstd'`. Python 3.14 and newer have it in the standard library. On older
+    versions, install the optional `zstd` extra: `pip install "apify-client[zstd]"`.


Nit: The brotli section above shows both pip install and uv add. Show uv add "apify-client[zstd]" here too. The same applies to docs/01_introduction/index.mdx.

✍️ Drafted by Claude Code

Added uv add "apify-client[zstd]" here and in docs/01_introduction/index.mdx in 7fe7fe1.

vdusek · 2026-11-11T08:34:24Z

@@ -14,9 +14,14 @@



Could we add integration test that sends a zstd body to the live API?

Not added in this PR: I can't run tests against the live API from here, and I don't want to add an integration test that has never been run. The natural place is next to test_key_value_store_set_and_get_pre_compressed_record in tests/integration/test_key_value_store.py: a client with compression='zstd' pushing a large record or dataset items and reading them back. It needs someone with API tokens to add and run it, either in this PR or as a follow-up.

…s import

…nsistent

…nstall hint

apify-factory · 2026-11-11T09:02:57Z

Re the review note on compressor_case in tests/unit/test_http_clients.py: added the zstd case in 345e092.

Cover a key-value store record and a dataset push through a client that compresses request bodies with zstd, for both transports and both the sync and async client. Co-Authored-By: Claude Fable 5.2 <[email protected]> Claude-Session: https://claude.ai/code/session_017ocuAK2fAevKdoyPCB2HvC

apify-factory added 10 commits November 10, 2026 10:22

feat: add zstd request body compression

f94f98c

feat: accept 'zstd' as a compression algorithm and add the zstd extra

08a3d81

docs: document zstd request body compression

0ec8a9d

docs: mention the zstd extra in the README

53a447b

test: cover the zstd compressor

8edd3bd

test: cover the optional-import handling of dotted and extra-less dep…

01a571f

…endencies

docs: note the zstd build requirement in the Actor server Raises section

bc033f7

docs: document the zstd compression option on the client constructors

214c2fd

docs: list zstd among the encodings set_record accepts

212ce68

build: let ty resolve without backports.zstd on Python 3.14+

ced1d0e

apify-factory requested a review from szaganek as a code owner November 10, 2026 09:40

apify-factory added the api-client-sync label Nov 10, 2026

github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 10, 2026

Pijukatel requested a review from vdusek November 10, 2026 10:17

vdusek reviewed Nov 10, 2026

View reviewed changes

apify-factory and others added 5 commits November 10, 2026 14:18

feat: tolerate a zstd module that fails to load on Python 3.14+

d68f6a0

chore: lock backports.zstd for the zstd extra

19d5957

feat: suggest gzip or brotli when the Python build lacks zstd

a155d9a

docs: tighten the zstd availability notes

7f623d3

docs: split the zstd install notes and shorten the zstd import comment

845edaf

Pijukatel requested a review from vdusek November 10, 2026 14:57

Pijukatel approved these changes Nov 10, 2026

View reviewed changes

vdusek requested changes Nov 11, 2026

View reviewed changes

vdusek reviewed Nov 11, 2026

View reviewed changes

apify-factory added 5 commits November 11, 2026 08:55

fix: keep the client importable when backports.zstd fails to load

b114508

test: cover zstd in the compressor fixture and use the plain backport…

345e092

…s import

docs: show uv add for the zstd extra

7fe7fe1

fix: keep backports.zstd imports type-checkable and the load error co…

7964ffa

…nsistent

fix: give a broken backports.zstd install its own error without the i…

db6ed4e

…nstall hint

	import backports.zstd as zstd # noqa: PLR0402 - the same import form as `http_compressors/_zstd.py`
	from backports import zstd

Conversation

apify-factory commented Nov 10, 2026 • edited by Pijukatel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference change

Ported change

Uh oh!

codecov Bot commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

apify-factory commented Nov 11, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

apify-factory commented Nov 10, 2026 •

edited by Pijukatel

Loading

codecov Bot commented Nov 10, 2026 •

edited

Loading