Support caller-supplied Content-Encoding for pre-compressed request bodies #996

Follow-up from a review comment on #987: #987 (comment)

Problem

There is no way for a caller to hand the client an already-encoded request body.

• Before perf: Skip request-body compression for already-compressed content types #987, _prepare_request_call compressed every bytes-like body and merged its own Content-Encoding last, so a caller-supplied value was silently overwritten.
• Since perf: Skip request-body compression for already-compressed content types #987, a caller-supplied Content-Encoding is dropped whenever compression is skipped, so a pre-compressed body goes out labeled as unencoded.

The API supports this and documents it. PUT key-value-store record documents a Content-Encoding request header accepting gzip, br, deflate, and identity: "To save bandwidth, storage, and speed up your upload, send the request payload compressed with Gzip compression and add the Content-Encoding: gzip header. It is possible to set up another compression type with Content-Encoding request header." And per the key-value store docs, "records are stored exactly as you upload them", with the encoding taken from that header — so on that endpoint the header is record metadata, not just a transport hint, and dropping it stores compressed bytes under an unencoded label.

Honoring the header also unlocks encodings the client ships no compressor for (deflate, zstd, ...) without having to add one.

How the JS client solves it

apify-client-js treats a caller-supplied content-encoding as "hands off" — src/interceptors.ts#L81-L82:

async function maybeCompressRequest(config: ApifyRequestConfig): Promise<ApifyRequestConfig> {
    if (config.headers?.['content-encoding']) return config;

    const maybeCompressed = await maybeCompressValue(config.data);
    ...
}

No opt-out parameter and no dedicated compressor — the header alone is the signal. (maybeCompressValue also skips bodies below MIN_COMPRESS_BYTES = 1024, which is #934 on the Python side.)

Proposal

1. Honor a caller-supplied Content-Encoding in _prepare_request_call, matching JS. Precedence: caller Content-Encoding -> content-type heuristic (is_compressible_content_type) -> compress. The header is forwarded verbatim, so any encoding the backend accepts works, and identity becomes a per-request opt-out for free.

   caller_encoding = next((v for k, v in headers.items() if k.lower() == 'content-encoding'), None)
   if caller_encoding is None and is_compressible_content_type(content_type):
       data = self._http_compressor.compress(data)
       headers = self._merge_headers(headers, {'Content-Encoding': self._http_compressor.content_encoding})

   Explicitly out of scope: stacking the client's encoding on top of the caller's (Content-Encoding: gzip, br) — the backend does not document the list form.

   This reverses the drop-the-header behavior added in perf: Skip request-body compression for already-compressed content types #987 and flips test_prepare_request_call_replaces_caller_content_encoding plus the two ..._drops_caller_content_encoding_... tests.

2. Expose content_encoding on KeyValueStoreClient.set_record (sync and async). Step 1 on its own is barely reachable: no public resource method takes per-request headers, so the only entry points are the client-wide ApifyClient(headers=...), which would apply the header to every request, and the private _http_client.call. set_record is where the API documents the header, so that is where it belongs.

Rejected: a per-request compress=False keyword on HttpClient.call. HttpClient/HttpClientAsync are a public ABC, and once resource clients pass the new keyword, existing custom implementations break. Headers already flow through, so step 1 covers the same need without touching that contract.

Docs: add a pre-compressed bodies section to the HTTP compression concept page.

✍️ Drafted by Claude Code