feat!: make request-body compression configurable

docs/02_concepts/05_bundled-environments.md

@@ -25,7 +25,7 @@ The bundle is self-contained and needs no polyfill configuration.
 
 ## Bundling the ES module build yourself
 
-The client's own code runs on Web APIs. The parts that need Node.js built-ins, the keep-alive HTTP agents with proxy support and request body compression, live in a single module that the `#runtime` entry of the package's `imports` field selects at bundle time. The `node` condition gets the Node.js implementation, every other target gets the Web API one. A bundler targeting a browser, Cloudflare Workers, or another edge runtime therefore never sees `node:zlib`, `node:os`, `node:util`, or `proxy-agent`.
+The client's own code runs on Web APIs. The parts that need Node.js built-ins, the keep-alive HTTP agents with proxy support and request body compression, live in a single module that the `#runtime` entry of the package's `imports` field selects at bundle time. The `node` condition gets the Node.js implementation, every other target gets the Web API one. A bundler targeting a browser, Cloudflare Workers, or another edge runtime therefore never sees `node:zlib`, `node:os`, or `proxy-agent`.
 
 Reaching the ES module build takes a bundler that doesn't set the `browser` condition, which resolves `apify-client` to the pre-built bundle. esbuild's `neutral` platform sets no conditions, and webpack and Vite let you list them through `resolve.conditionNames` and `resolve.conditions`.
 

docs/02_concepts/07_http-compression.md

@@ -0,0 +1,136 @@
+---
+id: http-compression
+title: HTTP compression
+sidebar_label: HTTP compression
+description: 'Configure how the Apify API client for JavaScript compresses request bodies: brotli by default, gzip, or a custom compressor.'
+---
+
+import ApiLink from '@theme/ApiLink';
+
+The client compresses request bodies before sending them to the API. Compression reduces the amount of data transferred over the network, which makes requests faster and saves bandwidth, especially for large payloads such as Actor inputs, dataset items, or key-value store records.
+
+## How it works
+
+The client compresses request bodies with the compressor configured via the `compression` option of the <ApiLink to="class/ApifyClient">`ApifyClient`</ApiLink> constructor, brotli by default. The API accepts both brotli and gzip and decompresses the body transparently. A body is compressed only when it's large enough to benefit, its content type isn't already compressed, and the request carries no `Content-Encoding` of its own. For details, see [Minimum body size](#minimum-body-size), [Already-compressed payloads](#already-compressed-payloads), and [Pre-compressed bodies](#pre-compressed-bodies).
+
+Compression needs `node:zlib`, so it runs where the client resolves its Node.js implementation. In a browser or an edge runtime it sends every body as it is, and the `compression` option has no effect. See [Bundled environments](./05_bundled-environments.md) for what decides which implementation a bundle gets.
+
+## Minimum body size
+
+The client sends bodies smaller than 1024 bytes without compression and without the `Content-Encoding` header. A body of this size fits in one network packet, so compression doesn't remove a network round trip and only costs CPU time. For very small bodies, the compression format adds bytes and can make the body larger.
+
+## Already-compressed payloads
+
+Some payloads carry their own compression, so compressing them again costs CPU and memory while making the request slightly larger. The client skips compression when the request's `Content-Type` is one of these:
+
+- any `image/*`, `audio/*`, or `video/*` type
+- archives such as `application/zip`, `application/gzip`, or `application/x-7z-compressed`
+- office documents and packages built on ZIP, such as `.docx`, `.xlsx`, `.epub`, or `.apk`
+- web fonts (`font/woff`, `font/woff2`)
+
+Two kinds of media type are compressed anyway: raw formats such as `image/bmp`, `image/tiff`, and `audio/wav`, and subtypes with a structured syntax suffix such as `image/svg+xml`. Set an accurate `contentType` when uploading media to a key-value store:
+
+```js
+import { readFile } from 'node:fs/promises';
+
+const screenshot = await readFile('screenshot.png');
+
+// The explicit content type lets the client skip compressing the PNG.
+await client.keyValueStore('my-store-id').setRecord({
+    key: 'screenshot.png',
+    value: screenshot,
+    contentType: 'image/png',
+});
+```
+
+Without an explicit content type, a `Buffer` is sent as `application/octet-stream`, which the client can't tell apart from uncompressed binary data and therefore still compresses. A stream is sent as it is, whatever its content type.
+
+## Pre-compressed bodies
+
+A payload can reach the client already encoded, for example a gzipped file read from disk. Set the `Content-Encoding` header to name the encoding the payload carries. The client then sends the body as it is and forwards the header, so nothing gets compressed twice. The resource clients don't expose the header, so send such a request through `client.httpClient.call()`:
+
+```js
+import { readFile } from 'node:fs/promises';
+
+// A file that is gzipped on disk already.
+const report = await readFile('report.json.gz');
+
+// The explicit content encoding stops the client from compressing the bytes again.
+await client.httpClient.call({
+    url: `${client.baseUrl}/key-value-stores/my-store-id/records/report.json`,
+    method: 'PUT',
+    data: report,
+    headers: { 'content-type': 'application/json', 'content-encoding': 'gzip' },
+    timeoutSecs: 'long',
+});
+```
+
+The header is forwarded verbatim, so it also covers encodings the client ships no compressor for, such as `deflate`. The API accepts `gzip`, `br`, `deflate`, and `identity`. Passing `identity` turns compression off for a single request without changing how the client is configured.
+
+The client can't verify that the bytes match the header, so set `Content-Encoding` only when the payload really is encoded that way. Key-value store records are stored exactly as you upload them, which makes the header part of the stored record rather than a transport detail.
+
+## Configuration
+
+To choose the compression algorithm, pass `compression` to the client constructor:
+
+```js
+import { ApifyClient } from 'apify-client';
+
+// Default: brotli
+const client = new ApifyClient({ token: 'MY-APIFY-TOKEN' });
+
+// Gzip
+const gzipClient = new ApifyClient({ token: 'MY-APIFY-TOKEN', compression: 'gzip' });
+```
+
+Both algorithms come from `node:zlib`, so neither needs an extra dependency. If the runtime's `node:zlib` doesn't implement brotli, which can happen with a Node.js compatibility shim or a bundler polyfill, the request fails with the error from the runtime. The client doesn't fall back to gzip on its own, so pass `compression: 'gzip'` on such a runtime.
+
+## Compression quality and custom compressors
+
+To set the compression quality, pass an <ApiLink to="interface/HttpCompressor">`HttpCompressor`</ApiLink> instead of a name:
+
+```js
+import { ApifyClient, BrotliHttpCompressor, GzipHttpCompressor } from 'apify-client';
+
+// Brotli at maximum quality
+const client = new ApifyClient({
+    token: 'MY-APIFY-TOKEN',
+    compression: new BrotliHttpCompressor({ quality: 11 }),
+});
+
+// Gzip at the fastest level
+const fastClient = new ApifyClient({
+    token: 'MY-APIFY-TOKEN',
+    compression: new GzipHttpCompressor({ quality: 1 }),
+});
+```
+
+A custom compressor is any object with a `contentEncoding` string and an async `compress()` method, which is what the <ApiLink to="interface/HttpCompressor">`HttpCompressor`</ApiLink> interface describes. The client calls it only for bodies that reach the [minimum body size](#minimum-body-size) and aren't [already compressed](#already-compressed-payloads) or [pre-compressed by the caller](#pre-compressed-bodies):
+
+```js
+import { ApifyClient } from 'apify-client';
+
+const identityCompressor = {
+    /** Value sent in the `Content-Encoding` header. */
+    contentEncoding: 'identity',
+
+    /** Returns the body as it is, so nothing gets compressed. */
+    async compress(data) {
+        return data;
+    },
+};
+
+const client = new ApifyClient({ token: 'MY-APIFY-TOKEN', compression: identityCompressor });
+```
+
+## Comparison
+
+|                               | Brotli                                  | Gzip                                  |
+|-------------------------------|-----------------------------------------|---------------------------------------|
+| **Compression ratio**         | Typically better than gzip              | Good                                  |
+| **CPU cost**                  | Moderate, depends on quality            | Low                                   |
+| **`Content-Encoding` header** | `br`                                    | `gzip`                                |
+| **Quality range**             | `0` to `11`                             | `1` to `9`                            |
+| **Default quality**           | `6`                                     | `6`                                   |
+| **Enable via config**         | `compression: 'brotli'` (default)       | `compression: 'gzip'`                 |
+| **Best for**                  | Large payloads where bandwidth matters  | Low CPU cost, runtimes without brotli |

docs/04_upgrading/upgrading_v3.md

@@ -339,6 +339,22 @@ The `timeoutSecs` option of <ApiLink to="class/KeyValueStoreClient#setRecord">`K
 
 `client.httpClient.call()` takes `timeoutSecs` where it read the axios `timeout` in milliseconds. A direct call that passed `timeout: 30000` has to become `timeoutSecs: 30`.
 
+## Request compression is configurable
+
+The <ApiLink to="class/ApifyClient">`ApifyClient`</ApiLink> constructor takes a `compression` option: `'brotli'`, `'gzip'`, or an <ApiLink to="interface/HttpCompressor">`HttpCompressor`</ApiLink> for a custom quality or algorithm. The default stays brotli at quality 6, so a client constructed without the option sends the same requests as in v2. For the rules that decide which bodies get compressed, see [HTTP compression](../02_concepts/07_http-compression.md).
+
+```js
+import { ApifyClient, BrotliHttpCompressor } from 'apify-client';
+
+const client = new ApifyClient({ token: 'MY-APIFY-TOKEN', compression: 'gzip' });
+const bestClient = new ApifyClient({
+    token: 'MY-APIFY-TOKEN',
+    compression: new BrotliHttpCompressor({ quality: 11 }),
+});
+```
+
+The silent fallback is gone. In v2, a body was sent gzipped when brotli failed, and uncompressed when gzip failed too, which could happen on a runtime whose `node:zlib` is a polyfill or a Node.js compatibility shim. In v3 the configured compressor is the only one the client runs, and its error fails the request. On such a runtime, pass `compression: 'gzip'` or a custom compressor.
+
 ## `versions().list()` and `envVars().list()` lose their pagination options
 
 <ApiLink to="class/ActorVersionCollectionClient#list">`ActorVersionCollectionClient.list()`</ApiLink> and <ApiLink to="class/ActorEnvVarCollectionClient#list">`ActorEnvVarCollectionClient.list()`</ApiLink> accept only the `timeoutSecs` option. Neither endpoint reads `offset`, `limit` or `desc`, and both return every item in one response, so `chunkSize` had nothing to size either. The `ActorVersionCollectionListOptions` and `ActorEnvVarCollectionListOptions` types that declared those four options, deprecated since v2.21.0, are gone from the package. A call that passed any of them no longer compiles, and throws an `ArgumentValidationError` about an unrecognized key in JavaScript. Drop them and the call returns the same items as before.

docs/public-api/apify-client.api.md

@@ -610,6 +610,7 @@ export class ApifyClient {
 // @public
 export interface ApifyClientOptions {
     baseUrl?: string;
+    compression?: HttpCompressionAlgorithm | HttpCompressor;
     // (undocumented)
     maxRetries?: number;
     minDelayBetweenRetriesMillis?: number;
@@ -666,6 +667,20 @@ interface BaseOptions {
     publicBaseUrl: string;
 }
 
+// @public
+export class BrotliHttpCompressor implements HttpCompressor {
+    constructor(options?: BrotliHttpCompressorOptions);
+    // (undocumented)
+    compress(data: Buffer): Promise<Buffer>;
+    // (undocumented)
+    readonly contentEncoding = "br";
+}
+
+// @public
+export interface BrotliHttpCompressorOptions {
+    quality?: number;
+}
+
 // @public
 export interface Build extends Omit<Schemas['Build'], keyof BuildRePointed>, BuildRePointed {
 }
@@ -2849,6 +2864,20 @@ export interface GetStreamedLogOptions extends TimeoutOptions {
     toLog?: Log | null | 'default';
 }
 
+// @public
+export class GzipHttpCompressor implements HttpCompressor {
+    constructor(options?: GzipHttpCompressorOptions);
+    // (undocumented)
+    compress(data: Buffer): Promise<Buffer>;
+    // (undocumented)
+    readonly contentEncoding = "gzip";
+}
+
+// @public
+export interface GzipHttpCompressorOptions {
+    quality?: number;
+}
+
 // Not exported by the entry point; reachable only as a referenced type.
 // @public (undocumented)
 class HttpClient {
@@ -2859,6 +2888,7 @@ class HttpClient {
     call<T = any>(config: ApifyRequestConfig): Promise<ApifyResponse<T>>;
     // (undocumented)
     httpAgent?: http.Agent;
+    readonly httpCompressor: HttpCompressor;
     // (undocumented)
     httpsAgent?: https.Agent;
     // (undocumented)
@@ -2883,6 +2913,8 @@ interface HttpClientOptions {
     // (undocumented)
     apifyClientStats: Statistics;
     // (undocumented)
+    httpCompressor: HttpCompressor;
+    // (undocumented)
     logger: Log;
     // (undocumented)
     maxRetries: number;
@@ -2904,6 +2936,15 @@ interface HttpClientOptions {
     workflowKey?: string;
 }
 
+// @public
+export type HttpCompressionAlgorithm = 'brotli' | 'gzip';
+
+// @public
+export interface HttpCompressor {
+    compress(data: Buffer): Promise<Buffer>;
+    readonly contentEncoding: string;
+}
+
 // @public
 export class InvalidRequestError extends ApifyApiError {
 }

rsbuild.config.ts

@@ -1,7 +1,7 @@
 import { defineConfig } from '@rsbuild/core';
 import { pluginNodePolyfill } from '@rsbuild/plugin-node-polyfill';
 
-const MAX_BUNDLE_BYTES = 350 * 1024;
+const MAX_BUNDLE_BYTES = 360 * 1024;
 
 // eslint-disable-next-line import/no-default-export
 export default defineConfig({
@@ -55,7 +55,7 @@ export default defineConfig({
                 ...config.optimization,
                 splitChunks: false,
             };
-            // A regression guard, not a target: the bundle sits at ~325 kB, so this only fails the
+            // A regression guard, not a target: the bundle sits at ~338 kB, so this only fails the
             // build on an unnoticed jump. A `zod` minor is the likeliest cause, since it is a runtime
             // dependency on a caret range - bumping this constant is the expected response. The
             // generated response schemas growing with the OpenAPI specification is the other.

src/apify_client.ts

@@ -6,6 +6,9 @@ import type { Log } from '@apify/log';
 import logger from '@apify/log';
 
 import { HttpClient } from './http_client.js';
+import type { HttpCompressor } from './http_compressors/base.js';
+import type { HttpCompressionAlgorithm } from './http_compressors/resolve.js';
+import { compressionSchema, resolveCompressor } from './http_compressors/resolve.js';
 import type { RequestInterceptorFunction } from './interceptors.js';
 import { ActorClient } from './resource_clients/actor.js';
 import { ActorCollectionClient } from './resource_clients/actor_collection.js';
@@ -42,6 +45,7 @@ import { getEnv, parseArgument } from './utils.js';
 
 const clientOptionsSchema = z.strictObject({
     baseUrl: z.string().default('https://api.apify.com'),
+    compression: compressionSchema.default('brotli'),
     publicBaseUrl: z.string().default('https://api.apify.com'),
     maxRetries: z.number().default(8),
     minDelayBetweenRetriesMillis: z.number().default(500),
@@ -101,6 +105,7 @@ export class ApifyClient {
 
         const {
             baseUrl,
+            compression,
             publicBaseUrl,
             maxRetries,
             minDelayBetweenRetriesMillis,
@@ -126,6 +131,7 @@ export class ApifyClient {
             maxRetries,
             minDelayBetweenRetriesMillis,
             requestInterceptors,
+            httpCompressor: resolveCompressor(compression),
             timeoutShortSecs,
             timeoutMediumSecs,
             timeoutLongSecs,
@@ -584,6 +590,13 @@ export class ApifyClient {
 export interface ApifyClientOptions {
     /** @default https://api.apify.com */
     baseUrl?: string;
+    /**
+     * Compression of request bodies: the name of a built-in algorithm, or an {@link HttpCompressor} for a custom
+     * quality or algorithm. Bodies are compressed only in Node.js, and only when they are large enough to benefit
+     * and their content type does not already carry its own compression.
+     * @default 'brotli'
+     */
+    compression?: HttpCompressionAlgorithm | HttpCompressor;
     /**
      * @default https://api.apify.com
      * @since Added in 2.17.0

src/http_client.ts

@@ -11,8 +11,9 @@ import type { Log } from '@apify/log';
 import { concatStreamToBuffer } from '@apify/utilities';
 
 import { ApifyApiError } from './apify_api_error.js';
+import type { HttpCompressor } from './http_compressors/base.js';
 import type { RequestInterceptorFunction } from './interceptors.js';
-import { InvalidResponseBodyError, requestInterceptors, responseInterceptors } from './interceptors.js';
+import { createRequestInterceptors, InvalidResponseBodyError, responseInterceptors } from './interceptors.js';
 import { runtime } from '#runtime';
 import type { Statistics } from './statistics.js';
 import type { Timeout, TimeoutTier } from './timeouts.js';
@@ -29,6 +30,12 @@ export class HttpClient {
 
     userProvidedRequestInterceptors: RequestInterceptorFunction[];
 
+    /**
+     * Compressor applied to request bodies that are worth compressing. The request interceptor captures it at
+     * construction, so it is fixed for the lifetime of the client.
+     */
+    readonly httpCompressor: HttpCompressor;
+
     logger: Log;
 
     /** Duration of each timeout tier, in milliseconds. */
@@ -53,6 +60,7 @@ export class HttpClient {
         this.maxRetries = options.maxRetries;
         this.minDelayBetweenRetriesMillis = options.minDelayBetweenRetriesMillis;
         this.userProvidedRequestInterceptors = options.requestInterceptors;
+        this.httpCompressor = options.httpCompressor;
         this.timeoutMillis = {
             short: options.timeoutShortSecs * 1000,
             medium: options.timeoutMediumSecs * 1000,
@@ -120,7 +128,7 @@ export class HttpClient {
             this.axios.defaults.headers['User-Agent'] = userAgent;
         }
 
-        requestInterceptors.forEach((i) => this.axios.interceptors.request.use(i as any));
+        createRequestInterceptors(this.httpCompressor).forEach((i) => this.axios.interceptors.request.use(i as any));
         this.userProvidedRequestInterceptors.forEach((i) => this.axios.interceptors.request.use(i as any));
         responseInterceptors.forEach((i) => this.axios.interceptors.response.use(i as any));
     }
@@ -346,6 +354,7 @@ export interface HttpClientOptions {
     maxRetries: number;
     minDelayBetweenRetriesMillis: number;
     requestInterceptors: RequestInterceptorFunction[];
+    httpCompressor: HttpCompressor;
     timeoutShortSecs: number;
     timeoutMediumSecs: number;
     timeoutLongSecs: number;