diff --git a/docs/02_concepts/05_bundled-environments.md b/docs/02_concepts/05_bundled-environments.md index a33e1b71b..44f810d01 100644 --- a/docs/02_concepts/05_bundled-environments.md +++ b/docs/02_concepts/05_bundled-environments.md @@ -25,7 +25,7 @@ The bundle is self-contained and needs no polyfill configuration. ## Bundling the ES module build yourself -The client's own code runs on Web APIs. The parts that need Node.js built-ins, the keep-alive HTTP agents with proxy support and request body compression, live in a single module that the `#runtime` entry of the package's `imports` field selects at bundle time. The `node` condition gets the Node.js implementation, every other target gets the Web API one. A bundler targeting a browser, Cloudflare Workers, or another edge runtime therefore never sees `node:zlib`, `node:os`, `node:util`, or `proxy-agent`. +The client's own code runs on Web APIs. The parts that need Node.js built-ins, the keep-alive HTTP agents with proxy support and request body compression, live in a single module that the `#runtime` entry of the package's `imports` field selects at bundle time. The `node` condition gets the Node.js implementation, every other target gets the Web API one. A bundler targeting a browser, Cloudflare Workers, or another edge runtime therefore never sees `node:zlib`, `node:os`, or `proxy-agent`. Reaching the ES module build takes a bundler that doesn't set the `browser` condition, which resolves `apify-client` to the pre-built bundle. esbuild's `neutral` platform sets no conditions, and webpack and Vite let you list them through `resolve.conditionNames` and `resolve.conditions`. diff --git a/docs/02_concepts/07_http-compression.md b/docs/02_concepts/07_http-compression.md new file mode 100644 index 000000000..2a0cbea2b --- /dev/null +++ b/docs/02_concepts/07_http-compression.md @@ -0,0 +1,136 @@ +--- +id: http-compression +title: HTTP compression +sidebar_label: HTTP compression +description: 'Configure how the Apify API client for JavaScript compresses request bodies: brotli by default, gzip, or a custom compressor.' +--- + +import ApiLink from '@theme/ApiLink'; + +The client compresses request bodies before sending them to the API. Compression reduces the amount of data transferred over the network, which makes requests faster and saves bandwidth, especially for large payloads such as Actor inputs, dataset items, or key-value store records. + +## How it works + +The client compresses request bodies with the compressor configured via the `compression` option of the `ApifyClient` constructor, brotli by default. The API accepts both brotli and gzip and decompresses the body transparently. A body is compressed only when it's large enough to benefit, its content type isn't already compressed, and the request carries no `Content-Encoding` of its own. For details, see [Minimum body size](#minimum-body-size), [Already-compressed payloads](#already-compressed-payloads), and [Pre-compressed bodies](#pre-compressed-bodies). + +Compression needs `node:zlib`, so it runs where the client resolves its Node.js implementation. In a browser or an edge runtime it sends every body as it is, and the `compression` option has no effect. See [Bundled environments](./05_bundled-environments.md) for what decides which implementation a bundle gets. + +## Minimum body size + +The client sends bodies smaller than 1024 bytes without compression and without the `Content-Encoding` header. A body of this size fits in one network packet, so compression doesn't remove a network round trip and only costs CPU time. For very small bodies, the compression format adds bytes and can make the body larger. + +## Already-compressed payloads + +Some payloads carry their own compression, so compressing them again costs CPU and memory while making the request slightly larger. The client skips compression when the request's `Content-Type` is one of these: + +- any `image/*`, `audio/*`, or `video/*` type +- archives such as `application/zip`, `application/gzip`, or `application/x-7z-compressed` +- office documents and packages built on ZIP, such as `.docx`, `.xlsx`, `.epub`, or `.apk` +- web fonts (`font/woff`, `font/woff2`) + +Two kinds of media type are compressed anyway: raw formats such as `image/bmp`, `image/tiff`, and `audio/wav`, and subtypes with a structured syntax suffix such as `image/svg+xml`. Set an accurate `contentType` when uploading media to a key-value store: + +```js +import { readFile } from 'node:fs/promises'; + +const screenshot = await readFile('screenshot.png'); + +// The explicit content type lets the client skip compressing the PNG. +await client.keyValueStore('my-store-id').setRecord({ + key: 'screenshot.png', + value: screenshot, + contentType: 'image/png', +}); +``` + +Without an explicit content type, a `Buffer` is sent as `application/octet-stream`, which the client can't tell apart from uncompressed binary data and therefore still compresses. A stream is sent as it is, whatever its content type. + +## Pre-compressed bodies + +A payload can reach the client already encoded, for example a gzipped file read from disk. Set the `Content-Encoding` header to name the encoding the payload carries. The client then sends the body as it is and forwards the header, so nothing gets compressed twice. The resource clients don't expose the header, so send such a request through `client.httpClient.call()`: + +```js +import { readFile } from 'node:fs/promises'; + +// A file that is gzipped on disk already. +const report = await readFile('report.json.gz'); + +// The explicit content encoding stops the client from compressing the bytes again. +await client.httpClient.call({ + url: `${client.baseUrl}/key-value-stores/my-store-id/records/report.json`, + method: 'PUT', + data: report, + headers: { 'content-type': 'application/json', 'content-encoding': 'gzip' }, + timeoutSecs: 'long', +}); +``` + +The header is forwarded verbatim, so it also covers encodings the client ships no compressor for, such as `deflate`. The API accepts `gzip`, `br`, `deflate`, and `identity`. Passing `identity` turns compression off for a single request without changing how the client is configured. + +The client can't verify that the bytes match the header, so set `Content-Encoding` only when the payload really is encoded that way. Key-value store records are stored exactly as you upload them, which makes the header part of the stored record rather than a transport detail. + +## Configuration + +To choose the compression algorithm, pass `compression` to the client constructor: + +```js +import { ApifyClient } from 'apify-client'; + +// Default: brotli +const client = new ApifyClient({ token: 'MY-APIFY-TOKEN' }); + +// Gzip +const gzipClient = new ApifyClient({ token: 'MY-APIFY-TOKEN', compression: 'gzip' }); +``` + +Both algorithms come from `node:zlib`, so neither needs an extra dependency. If the runtime's `node:zlib` doesn't implement brotli, which can happen with a Node.js compatibility shim or a bundler polyfill, the request fails with the error from the runtime. The client doesn't fall back to gzip on its own, so pass `compression: 'gzip'` on such a runtime. + +## Compression quality and custom compressors + +To set the compression quality, pass an `HttpCompressor` instead of a name: + +```js +import { ApifyClient, BrotliHttpCompressor, GzipHttpCompressor } from 'apify-client'; + +// Brotli at maximum quality +const client = new ApifyClient({ + token: 'MY-APIFY-TOKEN', + compression: new BrotliHttpCompressor({ quality: 11 }), +}); + +// Gzip at the fastest level +const fastClient = new ApifyClient({ + token: 'MY-APIFY-TOKEN', + compression: new GzipHttpCompressor({ quality: 1 }), +}); +``` + +A custom compressor is any object with a `contentEncoding` string and an async `compress()` method, which is what the `HttpCompressor` interface describes. The client calls it only for bodies that reach the [minimum body size](#minimum-body-size) and aren't [already compressed](#already-compressed-payloads) or [pre-compressed by the caller](#pre-compressed-bodies): + +```js +import { ApifyClient } from 'apify-client'; + +const identityCompressor = { + /** Value sent in the `Content-Encoding` header. */ + contentEncoding: 'identity', + + /** Returns the body as it is, so nothing gets compressed. */ + async compress(data) { + return data; + }, +}; + +const client = new ApifyClient({ token: 'MY-APIFY-TOKEN', compression: identityCompressor }); +``` + +## Comparison + +| | Brotli | Gzip | +|-------------------------------|-----------------------------------------|---------------------------------------| +| **Compression ratio** | Typically better than gzip | Good | +| **CPU cost** | Moderate, depends on quality | Low | +| **`Content-Encoding` header** | `br` | `gzip` | +| **Quality range** | `0` to `11` | `1` to `9` | +| **Default quality** | `6` | `6` | +| **Enable via config** | `compression: 'brotli'` (default) | `compression: 'gzip'` | +| **Best for** | Large payloads where bandwidth matters | Low CPU cost, runtimes without brotli | diff --git a/docs/04_upgrading/upgrading_v3.md b/docs/04_upgrading/upgrading_v3.md index 58bc7434e..6ce414050 100644 --- a/docs/04_upgrading/upgrading_v3.md +++ b/docs/04_upgrading/upgrading_v3.md @@ -339,6 +339,22 @@ The `timeoutSecs` option of `K `client.httpClient.call()` takes `timeoutSecs` where it read the axios `timeout` in milliseconds. A direct call that passed `timeout: 30000` has to become `timeoutSecs: 30`. +## Request compression is configurable + +The `ApifyClient` constructor takes a `compression` option: `'brotli'`, `'gzip'`, or an `HttpCompressor` for a custom quality or algorithm. The default stays brotli at quality 6, so a client constructed without the option sends the same requests as in v2. For the rules that decide which bodies get compressed, see [HTTP compression](../02_concepts/07_http-compression.md). + +```js +import { ApifyClient, BrotliHttpCompressor } from 'apify-client'; + +const client = new ApifyClient({ token: 'MY-APIFY-TOKEN', compression: 'gzip' }); +const bestClient = new ApifyClient({ + token: 'MY-APIFY-TOKEN', + compression: new BrotliHttpCompressor({ quality: 11 }), +}); +``` + +The silent fallback is gone. In v2, a body was sent gzipped when brotli failed, and uncompressed when gzip failed too, which could happen on a runtime whose `node:zlib` is a polyfill or a Node.js compatibility shim. In v3 the configured compressor is the only one the client runs, and its error fails the request. On such a runtime, pass `compression: 'gzip'` or a custom compressor. + ## `versions().list()` and `envVars().list()` lose their pagination options `ActorVersionCollectionClient.list()` and `ActorEnvVarCollectionClient.list()` accept only the `timeoutSecs` option. Neither endpoint reads `offset`, `limit` or `desc`, and both return every item in one response, so `chunkSize` had nothing to size either. The `ActorVersionCollectionListOptions` and `ActorEnvVarCollectionListOptions` types that declared those four options, deprecated since v2.21.0, are gone from the package. A call that passed any of them no longer compiles, and throws an `ArgumentValidationError` about an unrecognized key in JavaScript. Drop them and the call returns the same items as before. diff --git a/docs/public-api/apify-client.api.md b/docs/public-api/apify-client.api.md index 79e0f5c3e..551664a97 100644 --- a/docs/public-api/apify-client.api.md +++ b/docs/public-api/apify-client.api.md @@ -610,6 +610,7 @@ export class ApifyClient { // @public export interface ApifyClientOptions { baseUrl?: string; + compression?: HttpCompressionAlgorithm | HttpCompressor; // (undocumented) maxRetries?: number; minDelayBetweenRetriesMillis?: number; @@ -666,6 +667,20 @@ interface BaseOptions { publicBaseUrl: string; } +// @public +export class BrotliHttpCompressor implements HttpCompressor { + constructor(options?: BrotliHttpCompressorOptions); + // (undocumented) + compress(data: Buffer): Promise; + // (undocumented) + readonly contentEncoding = "br"; +} + +// @public +export interface BrotliHttpCompressorOptions { + quality?: number; +} + // @public export interface Build extends Omit, BuildRePointed { } @@ -2849,6 +2864,20 @@ export interface GetStreamedLogOptions extends TimeoutOptions { toLog?: Log | null | 'default'; } +// @public +export class GzipHttpCompressor implements HttpCompressor { + constructor(options?: GzipHttpCompressorOptions); + // (undocumented) + compress(data: Buffer): Promise; + // (undocumented) + readonly contentEncoding = "gzip"; +} + +// @public +export interface GzipHttpCompressorOptions { + quality?: number; +} + // Not exported by the entry point; reachable only as a referenced type. // @public (undocumented) class HttpClient { @@ -2859,6 +2888,7 @@ class HttpClient { call(config: ApifyRequestConfig): Promise>; // (undocumented) httpAgent?: http.Agent; + readonly httpCompressor: HttpCompressor; // (undocumented) httpsAgent?: https.Agent; // (undocumented) @@ -2883,6 +2913,8 @@ interface HttpClientOptions { // (undocumented) apifyClientStats: Statistics; // (undocumented) + httpCompressor: HttpCompressor; + // (undocumented) logger: Log; // (undocumented) maxRetries: number; @@ -2904,6 +2936,15 @@ interface HttpClientOptions { workflowKey?: string; } +// @public +export type HttpCompressionAlgorithm = 'brotli' | 'gzip'; + +// @public +export interface HttpCompressor { + compress(data: Buffer): Promise; + readonly contentEncoding: string; +} + // @public export class InvalidRequestError extends ApifyApiError { } diff --git a/rsbuild.config.ts b/rsbuild.config.ts index 4ee62f39e..ee369d16a 100644 --- a/rsbuild.config.ts +++ b/rsbuild.config.ts @@ -1,7 +1,7 @@ import { defineConfig } from '@rsbuild/core'; import { pluginNodePolyfill } from '@rsbuild/plugin-node-polyfill'; -const MAX_BUNDLE_BYTES = 350 * 1024; +const MAX_BUNDLE_BYTES = 360 * 1024; // eslint-disable-next-line import/no-default-export export default defineConfig({ @@ -55,7 +55,7 @@ export default defineConfig({ ...config.optimization, splitChunks: false, }; - // A regression guard, not a target: the bundle sits at ~325 kB, so this only fails the + // A regression guard, not a target: the bundle sits at ~338 kB, so this only fails the // build on an unnoticed jump. A `zod` minor is the likeliest cause, since it is a runtime // dependency on a caret range - bumping this constant is the expected response. The // generated response schemas growing with the OpenAPI specification is the other. diff --git a/src/apify_client.ts b/src/apify_client.ts index 918675e93..34a6878e9 100644 --- a/src/apify_client.ts +++ b/src/apify_client.ts @@ -6,6 +6,9 @@ import type { Log } from '@apify/log'; import logger from '@apify/log'; import { HttpClient } from './http_client.js'; +import type { HttpCompressor } from './http_compressors/base.js'; +import type { HttpCompressionAlgorithm } from './http_compressors/resolve.js'; +import { compressionSchema, resolveCompressor } from './http_compressors/resolve.js'; import type { RequestInterceptorFunction } from './interceptors.js'; import { ActorClient } from './resource_clients/actor.js'; import { ActorCollectionClient } from './resource_clients/actor_collection.js'; @@ -42,6 +45,7 @@ import { getEnv, parseArgument } from './utils.js'; const clientOptionsSchema = z.strictObject({ baseUrl: z.string().default('https://api.apify.com'), + compression: compressionSchema.default('brotli'), publicBaseUrl: z.string().default('https://api.apify.com'), maxRetries: z.number().default(8), minDelayBetweenRetriesMillis: z.number().default(500), @@ -101,6 +105,7 @@ export class ApifyClient { const { baseUrl, + compression, publicBaseUrl, maxRetries, minDelayBetweenRetriesMillis, @@ -126,6 +131,7 @@ export class ApifyClient { maxRetries, minDelayBetweenRetriesMillis, requestInterceptors, + httpCompressor: resolveCompressor(compression), timeoutShortSecs, timeoutMediumSecs, timeoutLongSecs, @@ -584,6 +590,13 @@ export class ApifyClient { export interface ApifyClientOptions { /** @default https://api.apify.com */ baseUrl?: string; + /** + * Compression of request bodies: the name of a built-in algorithm, or an {@link HttpCompressor} for a custom + * quality or algorithm. Bodies are compressed only in Node.js, and only when they are large enough to benefit + * and their content type does not already carry its own compression. + * @default 'brotli' + */ + compression?: HttpCompressionAlgorithm | HttpCompressor; /** * @default https://api.apify.com * @since Added in 2.17.0 diff --git a/src/http_client.ts b/src/http_client.ts index 062261e9f..ee24120e6 100644 --- a/src/http_client.ts +++ b/src/http_client.ts @@ -11,8 +11,9 @@ import type { Log } from '@apify/log'; import { concatStreamToBuffer } from '@apify/utilities'; import { ApifyApiError } from './apify_api_error.js'; +import type { HttpCompressor } from './http_compressors/base.js'; import type { RequestInterceptorFunction } from './interceptors.js'; -import { InvalidResponseBodyError, requestInterceptors, responseInterceptors } from './interceptors.js'; +import { createRequestInterceptors, InvalidResponseBodyError, responseInterceptors } from './interceptors.js'; import { runtime } from '#runtime'; import type { Statistics } from './statistics.js'; import type { Timeout, TimeoutTier } from './timeouts.js'; @@ -29,6 +30,12 @@ export class HttpClient { userProvidedRequestInterceptors: RequestInterceptorFunction[]; + /** + * Compressor applied to request bodies that are worth compressing. The request interceptor captures it at + * construction, so it is fixed for the lifetime of the client. + */ + readonly httpCompressor: HttpCompressor; + logger: Log; /** Duration of each timeout tier, in milliseconds. */ @@ -53,6 +60,7 @@ export class HttpClient { this.maxRetries = options.maxRetries; this.minDelayBetweenRetriesMillis = options.minDelayBetweenRetriesMillis; this.userProvidedRequestInterceptors = options.requestInterceptors; + this.httpCompressor = options.httpCompressor; this.timeoutMillis = { short: options.timeoutShortSecs * 1000, medium: options.timeoutMediumSecs * 1000, @@ -120,7 +128,7 @@ export class HttpClient { this.axios.defaults.headers['User-Agent'] = userAgent; } - requestInterceptors.forEach((i) => this.axios.interceptors.request.use(i as any)); + createRequestInterceptors(this.httpCompressor).forEach((i) => this.axios.interceptors.request.use(i as any)); this.userProvidedRequestInterceptors.forEach((i) => this.axios.interceptors.request.use(i as any)); responseInterceptors.forEach((i) => this.axios.interceptors.response.use(i as any)); } @@ -346,6 +354,7 @@ export interface HttpClientOptions { maxRetries: number; minDelayBetweenRetriesMillis: number; requestInterceptors: RequestInterceptorFunction[]; + httpCompressor: HttpCompressor; timeoutShortSecs: number; timeoutMediumSecs: number; timeoutLongSecs: number; diff --git a/src/http_compressors/base.ts b/src/http_compressors/base.ts new file mode 100644 index 000000000..9b333a9b1 --- /dev/null +++ b/src/http_compressors/base.ts @@ -0,0 +1,50 @@ +import { z } from 'zod'; + +/** + * Strategy for compressing HTTP request bodies. + * + * Implement this interface to create a custom compressor. Set `contentEncoding` to the value that should be sent in + * the `Content-Encoding` header and implement `compress()`. The client calls it only for bodies that are large + * enough to benefit from compression, whose content type does not already carry its own compression, and which the + * caller did not send with a `Content-Encoding` of their own. + * + * @example + * ```javascript + * import { ApifyClient } from 'apify-client'; + * + * const identityCompressor = { + * contentEncoding: 'identity', + * + * async compress(data) { + * return data; + * }, + * }; + * + * const client = new ApifyClient({ token: 'my-token', compression: identityCompressor }); + * ``` + */ +export interface HttpCompressor { + /** Value sent in the `Content-Encoding` header, for example `gzip` or `br`. */ + readonly contentEncoding: string; + + /** + * Compresses a request body. + * + * @param data - The raw bytes to compress. + * @returns The compressed bytes. + */ + compress(data: Buffer): Promise; +} + +/** + * Schema accepting anything shaped like an {@link HttpCompressor}. + * @internal + */ +export const httpCompressorSchema = z.custom( + (value) => + typeof value === 'object' && + value !== null && + typeof (value as HttpCompressor).contentEncoding === 'string' && + typeof (value as HttpCompressor).compress === 'function', + { error: 'Invalid input: expected an HttpCompressor' }, +); diff --git a/src/http_compressors/brotli.ts b/src/http_compressors/brotli.ts new file mode 100644 index 000000000..a20b9364e --- /dev/null +++ b/src/http_compressors/brotli.ts @@ -0,0 +1,60 @@ +import { z } from 'zod'; + +import { runtime } from '#runtime'; +import { parseArgument } from '../utils.js'; +import type { HttpCompressor } from './base.js'; + +/** Lowest valid brotli quality, the fastest with the least compression. */ +const MIN_QUALITY = 0; + +/** Highest valid brotli quality, the slowest with the best compression. */ +const MAX_QUALITY = 11; + +/** Middle of the range, where brotli already beats gzip at a comparable CPU cost. */ +const DEFAULT_QUALITY = 6; + +const optionsSchema = z.strictObject({ + quality: z.int().min(MIN_QUALITY).max(MAX_QUALITY).default(DEFAULT_QUALITY), +}); + +/** + * Compresses request bodies using brotli. + * + * Built on the `node:zlib` module, so it works wherever the client runs on Node.js. + * + * @example + * ```javascript + * import { ApifyClient, BrotliHttpCompressor } from 'apify-client'; + * + * const client = new ApifyClient({ token: 'my-token', compression: new BrotliHttpCompressor({ quality: 11 }) }); + * ``` + */ +export class BrotliHttpCompressor implements HttpCompressor { + readonly contentEncoding = 'br'; + + readonly #quality: number; + + /** + * @param options - Compressor options. + * @throws {ArgumentValidationError} If `quality` is out of the valid range. + */ + constructor(options: BrotliHttpCompressorOptions = {}) { + const { quality } = parseArgument(options, optionsSchema, 'BrotliHttpCompressorOptions'); + this.#quality = quality; + } + + async compress(data: Buffer): Promise { + return runtime.compress(data, { algorithm: 'br', quality: this.#quality }); + } +} + +/** + * Options for {@link BrotliHttpCompressor}. + */ +export interface BrotliHttpCompressorOptions { + /** + * Compression level, from `0` (the fastest) to `11` (the best compression). + * @default 6 + */ + quality?: number; +} diff --git a/src/http_compressors/gzip.ts b/src/http_compressors/gzip.ts new file mode 100644 index 000000000..f99587763 --- /dev/null +++ b/src/http_compressors/gzip.ts @@ -0,0 +1,60 @@ +import { z } from 'zod'; + +import { runtime } from '#runtime'; +import { parseArgument } from '../utils.js'; +import type { HttpCompressor } from './base.js'; + +/** Lowest valid gzip quality, the fastest with the least compression. */ +const MIN_QUALITY = 1; + +/** Highest valid gzip quality, the slowest with the best compression. */ +const MAX_QUALITY = 9; + +/** zlib's own default. The levels above it roughly double the CPU cost for about a percent fewer bytes. */ +const DEFAULT_QUALITY = 6; + +const optionsSchema = z.strictObject({ + quality: z.int().min(MIN_QUALITY).max(MAX_QUALITY).default(DEFAULT_QUALITY), +}); + +/** + * Compresses request bodies using gzip. + * + * Built on the `node:zlib` module, so it works wherever the client runs on Node.js. + * + * @example + * ```javascript + * import { ApifyClient, GzipHttpCompressor } from 'apify-client'; + * + * const client = new ApifyClient({ token: 'my-token', compression: new GzipHttpCompressor({ quality: 1 }) }); + * ``` + */ +export class GzipHttpCompressor implements HttpCompressor { + readonly contentEncoding = 'gzip'; + + readonly #quality: number; + + /** + * @param options - Compressor options. + * @throws {ArgumentValidationError} If `quality` is out of the valid range. + */ + constructor(options: GzipHttpCompressorOptions = {}) { + const { quality } = parseArgument(options, optionsSchema, 'GzipHttpCompressorOptions'); + this.#quality = quality; + } + + async compress(data: Buffer): Promise { + return runtime.compress(data, { algorithm: 'gzip', quality: this.#quality }); + } +} + +/** + * Options for {@link GzipHttpCompressor}. + */ +export interface GzipHttpCompressorOptions { + /** + * Compression level, from `1` (the fastest) to `9` (the best compression). + * @default 6 + */ + quality?: number; +} diff --git a/src/http_compressors/index.ts b/src/http_compressors/index.ts new file mode 100644 index 000000000..71efb22ac --- /dev/null +++ b/src/http_compressors/index.ts @@ -0,0 +1,6 @@ +export type { HttpCompressor } from './base.js'; +export { BrotliHttpCompressor } from './brotli.js'; +export type { BrotliHttpCompressorOptions } from './brotli.js'; +export { GzipHttpCompressor } from './gzip.js'; +export type { GzipHttpCompressorOptions } from './gzip.js'; +export type { HttpCompressionAlgorithm } from './resolve.js'; diff --git a/src/http_compressors/resolve.ts b/src/http_compressors/resolve.ts new file mode 100644 index 000000000..d365c0975 --- /dev/null +++ b/src/http_compressors/resolve.ts @@ -0,0 +1,39 @@ +import { z } from 'zod'; + +import type { HttpCompressor } from './base.js'; +import { httpCompressorSchema } from './base.js'; +import { BrotliHttpCompressor } from './brotli.js'; +import { GzipHttpCompressor } from './gzip.js'; + +/** + * Compression algorithms the `compression` option of {@link ApifyClient} accepts by name. Each name selects the + * matching built-in compressor with its default quality: `'brotli'` a {@link BrotliHttpCompressor}, `'gzip'` a + * {@link GzipHttpCompressor}. The Apify API also accepts `deflate` and `identity`, which a custom + * {@link HttpCompressor} covers. + */ +export type HttpCompressionAlgorithm = 'brotli' | 'gzip'; + +/** + * Schema of the `compression` option: an algorithm name or an {@link HttpCompressor}. + * @internal + */ +export const compressionSchema = z.union([ + z.enum(['brotli', 'gzip'] as const satisfies readonly HttpCompressionAlgorithm[]), + httpCompressorSchema, +]); + +/** + * Turns the `compression` option into a ready-to-use {@link HttpCompressor}: a name into the matching built-in + * compressor with its default quality, a compressor into itself. + * @internal + */ +export function resolveCompressor(compression: HttpCompressionAlgorithm | HttpCompressor): HttpCompressor { + if (typeof compression !== 'string') return compression; + + switch (compression) { + case 'gzip': + return new GzipHttpCompressor(); + case 'brotli': + return new BrotliHttpCompressor(); + } +} diff --git a/src/index.ts b/src/index.ts index 1377384e1..fdd2838ee 100644 --- a/src/index.ts +++ b/src/index.ts @@ -30,5 +30,6 @@ export * from './apify_api_error.js'; export { ArgumentValidationError } from '@apify/validations'; export * from './response_validation_error.js'; export { InvalidResponseBodyError } from './interceptors.js'; +export * from './http_compressors/index.js'; export type { PaginatedList, Dictionary } from './utils.js'; export type { Timeout, TimeoutOptions, TimeoutTier } from './timeouts.js'; diff --git a/src/interceptors.ts b/src/interceptors.ts index d0dd74c53..7ed0dc566 100644 --- a/src/interceptors.ts +++ b/src/interceptors.ts @@ -5,7 +5,9 @@ import type { JsonObject } from 'type-fest'; import { maybeParseBody } from './body_parser.js'; import type { ApifyRequestConfig, ApifyResponse } from './http_client.js'; -import { isCompressibleContentType, maybeCompressValue } from './utils.js'; +import type { HttpCompressor } from './http_compressors/base.js'; +import { runtime } from '#runtime'; +import { isCompressibleContentType, MIN_COMPRESS_BYTES, toBytes } from './utils.js'; /** * This error exists for the quite common situation, where only a partial JSON response is received and @@ -42,6 +44,17 @@ function getHeader(config: ApifyRequestConfig, name: string): string | undefined return typeof value === 'string' ? value : undefined; } +/** Removes a request header regardless of the casing it was set with. */ +function deleteHeader(config: ApifyRequestConfig, name: string): void { + const { headers } = config; + if (!headers) return; + + const wanted = name.toLowerCase(); + for (const key of Object.keys(headers)) { + if (key.toLowerCase() === wanted) delete headers[key]; + } +} + function serializeRequest(config: ApifyRequestConfig): ApifyRequestConfig { // A string body with an explicit content type is already serialized and goes out as it is. The axios default // transform would otherwise parse a JSON one in full just to check that it is valid, which for a body assembled @@ -95,18 +108,34 @@ function stringifyWithFunctions(obj: JsonObject) { }); } -async function maybeCompressRequest(config: ApifyRequestConfig): Promise { +/** + * Compresses the request body with the client's compressor and labels it with the compressor's `Content-Encoding`. + * + * Runs after `serializeRequest`, so a JSON body is already a string here. A caller-set `Content-Encoding` is + * forwarded verbatim, which is how a pre-encoded body is uploaded, and `Content-Encoding: identity` opts a single + * request out of compression. The built-in compressors need `node:zlib`, so a body sent from a browser or an edge + * runtime goes out as it is. + * + * Compressing changes the body length, so any `Content-Length` the caller set describes the wrong body and has to + * go. Axios keeps a caller-set one over the size it computes, which would stall the request until it times out. + */ +async function maybeCompressRequest( + config: ApifyRequestConfig, + compressor: HttpCompressor, +): Promise { + if (!runtime.isNode) return config; + + const bytes = toBytes(config.data); + if (!bytes || bytes.byteLength < MIN_COMPRESS_BYTES) return config; + // A caller-supplied encoding means the body is already encoded and the header describes it, so leave both alone. if (getHeader(config, 'content-encoding')) return config; - if (!isCompressibleContentType(getHeader(config, 'content-type'))) return config; - const maybeCompressed = await maybeCompressValue(config.data); - if (maybeCompressed) { - config.headers ??= {}; - config.headers['content-encoding'] = maybeCompressed.encoding; - config.data = maybeCompressed.data; - } + config.data = await compressor.compress(Buffer.from(bytes.buffer, bytes.byteOffset, bytes.byteLength)); + config.headers ??= {}; + config.headers['content-encoding'] = compressor.contentEncoding; + deleteHeader(config, 'content-length'); return config; } @@ -140,9 +169,14 @@ function parseResponseData(response: ApifyResponse): ApifyResponse { export type RequestInterceptorFunction = Parameters['use']>[0]; export type ResponseInterceptorFunction = Parameters['use']>[0]; -export const requestInterceptors: RequestInterceptorFunction[] = [ - maybeCompressRequest, - serializeRequest, - ensureHeadersPrototype, -]; +/** + * The client's own request interceptors, in registration order. Axios runs request interceptors in the reverse + * order of registration, so the body is serialized before it is compressed, and interceptors registered later, + * such as the user-provided ones, run before both. + * @internal + */ +export function createRequestInterceptors(compressor: HttpCompressor): RequestInterceptorFunction[] { + return [async (config) => maybeCompressRequest(config, compressor), serializeRequest, ensureHeadersPrototype]; +} + export const responseInterceptors: ResponseInterceptorFunction[] = [parseResponseData]; diff --git a/src/runtime/node.ts b/src/runtime/node.ts index 2dbd86ccf..468fd8272 100644 --- a/src/runtime/node.ts +++ b/src/runtime/node.ts @@ -1,7 +1,6 @@ import type http from 'node:http'; import type { Socket } from 'node:net'; import os from 'node:os'; -import { promisify } from 'node:util'; import { brotliCompress, constants, gzip } from 'node:zlib'; import type { Runtime } from './types.js'; @@ -15,23 +14,15 @@ export const runtime: Runtime = { platform: `${os.platform()}; Node/${process.version}`, - async compress(data) { - try { - // `promisify()` belongs inside the fallback chain: a partial `node:zlib` can export - // `brotliCompress` without implementing it, and `promisify(undefined)` throws. At module scope - // that would fail the whole import instead of falling through to gzip. - const options = { params: { [constants.BROTLI_PARAM_QUALITY]: 6 } }; - return { data: await promisify(brotliCompress)(data, options), encoding: 'br' }; - } catch { - // Runtimes that only provide a partial `node:zlib` (Node.js compatibility shims) may not implement - // brotli, but usually do implement gzip. - } - - try { - return { data: await promisify(gzip)(data), encoding: 'gzip' }; - } catch { - return undefined; - } + async compress(data, { algorithm, quality }) { + return new Promise((resolve, reject) => { + const done = (error: Error | null, result: Buffer) => (error ? reject(error) : resolve(result)); + if (algorithm === 'br') { + brotliCompress(data, { params: { [constants.BROTLI_PARAM_QUALITY]: quality } }, done); + } else { + gzip(data, { level: quality }, done); + } + }); }, async createHttpAgents({ timeoutMillis }) { diff --git a/src/runtime/types.ts b/src/runtime/types.ts index 19923f2d9..a1fbdcce3 100644 --- a/src/runtime/types.ts +++ b/src/runtime/types.ts @@ -2,12 +2,14 @@ import type http from 'node:http'; import type https from 'node:https'; /** - * A request body compressed for transport, with the `content-encoding` value that declares it. + * What to compress a request body with, as the built-in `HttpCompressor` implementations ask for it. * @internal */ -export interface CompressedValue { - data: Uint8Array; - encoding: 'br' | 'gzip'; +export interface CompressionOptions { + /** Algorithm to compress with, named by the `Content-Encoding` value it produces. */ + algorithm: 'br' | 'gzip'; + /** Compression level, in the range the algorithm defines. */ + quality: number; } /** @@ -28,9 +30,8 @@ export interface HttpAgents { export interface Runtime { /** * Whether the Node.js implementation was selected, which the `node` condition settles when `#runtime` is - * resolved - so a Node.js application bundled for a browser or a neutral target reports `false`. The - * features that need a Node.js `Readable` response - log streaming and the `stream` record option - are - * gated on it. + * resolved - so a Node.js application bundled for a browser or a neutral target reports `false`. Log + * streaming, the `stream` record option and request body compression are gated on it. */ isNode: boolean; /** @@ -39,10 +40,11 @@ export interface Runtime { */ platform: string | undefined; /** - * Compresses a request body, or resolves to `undefined` where compression is not available. Compression - * is a best-effort optimization, so it never throws. + * Compresses a request body. The built-in compressors go through it, so a bundle for a browser or an edge + * runtime carries no `node:zlib`. Throws where the runtime has no compression, which the client never + * reaches, since it compresses only where {@link isNode} holds. */ - compress(data: Uint8Array): Promise; + compress(data: Uint8Array, options: CompressionOptions): Promise; /** * Creates the agents axios's Node.js adapter sends requests through, or resolves to `undefined` where * axios does not use agents. diff --git a/src/runtime/web.ts b/src/runtime/web.ts index 64a017f23..2f5cd55db 100644 --- a/src/runtime/web.ts +++ b/src/runtime/web.ts @@ -9,10 +9,11 @@ export const runtime: Runtime = { // Browsers do not let a page set the `User-Agent` header. platform: undefined, - // No request compression: brotli has no Web API, and a `content-encoding` request header is not on the - // list of headers the Apify API allows in a cross-origin request, so a browser would fail the preflight. - async compress() { - return undefined; + // No request compression: brotli and gzip have no Web API, and a `content-encoding` request header is not + // on the list of headers the Apify API allows in a cross-origin request, so a browser would fail the + // preflight. The client compresses in Node.js alone, so nothing reaches this. + async compress(): Promise { + throw new Error('Request body compression is only available in Node.js.'); }, // The XHR and fetch adapters of axios do not use agents. diff --git a/src/utils.ts b/src/utils.ts index 0df27bf2b..85d6e004c 100644 --- a/src/utils.ts +++ b/src/utils.ts @@ -7,8 +7,6 @@ import type { ApifyApiError } from './apify_api_error.js'; import { NotFoundError } from './apify_api_error.js'; import { parseArgument } from '@apify/validations'; import type { ApifyResponse } from './http_client.js'; -import type { CompressedValue } from './runtime/types.js'; -import { runtime } from '#runtime'; import { ResponseValidationError } from './response_validation_error.js'; import type { RequestQueueClientListRequestsOptions, @@ -19,7 +17,14 @@ import type { WebhookUpdateData } from './resource_clients/webhook.js'; // @ts-ignore if we enable `resolveJsonModule`, we end up with a `src` folder in `dist` import packageJson from '../package.json' with { type: 'json' }; -const MIN_COMPRESS_BYTES = 1024; +/** + * Smallest request body, in bytes, that is worth compressing. A smaller body already fits in a single network + * packet, so compressing it costs CPU time without saving a round trip, and the compression format itself can + * make it larger. + * @internal + */ +export const MIN_COMPRESS_BYTES = 1024; + const textEncoder = new TextEncoder(); // Only the version, so a bundler can drop the rest of the manifest. @@ -197,8 +202,9 @@ export function concatBytes(chunks: Uint8Array[]): Uint8Array { /** * Views a request body as bytes: a string is UTF-8 encoded, binary values are viewed in place. Anything else * - a stream, a `Blob`, form data - is `undefined`. + * @internal */ -function toBytes(value: unknown): Uint8Array | undefined { +export function toBytes(value: unknown): Uint8Array | undefined { if (typeof value === 'string') return textEncoder.encode(value); if (!isBuffer(value)) return undefined; return ArrayBuffer.isView(value) @@ -228,19 +234,6 @@ export function isCompressibleContentType(contentType?: string): boolean { return !ALREADY_COMPRESSED_MEDIA_TYPE_PREFIXES.some((prefix) => mediaType.startsWith(prefix)); } -/** - * Compresses the passed value with the runtime's best available algorithm. Returns `undefined` if the data - * is too small, is not a string or binary value, or if the runtime does not offer compression. - */ -export async function maybeCompressValue(value: unknown): Promise { - // Request compression is not that important so let's - // skip it instead of throwing for unsupported types. - const bytes = toBytes(value); - if (!bytes || bytes.byteLength < MIN_COMPRESS_BYTES) return undefined; - - return runtime.compress(bytes); -} - /** * Reads an environment variable, on runtimes that have them. */ diff --git a/test/http_compressors.test.ts b/test/http_compressors.test.ts new file mode 100644 index 000000000..b941c336a --- /dev/null +++ b/test/http_compressors.test.ts @@ -0,0 +1,245 @@ +import type { AddressInfo } from 'node:net'; +import { brotliCompressSync, brotliDecompressSync, constants, gunzipSync, gzipSync } from 'node:zlib'; + +import type { HttpCompressor } from 'apify-client'; +import { ApifyClient, ArgumentValidationError, BrotliHttpCompressor, GzipHttpCompressor } from 'apify-client'; +import { afterAll, beforeAll, describe, expect, test, vi } from 'vitest'; + +import { resolveCompressor } from '../src/http_compressors/resolve.js'; +import { MIN_COMPRESS_BYTES } from '../src/utils.js'; +import { mockServer } from './mock_server/server.js'; + +const payload = Buffer.from('hello world'); + +/** Long and repetitive, so the compression quality changes the output bytes. */ +const repetitivePayload = Buffer.from('lorem ipsum dolor sit amet '.repeat(500)); + +describe('GzipHttpCompressor', () => { + test('reports gzip as its content encoding', () => { + expect(new GzipHttpCompressor().contentEncoding).toBe('gzip'); + }); + + test.each([ + { name: 'the fastest', quality: 1 }, + { name: 'the best', quality: 9 }, + ])('round-trips data at $name quality', async ({ quality }) => { + const compressed = await new GzipHttpCompressor({ quality }).compress(payload); + expect(gunzipSync(compressed)).toEqual(payload); + }); + + test('passes the quality through to zlib', async () => { + const fastest = await new GzipHttpCompressor({ quality: 1 }).compress(repetitivePayload); + const best = await new GzipHttpCompressor({ quality: 9 }).compress(repetitivePayload); + + expect(fastest).toEqual(gzipSync(repetitivePayload, { level: 1 })); + expect(best).toEqual(gzipSync(repetitivePayload, { level: 9 })); + }); + + test('defaults to level 6', async () => { + const compressed = await new GzipHttpCompressor().compress(repetitivePayload); + expect(compressed).toEqual(gzipSync(repetitivePayload, { level: 6 })); + }); + + test.each([ + { name: 'below the minimum', quality: 0 }, + { name: 'above the maximum', quality: 10 }, + { name: 'negative', quality: -1 }, + { name: 'fractional', quality: 5.5 }, + ])('rejects a quality $name', ({ quality }) => { + expect(() => new GzipHttpCompressor({ quality })).toThrow(ArgumentValidationError); + }); +}); + +describe('BrotliHttpCompressor', () => { + test('reports br as its content encoding', () => { + expect(new BrotliHttpCompressor().contentEncoding).toBe('br'); + }); + + test.each([ + { name: 'the fastest', quality: 0 }, + { name: 'the best', quality: 11 }, + ])('round-trips data at $name quality', async ({ quality }) => { + const compressed = await new BrotliHttpCompressor({ quality }).compress(payload); + expect(brotliDecompressSync(compressed)).toEqual(payload); + }); + + test('passes the quality through to zlib', async () => { + const brotli = (quality: number) => + brotliCompressSync(repetitivePayload, { params: { [constants.BROTLI_PARAM_QUALITY]: quality } }); + + expect(await new BrotliHttpCompressor({ quality: 0 }).compress(repetitivePayload)).toEqual(brotli(0)); + expect(await new BrotliHttpCompressor({ quality: 11 }).compress(repetitivePayload)).toEqual(brotli(11)); + }); + + test('defaults to quality 6', async () => { + const compressed = await new BrotliHttpCompressor().compress(repetitivePayload); + expect(compressed).toEqual( + brotliCompressSync(repetitivePayload, { params: { [constants.BROTLI_PARAM_QUALITY]: 6 } }), + ); + }); + + test.each([ + { name: 'negative', quality: -1 }, + { name: 'above the maximum', quality: 12 }, + { name: 'fractional', quality: 5.5 }, + ])('rejects a quality $name', ({ quality }) => { + expect(() => new BrotliHttpCompressor({ quality })).toThrow(ArgumentValidationError); + }); +}); + +describe('resolveCompressor()', () => { + test("resolves 'gzip' to a GzipHttpCompressor", () => { + expect(resolveCompressor('gzip')).toBeInstanceOf(GzipHttpCompressor); + }); + + test("resolves 'brotli' to a BrotliHttpCompressor", () => { + expect(resolveCompressor('brotli')).toBeInstanceOf(BrotliHttpCompressor); + }); + + test('passes a compressor through unchanged', () => { + const compressor = new BrotliHttpCompressor({ quality: 11 }); + expect(resolveCompressor(compressor)).toBe(compressor); + }); +}); + +describe('ApifyClient compression option', () => { + let baseUrl: string; + + beforeAll(async () => { + const server = await mockServer.start(); + baseUrl = `http://localhost:${(server.address() as AddressInfo).port}`; + }); + + afterAll(async () => { + await mockServer.close(); + }); + + class IdentityCompressor implements HttpCompressor { + readonly contentEncoding = 'identity'; + + async compress(data: Buffer): Promise { + return data; + } + } + + // Serializes to well above the compression threshold, so the body goes through the compressor. + const largeBody = Array.from({ length: 200 }, (_, index) => ({ index, url: `https://example.com/item/${index}` })); + + test('defaults to brotli', () => { + expect(new ApifyClient().httpClient.httpCompressor).toBeInstanceOf(BrotliHttpCompressor); + }); + + test.each([ + { compression: 'gzip', contentEncoding: 'gzip' }, + { compression: 'brotli', contentEncoding: 'br' }, + ] as const)('sends a large body compressed with $compression', async ({ compression, contentEncoding }) => { + const client = new ApifyClient({ baseUrl, compression, maxRetries: 0 }); + + await client.dataset('some-id').pushItems(largeBody); + + // The mock server decodes the body by its `Content-Encoding`, so a body in any other encoding would not + // parse back into the items. + const request = mockServer.getLastRequest(); + expect(request?.headers['content-encoding']).toBe(contentEncoding); + expect(request?.body).toEqual(largeBody); + }); + + test('replaces a caller-set content-length with the size of the compressed body', async () => { + const client = new ApifyClient({ baseUrl, maxRetries: 0 }); + const body = JSON.stringify(largeBody); + + await client.httpClient.call({ + url: `${baseUrl}/v2/key-value-stores/some-id/records/some-key`, + method: 'PUT', + data: body, + headers: { 'content-type': 'application/json', 'content-length': String(Buffer.byteLength(body)) }, + // A stale content-length leaves the server waiting for bytes that never arrive, so a short timeout + // turns that into a fast failure instead of a hang. + timeoutSecs: 5, + }); + + const request = mockServer.getLastRequest(); + expect(request?.headers['content-encoding']).toBe('br'); + expect(Number(request?.headers['content-length'])).toBeLessThan(Buffer.byteLength(body)); + expect(request?.body).toEqual(largeBody); + }); + + test('sends a large body through a custom compressor and labels it with its content encoding', async () => { + const compressor = new IdentityCompressor(); + const compress = vi.spyOn(compressor, 'compress'); + const client = new ApifyClient({ baseUrl, compression: compressor, maxRetries: 0 }); + expect(client.httpClient.httpCompressor).toBe(compressor); + + await client.dataset('some-id').pushItems(largeBody); + + expect(compress).toHaveBeenCalledOnce(); + const [data] = compress.mock.calls[0]; + expect(Buffer.isBuffer(data)).toBe(true); + expect(JSON.parse(data.toString())).toEqual(largeBody); + const request = mockServer.getLastRequest(); + expect(request?.headers['content-encoding']).toBe('identity'); + expect(request?.body).toEqual(largeBody); + }); + + test('fails the request with the error the compressor throws', async () => { + const compressor = new IdentityCompressor(); + const compress = vi.spyOn(compressor, 'compress').mockRejectedValue(new Error('compression failed')); + const client = new ApifyClient({ baseUrl, compression: compressor, maxRetries: 2 }); + + await expect(client.dataset('some-id').pushItems(largeBody)).rejects.toThrow('compression failed'); + // Retries are on, so a single call is what proves a compressor error is not retried. + expect(compress).toHaveBeenCalledOnce(); + }); + + test.each([ + { name: 'a Buffer', value: Buffer.alloc(4096, 'a') }, + { name: 'a typed array', value: new Uint8Array(4096).fill(0x61) }, + { name: 'an ArrayBuffer', value: new Uint8Array(4096).fill(0x61).buffer }, + ])('compresses a large binary body passed as $name', async ({ value }) => { + const client = new ApifyClient({ baseUrl, maxRetries: 0 }); + + await client + .keyValueStore('some-id') + .setRecord({ key: 'some-key', value, contentType: 'application/octet-stream' }); + + expect(mockServer.getLastRequest()?.headers['content-encoding']).toBe('br'); + }); + + test('compresses a body of exactly the threshold size', async () => { + const compressor = new IdentityCompressor(); + const compress = vi.spyOn(compressor, 'compress'); + const client = new ApifyClient({ baseUrl, compression: compressor, maxRetries: 0 }); + + await client + .keyValueStore('some-id') + .setRecord({ key: 'some-key', value: Buffer.alloc(MIN_COMPRESS_BYTES, 'a') }); + + expect(compress).toHaveBeenCalledOnce(); + expect(mockServer.getLastRequest()?.headers['content-encoding']).toBe('identity'); + }); + + test.each([ + { + name: 'a body below the threshold', + value: Buffer.alloc(MIN_COMPRESS_BYTES - 1, 'a'), + contentType: 'application/octet-stream', + }, + { name: 'an already-compressed content type', value: Buffer.alloc(4096, 'a'), contentType: 'image/png' }, + ])('leaves the compressor out for $name', async ({ value, contentType }) => { + const compressor = new IdentityCompressor(); + const compress = vi.spyOn(compressor, 'compress'); + const client = new ApifyClient({ baseUrl, compression: compressor, maxRetries: 0 }); + + await client.keyValueStore('some-id').setRecord({ key: 'some-key', value, contentType }); + + expect(compress).not.toHaveBeenCalled(); + expect(mockServer.getLastRequest()?.headers['content-encoding']).toBeUndefined(); + }); + + test.each([ + { name: 'an unknown algorithm', compression: 'deflate' }, + { name: 'an object that is not a compressor', compression: { contentEncoding: 'gzip' } }, + ])('rejects $name', ({ compression }) => { + expect(() => new ApifyClient({ compression: compression as any })).toThrow(ArgumentValidationError); + }); +}); diff --git a/test/runtime.test.ts b/test/runtime.test.ts index da47e1a53..3b19eeb8f 100644 --- a/test/runtime.test.ts +++ b/test/runtime.test.ts @@ -3,7 +3,7 @@ import http from 'node:http'; import os from 'node:os'; import { resolve } from 'node:path'; import type * as Zlib from 'node:zlib'; -import { gunzipSync } from 'node:zlib'; +import { brotliDecompressSync, gunzipSync } from 'node:zlib'; import { build } from 'esbuild'; import { describe, expect, test, vi } from 'vitest'; @@ -19,12 +19,19 @@ describe('Node.js runtime', () => { test('compresses with brotli', async () => { const data = new TextEncoder().encode('x'.repeat(2048)); - const compressed = await nodeRuntime.compress(data); - expect(compressed?.encoding).toBe('br'); - expect(compressed!.data.byteLength).toBeLessThan(data.byteLength); + const compressed = await nodeRuntime.compress(data, { algorithm: 'br', quality: 6 }); + expect(brotliDecompressSync(compressed)).toEqual(Buffer.from(data)); + expect(compressed.byteLength).toBeLessThan(data.byteLength); }); - test('falls back to gzip where `node:zlib` does not implement brotli', async () => { + test('compresses with gzip', async () => { + const data = new TextEncoder().encode('x'.repeat(2048)); + const compressed = await nodeRuntime.compress(data, { algorithm: 'gzip', quality: 6 }); + expect(gunzipSync(compressed)).toEqual(Buffer.from(data)); + expect(compressed.byteLength).toBeLessThan(data.byteLength); + }); + + test('rejects where `node:zlib` does not implement the algorithm', async () => { vi.resetModules(); vi.doMock('node:zlib', async () => { const zlib = await vi.importActual('node:zlib'); @@ -34,9 +41,7 @@ describe('Node.js runtime', () => { try { const { runtime } = await import('../src/runtime/node.js'); const data = new TextEncoder().encode('x'.repeat(2048)); - const compressed = await runtime.compress(data); - expect(compressed?.encoding).toBe('gzip'); - expect(gunzipSync(compressed!.data)).toEqual(Buffer.from(data)); + await expect(runtime.compress(data, { algorithm: 'br', quality: 6 })).rejects.toThrow(TypeError); } finally { vi.doUnmock('node:zlib'); vi.resetModules(); @@ -58,7 +63,9 @@ describe('Web API runtime', () => { test('offers no platform, compression or agents', async () => { expect(webRuntime.isNode).toBe(false); expect(webRuntime.platform).toBeUndefined(); - await expect(webRuntime.compress(new Uint8Array(4096))).resolves.toBeUndefined(); + await expect(webRuntime.compress(new Uint8Array(4096), { algorithm: 'br', quality: 6 })).rejects.toThrow( + 'only available in Node.js', + ); await expect(webRuntime.createHttpAgents({ timeoutMillis: 1000 })).resolves.toBeUndefined(); }); }); diff --git a/test/utils.test.ts b/test/utils.test.ts index 1f02d707d..0678f82ad 100644 --- a/test/utils.test.ts +++ b/test/utils.test.ts @@ -98,38 +98,6 @@ describe('utils.parseResponse()', () => { }); }); -describe('utils.maybeCompressValue()', () => { - test('returns undefined for small values', async () => { - expect(await utils.maybeCompressValue('small')).toBeUndefined(); - }); - - test('returns undefined for non-string non-binary values', async () => { - expect(await utils.maybeCompressValue({ foo: 'bar' })).toBeUndefined(); - expect(await utils.maybeCompressValue(Readable.from(['x'.repeat(2048)]))).toBeUndefined(); - }); - - test('compresses large string using brotli in Node.js', async () => { - const largeValue = 'x'.repeat(2048); - const result = await utils.maybeCompressValue(largeValue); - expect(result).not.toBeUndefined(); - expect(result!.encoding).toBe('br'); - expect(result!.data).toBeInstanceOf(Uint8Array); - expect(result!.data.byteLength).toBeLessThan(Buffer.byteLength(largeValue)); - }); - - test.each([ - { kind: 'Buffer', value: Buffer.alloc(2048, 'a') }, - { kind: 'Uint8Array', value: new Uint8Array(2048).fill(0x61) }, - { kind: 'ArrayBuffer', value: new Uint8Array(2048).fill(0x61).buffer }, - ])('compresses a large $kind using brotli in Node.js', async ({ value }) => { - const result = await utils.maybeCompressValue(value); - expect(result).not.toBeUndefined(); - expect(result!.encoding).toBe('br'); - expect(result!.data).toBeInstanceOf(Uint8Array); - expect(result!.data.byteLength).toBeLessThan(2048); - }); -}); - describe('utils.bytesToBase64()', () => { test('matches the Node.js encoding for an input longer than one slice', () => { const bytes = new Uint8Array(100_000).map((_, i) => i * 7919);