Skip to content

Avoid double JSON serialization in batchAddRequests #972

Closed

Description

@vdusek

Follow-up from the Python client's equivalent fix — see apify/apify-client-python#953 and the discussion there. The JS client has the same inefficiency.

In batchAddRequests, every request is JSON-serialized twice:

  1. Once to measure payload size when splitting into batches — sliceArrayByByteLength (src/utils.ts) calls JSON.stringify on the whole slice, and on the splitting path on each request individually.
  2. Once more when sending — the batch is passed as data to httpClient.call, and the serializeRequest axios interceptor (src/interceptors.ts) stringifies the same objects again to build the request body.

That's redundant CPU work in a hot path — Crawlee's RequestQueue goes through this method.

Suggested fix, mirroring apify/apify-client-python#953: serialize each request once up front, split batches by the serialized byte lengths (counting the , separators and the [] brackets), and send the pre-assembled JSON string as the body with an explicit content-type: application/json header so the interceptor doesn't re-serialize it. As a side effect this also fixes the current measurement slightly undercounting the actual body: it accounts for the [] brackets but not the commas between items (harmless today thanks to the safety buffer, but fixed for free by this approach).

✍️ Drafted by Claude Code

Activity

  1. added
    performanceIssues related to performance.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Jul 17, 2026
  2. added this to the 3.0 milestone on Jul 28, 2026
  3. added a commit that references this issue on Sep 14, 2026
    2ca4b04
  4. vdusek commented on Sep 14, 2026

    @vdusek
    ContributorAuthor

    resolved by #1051

  5. added a commit that references this issue on Sep 30, 2026
    bddfd7f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

performanceIssues related to performance.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions