You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Avoid double JSON serialization in batchAddRequests #972
In batchAddRequests, every request is JSON-serialized twice:
Once to measure payload size when splitting into batches — sliceArrayByByteLength (src/utils.ts) calls JSON.stringify on the whole slice, and on the splitting path on each request individually.
Once more when sending — the batch is passed as data to httpClient.call, and the serializeRequest axios interceptor (src/interceptors.ts) stringifies the same objects again to build the request body.
That's redundant CPU work in a hot path — Crawlee's RequestQueue goes through this method.
Suggested fix, mirroring apify/apify-client-python#953: serialize each request once up front, split batches by the serialized byte lengths (counting the , separators and the [] brackets), and send the pre-assembled JSON string as the body with an explicit content-type: application/json header so the interceptor doesn't re-serialize it. As a side effect this also fixes the current measurement slightly undercounting the actual body: it accounts for the [] brackets but not the commas between items (harmless today thanks to the safety buffer, but fixed for free by this approach).
Activity
perf!: serialize requests once in batchAddRequests (#1051)
vdusek commentedon Sep 14, 2026
resolved by #1051
perf!: serialize requests once in batchAddRequests (#1051)