[sentry-junior]

sentry snapshots upload now uses objectstore's batch endpoint, so a snapshot suite takes a few HTTP requests instead of one HEAD and one PUT per image. Before this change, suites of about 60 images got HTTP 429 from the Sentry objectstore proxy.

Root cause

The upload sent one HEAD per image (8 at a time), then one PUT per missing image. Example failure from getsentry/sentry-evals (job), with [email protected]:

[info] [snapshots.upload] Uploading 62 image(s)...
Error: Objectstore HEAD failed
  Endpoint: https://us.sentry.io/api/0/organizations/1/objectstore/v1/objects/preprod_snapshots/...
  Too Many Requests

The 429 arrives about 1.4s after the upload starts. The same error appears in two other recent failed runs of that workflow.

Those requests go through ObjectstoreEndpoint in getsentry/sentry. It sets authentication_classes = () and RateLimitConfig(group="CLI"). Since Django sees no authenticated user, the request should fall into the IP category of the CLI group, which is SENTRY_RATELIMITER_DEFAULT. That is 40 requests per second in getsentry. This category mapping comes from reading the code. I did not confirm it from the 429 response body, because the CLI only prints the status text.

The Rust sentry-cli does not hit this because it uses objectstore-client's Session::many(). That sends HEADs and small PUTs as multipart requests to /v1/objects:batch/{usecase}/{scope}/. The proxy accepts that path (is_valid_object_path allows objects:*).

Change

• lib/objectstore.ts: adds a small client for the batch protocol. It encodes per-part headers like x-sn-batch-operation-kind and x-sn-batch-operation-key, escaping keys the way encode_header_value does. It parses the multipart response by x-sn-batch-operation-index and x-sn-batch-operation-status.
  • findExistingObjects: sends batched HEADs, at most 1000 per request.
  • putObjects: inserts files up to 1 MB in batches of up to 1000 operations and 100 MB. Batches run one at a time to keep memory bounded. Larger files still get individual PUTs, 5 at a time. These limits match the Rust client and the server's MAX_OPERATIONS and MAX_FIELD_SIZE.
  • A batch-level failure, a per-operation error other than 404 on HEAD, or a missing per-operation result raises an ApiError.
• snapshots/upload.ts: uses the two functions above. runPooled and UPLOAD_CONCURRENCY are removed, and so is objectExists, which nothing else used.

Testing

• Unit tests cover the batch request shape (checked by decoding it as multipart form data), response parsing, error paths, and the split between batched and individual uploads. The snapshots upload command tests are updated to the new functions.
• I ran an end-to-end script against a real objectstore server (ghcr.io/getsentry/objectstore:nightly, filesystem backend) with 62 images of about 20 KB plus one 1.5 MB image:
  • 4 HTTP requests in total: 1 HEAD batch, 1 insert batch, 1 individual PUT for the large file, and 1 HEAD batch to recheck. The old code would send up to 126.
  • All 63 objects read back byte-identical.
  • x-sn-expiration was stored on both batched and individual uploads.
• tsc --noEmit and pnpm run lint pass. I ran the CLI vitest suites for objectstore and snapshots, not the full root pnpm run test.

Not covered

• I did not test through the real us.sentry.io proxy. Proxy support for the batch path comes from reading the code and from the Rust CLI using it.
• Objectstore also rate-limits each operation inside a batch. This change removes the per-request pressure on the Sentry proxy, which is where the 429s in the logs come from. It does not add retry or backoff on 429.
• The original fix was written against the archived getsentry/cli and ported here. Only formatting and lint-suppression comments differ.

via jamieq.

--

View Session [Sentry]

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
sentry-local	Ready	Preview	Nov 7, 2026 12:18am UTC

[BYK]

1. Keep concurrent uploads still (we can do concurrent batches?)
2. Move all this logic to @sentry/api package and use it from there

[sentry-junior]

Thanks @BYK.

1. Concurrent batches — done in 5459297. Up to 4 batch requests run at once for both the existence checks and the uploads, and files over 1 MB upload in parallel with the batches (5 at a time). I lowered the per-batch body limit from 100 MB to 25 MB, so the upload bodies held in memory stay around 100 MB, the same as before. I ran it against a local objectstore:nightly server with 2,483 existence checks and 83 files (about 77 MB). The checks went out as 3 batches, all in flight at once. The uploads took 6 requests (3 batches plus 3 single uploads), with 4 in flight at most. Every object read back byte-identical. I also added a unit test that checks batches are in flight at the same time. CLI typecheck and lint pass, and so do the objectstore and snapshots tests.

2. Move to @sentry/api — in progress. That package is published from getsentry/sentry-api-schema, so this needs a PR there and a release before this PR can import it. I'm opening that PR now, with the fetch function passed in so the CLI can keep its custom-CA fetch. Once it's released I'll bump @sentry/api here and swap src/lib/objectstore.ts for the package version.

[sentry-junior]

Follow-up on item 2: I opened getsentry/sentry-api-schema#133. It adds the objectstore client, exported from @sentry/api/client. You pass in your own fetch, it doesn't use Node or Buffer APIs, and it adds no dependencies. The batch limits and the concurrent batches are the same as 5459297. Once it merges and @sentry/api gets a release, I'll bump the package here and replace src/lib/objectstore.ts with the package version.