Skip to content

fix(cli): Split paginate() limits evenly and return Sentry-compatible cursors - #1642

Merged
BYK merged 5 commits into
mainfrom
fix/cli-paginate-cursor-compat
Oct 23, 2026
Merged

BYK merged 5 commits into
mainfrom
fix/cli-paginate-cursor-compat

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #1625. This fixes two things before paginate()'s cursors ship in a release.

Overfetching on the last page. paginate() used 100-row pages for any --limit over 100. So --limit 201 made three requests of 100 and threw away 99 rows. A fresh walk now uses the same, minimal number of requests and splits the limit evenly across them: 201 → 3 × 67, 250 → 3 × 84, 101 → 2 × 51. A property test checks the request count (ceil(limit / 100)) and that a walk overfetches by less than one row per request.

Cursors that work across CLI versions. #1625 returned CLI-only cli1~<perPage>~<skip>~<server cursor> cursors. Older CLIs send any cursor straight to the API, so they couldn't use those, and the new CLI couldn't trust the raw cursors older CLIs stored. Now every cursor paginate() returns is a Sentry cursor:

  • Page boundary: the server's cursor, unchanged.
  • Mid-page: the cursor of the partly returned page, with the number of rows already returned written in binary into is_prev. After a leading 0, a 1 bit is _0 and a 0 bit is 0, most significant first. So 84:2:0_00_000_00 means page 84:2:0, skip 82 (0b1010010), and the worst case is about 15 characters.
    • Sentry parses is_prev with Python's int() (Cursor.from_string in src/sentry/utils/cursors.py). int() allows single underscores between digits (PEP 515), so every such string is 0, and every endpoint accepts it as a normal next-page cursor.
    • paginate() refetches the page and skips those rows. An older CLI, or anything else that sends the cursor as is, sees those rows again but never misses any.
    • encodeSkip() documents the format.
  • Cursors that can't carry a skip: a server cursor that isn't a Sentry next-page cursor (e.g. a pasted is_prev=1 cursor) is returned as is. Resuming refetches that page.
  • Page size: no longer stored in the cursor. OffsetPaginator cursors (<per_page>:<page>:<is_prev>) already encode it, so paginate() reads it back, and pasted or stored raw cursors resume with the right window. Other paginators (BasePaginator, GenericOffsetPaginator, SequencePaginator) treat offset as a row offset and take any page size. When resuming mid-page, the first request also fetches the rows to skip, so it uses as few requests as possible.

Because both directions work, stored -c next history is shared across versions again: no namespacing, and nothing is discarded on upgrade or downgrade. Cursors stored by older CLIs are raw server cursors and resume correctly. cli1~ cursors saved by nightly builds since #1625 are still read but never written. A damaged one is ignored with a warning, and the walk starts from the first page.

Known limit: an older CLI that sends a page-number cursor with a different --limit still misreads it. That's the pre-#1625 bug, and this PR can't fix it on the old side.

Tests:

  • Properties over ports of OffsetPaginator, GenericOffsetPaginator, and a server that ignores per_page. They check that every row comes back exactly once across -c next with changing limits, that every returned cursor is a Sentry cursor (with is_prev a Python int literal equal to 0) that the API resumes from without skipping a row, and that a pasted page-number cursor resumes without skipping.
  • Unit tests for the request sequences, encoded skips, cursors that can't carry a skip, and nightly cli1~ cursors.
  • I checked the encoding against real Python int() for skips from 1 to 1000.

Local checks: tsc and lint pass. CLI unit suite: 11,300 pass. The only failures are 6 chmod/permission tests (completions, code-scanner, file-changes, telemetry, upgrade) that also fail on clean main in this sandbox.

via BYK.

--

View Junior Session in Sentry

…ersion

paginate() used 100-row pages for any --limit over 100, so --limit 201
asked for a full third page to return one row. It now uses the fewest
requests and splits the limit evenly across them (201 -> 3 x 67).

CLI cursors from another format version, or damaged ones, are ignored
with a warning and the walk restarts from the first page instead of
failing or reaching the API. Stored -c next history is namespaced by
CURSOR_FORMAT, so CLI versions never resume from each other's cursors
after an upgrade or downgrade.

Co-Authored-By: BYK <[email protected]>
@vercel

vercel Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
sentry-local Skipped Skipped Oct 23, 2026 10:49pm UTC

Request Review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread packages/cli/src/lib/api/infrastructure.ts Outdated

@BYK BYK left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to make our new cursor format somehow compatible with the old one? And also this new version being able to "salvage" the old cursors? I'm guessing they are just raw cursor from the server so we should be able to do something with them? You might be able to find more tricks/hacks by looking at getsentry/sentry code

A raw OffsetPaginator cursor (`<per_page>:<page>:<is_prev>`) passed to
paginate() was paired with the even-split page size for the limit, so
the server read the wrong window and skipped or repeated rows. Recover
the page size from the cursor instead; other paginators put 0 or a sort
value there and are unaffected.
Review feedback: make cursors work across CLI versions in both
directions instead of isolating them.

paginate() now returns Sentry cursors. A page-boundary cursor is the
server's cursor unchanged. A mid-page cursor is the page's cursor with
extra zeros in is_prev recording how many rows were already returned
(`84:2:000` skips 2). Sentry parses is_prev with int()
(Cursor.from_string), so older CLIs and the API accept these and see the
skipped rows again rather than missing any.

The page size comes back from page-number cursors themselves
(OffsetPaginator encodes it), so it no longer needs to live in the
cursor. That makes cursors stored by older CLIs resumable, so the
CURSOR_FORMAT namespace for stored -c next history is removed and old
and new versions share it again. cli1~ cursors from nightly builds are
still read, and remain the fallback for cursors Sentry did not issue.

Co-Authored-By: BYK <[email protected]>
@sentry-junior sentry-junior Bot changed the title fix(cli): Split paginate() limits evenly and isolate cursors by CLI version fix(cli): Split paginate() limits evenly and return Sentry-compatible cursors Oct 23, 2026
*
* Sentry parses `is_prev` with `int()` (`Cursor.from_string` in
* `src/sentry/utils/cursors.py`), so `0`, `00`, and `000` all mean "next".
* {@link paginate} uses the extra zeros to record how many rows of the page

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, so this means we can have up to 99 extra 0s there? 🤣 - I think its fine and clever but if you can think of a better way, I'd definitely take it.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f9333f0. Configure here.

Comment thread packages/cli/src/lib/api/infrastructure.ts Outdated
The cli1~ fallback for server cursors that cannot carry a skip (opaque
strings, is_prev=1) was the one path that still produced cursors older
CLIs cannot send to the API, and stored -c next history is shared across
versions again. Return the page's server cursor instead; resuming
returns that page's rows again rather than missing any. cli1~ cursors
from nightly builds are still read.
A skip of up to ~99 rows took up to 100 zeros in is_prev. Python's int()
also allows a single `_` between digits (PEP 515), so write the skip in
binary with `_0` for 1 and `0` for 0 after a leading `0`: 82 becomes
`0_00_000_00`. Sentry still reads it as is_prev=0. encodeSkip() documents
the format.

Co-Authored-By: BYK <[email protected]>
@BYK
BYK enabled auto-merge (squash) October 23, 2026 22:58
@BYK
BYK merged commit 8659970 into main Oct 23, 2026
26 of 27 checks passed
@BYK
BYK deleted the fix/cli-paginate-cursor-compat branch October 23, 2026 23:02

This branch was previously deployed

1 inactive deployment
Preview — ebc5b262 Deployed Oct 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: medium PR risk score: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant