Skip to content

Resolve a cursor the hub declined without waiting on files - #883

Merged
sduchesneau merged 6 commits into
developfrom
fix/retryable-unresolvable-cursor
Aug 17, 2026
Merged

Resolve a cursor the hub declined without waiting on files#883
sduchesneau merged 6 commits into
developfrom
fix/retryable-unresolvable-cursor

Conversation

@sduchesneau

@sduchesneau sduchesneau commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Problem

NewCursorResolver asks the hub first and falls back to bstream.NewFileSourceFromCursor when it returns nil. The hub returns nil for two different reasons, and one of them is not "this cursor is historical": it also declines a cursor whose block number it covers but whose ID it has never seen — a corrupted or forged cursor, or one carried over from a chain this process never served.

That file source cannot start before the merged-blocks bundle covering the cursor's block number is written. Near chain head it is not, so the request sits silent for a whole bundle — measured at ~16 minutes against mainnet.eth.streamingfast.io with a cursor whose block hash had its last 4 hex rewritten — and the files cannot resolve it when they arrive either.

Reproduced locally on firehose-core/devel/standard with the merger stopped, so the bundle never lands: develop returns nothing at all in 90s, no undo, no error.

Change

Before the file-source fallback, ask whether anything can still produce that block (bstream.CheckCursorResolvable, added in bstream#63):

  • the hub is authoritative over [LowestBlockNum, HeadNum], so a cursor block in that range whose ID it does not know is on no chain it ever saw;
  • the forked-blocks store is consulted before giving up, since a hub restarted after a fork no longer holds it while the store does;
  • both empty resolves to cursor.LIB, the existing "resolver failed" outcome, which resolveStartBlockNum turns into a BlockUndoSignal to LIB plus a start block of LIB+1.

The client gets the undo immediately instead of after the merger catches up — the same outcome, minus the twenty minutes.

A cursor above the hub's head is deliberately not that case: the block exists, this instance has not reached it, which is what a client reconnecting to an instance a few blocks behind the one that served it looks like. bstream waits it out briefly and reports ErrCursorAboveHead; this PR propagates that as connect.CodeUnavailable (retryable) instead of the usual CodeInvalidArgument, so a lagging tier1 neither kills a sink nor reverts one that is ahead of it.

Measured, local stack with the merger stopped

result
develop 0 blocks in 90s, no undo
this branch, unknown hash inside the hub's range UNDO … number:2910 then blocks, immediate
this branch, unknown hash 500 blocks above head 5s wait, then code = Unavailable, which the sink logs as a retryable error

Verified against the real thing too: the same forged-cursor scenario on mainnet.eth.streamingfast.io is what surfaced this.

The file source it fell back to cannot start before the merged bundle covering
that block number exists — twenty minutes on Ethereum — and cannot resolve the
cursor when it gets there. Undo to its LIB right away instead.
@sduchesneau

sduchesneau commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

🔍 Vulnerabilities of ghcr.io/streamingfast/substreams:15ee822

📦 Image Reference ghcr.io/streamingfast/substreams:15ee822
digestsha256:7941a3088149abd9dd7e774038458f8c2c8cd59851f8c2843c88e77b0bec0db8
vulnerabilitiescritical: 0 high: 0 medium: 0 low: 0
platformlinux/amd64
size123 MB
packages382
📦 Base Image oisupport/staging-amd64:24.04
also known as
  • a215e986b44aae6f10795ded1e39ce93d9c236d8163d21a522ffd0ab3659f546
  • noble
  • noble-20260730.1
digestsha256:019e8eb29a85e74d64925745884f2ec79aa27e3feab36353d24656f4d6b89467
vulnerabilitiescritical: 0 high: 0 medium: 5 low: 4

@sduchesneau
sduchesneau marked this pull request as ready for review August 17, 2026 19:12
Comment thread pipeline/resolve.go
return cursor.LIB, headBlock, nil
}

src = bstream.NewFileSourceFromCursor(mergedBlocksStore, forkedBlocksStore, cursor, jctBlkGetter, zap.NewNop(), fileSourceOptions...)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So are we double doing the CheckCursorResolvable now? Isn't NewFileSourceFromCursor doing the same?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The FileSourceFromCursor just determines that the file is in the future, so it waits for the merged block to appear...
Also, it cannot return an error, only a filesource or nil.

So that's 20 minutes on eth-mainnet !

CheckCursorResolvable checks if we are in the HEAD section, if so: if a forked-block contains it. If we are older than the LIB, we return nil (not our concern) so it goes back to the fileSourceFromCursor.

This prevents waiting for merged-blocks to be produced.

@sduchesneau
sduchesneau merged commit 74aaf43 into develop Aug 17, 2026
8 checks passed
@sduchesneau
sduchesneau deleted the fix/retryable-unresolvable-cursor branch August 17, 2026 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants