Resolve or refuse a cursor without waiting on merged files

[sduchesneau]

Problem

A stream resuming from a cursor whose block the live buffer should hold, but does not — a corrupted or forged cursor, or one carried over from a chain the process never saw — goes silent instead of failing.

ForkableHub declines such a cursor (forkable.blocksFromCursor → cannot find block with ID …), so JoiningSource falls back to the file source. That one can only start once the merged-blocks file holding the cursor's block number exists, which near chain head it does not: the stream then waits for the merger to write a whole bundle — measured at ~20 minutes on Ethereum with 100-block bundles — and the cursor is unresolvable from files when it gets there anyway.

Measured against firehose-core/devel/standard (dummy chain, 2 blocks/s), taking a live cursor off the stream and rewriting the last 4 hex of its block hash:

	time to first response
develop	43s — the wait for the bundle, then InvalidArgument: cannot resolve cursor from merged block files
this branch	0s — InvalidArgument: cannot resolve cursor: block #5266 (…dead) sits inside the live range [4770, 5286], where neither the live buffer nor the forked blocks hold it

With the merger stopped so the bundle never lands, develop stays silent indefinitely.

Change

Three commits, each a distinct decision.

41a2252 — ask before falling back to files. Once the live source has declined a cursor, JoiningSource asks whether anything can still produce that block:

• the live source is authoritative over the range it holds, so a cursor block inside [LowestBlockNum, HeadNum] whose ID it does not know is on no chain it ever saw;
• a cursor sitting on a fork the live buffer no longer holds is the one legitimate case, so the forked-blocks store is consulted before giving up;
• both empty is what makes the cursor unresolvable, reported as ErrResolveCursor — which stream.Run already surfaces as an invalid argument, unchanged by this PR.

Everything else is untouched: a cursor below the live range, a live buffer not yet ready, or a forked-blocks store that cannot be read all stay with the file source. The last one matters — an unreadable store is not evidence of absence, and treating an S3 hiccup as "that block never existed" would fail requests carrying perfectly good cursors.

The check is exported as CheckCursorResolvable so callers resolving cursors outside JoiningSource can make the same one — substreams' tier1 CursorResolver has the identical fallback and the identical hang (streamingfast/substreams#883).

36ebf04 — look the forked block up at its own height. One-block files are named after their block number, so the only file that can hold the cursor's block is the one at cursor.Block.Num(). Scanning [LIB.Num, Block.Num] was both wasted I/O and unsound: a truncated-ID suffix matching at any height in that window declared the cursor resolvable, sending it back to the file source to hang exactly as before.

c931e9c — wait out a cursor above the live head, and make that one retryable. A cursor block above head says nothing about the block existing; it says this process has not reached it, which is what a client reconnecting to an instance a few blocks behind the one that served it looks like. Those are given CursorHeadWaitTimeout (5s, exported) to arrive — polling the live head — and only then reported, as ErrCursorAboveHead rather than as a cursor no source can resolve.

That one maps to a new stream.ErrUnavailable, meant for codes.Unavailable: we may simply be lagging while another instance already serves that block, and an invalid argument would have the client discard a cursor that is good. ErrResolveCursor keeps its ErrInvalidArg mapping.

Servers need to map the new type; without it the error falls to their default branch (codes.Internal in firehose-core — still retryable, but the message is replaced). The matching mapping is streamingfast/firehose-core#212.

Callers with their own resolution path distinguish the two: substreams reverts an unresolvable cursor to its LIB, but must not revert a sink that is merely ahead of a lagging instance.

Tests

• TestJoiningSourceCheckCursorResolvable, 10 cases: block known to the live buffer, below the range, above head and reached while waiting, above head and never reached, reached but unknown there, buffer not ready, held by the forked blocks, held at another height (which is another block), forked-blocks store failing, and unknown to both.
• TestFileSourceFactory_HasForkedBlock against a MockStore with real one-block filenames: the same ID suffix present at two heights answers only for its own.

Note: TestFileSource_Run is flaky on develop already — it asserts a counter incremented from parallel preprocessor goroutines without synchronization (go test -count=50 -run TestFileSource_Run fails on develop, and go test -race reports pre-existing races in TestEternalSource). Adding tests to the package shifts scheduling enough to surface it more often. Left alone here as unrelated.

🤖 Generated with Claude Code

[UlysseCorbeil]

minor, but should pass ctx here so dosent continue on cancel