Streamer.Peek in the Espresso streamer we pin (espresso-streamers at 22c396a, pulled in by
#480) returns a bare *EspressoBatch with no second value, and five distinct conditions collapse
into a nil:
- no candidates at the position,
op/batch_store.go:110 with the return at :112
- tip hash unset,
:116
- no candidate extends the tip,
:137
BatchUndecided, op/op_streamer_v2.go:212BatchPast, the same :212 return, reached when advanceOnFinalization moves lastFinalizedL2
past the batch between store.peek() returning it and the re-check at :203
Some of those mean wait and some mean the consumer should re-anchor, and the consumer can't tell
which it got. Ours treats all five as retry next tick.
Why it matters to us
Two candidates can sit at one height, both signed by the authorized batcher. The store supports
that deliberately, and Peek's own doc comment at :190 says so. peek picks the earlier by
Espresso arrival order and serves it. If the chain actually continued from the other one, then once
our tip has advanced to the one we took, everything arriving at the next height declares the other
parent, peek falls into the no-fork-matches-tip branch, and it stays there. The channel manager
stays empty and the safe head stops.
This is #459's failure mode in a quieter form, and quiet is harder to operate.
What needs to change
Return a reason alongside the batch so a consumer can distinguish wait from re-anchor. Failing that,
a counter of consecutive no-fork-matches-tip peeks would at least give us something to alert on.
This is an API shape change, which is why it's worth settling before we build more on top of it.
Two corrections to how we first described this
It isn't indistinguishable from an empty store. The no-fork-matches-tip branch logs at Info on every
Peek with the block number and the stale tip, whereas the empty case logs nothing. So log-based
monitoring would catch it. There's no metric and no error, which is the real gap.
And our claim that none of our re-anchor triggers can fire was wrong as stated.
espressoSyncChannelManager calls SetBatchPosition off computeSyncActions without any batch
being served first. It just won't fire in this scenario, because an empty channel manager gives
computeSyncActions nothing to reconcile, which is what the comment in our own drain loop says. The
conclusion holds, the reasoning was wrong.
The fifth condition, BatchPast, is one we missed the first time round. Line numbers are against
22c396a, the current head of espresso-streamers#36. No upstream issue for this one either.
Streamer.Peekin the Espresso streamer we pin (espresso-streamersat22c396a, pulled in by#480) returns a bare
*EspressoBatchwith no second value, and five distinct conditions collapseinto a nil:
op/batch_store.go:110with the return at:112:116:137BatchUndecided,op/op_streamer_v2.go:212BatchPast, the same:212return, reached whenadvanceOnFinalizationmoveslastFinalizedL2past the batch between
store.peek()returning it and the re-check at:203Some of those mean wait and some mean the consumer should re-anchor, and the consumer can't tell
which it got. Ours treats all five as retry next tick.
Why it matters to us
Two candidates can sit at one height, both signed by the authorized batcher. The store supports
that deliberately, and
Peek's own doc comment at:190says so.peekpicks the earlier byEspresso arrival order and serves it. If the chain actually continued from the other one, then once
our tip has advanced to the one we took, everything arriving at the next height declares the other
parent,
peekfalls into the no-fork-matches-tip branch, and it stays there. The channel managerstays empty and the safe head stops.
This is #459's failure mode in a quieter form, and quiet is harder to operate.
What needs to change
Return a reason alongside the batch so a consumer can distinguish wait from re-anchor. Failing that,
a counter of consecutive no-fork-matches-tip peeks would at least give us something to alert on.
This is an API shape change, which is why it's worth settling before we build more on top of it.
Two corrections to how we first described this
It isn't indistinguishable from an empty store. The no-fork-matches-tip branch logs at Info on every
Peekwith the block number and the stale tip, whereas the empty case logs nothing. So log-basedmonitoring would catch it. There's no metric and no error, which is the real gap.
And our claim that none of our re-anchor triggers can fire was wrong as stated.
espressoSyncChannelManagercallsSetBatchPositionoffcomputeSyncActionswithout any batchbeing served first. It just won't fire in this scenario, because an empty channel manager gives
computeSyncActionsnothing to reconcile, which is what the comment in our own drain loop says. Theconclusion holds, the reasoning was wrong.
The fifth condition,
BatchPast, is one we missed the first time round. Line numbers are against22c396a, the current head of espresso-streamers#36. No upstream issue for this one either.