Skip to content

relay: a reply that never comes fails after its timeout, not the sync's - #50

Open
ChakraFusion wants to merge 1 commit into
Liquid-co:dev-publicfrom
ChakraFusion:pr/relay-lost-reply
Open

ChakraFusion wants to merge 1 commit into
Liquid-co:dev-publicfrom
ChakraFusion:pr/relay-lost-reply

Conversation

@ChakraFusion

Copy link
Copy Markdown

Found while chasing a CI timeout: an e2e shard hit the 30-minute limit on Windows, with TestThreeDevices_PerPeerStateStaysSeparate stuck for 7m47s in one manifest fetch over the relay. The goroutine dump showed nothing serving that request on the other side, so the request or its reply had been dropped.

  • Cause: exchange waited for the larger of wanRequestTimeout and the caller's deadline. Every sync runs under a 30-minute per-peer deadline, so a request or reply the relay dropped (it drops frames for a client that can't keep up, by design) held the sync for up to half an hour instead of failing it to be retried. This also made FetchBlocks' payload-sized deadline dead code: it only applied to a context without a deadline, which a sync never is.
  • Fix: a request waits wanRequestTimeout, or longer only when the caller asks with withReplyWait. Block fetches ask for their payload-sized time as before; manifest requests ask for ServeSettleWait more (the time a peer may hold its answer while a sync writes the save). The caller's context still ends it early.
  • Why frames were dropped: trackPresence wrote the peer row (a synced database commit) for every frame from a paired peer, including each block reply and heartbeat, on the one goroutine that reads the relay. In the dump, two of the three daemons' relay readers were inside FlushFileBuffers. The last-seen time is now written when something changed, and otherwise every 5 s (the LAN path uses 10 s; peers expire after 20 s).

Tests: TestReplyWait_NotTheCallersWholeDeadline, …LongerWhenAskedFor, and TestTrackPresence_WritesOnlyWhenDue (fails without the change).

🤖 Generated with Claude Code

exchange waited for the larger of wanRequestTimeout and the caller's own
deadline. Every sync runs under a 30-minute per-peer deadline, so a
manifest request or reply the relay dropped (it drops frames for a client
that can't keep up, relay/server.go) held the sync for up to half an hour
instead of failing it to be retried. It also made FetchBlocks' payload-sized
deadline dead code: it applied only to a context without a deadline, which
a sync never is.

A request now waits wanRequestTimeout, or longer only when its caller asks
with withReplyWait: block fetches for their payload as before, manifest
requests for ServeSettleWait more, the time a peer may hold its answer
while a sync writes the save. The caller's context still ends it early.

Found as an e2e shard hitting the 30-minute limit on Windows CI, with
TestThreeDevices_PerPeerStateStaysSeparate waiting 7m47s in one manifest
fetch over the relay.

Also: presence is written when it changes, not on every frame.
trackPresence wrote the peer row (a synced database commit) for every frame
a paired peer sent, block replies and heartbeats alike, on the one goroutine
that reads the relay. A slow disk held that reading up, and the relay drops
frames for a client that falls behind. The last-seen time is now written
when something changed and otherwise every 5 seconds, as the LAN path does
every 10; peers expire after 20.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@vercel

vercel Bot commented Oct 14, 2026

Copy link
Copy Markdown

@ChakraFusion is attempting to deploy a commit to the sivadaboi's projects Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant