Repository navigation
sync: batch small files, skip unchanged manifests, probe presence on its own - #26
Open
ChakraFusion wants to merge 2 commits into
Open
ChakraFusion wants to merge 2 commits into
ChakraFusion wants to merge 2 commits into
Conversation
|
@ChakraFusion is attempting to deploy a commit to the sivadaboi's projects Team on Vercel. A member of the Team first needs to authorize it. |
This was referenced Oct 4, 2026
…its own Performance - Batched pulls (protocol revision 2): small files are fetched up to 256 per request (4MB), 4 requests in flight, over a new /api/p2p/files route. Each file is still written through the PatchWriter (verified, fsynced, renamed). One request per file cost ~200ms on a real LAN; on loopback 5000 files went from 42s to 12s. Peers below revision 2 and relay connections keep the per-file path. - Unchanged games: when this device still holds the agreed base, the manifest request carries ifHash and a peer that still holds it too answers "unchanged" instead of sending the manifest. Older peers ignore the parameter; their full answer is used, not fetched twice. - Manifests are gzipped for clients that accept it (every Go client does, so older versions read them too): ~5.6x smaller with real hashes. - Manifests and batches get a 10-minute client timeout instead of 30s; a large manifest over a VPN never arrived inside 30s, so those games never synced. - An in-sync pass also mirrors the peer's latest snapshot, which only a pull did before; games identical everywhere from the start showed history on one device only. - A batch is bounded by what the responder counts (each requested block at the file's full block size, at most the file's real size), so a batch of many small files with a few larger ones is never refused as too large; a batch refused anyway is fetched again as two halves. Robustness - Presence: paired LAN peers are probed every 10s on their own ticker. Probes used to ride along with the minute-long reconcile, which waits for a whole sync-all pass; peers reached over a VPN (no UDP broadcast) showed online/offline minutes late. Probe rounds are serialized. - Games whose save folder is not on this device are skipped by sync-all and dropped from the retry queue instead of failing every 20s. - "syncing X with Y" is logged only when there is something to do; most passes produced only "already in sync" lines, which rotated everything else out of the log within hours. - The interrupted-pull safety test also runs over batched pulls. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Every peer started each run with whatever status the database held, and a peer counts as offline only after three failed probes. For a PC switched off overnight that meant being shown online after every start until three probes - the first ten seconds after the background loops started - had failed. - The presence loop probes at once on start, then every 10s. - The three-miss tolerance is for devices heard from during this run (answered a probe, or seen by discovery, a request or the relay since start). A peer with no sign of life since start goes offline on its first unanswered probe. Adds an opt-in e2e measurement (OPENSAVE_PRESENCE_TIMING) of how long a stopped device is shown online: 30s, by design, for one seen this run.
ChakraFusion
force-pushed
the
pr/sync-speed
branch
from
October 13, 2026 17:55
cf9269a to
14c12ba
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #21.
Based on the "Keep mine" PR (#22): its first commit is that PR. Only the last two commits are new here. I'll rebase once that PR is merged.
Why
A save of many small files (Project Zomboid: ~240k map chunks) cost one HTTP request per file. That is ~200 ms each on a real LAN, so a pull of 50 MB took hours. On top of that, every periodic pass fetched and compared the full manifest for every game, even when nothing had changed.
What
Batched pulls (protocol revision 2)
/api/p2p/filesroute.PatchWriter(verified, fsynced, renamed).Unchanged games
ifHashand the peer answers "unchanged" instead of sending the manifest.Gzip and timeouts
Smaller items
Presence (second commit)
Compatibility
Mixed versions work in both directions:
Accept-Encoding.Tests
batch_test.go: small files are batched, older peers are not, a batch stays under what the responder counts, and a refused batch is split.quick_test.go: the unchanged shortcut, real changes still sync, and an older peer's manifest is not fetched twice.manifest_gzip_test.go,presence_test.go,missing_skip_test.go.partial_safety_test.go: an interrupted batched pull never turns into deletions.e2e/batch_bench_test.go: opt-in measurement.🤖 Generated with Claude Code