Skip to content

Optimize snapshot unpack by using multiple accounts directories - #245

Open
palmerlao wants to merge 8 commits into
alpenglow-devfrom
palmer/shard-bigfile-unpack
Open

Optimize snapshot unpack by using multiple accounts directories#245
palmerlao wants to merge 8 commits into
alpenglow-devfrom
palmer/shard-bigfile-unpack

Conversation

@palmerlao

@palmerlao palmerlao commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

This change speeds up snapshot unpacking by

  1. Adding the ability to configure mithril to unpack the snapshot over multiple different directories
    • primarily useful when the machine has multiple NVMEs of different sizes and write speeds each mounted to a different directory. RAID0 can be unpredictable or inefficient in this situation
    • also changing the unpack process to use one big file per directory to hold all the appendvecs rather than storing each appendvec in its own file
    • the big file can optionally be written using O_DIRECT
  2. Improving the index log-to-SST process with pipelining and reducing the number of copies made during the sort

On my test machine I find that dev takes 15m23s to unpack a snapshot while 70f801f can unpack the same snapshot in 6m53s.

@palmerlao palmerlao changed the title Optimize snapshot unpack Optimize snapshot unpack by using multiple accounts directories Jul 6, 2026
palmerlao and others added 8 commits September 7, 2026 07:01
@palmerlao @7layermagik
Store accountsdb append-vecs as per-disk big files
9d004c8 
Unpack coalesces the snapshot's append-vecs into one big file per disk
shard, written by an async per-shard writer fed from a pooled tar reader.
The full and incremental snapshots share a shard's "data" file (offsets
carry across the two sequential phases); a fileId encodes shard = id % N
and segment = id / N, with segment 0 the coalesced file and segment >= 2
runtime per-file append-vecs.

Records go to the least-backlogged shard by in-flight (queued) bytes,
steering toward the fastest-draining disk so throughput approaches the
disks' weighted sum; when no disk has a backlog it falls back to an even
byte split.
@palmerlao @7layermagik
Rewrite the snapshot index flush as a read-sort-write pipeline
cbd55c6 
Sort each shard buffer in place. The stages overlap so the disk
keeps working while the CPU sorts, instead of each shard reading,
sorting, and writing serially.
@palmerlao @7layermagik
Add a flush-sort-workers flag and tuning setting
653ec6f 
Tunes how many shard buffers are sorted at once during the flush.
@palmerlao @7layermagik
Add a flush correctness test
8404c91 
Ingests the flushed SSTs and asserts one entry per key at its
highest slot.
@7layermagik
Exercise sharded snapshot recovery and preserve legacy state validation
796d9b6
@7layermagik
Document PR 245 review findings and Zen 5 validation
18631a3
@7layermagik
Fix sharded snapshot safety and Alpenglow compaction
e077951 
Reject aliased storage paths before cleanup and truncation, isolate Linux direct I/O, and reclaim coalesced snapshot files through bounded durable index-driven compaction. Preserve account slots, rewind pins, and concurrent read safety.

Release finished snapshot writer buffers before indexing. Add regression and abrupt-exit recovery coverage, and document Zen 5 correctness, performance, and retained-memory measurements.
@7layermagik
Default snapshot indexing to 256 shards and eight flushers
6c3751e
@7layermagik
7layermagik force-pushed the palmer/shard-bigfile-unpack branch from 46157e1 to 6c3751e Compare September 7, 2026 15:31
@7layermagik
7layermagik changed the base branch from dev to alpenglow-dev September 7, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@palmerlao @7layermagik