Add a split durable-dir capture that hardlinks file contents - #1519
Open
Chenyi Wang (chw120) wants to merge 1 commit into
Open
Add a split durable-dir capture that hardlinks file contents#1519Chenyi Wang (chw120) wants to merge 1 commit into
Chenyi Wang (chw120) wants to merge 1 commit into
Conversation
Sealing a durable-dir volume as one tar reads and rewrites every byte the actor holds, on the paused critical path, so suspend latency scales with the actor's data rather than with what it changed. Add a second arrangement, selected by ATEOM_DURABLE_BACKEND=files: a metadata-only index tar plus one blob per non-empty regular file, hardlinked out of the directory instead of copied. Sealing costs one link per file, and restore adopts the blobs the same way, so neither direction spends a full copy of the tree. Both ends are guarded: CreateSplit falls back to copying on EXDEV/EMLINK/EPERM, and ExtractSplit copies any blob whose link count shows a second owner still holds it, which keeps the "this activation owns the blobs" contract enforced by the code rather than by convention. The blob name travels in a PAX record per entry, so a reader tells the two arrangements apart entry by entry and ExtractSplit reads a plain archive too. A restore dispatches on what the snapshot contains rather than on the variable, so a node configured either way can read back an actor captured the other way. atelet has to know the new names as well: carving durable data out of a FULL micro-VM capture for a DATA upload now selects every durable-dir snapshot file, not just the tar, and fails the upload when none are present instead of silently uploading an empty set. Also split the micro-VM cold-boot restore timing into its durable and boot halves, which is what shows the durable side going to nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sealing a durable-dir volume as one tar reads and rewrites every byte the actor holds, on the paused critical path, so suspend latency scales with the actor's data rather than with what it changed. This adds a second arrangement that
hardlinks contents instead of copying them, at both ends.
Off by default.
ATEOM_DURABLE_BACKENDunset — or set to anything butfiles— keeps the existing tar path, byte for byte. Only ateom-microvm reads the variable; ateom-gvisor is untouched.What it does
ATEOM_DURABLE_BACKEND=fileswritesdurable-dir.index.tar(metadata only, regular-file entries carrySize=0) plus onedurable-dir.blob-NNNNper non-empty regular file, hardlinked out of the volume rather than copied.Sealing costs one
link(2)per file instead of a copy of the tree. Restore mirrors it:ExtractSplitadopts each blob withLinkatinstead of writing it out again.Both ends are guarded.
CreateSplitfalls back to copying onEXDEV/EMLINK/EPERM.ExtractSplitcopies any blob whose link count shows a second owner still holds it, so the "this activation owns the blobs" contract is enforced by the code rather than by convention.The blob name travels in a PAX record per entry, so dispatch is per entry, not per archive:
ExtractSplitreads a plainCreatearchive too, and an archive can hold both kinds (empty files never get a blob). A restore dispatches on whatthe snapshot contains rather than on the variable, so a node configured either way reads back an actor captured the other way.
atelet has to know the new names as well: carving durable data out of a FULL micro-VM capture for a DATA upload now selects every durable-dir snapshot file by predicate, and fails the upload when none are present instead of silently
uploading an empty set.
Measured
Old-new-old bracket,
durdir_partial_512mb_pause_microvm, 8 min per arm, single-node micro-VM pool. The two tar arms agree within 2.5%, so the session did not drift.durable_dir(seal)durablehalftotalThroughput +124%. Workload control (
DurDirWrite) held at 848 / 842 / 851 ms p50 with no failures, so the guest was doing the same work in all three arms.The cost is a snapshot of many objects rather than one, paid on upload — a directory of many small files would pay more for it than this workload's eight large ones does.
Testing
internal/tarutil/split_test.gocovers round-trip, empty files getting no blob, hardlink trees, symlinks and device nodes, thenlink != 1guard falling back to copy, theEXDEVfallback (linkatis a package variable the test stubs), and mixed archives.cmd/ateom-microvm/durable_test.gocovers arrangement selection and restore dispatching on snapshot contents in both directions.cmd/atelet/main_test.gocovers the widened DATA carve, and the existing "no durable files" case still returnsFailedPrecondition.make verifyand the full unit suite pass on linux.Caveat on the cluster numbers. They were produced by an equivalent build, not by this commit: the rebased tree needs a control plane carrying
ateapi.WorkerService.SetWorkerCapacity, which the benchmark cluster does nothave yet, so the measured image is this branch's
internal/tarutiland durable path grafted onto the pre-rebase base. The rebase touches the measured path in exactly three places — an import path, one comment, and upstream's tar buffering— and the buffering was grafted in and measured separately: it is inert here by its own design, since
copyBufSize(128 KiB) exceedsstreamBufSize(64 KiB) specifically to route file contents around the stream buffer, and this snapshotis eight large files. The atelet DATA carve was exercised on the cluster in its pre-rebase form (188 index files and 8×94 blobs uploaded); the gVisor widening is unit-test-only.
Potential Next Step (Not in scope)
Making the split arrangement the default, and whether the upload side should batch the blobs.