fix(ocap-kernel): boot past a vat whose code can no longer be loaded - #1031
Draft
grypez wants to merge 2 commits into
Draft
fix(ocap-kernel): boot past a vat whose code can no longer be loaded#1031grypez wants to merge 2 commits into
grypez wants to merge 2 commits into
Conversation
grypez
force-pushed
the
grypez/missing-bundle-boot-2
branch
from
August 25, 2026 16:57
958b4ea to
8780f84
Compare
Contributor
Coverage Report
File Coverage
|
||||||||||||||||||||||||||||||||||||||
grypez
force-pushed
the
grypez/missing-bundle-boot-2
branch
from
August 25, 2026 19:25
8780f84 to
45d038f
Compare
grypez
force-pushed
the
grypez/missing-bundle-boot-2
branch
from
August 26, 2026 14:26
45d038f to
cc47388
Compare
grypez
force-pushed
the
grypez/missing-bundle-boot-2
branch
from
August 26, 2026 16:27
cc47388 to
71e612a
Compare
grypez
force-pushed
the
grypez/missing-bundle-boot-2
branch
from
August 26, 2026 19:00
71e612a to
41ec52d
Compare
A vat outlives the code it was launched from. The kernel stores a vat's durable state but not its code, re-fetching from the recorded `bundleSpec` on every incarnation — so a bundle that is rebuilt to a new path, pruned, or recorded as an absolute path that did not survive relocation leaves a persisted vat that cannot be restored. `initializeAllVats` restores every vat inside one `Promise.all`, so one such vat rejects out of `Kernel.#init`, `Kernel.make` rejects, and the whole kernel — every healthy subcluster included — is unbootable. Under the daemon that surfaces only as "Daemon did not start within 30s". Three layers, because the interesting parts are at different ones. `Kernel.test.ts` covers the claim itself: boot completes with the healthy vat up and the unrestorable one absent, and one error entry names both the vat and its bundle. `VatManager.test.ts` covers both failure timings — before the worker comes up, and after it is live, which is the production one, since the bundle is fetched inside the vat's own worker. Also that the reap takes only the failed vat's worker, that boot survives a reap that cannot happen, that the record is not pruned, all three `VatConfig` source shapes, and several unrestorable vats at once, which is the case that cannot be satisfied by catching around the whole batch. `kernel-test/persistence.test.ts` covers the real path end to end: real bundles, real workers, a real file deleted between two real incarnations, and the genuine `fetchBlob` ENOENT arriving from inside the worker. A vat whose bundle returns comes back with the state it left off with; and one whose subcluster was torn down while it was skipped stays gone when its bundle returns, rather than reappearing in a subcluster that no longer exists and failing every `getStatus` thereafter. Co-Authored-By: Claude Opus 5 <[email protected]>
`initializeAllVats` restored every persisted vat inside one `Promise.all`, so a vat whose code could no longer be loaded rejected out of `Kernel.#init` and took the whole kernel with it — every healthy subcluster included. Each vat is now restored on its own: the one that fails is skipped and the rest of the kernel boots. The vat's record is kept rather than pruned, so a vat whose code becomes reachable again is restored by a later boot, resuming from the durable state it left off with. Discarding persisted state is not a call the restore path gets to make, and a missing bundle is usually a build or packaging problem rather than a decision about the vat. The leftover worker is reaped. A bundle is fetched inside the vat's own worker, so by the time the load fails `launch` has long since resolved and the worker is live; one left behind is the wedged process holding the sqlite lock that this failure mode is known by. When `launch` itself was what failed there is no worker to reap, so that is logged at debug. An error names the vat, its subcluster, and its code source. Trading an unbootable kernel for one that is quietly missing a vat would be no trade at all, and the spec that names the unreachable code is the actionable part of the report. Whether an unrestorable vat should instead take its subcluster down with it is the coterminous-lifecycle question in #979, and is deliberately not decided here: the tests put the unrestorable vat in a different subcluster from the healthy one, so no policy under discussion makes the healthy vat's loss correct. Co-Authored-By: Claude Opus 5 <[email protected]>
grypez
force-pushed
the
grypez/missing-bundle-boot-2
branch
from
August 26, 2026 19:37
41ec52d to
81acebb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of three, stacked on #1030. This is the fix for the issue #1025 was opened
for; the two beneath it are bugs reachable on
maintoday that #1025 had to fixin order for this one to be an improvement rather than a worse failure mode.
The defect
A vat outlives the code it was launched from. The kernel stores a vat's durable
state but not its code, re-fetching from the recorded
bundleSpecon everyincarnation — so a bundle that is rebuilt to a new path, pruned, or recorded as
an absolute path that did not survive relocation leaves a persisted vat that
cannot be restored.
VatManager.initializeAllVatsrestored every vat inside onePromise.all, soone such vat rejected out of
Kernel.#init,Kernel.makerejected, and thewhole kernel — every healthy subcluster included — was unbootable. Under the
daemon that surfaced only as "Daemon did not start within 30s".
Approach
Each vat is restored in isolation. The one that fails is skipped and the rest of
the kernel boots.
Its record is kept, not pruned, so a vat whose code becomes reachable again
is restored by a later boot, resuming from the durable state it left off with.
Discarding persisted state is not a call the restore path gets to make, and a
missing bundle is usually a build or packaging problem rather than a decision
about the vat.
The leftover worker is reaped. A bundle is fetched inside the vat's own
worker (
VatSupervisor.#initVat→fetchBlob), so by the time the load failslaunchhas long since resolved and the worker is live; one left behind is thewedged process holding the sqlite lock that this failure mode is known by. When
launchitself was what failed there is no worker to reap, so that is logged atdebug.
An error names the vat, its subcluster, and its code source. Trading an
unbootable kernel for one that is quietly missing a vat would be no trade at
all.
Scope
Deliberately not decided here: whether an unrestorable vat should take its
whole subcluster down with it. That is the coterminous-lifecycle question in
#979, and the tests are written so no lifecycle policy is presumed — the
unrestorable vat sits in a different subcluster from the healthy one, so no
policy under discussion makes the healthy vat's loss correct.
Worth being precise about what a skipped vat is, since the changelog wording
invites a stronger reading than the code supports. Its data is preserved; its
interactions are not. Promises it was already deciding stay unresolved forever,
but new sends to it are rejected immediately with
ENDPOINT_UNREACHABLE,notifications owed to it are dropped, and GC actions against it still perform
the kernel's own half. What comes back on the day the bundle returns is not
what left. Making a skipped vat genuinely resumable is a lifecycle decision and
belongs with #979.
restartVatalso cannot recover one without a kernel restart, since it requiresa running vat.
The observability half of #964 (rendering the real fatal to
daemon.logratherthan
[object Object]on a discarded stderr) landed separately in #966.fetchBlobneeds no change: Node's errno already carries the path, and the[object Object]came from the error crossing the worker's JSON-RPC boundary.Testing
First commit is the failing repro on its own, at three layers.
Kernel.test.ts— boot completes with the healthy vat up and theunrestorable one absent, and one error entry names both the vat and its bundle.
VatManager.test.ts— both failure timings (before the worker comes up,and after it is live, which is the production one); the reap takes only the
failed vat's worker; boot survives a reap that cannot happen; the record is
not pruned; all three
VatConfigsource shapes and the no-subclusterfallback; and several unrestorable vats at once, the case that cannot be
satisfied by catching around the whole batch.
kernel-test/persistence.test.ts— the real path end to end: realbundles, real workers, a real file deleted between two real incarnations, and
the genuine
fetchBlobENOENT arriving from inside the worker. A vat whosebundle returns comes back with the state it left off with; and one whose
subcluster was torn down while it was skipped stays gone when its bundle
returns, rather than reappearing in a subcluster that no longer exists and
failing every
getStatusthereafter.@metamask/ocap-kerneland@ocap/kernel-testgreen, withauditRefCountsonfor every kernel
kernel-testbuilds.lintandbuildclean.Checklist