Skip to content

fix(ocap-kernel): boot past a vat whose code can no longer be loaded - #1031

Draft
grypez wants to merge 2 commits into
grypez/retire-persisted-vatfrom
grypez/missing-bundle-boot-2
Draft

fix(ocap-kernel): boot past a vat whose code can no longer be loaded#1031
grypez wants to merge 2 commits into
grypez/retire-persisted-vatfrom
grypez/missing-bundle-boot-2

Conversation

@grypez

@grypez grypez commented Aug 25, 2026

Copy link
Copy Markdown
Member

Third of three, stacked on #1030. This is the fix for the issue #1025 was opened
for; the two beneath it are bugs reachable on main today that #1025 had to fix
in order for this one to be an improvement rather than a worse failure mode.

The defect

A vat outlives the code it was launched from. The kernel stores a vat's durable
state but not its code, re-fetching from the recorded bundleSpec on every
incarnation — so a bundle that is rebuilt to a new path, pruned, or recorded as
an absolute path that did not survive relocation leaves a persisted vat that
cannot be restored.

VatManager.initializeAllVats restored every vat inside one Promise.all, so
one such vat rejected out of Kernel.#init, Kernel.make rejected, and the
whole kernel — every healthy subcluster included — was unbootable. Under the
daemon that surfaced only as "Daemon did not start within 30s".

Approach

Each vat is restored in isolation. The one that fails is skipped and the rest of
the kernel boots.

Its record is kept, not pruned, so a vat whose code becomes reachable again
is restored by a later boot, resuming from the durable state it left off with.
Discarding persisted state is not a call the restore path gets to make, and a
missing bundle is usually a build or packaging problem rather than a decision
about the vat.

The leftover worker is reaped. A bundle is fetched inside the vat's own
worker (VatSupervisor.#initVatfetchBlob), so by the time the load fails
launch has long since resolved and the worker is live; one left behind is the
wedged process holding the sqlite lock that this failure mode is known by. When
launch itself was what failed there is no worker to reap, so that is logged at
debug.

An error names the vat, its subcluster, and its code source. Trading an
unbootable kernel for one that is quietly missing a vat would be no trade at
all.

Scope

Deliberately not decided here: whether an unrestorable vat should take its
whole subcluster down with it. That is the coterminous-lifecycle question in
#979, and the tests are written so no lifecycle policy is presumed — the
unrestorable vat sits in a different subcluster from the healthy one, so no
policy under discussion makes the healthy vat's loss correct.

Worth being precise about what a skipped vat is, since the changelog wording
invites a stronger reading than the code supports. Its data is preserved; its
interactions are not. Promises it was already deciding stay unresolved forever,
but new sends to it are rejected immediately with ENDPOINT_UNREACHABLE,
notifications owed to it are dropped, and GC actions against it still perform
the kernel's own half. What comes back on the day the bundle returns is not
what left. Making a skipped vat genuinely resumable is a lifecycle decision and
belongs with #979.

restartVat also cannot recover one without a kernel restart, since it requires
a running vat.

The observability half of #964 (rendering the real fatal to daemon.log rather
than [object Object] on a discarded stderr) landed separately in #966.
fetchBlob needs no change: Node's errno already carries the path, and the
[object Object] came from the error crossing the worker's JSON-RPC boundary.

Testing

First commit is the failing repro on its own, at three layers.

  • Kernel.test.ts — boot completes with the healthy vat up and the
    unrestorable one absent, and one error entry names both the vat and its bundle.
  • VatManager.test.ts — both failure timings (before the worker comes up,
    and after it is live, which is the production one); the reap takes only the
    failed vat's worker; boot survives a reap that cannot happen; the record is
    not pruned; all three VatConfig source shapes and the no-subcluster
    fallback; and several unrestorable vats at once, the case that cannot be
    satisfied by catching around the whole batch.
  • kernel-test/persistence.test.ts — the real path end to end: real
    bundles, real workers, a real file deleted between two real incarnations, and
    the genuine fetchBlob ENOENT arriving from inside the worker. A vat whose
    bundle returns comes back with the state it left off with; and one whose
    subcluster was torn down while it was skipped stays gone when its bundle
    returns, rather than reappearing in a subcluster that no longer exists and
    failing every getStatus thereafter.

@metamask/ocap-kernel and @ocap/kernel-test green, with auditRefCounts on
for every kernel kernel-test builds. lint and build clean.

Checklist

  • I've updated the test suite for new or updated code as appropriate
  • I've updated documentation (JSDoc, Markdown, etc.) for new or updated code as appropriate
  • I've communicated my changes to consumers by updating changelogs for packages I've changed, highlighting breaking changes as necessary
  • I've prepared draft pull requests for clients and consumer packages to resolve any breaking changes

@grypez
grypez force-pushed the grypez/missing-bundle-boot-2 branch from 958b4ea to 8780f84 Compare August 25, 2026 16:57
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 72.64%
⬆️ +0.10%
9738 / 13405
🔵 Statements 72.49%
⬆️ +0.09%
9901 / 13657
🔵 Functions 73.11%
⬆️ +0.12%
2284 / 3124
🔵 Branches 67.05%
⬆️ +0.16%
4016 / 5989
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
packages/ocap-kernel/src/vats/VatManager.ts 100%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
Generated in workflow #4696 for commit 81acebb by the Vitest Coverage Report Action

@grypez
grypez force-pushed the grypez/missing-bundle-boot-2 branch from 8780f84 to 45d038f Compare August 25, 2026 19:25
@grypez
grypez force-pushed the grypez/missing-bundle-boot-2 branch from 45d038f to cc47388 Compare August 26, 2026 14:26
@grypez
grypez force-pushed the grypez/missing-bundle-boot-2 branch from cc47388 to 71e612a Compare August 26, 2026 16:27
@grypez
grypez force-pushed the grypez/missing-bundle-boot-2 branch from 71e612a to 41ec52d Compare August 26, 2026 19:00
grypez and others added 2 commits August 26, 2026 15:33
A vat outlives the code it was launched from. The kernel stores a vat's
durable state but not its code, re-fetching from the recorded `bundleSpec`
on every incarnation — so a bundle that is rebuilt to a new path, pruned,
or recorded as an absolute path that did not survive relocation leaves a
persisted vat that cannot be restored.

`initializeAllVats` restores every vat inside one `Promise.all`, so one
such vat rejects out of `Kernel.#init`, `Kernel.make` rejects, and the
whole kernel — every healthy subcluster included — is unbootable. Under the
daemon that surfaces only as "Daemon did not start within 30s".

Three layers, because the interesting parts are at different ones.

`Kernel.test.ts` covers the claim itself: boot completes with the healthy
vat up and the unrestorable one absent, and one error entry names both the
vat and its bundle.

`VatManager.test.ts` covers both failure timings — before the worker comes
up, and after it is live, which is the production one, since the bundle is
fetched inside the vat's own worker. Also that the reap takes only the
failed vat's worker, that boot survives a reap that cannot happen, that the
record is not pruned, all three `VatConfig` source shapes, and several
unrestorable vats at once, which is the case that cannot be satisfied by
catching around the whole batch.

`kernel-test/persistence.test.ts` covers the real path end to end: real
bundles, real workers, a real file deleted between two real incarnations,
and the genuine `fetchBlob` ENOENT arriving from inside the worker. A vat
whose bundle returns comes back with the state it left off with; and one
whose subcluster was torn down while it was skipped stays gone when its
bundle returns, rather than reappearing in a subcluster that no longer
exists and failing every `getStatus` thereafter.

Co-Authored-By: Claude Opus 5 <[email protected]>
`initializeAllVats` restored every persisted vat inside one `Promise.all`,
so a vat whose code could no longer be loaded rejected out of
`Kernel.#init` and took the whole kernel with it — every healthy subcluster
included. Each vat is now restored on its own: the one that fails is
skipped and the rest of the kernel boots.

The vat's record is kept rather than pruned, so a vat whose code becomes
reachable again is restored by a later boot, resuming from the durable
state it left off with. Discarding persisted state is not a call the
restore path gets to make, and a missing bundle is usually a build or
packaging problem rather than a decision about the vat.

The leftover worker is reaped. A bundle is fetched inside the vat's own
worker, so by the time the load fails `launch` has long since resolved and
the worker is live; one left behind is the wedged process holding the
sqlite lock that this failure mode is known by. When `launch` itself was
what failed there is no worker to reap, so that is logged at debug.

An error names the vat, its subcluster, and its code source. Trading an
unbootable kernel for one that is quietly missing a vat would be no trade
at all, and the spec that names the unreachable code is the actionable
part of the report.

Whether an unrestorable vat should instead take its subcluster down with it
is the coterminous-lifecycle question in #979, and is deliberately not
decided here: the tests put the unrestorable vat in a different subcluster
from the healthy one, so no policy under discussion makes the healthy vat's
loss correct.

Co-Authored-By: Claude Opus 5 <[email protected]>
@grypez
grypez force-pushed the grypez/missing-bundle-boot-2 branch from 41ec52d to 81acebb Compare August 26, 2026 19:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant