Skip to content

fix(ocap-kernel): finish retiring a vat that is not running - #1030

Draft
grypez wants to merge 2 commits into
grypez/absent-endpoint-deliveriesfrom
grypez/retire-persisted-vat
Draft

fix(ocap-kernel): finish retiring a vat that is not running#1030
grypez wants to merge 2 commits into
grypez/absent-endpoint-deliveriesfrom
grypez/retire-persisted-vat

Conversation

@grypez

@grypez grypez commented Aug 25, 2026

Copy link
Copy Markdown
Member

Second of three, stacked on #1029. Splits #1025.

The defect

restartVat stops a vat and then runs it again. A relaunch that fails — the
bundle has moved since the vat was launched, the worker will not spawn — leaves
the vat gone from the running map with its record, its own store and its root
pin all still in place. stopVat(vatId, false) deletes the handle and does not
touch persistence, and nothing puts it back.

Nothing retires a vat in that state. terminateVat goes through stopVat
getVat and throws VatNotFoundError, so the only way to be rid of one is to
discard the whole store — and terminateSubcluster, which walks persisted
membership rather than the running map, rejects part-way through on reaching
one, after deleting the system-subcluster mapping and before removing the
subcluster record.

Approach

Retiring such a vat does everything stopVat does apart from stopping a worker.
None of it is visible from terminateVat alone, which is why the state was easy
to leave half-done:

  • The record goes. markVatAsTerminated alone does not retire a vat: the
    cleanup it schedules walks keys prefixed ${vatId}., which never matches
    vatConfig.${vatId}. Only deleteVat takes that, along with the vat's own
    store and its subcluster membership. A record left behind restores the vat at
    the next boot whose code is reachable.
  • The decider promises are rejected, carrying the termination reason.
    cleanupTerminatedVat deletes their c-list entries and drops the decider's
    refcount on the stated understanding that its caller rejected them first.
    Nothing did, so they were left unresolved with a decider that no longer
    existed and no way to ever settle — their waiters hang for good.
  • The root pin goes. fix(ocap-kernel): make c-list import accounting symmetric #1020 made launchVat pin a vat's root for its
    lifetime and stopVat release it when terminating, so a vat retired without
    passing through stopVat left its root pinned, and uncollectable, for the
    life of the store.

A vat that is neither running nor persisted still throws.

removeVatFromSubcluster no longer reports a vat that is in no subcluster. It
is reached from deleteVat while a vat is being discarded — the one moment a
failure cannot be retried past — and a vat in no subcluster is already in the
state it asks for.

Relationship to #1023

#1023's #retireVat does the same three things and additionally makes them
atomic, which mine is not, and its stopVat gains the same isVatActive
tolerance. Its version is the better one. If #1023 lands first this PR should be
dropped in favour of it; what may be worth keeping is the exemplary test below,
which drives the state through restartVat rather than constructing it, and the
removeVatFromSubcluster tolerance. Flagging rather than resolving here, since
the two were developed in parallel.

Testing

First commit is the failing repro on its own: 6 tests. The exemplary one,
can be terminated after a failed restart, produces the state the way
production does — run a vat, fail its relaunch, then terminate it — rather than
asserting against a hand-built mock. The rest pin each of the three things a
retire has to do, plus the unknown-vat case as a control.

@metamask/ocap-kernel and @ocap/kernel-test green, with auditRefCounts on
for every kernel kernel-test builds. lint and build clean.

Checklist

  • I've updated the test suite for new or updated code as appropriate
  • I've updated documentation (JSDoc, Markdown, etc.) for new or updated code as appropriate
  • I've communicated my changes to consumers by updating changelogs for packages I've changed, highlighting breaking changes as necessary
  • I've prepared draft pull requests for clients and consumer packages to resolve any breaking changes

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 72.62%
⬆️ +0.08%
9726 / 13392
🔵 Statements 72.47%
⬆️ +0.07%
9889 / 13644
🔵 Functions 73.08%
⬆️ +0.09%
2281 / 3121
🔵 Branches 67.05%
⬆️ +0.16%
4013 / 5985
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
packages/ocap-kernel/src/store/methods/subclusters.ts 98.82%
⬆️ +0.02%
90.62%
⬆️ +0.62%
96.15%
🟰 ±0%
98.78%
⬆️ +0.02%
264
packages/ocap-kernel/src/vats/VatManager.ts 100%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
Generated in workflow #4695 for commit 6819b5d by the Vitest Coverage Report Action

@grypez
grypez force-pushed the grypez/retire-persisted-vat branch 3 times, most recently from 4b29d4c to 4926948 Compare August 26, 2026 16:27
@grypez
grypez force-pushed the grypez/retire-persisted-vat branch from 4926948 to a3e80cc Compare August 26, 2026 19:00
grypez and others added 2 commits August 26, 2026 15:32
`restartVat` stops a vat and then runs it again. A relaunch that fails —
the bundle has moved since the vat was launched, the worker will not spawn
— leaves the vat gone from the running map with its record, its vat store
and its root pin all still in place.

Nothing retires a vat in that state. `terminateVat` goes through `stopVat`
→ `getVat` and throws `VatNotFoundError`, so the only way to be rid of one
is to discard the whole store, and `terminateSubcluster` — which walks
persisted membership rather than the running map — rejects part way through
on reaching one, after deleting the system-subcluster mapping and before
removing the subcluster record.

Retiring one has to do what `stopVat` does besides stopping a worker, and
none of it is visible from `terminateVat` alone:

`deleteVat` takes the vat's config, its own store, and its subcluster
membership. `markVatAsTerminated` does not: the cleanup it schedules walks
keys prefixed `${vatId}.`, which never matches `vatConfig.${vatId}`. A
record left behind restores the vat at the next boot whose code is
reachable.

The promises the vat was deciding are rejected. `cleanupTerminatedVat`
deletes their c-list entries and drops the decider's refcount on the stated
understanding that its caller has already rejected them.

The pin `launchVat` took on the vat's root is released, or the root stays
pinned and uncollectable for the life of the store.

`removeVatFromSubcluster` reports a vat that is in no subcluster, which
`deleteVat` reaches while discarding a vat — the one moment a failure
cannot be retried past.

Co-Authored-By: Claude Opus 5 <[email protected]>
`terminateVat` went through `stopVat`, so a vat that is persisted but not
running threw `VatNotFoundError` and could not be retired at all. It now
does everything `stopVat` does apart from stopping a worker.

The record goes. `markVatAsTerminated` alone does not retire a vat: the
cleanup it schedules walks keys prefixed `${vatId}.`, which never matches
`vatConfig.${vatId}`, and that record is what the next boot restores from.

The decider promises are rejected, carrying the termination reason.
`cleanupTerminatedVat` deletes their c-list entries and drops the decider's
refcount on the stated understanding that its caller rejected them first;
nothing did, so they were left unresolved with a decider that no longer
existed and no way to ever settle.

The root pin goes. `launchVat` pins a vat's root for its lifetime and
`stopVat` releases it when terminating, so a vat retired without passing
through `stopVat` left its root pinned, and so uncollectable, for the life
of the store.

`removeVatFromSubcluster` no longer reports a vat that is in no subcluster.
It is reached from `deleteVat` while a vat is being discarded — the one
moment a failure cannot be retried past — and a vat that is in no
subcluster is already in the state it asks for.

Co-Authored-By: Claude Opus 5 <[email protected]>
@grypez
grypez force-pushed the grypez/retire-persisted-vat branch from a3e80cc to 6819b5d Compare August 26, 2026 19:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant