Skip to content

A notify skipped for an absent endpoint is lost, not deferred #1033

Description

@grypez

Follow-up from #1029, and input to #979.

The loss

A notify addressed to an endpoint that is not running is dropped. Not deferred
— dropped, and the resolution it carried is then collectable, so the answer
ceases to exist anywhere in the kernel.

The chain, all on grypez/absent-endpoint-deliveries (#1029) and verified
against the code:

  1. Vat v2 subscribes to kernel promise kp7.
  2. v2 stops running while still named by persisted state — a terminated vat
    awaiting cleanupTerminatedVat (which runs one vat per crank, so a
    subcluster teardown leaves an N-crank window), or a vat whose relaunch
    failed, or a vat skipped at boot once fix(ocap-kernel): boot past a vat whose code can no longer be loaded #1031 lands.
  3. kp7 resolves. KernelQueue.resolvePromises walks the subscriber list and
    calls enqueueNotify(v2, kp7) once per subscriber, taking a 'notify'
    reference on kp7. The subscription is consumed at that point; nothing
    re-notifies later.
  4. The crank dequeues the notify. #deliverNotify releases the 'notify'
    reference up front (decrementRefCount(kpid, 'deliver|notify'), moved there
    by fix(ocap-kernel): make c-list import accounting symmetric #1020 so the early-return paths don't leak it), finds the endpoint absent,
    and returns.
  5. If that was the last reference, collectGarbage reaches the promise branch,
    sees refCount === 0 for a settled promise, decrements its value's slots and
    calls deleteKernelPromise(kp7) — which deletes kp7.state, kp7.value,
    kp7.subscribers and its queue.

Net: when v2 comes back, its liveslots still holds a p-NN for kp7 that can
never settle, and the kernel has no record of kp7 at all. The vat waits
forever on an answer that was computed and then discarded.

This is not a regression from #1029

Before that PR the same notify threw VatNotFoundError out of the crank and
killed the run loop permanently — and, since the crank rolled back, killed the
next boot too. Skipping is a strict improvement. But it converts kernel death
into silent, unrecoverable loss, and that residual is worth naming rather than
leaving implied.

Why the other two housekeeping deliveries are different

Only notify carries a promise:

Delivery Carries Anything to preserve?
notify kpid yes — the resolution
dropExports / retireExports / retireImports krefs, and parseAction does insistKernelType('object', kref) no — and since #1020 the kernel's own half must run at delivery time or the action is re-derived on every sweep
bringOutYourDead endpointId only no — a stale reap is worthless; scheduleReap on return instead

So this is specifically a notify problem, not a general "queue everything for
absent endpoints" problem.

Design sketch

The shape that seems to fit, offered as a starting point rather than a decision:

  • Park skipped notifies in a per-endpoint durable queue, holding references
    the way enqueuePromiseMessage already does for messages queued on an
    unresolved promise (promise.ts, incrementing target/result/slots). That
    precedent is the closest existing machinery.
  • Replay on restore, before the endpoint takes any other delivery.
  • Drain to rejections when we give up on the endpoint. terminateVat is
    already that moment — fix(ocap-kernel): finish retiring a vat that is not running #1030 makes it work for a vat that is persisted but not
    running, and it already rejects the promises the vat was deciding.

What not to do: leave the item on the run queue. That is what the pre-#1029
code did by accident via crank rollback, and it livelocks — re-dequeued and
re-thrown every crank.

Costs to weigh

These are why this doesn't belong in the #1029#1031 stack:

  • Durable and unbounded. The queue must survive reboots, since the endpoint
    is absent across them. An endpoint that never returns accumulates forever,
    until something gives up on it.
  • It pins the closure. To be replayable, parked notifies must hold
    references, so a derelict vat keeps alive everything it was about to be told
    about. That is the real content of "awaiting the day the bundle returns" — and
    the reason the current code releases up front.
  • The refcount audit needs a new credit. fix(ocap-kernel): make c-list import accounting symmetric #1020's recomputeRefCounts
    derives ground truth from c-lists, run-queue and promise-queue messages,
    resolution values and pins. A parking queue is a new category, or every kernel
    kernel-test builds starts reporting violations.
  • It presumes an answer to Define explicit subcluster lifecycle #979. Whether a vat that cannot run is
    suspended or dying is the lifecycle question; parking is only correct
    under the first reading.

Scope note

#1031's description currently states the committed semantics plainly — data
preserved, interactions not — rather than claiming resumability. If this issue
is resolved in favour of parking, that wording is what should change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions