Skip to content

compute: a reconcile barrier between the two compute runtimes - #40558

Draft
antiguru wants to merge 1 commit into
mh/interactive-06-runtimefrom
mh/interactive-07-reconcile-barrier
Draft

antiguru wants to merge 1 commit into
mh/interactive-06-runtimefrom
mh/interactive-07-reconcile-barrier

Conversation

@antiguru

@antiguru antiguru commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

9 of 14 in the interactive-runtime stack. Stacks on #38391, followed by #38393. Tracked by CPU-216. The design document is doc/developer/design/20261107_interactive_runtime.md in #40559.

Both runtimes reconcile each controller connection on their own. Without coordination, the interactive runtime can bind the peer of an index recreated at the same as_of to the slot of the incarnation the maintenance runtime has yet to drop, and then read that incarnation's frozen view.

  • Worker::reconcile runs the compactions and drops it decides, then waits at a ReconcileBarrier it shares with the worker of the same local ordinal on the other runtime, then runs the creates. A waiting worker steps its timely worker but reads no command, so every worker of a runtime still executes the same commands in the same order. Solo skips the barrier.
  • Connections are ordered by a ConnectionOrder that clusterd's multiplexing client records each Hello in before forwarding it. Global worker 0 of each runtime runs in process 0, looks up a connection's position there when it sequences the connection's commands, and the command channel delivers the position with them to every worker of its runtime. A worker never reads its own process's order, because the controller sends the other processes a connection's Hello but not its commands.
  • A worker passes once the other side arrived at the same connection or a later one. A later one counts because the barrier keeps only each side's latest arrival, which the other side can overwrite before the waiting worker rechecks, and because a runtime's channel adapter drops a connection's queued commands once a response send to it fails, so one runtime can finish a connection the other never does.

Inert: enable_compute_interactive_runtime is off, so no replica runs two runtimes.

Tests:

  • reconcile_barrier/tests.rs covers passing on the same connection, both reasons a later arrival counts, the wake of a parked waiter.
  • server/tests.rs runs two runtimes' reconciliations: the interactive runtime creates only after maintenance dropped the old incarnation, and a worker whose process never recorded the connection still reconciles. sharing/tests.rs adds the same scenario at the registry.
  • The clusterd-test-driver gains workflow_two_runtime_compute, which starts clusterd with both runtimes (Clusterd(interactive_compute=True)) and runs reconciliation.spec and the new two_runtime_recreate_index.spec with one and two workers. A peek's timeout-secs makes a read bound to a frozen incarnation fail instead of hang.

A reviewer should check the NOTE that the drop phase waits on no other worker, and the argument for later arrivals in the module doc of reconcile_barrier.rs.

Posted by Claude Code.

🤖 Generated with Claude Code

Both runtimes reconcile each controller connection on their own. A worker
runs the drops its reconciliation decides, then waits at a barrier it shares
with the worker of the same local ordinal on the other runtime, and creates
dataflows only once that worker finished the drops of the same connection or a
later one. Without the barrier, the interactive runtime could bind the peer of
an index recreated at the same `as_of` to the slot of the incarnation the
maintenance runtime had yet to drop, and then read that incarnation's frozen
view.

The multiplexing client records each connection's `Hello` in a process-wide
order before forwarding it. Global worker 0 of each runtime runs in process 0,
looks up a connection's position in that process's order when it sequences the
connection's commands, and the command channel delivers the position with them
to every worker of its runtime. The barrier compares connections by these
positions. A worker never looks one up in its own process's order: the
controller sends the other processes a connection's `Hello` but not its
commands, so their orders can lack a connection their workers already
reconcile. A later arrival counts because the barrier keeps only each side's
latest arrival, which the other side can overwrite before the waiting worker
rechecks, and because a runtime's channel adapter can drop a connection's
queued commands once a response to it fails, so one runtime can finish a
connection the other never does. A waiting worker steps its dataflows but reads no command, and passing
is the only way out, so every worker of a runtime still executes the same
commands in the same order.

The clusterd-test-driver runs `reconciliation.spec` and the new
`two_runtime_recreate_index.spec` against a clusterd with both runtimes, and a
peek takes a `timeout-secs` so a read bound to a frozen incarnation fails
instead of hanging.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant