Repository navigation
Conversation
Both runtimes reconcile each controller connection on their own. A worker runs the drops its reconciliation decides, then waits at a barrier it shares with the worker of the same local ordinal on the other runtime, and creates dataflows only once that worker finished the drops of the same connection or a later one. Without the barrier, the interactive runtime could bind the peer of an index recreated at the same `as_of` to the slot of the incarnation the maintenance runtime had yet to drop, and then read that incarnation's frozen view. The multiplexing client records each connection's `Hello` in a process-wide order before forwarding it. Global worker 0 of each runtime runs in process 0, looks up a connection's position in that process's order when it sequences the connection's commands, and the command channel delivers the position with them to every worker of its runtime. The barrier compares connections by these positions. A worker never looks one up in its own process's order: the controller sends the other processes a connection's `Hello` but not its commands, so their orders can lack a connection their workers already reconcile. A later arrival counts because the barrier keeps only each side's latest arrival, which the other side can overwrite before the waiting worker rechecks, and because a runtime's channel adapter can drop a connection's queued commands once a response to it fails, so one runtime can finish a connection the other never does. A waiting worker steps its dataflows but reads no command, and passing is the only way out, so every worker of a runtime still executes the same commands in the same order. The clusterd-test-driver runs `reconciliation.spec` and the new `two_runtime_recreate_index.spec` against a clusterd with both runtimes, and a peek takes a `timeout-secs` so a read bound to a frozen incarnation fails instead of hanging. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This was referenced Nov 7, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
9 of 14 in the interactive-runtime stack. Stacks on #38391, followed by #38393. Tracked by CPU-216. The design document is
doc/developer/design/20261107_interactive_runtime.mdin #40559.Both runtimes reconcile each controller connection on their own. Without coordination, the interactive runtime can bind the peer of an index recreated at the same
as_ofto the slot of the incarnation the maintenance runtime has yet to drop, and then read that incarnation's frozen view.Worker::reconcileruns the compactions and drops it decides, then waits at aReconcileBarrierit shares with the worker of the same local ordinal on the other runtime, then runs the creates. A waiting worker steps its timely worker but reads no command, so every worker of a runtime still executes the same commands in the same order.Soloskips the barrier.ConnectionOrderthat clusterd's multiplexing client records eachHelloin before forwarding it. Global worker 0 of each runtime runs in process 0, looks up a connection's position there when it sequences the connection's commands, and the command channel delivers the position with them to every worker of its runtime. A worker never reads its own process's order, because the controller sends the other processes a connection'sHellobut not its commands.Inert:
enable_compute_interactive_runtimeis off, so no replica runs two runtimes.Tests:
reconcile_barrier/tests.rscovers passing on the same connection, both reasons a later arrival counts, the wake of a parked waiter.server/tests.rsruns two runtimes' reconciliations: the interactive runtime creates only after maintenance dropped the old incarnation, and a worker whose process never recorded the connection still reconciles.sharing/tests.rsadds the same scenario at the registry.workflow_two_runtime_compute, which starts clusterd with both runtimes (Clusterd(interactive_compute=True)) and runsreconciliation.specand the newtwo_runtime_recreate_index.specwith one and two workers. A peek'stimeout-secsmakes a read bound to a frozen incarnation fail instead of hang.A reviewer should check the NOTE that the drop phase waits on no other worker, and the argument for later arrivals in the module doc of
reconcile_barrier.rs.Posted by Claude Code.
🤖 Generated with Claude Code