Skip to content

compute-client: multiplex one command stream across two runtimes - #38388

Draft
antiguru wants to merge 1 commit into
mh/interactive-02-sharingfrom
mh/interactive-03-multiplex
Draft

antiguru wants to merge 1 commit into
mh/interactive-02-sharingfrom
mh/interactive-03-multiplex

Conversation

@antiguru

@antiguru antiguru commented Aug 21, 2026 •

Copy link
Copy Markdown
Member

5 of 14 in the interactive-runtime stack. Stacks on #38387, followed by #38389. Tracked by CPU-216. The design document is doc/developer/design/20261107_interactive_runtime.md in #40559.

This adds the multiplexer that fans a single command stream out to both runtimes and merges their responses into one, and the DataflowClass that decides which runtime renders a dataflow.

DataflowDescription gains a class: Maintained for indexes, materialized views, sinks, subscribes and copy-tos, OneShotRead for a dataflow that exists to answer one read. It is a class, not a destination, because the command history is shared by replicas with different runtime layouts, and a single-runtime replica renders every class. The adapter marks a peek dataflow OneShotRead where it already sets until one step past the as_of. A read at Timestamp::MAX has no finite until and stays maintained. The compute controller soft-asserts that a OneShotRead dataflow has the shape the interactive runtime can serve (transient exports, a single time, no subscribe, no copy-to), and rewrites a misfit to Maintained. The interactive runtime publishes nothing, so nothing else could import a durable export it rendered. compatible_with ignores the class, so a class change across a reconnect does not rebuild a dataflow a runtime already renders.

The multiplexer decides nothing. It forwards every command to both runtimes, except Peek and CancelPeek, which go to the interactive runtime because it serves every peek in a two-runtime process. Each runtime consumes its commands at its own rate, so the order of the two sends guarantees nothing. Each runtime reads the class: the runtime that renders it builds the dataflow, and the other records its exports as peers (#38391). Both runtimes seeing the same commands in the same order is necessary for an index's since to stay at or below the as_of of every dataflow importing it, but not sufficient: #38390 adds the peer hold that covers a runtime lagging behind the other. Each runtime reports frontiers only for the collections it renders, which are disjoint, so responses are forwarded verbatim.

Inert: nothing constructs a multiplexer. Independent of #38386 and #38387, it sits above them only to keep the stack linear.

Tests in multiplex/tests.rs cover routing, per-runtime command order, verbatim frontier forwarding, no lost message when both runtimes are ready, and either runtime ending the endpoint.

A reviewer should check the shape check and its fallback in Instance::create_dataflow.

Posted by Claude Code.

🤖 Generated with Claude Code

@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 92a68e1 to 1db2f7e Compare August 21, 2026 11:23
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 1db2f7e to b6ef7c0 Compare August 21, 2026 13:24
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from b6ef7c0 to 017ca4b Compare August 21, 2026 13:42
@antiguru
antiguru requested a review from DAlperin August 21, 2026 13:46
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch 2 times, most recently from 92b0861 to c595599 Compare August 21, 2026 17:54
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from c595599 to 2e15845 Compare August 28, 2026 14:06
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 2e15845 to 9fb1ae9 Compare September 3, 2026 08:48
@linear-code

linear-code Bot commented Sep 3, 2026

Copy link
Copy Markdown

CPU-215

@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 9fb1ae9 to 1a75556 Compare September 3, 2026 15:52
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 1a75556 to 05faeed Compare September 3, 2026 15:55
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 05faeed to ea5a4a5 Compare September 3, 2026 15:56
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from ea5a4a5 to 2068331 Compare September 4, 2026 15:50
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 2068331 to 12529cc Compare September 4, 2026 17:38
Comment thread src/compute-client/src/multiplex.rs Resolved
Comment thread src/compute-client/src/multiplex.rs Resolved
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 12529cc to f0fc9ed Compare September 4, 2026 19:37
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 23931cf to 16c7784 Compare September 23, 2026 14:09
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 16c7784 to 9e41882 Compare September 24, 2026 11:36
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 9e41882 to ed8b178 Compare September 25, 2026 10:54
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from ed8b178 to 6e6ab0a Compare September 25, 2026 17:05
Comment thread src/compute-client/src/multiplex.rs Resolved
Comment thread src/compute-client/src/multiplex.rs Resolved
Comment thread src/compute-client/src/multiplex.rs Resolved
Comment thread src/compute-client/src/multiplex.rs Resolved
Comment thread src/compute-client/src/multiplex.rs Resolved
Comment thread src/compute-client/src/multiplex.rs Resolved
Comment thread src/compute-client/src/multiplex.rs Resolved
Comment thread src/compute-client/src/multiplex.rs Resolved
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 6e6ab0a to 847362f Compare September 29, 2026 09:04
Comment thread src/compute-client/src/multiplex.rs Resolved
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch 4 times, most recently from 847362f to 6dfc394 Compare October 1, 2026 12:54
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 6dfc394 to 6bb7eaa Compare October 1, 2026 13:24
@antiguru
antiguru force-pushed the mh/interactive-03-multiplex branch from 6bb7eaa to 13ecc17 Compare October 1, 2026 14:16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants