[antiguru]

Motivation

A compute replica maintains indexes and serves reads on the same timely workers, and timely does not preempt a running task. So a read waits behind maintenance work, not because the machine is out of CPU but because the one run loop is busy and cannot be interrupted. The sharpest form is introspection: mz_introspection describes a replica's own state and cannot be served from any other replica, so it is unavailable exactly when an operator needs it.

This is design only. No production code. It is the design of record for the implementation stack, #38386 through #38393, with the benchmarks in #38676 and #38678 and the hold-gap metric in #38736, and it describes what those pull requests ship.

Description

One document, design.md. It targets the interactive runtime: a second in-process timely runtime that renders temporary dataflows and serves reads off the maintenance runtime's arrangements, zero-copy through a per-process sharing registry.

It opens with the decomposition that shaped everything else: twelve reported symptoms, the mechanisms behind them, and which of sixteen candidate solutions reaches each. Every cell carries an evidence label distinguishing measured from argued from read-off-the-code, and the table deliberately includes mechanisms nothing here addresses. Two of the widest-blast-radius ones are among those, which is the point: a decomposition listing only the causes we have answers for is not a decomposition.

Several claims the work started with are retired in it:

• The peek walk substrate, not the second runtime, is what removes head-of-line blocking between peeks.
• Moving the walk off the worker is reproducibly worse than doing nothing when a peek queues behind a long operator activation.
• The second runtime's case is temporary dataflows and introspection during hydration. Judging the sharing registry and its protocol against a peek-latency claim weighs them against the wrong benefit.

Peek placement is therefore treated as an orthogonal axis and discussed once. It needs no second runtime and this design needs none of it. It shipped separately, as budgeted peek slices on the worker with offload past the budget, and #38449 is its design. The interactive runtime inherits that path unchanged.

The protocol section describes what shipped: the compute controller tags each dataflow with a DataflowClass, the multiplexer forwards every command to both runtimes, and the runtime that does not render a dataflow records its exports as peers, holding each peer index at its as_of in the TraceManager. It includes the counterexample that rules out broadcasting without the hold and the liveness coupling the hold introduces. Capping the frontier at the multiplexer, a synthesized AcquireHolds command, an in-process sequence barrier, and routing creates while broadcasting only compaction are recorded as rejected alternatives with the reason each failed.

What the current revision changed

The document now describes the reworked stack.

• Placement is the controller's: DataflowClass on the dataflow description, a passive broadcasting multiplexer, peers, and enable_compute_interactive_dataflows to keep peek dataflows off the interactive runtime.
• The registry is one per worker ordinal, holds slots weakly, gives every published id its own point instead of aliasing re-exports, and wakes the reading thread with an unpark instead of a dirty set.
• The shared import is an IndexTrace in the TraceManager, with one import_index for both provenances, so it honours SnapshotMode.
• A saturated staging run is recorded. Reads off arrangements stayed at the client floor through hydration, while persist-backed reads wait on the process-wide fetch semaphore (CPU-298).
• The latest stack review's fixes are described: the publisher registers the peer hold so a lagging reader cannot see a compacted index, registry slots are keyed by incarnation, and peer bundles are padded on reconnect.
• Pinning is described as unmeasured: the doc states what the code does and what an experiment would compare.
• The nightly numbers are from runs 18441 and 18442. The introspection read under hydration churn is the headline: p50 99 ms against 481 s at the merge base.

What moved out

The experimental evaluation is a project document under Interactive read isolation, which also tracks the work as issues. Evaluation artifacts do not belong in a design.

Both model checks are removed with their CI jobs. The TLA+ spec modelled the shipped mechanism; the Lean model's scope was the controller's read hold and the multiplexer's cap, and both of its counterexamples target behaviours no longer present in any form. The reasoning each contributed survives as prose in the rejected alternatives.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDm7opomJLxbNUEP3r9BLk

[bosconi]

The design needs a few corrections before it merges, mainly the inverted M4 comparison and the unsupported Solo claim; details inline.

• Add a Verification section to the PR description saying the document was checked against the stack, since that is the failure the revision notes describe fixing.
• The revision notes do not mention the new Simplifying the sharing protocol section, the only forward-looking design in the document.

[bosconi]

This section repeats 296-366 nearly verbatim, and the copies already disagree: 309 says concurrent scans and 392 says concurrent peeks. Cut it to the two new conclusions, that S3 stands alone on M3 and S4 is not separable, and link back to the evidence above.

[bosconi]

nit: the body says reading runtime; the heading still says rendering.

Suggested change

	#### The price: compaction is coupled to the rendering runtime's drain rate
	#### The price: compaction is coupled to the reading runtime's drain rate

[bosconi]

Give the hold-gap gauges an actionable threshold, even a provisional one, for when the interactive runtime has stopped draining and what to do then. This is the only detection for the one new price the design names. Also say here that the gauges are added in #38736, which is still a draft.

[bosconi]

Landing the specialization first removes ImportQueue, incremental replay and the per-read logical hold (1272-1282), which S14 (235-246), Compaction feedback and Non-goals (1490-1492) present as in place. Either say here that the general design is kept alongside the specialization, or say in those three places that they depend on the general design landing.

[bosconi]

As written this reads as the deferred build that Principle 3 rules out (766-769), and the import needs a real trace at construction (1033-1036). Say what the import holds between construction and the snapshot, and that the wait is inside the operator, not the build.

[bosconi]

M4's floor of about 900 ms is smaller than the roughly 1400 ms tail that placement recovers (321-323, E7's 2835 to 1456 ms), so it is not the dominant term. The same correction is needed at 320-323, 379-380 and 439-441.

Suggested change

	loaded, on either runtime. It is larger than the tail that runtime placement
	recovers, so for the workload M3's remedy is justified by, this is the dominant
	term.
	loaded, on either runtime. At about 900 ms it is roughly two thirds of the tail
	that runtime placement recovers, so for the workload M3's remedy is justified
	by, it is a first-order cost next to that benefit.

[bosconi]

I1c names a rate, not a relation, so no assertion or test can check it. State it as a predicate, matching the restatement at 1205-1206 and what a_peer_bundle_pins_until_the_importing_runtime_applies asserts.

Suggested change

	> **I1c.** A shared arrangement compacts only as fast as the slowest runtime's stream
	> position.
	> **I1c.** For every published index `I`, the publishing trace's `since` is at or below the
	> reading runtime's peer hold on `I`: the `as_of` of `I`'s `CreateDataflow`, advanced by every
	> `AllowCompaction` for `I` that the reading runtime has applied.

[bosconi]

typedefs.rs:46-55 wraps every spine in SharedSpine with no role or flag gate, so Solo is not behaviorally identical to a pre-work deployment, and the wrapper's cost is unmeasured. The summary at 20-21 makes the same claim.

Suggested change

	* `Solo` is the sole runtime of a single-runtime process and is behaviorally identical to a
	deployment from before this work. Its metric and log label is `None`, so a single-runtime
	registration collides with nothing and looks unchanged.
	* `Solo` is the sole runtime of a single-runtime process. It renders every class, publishes
	nothing and reads nothing from the registry, and its metric and log label is `None`, so a
	single-runtime registration collides with nothing and looks unchanged. It is not unchanged
	underneath: every spine in `mz_compute::typedefs` is a `SharedSpine` regardless of role, so
	`Solo` runs every trace call through the unattached wrapper.

[bosconi]

1077-1080 says the published since is the frontier the wrapper is about to apply, matching apply_holds. "Reads both off the spine" describes the previous order.

Suggested change

	only approximate. The wrapper reads both off the spine, and the registry stopped
	forwarding the controller's frontier.
	only approximate. The wrapper publishes the spine's chain and the exact `since` it is about
	to apply, and the registry stopped forwarding the controller's frontier.

[bosconi]

This cell is censored by the load phase, not a measured latency; mark it as a lower bound so nobody derives a speedup ratio from it.

Suggested change

	| Introspection read under hydration churn | 99 / 168 ms | 481 / 519 s |
	| Introspection read under hydration churn | 99 / 168 ms | > 481 / > 519 s |