[antiguru]

7 of 14 in the interactive-runtime stack. Stacks on #38389, followed by #38391. Tracked by CPU-215. The design document is doc/developer/design/20261107_interactive_runtime.md in #40559.

TraceBundle holds OksTrace and ErrsTrace, both instances of a generic IndexTrace enum: a trace this runtime maintains, or a SharedReader over one the other runtime publishes. The two share their batch and cursor types and implement TraceReader by delegation, so ArrangementFlavor::Trace carries either, and joins, delta joins, thresholds and reduces read both through one set of bodies. import_index has one body for every runtime: it reads compute_state.traces and imports what it finds there.

A runtime holds an index its peer publishes through ArrangementSharingRegistry::peer_bundle, a bundle whose logical compaction is that runtime's hold on the publication. The hold sits at the index's as_of. publish registers it before the publishing runtime can apply any compaction for the index, and peer_bundle takes it over, so a reading runtime that lags by a whole create still finds the index at the as_of of a read the controller has since cancelled. Reconciliation pads a peer bundle the way it pads a local trace. The next PR inserts these bundles into traces. TraceManager::maintenance skips them, since they hold nothing physically and setting their physical compaction would only take the point's lock.

A shared import never reads through the bundle's own handle. It mints a reader that holds the publisher's chain physically at the boundary it was seeded with, as a join's cut requires. The import is live: it follows the publisher's chain, presented at as_of and bounded by the dataflow's until, as the maintenance import is. The read hold is the returned Arranged's own trace rather than a separate token, so consumers can downgrade it as their frontiers advance. A token nobody downgrades would be a floor under every hold that is downgraded, since the publisher forwards the meet.

Reachable only when a runtime holds the Interactive role, which nothing constructs yet.

Tests in render/tests.rs and sharing/tests.rs cover import holds (at as_of, after construction, while live, on drop), the reserved hold in both orders of publish and peer, padded peer bundles, and joins and reduces over imports against the direct result, including over a merged chain.

A reviewer should check the reserved-hold pairing in publish and peer_bundle.

Posted by Claude Code.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- Delta join's SharedTrace arms re-emit input errors that bundle_errs already seeded

src/compute/src/render/join/delta_join.rs:431 (and :785)

Details

The two new SharedTrace arms concatenate the input arrangement's own error collection into the errors they return, but render_delta_join already seeds inner_errs once per input from bundle_errs, which this PR extended with a SharedTrace arm too. An error carried by an imported shared index is therefore emitted 1 + (number of delta paths that read it) times instead of once, which is exactly the per-path multiplication the surrounding code is written to avoid.

The sibling arms deliberately drop the bundle's errors: Some(ArrangementFlavor::Local(oks, _errs)) at :359 and Some(ArrangementFlavor::Trace(_, oks, _errs)) at :383, likewise at :760/:768 in build_update_stream. The contract is stated on both helpers ("The returned error collection holds only the errors this stage produces. The errors bundle already carries are the caller's to propagate, once") and on bundle_errs itself at :294, which spells out the consequence: "Propagating them per path would multiply their multiplicities by the number of paths, and because a join's output is another join's input, those factors compound multiplicatively through a nested plan until the Diff overflows."

bundle_errs gained the matching arm at :323, so both sources are live for this flavor. Concretely, a three-way delta join over imported indexes on the interactive runtime reads each input as one path's source and two paths' lookup, so each input's pre-existing errors land in the output four times; the inflation is visible in mz_compute_error_counts for an index export and compounds through nested joins.

Fix: bind the errors away, as the Trace arms do.

-        Some(ArrangementFlavor::SharedTrace(_, oks, errs)) => {
+        Some(ArrangementFlavor::SharedTrace(_, oks, _errs)) => {
             let (oks, errs2) = if source_precedes_lookup {
...
-            (oks, errs2.concat(errs.as_collection(|k, _v| k.clone())))
+            (oks, errs2)
         }

and the same in build_update_stream:

-        Some(ArrangementFlavor::SharedTrace(_, oks, errs)) => {
-            let (oks, errs2) = build_update_stream_trace::<_, SharedOksEnter<_>>(
-                oks,
-                as_of,
-                source_relation,
-                initial_closure,
-            );
-            (oks, errs2.concat(errs.as_collection(|k, _v| k.clone())))
-        }
+        Some(ArrangementFlavor::SharedTrace(_, oks, _errs)) => {
+            build_update_stream_trace::<_, SharedOksEnter<_>>(
+                oks,
+                as_of,
+                source_relation,
+                initial_closure,
+            )
+        }

2. LOW -- import_shared_index's contract is attached to report_compacted_past

src/compute/src/render.rs:563

The doc comment block that opens at :563 runs straight into report_compacted_past's own summary at :605 without a break, so the entire import contract, including the "caller MUST retain [the slot] for as long as the import is alive" requirement, renders as documentation for the panic helper, and import_shared_index itself is left undocumented.

Details

Split the block so that /// Reports a publication point refusing to serve as_of, and aborts. starts report_compacted_past's own comment and the preceding paragraphs move onto import_shared_index.

Related: the NOTE: on the new variant at src/compute/src/render/context.rs:244 says "the imported handles carry no TraceFrontier/until bound". They do: SharedOksEnter<T> = TraceEnter<TraceFrontier<SharedOksHandle>, T>, structurally identical to RowRowEnter, and import_index_shared passes a real until (as_of stepped forward). The accurate caveat is that the bound is derived from as_of rather than from the dataflow's self.until, which is what a multi-time import would need.

[antiguru]

All three fixed in 6807118. The delta join's SharedTrace arms bind the input errors away as the Trace arms do, so bundle_errs remains the one place that propagates them. The import contract now sits on import_shared_index, and report_compacted_past has its own doc. The NOTE on the variant says what is true: the bound is derived from as_of rather than from until, so the variant serves single-time dataflows only.

Posted by Claude Code.

[antiguru]

On the 09-05 finding about interactive_import_holds_after_construction: the publisher's standing hold no longer exists, and it was the reason the test passed without the import's own hold. Peers now hold logically through their peer handle, and the import holds through the reader it owns. I re-ran the mutation from the report on the current head: with the import's hold replaced by None in SharedReader::import_frontier_core, this test fails ("a built import must leave a read hold behind ...") and the other 20 tests in render::tests and sharing::tests still pass. The regression has coverage now, with no test change needed.

The peer-count finding is answered above. The other replies in this review describe intermediate designs (role dispatch in import_index, re-export aliases, shared_reexports) that the rework removed. Now there is one import_index over compute_state.traces, where a peer index is an ordinary TraceBundle.

Posted by Claude Code