[antiguru]

3 of 14 in the interactive-runtime stack. Stacks on #40513, followed by #38387. Tracked by CPU-215. The design document is doc/developer/design/20261107_interactive_runtime.md in #40559.

An arrangement is normally readable only from the timely worker that maintains it. This PR lets the owned TraceAgent from #40512 publish its trace to a Shared point, a rendezvous (#40513) whose items are spans and whose hold kinds are logical and physical compaction.

• Once published, the agent's TraceBox mirrors the spine's span chain, upper, and compaction frontiers into the point on every insert, and on every exertion that changed them. The lock is held for a chain swap, never for merge work.
• An exertion reads the readers' holds only when one of them moved a meet (take_holds_moved), and publishes only when the frontiers, the chain's shape, or a point's reachability changed. An activation that changes nothing takes no lock.
• The box applies the meet of the agents' and the readers' holds to the spine, joined with the spine's own since, so a hold below it constrains nothing rather than rewinding the spine. The point publishes the since it is about to apply before applying it, the safe direction.
• SharedReader implements TraceReader, overrides read_upper to read the published upper without cloning the chain, and imports the point with the analogue of TraceAgent::import_frontier_core. A peer handle (Shared::reader_at_least) holds logically only. SharedReader::fresh_reader mints an import's reader that also holds the chain physically at the coverage it was seeded with, with a # Panics contract on where its hold may sit.
• The batch-size logger observes the trace through a weak observer, replacing trace_box_unstable. Published and adopt_trace in mz_compute::shared_trace are the Materialize-side glue.

Not inert in code paths: every production TraceWriter insert, exert and drop, and the batch-size logger, now run through the new code, which does nothing for a trace nobody published. Nothing in mz-compute publishes outside tests. The glue module carries #![cfg_attr(not(test), expect(unused))] with a TODO(CPU-215), which #38390 removes.

Tests in shared_trace/tests.rs cover reads and seals across threads, holds and their release, reader floors on merging, holds below the since, imports after the writer or the trace dropped, and a join over an import against the direct result.

A reviewer should check the lock order and ordering argument in sync_holds, and that exert's skip misses no reader move.

Posted by Claude Code.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- a live import permanently pins the published arrangement's physical compaction

src/compute/src/shared_trace.rs:1075

The read hold import_snapshot_at retains for the life of the import advances only on the logical axis, and Clone seeds its physical hold from the mint-time since rather than the chain coverage. The accumulated physical frontier therefore sticks at that stale since forever, the publisher never forwards anything higher, and the published spine stops merging: batches pile up in Spine::pending, one per seal, for the life of the import.

Details

Measured on a RowRowSpine published through adopt, with a live import_snapshot_at (until empty) consumed by as_collection, over 40 seal ticks against an identical unimported control:

after register: physical_holds={0: [4]}          coverage_hold=[4]  accumulated=[4]  since=[0]
after import:   physical_holds={2: [0]}          coverage_hold=[4]  accumulated=[0]  since=[0]
end:            physical_holds={2: [0]}          coverage_hold=[40] accumulated=[0]  since=[0]
chain_len: imported=39  control=5

Registration id 2 is the hold clone at shared_trace.rs:1075. register/register_at do install the hold at the chain coverage (id 0 above sits at [4], as shared_trace.rs:151 documents), but Clone at shared_trace.rs:625 writes self.physical into the new registration, and self.physical is initialised to since at shared_trace.rs:482 and shared_trace.rs:537. Nothing then moves it: shared_trace.rs:1167 follows acknowledged on the logical axis only, and the TraceFrontier clone that a join would advance is dropped at build time for every consumer that keeps only the stream. So physical_compaction's meet is pinned at the since observed when the handle was minted, agent.set_physical_compaction joins and cannot be pulled back down, and consider_merges never drains pending again because pending[0].upper() <= physical_frontier stays false.

Cost is unbounded rather than constant: retractions in stranded pending batches never consolidate, and every cursor_through builds a CursorList over a batch count that grows one per seal. On an index with a live shared import and a one-second seal cadence that is thousands of batches per hour.

Suggested fix, verified against the same probe (chain folds to 5, exactly matching the control, and the hold tracks to [40]):

                                 if let Some(hold) = hold.as_mut() {
                                     hold.set_logical_compaction(acknowledged.borrow());
+                                    hold.set_physical_compaction(acknowledged.borrow());
                                 }

acknowledged is the right value on both axes: it is exactly the frontier below which this import will never cut again, which is what the physical hold is supposed to express. Worth separately reconciling Clone at shared_trace.rs:625 with the coverage-seeded hold register/register_at install, since as written a clone silently lowers a registration's physical hold to since and the two paths disagree about the documented invariant at shared_trace.rs:151.

[antiguru]

Confirmed and fixed in 285af14. The measurement reproduces exactly: 39 batches against an unimported control's 5, over 40 seals.

Both of the report's points were real, and the second one is the root cause rather than a separate cleanup. A handle carries two physical frontiers that are not interchangeable. The one it reports through get_physical_compaction is seeded at the published since, because a reported frontier may never lead the chain coverage (mz_join_core asserts exactly that). The one it holds is seeded at that coverage, because a merge spanning the coverage destroys the boundary the reader was seeded with. Clone and set_physical_compaction both wrote the reported frontier into the hold, so the hold silently dropped to the weaker value, and since the accumulation is a meet, one such registration is a floor under every other hold.

The import's read hold hit this on both counts: it is a clone, so it registered at since, and it advanced only on the logical axis, so nothing raised it afterwards.

The fix keeps the two frontiers in separate fields, has Clone inherit the hold, has the setter join into both, and takes the suggested set_physical_compaction(acknowledged) on the import's hold. acknowledged is right on the physical axis for the reason given: it is exactly the frontier below which that import will never cut again.

Two regression tests, each verified red without its half of the fix:

• live_import_does_not_pin_merging — the two-arm chain-length comparison, 39 against 5 before, folding to the control after. One note on reproducing it: the minting handle has to be dropped after building the import, as render::import_shared_index does. A live mint holds its own coverage-seeded registration and pins the floor by itself, which masks the bug under test. My first attempt at this test failed for that reason rather than the one it was written for.
• clone_inherits_the_hold_not_the_reported_frontier — asserts the registered hold directly, since the reported frontier cannot be read back through TraceReader. This is what pins the Clone half; the merge test alone does not, because the added physical advance repairs that scenario regardless of what Clone seeded.

The coverage_hold this PR already gives the publisher caps the forwarded physical frontier at the chain coverage, so advancing a reader hold to acknowledged cannot push the forwarded value past what the chain carries even though acknowledged tracks the stream frontier, which leads the coverage by up to a scheduling round.

The stack above this PR is rebased and pushed. Full run at the tip: 124 tests, workspace cargo check --all-targets, clippy, rustdoc with -D warnings.

(Posted by Claude Code.)

[def-]

QA LLM Review

1. MEDIUM -- empty_logical_request_releases_the_hold can no longer fail

src/compute/src/shared_trace/tests.rs:687

Rewriting the assertion against the accumulated MutableAntichain made it vacuous: the handle registers at since and the standing hold sits at the same minimum time, so logical_holds() == standing_hold() holds identically whether or not the empty request released anything. The test named for the release property can no longer observe it, and no other test in the file does either.

Details

Verified by mutation. With set_logical_compaction changed to ignore an empty request outright, so the handle's hold is never released:

let next = self.logical.join(&frontier.to_owned());
if next.is_empty() { return; }   // hold is never released

all 17 shared_trace tests still pass. The concealed regression is real: an unreleased logical hold sits in logical_compaction for the life of the handle, the publisher forwards that frontier to its agent every activation, and the published arrangement stops compacting past it. That is exactly what the reduce operator triggers on every dataflow whose input finishes, per the test's own doc.

The accumulation is a meet, so a single hold at or above the standing hold is unobservable through logical_holds(). Releasing the standing hold first makes the accumulation carry the handle's contribution alone:

published.note_standing_hold(&Antichain::new());
assert!(published.logical_holds().is_empty());

let mut hold = published.handle();
assert!(!published.logical_holds().is_empty());
hold.set_logical_compaction(Antichain::new().borrow());
assert_eq!(published.logical_holds(), Antichain::new(), "...");

That version passes on the current code and fails on the mutation above.

[linear-code]

CPU-215

[DAlperin]

One question I have looking at the diagrams is why the import operator has to hold a direct handle to the ImportQueue when the registry already contains some of these as well

[antiguru]

@DAlperin The registry does not hold import queues. State keeps them as Vec<Weak<ImportQueue>>, and the import operator's Arc is the only strong reference (src/timely-util/src/shared_trace.rs, ImportQueue and import_frontier_core). The diagram makes this look otherwise. The State -> ImportQueue edge should read "Weak, pruned on publish", and I've updated it. The import operator holds the queue directly for three reasons.

• A queue belongs to one importer, not to an index. Each queue is seeded with the chain as it stood when that importer registered, and it holds only the instructions that importer has not drained yet. Two imports of the same index are at different positions. One queue per point would need a cursor per reader, plus trimming.
• Ownership is the lifetime. Queued batches are Arcs that keep memory alive. When the operator drops, its queue drops with it. At the next publish the writer's Weak fails to upgrade, and the writer prunes the entry (State::live_queues). If the registry held the queue strongly, every dropped import would need an explicit unregister, and a missed one would pin batches for the life of the slot.
• Draining takes no shared lock. Each activation locks only the operator's own queue. Reaching the queue through the registry would mean taking the registry mutex and then the State mutex on every drain, which contends with the writer publishing. It would also mean an id lookup, and the registry maps an id to its newest incarnation. An importer has to stay on the point that seeded it. When a new writer attaches, the old queues are cleared on purpose rather than moved over to the new chain.

The alternative is one append-only log in State with a cursor per importer, instead of a queue per importer. That saves a clone per importer when an index has many importers, but it moves the trimming and lifetime bookkeeping into State. It hasn't been measured, and nothing in the stack depends on it.

Posted by Claude Code.