Skip to content

compute: arrange through a Materialize-owned copy of differential's trace agent - #40504

Closed
antiguru wants to merge 1 commit into
MaterializeInc:mh/dd-nextfrom
antiguru:mh/dd-next-vendored-agent
Closed

antiguru wants to merge 1 commit into
MaterializeInc:mh/dd-nextfrom
antiguru:mh/dd-next-vendored-agent

Conversation

@antiguru

@antiguru antiguru commented Nov 5, 2026

Copy link
Copy Markdown
Member

Compute's arrangements, reduces and imports now go through copies of differential master-next's TraceAgent, TraceWriter and the two drivers that construct an agent, in mz_timely_util::arrange_agent. Behavior is unchanged. The interactive runtime needs to own the agent and writer so a second runtime can share an arrangement, and copying them lets that work proceed without a differential change. This builds on the master-next port (#38805) and is the base the interactive stack rebases onto.

  • agent.rs and writer.rs copy operators/arrange/agent.rs and writer.rs, including TraceBox. TraceReplayInstruction, ShutdownButton, Arranged and the frontier wrappers stay differential's.
  • arrangement.rs and reduce.rs copy arrange_core and reduce_with_tactic as arrange_core_with and reduce_with_tactic_with, which build their agent through a factory. reduce_trace_with uses differential's public CursorTactic.
  • arrange_agent.rs holds the Writer, Agent and ArrangeContext traits, with the shape proposed in Let arranging operators take a caller-chosen agent and writer TimelyDataflow/differential-dataflow#959, and TraceAgent::factory().
  • Copies keep upstream formatting under #[rustfmt::skip] and mark each deviation Not upstream, so a differential bump can re-derive them by diffing against the revision named in each header.
  • Storage keeps using differential's TraceAgent.

Posted by Claude Code.

🤖 Generated with Claude Code

…race agent

Copy differential's TraceAgent, TraceWriter, arrange_core and
reduce_with_tactic (with reduce_trace) into mz_timely_util::arrange_agent,
taken from the differential-dataflow revision the workspace pins. The
drivers take a factory that builds the agent and writer, behind local
Writer and Agent traits shaped after differential issue MaterializeInc#959, and
TraceAgent::factory builds them exactly as upstream does. The copies keep
upstream's formatting and mark every deviation with a "Not upstream"
comment, so a later bump can re-derive them by diffing against upstream.

Compute's arrangements, reductions and imports now go through the copies.
Behavior is unchanged. Storage stays on differential's TraceAgent.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@antiguru

antiguru commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Superseded by #40512, the same commit from a branch in this repository so the interactive stack can use it as a base. Posted by Claude Code.

@antiguru antiguru closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant