Skip to content

compute: arrange through a Materialize-owned copy of differential's trace agent - #40512

Draft
antiguru wants to merge 1 commit into
mainfrom
mh/dd-next-vendored-agent
Draft

antiguru wants to merge 1 commit into
mainfrom
mh/dd-next-vendored-agent

Conversation

@antiguru

@antiguru antiguru commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

1 of 14 in the interactive-runtime stack, which supersedes #37770. Followed by #40513. The design document is doc/developer/design/20261107_interactive_runtime.md in #40559.

Compute's arrangements, reduces and imports now go through copies of differential's TraceAgent, TraceWriter and the two drivers that construct an agent, in mz_timely_util::arrange_agent. Behavior is unchanged. The interactive runtime needs to own the agent and writer so a second runtime can share an arrangement, and copying them lets that work proceed without a differential change. The copies come from differential revision 6074cc78, which the 0.26.0 release the workspace depends on ships unchanged.

  • agent.rs and writer.rs copy operators/arrange/agent.rs and writer.rs, including TraceBox. TraceReplayInstruction, ShutdownButton, Arranged and the frontier wrappers stay differential's.
  • arrangement.rs and reduce.rs copy arrange_core and reduce_with_tactic as arrange_core_with and reduce_with_tactic_with, which build their agent through a factory. reduce_trace_with uses differential's public CursorTactic. clippy.toml disallows calling arrange_core_with and reduce_trace_with directly, as it does their differential originals.
  • arrange_agent.rs holds the Writer, Agent and ArrangeContext traits, with the shape proposed in Let arranging operators take a caller-chosen agent and writer TimelyDataflow/differential-dataflow#959, and TraceAgent::factory(). Copying alone would give compute the agent. The factory and traits exist so the operator copies can go once differential takes coord: merge transient and coordinator modules #959. Until then TraceAgent::factory() is the only factory, and compute: a shared-trace primitive for cross-thread arrangement reads #38386 adds publishing by editing the copied agent and writer in place.
  • Copies are run through rustfmt and mark each deviation Not upstream, so a differential bump can re-derive them by diffing against the rustfmt-formatted revision named in each header. They carry mz-timely-util's Apache 2.0 header next to the MIT attribution.
  • Storage keeps using differential's TraceAgent.

A reviewer should check that every difference from the named upstream revision carries a Not upstream marker.

Posted by Claude Code.

🤖 Generated with Claude Code

…race agent

Copy differential's TraceAgent, TraceWriter, arrange_core and
reduce_with_tactic (with reduce_trace) into mz_timely_util::arrange_agent,
taken from differential-dataflow revision 6074cc78, whose copied sources
the 0.26.0 release the workspace depends on ships unchanged.
The drivers take a factory that builds the agent and writer, behind local
Writer and Agent traits shaped after differential issue #959, and
TraceAgent::factory builds them exactly as upstream does. The copies are
run through rustfmt and mark every deviation with a "Not upstream"
comment, so a later bump can re-derive them by diffing against a
rustfmt-formatted upstream. They carry mz-timely-util's Apache 2.0 header
next to the MIT attribution.

Compute's arrangements, reductions and imports now go through the copies.
Behavior is unchanged. Storage stays on differential's TraceAgent.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant