[antiguru]

11 of 14 in the interactive-runtime stack. Stacks on #38393, followed by #38678. Tracked by CPU-216. The design document is doc/developer/design/20261107_interactive_runtime.md in #40559.

Benchmarks and regression tests for the read paths the interactive runtime changes, so a regression shows up in the nightly comparison against the previous build.

Feature benchmark. scenarios/interactive_runtime.py adds a peek dataflow joining two indexed views (the temporary-dataflow floor) and a per-replica introspection read. The benchmark's replica is an unmanaged one backed by the composition's clusterd container, which enable_compute_interactive_runtime never reaches, so the container's second runtime is keyed off that parameter, defaulting to the value get_default_system_parameters() gives environmentd. --this-params/--other-params with enable_compute_interactive_runtime=false measure the runtime's cost on a single build.

Parallel benchmark. TemporaryDataflowFloor is a closed loop of a join over two small indexed tables on a quiet replica, so the reported latency is the dataflow's fixed cost. IntrospectionUnderHydration reads mz_dataflow_arrangement_sizes at a fixed rate on the replica that hydration churn saturates. FreshnessUnderPeekDataflows runs peek dataflows, which offload does not move off the worker, on the replica that maintains a materialized view, measured by a strict serializable read of the view from another cluster.

Two scenarios guard peek offload rather than the second runtime and run on a replica with one runtime. PeekIsolationUnderExpensivePeeks checks that a point lookup does not queue behind full index walks. Both its queries are peeks, so a second runtime would move them together and change nothing it measures. FreshnessUnderPeekWalks checks that those walks do not hold back a maintained index's frontier, and a second runtime would hide that regression by serving the walks away from the maintenance worker. A new SystemParameterPhase sets the flag around the replica's creation.

The measured loops of every scenario with contention, ReadIsolationUnderHydration included, now gate regressions at CONTENDED_THRESHOLDS, 1.5 times the previous build on qps, avg, p50 and p95, with p99 reported but ungated. The thresholds are not yet backed by measured run-to-run spread.

Regression tests. interactive_runtime.slt pins the flag on a two-worker replica and covers duplicate indexes and re-exports across drops, error results on the fast path and through a peek dataflow, enable_compute_interactive_dataflows off, strict serializable reads, cluster re-provisioning, and introspection reads across the runtime boundary.

Not covered: an assertion that a query ran on the interactive runtime (CPU-222), and cancellation of a parked shared peek.

Benchmarks and tests only, no product code changes. A reviewer should check the gating choices against the scenario docstrings.

Posted by Claude Code.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. HIGH -- feature-benchmark scenarios never run on a two-runtime replica

misc/python/materialize/feature_benchmark/scenarios/interactive_runtime.py:12

The four new scenarios run on a replica that can never have an interactive runtime, so they measure a single-runtime clusterd on both sides of the nightly comparison and give no signal for the feature they exist to guard. The module docstring and the PR body both assert the opposite, so a green result here will be read as evidence the interactive read paths did not regress.

Details

test/feature-benchmark/mzcompose.py:298-300 creates cluster_default as an unmanaged replica (COMPUTECTL ADDRESSES ['clusterd:2101'] … WORKERS 1) and makes it the system default cluster, so every benchmark query runs on the clusterd container. create_replica ignores interactive_runtime on the ReplicaLocation::Unmanaged arm (src/controller/src/clusters.rs:467-478); only provision_replica acts on it. So enable_compute_interactive_runtime=true, which get_default_system_parameters() does put in environmentd's MZ_SYSTEM_PARAMETER_DEFAULT, has no effect at all in this composition.

The container is built by create_clusterd_service (test/feature-benchmark/mzcompose.py:393-398) as a bare Clusterd(image=…), i.e. interactive_compute=False, so no CLUSTERD_INTERACTIVE_COMPUTE_TIMELY_CONFIG is set and clusterd starts one runtime (src/clusterd/src/lib.rs:516-557, where the second runtime is gated purely on that argument).

Note the parallel-benchmark half of the PR is unaffected: test/parallel-benchmark/mzcompose.py uses orchestrated replicas, so the flag does reach them.

Fix — configure the second runtime on the container, since environmentd is not in a position to:

 def create_clusterd_service(
     clusterd_image: str | None,
     default_size: int,
     additional_system_parameter_defaults: dict[str, str] | None,
 ) -> Clusterd:
-    return Clusterd(image=clusterd_image)
+    # Benchmark queries run on `cluster_default`, an unmanaged replica backed by
+    # this container. environmentd never provisions it, so
+    # `enable_compute_interactive_runtime` never reaches it and the runtime has
+    # to be configured here. An older image ignores the unknown env var and
+    # stays single-runtime, which is the intended this-vs-other comparison.
+    return Clusterd(image=clusterd_image, interactive_compute=True)

Clusterd derives the interactive timely config from the same workers value as the maintenance one (default 1), which matches the WORKERS 1 in the replica definition.

2. MEDIUM -- qps is not in CONTENDED_THRESHOLDS, so the loosening does nothing for FreshnessUnderPeekWalks

misc/python/materialize/parallel_benchmark/scenarios.py:1278

CONTENDED_THRESHOLDS covers avg/p50/p95/p99 but not qps, which falls through to the default 1.2. For a closed loop qps is just 1/mean latency, so FreshnessUnderPeekWalks still fails on a 20% mean-latency increase even though its avg threshold was deliberately set to 1.5, and the scenario will report regressions at exactly the run-to-run variance the constant exists to tolerate.

Details

check_regressions resolves the threshold per stat as scenario.regression_thresholds.get(query, {}).get(stat) or REGRESSION_THRESHOLDS[stat] (test/parallel-benchmark/mzcompose.py:627-633), and less_than_is_regression("qps") is true, so the test is this_qps < other_qps / 1.2, i.e. mean_this > 1.2 * mean_other. The measured action at scenarios.py:1786-1792 is a ClosedLoop, whose Statistics.qps = queries / elapsed, so qps and avg are the same signal inverted and the stricter of the two binds.

The two existing scenarios that gained report_regressions=True are unaffected: their measured loops are OpenLoop at a fixed Periodic rate, where qps is the offered rate and does not move. TemporaryDataflowFloor is also a closed loop but uses the defaults throughout, so it is self-consistent.

Adding "qps": 1.5 to CONTENDED_THRESHOLDS fixes it without affecting the open-loop scenarios.

[antiguru]

Both confirmed and fixed in 7d7df7c.

1. create_clusterd_service now passes interactive_compute=True, with a comment on why the system parameter cannot reach the unmanaged replica. The module doc and PR body now describe the container configuration instead of the flag.
2. CONTENDED_THRESHOLDS gains "qps": 1.5.

Posted by Claude Code.

[def-]

QA LLM Review

1. MEDIUM -- CONTENDED_THRESHOLDS turns the p99 gate on for four scenarios rather than loosening it

misc/python/materialize/parallel_benchmark/scenarios.py:1282

REGRESSION_THRESHOLDS["p99"] is None, i.e. p99 is deliberately not checked for any query in the suite. Putting "p99": 2.0 in CONTENDED_THRESHOLDS therefore does not loosen anything: it newly subjects the four scenarios that use the constant to a p99 gate that no other query in the benchmark has, and it does so on exactly the loops whose tails are the noisiest in the suite. The likely outcome is red nightlies from run-to-run tail variance rather than from a real regression.

Details

check_regressions resolves a threshold as scenario.regression_thresholds.get(query, {}).get(stat) or REGRESSION_THRESHOLDS[stat] (test/parallel-benchmark/mzcompose.py:627-633), and Statistics.__dir__ (test/parallel-benchmark/mzcompose.py:260-272) includes p99, so a per-query 2.0 wins over the global None and the comparison runs.

The four other entries do loosen: qps 1.5 vs 1.2, avg 1.5 vs 1.2, p50 1.5 vs 1.2, p95 1.5 vs 1.3. p99 is the only one that goes from unchecked to checked, which also contradicts the constant's own rationale ("contended tails vary more between runs than a quiet query's") -- a quiet query's p99 is not checked at all.

This bites hardest on the two scenarios this PR flips from report_regressions=False to True. ReadIsolationUnderHydration and PeekIsolationUnderExpensivePeeks previously had no gate on any stat; they now get a p99 gate whose sensitivity their own docstrings warn about ("Check queries on both sides before comparing percentiles. A query that raises is logged and dropped rather than recorded, so a run that lost samples reports percentiles over the ones that survived, and the ones that fail are the ones taken when the replica was worst.").

Dropping the "p99" key restores the intended behaviour: the four remaining entries loosen the checked stats, and p99 stays reportable-but-not-gating like everywhere else. Keeping it is fine too, but then it is a new gate and worth saying so rather than filing it under "looser thresholds for contended tails".

[def-]

QA LLM Review

1. MEDIUM -- PointLookup and CreateIndexPublish are exact duplicates of existing feature-benchmark scenarios

misc/python/materialize/feature_benchmark/scenarios/interactive_runtime.py:69

Two of the four new scenarios render byte-identical testdrive scripts to FastPathFilterIndex and CreateIndex, at the same SCALE, on the same replica. Now that create_clusterd_service turns the second runtime on for the whole composition, those two existing scenarios already measure these shapes on a two-runtime clusterd, so the new pair adds no signal and doubles the nightly cost of two of the more expensive workloads.

Details

PointLookup (interactive_runtime.py:69) vs FastPathFilterIndex (benchmark_main.py:120): both init build v1 AS SELECT <unique_values> AS f1 FROM <join> plus CREATE DEFAULT INDEX ON v1, and both benchmark bodies are SET auto_route_introspection_queries TO false / BEGIN / SELECT 1 /* A */ / 1000 x SELECT * FROM v1 WHERE f1 = 1 / SELECT 1 /* B */. REPEAT = 1000 equals the existing range(0, 1000), and both inherit SCALE = 6.

CreateIndexPublish (interactive_runtime.py:109) vs CreateIndex (benchmark_main.py:840): identical init (t1 filled from <unique_values>, then SELECT 1 FROM t1 WHERE f1 = 0) and identical benchmark (DROP INDEX IF EXISTS i1 /* A */, CREATE INDEX i1 ON t1(f1), the two-way self-join on f1 = 0 at /* B */). Normalising for ;, case and whitespace, the only textual difference between the two classes is that the original carries a # Make sure the dataflow is fully hydrated comment.

Both duplicates also run under --root-scenario Scenario (the default), so a regression in the fast path or in CREATE INDEX now shows up as two report rows under different names, which reads as two independent signals during triage. PeekDataflowJoin and IntrospectionRead are genuinely new shapes and are unaffected.

Dropping the two duplicated classes keeps the coverage: their twins now run against the two-runtime container. If the intent was a distinct shape, they need to differ from it -- for example a lookup outside an explicit transaction, so per-read timestamp selection is in the measurement, or a CREATE INDEX whose first reader is a fast-path peek rather than a join.

[antiguru]

Both remaining review items confirmed and fixed in 18ac9dc.

• PointLookup and CreateIndexPublish are gone. They rendered the same testdrive as FastPathFilterIndex and CreateIndex, which run on the same two-runtime container, so they were two report rows for one signal. The module doc names the originals so nobody re-adds them.
• "p99" is out of CONTENDED_THRESHOLDS. The global threshold for p99 is None, so the entry created a gate on the noisiest loops rather than loosening one. The four remaining entries loosen stats the suite already gates.

Posted by Claude Code.

[def-]

QA LLM Review

1. MEDIUM -- interactive_compute=True ignores --this-params/--other-params, unlike the unified_cluster line next to it

test/feature-benchmark/mzcompose.py:411

After the rebase, create_clusterd_service reads unified_cluster from the effective system parameters, so that "any --this-params or --other-params override apply to the cluster being measured". The new interactive_compute is hardcoded instead. Setting enable_compute_interactive_runtime for either side therefore has no effect, and a same-build on-vs-off A/B compares two identical two-runtime replicas and reports no delta.

Details

doc/developer/feature-benchmark.md:51-55 documents --this-params/--other-params as the way to benchmark a product feature. The PR body says the on-vs-off delta shows up only on "the first build carrying the runtime". After that build, both nightly sides run two runtimes, and params are the only way left to measure the runtime's cost on a build. environmentd does receive the override, but the benchmark replica is unmanaged, and create_replica drops interactive_runtime on the Unmanaged arm (src/controller/src/clusters.rs:467). The container's CLUSTERD_INTERACTIVE_COMPUTE_TIMELY_CONFIG is fixed at service construction.

Keying the flag off the same dict, and defaulting it to the CI default ("true", misc/python/materialize/mzcompose/__init__.py:400), keeps today's behaviour and makes the override work:

+    interactive_compute = (additional_system_parameter_defaults or {}).get(
+        "enable_compute_interactive_runtime", "true"
+    ) == "true"
     return Clusterd(
         image=clusterd_image,
         unified_cluster=unified_cluster,
-        interactive_compute=True,
+        interactive_compute=interactive_compute,
     )

[antiguru]

Confirmed and fixed in 3b916a5. create_clusterd_service now reads enable_compute_interactive_runtime from the effective system parameters, like enable_unified_cluster next to it, defaulting to the CI default of true. A --this-params/--other-params override now reaches the benchmark replica, so a same-build on-versus-off comparison measures the runtime. The PR body says so instead of pointing at the first build carrying the runtime.

Posted by Claude Code.

[def-]

QA LLM Review

1. MEDIUM -- FreshnessUnderPeekWalks's walks are offloaded off the timely worker, so they never hold back the frontier it measures

misc/python/materialize/parallel_benchmark/scenarios.py:1811

The contention is a fast-path index walk. Since #38509, any fast-path index walk that passes 1024 cursor positions moves to a tokio task, and that is on by default in production and CI. So on both sides of every comparison the walks spend almost no time on the worker that maintains fresh_w. The scenario therefore never produces the walk-induced frontier lag that its docstring and the PR body say the read's latency measures. A gated result from it says nothing about whether peeks still delay maintenance.

Details

SELECT v FROM fresh_hot WHERE v = -1 filters on a non-key column. It is a fast-path PeekExisting that walks all 100,000 positions, and fuel counts positions visited, not rows returned. enable_compute_index_peek_offload defaults to true (src/compute-types/src/dyncfgs.rs:700), and compute_index_peek_inline_budget is 1024 (:719). Both are left at their code defaults in get_default_system_parameters, and ADDITIONAL_BENCHMARKING_SYSTEM_PARAMETERS does not touch them. A scan that exhausts its fuel returns PeekStatus::Offload (src/compute/src/compute_state.rs:1869), and the rest of the walk runs in OffloadedPeek::start's spawned task (src/compute/src/compute_state/peek_offload.rs:242). That holds for the single-runtime build, for the two-runtime build, and for a same-build --this-params enable_compute_interactive_runtime=false A/B. Whatever the measured read reports, it is strict serializable read latency under a concurrent INSERT loop. Removing the walks would very likely give the same numbers.

The same mechanism probably explains the observation recorded in PeekIsolationUnderExpensivePeeks (scenarios.py:1554) that at 100,000 positions and 2/s "the lookups did not notice the walks at all". The walks were not on the worker, so "60% of that worker" does not describe the nightly configuration either.

To get contention onto the maintenance worker, the walk has to render a dataflow, since peek dataflows are not offloaded. For example, SELECT count(*) FROM fresh_hot WHERE v = -1 renders on the maintenance worker on a single-runtime replica and on the interactive worker on a two-runtime replica. The measured read is itself a peek dataflow on the serving runtime, so it will still share a worker with that contention when two runtimes run. Observing fresh_w's frontier from outside the replica avoids conflating the two effects.

[antiguru]

Re the QA review: the offload mechanism is right, and the walks leaving the worker is the behavior both scenarios guard. PeekIsolationUnderExpensivePeeks checks that a lookup does not queue behind a walk, and FreshnessUnderPeekWalks checks that bounding a walk's time on the worker keeps a maintained index fresh. A regression in offload shows up in either.

What changed:

• Both run on a replica of their own with the interactive runtime off, because a second runtime would take the walks off the maintenance worker whether or not offload bounds them. A new SystemParameterPhase sets the flag before the replica is created and skips it on a build that predates it.
• The docstrings now say what each scenario guards. The sizing is the cost the walks would have on the worker in the regression case.
• The second runtime's effect on freshness has its own scenario, FreshnessUnderPeekDataflows. Its contention is peek dataflows, which offload does not move off the worker, and it reads a materialized view from another cluster so the measured read shares a worker with neither the contention nor the maintenance.

Posted by Claude Code.

[def-]

QA LLM Review

1. MEDIUM -- FreshnessUnderPeekWalks raises in its constructor, so it never runs and takes its nightly shard down with it

misc/python/materialize/parallel_benchmark/scenarios.py:1803

replace(conn_infos["materialized"], cluster="fresh") gives the two ReuseConnQuery actions a PgConnInfo whose connect() runs SET cluster = fresh with autocommit off, so psycopg opens a transaction. ReuseConnQuery._reconnect then sets autocommit = True, and psycopg 3 rejects that with ProgrammingError: can't change 'autocommit' now: connection in transaction status INTRANS. The scenario fails before any phase runs. The exception escapes run_once, which aborts the rest of the shard and drops the results of scenarios that already ran in it.

Details

PgConnInfo.connect (misc/python/materialize/util.py:170-172) issues the SET on a fresh non-autocommit connection, and psycopg sends BEGIN first. ReuseConnQuery.__init__ calls _reconnect eagerly (framework.py:243). Scenario construction (test/parallel-benchmark/mzcompose.py:531) sits outside the try, so the error propagates out of workflow_default. I reproduced this against a Materialize container with the repo's own PgConnInfo and ReuseConnQuery. The same construction without cluster succeeds. No nightly has run on this branch, so it hasn't shown up yet.

With the nightly's 5-way sharding by sorted name, FreshnessUnderPeekWalks shares shard 1 with CommandQueryResponsibilitySegregation, which runs first and loses its results, and with MySQLReadReplica, PgReadReplica and StatementLogging, which never run. FreshnessUnderPeekDataflows and the other new scenarios are unaffected because they use the unmodified conn_infos["materialized"].

Setting autocommit before the SET fixes it. I checked this against the same container: the session reports cluster = fresh, status IDLE, and isolation strict serializable.

-        mz = replace(conn_infos["materialized"], cluster="fresh")
+        mz = replace(conn_infos["materialized"], cluster="fresh", autocommit=True)

Fixing it in PgConnInfo.connect (for example by committing after the SET) would also cover the same pattern in StagingBench.

[antiguru]

Confirmed and fixed with the suggested autocommit=True (2e10a2f, now 606109c after a restack). Nightly 18441 hit it in shard 2, as predicted: the constructor raised and the shard lost the scenarios after it. Nightly 18442 on the fix ran the whole shard. FreshnessUnderPeekWalks and PeekIsolationUnderExpensivePeeks both compare flat against main, which is expected with the interactive runtime off.

I left PgConnInfo.connect unchanged. Committing after the SET would change behavior for every caller of a shared helper, which belongs in a change of its own.

Posted by Claude Code.

[bosconi]

Nothing blocking. The FreshnessUnderPeekWalks measured read is the one change that affects what a scenario measures, and the rest are sizing, a scenario version and a deprecated variable name, details inline.

• Please run a nightly at the current head before merge and record it in a Verification section. The last one, nightly 18245, predates the rework of the stack underneath.

[bosconi]

count(*) makes every measured read a peek dataflow on the same single worker whose frontier is being measured, and the unthrottled (0, 0) inserts grow the counted group during the run. A fast-path lookup of an untouched key keeps the work constant and still waits on the same frontier. The threshold key at L1887 changes with it.

Suggested change

	"SELECT count(*) FROM fresh_w WHERE k = 0",
	"SELECT v FROM fresh_w WHERE k = 1",

[bosconi]

Goes with the measured query change above.

Suggested change

	"SELECT count(*) FROM fresh_w WHERE k = 0 (reuse connection)": CONTENDED_THRESHOLDS,
	"SELECT v FROM fresh_w WHERE k = 1 (reuse connection)": CONTENDED_THRESHOLDS,

[bosconi]

nit: auto_route_introspection_queries is the deprecated name kept by compat_translate, and auto_run_on_catalog_server never routes a read with a per-replica log dependency, so the comment's reason does not apply.

Suggested change

	"SET TRANSACTION_ISOLATION TO 'SERIALIZABLE'",
	# The read must run on the replica being hydrated, not be routed
	# to the catalog server.
	"SET auto_route_introspection_queries TO false",
	"SET TRANSACTION_ISOLATION TO 'SERIALIZABLE'",
	# A relation with a per-replica log dependency is never
	# auto-routed to the catalog server, so this only makes the
	# replica choice explicit.
	"SET auto_route_catalog_queries TO false",

[bosconi]

At SCALE = 5 the 100,000-row join dominates the measurement, not the dataflow build and teardown the docstring describes. If the fixed cost is the target, shrink the inputs and raise the repeats. If the join work is meant to count, update the docstring instead.

Suggested change

	SCALE = 5
	REPEAT = 10
	SCALE = 3
	REPEAT = 100

[bosconi]

The inherited SCALE = 6 builds a 1,000,000-row view per side, but the measured mz_dataflow_arrangement_sizes read does not depend on row count.

Suggested change

	REPEAT = 100
	FIXED_SCALE = True
	SCALE = 3
	REPEAT = 100

[bosconi]

The row count, walk rate and cluster all change in this PR, so this scenario's stored percentiles are no longer comparable. Bumping the version separates the two workloads in the scenario_version history.

Suggested change

	def __init__(self, c: Composition, conn_infos: dict[str, PgConnInfo]):
	version = "1.1.0"
	def __init__(self, c: Composition, conn_infos: dict[str, PgConnInfo]):