Reconsider prepare/node/verify setup

[aviggiano]

Context

A large default-profile campaign was analyzed to understand whether the prepare: and verify: stages consume too much time relative to the producer node: stage.

Durations were measured from each NodeStarted event to its matching NodeFinished or NodeFailed event. The dataset contained 678 completed executions; 34 in-flight starts were excluded. Standard deviations below are population standard deviations. Aggregate hours are summed task/agent time, so concurrently executing tasks overlap in wall-clock time.

The audited project and all campaign findings are intentionally omitted.

Successful executions

Stage	Executions	Average	Median	Standard deviation	P90
Prepare	150	1m 48s	53.5s	2m 28s	6m 45s
Node	120	53m 45s	55m 35s	24m 46s	85m 43s
Verify	30	8.1s	4.6s	10.9s	13.4s

All completed attempts

This includes failures and retries.

Stage	Attempts	Average	Median	Standard deviation	Total task/agent time
Prepare	266	1m 38s	53.4s	2m 7s	7.22h
Node	341	20m 34s	1m 1s	29m 23s	116.85h
Verify	71	7.4s	5.8s	7.7s	0.15h

The all-attempt node median is depressed by 221 failed attempts that often terminated quickly, so the successful-execution table better represents normal producer work.

Time allocation

Across all completed attempts:

• Node: 94.07%
• Prepare: 5.82%
• Verify: 0.12%
• Combined prepare/verify overhead: 5.93%

Across the 30 fully successful prepare -> node -> verify triplets:

• Node: 96.75%
• Prepare: 2.93%
• Verify: 0.32%
• Combined overhead: 3.25%
• Median per-triplet overhead: 2.02%
• Average per-triplet overhead: 3.55%
• P90 per-triplet overhead: 9.54%
• Maximum observed per-triplet overhead: 13.78%

Interpretation

Verification is effectively negligible. Preparation is modest in aggregate, but it has a meaningful long tail: its median is under one minute while P90 is approximately 6m 45s and the observed maximum is approximately 8m 36s.

The current structure is not an obvious global bottleneck, but the preparation outliers and repeated work across retries make the setup worth reconsidering.

Questions / possible directions

• Can preparation results be safely reused or cached across retries and related fanout nodes?
• Can preparation be made incremental so unchanged workspace state avoids repeated setup?
• Are any preparation steps better performed once at the fanout or campaign level?
• Can stage-level timing and cache-hit metrics be exposed directly by Ultrafuzz?
• Should long-running preparation emit more granular progress and attribution so filesystem, dependency, snapshot, and workspace costs can be distinguished?
• Can regression tests or benchmarks establish an acceptable preparation-overhead budget?

[aviggiano]

Closing as not planned. The measurements here show prepare plus verify at about 3% of successful triplet time (median 2%), and the per-attempt prepare: step has changed since they were taken. #1168 (f9bd742) runs the JSON validator preflight once per engine process instead of on every prepare and retry reset; that CLI cold start was measured at up to about 35 s under contention. #1166 (aa58130) stops Smithers running git fetch origin on worktree creation and fetch/rebase on every re-entry. If a prepare tail still shows up on v0.1.3, please open a concrete issue with fresh per-step timings that names the slow step.