fix(topology): restore the review group time budget and remove prompt/gate contradictions

[aviggiano]

Refs #1150
Refs #675

Problem

• Review stages were under-budgeted. In the four non-smoke topologies (.ultrafuzz/topology.yml and the packaged default, exhaustive and invariant-only), the review group (dedupe-findings → triage → severity-classification → aggregate-test-files → final-report) had no timeout pin. It ran on run.default_timeout_seconds (3,600), while every goal, strategy and specialist lane ran with 7,200. review/triage.md and review/severity-classification.md still tell the agent "The topology gives this review node an extended timeout", which was false. Expanding main's default.yml gives review:3600 ×5, strategies:7200 ×39, goals:7200 ×3, specialists:7200 ×5. Dedupe is the widest fan-in. When a review stage needs more than an hour, it times out every time and is retried with the same budget. The review group halts on failure, so shouldSkipWorkflowTask then skips every later review stage and the final-report node. That is the "one-hour outer timeout" repro in Checkpoint long fan-in stages and recover according to failure type #1150.
• Stateful prompts contradicted the gate. strategies/invariants/setup.md and implement-properties.md allowed production edits "except for interfaces if they are genuinely required". Both nodes publish workspace.patch, and captureWorkspacePatch rejects every change under the production source roots, added files included. Following the prompt therefore failed the handoff, and a retry starts from the same prompt. The two Vyper setup prompts (setup/prepare-foundry-harness.md, setup/discover-base-test.md) ask for Solidity interfaces without saying where to put them; their nodes publish workspace.patch too, and the setup group halts the run on failure.
• Prompt drift was invisible. Runs use the project copy of each prompt, and ultrafuzz init keeps existing copies unless run with --force. After an upgrade, stale copies ran against newer gates, and nothing warned about it until a node failed its artifact gate.
• Dead config. 36 timeout_seconds: 300 lines sat on reference nodes that never run as tasks.

Root cause

• The history bears out the drift. At v0.0.1, triage, severity-classification, aggregate-test-files and final-report each pinned timeout_seconds: 3600. v0.0.2 removed those pins, and the "extended timeout" sentence first appears in v0.0.3 (02d8927). timeoutSecondsFor (topology expand.ts) reads only a node pin or a group pin. buildSmithersTask then falls back to the model-profile timeout, then to run.default_timeout_seconds.
• The interface exception predates the source-snapshot guard in workspace-handoff.ts (assertProductionSourcePreserved), and the prompts were never updated to match it.

Change

1. Review group pin. Add defaults: { timeout_seconds: 7200 } to the review group in .ultrafuzz/topology.yml and packages/config/topologies/{default,exhaustive,invariant-only}.yml. default.yml stays byte-identical to .ultrafuzz/topology.yml. smoke.yml is untouched because it has no triage or severity node. 7,200 is the value the existing two-sided guard ("pins every agentic timeout to the reviewed window…") requires of every agentic group pin.
2. Reference-node timeouts. Remove the 36 timeout_seconds: 300 lines. Nothing uses them: plan-run materializes reference nodes before launch (materializeReferenceNodesForPlan), and compileSmithersWorkflow compiles only kind === "agentic" nodes. The value was only copied into the graph fingerprint and planned graph, and shown by the dashboard. topology-yaml.md now says the field has no effect on reference nodes. Existing topologies that still carry the lines keep validating.
3. Prompts. Delete the interface exception from both invariant prompts. They now say the handoff rejects every change under the production source roots, and tell the agent to declare harness interfaces in the test tree instead. The two Vyper setup prompts get the same rule in one sentence each.
4. Drift warning. A new projectPromptsDifferingFromBuiltIns(catalog) in @ultrafuzz/prompts lists project prompts whose bytes differ from the built-in prompt at the same path. validate emits one PROMPT_DIFFERS_FROM_BUILT_IN warning per such file and sets the prompts posture to warn. Warnings do not fail validate, plan, doctor or the dashboard's prompt save. The text summary gives the count and points to --json for the paths, because text-mode validate prints diagnostics only on failure. doctor's validate check used to keep the summary "config, topology, prompts, paths, agents, and trust posture pass" next to a warning status; it now says "configuration validation passed with warnings; run ultrafuzz validate --json for detail". Documented in docs/reference/cli.md and docs/how-to/edit-prompts-topology.md.
5. Upgrade notes. CHANGELOG gains a Breaking-changes entry for the goals/strategies group timeout_seconds: 7200 shadows the profile default and kills high-reasoning nodes at 2h with total loss of node work #675 precedence effect (see Risk) with its escape hatch. The Other-changes entry gives ultrafuzz topology copy default .ultrafuzz/topology.yml --force as the refresh for an uncustomized project topology, which init keeps; the how-to says the same.

Deliberately not built (and why)

• Most of Checkpoint long fan-in stages and recover according to failure type #1150's proposal: batching the fan-in, persisting intermediate results, a host-enforced finalization window, a repair pass, and checkpoints across cancellation. Dedupe, triage, severity and the report are global operations over every raw finding. Batching them needs a merge stage, per-batch contracts and cross-batch reconciliation. That is more strict gate surface, the kind that has repeatedly stopped runs. Backoff for transient errors, the finalization reserve and clock deadlines already exist. Deterministic retry cascades are Classify deterministic failures correctly and stop retry cascades #1144; retry backoff tuning is Concurrent agents race on OAuth token refresh; three immediate retries all re-race and kill the run with an opaque "Claude run failed" #1084. Because the acceptance criteria are not built, and projects scaffolded before this change keep the one-hour review budget until they refresh their topology, this PR uses Refs #1150 rather than Closes. Whether to close the rest of Checkpoint long fan-in stages and recover according to failure type #1150 is your call.
• A topology drift warning like the prompt one. Customizing .ultrafuzz/topology.yml is the normal workflow, so a byte comparison against the packaged default would warn on most real projects forever. The CHANGELOG and how-to give the refresh command instead.
• Rewriting existing .ultrafuzz/topology.yml files in init. That is merge machinery over user-owned YAML.
• The goals/strategies group timeout_seconds: 7200 shadows the profile default and kills high-reasoning nodes at 2h with total loss of node work #675 plan-time shadowing warning. It is not in this item's scope, and goals/strategies group timeout_seconds: 7200 shadows the profile default and kills high-reasoning nodes at 2h with total loss of node work #675 stays open. This PR adds one more group pin with the shadowing behaviour goals/strategies group timeout_seconds: 7200 shadows the profile default and kills high-reasoning nodes at 2h with total loss of node work #675 describes, now recorded under Breaking changes.
• A guard that dedupe-findings, aggregate-test-files and final-report are pinned. The anchor test covers only nodes whose built-in prompt promises an extended timeout (triage, severity-classification), and its comment now says so. The existing two-sided guard checks the value of every group pin, not its presence. Selecting nodes by group would need a smoke.yml carve-out for little gain.
• A test pinning the new prompt wording. It would only be a phrase ban over prompt text. The behavioural premise is covered instead, below.
• Rejecting timeout_seconds on reference nodes in topology validation. Every scaffolded project has these lines, so rejecting them would break valid projects for no stability gain.
• The CI markdownlint fix. See Verification → CI.

Verification

Discriminating tests, each run by me on the rebased branch (base b6dd1da) failing without the change and passing with it:

• packages/topology/test/packaged-topologies.test.ts → "gives every node whose prompt promises an extended timeout more than the default". It expands every packaged topology with the real expandTopology. For each node whose built-in prompt matches /extended\s+timeout/, the resolved timeout must exceed the largest shipped default. It also fails if no node promises one, so it cannot pass vacuously. With origin/main's three packaged YAMLs swapped in, it fails with default.yml node `triage` promises an extended timeout but resolves to ultrafuzz.toml `run.default_timeout_seconds`=3600: expected 3600 to be greater than 3600. With the branch's topologies, all 15 tests in the file pass.
• packages/runtime/test/prompt-catalog-validation.test.ts (new, 2 tests).
  • "validate warns about a project prompt that differs from the built-in prompt at its path": a fresh scaffold → posture pass and no warning. Edit review/triage.md and add strategies/project-only.md → ok: true, posture warn, exactly one warning, for review/triage.md. Delete the edited file and rerun initProject → pass again, which checks the remedy the warning names. With origin/main's validate.ts it fails with actual: 'pass', expected: 'warn'.
  • "doctor does not summarize a validation that only warned as a pass". With origin/main's doctor.ts it fails with actual: 'config, topology, prompts, paths, agents, and trust posture pass'. With origin/main's validate.ts it fails earlier, with actual: 'ok', expected: 'warning'.

Supporting evidence (not discriminating):

• Expansion with the built expander. Default, exhaustive and invariant-only: all five review nodes resolve to 7,200. Smoke: dedupe-findings and final-report stay unset. In default, the setup (6) and properties (9) groups stay unset (3,600 default), as before.
• Upgrade simulation with the built CLI. I ran init on a scratch repo, then put back origin/main's default.yml and the four edited prompts. validate exits 0 with prompts: warn and four PROMPT_DIFFERS_FROM_BUILT_IN diagnostics, and doctor prints validate: warning - configuration validation passed with warnings; run ultrafuzz validate --json for detail. ultrafuzz topology copy default .ultrafuzz/topology.yml --force then makes the file byte-identical to the packaged default, validate still passes, and all five review nodes expand to 7,200.
• captureWorkspacePatch against a scratch repo using the built runtime (implementer's probe; I did not re-run it): adding src/interfaces/IVault.sol fails with source-snapshot violation: workspace patch modifies protected production source: src/interfaces/IVault.sol, and adding test/recon/interfaces/IVault.sol is accepted.
• git merge-tree against every other claude/* branch: CHANGELOG.md is the only conflict. Before the helper moved, packages/prompts/src/catalog.ts also conflicted with refactor: delete dead prompt rename, config helpers, and CI scripts orphaned by #1131 #1175.

Ran, all passing:

• vitest: @ultrafuzz/topology 8 files / 74 tests, @ultrafuzz/prompts 6 / 104 (including semantic-anchors), @ultrafuzz/config 5 / 94.
• node:test runtime:
  • runtime.test.js subset, 18/18: all validate … tests, init preserves existing project-owned files…, the four a clean scaffold … tests (one plans the full shipped default topology), the shipped default topology expands…, plan materializes pinned reference nodes…, plan provisions a validated trusted expectation catalog… and planned graphs persist topology overrides….
  • lifecycle-inspection.test.js diagnoseProject*, 12/12.
  • prompt-catalog-validation 2/2 and model-override-validation 4/4.
• node:test CLI: init and validate emit schema-versioned launch JSON and doctor reports install posture in human and JSON output.
• Checks: npx prettier --check and npx eslint on the changed files, CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/main pnpm -w lint:strict:ci, pnpm -w knip, typecheck for prompts, runtime and topology, the three docs:check scripts (the audit-profile and prompt-catalog generators produce no diff), and node scripts/validate-audit-profile-package.mjs.

Not run: the full runtime and CLI suites, and the modal, evals, evmbench and dashboard suites.

CI. "External static analysis" fails on this PR for a repo-wide reason. Super-linter's markdownlint checks every changed file in full, and CHANGELOG.md already had many lines over MD013's 400-character limit when super-linter was added (#1005). Any PR that adds a changelog entry therefore fails, release-validation (runtime shards, runtime-supporting, CLI) is skipped, and release-gates fails. #1184 fixes this once with .github/linters/.markdown-lint.yml (MD013 off); its External static analysis passes with a CHANGELOG edit. This PR does not duplicate that change. Until #1184 merges and CI is re-run here, the runtime and CLI suites have not run in CI for this PR. The new runtime test runs in the runtime-supporting lane.

Risk / compatibility

• Shadowing (goals/strategies group timeout_seconds: 7200 shadows the profile default and kills high-reasoning nodes at 2h with total loss of node work #675), now under Breaking changes. Like the goal, strategy and specialist pins, the review pin wins over run.default_timeout_seconds and model-profile timeout_seconds. A project that raised either above 7,200 now gets 7,200 on review nodes when it runs a packaged topology (exhaustive, invariant-only) or a new scaffold. The CHANGELOG gives the escape hatch: copy the topology, raise the pin, and set topology_path. For everyone on the shipped 3,600 default, review goes from 1h to 2h.
• Cloud caps, also under Breaking changes. In cloud mode, an explicit global execution.resources.timeout_seconds below 7,200 already fails launch (CLOUD_TASK_TIMEOUT_BUDGET_EXCEEDED) on the strategy nodes. A config that works around this with per-node overrides on the pinned nodes now needs them on the review nodes as well.
• Cost. A review agent that truly hangs now burns up to 2h per attempt instead of 1h.
  • Per node, the worst case (window × topology attempts × same-agent attempts) stays below every profile's workflow_deadline_seconds. The existing "keeps one stuck agentic node from consuming a whole profile's workflow deadline" guard checks this and now covers review nodes too.
  • If all five sequential review stages time out on every attempt under the default 3 attempts, that is up to 30h (15h before), past the default 24h workflow_deadline_seconds. The deadline is checked only when a command synchronizes the run: status, inspect, why, stats, the dashboard's inspect action or the eval poll loop (docs: state that workflow_deadline_seconds is only enforced on sync #1157, workflow_deadline_seconds is only enforced while an operator runs a CLI command; unattended runs have no wall clock #1110). An unattended run keeps executing, and spending, past it.
• Modal benchmark lanes (by code reading only; no Modal run).
  • The public lanes write run.default_timeout_seconds = node_timeout_seconds (1,800) and do not cap topology pins. threat-model runs the default topology from a fresh init --force scaffold and full runs exhaustive, so each of their five review stages can now take up to 2h per attempt instead of 30 min, inside the 15,000-second row envelope.
  • The private-lane recovery overseer replaces a live owner as owner-stalled when neither a run-state transition nor a node success has happened within max(30 min, node_timeout_seconds + 15 min). With node_timeout_seconds below 7,200 (the documented example uses 7,200), a single review stage that runs past that window could have its owner replaced mid-attempt. The strategy, goal and specialist pins already have the same exposure.
  • evmbench caps every topology timeout_seconds to its node_timeout_seconds, so its review budget is unchanged.
• Existing projects. A scaffolded .ultrafuzz/topology.yml (default and low-cost profiles) is not rewritten. It keeps the 1h review budget until it is refreshed (ultrafuzz topology copy default .ultrafuzz/topology.yml --force if uncustomized) or defaults: { timeout_seconds: 7200 } is added to its review group. Projects also keep their old copies of the four edited prompts. The new validate warning flags those copies, along with any other stale or customized prompts. Expect warnings, not failures, after upgrading.
• Digests. The packaged default, exhaustive and invariant-only topology digests change. In-flight runs are unaffected, because resume uses the sealed plan and does not re-expand the topology. An eval run launched before this merges and published after it would fail the public-history "current packaged policy" check, as with any topology change.
• No schema, validator-build or contract-description changes.
• History. The branch was rebased onto b6dd1da and force-pushed to reword one commit message ("the other agentic groups" became "the goal, strategy and specialist groups"). The original five commits' content is unchanged, and four review-fix commits follow them.

🤖 Generated with Claude Code

Confidence Score: 3/5

The PR does not appear safe to merge while the previously identified review-timeout gaps remain.

Findings

1.  Existing projects keep shorter timeouts ▶
2.  Review pin exceeds cloud caps ▶

Fix with agent prompt

### Issue 1
.ultrafuzz/topology.yml:43-45
Default and low-cost runs use each project's editable `.ultrafuzz/topology.yml`, and `ultrafuzz init` preserves that file unless run with `--force`. Projects created before this change therefore do not get the new review pin. Their review stages still use the existing 3,600-second default, so the one-hour timeout this PR aims to fix persists for those runs.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 2
packages/config/topologies/default.yml:44-45
If a cloud project has an explicit resource timeout below 7,200 seconds and per-node caps for the already-pinned strategy tasks, its review tasks now exceed that resource timeout. Task compilation throws `CLOUD_TASK_TIMEOUT_BUDGET_EXCEEDED`, so a previously valid run cannot launch. The same review pin is added to the exhaustive and invariant-only topologies.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR extends the review group's timeout, aligns harness prompts with the production-source handoff, and warns when project prompts differ from built-ins.

• It also removes ineffective reference-node timeout pins and adds validation and topology tests.
• The latest change removes the release notes explaining the review timeout's breaking configuration effects.

_{Reviews (3) · Last reviewed commit: "chore: move the changelog entry to the c..."}

[greptile-apps]

Existing projects keep shorter timeouts

Default and low-cost runs use each project's editable .ultrafuzz/topology.yml, and ultrafuzz init preserves that file unless run with --force. Projects created before this change therefore do not get the new review pin. Their review stages still use the existing 3,600-second default, so the one-hour timeout this PR aims to fix persists for those runs.

Prompt To Fix With AI

This is a comment left during a code review.
Path: .ultrafuzz/topology.yml
Line: 43-45

Comment:
**Existing projects keep shorter timeouts**

Default and low-cost runs use each project's editable `.ultrafuzz/topology.yml`, and `ultrafuzz init` preserves that file unless run with `--force`. Projects created before this change therefore do not get the new review pin. Their review stages still use the existing 3,600-second default, so the one-hour timeout this PR aims to fix persists for those runs.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Review pin exceeds cloud caps

If a cloud project has an explicit resource timeout below 7,200 seconds and per-node caps for the already-pinned strategy tasks, its review tasks now exceed that resource timeout. Task compilation throws CLOUD_TASK_TIMEOUT_BUDGET_EXCEEDED, so a previously valid run cannot launch. The same review pin is added to the exhaustive and invariant-only topologies.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/config/topologies/default.yml
Line: 44-45

Comment:
**Review pin exceeds cloud caps**

If a cloud project has an explicit resource timeout below 7,200 seconds and per-node caps for the already-pinned strategy tasks, its review tasks now exceed that resource timeout. Task compilation throws `CLOUD_TASK_TIMEOUT_BUDGET_EXCEEDED`, so a previously valid run cannot launch. The same review pin is added to the exhaustive and invariant-only topologies.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.