fix(runtime): workflow renders no longer crash on prompt text, and per-render cost drops

[aviggiano]

Problem

• Prompt text could fail the whole run. Any literal {{word}} that reached an agent prompt made the Smithers render throw. Three sources can produce one:

  • an escaped \{{word}} example in a prompt template, which the prompt renderer emits as literal {{word}};
  • the same escape inside a goal-plan value, which is kept verbatim;
  • an operator --prompt note.

  Every render builds the prompt of every task it renders, so one such prompt fails the render (WORKFLOW_RENDER_FAILED). The run stops, and every resume fails the same way. Goal-plan replacement values had a second route: Late tick lets {{amount}} round down passes validateGoalPlan, and then the dynamic prompt renderer throws dynamic item value liquidation:overdue references non-item variable: amount inside the same render. I reproduced both throws at function level on origin/main.

• The validator CLI self-test ran over and over. prepareArtifactMirror spawned ultrafuzz json validate in every prepare:, in every agent-attempt reset, when an agent was built after a controller restart, and again in the zero-retry verify: task. The trusted launcher's cold start was measured at ~35 s under contention (Smoke lane reaches agent work but no target can score: 15s validator preflight budget and a self-contradicting final-report prompt #1026), so each spawn was one more way to fail an attempt, including a verify whose agent work had already finished.

• Per-process and per-render cost.

  • Every ultrafuzz CLI process imports @ultrafuzz/runtime, and that includes every agent validator command. That import loaded the TypeScript compiler. (Smithers engine processes import the runtime index too, but they load the compiler through smthrs regardless; see the Enforce aggregate memory limits and avoid type-checking huge generated workflows #1146 section.)
  • Every render rewrote and fsynced tasks.json and graph.json, even when nothing had changed.
  • taskSpecsFromCompiled looked up each task with a linear find.
  • Nothing bounded the size of the generated workflow (Enforce aggregate memory limits and avoid type-checking huge generated workflows #1146).

Root cause

• renderAgentPrompt (workflow.tsx, lines 1673-1688 on main) substituted the values first, then regex-scanned the substituted result. So it validated the inserted text rather than the template.
• The ultrafuzz/goal-plan@1 zod contract accepted replacement values containing template syntax, which the dynamic renderer resolves and can fail on.
• preflightJsonValidator ran on every call, although its answer only depends on the process. It also ran in the post-agent verify pass, which never uses the CLI. materializePromptSchemas digest-checks each workspace's schema copy just before it, and start-run already runs the trusted-launcher preflight at launch and resume.
• agent-registry.ts and smithers.ts both had a module-scope import * as ts from "typescript", and both are reachable from the runtime index. Only registry analysis (validate/init) and assertRefreshedModuleAuthority use it. The latter is reached only through refreshedSmithersControllerSnapshot, which has no caller in packages/*/src.
• materializeDynamicRuntime's publish mode called writeJsonDurable unconditionally.

Change

1. renderAgentPrompt checks the template's own placeholders inside the replace callback, the same way renderAgentPreambleTemplate does. Inserted text is never re-scanned. A template placeholder with no value still throws, and the error now names it.
2. goal-plan.ts adds a zod refinement: replacement values must not contain {{. A bad label now fails goal-plan's own verify (assertGoalPlan, reached from materializeGoalPlanDatabaseArtifacts) instead of every render. The evals readGoalPlan applies the same zod contract. The agent-facing ultrafuzz json validate --schema goal-plan.schema.json and ultrafuzz artifact validate ultrafuzz/goal-plan@1 do not run it: both return ok: true for such a plan (I ran both), the same gap the existing JSON-record refinement already has. z.toJSONSchema drops refinements, so goal-plan.schema.json and the validator build identity are unchanged. retry-failed replays failed verifier without rerunning artifact producer #925 made resume --retry-failed reopen a failed verifier's producer; I did not re-test that path here. docs/reference/prompt-variables.md says where the rule runs.
3. Validator preflight. preflightJsonValidator remembers its first success for the engine process, and a failure is not remembered. The post-agent verify pass (finalizeAndVerifyArtifacts calls prepareArtifactMirror with pinnedSubmodules: "verify") no longer runs it at all. prepare:* and the reset before each agent attempt keep it, because they run ahead of an agent that uses the CLI.
4. TypeScript loading. The TypeScript compiler is now createRequired on first use in agent-registry.ts and smithers.ts. The value import became import type, so agent-registry.ts renames its type annotations to TypeScript.*. That touched the signature line of topLevelNameIsBound, so the strict diff lint reported the function's existing complexity (25). I moved its import-clause check into importClauseBindsName, with the same conditions, and added test cases for it (item 9). The saving lands in ultrafuzz CLI processes, including every agent validator call. It does not reach Smithers engine processes.
5. materializeDynamicRuntime skips the write when the file already holds the exact bytes writeJsonDurable would write.
6. taskSpecsFromCompiled indexes the serialized specs by id once per call.
7. New generated-workflow-footprint.test.ts. It compiles the packaged default topology and asserts the generated workflow stays under 2,300,000 bytes. It measures with a fixed-length project root, because the absolute root appears 4,841 times and would otherwise make the budget depend on the length of $TMPDIR. The normalized size is about 2.06-2.07 MB; the exact figure differs between environments (2,060,237 bytes on this host at the current head, 2,070,382 in the first round's environment). It also asserts that importing the runtime index does not load typescript.
8. CHANGELOG.md gets <!-- markdownlint-disable MD013 --> below its title. The first push failed External static analysis. Super-linter lints every changed Markdown file with MD013 at 400 characters, and markdownlint reports 42 existing entries on main as longer than that. So any change to the file fails the check, which fails release-gates and skips the release validation lanes. No commit has touched CHANGELOG.md since that gate landed in chore: adopt external static-analysis gates #1005. The directive line is now spelled exactly as fix(runtime): resume --retry-failed reopens descendants skipped behind a recovered producer #1169, fix(runtime): name the runner error when a workflow fails with no failed node, and pin same-id recovery #1172 and refactor(artifacts): delete never-dispatched gates, production-dead code, and the unread event index #1181 add it, so the file does not end up with two spellings of the policy. With it, the file passes super-linter's pinned markdownlint config ([email protected] locally: 43 MD013 errors without the directive, 0 with it). This is a lint-policy call, kept to one line in its own commit, so it is easy to drop if you would rather reflow the file.
9. Registry-shadowing test cases for imports. The existing test only declared Object locally. It now also covers default, namespace, named, aliased and default-plus-named imports of Object, plus an unrelated named import that must not count as shadowing.

#1146 investigation: does anything type-check the generated workflow?

No hot path I could find does. Nothing to remove or bound.

• packages/*/src: nothing runs tsc, ts.createProgram, ts.transpileModule, Bun.build or Bun.Transpiler over the generated workflow. The only TypeScript compiler API calls in production code are:

  • packages/runtime/src/agent-registry.ts:53 on main (ts.createSourceFile over the at-most-256 KiB .smithers/agents/index.ts, called from validate and init);
  • packages/runtime/src/smithers.ts:3760 on main (ts.preProcessFile over sealed node_modules .js files in assertRefreshedModuleAuthority, with no production caller).

  This PR makes both lazy.

• Pinned Smithers 0.35.0 only parses the workflow:

  • @smthrs/engine/src/workflow-hash.js:59-61 and @smthrs/time-travel/src/validateWorkflowIdentity.js:15-26 call Bun.Transpiler#scanImports;
  • @smthrs/cli/src/packs.js:186-188 does the same for packs;
  • Bun transpiles the .tsx when the engine imports it.

  @smthrs/cli/src/workflow-pack.js:165 holds typecheck: "tsc --noEmit" only as a script string for generated pack package.json files; ultrafuzz never runs it.

• Every engine process still loads the TypeScript compiler, and this PR cannot change that. The generated workflow's import { createSmithers } from "smthrs" (workflow.tsx:23, and the agent templates import smthrs the same way) loads [email protected]/src/index.js. Its export { … } from "@smthrs/scorers" (lines 506-520) loads the scorers index, which re-exports workflowUiCompliance.js and sideEffectAnalysis.js (@smthrs/scorers/src/index.js:54-55). Both start with import ts from "typescript" (workflowUiCompliance.js:2, sideEffectAnalysis.js:1). sideEffects: false in the package manifest only affects bundlers. I checked it with a require.cache probe: after import("smthrs") alone, [email protected]/lib/typescript.js is loaded under both Bun 1.3.14 and Node 24.

• Measured parse cost over the ~2.07 MB default workflow with Bun 1.3.14 (first round): scanImports 7-9 ms, transformSync 14-15 ms, process RSS under 90 MB.

• In this repo, only tests call ts.transpileModule on the template or on generated output. That includes dynamic-workflow.test.ts:153, which transpiles a whole generated workflow.

• Conclusion. One engine or CLI process does not come near "tens of GB". My inference, which I have not verified, is that such a figure needs an external type-checker (an agent running tsc, or an editor LSP) over .smithers/workflows/*.tsx. The budget test is the guard added here.

Deliberately not built (and why)

• Dropping the duplicated task-spec literal. It is 886 KB of the file, next to the 702 KB compiled-tasks literal, and it is the real size lever for Enforce aggregate memory limits and avoid type-checking huge generated workflows #1146. But it changes the generated-program layout and the helpers the template destructures. The budget test only stops growth.

• Catching per-task render errors in renderReadyRuntimePrompts, so that one bad dynamic prompt fails only its own task. That needs a new way to represent a task that failed at render. The goal-plan refinement moves the one model-authored trigger I found in the default topology to its producer instead.

• Fixing the dynamic renderer instead of adding the goal-plan rule (review suggestion: keep a non-item reference inside a value as literal text, in resolveDynamicVariable). I ran the built renderPrompt with a {{item.goal_prompt}} → {{liquidation:overdue}} chain and four labels:

  • Late tick lets {{amount}} round down throws references non-item variable: amount, which that change would fix;
  • Late tick {{ never closed throws unclosed-template-variable;
  • Tick {{}} rounding throws empty-template-variable;
  • Tick {{item.missing}} rounding throws missing dynamic item template variable.

  The one-branch change fixes one of these four, and the refinement rejects all four. A renderer change that covers them all means not scanning replacement values at all. That changes the documented nested-replacement semantics ("unresolved, cyclic, non-scalar, or non-item references fail") for custom topologies, in render.ts, which refactor: delete dead prompt rename, config helpers, and CI scripts orphaned by #1131 #1175 is also editing. The refinement's costs are real, and I am leaving them in place:

  • it also rejects labels the renderer handles, such as Overdue {{item.node_id}};
  • the goal-plan agent cannot catch it with its own validator commands;
  • goal-plan is in the setup group, which has no failure_policy: continue, so a rejected plan stops the run until resume --retry-failed;
  • a run whose already-verified plan holds such a label is not rescued.

• Rejecting escaped braces in goal_prompt. After change 1 they render as literal text and no longer throw.

• Editing the goal-plan prompt. It already asks for "one line of plain prose", and the verify error names the rule.

• Deleting the workflow preflight outright, or running it only in create mode (review suggestions). After this change it spawns the CLI only until its first success in each engine process, and only from prepare:* (which has at least one retry) and the reset before an agent attempt. On resume, start-run swallows a failed trusted-launcher preflight (submitSmithersContinuation, start-run.ts:584-599). On a resumed run, the engine check is then the only thing that confirms, before an agent spends an attempt, that the ultrafuzz it will call works. A create-only gate would skip the pre-attempt reset. In a restarted engine process that reset runs before the lazy agent rebuild, and its assert-task-inputs step records the admission that makes the rebuild unnecessary, so the process's first agent attempt could go unchecked.

• Sharing fixture setup between the new dynamic-expansion test and the existing retry test. fix(runtime): published dynamic expansions are the lock-free authority for fan-out membership #1163 changes about 250 lines of that file, and a shared helper would conflict with it.

• Deleting the dead controller-refresh code, the other TypeScript user. It is out of scope, and several parallel items edit smithers.ts.

• cgroup or process-tree memory budgets (Enforce aggregate memory limits and avoid type-checking huge generated workflows #1146's main ask).

Verification

Discriminating tests. Each new behavioural test fails on origin/main and passes on this branch. Both reviewers re-ran the first five against b6dd1da9; I ran the sixth there:

Test	origin/main	this branch
generated agent prompt inserts literal braces from task and operator prompts verbatim (runtime generated-workflow-verifier.test.ts)	throws agent prompt template contains an unresolved variable	pass
generated validator preflight spawns the CLI once per engine process and never remembers a failure (same file)	3 spawns, expected 1	pass
goal-plan replacement values cannot carry template placeholders into the dynamic render (artifacts threat-goal-artifacts.test.ts)	accepts Late tick lets {{amount}} round down	pass
re-rendering an unchanged dynamic runtime does not replace its published task plan or graph (runtime dynamic-expansion.test.ts)	inode of both files replaced	pass
importing the runtime package does not load the TypeScript compiler (new generated-workflow-footprint.test.ts)	typescript.js in require.cache right after import	pass
the post-agent verify pass does not preflight the agent-facing validator CLI (runtime workspace-preparation-lifecycle.test.ts)	3 preflights, expected 2 (on b6dd1da9, and on this branch with the template change stashed)	pass

The last test runs the production prepareArtifactMirror body, with its external CLI and submodule work stubbed, using the three option shapes the template passes. The verify-shaped call is the one finalizeAndVerifyArtifacts makes, and an existing verifier test pins that call's options.

The byte-budget test is a regression guard, so it passes on both.

The registry import cases (item 9) cover a refactor, so they pass on origin/main too. To check that they actually exercise the helper, I made four one-line mutations to the compiled importClauseBindsName. Removing the default-import check fails the test on import Object from …. Removing the namespace check fails it on import * as Object …. Removing the named-import check fails it on import { Object } …. Matching every named import fails it on the unrelated-import control.

The two template tests run renderAgentPrompt and preflightJsonValidator extracted from workflow.tsx with stubbed collaborators, which is how the existing verifier tests work. No test here runs the generated workflow under the real engine.

TypeScript load, checked with a require.cache probe on this host:

• node packages/cli/dist/index.js json validate --schema findings.schema.json --file validator-smoke.valid.json --json loads typescript on origin/main and does not on this branch.
• Importing smthrs alone loads it under both Bun and Node, on either branch.

First-round measurements on this 32-core host, shared with other jobs:

What	origin/main	this branch
Import built runtime index alone, Node 24 (5 runs)	528-547 ms, 248-250 MB RSS	409-437 ms, 196-198 MB
Same, Bun 1.3.14 (5 runs)	449-483 ms, 252-261 MB	336-351 ms, 209-213 MB
The CLI json validate command above, runs 2-5 of 5	2.38-2.46 s, 507-513 MB max RSS	2.20-2.28 s, 481-484 MB

The runtime-index rows do not describe an engine process, which loads smthrs as well.

What I ran on the current head:

• pnpm -w build, pnpm --filter @ultrafuzz/runtime typecheck and pnpm --filter @ultrafuzz/artifacts typecheck.
• runtime tests:
  • generated-workflow-verifier (142/142);
  • workspace-preparation-lifecycle (7/7);
  • generated-workflow-footprint (2/2);
  • cloud-worker-handoff, dynamic-expansion, dynamic-lifecycle, dynamic-workflow, task-workflow-identity and workflow-dependency-policy together (58/58, --test-concurrency=2);
  • the runtime.test.ts registry-shadowing test.
• npx prettier --check and npx eslint on every changed file, then CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/main pnpm -w lint:strict:ci, pnpm -w knip, node scripts/docs-check.mjs, node scripts/audit-profile-docs.mjs --check and node scripts/prompt-catalog-docs.mjs --check.
• [email protected] with super-linter's config on CHANGELOG.md and docs/reference/prompt-variables.md.

First round, not repeated on this head. This round's only code change is the verify-pass gate in the template. Everything else is comments, docs and tests, and the artifacts package did not change:

• 31 targeted runtime.test.ts tests: the registry-analysis validate/init tests, the controller-refresh tests that reach the lazy ts.preProcessFile, compileSmithersWorkflow*, and startRun compiles normal Smithers tasks…;
• artifacts threat-goal-artifacts (25/25) and schema.test.js (51/51, including the goal-plan JSON Schema snapshot).

PR smoke lane. It passed in CI on the first-round heads and on this head. When I ran it locally in the first round, 2 of its 12 named runtime.test.ts tests failed with WORKFLOW_SUBMISSION_FAILED ... changed while it was read on files under this worktree's shared node_modules. Both passed when re-run on their own with --test-concurrency=1. A reviewer saw the same class of flake in dynamic-lifecycle.

CI on 474f4f85: every check passes. That includes External static analysis, the PR smoke lane, all four runtime integration shards, the runtime support tests with the Bun 1.3.14 adapter contracts, and release-gates. Integration shard 1 took 47 min, against about 30 min on 59f7fe25; I did not look into why.

Not run anywhere: the CLI suite and any real campaign.

Risk / compatibility

• In-flight runs keep the old template. A run's generated workflow embeds the template. renderAgentPrompt, the preflight memo, the verify-pass skip and the task-spec index therefore take effect for new runs or re-rendered controllers. A native resume keeps the old template code.
• Runtime-module changes do reach resumed runs, because native continuation imports the installed packages. Those are the lazy TypeScript load, skipping unchanged dynamic writes, and the goal-plan refinement. In the runtime the refinement runs only in goal-plan's own verify. Evals readGoalPlan applies it too, and turns a validation failure into goal-plan-unreadable evidence rather than throwing. So a historical plan with {{ in a value would read as unreadable there.
• Verify no longer proves the validator CLI works. It never used it: its artifact checks run in-process.
• No export was removed or renamed, and no schema file, validator-build module or contract description changed.
• The CHANGELOG.md directive changes lint policy for that one file only (item 8).
• The skip-write path compares against the exact bytes writeJsonDurable produces (JSON.stringify(value, null, 2) + "\n"), so the on-disk format is unchanged.
• History. This round reworded the messages of two earlier commits, and changed the directive spelling in the MD013 commit, so the branch was force-pushed. Commit contents are otherwise unchanged. The perf commit's message had implied an engine-process saving.

Refs #1146

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issue was established in the changes since the previous review.

Summary

The PR changes prompt rendering so literal braces in inserted text do not fail a workflow render, validates goal-plan replacement labels earlier, and reduces validator startup and repeated-render costs.

• Adds regression tests for prompt rendering, validator preflight, dynamic-runtime publication, and generated-workflow size.
• Documents where goal-plan replacement validation runs.

_{Reviews (4) · Last reviewed commit: "chore: move the changelog entry to the c..."}