test(prompts): fail when the goal hunter prompt loses its node_id lookup key

[aviggiano]

Problem

• The goal hunter's {{item.node_id}} lost its test in test: delete vacuous and prompt-prose tests outside the runtime package #1196 (Greptile, P2). test: delete vacuous and prompt-prose tests outside the runtime package #1196 (071080b) deleted the prompt semantic-anchor tests and moved some of their variable pins into REQUIRED_PROMPT_VARIABLES. The deleted test "makes goal searches read their records from files and stop at a measurable deadline" pinned {{item.node_id}} in strategies/goal-hunter.mdx, with the comment that it "is what makes it resolvable at all". test: delete vacuous and prompt-prose tests outside the runtime package #1196's body listed that test as "wording" only, and the pin was not moved to the table. Each generated threat-goals / class-goals node (.ultrafuzz/topology.yml, default.yml, exhaustive.yml) uses it to find "the single element of class_goals or threat_goals whose node_id is {{item.node_id}}" in a goal plan that lists every goal. On origin/main 2cacf4c that line can be removed and all 58 prompts tests still pass. The hunter then has to guess its entry from the goal label. The goal-search census still records that node's result as coverage of the goal it was generated for.

Change

• packages/prompts/test/prompt-structure.test.ts: add "strategies/goal-hunter.mdx": ["item.node_id"] to REQUIRED_PROMPT_VARIABLES. Add one bullet to the table comment saying why it belongs: the goal-search census records the node's result as coverage of the goal it was generated for, and item.node_id is what picks that goal out of the plan. extractPromptVariables already reports this reference as item.node_id, so requirementKeys is unchanged. Test-only (+4/-1).

Deliberately not fixed

The same Greptile comment also asked to pin these. I left them out:

• Final report's aggregation and setup authority selectors. These are already pinned more strictly. render.test.ts:543 ("renders smoke final review with bounded classification semantics and context") renders the shipped review/final-report.md and asserts the sha256 selector ids of the exact 10-path machine selector and the exact 4-path setup selector. I checked by editing the prompt file. Dropping aggregation.json from the machine selector fails that test (expected '…' to contain 'path entry whose `id` is `3311f635787…'). Deleting the setup selector also fails it (expected 1 to be 2). A table entry would add nothing. test: delete vacuous and prompt-prose tests outside the runtime package #1196 also dropped these two pins on purpose ("its two path-selector pins are not kept").
• Dedupe's setup authority selector (setup/project-discovery.md,setup/base-test-setup.md). It is an ancestor_*_authority selector, so it fits the table's comment, but test: delete vacuous and prompt-prose tests outside the runtime package #1196 dropped this pin on purpose ("the optional setup path-selector pin is not kept"), and running without it is a supported degraded mode, so I left that decision alone. The prompt calls it "Optional native-validation context" and says a bounded topology may omit those handoffs. When they are absent, dedupe does one model-only consolidation pass instead (dedupe-findings.md:56-66).

Greptile did not flag the deleted test's other lookup pin, {{artifact_path:goal-plan}}/goal-plan.json, and it stays unpinned. With all three {{artifact_path:goal-plan}} references in goal-hunter.mdx rewritten as prose, the prompts suite still passes 58/58, and the topology packaged-topologies, artifact-handoffs and validate tests pass 43/43. A table entry would catch only the loss of all three, which takes a deliberate restructure, while {{item.node_id}} appears once and a one-line rewording drops it. If wanted, "strategies/goal-hunter.mdx": ["artifact_path:goal-plan", "item.node_id"] passes with the shipped prompt and fails (strategies/goal-hunter.mdx: {{artifact_path:goal-plan}}: expected 0 to be greater than 0) when those references are gone.

No overlap with in-flight work: none of x01-x05, #1202, #1197, #1201, #1198, #1183 or #1162 touches prompt-structure.test.ts or goal-hunter.mdx. #1162 touches only packages/prompts/test/agent-preamble.test.ts in that directory.

Verification

• prompt-structure.test.ts › "keeps the template variables that gates, budgets and sealed authorities depend on". This PR closes a test gap, so the discriminating run changes the prompt, not the code. Step 1 of goal-hunter.mdx was rewritten to "…whose node_id matches this node.", which removes {{item.node_id}}:
  • origin/main 2cacf4c: the regression is not caught. prompt-structure.test.ts passes 4/4 and the whole prompts suite passes 58/58.
  • this branch: fails with AssertionError: strategies/goal-hunter.mdx: {{item.node_id}}: expected 0 to be greater than 0.
  • this branch with the shipped prompt: passes. prompt-structure.test.ts passes 4/4 and the whole prompts suite passes 58/58.
• The 58/58 counts need packages/prompts itself built. Its test script builds only its dependencies, and without packages/prompts/dist two packaging tests fail with ENOENT (render.test.ts "renders output-contract guidance from the packaged layout with no repository above it" and scaffold-catalog.test.ts "packs the canonical prompt tree and scaffolds every prompt from the extracted package"). This PR does not touch either test.
• npx prettier --check and npx eslint on the changed file, CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/main pnpm -w lint:strict:ci, pnpm -w lint, pnpm --filter @ultrafuzz/prompts typecheck and pnpm -w knip all pass. The package typecheck does not cover tests, so I also ran tsc --noEmit over prompts src and test with the base config. It passes.

Risk / compatibility

• Test-only. No prompt, runtime, schema, validator or contract-description bytes change, so VALIDATOR_BUILD_IDENTITY is unaffected.
• A later edit that deliberately stops using {{item.node_id}} in the goal hunter now has to remove the table entry too. That is intended.

Changelog entry

Test-only; no product behaviour change. The prompt structure test now requires strategies/goal-hunter.mdx to keep {{item.node_id}}, which each generated threat or class goal node uses to find its own entry in the goal plan. Since #1196, no test caught its removal.

Also in this PR

• A schema reference is checked as a whole path (Greptile, P2 on #1196, and Greptile on this PR). The shipped-schema check in prompt-structure.test.ts captured the name after {{schema_path}}/ and stopped at a slash. So {{schema_path}}/findings.schema.json/obsolete passed as findings.schema.json. A first attempt required the name to end the path, but then {{schema_path}}/subdir/findings.schema.json matched nothing and escaped the check. The regex now reads dots and slashes between path segments: any longer path is captured whole and fails the shipped-file check, while a sentence-final period still ends the path. All current references pass (4/4 in prompt-structure.test.ts).

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge.

Summary

The PR pins the goal hunter’s item.node_id variable in the prompt-structure test and updates schema-reference matching to validate the whole path, including directory segments.

_{Reviews (3) · Last reviewed commit: "test(prompts): a schema reference with a..."}