Skip to content

Check that the seeded plan is one the planner would emit - #1784

Open
Nathan (nturinski) wants to merge 1 commit into
feat/CoRfrom
nturinski-seed-contract-check
Open

Check that the seeded plan is one the planner would emit#1784
Nathan (nturinski) wants to merge 1 commit into
feat/CoRfrom
nturinski-seed-contract-check

Conversation

@nturinski

Copy link
Copy Markdown
Member

Four stimuli — scaffold-fullstack, scaffold-autopilot, debug-plan-approval-gate and debug-generate-artifacts — begin from a single seeded plan standing in for azure-project-plan''s output. That document drifted away from the template the planner emits: no per-service sections, a bullet list where the ### Run / ### Debug prerequisite tables belong, and no ## 9. Next Steps at all. It failed validate-project-plan outright, and had been failing for months.

Nothing noticed, because nothing had ever pointed a grader at the seed. certify covers the grader-certification fixtures; drift covers resources/agents/**. The document between them — a checked-in fixture standing in for agent output — was covered by neither.

npm run seed:contract runs the production validator against the plan each recipe would actually stage. Reproducing the original defect on the fixed seed (deleting ## 9. Next Steps) now gets caught, with the finding named:

✖ approved-fullstack does not satisfy the project-plan contract (checked-in fixture):
    [missingSection] $: Missing required "next steps" section.
✖ unapproved-plan does not satisfy the project-plan contract (checked-in fixture):
    [missingSection] $: Missing required "next steps" section.

Why a validator rather than a hash

The obvious guard is to record the agentAssetsHash a fixture was captured under and fail when the lock moves — what seeds/provenance.json already does for a harvested seed. That is the wrong instrument here.

Most agent-asset edits do not touch the plan template, so the check would fire constantly for non-defects — and a check that cries wolf gets its hash bumped reflexively, at which point it guards nothing.

Running the validator asks the question that actually matters, and is the one that was silently answered "no": is this still a document the planner could have emitted? It fires exactly when the answer changes, and names the missing section so the failure is actionable.

Validating what runs, not a copy of it

planSeedDocuments() is exported from stage-workspace.ts and returns the bytes the recipes stage. Re-deriving "which plan wins, and what its status line says" in the checker would have been a second source of truth — the specific failure that module''s header warns about — and the checker would then certify its own copy while the suite ran something else.

Both recipes are checked, not one. They differ only in the **Status**: line, and that line is what --expect-status reads, so validating one says nothing about the other. Confirmed by breaking it: renaming the **Status** row fails loudly through withStatus''s existing guard, which already explains that a silent no-op would make scaffold-unapproved-plan a duplicate of scaffold-fullstack that reports green while testing nothing.

Scope

Structural conformance only. This cannot detect a plan that is well-formed but describes a stack no planner would choose — for that, harvest-seed.ts --check and a real harvest remain the answer. What it guarantees is the floor: the seed is shaped like the planner''s output, so a scaffold run that fails is failing about the product.

Wired into the contracts workflow. All ten suites pass.

Four stimuli -- `scaffold-fullstack`, `scaffold-autopilot`,
`debug-plan-approval-gate` and `debug-generate-artifacts` -- begin from a single
seeded plan that stands in for `azure-project-plan`'s output. That document
drifted away from the template the planner emits: no per-service sections, a
bullet list where the `### Run` / `### Debug` prerequisite tables belong, and no
`## 9. Next Steps` at all. It failed `validate-project-plan` outright, and had
been failing for months.

Nothing noticed, because nothing had ever pointed a grader at the seed.
`npm run certify` covers the grader-certification fixtures; `npm run drift`
covers `resources/agents/**`. The document between them -- a checked-in fixture
standing in for agent output -- was covered by neither.

`npm run seed:contract` runs the production validator against the plan each
recipe would actually stage. Reproducing the original defect on the fixed seed
(deleting `## 9. Next Steps`) now gets caught, with the finding named:

    ✖ approved-fullstack does not satisfy the project-plan contract (checked-in fixture):
        [missingSection] $: Missing required "next steps" section.
    ✖ unapproved-plan does not satisfy the project-plan contract (checked-in fixture):
        [missingSection] $: Missing required "next steps" section.

## Why a validator rather than a hash

The obvious guard is to record the `agentAssetsHash` a fixture was captured under
and fail when the lock moves -- what `seeds/provenance.json` already does for a
*harvested* seed. That is the wrong instrument here. Most agent-asset edits do not
touch the plan template, so the check would fire constantly for non-defects, and a
check that cries wolf gets its hash bumped reflexively -- at which point it guards
nothing.

Running the validator asks the question that actually matters, and is the question
that was silently answered "no": is this still a document the planner could have
emitted? It fires exactly when the answer changes, and names the missing section so
the failure is actionable.

## Validating what runs, not a copy of it

`planSeedDocuments()` is exported from `stage-workspace.ts` and returns the bytes
the recipes stage. Re-deriving "which plan wins, and what its status line says" in
the checker would have been a second source of truth -- the specific failure that
module's header warns about -- and the checker would then certify its own copy
while the suite ran something else.

Both recipes are checked, not one. They differ only in the `**Status**:` line, and
that line is what `--expect-status` reads, so validating one says nothing about the
other's status rewrite. Confirmed by breaking it: renaming the `**Status**` row
fails loudly through `withStatus`'s existing guard, which already explains that a
silent no-op would make `scaffold-unapproved-plan` a duplicate of
`scaffold-fullstack` that reports green while testing nothing.

## Scope

Structural conformance only. This cannot detect a plan that is well-formed but
describes a stack no planner would choose -- for that, `harvest-seed.ts --check`
and a real harvest remain the answer. What it guarantees is the floor: the seed is
shaped like the planner's output, so a scaffold run that fails is failing about the
product.

Wired into the contracts workflow. All ten suites pass.

Co-authored-by: Copilot App <[email protected]>
@nturinski
Nathan (nturinski) requested a review from a team as a code owner September 3, 2026 20:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant