Check that the seeded plan is one the planner would emit - #1784
Open
Nathan (nturinski) wants to merge 1 commit into
Open
Check that the seeded plan is one the planner would emit#1784Nathan (nturinski) wants to merge 1 commit into
Nathan (nturinski) wants to merge 1 commit into
Conversation
Four stimuli -- `scaffold-fullstack`, `scaffold-autopilot`,
`debug-plan-approval-gate` and `debug-generate-artifacts` -- begin from a single
seeded plan that stands in for `azure-project-plan`'s output. That document
drifted away from the template the planner emits: no per-service sections, a
bullet list where the `### Run` / `### Debug` prerequisite tables belong, and no
`## 9. Next Steps` at all. It failed `validate-project-plan` outright, and had
been failing for months.
Nothing noticed, because nothing had ever pointed a grader at the seed.
`npm run certify` covers the grader-certification fixtures; `npm run drift`
covers `resources/agents/**`. The document between them -- a checked-in fixture
standing in for agent output -- was covered by neither.
`npm run seed:contract` runs the production validator against the plan each
recipe would actually stage. Reproducing the original defect on the fixed seed
(deleting `## 9. Next Steps`) now gets caught, with the finding named:
✖ approved-fullstack does not satisfy the project-plan contract (checked-in fixture):
[missingSection] $: Missing required "next steps" section.
✖ unapproved-plan does not satisfy the project-plan contract (checked-in fixture):
[missingSection] $: Missing required "next steps" section.
## Why a validator rather than a hash
The obvious guard is to record the `agentAssetsHash` a fixture was captured under
and fail when the lock moves -- what `seeds/provenance.json` already does for a
*harvested* seed. That is the wrong instrument here. Most agent-asset edits do not
touch the plan template, so the check would fire constantly for non-defects, and a
check that cries wolf gets its hash bumped reflexively -- at which point it guards
nothing.
Running the validator asks the question that actually matters, and is the question
that was silently answered "no": is this still a document the planner could have
emitted? It fires exactly when the answer changes, and names the missing section so
the failure is actionable.
## Validating what runs, not a copy of it
`planSeedDocuments()` is exported from `stage-workspace.ts` and returns the bytes
the recipes stage. Re-deriving "which plan wins, and what its status line says" in
the checker would have been a second source of truth -- the specific failure that
module's header warns about -- and the checker would then certify its own copy
while the suite ran something else.
Both recipes are checked, not one. They differ only in the `**Status**:` line, and
that line is what `--expect-status` reads, so validating one says nothing about the
other's status rewrite. Confirmed by breaking it: renaming the `**Status**` row
fails loudly through `withStatus`'s existing guard, which already explains that a
silent no-op would make `scaffold-unapproved-plan` a duplicate of
`scaffold-fullstack` that reports green while testing nothing.
## Scope
Structural conformance only. This cannot detect a plan that is well-formed but
describes a stack no planner would choose -- for that, `harvest-seed.ts --check`
and a real harvest remain the answer. What it guarantees is the floor: the seed is
shaped like the planner's output, so a scaffold run that fails is failing about the
product.
Wired into the contracts workflow. All ten suites pass.
Co-authored-by: Copilot App <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four stimuli —
scaffold-fullstack,scaffold-autopilot,debug-plan-approval-gateanddebug-generate-artifacts— begin from a single seeded plan standing in forazure-project-plan''s output. That document drifted away from the template the planner emits: no per-service sections, a bullet list where the### Run/### Debugprerequisite tables belong, and no## 9. Next Stepsat all. It failedvalidate-project-planoutright, and had been failing for months.Nothing noticed, because nothing had ever pointed a grader at the seed.
certifycovers the grader-certification fixtures;driftcoversresources/agents/**. The document between them — a checked-in fixture standing in for agent output — was covered by neither.npm run seed:contractruns the production validator against the plan each recipe would actually stage. Reproducing the original defect on the fixed seed (deleting## 9. Next Steps) now gets caught, with the finding named:Why a validator rather than a hash
The obvious guard is to record the
agentAssetsHasha fixture was captured under and fail when the lock moves — whatseeds/provenance.jsonalready does for a harvested seed. That is the wrong instrument here.Most agent-asset edits do not touch the plan template, so the check would fire constantly for non-defects — and a check that cries wolf gets its hash bumped reflexively, at which point it guards nothing.
Running the validator asks the question that actually matters, and is the one that was silently answered "no": is this still a document the planner could have emitted? It fires exactly when the answer changes, and names the missing section so the failure is actionable.
Validating what runs, not a copy of it
planSeedDocuments()is exported fromstage-workspace.tsand returns the bytes the recipes stage. Re-deriving "which plan wins, and what its status line says" in the checker would have been a second source of truth — the specific failure that module''s header warns about — and the checker would then certify its own copy while the suite ran something else.Both recipes are checked, not one. They differ only in the
**Status**:line, and that line is what--expect-statusreads, so validating one says nothing about the other. Confirmed by breaking it: renaming the**Status**row fails loudly throughwithStatus''s existing guard, which already explains that a silent no-op would makescaffold-unapproved-plana duplicate ofscaffold-fullstackthat reports green while testing nothing.Scope
Structural conformance only. This cannot detect a plan that is well-formed but describes a stack no planner would choose — for that,
harvest-seed.ts --checkand a real harvest remain the answer. What it guarantees is the floor: the seed is shaped like the planner''s output, so a scaffold run that fails is failing about the product.Wired into the contracts workflow. All ten suites pass.