Add fidelity gates: did the agent build what it planned? by alexweininger · Pull Request #1721 · microsoft/vscode-azureresourcegroups

Alex Weininger (alexweininger) · 2026-08-26T05:38:45Z

The short version

Our gates can tell you the project builds. They can't tell you it built what it promised.

Every scaffold gate we have today grades the tree against itself: it compiles, the frontend is embeddable, the API seam holds. None of them read the plan. So a scaffold that quietly drops a service, or plans PostgreSQL and wires SQLite, passes every single gate we currently run — because two working services look exactly like two working services, and an app on the wrong database installs, builds, starts and serves traffic perfectly well. It only breaks later, somewhere the plan is no longer in the room.

That's the actual product question, and until this PR it was unanswered.

Something that passes today and shouldn't

Take the plan in the new reference-node-multiservice fixture: three services (API, web, worker), PostgreSQL, Blob Storage, Queue Storage. Delete services/worker entirely and change one import in services/api/src/db.ts from pg to sqlite3.

Today: green. It builds, the frontend is embeddable, the seam holds, the plan itself is still perfectly well-formed. Every existing gate passes.

With this PR: three gates go red — the missing worker, PostgreSQL declared but never imported, and SQLite wired without ever being planned.

The gates

validate-service-fidelity — plan model vs. tree

Code	Fires when	Mutation that proves it fails
`plannedServiceMissing`	plan declares 3 services, tree has 2	delete `services/worker`
`unplannedServiceScaffolded`	a service directory the plan never declares	strip the worker section from the plan
`frontendMissingFromScaffold`	App Type promises a UI, none was built	delete `services/web`
`frontendNotPlanned`	a UI exists on an API-only project	`App Type: SPA + API` → `API only`
`serviceLanguageMismatch`	plan says Python, directory is TypeScript	flip the backend's `Language` row
`serviceFrameworkMismatch`	plan says React + Vite, deps say Angular	flip the `Framework` row
`planDeclaresNoServices`	plan has no service sections at all	excise sections 2–4
`noServicesScaffolded`	plan promises services, tree has no project	delete the manifest

validate-datastore-fidelity — Services Required vs. tree

Code	Fires when	Mutation that proves it fails
`plannedDatastoreNotWired`	planned family never imported at runtime	`from 'pg'` → `from 'sqlite3'`; and the same swap on Python and .NET
`unplannedDatastoreWired`	a family the plan never named, wired	add a `mongodb` import; `psycopg` → `aiosqlite`
`datastoreDependencyMissing`	driver imported, no manifest declares it	drop `"pg"` from `package.json`
`plannedResourceNotWired`	a promised resource's env var appears nowhere	rename the planned env var
`plannedResourcesUnreadable`	no readable Services Required table	replace its header row

Certification goes 44/44 → 70/70. Every gate above has a mutation that turns it red, and the not-applicable path is pinned too (below).

Two corrections to the original framing, both load-bearing

## N. Services Required is the Azure resources table, not the app's services. Its columns are Azure Service | Role in App | Environment Variable | Default Value (Local) | Classification. The app's services are the per-service stack sections ## N. <Service> — <role>, identified by their Language row — which is the plan template's own discriminator, the same one the plan webview uses. Reading the wrong table would have produced services named "PostgreSQL".

This turned out to be good news: the plan hands us an env var name and a local connection string as structured cells, so the datastore check is a table lookup rather than a grep.

sample-agent-output could not be the fidelity fixture — see the known limitation at the bottom.

How the datastore check works, and why not a grep

Both sides normalise to a closed set of families and compare families, never strings. The plan's wording and the package registry's wording never match ("Azure Database for PostgreSQL Flexible Server" vs pg), so a string comparison is a lookup table in disguise that silently returns "no match" for every spelling nobody anticipated.

Planned side resolves from the local connection string's scheme first, name second. This is what makes "Azure Cosmos DB for MongoDB" resolve to mongodb: the wire protocol decides which client the code must speak. A name-first parser resolves it to cosmos, finds no @azure/cosmos, and reports a confident failure against a correctly built project.
Actual side is a per-ecosystem driver registry for Node, Python and .NET, and evidence is manifests ∪ imports — because import sqlite3 (Python stdlib) and node:sqlite never appear in any manifest, and a quiet swap to SQLite is the single likeliest form of this bug.
Wired means imported at runtime, not declared. A dependency installed and never imported is a leftover. This is what makes "swap the import" a red gate rather than a no-op.
Asymmetric scoping: a planned family needs runtime evidence; an unplanned one is only reported on runtime evidence too, so a test suite using in-memory SQLite on a PostgreSQL project doesn't turn the gate red. The direction that can be noisy gets the stricter bar.
ORMs are handled explicitly. A Prisma or EF Core project legitimately never imports pg, so a connection string naming the family counts instead — but only when an ORM is actually present, so this never degrades into "the tree mentions postgres somewhere".

Not-applicable: exit 3, and the escape hatch is itself certified

On a stack with no analyser these report not-applicable, never a pass — a datastore check that silently approves every Go project is indistinguishable from no check at all.

Per the shape agreed with the runtime-gates and gate-health sessions, that verdict exits 3 and prints:

NOT_APPLICABLE gate=datastore-fidelity class=coverageGap reason=ecosystemNotSupported detail="This gate has no dependency analyser for go yet… The fix is unwritten code in evals/src/artifacts/datastoreFidelity.ts, not a missing tool on this machine."
SKIP: gate=datastore-fidelity — … did not apply here.
  This gate applies here but could not run, so we are not testing something we claim to test. This is a gap to close, not a gate to unwire.
  Nothing here is evidence about the generated app.

A note for whoever runs the follow-up queries below, because this PR walked into it twice: being careful about the inference doesn't help if you don't also check the instrument. A null result from a measurement that fails toward null isn't a null result, it's no result — the same recoverable-versus-unrecoverable asymmetry that drove the exit-code decision, applied to measurement rather than scoring.

Exit 0 was tried first and reversed. The reason is worth recording: exit 0 was justified by the marker making an N/A detectable, but detection is not correction. MSBench writes exitCode = 0 as passed: true, resolved is computed from it, and the run-analysis site and Kusto publish that number — a separate report saying "not applicable" cannot correct a headline that says green, because nobody investigates green. Exit 3 makes the raw score pessimistic and recoverable; exit 0 makes it optimistic and unrecoverable.

class= answers exactly one question: dead weight, or a hole worth closing? outOfScope red is a complaint about the wiring; coverageGap red is a true statement that we aren't testing something we claim to, and is correct until fixed. The test for any new reason code is whether it gets fixed by unwiring the gate or by closing a hole.
The class is passed explicitly by the caller rather than looked up centrally, so each family of gates owns its vocabulary and adding a code is never a line two sessions edit at once.
gate= now appears on PASS:/FAIL: too, derived from the grader's filename rather than its prose description — a reworded description silently becomes a different gate to anything aggregating history (this has already split one real gate's record into 7 runs and 3). It already matches the certification validator id, so it doubles as a join key to report.json. Zero edits to the eleven existing graders. Note this only unifies identity for program/exec: gates; SQL assertions have no stderr and keep comment-string identity.
The sentinel is the first line on the N/A path and is emitted from exactly one place, so a grader that dies before reaching it stays a harness fault rather than being laundered into "not applicable".

The important part: a not-applicable branch that nothing exercises is just a differently-shaped vacuous gate. So reference-go-unsupported's golden case asserts ecosystemNotSupported rather than "passed", via the new offlineExpectations. Under exit 3 that pins a path that fails runs, so the tempting fix is to make N/A fall through to green — and this is what stops it.

Cross-stack coverage is executed, not asserted

Node, Python and .NET each get a fixture with mutations, because "works across ecosystems" written in a comment is worth nothing. A datastore check that only understands npm would pass the Python fixture's golden case while being unable to fail any mutation of it — vacuous in exactly the way this suite exists to prevent.

The two ORM controls (fidelity-orm-owns-the-driver-python, …-dotnet) are expectedCode: "passed" negative controls, and I verified each actually goes red when its fix is reverted — the first version of the .NET one didn't, and was rewritten.

That check is worth generalising, because certification cannot do it for you: certification asserts that a mutation produces an expected code, but nothing asserts the mutation is reachable by the code path you think you are testing. The .NET control passed either way, because using Microsoft.EntityFrameworkCore; matched the ORM list by exact name regardless of the provider-prefix handling it was supposed to pin — so it would have stayed green forever, for the wrong reason. A negative control you have not personally watched fail is a decoration.

Deliberately not built

Route-definition fidelity. The plan's Route Definitions table is tempting, but detecting real routes across Express / Azure Functions / FastAPI / ASP.NET isn't cheaply robust, and a gate that's right 80% of the time is worse than no gate. Recording this as a decision, not an oversight.

Known limitation: `sample-agent-output` is not a realistic fixture

Worth writing down rather than rediscovering. Its .azure/project-plan.md describes a root-level vanilla Node app (src/server.js, public/index.html), while its actual tree is a React SPA under services/web with no backend — the plan describes reference-node-fullstack's tree, not its own. It also has no per-service sections at all.

It is, in other words, itself an instance of the bug this PR builds gates for, which is why fidelity needed its own fixtures. This doesn't invalidate the existing scaffold gates — by design none of them read the plan, so a wrong plan can't make them pass wrongly — but it does mean our scaffold certification runs against a fixture that could not have come from a real run. Not fixing it here: six validators and ~15 mutations are pinned to it.

Follow-ups this PR deliberately doesn't do

readArtifact reports an unstaged workspace as a product failure. A missing artifact throws ProductFailure → exit 1, and all eleven graders use it. So a wrong-path or unstaged workspace is indistinguishable from the agent producing a bad artifact — the harness billing its own defect to the corpus, silently, because exit 1 is a plausible verdict. Pre-existing and harness-wide, so not widened into here.

Queried, but not yet measurable. A first pass over the 26-instance corpus found 10 validate-* exec rows and zero matching does not exist (workspace …). That null result is not evidence of absence, for two independent reasons: only one exit-1 row exists in the whole corpus, so the test has almost no power; and the corpus reader used to run it has a known order-dependent extraction bug that can report an archive as having no instances at all. Both failure modes point the same way — toward "no finding" — which is the direction that quietly retires a live hypothesis.

So this stays open: re-run once the reader is fixed and exec: rows have accumulated. The (workspace …) suffix readArtifact already prints is what keeps it answerable and should stay exactly as it is.
regrade.ts attributes purely on exit code, so it will now report an N/A as grader-error. Safe direction, but it should learn to read the NOT_APPLICABLE marker and report the three outcomes separately.
Wiring, including which stacks these attach to. Applicability is a wiring-time decision; a gate that doesn't apply to a stack shouldn't be wired to it rather than relying on a runtime skip. Owned by the msbench-config session.
Certification should score mutation reachability, not just outcome. Certification asserts a mutation produces an expected code; nothing asserts the mutation reaches the code path it is supposed to pin — which is exactly how the .NET ORM control here was vacuous while green. "Does this case go red when its fix is reverted" would make that a first-class output rather than a manual exercise. Suggested by the gate-health session as the fixture-level analogue of its never-failed verdict.
Plan-prose ↔ config port agreement — reference-node-fullstack's debug plan says port 3000 while launch.json says 7071 (found by the runtime-gates session). A real instance of this PR's category, worth an adversarial fixture rather than only a golden one if it lands.

Scope

evals/graders/, evals/src/artifacts/, evals/grader-certification/, plus evals/src/graderCertification.ts (pre-agreed): registering the two validators, making delete recursive so a mutation can remove a service directory, and adding optional per-fixture offlineExpectations so a golden case can assert something other than "passed". Both additions are backward-compatible; no existing manifest entry changed. offlineExpectations has a second consumer already — the runtime gates need the same "unsupported stack correctly reports not-applicable" pin on the non-Node fixtures. Also de-duplicated the non-visual App Type list so projectPlan.ts and the fidelity gates can't drift on what "API only" means.

npm run typecheck, npm run certify (70/70) and npm run drift all pass. No MSBench run submitted — everything here is verifiable offline. The certification report is regenerated only where it changed materially; timestamp-only churn is excluded.

An independent review pass found eight real bugs in the first draft, seven of which were false positives that would have failed correct projects (a renamed worker directory, a Next.js frontend, a flat backend/frontend/worker layout, a mixed-stack repo, EF Core provider packages, unpinned pyproject.toml dependencies, and test-only driver imports satisfying the wiring check). All are fixed and each fix was re-verified against the case that triggered it.

Every scaffold gate today grades the tree against itself — it builds, the frontend is embeddable, the API seam holds. All of them pass a project that dropped a service or wired the wrong datastore, because two working services look exactly like two working services. The plan is the only artifact that knows there should have been three. Two validators, both comparing `.azure/project-plan.md` to the tree: - service fidelity — every planned service exists, nothing was invented, the frontend exists iff the plan says so, and each service's language and framework match what was promised. - datastore fidelity — the wired datastore is the planned one, planned resources are actually referenced, and no unplanned datastore is wired. Both sides of the datastore check normalise to a closed set of families and compare families, never strings. The planned family is resolved from the local connection string's scheme before its marketing name, so "Cosmos DB for MongoDB" resolves to mongodb — the wire protocol is what decides which client the code must speak. Evidence is manifests union imports, because `import sqlite3` and `node:sqlite` never appear in a manifest, and a quiet swap to SQLite is the likeliest form of this bug. On a stack with no analyser these report not-applicable, never a pass, using the shared `NOT_APPLICABLE:` stderr marker agreed with the runtime-gates and gate-health sessions. A Go fixture pins that behaviour as a certified golden expectation, so the escape hatch cannot quietly become green. Every gate has a mutation that turns it red; certification goes 44/44 -> 70/70. Co-authored-by: Copilot App <[email protected]>

…er shape Exit 0 was wrong, and the reason it was wrong is worth recording. It was chosen on the grounds that the stderr marker made an N/A detectable — but detection is not correction. MSBench writes exitCode = 0 as passed: true, `resolved` is computed from it, and the run-analysis site and Kusto publish that number. A separate report saying "not applicable" cannot correct a headline that says green, because nobody investigates green. Exit 3 makes the raw score pessimistic and recoverable; exit 0 makes it optimistic and unrecoverable. A red run carrying a NOT_APPLICABLE marker is explainable in seconds. Also aligns with the marker shape the runtime-gates session landed: NOT_APPLICABLE gate=<id> class=<outOfScope|environmentGap> reason=<code> detail="…" `class` replaces the earlier outOfScope/notAttempted split and is now passed explicitly by the caller rather than looked up in a central registry, so each family of gates owns its reason vocabulary and adding a code is never a line two sessions edit at once. The central registry is removed. `ecosystemNotSupported` is classified `environmentGap`, not `outOfScope`: a Go project is not a scenario with nothing to test, it is one we are failing to test. The remedy is "write the analyser", not "unwire the gate", and under exit 3 that distinction is what tells a wiring complaint apart from a true statement that we are not testing something we claim to. Certification is unaffected at 70/70 — it grades issue codes, not exit codes, so the Go fixture still pins the N/A path. Co-authored-by: Copilot App <[email protected]>

Under exit 3 a human reads this on a failed run, and the expensive mistake is concluding the red says something about the agent's output. Name the class in prose and state plainly that nothing in the verdict is evidence about the generated app. Mirrors the wording the runtime gates use. Co-authored-by: Copilot App <[email protected]>

Copilot

Pull request overview

Adds “fidelity” evaluation gates so Copilot on Rails scaffolds are graded against what the plan promised (services + datastores), and expands the Copilot on Rails extension surface area (webviews, MCP tools, tree progress, diagnostics), alongside new eval infrastructure/fixtures and CI wiring.

Changes:

Add new scaffold fidelity graders/validators (service + datastore fidelity) plus expanded grader-certification fixtures and MSBench/Vally config to exercise them.
Introduce/expand Copilot on Rails UI + tooling surfaces (webview bundling/registry, loading/next-steps controllers, progress tree nodes, MCP tools, diagnostics utilities).
Add supporting repo plumbing (eslint ignores/overrides, tsconfig excludes, .vscode tasks, .vscodeignore/.gitignore, agent metadata + reference docs, agent-contracts workflow).

Docs / screenshots to refresh (per CoR doc-sync rule):

docs/copilot-create-project.md should be updated to include the newly introduced Loading view surface (and add/flag a screenshot placeholder for that new state).
If the Azure Project progress tree or stage UI changed materially, 12-azure-project-progress-tree.png may need a refresh (verify against current UI).

Show a summary per file

File	Description
tsconfig.json	Suppress TS deprecations; exclude evals/views from main project roots.
test/testUtils.ts	Add VS Code test helper for workspace folders.
test/testProjects/copilotOnRails/scrapbook/README.md	Add scrapbook fixture README.
test/testProjects/copilotOnRails/attendance/README.md	Add attendance fixture README.
test/test.code-workspace	Add fixture folders to test workspace.
test/copilotOnRails/planStatus.test.ts	Add unit tests for plan status rewriting.
test/agentInstructionsDisclaimer.test.ts	Add unit tests for agent-instructions disclaimer insertion.
src/webviews/copilotOnRails/views/WebviewRegistry.ts	Centralize React view registry for webview rendering.
src/webviews/copilotOnRails/views/webviewEntry.tsx	Webview render entrypoint using the registry.
src/webviews/copilotOnRails/views/utils/projectPlanStatus.ts	Shared status constants + comparison helpers for plan status.
src/webviews/copilotOnRails/views/utils/emulatorSupport.ts	Shared “limited emulator support” detection + message.
src/webviews/copilotOnRails/views/utils/deploymentPlanTypes.ts	Types for deployment plan parsing/rendering.
src/webviews/copilotOnRails/views/tsconfig.json	Dedicated TS config for webview source typechecking.
src/webviews/copilotOnRails/views/styles/loadingView.scss	Styles for the transient loading view.
src/webviews/copilotOnRails/views/styles/_fluentTooltipPortal.scss	Ensure Fluent tooltip portal styling matches VS Code theme tokens.
src/webviews/copilotOnRails/views/scss.d.ts	SCSS ambient module declaration for webviews.
src/webviews/copilotOnRails/views/react-shim.js	React injection shim for esbuild bundling.
src/webviews/copilotOnRails/views/fluentui-icons.d.ts	Workaround typings hole for `@fluentui/react-icons`.
src/webviews/copilotOnRails/views/components/StageProgress.tsx	Stage progress UI component.
src/webviews/copilotOnRails/extension/reloadResumePrompt.ts	Stash/consume create prompt across reload for resume.
src/webviews/copilotOnRails/extension/recentPrompts.ts	Track/prune recent prompts in globalState.
src/webviews/copilotOnRails/extension/openScaffoldNextStepsView.ts	Open/update scaffold next-steps webview; close loading first.
src/webviews/copilotOnRails/extension/openLocalDevNextStepsView.ts	Open/update local-dev next-steps; detect API tests; telemetry.
src/webviews/copilotOnRails/extension/openLoadingView.ts	Open/update/close transient loading webview.
src/webviews/copilotOnRails/extension/harnessSettings.ts	Force-disable experimental “Copilot Harness” settings in workspace.
src/webviews/copilotOnRails/extension/copilotOnRailsBundleLocation.ts	Provide webview bundle location (dist dir + filenames).
src/webviews/copilotOnRails/extension/controllers/LoadingViewController.ts	Webview controller for loading view + message handling.
src/utils/settingUtils.ts	Add optional configuration target when updating workspace settings.
src/utils/copilotOnRails/prepareNewCorProject.ts	Clear CoR workspace state + reset submission state before new runs.
src/utils/copilotOnRails/CopilotOnRailsContext.ts	Add CoR-specific diagnostics context wrapper + helper.
src/tree/project/StateStageNode.ts	Tree node for “Start/Resume” stage actions.
src/tree/project/StageNode.ts	Base stage node w/ state decoration + icon logic.
src/tree/project/registerProjectSubmissionStateWatcher.ts	Watch for plan files to clear “pending submission” state.
src/tree/project/projectSubmissionState.ts	In-memory “submission pending” state for the Azure Project tree.
src/tree/project/ProjectCreationStageItem.ts	Stage node for project creation step.
src/tree/project/ProgressNode.ts	Shared interface for progress tree nodes.
src/tree/project/OpenPlanNode.ts	Tree node that opens plan views.
src/tree/project/LocalDevelopmentStageItem.ts	Stage node for local development step.
src/tree/project/DeploymentStageItem.ts	Stage node for deployment step (start vs resume label).
src/tree/project/DebugConfigurationNode.ts	Tree node for starting a debug configuration.
src/extensionVariables.ts	Add `refreshProjectTree` action hook.
src/constants.ts	Add Azure Project IDs + agent names + Copilot-for-Azure extension ID.
src/commands/registerCommands.ts	Register Copilot on Rails commands in main command registration.
src/commands/copilotOnRails/startDebugConfiguration.ts	Command handler to start a named debug config.
src/commands/copilotOnRails/inspectDiagnostics.ts	Command to open workspace-cached diagnostics as JSON.
src/chat/tools/registerMcpTools.ts	Register CoR MCP tools and fix activity log tool import path.
src/chat/tools/copilotOnRails/startProjectScaffoldTool.ts	MCP tool to start scaffold agent.
src/chat/tools/copilotOnRails/startProjectIntegrateTool.ts	MCP tool to start integrate agent.
src/chat/tools/copilotOnRails/startLocalDevelopmentTool.ts	MCP tool to start local dev plan agent.
src/chat/tools/copilotOnRails/startDeploymentTool.ts	MCP tool to start deploy agent.
src/chat/tools/copilotOnRails/startAzureDebugGenerateTool.ts	MCP tool to start debug-generate agent.
src/chat/tools/copilotOnRails/registerCopilotOnRailsTools.ts	Central registration of all CoR MCP tools.
src/chat/tools/copilotOnRails/openScaffoldNextStepsViewTool.ts	MCP tool to open scaffold next-steps view.
src/chat/tools/copilotOnRails/openRequirementsViewTool.ts	MCP tool to open requirements view.
src/chat/tools/copilotOnRails/openPlanViewTool.ts	MCP tool to open plan preview view.
src/chat/tools/copilotOnRails/openLocalPlanViewTool.ts	MCP tool to open local debug plan view.
src/chat/tools/copilotOnRails/openLocalNextStepsViewTool.ts	MCP tool to open local next-steps view (optional `hasApiTests`).
src/chat/tools/copilotOnRails/openFrontendPreviewViewTool.ts	MCP tool to open frontend preview view (optional folder).
src/chat/tools/copilotOnRails/openDeployResultViewTool.ts	MCP tool to open deploy results view.
src/chat/tools/copilotOnRails/openDeployPlanViewTool.ts	MCP tool to open deploy plan view.
src/chat/chatStandIn.ts	Use shared constant for Copilot-for-Azure extension ID.
resources/agents/shared-references/.metadata.json	Agent bundle metadata.
resources/agents/azure-project-scaffold/.metadata.json	Agent bundle metadata.
resources/agents/azure-project-plan/.metadata.json	Agent bundle metadata.
resources/agents/azure-project-integrate/references/end-to-end.md	Add integration verification reference for agent.
resources/agents/azure-project-integrate/.metadata.json	Agent bundle metadata.
resources/agents/azure-deploy/scaffold/references/waf-checklist.md	Add deploy scaffold WAF checklist reference.
resources/agents/azure-deploy/scaffold/references/self-review-procedure.md	Add deploy scaffold self-review procedure reference.
resources/agents/azure-deploy/scaffold/references/self-healing.md	Add self-healing loop reference.
resources/agents/azure-deploy/scaffold/references/error-handling.md	Add scaffold error handling reference.
resources/agents/azure-deploy/scaffold/references/cicd-pipelines.md	Document CI/CD deferral + guidance path.
resources/agents/azure-deploy/scaffold/references/bicep-swa.md	Add SWA Bicep pattern reference.
resources/agents/azure-deploy/references/subscription-resolution.md	Add subscription resolution reference.
resources/agents/azure-deploy/references/intent-gathering.md	Add intent-gathering reference.
resources/agents/azure-deploy/references/iac-resources.md	Add IaC docs/resources reference.
resources/agents/azure-deploy/prereq/references/subscription-resolution.md	Add prereq subscription resolution reference.
resources/agents/azure-deploy/prereq/references/prereq-artifacts.md	Add prereq artifacts writing reference.
resources/agents/azure-deploy/prereq/references/cloud-sdk-migration.md	Add non-Azure SDK migration classification reference.
resources/agents/azure-deploy/prepare/references/validation-rubric.md	Add prepare validation rubric reference.
resources/agents/azure-deploy/.metadata.json	Agent bundle metadata.
resources/agents/azure-debug-plan/references/runtimes.md	Add runtimes detection reference.
resources/agents/azure-debug-plan/references/multi-service.md	Add multi-service orchestration reference.
resources/agents/azure-debug-plan/references/migrations.md	Add migrations detection reference.
resources/agents/azure-debug-plan/references/classify.md	Add workspace classification reference.
resources/agents/azure-debug-plan/.metadata.json	Agent bundle metadata.
resources/agents/azure-debug-generate/references/project-types/frontend-spa/debug-adapters/blazorwasm.md	Add planned Blazor WASM debug adapter reference.
resources/agents/azure-debug-generate/references/project-types/frontend-spa/debug-adapters/_template.md	Add debug adapter template reference.
resources/agents/azure-debug-generate/references/preflight.md	Add preflight checks reference (stale dirs, ports).
resources/agents/azure-debug-generate/references/emulators/azurite.md	Add Azurite emulator reference.
resources/agents/azure-debug-generate/references/emulators/_template.md	Add emulator doc template.
resources/agents/azure-debug-generate/.metadata.json	Agent bundle metadata.
package.nls.json	Add CoR + Azure Project UI strings.
evals/tsconfig.json	Add evals TS project config (typecheck graders/validators).
evals/src/artifacts/validationTypes.ts	Add shared validation result types/helpers.
evals/src/artifacts/planEvaluation.ts	Add plan-gate contract validation for preview/intent.
evals/scenarios/api-ts-functions-minimal.json	Add scenario definition for minimal TS Functions API.
evals/release-thresholds.v1.json	Add release threshold set v1.
evals/package.json	Add evals package (scripts + vally deps + TS).
evals/msbench/config/stimuli/no-datastore-converter.yaml	Add MSBench stimulus config for no-datastore scenario.
evals/msbench/config/stimuli/multi-service-order-processing.yaml	Add MSBench stimulus config for multi-service scenario.
evals/msbench/config/phases/plan.yaml	Add MSBench phase config for plan stage.
evals/msbench/assets/extensions/.gitkeep	Keep extensions assets dir in git.
evals/local-dev/fixtures/functions-postgres/.azure/project-plan.md	Add local-dev fixture plan for functions+postgres.
evals/graders/validate-webview-parseable.ts	Grader: plan renders via real webview parser.
evals/graders/validate-service-fidelity.ts	Grader: planned vs scaffolded services fidelity.
evals/graders/validate-project-plan.ts	Grader: plan artifact satisfies contract.
evals/graders/validate-integration-plan.ts	Grader: integration plan artifact contract.
evals/graders/validate-frontend-scaffold.ts	Grader: frontend scaffold embeddability + seam.
evals/graders/validate-debug-gate.ts	Grader: debug plan stops at approval gate.
evals/graders/validate-debug-config.ts	Grader: launch/tasks structural correctness.
evals/graders/validate-debug-artifacts.ts	Grader: generated debug artifacts match plan.
evals/graders/validate-datastore-fidelity.ts	Grader: planned vs wired datastore fidelity (+ N/A path).
evals/grader-certification/sample-agent-output/services/web/vite.config.ts	Certification fixture content (web).
evals/grader-certification/sample-agent-output/services/web/tsconfig.json	Certification fixture content (web).
evals/grader-certification/sample-agent-output/services/web/src/test/tickets.test.ts	Certification fixture content (web test).
evals/grader-certification/sample-agent-output/services/web/src/pages/TicketsPage.tsx	Certification fixture content (web page).
evals/grader-certification/sample-agent-output/services/web/src/mocks/data.ts	Certification fixture content (mocks).
evals/grader-certification/sample-agent-output/services/web/src/main.tsx	Certification fixture content (entry).
evals/grader-certification/sample-agent-output/services/web/src/App.tsx	Certification fixture content (app).
evals/grader-certification/sample-agent-output/services/web/src/api/types.ts	Certification fixture content (types).
evals/grader-certification/sample-agent-output/services/web/src/api/previewState.ts	Certification fixture content (preview state).
evals/grader-certification/sample-agent-output/services/web/src/api/mockClient.ts	Certification fixture content (mock client).
evals/grader-certification/sample-agent-output/services/web/src/api/index.ts	Certification fixture content (API seam).
evals/grader-certification/sample-agent-output/services/web/src/api/client.ts	Certification fixture content (client interface).
evals/grader-certification/sample-agent-output/services/web/package.json	Certification fixture content (web pkg).
evals/grader-certification/sample-agent-output/services/web/index.html	Certification fixture content (web html).
evals/grader-certification/sample-agent-output/services/shared/src/types/ticket.ts	Certification fixture content (shared types).
evals/grader-certification/sample-agent-output/services/shared/src/index.ts	Certification fixture content (shared index).
evals/grader-certification/sample-agent-output/services/shared/package.json	Certification fixture content (shared pkg).
evals/grader-certification/sample-agent-output/scenario.json	Certification fixture scenario definition.
evals/grader-certification/sample-agent-output/.azure/project-plan.md	Certification fixture plan artifact.
evals/grader-certification/sample-agent-output/.azure/integration-plan.md	Certification fixture integration artifact.
evals/grader-certification/sample-agent-output/.azure/deployment-plan.md	Certification fixture deployment artifact.
evals/grader-certification/sample-agent-output/.azure/.preview-temp/theme.css	Certification fixture preview artifact.
evals/grader-certification/sample-agent-output/.azure/.preview-temp/project-tracker.html	Certification fixture preview artifact.
evals/grader-certification/sample-agent-output/.azure/.preview-temp/manifest.json	Certification fixture preview manifest.
evals/grader-certification/reference-python-api/services/api/requirements.txt	Reference fixture (python) deps.
evals/grader-certification/reference-python-api/services/api/main.py	Reference fixture (python) code.
evals/grader-certification/reference-python-api/scenario.json	Reference fixture (python) scenario.
evals/grader-certification/reference-python-api/.env.example	Reference fixture (python) env.
evals/grader-certification/reference-python-api/.azure/project-plan.md	Reference fixture (python) plan.
evals/grader-certification/reference-node-multiservice/services/worker/src/index.ts	Reference fixture (node multi) worker code.
evals/grader-certification/reference-node-multiservice/services/worker/package.json	Reference fixture (node multi) worker pkg.
evals/grader-certification/reference-node-multiservice/services/web/vite.config.ts	Reference fixture (node multi) web config.
evals/grader-certification/reference-node-multiservice/services/web/src/main.tsx	Reference fixture (node multi) web entry.
evals/grader-certification/reference-node-multiservice/services/web/src/App.tsx	Reference fixture (node multi) web app.
evals/grader-certification/reference-node-multiservice/services/web/package.json	Reference fixture (node multi) web pkg.
evals/grader-certification/reference-node-multiservice/services/web/index.html	Reference fixture (node multi) web html.
evals/grader-certification/reference-node-multiservice/services/shared/src/types.ts	Reference fixture (node multi) shared types.
evals/grader-certification/reference-node-multiservice/services/shared/package.json	Reference fixture (node multi) shared pkg.
evals/grader-certification/reference-node-multiservice/services/api/src/storage.ts	Reference fixture (node multi) storage code.
evals/grader-certification/reference-node-multiservice/services/api/src/index.ts	Reference fixture (node multi) API server.
evals/grader-certification/reference-node-multiservice/services/api/src/db.ts	Reference fixture (node multi) pg usage.
evals/grader-certification/reference-node-multiservice/services/api/package.json	Reference fixture (node multi) API pkg.
evals/grader-certification/reference-node-multiservice/scenario.json	Reference fixture (node multi) scenario.
evals/grader-certification/reference-node-multiservice/.env.example	Reference fixture (node multi) env.
evals/grader-certification/reference-node-fullstack/test/server.test.js	Reference fixture (node fullstack) tests.
evals/grader-certification/reference-node-fullstack/scripts/lint.js	Reference fixture (node fullstack) lint.
evals/grader-certification/reference-node-fullstack/public/styles.css	Reference fixture (node fullstack) styles.
evals/grader-certification/reference-node-fullstack/public/index.html	Reference fixture (node fullstack) html.
evals/grader-certification/reference-node-fullstack/public/app.js	Reference fixture (node fullstack) client js.
evals/grader-certification/reference-node-fullstack/package.json	Reference fixture (node fullstack) pkg.
evals/grader-certification/reference-node-fullstack/package-lock.json	Reference fixture (node fullstack) lock.
evals/grader-certification/reference-node-fullstack/infra/main.bicep	Reference fixture (node fullstack) infra.
evals/grader-certification/reference-node-fullstack/azure.yaml	Reference fixture (node fullstack) azd.
evals/grader-certification/reference-node-fullstack/api-test-collections/golden-app/health/invoke.sh	Reference fixture API test probe.
evals/grader-certification/reference-node-fullstack/.vscode/tasks.json	Reference fixture debug tasks.
evals/grader-certification/reference-node-fullstack/.vscode/settings.json	Reference fixture debug settings.
evals/grader-certification/reference-node-fullstack/.vscode/launch.json	Reference fixture launch config.
evals/grader-certification/reference-node-fullstack/.vscode/extensions.json	Reference fixture ext recommendations.
evals/grader-certification/reference-node-fullstack/.env.example	Reference fixture env.
evals/grader-certification/reference-node-fullstack/.azure/vscode-debug-plan.md	Reference fixture debug plan artifact.
evals/grader-certification/reference-node-fullstack/.azure/integration-plan.md	Reference fixture integration artifact.
evals/grader-certification/reference-go-unsupported/services/api/main.go	Reference fixture for unsupported ecosystem path.
evals/grader-certification/reference-go-unsupported/services/api/go.mod	Reference fixture for unsupported ecosystem path.
evals/grader-certification/reference-go-unsupported/scenario.json	Reference fixture for unsupported ecosystem path.
evals/grader-certification/reference-go-unsupported/.azure/project-plan.md	Reference fixture plan (unsupported ecosystem).
evals/grader-certification/reference-dotnet-api/services/api/Program.cs	Reference fixture (.NET) code.
evals/grader-certification/reference-dotnet-api/services/api/Api.csproj	Reference fixture (.NET) project file.
evals/grader-certification/reference-dotnet-api/scenario.json	Reference fixture (.NET) scenario.
evals/grader-certification/reference-dotnet-api/.env.example	Reference fixture (.NET) env.
evals/grader-certification/reference-dotnet-api/.azure/project-plan.md	Reference fixture (.NET) plan.
eslint.config.mjs	Update lint ignores and add evals/webviews-specific overrides.
esbuild.copilotOnRailsViews.mjs	Add esbuild pipeline for bundling CoR React webviews + SCSS.
docs/images/copilot-create-project/README.md	Add README for screenshot asset folder.
.vscodeignore	Exclude evals/results from VSIX; include agent TS where needed.
.vscode/tasks.json	Add webviews esbuild watch task.
.vally.yaml	Add Vally project config pointing at evals/ + results dir.
.gitignore	Ignore eval outputs and MSBench staging caches.
.github/workflows/agent-contracts.yml	Add CI workflow for drift/typecheck/certify/lint of agent contracts.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Files reviewed: 42/42 changed files
Comments generated: 0
Review effort level: Lite

`environmentGap` said the machine was at fault. For the case these gates actually emit it — no analyser for this ecosystem — the fix is unwritten code in this repository, so whoever triaged the red would inspect the container, find nothing wrong, and conclude the marker was broken. A name that sends the reader to the wrong place suppresses its own investigation, which is the same defect the runtime session found when it renamed frontendServerNotStarted. `class=` now answers exactly one question: is this gate dead weight, or a hole worth closing? The test for a new reason code is whether it gets fixed by unwiring the gate or by closing a hole. Who closes it — install a binary vs write an analyser — is a difference of backlog, and `reason=` already carries that as a closed groupable vocabulary. A third class would make `class=` answer two questions at once. The shared harness prose is now owner-neutral for the same reason, with the repository-vs-machine specifics moved into each gate's `detail`, which the gate owns. Baking install-a-binary wording into the harness would have reintroduced the misdirection one layer down. Co-authored-by: Copilot App <[email protected]>

Two dialects of one shared field: this branch rewrote `"` to `'`, the runtime gates used JSON.stringify. Only the second round-trips. Rewriting quotes silently corrupts any detail containing one — and details legitimately carry shell commands, so the runtime gates' Functions detail already does — while looking perfectly fine in the log. "detail= is never parsed" was the reason to fix it rather than leave it: the gate-health reader is being written against this line now, and a field that cannot round-trip is a latent bug that fires the moment someone does the obvious thing. JSON.stringify supplies the surrounding quotes, so the field is a JSON string literal and escapes newlines too. Verified against a detail carrying quotes, backticks and a newline. Co-authored-by: Copilot App <[email protected]>

Copilot AI lite review requested due to automatic review settings August 26, 2026 05:38

Alex Weininger (alexweininger) requested a review from a team as a code owner August 26, 2026 05:38

Copilot started reviewing on behalf of Alex Weininger (alexweininger) August 26, 2026 05:39 View session

Alex Weininger (alexweininger) changed the base branch from main to feat/CoR August 26, 2026 05:39

Alex Weininger (alexweininger) and others added 2 commits August 25, 2026 22:42

Alex Weininger (alexweininger) mentioned this pull request Aug 26, 2026

Add a hostile certification fixture: prove the runtime probes report on the product, not on whatever else is running #1722

Open

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Alex Weininger (alexweininger) mentioned this pull request Aug 26, 2026

readArtifact bills an unstaged workspace to the product as exit 1 #1723

Open

Alex Weininger (alexweininger) merged commit 551e0e8 into feat/CoR Aug 26, 2026
3 checks passed

Alex Weininger (alexweininger) deleted the alexweininger-fidelity-gates branch August 26, 2026 06:17

Alex Weininger (alexweininger) mentioned this pull request Aug 26, 2026

Stop the grader scanner reading English as code #1727

Merged

Navigation Menu

Uh oh!

Add fidelity gates: did the agent build what it planned? - #1721

Add fidelity gates: did the agent build what it planned?#1721
Alex Weininger (alexweininger) merged 5 commits into
feat/CoRfrom
alexweininger-fidelity-gates

Alex Weininger (alexweininger) commented Aug 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Alex Weininger (alexweininger) commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The short version

Something that passes today and shouldn't

The gates

Two corrections to the original framing, both load-bearing

How the datastore check works, and why not a grep

Not-applicable: exit 3, and the escape hatch is itself certified

Cross-stack coverage is executed, not asserted

Deliberately not built

Known limitation: sample-agent-output is not a realistic fixture

Follow-ups this PR deliberately doesn't do

Scope

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Alex Weininger (alexweininger) commented Aug 26, 2026 •

edited

Loading

Known limitation: `sample-agent-output` is not a realistic fixture