Skip to content

fix: grade outputs by run kind - #183

Open
danielgaskins wants to merge 1 commit into
openprose:mainfrom
danielgaskins:fix/contract-grader-run-layouts
Open

danielgaskins wants to merge 1 commit into
openprose:mainfrom
danielgaskins:fix/contract-grader-run-layouts

Conversation

@danielgaskins

Copy link
Copy Markdown

Summary

  • Detect whether the subject is a mounted responsibility or a standalone function before locating contracts and outputs
  • Resolve mounted outputs from compiled intent and published world models
  • Verify receipt chains and published versions before using mounted output as grading evidence
  • Continue resolving standalone function returns from bindings
  • Report unsupported layouts, contradictory layouts, and evidence gaps explicitly
  • Add focused conformance coverage for both run kinds

Use Case / Run Evidence

Fixes #180.

The contract grader still required a retired manifest and read every contract output from bindings/. Current mounted responsibility runs instead declare their topology in compiled-intent.json and publish committed outputs through world-model/ and receipts/.

This could cause a valid mounted run to fail before any clauses were graded. It could also allow an output to receive satisfaction credit without establishing that the output was committed successfully.

Design Boundary

This change updates the standard contract-grader contract and its focused conformance coverage. It does not change contract syntax, VM semantics, filesystem layout, compilation, or host behavior.

Examples

For a mounted responsibility, the grader now resolves each node and its ### Maintains clauses from compiled intent and source snapshots. It reads output from world-model/{node}/ and checks the corresponding receipt chain and published version.

For a standalone function, the grader reads declared ### Returns outputs from bindings/{function}/ without requiring mounted-only artifacts.

If mounted output exists without trustworthy receipt or version evidence, the affected clause is reported as unevaluable. It cannot receive a satisfied or partially satisfied verdict.

Testing

  • corepack pnpm exec vitest run tests/open-prose/evals/contract-grader.test.ts --maxWorkers=1
  • corepack pnpm test -- --maxWorkers=1

The focused suite passed 7 tests. The full suite passed 308 tests across 20 files.

Residual Risk / Follow-ups

The tests verify that the grader contract follows the documented artifact and scoring rules. They do not execute contract grading through a live Prose Complete host.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make contract-grader resolve outputs by run kind

1 participant