V0.9.1/evals

CHANGELOG.md

@@ -4,6 +4,32 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.1] - 2026-08-22
+
+This patch release adds harness-agnostic Eval Runner execution boundary infrastructure without changing the paired evaluation methodology or existing report schemas, while optimizing skill descriptions and refactoring repository-level authoring guidance. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence. The deterministic fake runner is the conformance reference, with Codex, GitHub Copilot CLI, and OpenCode as supported real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed.
+
+### Added
+
+- `scripts/eval-runners/` with the common `describe`/`preflight`/`execute` process contract, execution-profile and execution-result schemas, deterministic fake runner, Codex adapter, GitHub Copilot CLI adapter, OpenCode adapter, runner resolution, artifact/hash validation, and bridge into the existing `eval-result/2` result shape,
+- GitHub Copilot CLI as a supported Eval Runner with authentication handling (GitHub tokens, OS keychain, CLI fallback), JSONL-based event output parsing, stdin-based prompt delivery for byte fidelity, repository instruction visibility, and conformance tests covering token management and authentication source detection,
+- deterministic fake-runner conformance coverage for fresh paired sessions, prompt fidelity, isolation boundaries, candidate-skill exposure, status normalization, unavailable telemetry, event warnings, artifact references, and report compatibility,
+- runner-aware package preparation that reuses `run.json`, keeps runner selection outside `evals/evals.json`, and preserves Anthropic-compatible benchmark/report artifacts.
+
+### Changed
+
+- Eval preparation now resolves Harness + Model before writing `RUN-THIS.prompt.md`, removes the redundant portable `provider` field from `execution-profile.json` and result reporting, treats model selectors as runner-native opaque strings, and adds `scripts/Get-HarnessModels.ps1` for current model discovery with Codebelt Reference verification and free-only OpenCode filtering,
+- `AGENTS.md`, `README.md`, and `CONTRIBUTING.md` now distinguish the Eval Runner, Eval Orchestrator, Grader, and Human Reviewer and clarify that runner execution is an explicit external-handoff boundary rather than repository automation,
+- All 21 repo-managed skill descriptions refactored to lean, trigger-oriented activation metadata following progressive-disclosure principles and specification compliance,
+- `AGENTS.md` Skill Authoring section restructured for clarity, brevity, and progressive disclosure of form fields, asset handling, and dynamic defaults,
+- report timing output omits unavailable duration and token telemetry instead of writing zero placeholders,
+- GitHub Copilot runner added to the eval runner lineup alongside existing Codex and OpenCode support,
+- Runner conformance tests enhanced with additional event fixtures and isolation capability assessment.
+
+### Fixed
+
+- Codex runner now ensures the evidence directory is created before writing output files, preventing file-not-found errors,
+- Test runner conformance validation now requires the output parent directory to exist upfront with explicit error reporting when the directory structure is misconfigured.
+
 ## [0.9.0] - 2026-08-20
 
 This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-remote-testing`, and `dotnet-segregated-assets` — replaces the repository's model-backed eval benchmark workflow with deterministic, local-only validation, and finalizes the portable eval handoff. The selected external evaluator now runs the paired workers, grades their results, and invokes Anthropic's skill-creator aggregator and eval viewer without sending the user back for a second collection command. `dotnet-test` bootstraps and modernizes xUnit test projects against Codebelt conventions with role-aware fixtures; `dotnet-remote-testing` runs .NET tests inside official Microsoft SDK containers using either an existing `testenvironments.json` or zero-config, offline-safe release discovery; and `dotnet-segregated-assets` migrates ASP.NET Core applications to an artifact-first topology where `wwwroot` stays the authoring root while deployed static content is served by a separate hardened origin. Alongside those, `git-keep-a-changelog` and `git-nuget-release-notes` gained deterministic release-entity classification, and `git-visual-commits` gained an invocation routing lock. No published skill was removed or renamed, so adopting this release is non-breaking for existing installs.
@@ -612,6 +638,7 @@ This is a minor release that introduces two complementary git workflow skills, e
 
 - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output.
 
+[0.9.1]: https://github.com/codebeltnet/agentic/compare/v0.9.0...v0.9.1
 [0.9.0]: https://github.com/codebeltnet/agentic/compare/v0.8.2...v0.9.0
 [0.8.2]: https://github.com/codebeltnet/agentic/compare/v0.8.1...v0.8.2
 [0.8.1]: https://github.com/codebeltnet/agentic/compare/v0.8.0...v0.8.1

CONTRIBUTING.md

@@ -89,28 +89,30 @@ Aim for 3–5 evals that cover distinct scenarios: happy path, edge cases, and c
 Evals are prepared, not executed, from this repository. Adding or modifying a repo-managed skill requires preparing the packages for every skill the branch touched, which is a completion gate rather than an optional extra:
 
 ```console
-pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed
+pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot -Model claude-haiku-4.5
 ```
 
 Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, which stays last. For a single skill on demand, use:
 
 ```console
-pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill <skill-name>
+pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill <skill-name> -Runner <runner-id> -Model <runner-native-model>
 ```
 
-The script writes `.bot/<skill-name>-workspace/iteration-<n>/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two hermetic run directories, `with_skill/` and `without_skill/`. A run directory is the worker's sandbox root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/<name>/` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the agent of your choice. That agent starts immediately, creates one isolated worker for every run, launches it from its run directory with `repo/` as the working directory and `home/` as an isolated profile, gives each worker only its `prompt.md` and staged files, writes the results back, grades after collection using the packaged grader guidance, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage.
+Before running the script, choose a Harness + Model. Use `scripts/Get-HarnessModels.ps1 -Runner <runner-id>` to list current selectors; OpenCode is free-only, while GitHub Copilot and Codex list all currently available models. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`; `-CodebeltReference` verifies that the model still exists and fails instead of silently substituting a different model. The script writes `.bot/<skill-name>-workspace/iteration-<n>/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/<name>/` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage.
 
-Repository scripts, CI jobs, and the agent that prepares a package never run those prompts. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A user-selected harness handed a specific package is the executor, not the preparer; its current context orchestrates fresh workers while the workers run the prompt files.
+Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals.
 
-Run both configurations on the same model, same version, and same configuration. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result.
+Run both configurations on the same model, same version, and same configuration. Independent arms must be dispatched concurrently up to `execution-profile.json.concurrency` when the harness permits it. For any selected runner whose descriptor says `delegation.dispatch_owner=runner`, invoke `control-runner-owned-phase1.ps1` with a bounded `WaitSeconds` value and repeat that same idempotent call while status is `running`. The durable supervisor alone invokes the internal fan-out once; never invoke `invoke-runner-owned-arms.ps1` directly, create outer native subagents/tasks, or hand-author preflight, fan-out, state, or result bookkeeping. Copilot task/general-purpose workers, OpenCode Task/General workers, and Codex native mechanisms remain harness capabilities only; they are not the behavioral transport for runner-owned evaluation. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result.
 
-Record each external result in the matching `results/*.result.json`: `model`, `provider`, `harness`, and the complete `output`; include `transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, and `tool_calls` when the harness exposes them, and the `isolation` flags the harness confirmed. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. The normal external evaluator writes `grading[].passed` and evidence, then generates the report before handing the package back. If the results were transferred without those report artifacts, validate and compare with:
+For an orchestrator-owned worker, preserve its `codebeltnet/agentic/eval-native-worker-result/1` terminal envelope and pass it to `record-native-result.ps1` with the exact `run.json`, `execution-profile.json`, and manifest-declared output path. For a runner-owned worker, preserve the runner-produced `execution-result.json` directly at the exact manifest-declared path and do not invoke the recorder or synthesize an envelope. In either mode, the generated result must carry the protocol/schema, opaque run and fresh session ids, status, complete final response or explicit unavailability, runner/harness identity, requested and resolved model selection, timestamps and duration, exit/failure state, prompt/run/profile hashes, resolved isolation mechanisms, warnings, and artifact references. Include token, cache, cost, tool, command, file, and transcript evidence only when the harness exposes it; unavailable values remain explicitly unavailable and are never estimated. The deterministic bridge then writes the existing `results/*.result.json` shape, after which grading may add `grading[].passed` and evidence. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. If the selected external process cannot write valid runner-produced execution results back into the package, the evaluation is incomplete and must fail closed; no response-only or reconstructed result is accepted.
+
+An explicitly authorized forensic recovery of an old or broken package may validate an existing iteration with:
 
 ```console
 pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults <iteration-path>
 ```
 
-That writes `comparison.md`, the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`, while flagging missing arms, unrun configurations, and mixed models. The normal external evaluator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model.
+It may write a diagnostic `comparison.md` while flagging missing arms, unrun configurations, incompatible evidence, and mixed models, but it exits non-zero and does not write benchmark/report artifacts until the required completion gate is satisfied. Those diagnostic artifacts must not present an incomplete or unrun package as a successfully completed evaluation. The normal external Eval Orchestrator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. Codex, OpenCode, and GitHub Copilot are the conforming real runners; the deterministic fake runner is the CI conformance harness. Hard filesystem confinement is reported as strict versus pragmatic confidence and is not a universal Windows prerequisite. Freebuff remains planned/blocked until its official CLI provides a supported noninteractive machine-readable transport. Native skill activation, portability scoring, and additional runners are not part of v0.9.1.
 
 The eval package is a temp artifact. Do not commit it, its prompts, or its results unless the change explicitly calls for checked-in examples.
 
@@ -147,8 +149,8 @@ pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD
 - [ ] At least one eval in `evals/evals.json`
 - [ ] The skill's `evals/evals.json` exists and its `skill_name` matches the folder/frontmatter name
 - [ ] Any optional `files` entries in `evals/evals.json` point to real fixture files under the same skill folder
-- [ ] `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` was run after the last skill edit, and the prepared prompt paths were reported
-- [ ] If an external evaluation was run, each result includes the producing model and the package contains the first-party `report.html`, exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md`; use `-CollectResults` only when transferred results need the repository-side fallback
+- [ ] `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner <runner-id> -Model <runner-native-model>` or `-CodebeltReference` was run after the last skill edit, and the prepared prompt paths were reported
+- [ ] If an external evaluation was run, each result includes the producing model and the package contains the first-party `report.html`, exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md`; use `-CollectResults` only for explicitly authorized forensic recovery of an existing package
 - [ ] `scripts/validate-skill-templates.ps1` passes for the current working tree when changing scaffold or template behavior
 - [ ] If CI is enabled for the branch, the GitHub Actions validation job passes too
 - [ ] Eval packages live in `.bot/<skill-name>-workspace/` or a temp path, never anywhere else in the working tree