V0.9.1/evals - #39
Open
gimlichael wants to merge 71 commits into
Open
Conversation
Update AGENTS.md, CONTRIBUTING.md, README.md, and CHANGELOG.md to document the new portable evaluation handoff process, eval runner infrastructure, and updated automation prohibition. Clarify the roles of Eval Orchestrator, Eval Runner, Grader, and Human Reviewer in the skill evaluation workflow.
Add portable evaluation runner framework with support for multiple harnesses: Codex, OpenCode, and deterministic Fake runner. Includes contract schemas, runner adapters, result bridging, and conformance tests. Refactor eval preparation and report generation scripts to support the new runner abstraction.
Greptile SummaryThis PR introduces a runner-agnostic evaluation protocol with Codex, OpenCode, Copilot, and deterministic fake-runner adapters, plus package orchestration, evidence freezing, grading bridges, schemas, and regression coverage.
Confidence Score: 1/5The PR does not appear safe to merge until the outstanding Codex and OpenCode isolation paths stop accepting runs that can access package-level evaluation data. Codex subscription execution and OpenCode fallback execution can still run without an OS filesystem sandbox; OpenCode additionally records definitive paired-arm and grading-data exclusion claims despite that missing boundary, allowing contaminated evaluation evidence to be accepted. Files Needing Attention: scripts/eval-runners/codex/runner.ps1, scripts/eval-runners/opencode/runner.ps1, AGENTS.md Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[Repository preparation] --> B[Prepared eval package]
B --> C[External orchestrator]
C --> D[Eval runner preflight]
D --> E[Runner-owned or orchestrator-owned execution]
E --> F[Execution result and frozen evidence]
F --> G[Deterministic bridge and grading]
G --> H[Reports]
Reviews (30): Last reviewed commit: "🐛 fix iteration-11 bug: resolve writabl..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Update CONTRIBUTING.md and README.md to document runner infrastructure, installation procedures, and eval orchestrator workflow.
Add isolation capability assessment and preflight validation to detect unsupported runner configurations. Implement Cline runner adapter for eval orchestration. Enhance common runner utilities with sandbox path mapping, external command versioning, and environment management. Expand conformance tests with additional event fixtures.
Add GitHub Copilot CLI as a supported Eval Runner alongside Cline, Codex, and OpenCode. The runner handles Copilot-specific authentication (GitHub tokens), isolation constraints, and JSONL-based event output parsing. Includes conformance tests and reference documentation for the runner protocol compliance.
Extend prepare-skill-evals.ps1 and validate-skill-templates.ps1 to support GitHub Copilot runner configuration and execution profile selection. Enables dynamic runner resolution and integration with the new GitHub Copilot CLI evaluation path.
Establish GitHub Copilot with claude-haiku-4.5 as the Codebelt reference evaluation configuration for economical and stable comparison across evals. Update eval isolation terminology from 'hermetic' to 'isolated' and clarify Windows support as a first-class pragmatic evaluation target. Document filesystem confinement as a confidence enhancement rather than a universal prerequisite, enabling broader platform coverage.
Modernize the GitHub Copilot eval runner to deliver the prepared prompt through stdin instead of the --prompt argument. This improves byte fidelity and supports larger prompts without hitting command-line length limits. Update authentication handling to follow Copilot's normal precedence: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN environment variables, then OS keychain, then GitHub CLI fallback through GH_CONFIG_DIR. Remove --no-custom-instructions so repository-owned instructions remain visible to both paired arms. Add COPILOT_CACHE_HOME isolation. Refine --secret-env-vars to filter all listed token variables from shell and MCP child environments. Update preflight to report conditional native keychain readiness without requiring a live model request. Extend runner conformance tests to validate stdin prompt delivery with byte-exact verification, multiple token variables, repository instruction visibility, and authentication source detection.
The codex runner writes to the evidence directory but was not ensuring it existed first. Adding defensive directory creation prevents file-not-found errors when the evidence directory hasn't been created yet.
Change from defensive directory creation to explicit validation. The conformance test now requires the output parent directory to already exist and exits with a clear error message if it doesn't, improving error clarity when the directory structure is misconfigured.
Restructured AGENTS.md Skill Authoring section for clarity, brevity, and progressive disclosure. Updated README.md skill catalog to note that descriptions are lean activation metadata. Optimized all 21 repo-managed skill descriptions to be concise, trigger-oriented metadata following the new authoring standard. Changes align description quality with specification requirements and improve skill discoverability.
Added Layered Capability Classification section to distinguish classification of independently selectable adapters and runners from their parent directory or framework. This ensures that new child adapters are correctly classified as Added rather than grouped with pre-existing framework changes. Added comprehensive eval case covering framework refinement, new adapter introduction, defect repair, and layered state classification to validate the enhanced guidance.
Extended git-keep-a-changelog skill validation to verify the presence and correctness of the new Layered Capability Classification section and its guidance on adapter/runner boundary classification. Added checks for section presence, guidance on child adapter states, refinement classification, and avoidance of repeat classification patterns.
Regenerated [0.9.1] release entry from current git state (2026-08-22) to capture all commits through HEAD, including GitHub Copilot CLI runner support, Cline runner addition, skill description optimization, and AGENTS.md authoring guidance restructuring. Updated release highlight, Added/Changed/Fixed sections, and compare link to reflect the complete v0.9.1 release scope.
Updated repository guidelines, contributor expectations, and evaluation process documentation to distinguish Eval Runners, Orchestrators, and Graders while clarifying the explicit external-handoff boundary. All 21 repo-managed skill descriptions refactored for trigger-oriented activation metadata following progressive disclosure. Improved section organization in AGENTS.md for skill authoring, form handling, and dynamic defaults.
Added entry documenting the harness model resolution workflow, removal of the redundant provider field from execution-profile.json, and introduction of scripts/Get-HarnessModels.ps1 for current model discovery with Codebelt Reference verification and platform-specific filtering.
Modernized eval runner implementations across all harness types (GitHub Copilot, Codex, OpenCode, Cline) with updated contract schemas. Added scripts/Get-HarnessModels.ps1 for discovering current model selectors per harness with Codebelt Reference verification and platform-specific filtering. Improved prepare-skill-evals.ps1 to resolve Harness + Model before package generation, updated validation and conformance testing, and refined execution-profile.json to remove redundant provider field while treating model selectors as runner-native opaque strings.
Added manifest-paths.ps1 and bridge-manifest-results.ps1 for safe manifest path resolution and result artifact bridging. Updated prepare-skill-evals.ps1 with improved error handling and result collection. Enhanced generate-eval-report.ps1 for report generation and grading workflow. Expanded validate-skill-templates.ps1 with comprehensive fixture validation. Improved conformance testing in test-runner-conformance.ps1 to cover all eval runner contracts.
Update repository guidance in AGENTS.md, CONTRIBUTING.md, and README.md to explicitly clarify that the eval completion gate must be satisfied before a package is presented as successfully completed. Incomplete or unrun evaluation packages must be flagged as incomplete, not misrepresented as passing.
Update the generated README.md content in prepare-skill-evals.ps1 to reflect the clarified eval completion gate behavior. Packages generated by this script now include text clarifying that incomplete or unrun arms must be reported as such and that -CollectResults exits non-zero when the completion gate is not satisfied.
Implements deterministic queue and state management for eval workers. Adds native delegation surface with full-capability and model-lock guarantees. Each eval arm executes in an isolated harness-native worker with working-directory isolation, result capture, and fresh session constraints. Supports concurrent worker coordination with configurable capacity slots and rejection handling without eval attempt increment.
Adds native worker terminal evidence requirements and validation logic. Enhances delegation capability assessment with status tracking (supported/conditional/unsupported). Implements terminal evidence collection in orchestration tests including session tracking, model observation, directory isolation verification, and prompt fidelity checks. Updates result bridging to validate native worker evidence when required. Expands runner descriptor validation and adds comprehensive terminal evidence audit trail.
Update SKILL.md and evals.json to explicitly clarify that when the user provides 'yolo' or 'auto' in an explicit commit request, the skill should complete the workflow in the same turn after required checks pass, rather than awaiting additional approval or returning a pending plan.
Add validation assertions to ensure the git-visual-commits SKILL.md contains required phrases about completing the commit workflow in the same turn and that the plan summary is status output rather than a review request.
Update Git Operations Safeguards section to explicitly state that yolo or auto on an explicit commit request counts as approval to complete the commit workflow in the same turn, and clarifies that such approval must be attached to the same explicit commit request rather than treated as a standalone approval modifier.
Implement the Phase 1 deterministic preflight boundary that validates all pending evaluation arms before dispatching any execution. The Codex runner now includes preflighting logic; invoke-runner-owned-arms.ps1 serves as the complete Phase 1 entry point that asserts native delegation, validates every pending arm, and only then constructs the execution plan; and prepare-skill-evals.ps1 instructions now guide orchestrators to invoke this boundary first for runner-owned dispatch.
Add test cases and fixtures to validate the Phase 1 preflight boundary implementation. Tests cover runner-owned fixture setup, orchestration state management, and runner conformance to the new preflighting contract. These tests ensure that incompatible preflights are rejected before any execution commences and that native delegation assertions are enforced consistently.
Restructure eval runners to support runner-owned behavioral transport for GitHub Copilot, Codex, and OpenCode. This replaces orchestrator-owned delegation with runner-owned processes that manage their own child invocations, concurrency slots, and result capture. Implement headless child-process management via fanout-process.ps1 with slot allocation that frees immediately when any child completes, rather than waiting for the oldest. Fix Windows UTF-8 encoding issues in report generation by forcing CPython UTF-8 Mode without modifying packaged upstream Python sources. Add regression tests and validation gates to ensure the new architecture remains stable.
Update Eval Runner protocol documentation to explain the shift from orchestrator-owned to runner-owned behavioral delegation. Clarify that GitHub Copilot, Codex, and OpenCode now use runner-owned transports with their own child-process management, while orchestrator-owned envelope and record-native-result.ps1 remain available for runners that declare dispatch_owner=orchestrator. Document headless child-process spawning via CreateNoWindow and slot-allocation behavior for concurrent arms.
Update AGENTS.md, CONTRIBUTING.md, and README.md to clarify the runner-owned evaluation execution contract. Emphasize that for runners declaring dispatch_owner=runner, the external orchestrator invokes invoke-runner-owned-arms.ps1 exactly once and consumes its machine-readable summary without hand-authoring preflight, fan-out, or state bookkeeping. Clarify that runner-produced execution results must be persisted at manifest-declared paths and that response-only or reconstructed results cannot substitute for runner-provided evidence. Update guidance on result persistence and -CollectResults usage to reflect the new architecture.
Update scripts to implement and validate the runner-owned evaluation dispatch architecture. Remove OpenCode-specific sibling Task dispatch guidance from prepare-skill-evals.ps1 and update prompt generation to reflect that runners declaring dispatch_owner=runner own their own child-process management through the deterministic Phase 1 helper. Update validate-skill-templates.ps1 to enforce the new dispatch contract and remove outdated validation rules. Fix duplicate line in opencode/runner.ps1 and update runner conformance tests to align with the runner-owned dispatch model.
Implement comprehensive eval runner protocol enhancements including execution-freeze contracts, grading workflows, integrity finalization, and result interaction schemas. Extend runner-common with canonicalization utilities, JSON fingerprinting, and path normalization for deterministic evidence handling. Update all runner implementations and test suites to support the new grading and evidence lifecycle.
Update prepare-skill-evals.ps1 with improved runner resolution, multi-runner support, and deterministic package finalization. Extend validate-skill-templates.ps1 with comprehensive fixture validation and schema enforcement. Enhance generate-eval-report.ps1 to support the new grading contracts and runner-owned dispatch patterns. Align tooling with portable eval handoff protocol and multi-runner orchestration requirements.
Add evaluation test cases for dotnet-strong-name-signing skill covering signing verification, certificate handling, and configuration validation scenarios. Defines evaluation inputs and expected outcomes for validating skill effectiveness across baseline and with-skill execution paths.
Update top-level README.md with current feature set and project structure. Align repository documentation with recent skill additions and eval runner protocol enhancements.
Implement exact-session continuation for GitHub Copilot and OpenCode runners using model-free help inspection to prove installed continuation capability before execution. This enables deterministic scripted interactions across multiple turns using the same session ID rather than implicit 'last session' semantics. Adds continuation mechanism discovery via help parsing, capability verification in preflight checks, and session tracking with exact-ID resumption. Includes comprehensive test fixtures covering positive cases (exact session preserved across turns) and negative cases (unsupported versions or implicit-only continuation). All test assertions run deterministically with zero model invocations.
Add major enhancements to the OpenCode evaluation runner including projection management for isolated execution environments, cached preflight checks for improved performance, comprehensive timing instrumentation for execution metrics, and cleanup phase handling. Extend test-runner-conformance.ps1 with new conformance test infrastructure and add six new fixture files covering OpenCode help command scenarios for session resolution testing. Update prepare-skill-evals.ps1 documentation to clarify error handling requirements when Phase 1 reports incompatible status or fails. These changes collectively improve runner reliability, observability, and test coverage for the evaluation harness system.
Improve eval runner execution reliability with enhanced timeout handling, process management, concurrency controls, and comprehensive test coverage. Updates include bounded task wait utilities, async input/output handling, Phase 1 timeout allowance calculations, and expanded test coverage for orchestration and runner conformance validation.
|
|
||
| The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, and the prompt files needed to dispatch work, but it must not execute an eval prompt itself. In phase one, for every case it creates one new isolated worker for `with_skill` and another for `without_skill`, launching each from its own run directory with `repo/` as the working directory, `home/` as an isolated profile, and filesystem access confined to the run directory. It sends each worker only the matching `prompt.md` and the files already staged in that run directory. Workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after collection, the executor reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. | ||
| The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it follows `delegation.dispatch_owner`: for runner-owned dispatch it invokes the deterministic helper, whose entry boundary resolves the runner, validates `describe`, preflights every pending `run.json`, asserts native delegation for every result, and refuses to start any execution until all preflights pass; for orchestrator-owned dispatch it resolves the runner, validates `describe`, preflights each `run.json`, and uses the declared native subagent/task. A runner-owned process/thread is the single Eval Worker and model execution; no outer model worker may contain it. Orchestrator-owned workers must not invoke `runner.ps1 execute`; their transport envelope passes through `record-native-result.ps1`. Runner-owned execution results come directly from the runner and must never be synthesized, repaired, or reconstructed from assistant text. The runner launches each native session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after all available execution results are complete or failed, the executor validates and freezes the raw results, bridges them into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. | ||
|
|
There was a problem hiding this comment.
When a subscription-authenticated Codex evaluation runs, the compatible pragmatic-isolation path does not prevent the worker from reading outside its projected arm, allowing it to inspect package-level grading or sibling-arm data and contaminate an accepted blind-evaluation result.
How this was verified: The subscription path is explicitly accepted without an external filesystem sandbox while hard confinement is designated optional.
Context Used: AGENTS.md (source)
Knowledge Base Used: Skill installation and evaluation
Prompt To Fix With AI
This is a comment left during a code review.
Path: AGENTS.md
Line: 140
Comment:
**Pragmatic isolation exposes evaluation data**
When a subscription-authenticated Codex evaluation runs, the compatible pragmatic-isolation path does not prevent the worker from reading outside its projected arm, allowing it to inspect package-level grading or sibling-arm data and contaminate an accepted blind-evaluation result.
**How this was verified:** The subscription path is explicitly accepted without an external filesystem sandbox while hard confinement is designated optional.
**Context Used:** AGENTS.md ([source](https://github.com/codebeltnet/agentic/blob/main/AGENTS.md))
**Knowledge Base Used:** [Skill installation and evaluation](https://app.greptile.com/geekle/-/custom-context/knowledge-base/codebeltnet/agentic/-/docs/skill-installation-and-evaluation.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Refactored OpenCode runner and invoke-runner-owned-arms to improve timeout calculation and management. Removed session continuation functions in favor of comprehensive timeout handling with new utility functions for computing timeouts, checking remaining time, and detecting timeout-related cancellations. Extracted GetRunnerPreflightTimeoutSeconds helper to calculate the preflight timeout boundary with runner grace period, reducing duplication and improving maintainability.
Refactored test infrastructure to dynamically import runner functions from the AST instead of relying on static fixture functions. Removed session continuation parser fixture and help-text fixtures (opencode-help-*.txt) that are no longer needed. Added comprehensive timeout regression test suite with OpenCode HTTP timeout scenarios, test environment setup, and proper isolation verification. Tests verify timeout calculation accuracy and cancellation handling across various delay and timeout configurations.
Refactored OpenCode runner and invoke-runner-owned-arms to improve timeout calculation and management. Removed session continuation functions in favor of comprehensive timeout handling with new utility functions for computing timeouts, checking remaining time, and detecting timeout-related cancellations. Extracted GetRunnerPreflightTimeoutSeconds helper to calculate the preflight timeout boundary with runner grace period, reducing duplication and improving maintainability.
Fixed Phase 1 timeout calculation in generated eval packages to account for serial preflight overhead. Old calculation only considered child execution time; new calculation adds separate allowance for preflight execution across all arms, longest child execution, and orchestration grace period. Updated generated prompt guidance to provide accurate timeout expectations to users, preventing premature timeouts during preflight validation.
…into v0.9.1/evals
Preserve runner-reported terminal status exactly and record evidence validation separately. Fix requires different evidence requirements for completed vs non-success terminals to accept honest timed-out, failed, or cancelled results without forcing fabricated responses.
Verify that ledger preserves raw runner status and evidence_validation is recorded separately. Test mixed terminal outcomes (timed_out, failed, completed, cancelled) and OpenCode timeout handling.
Add infrastructure to aggregate Phase 1 orchestration outcomes and validate that all expected evaluation arms complete. Extract and refactor orchestration state inspection into reusable aggregation functions. Update fanout summary generation to track terminal, completed, failed, timed-out, cancelled, incompatible, and evidence-validation-failed counts. Include Phase 1 success test and fail-closed gate in orchestration path. Add deterministic regression test suite for Phase 1 aggregate calculations and manifest result validation for terminal non-completed states.
| $capabilities = Get-OpenCodeCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem | ||
| $mechanisms = [System.Collections.Generic.List[string]]::new() | ||
| foreach ($mechanism in @('opencode run --format json', '--auto', 'isolated OPENCODE_CONFIG_DIR', 'isolated OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'repository-owned project configuration preserved', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } | ||
| if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement'); $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } |
There was a problem hiding this comment.
When OpenCode runs without bwrap or sandbox-exec, execution continues without hard filesystem confinement but records paired_arm_visible=false and grading_material_visible=false; the shared validator trusts those values, allowing a worker that reads package-level grading or sibling-arm data to produce evaluation evidence accepted as uncontaminated.
How this was verified: The non-confined branch completes with pragmatic isolation while the terminal evidence hard-codes both visibility fields to false.
Knowledge Base Used: Skill installation and evaluation
Prompt To Fix With AI
This is a comment left during a code review.
Path: scripts/eval-runners/opencode/runner.ps1
Line: 2947
Comment:
**Pragmatic isolation asserts false exclusion**
When OpenCode runs without `bwrap` or `sandbox-exec`, execution continues without hard filesystem confinement but records `paired_arm_visible=false` and `grading_material_visible=false`; the shared validator trusts those values, allowing a worker that reads package-level grading or sibling-arm data to produce evaluation evidence accepted as uncontaminated.
**How this was verified:** The non-confined branch completes with pragmatic isolation while the terminal evidence hard-codes both visibility fields to false.
**Knowledge Base Used:** [Skill installation and evaluation](https://app.greptile.com/geekle/-/custom-context/knowledge-base/codebeltnet/agentic/-/docs/skill-installation-and-evaluation.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Introduce a bounded, idempotent Phase 1 controller surface that creates one durable background supervisor. The external orchestrator invokes this controller repeatedly while execution is running, and the supervisor alone manages fan-out, preflight, and result registration. This design ensures safe orchestrator restart boundaries and prevents concurrent supervisor processes for the same iteration.
Adapt prepare-skill-evals.ps1 and validate-skill-templates.ps1 to integrate with the new durable Phase 1 supervisor model. Updates include support for the bounded controller surface, improved timeout handling, and alignment with runner-owned phase lifecycle expectations.
Update AGENTS.md to clarify the runner-owned Phase 1 orchestration workflow, replacing the previous fan-out helper language with explicit control-runner-owned-phase1.ps1 controller invocation. Emphasize that the external orchestrator repeats the same bounded idempotent call, the durable supervisor manages preflight and fan-out internally, and Phase 1 fails closed if the supervisor dies without a terminal result. Minor updates to CONTRIBUTING.md and README.md for consistency.
Extract phase1 control logic into a shared common module (phase1-control-common.ps1) and enhance process isolation with Windows Job Object validation. Add Windows interop layer for process/job management, ownership state tracking, and lifecycle verification. Refactor control-runner-owned-phase1.ps1 to leverage shared helpers. Enhance supervise-runner-owned-phase1.ps1 ownership state detection. Add Windows Job Object test harness to test-phase1-controller-lifecycle.ps1.
Add comprehensive test coverage for nested Windows Job Object breakaway detection. Includes new test cases validating that Phase 1 controllers properly handle scenarios where breakaway is requested but forbidden by an outer job, ensuring durable supervisor independence is strictly enforced. Updates test assertions and harness property references to support enhanced validation logic.
Model-free harness probes (--version, --help, describe commands) were falling back to %WINDIR% via GetTempPath() when TEMP/TMP/USERPROFILE were stripped from the environment. Affected probes could not execute without write access to the system directory. Add Resolve-RunnerProbeTempRoot() to explicitly resolve a writable temp directory by consulting TEMP, TMP, and LOCALAPPDATA environment variables. Update New-RunnerProbeEnvironment to pin TEMP and TMP from the resolver, ensuring model-free probes always have OS scratch space without system directory fallback. Implement Get-RunnerSystemDirectorySet and Test-RunnerDirectoryWritable as supporting utilities. Add regression test (test-probe-environment.ps1) covering both the probe temp resolution and the probe-vs-eval isolation boundary. Integrate into validate-skill-templates.ps1 as a required validation gate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a harness-agnostic Eval Runner execution boundary, clarifying and formalizing how evaluation packages are prepared, executed, and reported. The main goal is to ensure that all repository automation remains model-free and deterministic, with only explicit, human-directed external orchestration allowed to invoke model-backed evaluation runs. The update adds new runner protocol tools, schemas, and adapters, and updates documentation to reflect these changes and the strict separation between preparation and execution.
Eval Runner Protocol and Execution Boundary:
scripts/eval-runners/directory containing the common Eval Runner protocol (describe,preflight,execute), schemas forexecution-profile.jsonandexecution-result.json, a deterministic fake runner (for conformance), and initial adapters for Codex and OpenCode.execution-profile.json, runner protocol tools), keeping runner selection outside ofevals/evals.jsonand maintaining compatibility with Anthropic's report formats. [1] [2]Repository Automation and Execution Separation:
AGENTS.md,CONTRIBUTING.md,README.md) to distinguish roles (Eval Runner, Eval Orchestrator, Grader, Human Reviewer) and clarify the separation between deterministic preparation and external execution. [1] [2]Evaluation and Reporting Workflow Updates:
execution-result.json), and that the deterministic bridge produces the existing result shape for grading and reporting. [1] [2]Methodology and Compatibility:
incompatibleand no fallback or substitution provided.Changelog:
0.9.1) inCHANGELOG.mdsummarizing these protocol, workflow, and documentation changes.