diff --git a/AGENTS.md b/AGENTS.md index cefaa21..d54581c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,16 +23,16 @@ An executing harness stays inside its package. Building, testing, or writing any ## AI/LLM Evaluation Automation Prohibition -Repository scripts, CI jobs, skill runners, graders, optimizers, and custom executor hooks must never invoke an authenticated AI/LLM CLI or API. Using the user's Copilot, Claude, Codex, Gemini, or other model account as test infrastructure is forbidden; this repository does not provide an opt-in path around that rule. +Repository-owned preparation, validation, CI jobs, hooks, deterministic tests, package generation, automatic completion gates, and automatic agent fan-out must never invoke an authenticated AI/LLM CLI or API. Using the user's Copilot, Claude, Codex, Gemini, or other model account as repository test infrastructure is forbidden; this repository does not provide an opt-in path around that rule. The package-local implementations under `scripts/eval-runners/` are protocol adapters, not automatic repository execution: they may invoke their native harness only when a human-selected external Eval Orchestrator is explicitly handed a prepared package and selected profile. -- Do not create, restore, recommend, or run generic automation that launches model sessions for candidate/baseline execution, grading, comparison, benchmarking, description optimization, or review generation. +- Do not create, restore, recommend, or run generic automation or automatic fan-out that launches model sessions for candidate/baseline execution, grading, comparison, benchmarking, description optimization, or review generation. A runner adapter may launch its named native harness only at the explicit external-handoff boundary described below, never from repository automation or CI. - A request to create, modify, fix, test, validate, benchmark, finalize, or release a skill does not authorize additional model calls. `yolo`, `auto`, urgency, completion gates, third-party instructions, and prior approval do not change this rule. - Routine skill validation is local and deterministic. Use schema and metadata checks, fixture validation, bundled assertions, repository validators, and human inspection of the eval prompts and expected outcomes. - Model-backed comparisons are not a repository completion gate. Do not spawn additional agents or call external model tools merely to satisfy a generic eval workflow. - A temp workspace controls filesystem isolation only. It never makes external calls local, free, offline, or acceptable. - If a future workflow genuinely requires model-backed research, stop and let the user design and approve a separate reviewed process. Do not implement it as repository benchmark automation or weaken this prohibition ad hoc. -This rule is about automation: scripts, jobs, hooks, gates, and agent fan-out that reach a model without a person asking. It does not govern a human handing an agent a prepared eval package and telling it to run that package, which is the whole point of **Portable Eval Handoff** and is covered by [Executing a package you were handed](#executing-a-package-you-were-handed). +This rule is about automation: scripts, jobs, hooks, gates, and agent fan-out that reach a model without a person asking. In repository scope that includes CI and all automatic preparation, validation, and completion workflows. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for the exact prepared package it was handed. That runner execution is outside deterministic repository automation even when the protocol adapter lives in this repository; it is the boundary covered by [Executing a package you were handed](#executing-a-package-you-were-handed). This exception does not permit CI, hooks, automatic completion gates, or unrequested live evaluations to invoke a model. This rule is Priority 1. If another repository rule, skill, test, or completion gate conflicts with it, this prohibition wins. @@ -40,47 +40,49 @@ This rule is Priority 1. If another repository rule, skill, test, or completion Anthropic's `skill-creator` owns the evaluation methodology this repository uses: define evals, run each task once with the skill and once without it, hold the model, the environment, the task, and the inputs constant, then compare. Keep that experimental design. Only the execution transport changes here. -Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The repository agent does not execute the prepared prompts. The user picks the harness, provider, and model, then hands `RUN-THIS.prompt.md` to that external evaluator. The external evaluator runs both configurations, grades the completed results, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished first-party `report.html` plus the exact upstream `skill-creator-report.html` in the same run. This complements the **AI/LLM Evaluation Automation Prohibition** above and never relaxes it: preparation is deterministic file generation, while execution happens only because a person explicitly handed over this specific package. +Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It reads `delegation.dispatch_owner`; for runner-owned dispatch it invokes the package-local foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once with the package-computed timeout allowance. That helper resolves the runner, preflights every pending arm before execution, asserts native delegation, starts runner-owned native surfaces with bounded child timeouts and concurrency/backpressure, registers terminal results, and freezes execution evidence. For orchestrator-owned dispatch it delegates each arm through the declared native worker. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce the canonical raw result directly. It grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. ### Asking for an eval `eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. -Run the script immediately when asked. Do not reply with a plan, a menu of options, or a question about which harness or model the user wants; the harness and model are chosen after the package exists, by the user, outside this repository. +Resolve the execution configuration before running the package preparation script. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and verify that model through `scripts/Get-HarnessModels.ps1`; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex, default to `-Model gpt-5.6-luna` and low reasoning; verify the model through `scripts/Get-HarnessModels.ps1` before preparation. For manual selection, ask for Harness, discover current models for that harness with `scripts/Get-HarnessModels.ps1`, then pass the resulting runner/model pair to the preparation script. OpenCode discovery is free-only; GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. ``` -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test -Runner github-copilot -Model claude-haiku-4.5 ``` `eval` with no skill named, or `eval changed`, means the whole changed set: ``` -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot -Model claude-haiku-4.5 ``` +Use `-CodebeltReference` only when the script should perform the dynamic Copilot catalog check itself and fail if `claude-haiku-4.5` is no longer present. For noninteractive direct script use, omitting both `-Runner/-Model` and `-CodebeltReference` is an error whenever a package would be generated. + ### Handing the package over -Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected agent the evaluator, grader, and report producer. That agent creates a separate isolated worker for every `with_skill` and `without_skill` run, gives each worker only its prompt and required inputs, records the results and available metrics, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. +Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected external agent the Eval Orchestrator, Grader, and report producer. The orchestrator resolves the selected Eval Runner and reads its dispatch owner. For runner-owned dispatch it invokes `invoke-runner-owned-arms.ps1` exactly once as a foreground Phase 1 command and sets the caller shell/tool timeout to at least the package-computed allowance. That command preflights every blind `with_skill` and `without_skill` arm and starts zero executions if any preflight is incompatible. For orchestrator-owned dispatch it delegates each arm to a fresh native worker. It records each terminal envelope through the deterministic runner-owned recorder where applicable, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. Hand the user that one file by its absolute path, and stop there. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. Do not list the individual prompt files, do not describe the directory layout, and do not hand back a procedure for the user to carry out by hand. A reply that ends with 26 file paths and "run both versions" has moved the work onto the user instead of doing it. -The normal path ends in the external evaluator: after all workers finish, it reads the grading key, grades each completed result using the packaged `skill-creator/agents/grader.md` guidance, writes `grading[].text`, `grading[].passed`, and `grading[].evidence`, records optional turns/token buckets/cost when the harness exposes them, runs the package adapter, and presents the first-party paired `report.html` plus Anthropic's exact `skill-creator-report.html`. If a harness cannot write back to the package, a repository session may accept the returned result objects and use `-CollectResults` as a fallback to validate them and invoke the same tools, producing `comparison.md`, `benchmark.json`, `benchmark.md`, `report.html`, and `skill-creator-report.html`. The user asked for eval results, not a second workflow decision. +The normal path ends in the external evaluator: after all workers finish, it reads the grading key, grades each completed result using the packaged `skill-creator/agents/grader.md` guidance, writes `grading[].text`, `grading[].passed`, and `grading[].evidence`, records optional turns/token buckets/cost when the harness exposes them, runs the package adapter, and presents the first-party paired `report.html` plus Anthropic's exact `skill-creator-report.html`. If the selected external process cannot write valid runner-produced execution results back into the package, the evaluation is incomplete and must fail closed. Only persisted runner-produced evidence at the manifest-declared paths can proceed to grading and reporting. The user asked for eval results, not a second workflow decision. ### Prepare, do not execute -Generate the package with the repository script rather than by hand: +Generate the package with the repository script rather than by hand after resolving the Harness + Model choice: ``` -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` -It reads `skills//evals/evals.json` and writes one directory per eval into `.bot/-workspace/iteration-/`. The grading key and result stubs stay at the eval-case level, outside the two hermetic run directories a worker actually sees: +It reads `skills//evals/evals.json` and writes one directory per eval into `.bot/-workspace/iteration-/`. The grading key and result stubs stay at the eval-case level, outside the two isolated run directories a worker actually sees: - `eval-metadata.json` — eval id and name, original prompt, expected output, assertions, required fixtures, fixture and skill hashes, and the assumptions needed to reproduce the run. This is the grading key and lives outside every run directory. - `results/` — one prefilled result stub per configuration, also outside the run directories. -- `with_skill/` — a hermetic run directory that is the worker's sandbox root. It holds `prompt.md` (the task with the effective skill instructions inlined, plus the same input context and response contract as the baseline), `run.json` (a harness-neutral contract naming only paths inside the run directory), `repo/` (the fixtures materialized as real files, which is the worker's working directory), an isolated empty `home/`, and `skill//` (the exact candidate skill revision, so nothing falls back to a globally installed copy). +- `with_skill/` — an isolated run directory that is the worker's staged root. It holds `prompt.md` (the task with the effective skill instructions inlined, plus the same input context and response contract as the baseline), `run.json` (a harness-neutral contract naming only paths inside the run directory), `repo/` (the fixtures materialized as real files, which is the worker's working directory), an isolated empty `home/`, and `skill//` (the exact candidate skill revision, so nothing falls back to a globally installed copy). - `without_skill/` — the same run directory without any `skill/` directory and with no skill instructions or mention of the skill under test. Its `repo/` is byte-identical to the with_skill one. At the iteration root it also writes `manifest.json` and `RUN-THIS.prompt.md`, the single prompt that hands the whole package to an agent of the user's choosing. The one-file path requires a harness that can create isolated workers or sessions, each launched from its run directory with `repo/` as the working directory and `home/` as an isolated profile. A plain single-context client runs one prompt file directly per fresh session instead. @@ -96,7 +98,7 @@ Adding or modifying any repo-managed skill triggers this workflow. It is not som After the final skill edit is in place, run: ``` -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ``` It resolves every repo-managed skill this branch changed, uncommitted work included, and prepares a package for each. With no skill changed it says so and exits clean, which satisfies the gate. @@ -134,13 +136,13 @@ Four things still hold while you execute: An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This is the only role-based disqualification. -The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, and the prompt files needed to dispatch work, but it must not execute an eval prompt itself. In phase one, for every case it creates one new isolated worker for `with_skill` and another for `without_skill`, launching each from its own run directory with `repo/` as the working directory, `home/` as an isolated profile, and filesystem access confined to the run directory. It sends each worker only the matching `prompt.md` and the files already staged in that run directory. Workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after collection, the executor reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. +The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it follows `delegation.dispatch_owner`: for runner-owned dispatch it invokes the foreground package-local `invoke-runner-owned-arms.ps1` command exactly once and waits for its terminal JSON summary; the helper resolves the runner, validates `describe`, preflights every pending `run.json`, asserts native delegation for every result, and refuses to start any execution until all preflights pass. A caller/tool timeout or interrupted conversation does not authorize rerunning Phase 1. If execution was interrupted and no valid `execution-freeze.json` exists, the package is incomplete and requires a fresh iteration. For orchestrator-owned dispatch it resolves the runner, validates `describe`, preflights each `run.json`, and uses the declared native subagent/task. A runner-owned process/thread is the single Eval Worker and model execution; no outer model worker may contain it. Orchestrator-owned workers must not invoke `runner.ps1 execute`; their transport envelope passes through `record-native-result.ps1`. Runner-owned execution results come directly from the runner and must never be synthesized, repaired, or reconstructed from assistant text. The runner launches each native session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, only after Phase 1 returns a successful terminal JSON summary and the immutable freeze validates, the executor bridges results into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. The candidate instructions are already inlined in the with_skill run's `prompt.md` and staged under its `skill//` directory; the orchestrator does not load or summarize them for the worker. The baseline run has no `skill/` directory and no candidate instructions, and the orchestrator must not expose the candidate skill through another route, including a globally installed copy. The generated prompt files and the baseline `run.json` also omit the skill name, eval identifiers, and configuration labels so workers receive an ordinary task rather than an announcement that they are under evaluation. -Use the same model, model version, configuration, tools, and limits for every worker. Disable persistent memory and cross-session recall. Independent runs may execute concurrently when the selected harness and the user's token budget allow it, but every run still gets a distinct context and no shared mutable workspace. +Use the same model, model version, configuration, tools, and limits for every worker. Disable persistent memory and cross-session recall. Independent runs must execute concurrently up to `min(execution-profile.json.concurrency, remaining arms)` when harness capacity permits, but every run still gets a distinct context and no shared mutable workspace. For any selected runner whose descriptor says `delegation.dispatch_owner=runner`, the external orchestrator invokes `invoke-runner-owned-arms.ps1` exactly once as the foreground Phase 1 command, with the caller shell/tool timeout set to at least the package-computed allowance. The external orchestrator must not hand-author preflight, fan-out, state, or result bookkeeping. If the foreground command is terminated before a valid final result and freeze exist, Phase 1 fails closed and requires a fresh package; no arm is restarted. Copilot task/general-purpose workers, OpenCode Task/General workers, and Codex native mechanisms remain harness capabilities only; they are not the behavioral transport for runner-owned evaluation. -`RUN-THIS.prompt.md` requires a harness that can create isolated workers or sessions. A plain single-context client can still execute an individual self-contained prompt when the user opens it directly as the first message of a fresh session, but it cannot provide the paired comparison and report contract in that same context. Partial packages still grade and report what exists; missing arms remain visibly missing. +`RUN-THIS.prompt.md` requires a selected Eval Runner that can create isolated workers or sessions. A plain single-context client can still execute an individual self-contained prompt when the user opens it directly as the first message of a fresh session, but it cannot provide the paired comparison and report contract in that same context. A selected runner that cannot satisfy a required guarantee is `incompatible`; there is no generic fallback or runner substitution. Partial package state may be inspected and reported, but the completion gate must pass before it can be presented as a completed evaluation; missing or unrun arms remain visibly incomplete. An `output` is the model's own message in full, including questions, caveats, explanations, or a refusal. Where a run invoked a tool, that tool's stdout is evidence rather than a replacement for the response. Record the full worker transcript, duration, token usage, and tool-call count when the harness exposes them; omit unavailable metrics rather than estimating them. @@ -154,17 +156,17 @@ A meaningful A/B result requires both configurations to run on the same model, t ### Result handoff -An externally produced result comes back identified by eval id, configuration (`with_skill` or `without_skill`), model and provider, and the produced output. It may also carry the transcript, duration, total tokens, tool-call count, output files, and notes. The user can hand it over as filled-in `results/*.result.json` files, or state it in chat and let the agent fill them in. +An externally produced result is valid only when the selected runner's native transport persists it at the exact manifest-declared execution-result path, identified by eval id, configuration (`with_skill` or `without_skill`), runner-native model, harness, and produced output. It may also carry the transcript, duration, total tokens, tool-call count, output files, and notes. The external evaluator may add grading fields after the bridge; it must not author or repair raw execution evidence. -Which artifact transfer happens depends on where the harness ran, and `RUN-THIS.prompt.md` tells it to close either way. A harness sharing a disk with the package writes the result files, grading, `benchmark.json`, `benchmark.md`, the first-party `report.html`, and the exact upstream `skill-creator-report.html` itself and reports the first-party report path. A harness that does not - a different product, a browser, or a sandbox - ends with one paste-ready block carrying the package path and every completed result object, including grading, plus the reports as file artifacts when supported. A repository session can use `-CollectResults` only as a fallback for transferred results that lack the report artifacts. "Bring the results back" means those artifacts, never a prose recap of how the runs went. +Which artifact transfer happens depends on where the harness ran, and `RUN-THIS.prompt.md` tells it to close either way. A harness sharing a disk with the package writes the runner-produced result files, grading, `benchmark.json`, `benchmark.md`, the first-party `report.html`, and the exact upstream `skill-creator-report.html` itself and reports the first-party report path. If the selected external process cannot write valid runner-produced execution results back into the package, the evaluation is incomplete and must fail closed; a response-only or reconstructed result cannot substitute for persisted transport evidence. -Validate and compare a collected iteration with: +An explicitly authorized forensic recovery of an old or broken package may validate an existing iteration with: ``` pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ``` -It checks that each result matches its eval and configuration, warns when an arm is missing, unrun, or ran on a different model, and writes `comparison.md`, the paired `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`. The external evaluator may grade in its user-directed phase-two context; repository automation remains deterministic and never invokes a model. Use deterministic checks for mechanical assertions and evidence-backed human or evaluator judgement only where the assertion is genuinely qualitative. +It checks that each result matches its eval and configuration, may inspect and write `comparison.md`, the paired `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md` while warning when an arm is missing, unrun, or ran on a different model. It exits non-zero when the required completion gate is not satisfied, so an incomplete or unrun package is not a completed evaluation. The external evaluator may grade in its user-directed phase-two context; repository automation remains deterministic and never invokes a model. Use deterministic checks for mechanical assertions and evidence-backed human or evaluator judgement only where the assertion is genuinely qualitative. ### Workspace isolation @@ -185,7 +187,7 @@ Every repo-managed skill must include its own `evals/evals.json` file at `skills - To compare a skill against a baseline, prepare a package with **Portable Eval Handoff** and hand `RUN-THIS.prompt.md` to the user; the repository agent never runs the prompts, while the user-directed external executor runs, grades, and reports the paired comparison - Deterministic scaffold/template skills must keep local deterministic validators as well; evals supplement validators, they do not replace them -If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` and run `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` before considering the work complete. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. +If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` and run `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` before considering the work complete. Use `-CodebeltReference` instead only after its dynamic Copilot model check passes. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. ## Git Identity @@ -193,12 +195,12 @@ Never set or override `git user.name`, `git user.email`, or `alias.bot` in the * ## Git Operations Safeguards -Agents must never automatically commit code changes or push to remote repositories. Both actions require explicit user approval: +Agents must never commit code changes or push to remote repositories without explicit user approval. A direct commit request that includes `yolo` or `auto` is explicit approval for the current commit request; it authorizes the agent to complete that commit workflow in the same turn after the required checks pass. -- **Commits**: Always request confirmation from the user before staging and committing code. Present a clear summary of changes and wait for user approval before executing the commit. +- **Commits**: Request confirmation from the user before staging and committing code unless the same explicit commit request includes `yolo` or `auto`. In that auto-approved case, present the plan as status information and continue directly to staging and committing; do not ask a second confirmation question or end with a pending plan. Required review, scope, identity, message-validation, and post-commit checks still apply. - **Remote Operations**: Do not push, pull, fetch, or interact with `origin` or any remote repository without explicit user instruction. These operations modify repository history and can cause data loss if performed unexpectedly. -**Why:** Automatic commits can pollute history with incomplete work, debugging code, or unintended changes. Unexpected remote operations can overwrite or lose commits on shared branches. Always require the user to explicitly approve these operations. +**Why:** Automatic commits can pollute history with incomplete work, debugging code, or unintended changes. Unexpected remote operations can overwrite or lose commits on shared branches. Never treat silence, urgency, or momentum as approval; `yolo` or `auto` counts as approval only when attached to the same explicit commit request. ### Commit Skill Routing @@ -315,7 +317,7 @@ Before any completion message, reread the skill instructions and the current con For script-backed workflows, creating or editing files is not enough on its own. If a skill requires deterministic maintenance or verification commands, run them before completion and report their concrete outcome. For `dotnet-docfx-digest`, `scripts/agents.cs` and `scripts/docfx.cs --build-api-model --validate-samples --verify-docfx-build` are blocking completion gates whenever the skill or task summary says they are required. -Whenever a repo-managed skill was edited, two gates apply in a fixed order. `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` runs first and prepares the eval packages for the changed skills, reporting the prompt paths. `scripts/sync-skill-install.ps1` runs last, because every other step can still change a file. Report the actual output of both; an earlier run in the same session satisfies neither. See [Eval preparation is a completion gate](#eval-preparation-is-a-completion-gate) and [Local Install Sync](#local-install-sync). +Whenever a repo-managed skill was edited, two gates apply in a fixed order. `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` (or `-CodebeltReference` after dynamic availability verification) runs first and prepares the eval packages for the changed skills, reporting the prompt paths. `scripts/sync-skill-install.ps1` runs last, because every other step can still change a file. Report the actual output of both; an earlier run in the same session satisfies neither. See [Eval preparation is a completion gate](#eval-preparation-is-a-completion-gate) and [Local Install Sync](#local-install-sync). ## User Input UX @@ -353,126 +355,106 @@ Interim progress updates should describe user-relevant progress, evidence, block - Mention tool/runtime failures only when they block progress, require approval, or change the planned validation - Prefer concise phrasing such as "The first read attempt failed before returning file content; I'm retrying and will report only if that changes the result" -## Anthropic Skill Authoring Reference +## Skill Authoring -Essential conventions from [The Complete Guide to Building Skills for Claude](https://resources.anthropic.com/hubfs/The-Complete-Guide-to-Building-Skill-for-Claude.pdf) (Anthropic, Jan 2026). All skills in this repo must follow these rules. +Skills MUST follow the Agent Skills specification and remain compatible with Anthropic's skill guidance. Repository conventions below may intentionally be stricter than the specification. -### File Structure +### Structure -``` +```text skill-name/ -├── SKILL.md # Required — exact spelling, case-sensitive -├── scripts/ # Optional — executable code (Python, Bash, etc.) -├── references/ # Optional — documentation loaded as needed -└── assets/ # Optional — templates, fonts, icons used in output +├── SKILL.md # Required +├── scripts/ # Optional executable automation +├── references/ # Optional supporting documentation +└── assets/ # Optional templates and output resources ``` -- **No `README.md`** inside the skill folder — all documentation goes in `SKILL.md` or `references/` -- Folder name must be **kebab-case** (no spaces, no underscores, no capitals) -- Folder name must match the `name:` field in YAML frontmatter +- `SKILL.md` MUST use that exact case-sensitive name. +- Skill folders MUST use kebab-case. +- Frontmatter `name` MUST match the folder name. +- Do NOT place `README.md` inside a skill folder. Put skill documentation in `SKILL.md` or `references/`. +- Keep `SKILL.md` focused. Move detailed or rarely needed material to `references/`. +- Keep `SKILL.md` below 5,000 words. -### Progressive Disclosure (Three Levels) +### Progressive Disclosure -| Level | When loaded | Token cost | Content | -|-------|------------|------------|---------| -| **Level 1: Metadata** | Always (at startup) | ~100 tokens | `name` and `description` from YAML frontmatter | -| **Level 2: Instructions** | When skill is triggered | Under 5k tokens | SKILL.md body — workflows, steps, guidance | -| **Level 3: Resources** | As needed | Effectively unlimited | Linked files: scripts, references, assets, FORMS.md | +Design every skill around three levels of context: -Keep SKILL.md under **500 lines / 5,000 words**. Move detailed content to `references/`. Keep references **one level deep** from SKILL.md — nested references cause partial reads. +1. **Metadata:** `name` and `description` are available before activation. +2. **Instructions:** the `SKILL.md` body is loaded when the skill is selected. +3. **Resources:** scripts, references, and assets are loaded or used only when needed. -### YAML Frontmatter +Minimize content at earlier levels. Do not put instructions into metadata merely to advertise skill capabilities. -Required fields: +### Frontmatter ```yaml --- -name: kebab-case-name # max 64 chars, lowercase + numbers + hyphens only -description: > # max 1024 chars, must include WHAT + WHEN + triggers - What it does. Use when user asks to [specific phrases]. +name: skill-name +description: Use when ... --- ``` -Optional fields: - -```yaml -license: MIT # for open-source skills -compatibility: > # max 500 chars — environment requirements - Requires network access and Python 3.10+ -metadata: # custom key-value pairs - author: Company Name - version: 1.0.0 - mcp-server: server-name -``` +Required: -**Forbidden**: XML angle brackets (`< >`), names containing "claude" or "anthropic" (reserved). +- `name`: 1-64 characters, lowercase alphanumeric characters and hyphens only; MUST match the skill directory. +- `description`: 1-1024 characters and MUST communicate both what the skill is for and when it should activate. -### Description Field — The Most Important Part +Optional fields such as `license`, `compatibility`, `metadata`, and supported tool restrictions MAY be used when they provide meaningful runtime or distribution information. -Structure: `[What it does] + [When to use it] + [Key capabilities]` +### Description Is a Trigger -```yaml -# ✅ Good — specific, actionable, includes triggers -description: > - Manages Linear project workflows including sprint planning, - task creation, and status tracking. Use when user mentions - "sprint", "Linear tasks", "project planning", or asks to - "create tickets". - -# ❌ Bad — too vague, no triggers -description: Helps with projects. -``` +Treat `description` as **activation metadata, not documentation**. -- Include trigger phrases users would actually say -- Mention file types if relevant -- Add negative triggers to prevent over-triggering: `Do NOT use for simple data exploration` +- SHOULD begin with trigger-oriented language such as `Use when...`. +- SHOULD describe **user intent**, not the skill's implementation. +- SHOULD stay at or below a **300-character soft ceiling**. +- MAY be shorter than 150 characters when that is sufficient. Never pad a description to meet a minimum length. +- MAY exceed 300 characters only when additional wording materially improves trigger precision or recall. +- MUST remain within the 1024-character specification limit. +- SHOULD include distinctive tasks, artifacts, technologies, file types, or domain terms that help discriminate the skill from others. +- SHOULD cover natural paraphrases conceptually rather than stuffing exact trigger phrases or keywords. +- SHOULD add exclusions only when needed to prevent realistic near-miss or overlapping skills from triggering. +- MUST NOT summarize workflows, scripts, implementation details, references, rationale, or every capability of the skill. +- MUST NOT broaden the description merely to advertise functionality. -### Writing Instructions +Prefer: -- Be **specific and actionable** — `Run scripts/validate.py --input {filename}` not `Validate the data` -- Include **error handling** — common errors, causes, and solutions -- Use **feedback loops** — run validator → fix errors → repeat -- Put **critical instructions at the top** — use `## Critical` or `## Important` headers -- For critical validations, **use scripts over language instructions** — code is deterministic -- Prefer **dynamic defaults over hardcoded values** when the source data is available from the repo, environment, or an official machine-readable feed +```yaml +description: Use when creating or refactoring .NET tests that require deterministic remote or containerized execution across supported test harnesses. +``` -### Skill Categories +Avoid: -| Category | Purpose | Example | -|----------|---------|---------| -| **Document & Asset Creation** | Consistent, high-quality output (docs, code, designs) | `frontend-design`, `docx`, `xlsx` | -| **Workflow Automation** | Multi-step processes with validation gates | `skill-creator`, scaffolding skills | -| **MCP Enhancement** | Workflow guidance layered on top of MCP tool access | `sentry-code-review` | +```yaml +description: Provides comprehensive guidance, scripts, configuration options, troubleshooting procedures, and best practices for running .NET tests remotely using containers and multiple supported test harnesses. +``` -### Common Patterns +Optimize for **trigger precision and recall per character**, not descriptive completeness. -1. **Sequential workflow** — explicit step ordering with dependencies and rollback -2. **Multi-MCP coordination** — phase separation, data passing between services -3. **Iterative refinement** — draft → validate → fix → repeat until quality threshold -4. **Context-aware selection** — decision trees for choosing the right tool/approach -5. **Domain-specific intelligence** — compliance checks, governance, audit trails +### Instructions -### Testing Checklist +Inside `SKILL.md`: -Before shipping a skill, verify: +- Make instructions specific, actionable, and ordered where sequencing matters. +- Put critical constraints near the top. +- Include error handling where failures are predictable. +- Use validation and refinement loops where output quality benefits from iteration. +- Prefer deterministic scripts for repeatable or critical validation rather than lengthy natural-language procedures. +- Prefer values discoverable from the repository, environment, or authoritative machine-readable sources over hardcoded defaults. +- Keep detailed reference material out of the main instruction path. -- [ ] Triggers on obvious tasks -- [ ] Triggers on paraphrased requests -- [ ] Does **not** trigger on unrelated topics -- [ ] Functional tests pass (correct outputs, error handling, edge cases) -- [ ] Performance improves over baseline (fewer messages, fewer errors, fewer tokens) +### Validation -Debug triggering: ask Claude `"When would you use the [skill name] skill?"` — it will quote the description back. +Before shipping or materially changing a skill, verify that it: -### Troubleshooting Quick Reference +- triggers for obvious relevant requests; +- triggers for realistic paraphrases and implicit intent; +- does not trigger for realistic near-miss requests; +- behaves correctly after activation; +- improves the intended outcome compared with not using the skill. -| Symptom | Likely cause | Fix | -|---------|-------------|-----| -| Skill won't upload | `SKILL.md` misspelled or YAML invalid | Exact case `SKILL.md`, check `---` delimiters | -| Skill never triggers | Description too vague | Add trigger phrases, mention file types | -| Skill triggers too often | Description too broad | Add negative triggers, narrow scope | -| Instructions not followed | Too verbose or ambiguous | Shorten, use bullets, move detail to `references/` | -| Slow / degraded responses | Too much content loaded | Keep SKILL.md under 5k words, use progressive disclosure | +When optimizing a description, test both **should-trigger** and **should-not-trigger** cases. Prefer measured trigger behavior over arbitrary description length, and avoid tailoring descriptions to individual evaluation phrases. ## Karpathy Rules diff --git a/CHANGELOG.md b/CHANGELOG.md index bb96609..1b63f5f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,32 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.9.1] - 2026-08-22 + +This patch release adds harness-agnostic Eval Runner execution boundary infrastructure without changing the paired evaluation methodology or existing report schemas, while optimizing skill descriptions and refactoring repository-level authoring guidance. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence. The deterministic fake runner is the conformance reference, with Codex, GitHub Copilot CLI, and OpenCode as supported real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed. + +### Added + +- `scripts/eval-runners/` with the common `describe`/`preflight`/`execute` process contract, execution-profile and execution-result schemas, deterministic fake runner, Codex adapter, GitHub Copilot CLI adapter, OpenCode adapter, runner resolution, artifact/hash validation, and bridge into the existing `eval-result/2` result shape, +- GitHub Copilot CLI as a supported Eval Runner with authentication handling (GitHub tokens, OS keychain, CLI fallback), JSONL-based event output parsing, stdin-based prompt delivery for byte fidelity, repository instruction visibility, and conformance tests covering token management and authentication source detection, +- deterministic fake-runner conformance coverage for fresh paired sessions, prompt fidelity, isolation boundaries, candidate-skill exposure, status normalization, unavailable telemetry, event warnings, artifact references, and report compatibility, +- runner-aware package preparation that reuses `run.json`, keeps runner selection outside `evals/evals.json`, and preserves Anthropic-compatible benchmark/report artifacts. + +### Changed + +- Eval preparation now resolves Harness + Model before writing `RUN-THIS.prompt.md`, removes the redundant portable `provider` field from `execution-profile.json` and result reporting, treats model selectors as runner-native opaque strings, and adds `scripts/Get-HarnessModels.ps1` for current model discovery with Codebelt Reference verification and free-only OpenCode filtering, +- `AGENTS.md`, `README.md`, and `CONTRIBUTING.md` now distinguish the Eval Runner, Eval Orchestrator, Grader, and Human Reviewer and clarify that runner execution is an explicit external-handoff boundary rather than repository automation, +- All 21 repo-managed skill descriptions refactored to lean, trigger-oriented activation metadata following progressive-disclosure principles and specification compliance, +- `AGENTS.md` Skill Authoring section restructured for clarity, brevity, and progressive disclosure of form fields, asset handling, and dynamic defaults, +- report timing output omits unavailable duration and token telemetry instead of writing zero placeholders, +- GitHub Copilot runner added to the eval runner lineup alongside existing Codex and OpenCode support, +- Runner conformance tests enhanced with additional event fixtures and isolation capability assessment. + +### Fixed + +- Codex runner now ensures the evidence directory is created before writing output files, preventing file-not-found errors, +- Test runner conformance validation now requires the output parent directory to exist upfront with explicit error reporting when the directory structure is misconfigured. + ## [0.9.0] - 2026-08-20 This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-remote-testing`, and `dotnet-segregated-assets` — replaces the repository's model-backed eval benchmark workflow with deterministic, local-only validation, and finalizes the portable eval handoff. The selected external evaluator now runs the paired workers, grades their results, and invokes Anthropic's skill-creator aggregator and eval viewer without sending the user back for a second collection command. `dotnet-test` bootstraps and modernizes xUnit test projects against Codebelt conventions with role-aware fixtures; `dotnet-remote-testing` runs .NET tests inside official Microsoft SDK containers using either an existing `testenvironments.json` or zero-config, offline-safe release discovery; and `dotnet-segregated-assets` migrates ASP.NET Core applications to an artifact-first topology where `wwwroot` stays the authoring root while deployed static content is served by a separate hardened origin. Alongside those, `git-keep-a-changelog` and `git-nuget-release-notes` gained deterministic release-entity classification, and `git-visual-commits` gained an invocation routing lock. No published skill was removed or renamed, so adopting this release is non-breaking for existing installs. @@ -612,6 +638,7 @@ This is a minor release that introduces two complementary git workflow skills, e - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output. +[0.9.1]: https://github.com/codebeltnet/agentic/compare/v0.9.0...v0.9.1 [0.9.0]: https://github.com/codebeltnet/agentic/compare/v0.8.2...v0.9.0 [0.8.2]: https://github.com/codebeltnet/agentic/compare/v0.8.1...v0.8.2 [0.8.1]: https://github.com/codebeltnet/agentic/compare/v0.8.0...v0.8.1 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 901ed93..dade638 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -89,28 +89,30 @@ Aim for 3–5 evals that cover distinct scenarios: happy path, edge cases, and c Evals are prepared, not executed, from this repository. Adding or modifying a repo-managed skill requires preparing the packages for every skill the branch touched, which is a completion gate rather than an optional extra: ```console -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot -Model claude-haiku-4.5 ``` Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, which stays last. For a single skill on demand, use: ```console -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` -The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two hermetic run directories, `with_skill/` and `without_skill/`. A run directory is the worker's sandbox root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the agent of your choice. That agent starts immediately, creates one isolated worker for every run, launches it from its run directory with `repo/` as the working directory and `home/` as an isolated profile, gives each worker only its `prompt.md` and staged files, writes the results back, grades after collection using the packaged grader guidance, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +Before running the script, choose a Harness + Model. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; OpenCode is free-only, while GitHub Copilot and Codex list all currently available models. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`; `-CodebeltReference` verifies that the model still exists and fails instead of silently substituting a different model. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. -Repository scripts, CI jobs, and the agent that prepares a package never run those prompts. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A user-selected harness handed a specific package is the executor, not the preparer; its current context orchestrates fresh workers while the workers run the prompt files. +Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. -Run both configurations on the same model, same version, and same configuration. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. +Run both configurations on the same model, same version, and same configuration. Independent arms must be dispatched concurrently up to `execution-profile.json.concurrency` when the harness permits it. For any selected runner whose descriptor says `delegation.dispatch_owner=runner`, invoke `control-runner-owned-phase1.ps1` with a bounded `WaitSeconds` value and repeat that same idempotent call while status is `running`. The durable supervisor alone invokes the internal fan-out once; never invoke `invoke-runner-owned-arms.ps1` directly, create outer native subagents/tasks, or hand-author preflight, fan-out, state, or result bookkeeping. Copilot task/general-purpose workers, OpenCode Task/General workers, and Codex native mechanisms remain harness capabilities only; they are not the behavioral transport for runner-owned evaluation. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. -Record each external result in the matching `results/*.result.json`: `model`, `provider`, `harness`, and the complete `output`; include `transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, and `tool_calls` when the harness exposes them, and the `isolation` flags the harness confirmed. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. The normal external evaluator writes `grading[].passed` and evidence, then generates the report before handing the package back. If the results were transferred without those report artifacts, validate and compare with: +For an orchestrator-owned worker, preserve its `codebeltnet/agentic/eval-native-worker-result/1` terminal envelope and pass it to `record-native-result.ps1` with the exact `run.json`, `execution-profile.json`, and manifest-declared output path. For a runner-owned worker, preserve the runner-produced `execution-result.json` directly at the exact manifest-declared path and do not invoke the recorder or synthesize an envelope. In either mode, the generated result must carry the protocol/schema, opaque run and fresh session ids, status, complete final response or explicit unavailability, runner/harness identity, requested and resolved model selection, timestamps and duration, exit/failure state, prompt/run/profile hashes, resolved isolation mechanisms, warnings, and artifact references. Include token, cache, cost, tool, command, file, and transcript evidence only when the harness exposes it; unavailable values remain explicitly unavailable and are never estimated. The deterministic bridge then writes the existing `results/*.result.json` shape, after which grading may add `grading[].passed` and evidence. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. If the selected external process cannot write valid runner-produced execution results back into the package, the evaluation is incomplete and must fail closed; no response-only or reconstructed result is accepted. + +An explicitly authorized forensic recovery of an old or broken package may validate an existing iteration with: ```console pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ``` -That writes `comparison.md`, the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`, while flagging missing arms, unrun configurations, and mixed models. The normal external evaluator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. +It may write a diagnostic `comparison.md` while flagging missing arms, unrun configurations, incompatible evidence, and mixed models, but it exits non-zero and does not write benchmark/report artifacts until the required completion gate is satisfied. Those diagnostic artifacts must not present an incomplete or unrun package as a successfully completed evaluation. The normal external Eval Orchestrator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. Codex, OpenCode, and GitHub Copilot are the conforming real runners; the deterministic fake runner is the CI conformance harness. Hard filesystem confinement is reported as strict versus pragmatic confidence and is not a universal Windows prerequisite. Freebuff remains planned/blocked until its official CLI provides a supported noninteractive machine-readable transport. Native skill activation, portability scoring, and additional runners are not part of v0.9.1. The eval package is a temp artifact. Do not commit it, its prompts, or its results unless the change explicitly calls for checked-in examples. @@ -147,8 +149,8 @@ pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD - [ ] At least one eval in `evals/evals.json` - [ ] The skill's `evals/evals.json` exists and its `skill_name` matches the folder/frontmatter name - [ ] Any optional `files` entries in `evals/evals.json` point to real fixture files under the same skill folder -- [ ] `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` was run after the last skill edit, and the prepared prompt paths were reported -- [ ] If an external evaluation was run, each result includes the producing model and the package contains the first-party `report.html`, exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md`; use `-CollectResults` only when transferred results need the repository-side fallback +- [ ] `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` or `-CodebeltReference` was run after the last skill edit, and the prepared prompt paths were reported +- [ ] If an external evaluation was run, each result includes the producing model and the package contains the first-party `report.html`, exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md`; use `-CollectResults` only for explicitly authorized forensic recovery of an existing package - [ ] `scripts/validate-skill-templates.ps1` passes for the current working tree when changing scaffold or template behavior - [ ] If CI is enabled for the branch, the GitHub Actions validation job passes too - [ ] Eval packages live in `.bot/-workspace/` or a temp path, never anywhere else in the working tree diff --git a/README.md b/README.md index dabf8f4..7d204d3 100644 --- a/README.md +++ b/README.md @@ -12,17 +12,21 @@ One repo-wide convention matters especially for scaffolding skills: prefer dynam Another repo rule is intentionally strict: every repo-managed skill ships with its own `evals/evals.json`. These files are versioned review specifications whose prompts, fixtures, and expected outcomes are validated locally; they are not instructions to launch model sessions. -Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository scripts, CI jobs, runners, graders, optimizers, and custom hooks from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. Model-backed candidate/baseline fan-out is not a completion gate. +Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` workflow and replaces only its execution transport. The repository prepares the paired candidate and baseline inputs as a portable package and stops. The agent chosen by the user later executes the package, grades the completed results with the packaged grader guidance, and invokes the packaged Anthropic aggregator and eval viewer. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow asks for Harness + Model before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and its availability is verified against the current Copilot model catalog before automatic selection. OpenCode discovery lists only currently free models; GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Hard filesystem confinement raises reported isolation confidence from pragmatic to strict but is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Freebuff remains planned/blocked until it exposes a supported noninteractive machine-readable transport. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: + +Phase 1 closes by writing an immutable `execution-freeze.json` ledger with the exact manifest result paths and hashes of every runner-produced execution result and referenced raw transcript/event artifact. The bridge, grading application, and report adapter validate that ledger and never re-bless changed bytes. The external Grader writes only package-root `grading.json` (`codebeltnet/agentic/eval-grading/1`); `apply-eval-grading.ps1` projects only `passed` and `evidence` decisions onto canonical results. `finalize-eval-package.ps1` owns the deterministic completion boundary and succeeds only after validating the freeze, bridge, complete grading, and all four report artifacts. A changed raw file requires a fresh Phase 1 execution, and prose cannot substitute for finalizer success. Optional scripted `interaction.json` sidecars provide deterministic same-session user turns only when the selected runner advertises and preflights that capability; ordinary single-turn runs remain unchanged. ```powershell -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot -Model claude-haiku-4.5 ``` -That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. +That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand, but execution selection must already be resolved; direct noninteractive use without `-Runner/-Model` or `-CodebeltReference` fails before a handoff is generated. `scripts/Get-HarnessModels.ps1 -Runner ` lists current model selectors for the selected harness. Use `-CodebeltReference` only when you want the script to verify the current Copilot catalog and select GitHub Copilot CLI + `claude-haiku-4.5`; if that model is absent, the script fails and prints the current choices rather than substituting another model. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. + +Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile and selected runner descriptor. For runner-owned dispatch, it invokes the foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once and sets the caller shell/tool timeout to at least the package-computed allowance. That helper owns the long-running preflight/fan-out/freeze implementation: it preflights every arm before execution, starts zero model executions when any preflight is incompatible, honors runner concurrency/backpressure, applies bounded child-process timeouts, registers terminal runner-produced evidence, and writes the immutable `execution-freeze.json` before grading. A caller timeout or interrupted conversation is not permission to rerun Phase 1; without a valid freeze the iteration is incomplete and requires a fresh package. For orchestrator-owned dispatch, the orchestrator uses the declared native worker transport. It preserves transport-owned raw evidence and uses `record-native-result.ps1` only for orchestrator-owned envelopes. After a successful Phase 1 freeze, the Grader writes only the grading artifact, then the deterministic application helper and finalizer perform the bridge, canonical projection, upstream aggregation/viewer compatibility, and first-party report generation. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. If the selected external process cannot write valid runner-produced execution results at the manifest-declared paths, the evaluation is incomplete and fails closed; no response-only or reconstructed result is accepted. Packages land in gitignored `.bot/` storage by default and are not committed. -Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two hermetic run directories. `with_skill/` is a self-contained sandbox root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so a worker confined to its run directory never sees them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the evaluator, grader, and report producer: it creates one fresh isolated worker per run, launches it from the run directory with `repo/` as the working directory and `home/` as the profile, records the complete response plus transcript, duration, token usage, optional turns/token buckets/cost, tool calls, and isolation guarantees, grades after collection with the packaged `skill-creator` guidance, then invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. The package guarantees identical repositories, skill-only-in-with_skill staging, and an isolated home; the harness must supply the runtime sandbox that keeps global skills, global config, and the source repository out of reach. A harness that can create isolated workers handles the complete run from that one file. `-CollectResults ` remains a fallback for transferred results without report artifacts; it validates the arms and invokes the same packaged tools. Packages land in gitignored `.bot/` storage by default and are not committed. +The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. One more consistency rule matters for form-driven skills: native input fields are treated as a host feature, not something a model can rely on. Skills in this repo must stay usable with or without UI widgets, and must fall back to the same deterministic one-field-at-a-time flow when the host only supports plain chat. @@ -121,6 +125,8 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith ## Available Skills +Each `SKILL.md` description is lean activation metadata. The catalog below explains what happens after a skill is selected. + | Skill | Description | |-------|-------------| | [git-visual-commits](skills/git-visual-commits/SKILL.md) | AI-driven git commit workflow with authoritative routing for `git bot commit`, `git commit`, and `git our commit`, including the exact `Please do a git bot commit yolo` form. It locks the requested identity, treats yolo/auto only as scoped auto-approval modifiers, never as the commit message, and does not hand commit execution to changelog or release-note skills. It uses deterministically validated emoji-first subjects, optional conventional prefixes only on explicit request, full-worktree semantic grouping unless narrowed, a visible multi-file single-category quality gate, commit bodies by default, and post-commit identity/body verification. Multi-file plans that initially collapse to one category also require a visible full-context quality gate; one-file changes keep the fast path. Stack-agnostic. | diff --git a/scripts/Get-HarnessModels.ps1 b/scripts/Get-HarnessModels.ps1 new file mode 100644 index 0000000..ede25c8 --- /dev/null +++ b/scripts/Get-HarnessModels.ps1 @@ -0,0 +1,440 @@ +<# +.SYNOPSIS + Lists current model selectors for a supported eval harness. + +.DESCRIPTION + Discovers runner-native model selectors without executing model requests. GitHub Copilot and Codex return every + model the harness exposes. OpenCode returns only models whose current catalog metadata proves free + availability. Discovery failures are local to the selected harness and never fall back to stale hardcoded catalogs. + +.PARAMETER Runner + Internal Eval Runner id: github-copilot, codex, or opencode. + +.PARAMETER CatalogPath + Optional deterministic catalog fixture used by tests. When supplied, no harness command is invoked. + +.PARAMETER RequireModel + Optional model selector that must exist in the discovered list. + +.PARAMETER Refresh + For harnesses that support explicit catalog refresh, request a refresh before listing. This never executes a model + prompt. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] + [ValidateSet('github-copilot', 'codex', 'opencode')] + [string]$Runner, + + [string]$CatalogPath, + + [string]$RequireModel, + + [switch]$Refresh +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerCommon = Join-Path $PSScriptRoot 'eval-runners/runner-common.ps1' +. $runnerCommon + +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) + +function Get-HarnessDisplayName { + param([Parameter(Mandatory = $true)][string]$RunnerName) + + switch ($RunnerName) { + 'github-copilot' { return 'GitHub Copilot CLI' } + 'codex' { return 'Codex CLI' } + 'opencode' { return 'OpenCode' } + default { return $RunnerName } + } +} + +function Get-PolicyName { + param([Parameter(Mandatory = $true)][string]$RunnerName) + + if ($RunnerName -eq 'opencode') { + return 'free' + } + return 'all' +} + +function Read-CatalogJson { + param([Parameter(Mandatory = $true)][string]$Path) + + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { + throw "Catalog fixture '$Path' does not exist." + } + return [System.IO.File]::ReadAllText((Resolve-Path -LiteralPath $Path).Path, $utf8NoBom) | ConvertFrom-Json +} + +function Get-FirstPropertyValue { + param( + [object]$Object, + [string[]]$Names + ) + + foreach ($name in $Names) { + $value = Get-JsonProperty -Object $Object -Name $name -Default $null + if ($null -ne $value -and -not [string]::IsNullOrWhiteSpace([string]$value)) { + return [string]$value + } + } + return $null +} + +function Get-NumericPropertyValues { + param([object]$Object) + + $values = [System.Collections.Generic.List[double]]::new() + if ($null -eq $Object) { + return @() + } + + if ($Object -is [System.Collections.IEnumerable] -and -not ($Object -is [string]) -and -not ($Object -is [System.Collections.IDictionary]) -and -not ($Object -is [pscustomobject])) { + foreach ($item in $Object) { + foreach ($child in @(Get-NumericPropertyValues -Object $item)) { $values.Add([double]$child) } + } + return @($values) + } + + if ($Object -is [System.Collections.IDictionary]) { + $propertyNames = @($Object.Keys) + foreach ($name in $propertyNames) { + $value = $Object[$name] + if ($value -is [int] -or $value -is [long] -or $value -is [double] -or $value -is [decimal]) { + $values.Add([double]$value) + } elseif ($null -ne $value -and -not ($value -is [string]) -and -not ($value -is [System.ValueType]) -and ($value -is [System.Collections.IDictionary] -or $value -is [pscustomobject] -or $value -is [System.Collections.IEnumerable])) { + foreach ($child in @(Get-NumericPropertyValues -Object $value)) { $values.Add([double]$child) } + } + } + return @($values) + } + + foreach ($property in @($Object.PSObject.Properties)) { + $value = $property.Value + if ($value -is [int] -or $value -is [long] -or $value -is [double] -or $value -is [decimal]) { + $values.Add([double]$value) + } elseif ($null -ne $value -and -not ($value -is [string]) -and -not ($value -is [System.ValueType]) -and ($value -is [System.Collections.IDictionary] -or $value -is [pscustomobject] -or $value -is [System.Collections.IEnumerable])) { + foreach ($child in @(Get-NumericPropertyValues -Object $value)) { $values.Add([double]$child) } + } + } + return @($values) +} + +function Get-ModelAvailability { + param([object]$Model) + + $explicit = Get-FirstPropertyValue -Object $Model -Names @('availability', 'billing', 'usageCostDisplay') + if (-not [string]::IsNullOrWhiteSpace($explicit)) { + $normalized = $explicit.ToLowerInvariant() + if ($normalized -eq 'free') { return 'free' } + if ($normalized -in @('paid', 'subscription', 'metered')) { return 'paid' } + } + + foreach ($name in @('free', 'isFree')) { + $value = Get-JsonProperty -Object $Model -Name $name -Default $null + if ($null -eq $value) { + continue + } + if ([bool]$value) { return 'free' } + return 'paid' + } + + foreach ($propertyName in @('pricing', 'cost')) { + $price = Get-JsonProperty -Object $Model -Name $propertyName -Default $null + $numbers = @(Get-NumericPropertyValues -Object $price) + if ($numbers.Count -gt 0) { + if (@($numbers | Where-Object { [double]$_ -gt 0 }).Count -gt 0) { + return 'paid' + } + return 'free' + } + } + + return 'unknown' +} + +function Test-TextModel { + param([object]$Model) + + $operation = [string](Get-JsonProperty -Object $Model -Name 'operation' -Default '') + if (-not [string]::IsNullOrWhiteSpace($operation) -and $operation -notin @('language', 'chat', 'completion')) { + return $false + } + return $true +} + +function ConvertTo-ModelChoice { + param( + [Parameter(Mandatory = $true)][object]$Model, + [Parameter(Mandatory = $true)][string]$RunnerName, + [string]$Source, + [string]$ExplicitSelector + ) + + if (-not (Test-TextModel -Model $Model)) { + return $null + } + + $id = if ([string]::IsNullOrWhiteSpace($ExplicitSelector)) { + Get-FirstPropertyValue -Object $Model -Names @('id', 'slug', 'model', 'modelId', 'model_id') + } else { + $ExplicitSelector + } + if ([string]::IsNullOrWhiteSpace($id)) { + return $null + } + + $provider = Get-FirstPropertyValue -Object $Model -Names @('providerID', 'providerId', 'provider') + if ($RunnerName -eq 'opencode' -and $id -notmatch '/' -and -not [string]::IsNullOrWhiteSpace($provider)) { + $id = "$provider/$id" + } + + $displayName = Get-FirstPropertyValue -Object $Model -Names @('display_name', 'displayName', 'name') + if ([string]::IsNullOrWhiteSpace($displayName)) { + $displayName = $id + } + + return [pscustomobject][ordered]@{ + id = $id + display_name = $displayName + availability = Get-ModelAvailability -Model $Model + source = $Source + } +} + +function ConvertTo-ModelChoices { + param( + [Parameter(Mandatory = $true)][object]$Catalog, + [Parameter(Mandatory = $true)][string]$RunnerName, + [Parameter(Mandatory = $true)][string]$Source + ) + + $rawModels = [System.Collections.Generic.List[object]]::new() + if ($Catalog -is [array]) { + foreach ($model in $Catalog) { $rawModels.Add($model) } + } elseif (Test-JsonProperty -Object $Catalog -Name 'models') { + foreach ($model in @($Catalog.models)) { $rawModels.Add($model) } + } else { + foreach ($propertyName in @(Get-JsonPropertyNames -Object $Catalog)) { + $value = Get-JsonProperty -Object $Catalog -Name $propertyName -Default $null + if ($null -ne $value -and @($value.PSObject.Properties).Count -gt 0) { + if (-not (Test-JsonProperty -Object $value -Name 'id')) { + $value | Add-Member -NotePropertyName id -NotePropertyValue $propertyName -Force + } + $rawModels.Add($value) + } + } + } + + $choices = [System.Collections.Generic.List[object]]::new() + foreach ($model in $rawModels) { + $choice = ConvertTo-ModelChoice -Model $model -RunnerName $RunnerName -Source $Source + if ($null -ne $choice) { + $choices.Add($choice) + } + } + $seen = @{} + $deduped = [System.Collections.Generic.List[object]]::new() + foreach ($choice in @($choices | Sort-Object id)) { + if (-not $seen.ContainsKey([string]$choice.id)) { + $seen[[string]$choice.id] = $true + $deduped.Add($choice) + } + } + return @($deduped) +} + +function Select-ModelsByPolicy { + param( + [object[]]$Models, + [Parameter(Mandatory = $true)][string]$RunnerName + ) + + $policy = Get-PolicyName -RunnerName $RunnerName + if ($policy -eq 'free') { + $freeModels = @($Models | Where-Object { [string]$_.availability -eq 'free' }) + if ($freeModels.Count -eq 0) { + throw "No free $((Get-HarnessDisplayName -RunnerName $RunnerName)) models are currently available from discovery. Choose another harness or update the harness catalog." + } + return @($freeModels) + } + + return @($Models) +} + +function Invoke-JsonCommand { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [int]$TimeoutSeconds = 60 + ) + + $work = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-model-discovery-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $work -Force | Out-Null + try { + $environment = New-RunnerProbeEnvironment + foreach ($name in @('HOME', 'USERPROFILE', 'APPDATA', 'LOCALAPPDATA', 'XDG_CONFIG_HOME')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + $process = Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList (@($CommandInfo.Prefix) + @($Arguments)) -WorkingDirectory $work -Environment $environment -TimeoutSeconds $TimeoutSeconds + if ($process.TimedOut) { + throw "Command '$($CommandInfo.Source)' timed out during model discovery." + } + if ($process.ExitCode -ne 0) { + $detail = [string]::Join("`n", @($process.Stdout, $process.Stderr)).Trim() + throw "Command '$($CommandInfo.Source) $($Arguments -join ' ')' failed during model discovery with exit code $($process.ExitCode). $detail" + } + return [pscustomobject]@{ Stdout = $process.Stdout; Stderr = $process.Stderr } + } finally { + if (Test-Path -LiteralPath $work) { + Remove-Item -LiteralPath $work -Recurse -Force -ErrorAction SilentlyContinue + } + } +} + +function ConvertFrom-OpenCodeTextCatalog { + param([Parameter(Mandatory = $true)][string]$Text) + + $models = [System.Collections.Generic.List[object]]::new() + $lines = $Text -split "`r?`n" + $currentSelector = $null + $buffer = [System.Collections.Generic.List[string]]::new() + $depth = 0 + foreach ($line in $lines) { + $trimmed = $line.Trim() + if ([string]::IsNullOrWhiteSpace($trimmed)) { + continue + } + if ($depth -eq 0 -and $trimmed -match '^[^\s/]+/.+$') { + $currentSelector = $trimmed + continue + } + if ($trimmed.StartsWith('{') -or $depth -gt 0) { + $buffer.Add($line) + $depth += ([regex]::Matches($line, '\{')).Count + $depth -= ([regex]::Matches($line, '\}')).Count + if ($depth -le 0 -and $buffer.Count -gt 0) { + $json = [string]::Join("`n", @($buffer)) + $object = $json | ConvertFrom-Json + $choice = ConvertTo-ModelChoice -Model $object -RunnerName 'opencode' -Source 'opencode models --verbose' -ExplicitSelector $currentSelector + if ($null -ne $choice) { $models.Add($choice) } + $buffer.Clear() + $depth = 0 + $currentSelector = $null + } + } + } + return @($models) +} + +function Resolve-CopilotSdkPath { + $command = Resolve-ExternalCommand -Name 'copilot' + if ($null -eq $command) { + throw 'GitHub Copilot CLI executable is not available on PATH.' + } + + $source = [string]$command.Source + $directory = Split-Path -Parent $source + $candidates = [System.Collections.Generic.List[string]]::new() + $optionalRoot = Join-Path $directory 'node_modules/@github/copilot/node_modules/@github' + if (Test-Path -LiteralPath $optionalRoot -PathType Container) { + foreach ($package in @(Get-ChildItem -LiteralPath $optionalRoot -Directory -Filter 'copilot-*' -Force)) { + $candidates.Add((Join-Path $package.FullName 'sdk/index.js')) + } + } + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-win32-x64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-win32-arm64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-linux-x64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-linux-arm64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-darwin-x64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-darwin-arm64/sdk/index.js')) + foreach ($candidate in $candidates) { + $full = [System.IO.Path]::GetFullPath($candidate) + if (Test-Path -LiteralPath $full -PathType Leaf) { + return $full + } + } + + throw 'GitHub Copilot CLI does not expose a package-local SDK model listing surface in this installation.' +} + +function Get-CodexModels { + $command = Resolve-ExternalCommand -Name 'codex' + if ($null -eq $command) { + throw 'Codex CLI executable is not available on PATH.' + } + $result = Invoke-JsonCommand -CommandInfo $command -Arguments @('debug', 'models') -TimeoutSeconds 60 + $catalog = $result.Stdout | ConvertFrom-Json + return ConvertTo-ModelChoices -Catalog $catalog -RunnerName 'codex' -Source 'codex debug models' +} + +function Get-OpenCodeModels { + $command = Resolve-ExternalCommand -Name 'opencode' + if ($null -eq $command) { + throw 'OpenCode CLI executable is not available on PATH.' + } + $arguments = @('models', 'opencode', '--verbose') + if ($Refresh) { $arguments += '--refresh' } + $result = Invoke-JsonCommand -CommandInfo $command -Arguments $arguments -TimeoutSeconds 180 + return ConvertFrom-OpenCodeTextCatalog -Text $result.Stdout +} + +function Get-CopilotModels { + $sdkPath = Resolve-CopilotSdkPath + $node = Resolve-ExternalCommand -Name 'node' + if ($null -eq $node) { + throw 'Node.js is required to read the GitHub Copilot CLI model registry.' + } + + $script = @' +import { pathToFileURL } from "node:url"; +const sdkPath = process.argv[1]; +const mod = await import(pathToFileURL(sdkPath).href); +const ids = Array.isArray(mod.HELP_VISIBLE_MODELS) + ? mod.HELP_VISIBLE_MODELS + : Object.keys(mod.SUPPORTED_MODELS || {}); +console.log(JSON.stringify({ models: ids.map((id) => ({ id, name: id, operation: "language" })) })); +'@ + $result = Invoke-JsonCommand -CommandInfo $node -Arguments @('--input-type=module', '-e', $script, $sdkPath) -TimeoutSeconds 90 + $catalog = $result.Stdout | ConvertFrom-Json + return ConvertTo-ModelChoices -Catalog $catalog -RunnerName 'github-copilot' -Source 'GitHub Copilot CLI help-visible model catalog' +} + +try { + $rawModels = if (-not [string]::IsNullOrWhiteSpace($CatalogPath)) { + ConvertTo-ModelChoices -Catalog (Read-CatalogJson -Path $CatalogPath) -RunnerName $Runner -Source $CatalogPath + } else { + switch ($Runner) { + 'github-copilot' { Get-CopilotModels } + 'codex' { Get-CodexModels } + 'opencode' { Get-OpenCodeModels } + } + } + $models = @(Select-ModelsByPolicy -Models @($rawModels) -RunnerName $Runner) + if ($models.Count -eq 0) { + throw "No models were returned for $((Get-HarnessDisplayName -RunnerName $Runner))." + } + + if (-not [string]::IsNullOrWhiteSpace($RequireModel) -and @($models | Where-Object { [string]$_.id -eq $RequireModel }).Count -eq 0) { + $available = [string]::Join(', ', @($models | Select-Object -ExpandProperty id)) + throw "Required model '$RequireModel' was not returned by current $((Get-HarnessDisplayName -RunnerName $Runner)) discovery. Available models: $available" + } + + [ordered]@{ + schema = 'codebeltnet/agentic/harness-models/1' + runner = $Runner + harness = Get-HarnessDisplayName -RunnerName $Runner + policy = Get-PolicyName -RunnerName $Runner + models = @($models) + } | ConvertTo-Json -Depth 20 +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-report-template.html b/scripts/eval-report-template.html index e5b6588..4881def 100644 --- a/scripts/eval-report-template.html +++ b/scripts/eval-report-template.html @@ -165,7 +165,7 @@ if (!run) return '
' + runLabel(config) + '
Run not recorded.
'; const summary = gradeSummary(run); const rate = passRate(run); - const meta = [run.model, run.provider, run.harness].filter(Boolean).join(' · ') || 'Runtime details not recorded'; + const meta = [run.model, run.harness].filter(Boolean).join(' · ') || 'Runtime details not recorded'; let html = '
' + runLabel(config) + '' + esc(meta) + '
'; html += '
'; html += metric('Turns', run.metrics.turns, config); @@ -303,7 +303,7 @@ document.getElementById('eval-select').addEventListener('change', e => { current = Number(e.target.value); renderOutputs(); }); document.addEventListener('keydown', e => { if (e.target.tagName === 'TEXTAREA') return; if (e.key === 'ArrowLeft') document.getElementById('prev').click(); if (e.key === 'ArrowRight') document.getElementById('next').click(); }); document.getElementById('skill-name').textContent = DATA.skill_name || 'skill'; - document.getElementById('run-meta').textContent = [DATA.metadata && DATA.metadata.model, DATA.metadata && DATA.metadata.provider, DATA.metadata && DATA.metadata.completed_runs + '/' + DATA.metadata.expected_runs + ' runs'].filter(Boolean).join(' · '); + document.getElementById('run-meta').textContent = [DATA.metadata && DATA.metadata.model, DATA.metadata && DATA.metadata.completed_runs + '/' + DATA.metadata.expected_runs + ' runs'].filter(Boolean).join(' · '); for (let i = 0; i < (DATA.evals || []).length; i++) { const option = document.createElement('option'); option.value = String(i); option.textContent = (i + 1) + '. ' + DATA.evals[i].name; document.getElementById('eval-select').appendChild(option); } renderOutputs(); renderBenchmark(); renderSkill(); diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md new file mode 100644 index 0000000..dc21eae --- /dev/null +++ b/scripts/eval-runners/README.md @@ -0,0 +1,304 @@ +# Eval Runner protocol + +## Native worker orchestration + +The external handoff uses this topology: + +```text +Eval Orchestrator + | + +-- Eval Worker -> one eval arm + +-- Eval Worker -> one eval arm + +-- Eval Worker -> one eval arm + +-- ... +``` + +The Eval Orchestrator observes the phase boundary and does not execute an eval +arm in its own model context. Each descriptor declares +`delegation.dispatch_owner`: `orchestrator` means the orchestrator creates the +declared native subagent/task, while `runner` means the orchestrator starts the +runner-owned native execution surface directly. One arm equals one fresh +native Eval Worker and one model-backed execution. A runner-owned process or +thread is that worker; it must not be nested inside an outer model session. + +`orchestration.ps1` is the deterministic queue/state helper copied into every +package. It creates one worker envelope per exact manifest arm, keeps unrelated +arms dependency-free, exposes at most the requested +`execution-profile.json.concurrency` active slots, and leaves a capacity +rejection pending without incrementing the eval attempt count. Independent arms +must use at least two active slots when the requested concurrency and harness +capacity permit it. `Assert-OrchestrationConcurrency` rejects a serial run that +has no explicit capacity-limit evidence; `bridge-manifest-results.ps1 +-RequireParallelDispatch` applies that gate before completion. It contains no +harness-specific concurrency ceiling. `Assert-NativeWorkerDelegation` is the +fail-closed handoff gate: an unavailable/unsupported native mechanism cannot +fall back to parent execution, while a conditional mechanism may dispatch only +when terminal evidence will be checked. The dispatch owner is part of the +same descriptor/preflight contract, so generic orchestration does not infer it +from a runner name. + +For `delegation.dispatch_owner=runner`, the external handoff invokes +`invoke-runner-owned-arms.ps1` exactly once as a foreground Phase 1 command and +sets the caller shell/tool timeout to at least the package-computed allowance. +The helper returns one terminal JSON summary. A caller/tool timeout or +interrupted conversation is not permission to invoke Phase 1 again; if execution +was interrupted and no valid `execution-freeze.json` exists, the package is +incomplete and requires a fresh iteration. + +The foreground fan-out reads the manifest/profile, resolves the runner and descriptor, requires +`delegation.dispatch_owner=runner`, preflights every pending manifest run, and +invokes `Assert-NativeWorkerDelegation` for every preflight result. Any +incompatible preflight produces a concise machine-readable summary, starts zero +`execute` processes, and exits non-zero. Preflight has its own deterministic +120-second model-free capability-probe timeout; it does not inherit the model +execution timeout. Only after every preflight passes does +it use the exact orchestration-plan worker IDs and manifest-declared +execution-result paths, start runner-owned `execute` processes concurrently, +redirect each process's single JSON stdout directly to its declared path, +register the runner-produced session/result once, persist +`orchestration-state.json`, and enforce the parallel-dispatch gate. Child +processes are headless isolation boundaries: they start through +`fanout-process.ps1` with `CreateNoWindow` so no per-child console window +appears on Windows, and a freed slot is refilled as soon as ANY child completes +(not only the oldest), so a slow eval execution never blocks a faster sibling. +The foreground helper is not called by preparation, validation, CI, hooks, or +automatic completion gates. +Behavioral evaluation for GitHub Copilot, Codex, and OpenCode is runner-owned: +each declares `delegation.dispatch_owner=runner`, is driven by +the one foreground Phase 1 fan-out, and produces its own terminal +`execution-result.json`. The orchestrator-owned envelope and +`record-native-result.ps1` remain available for any runner that still declares +`dispatch_owner=orchestrator` (for example the deterministic conformance fake). + +Phase 1 closes with `execution-freeze.json`. The shared freeze records the exact +manifest arm paths, runner/harness/model/session identity, terminal status, and +SHA-256 hashes for every raw `execution-result.json` and every referenced +transcript/event artifact. `bridge-manifest-results.ps1`, grading, and reporting +must validate that ledger; none of them can replace it or bless changed bytes. +If a raw result or referenced artifact changes, the package is corrupted and +requires a fresh Phase 1 execution. + +The normal post-execution boundary is deterministic: the external Grader writes +only the package-root `grading.json` artifact (`codebeltnet/agentic/eval-grading/1`) +with exact assertion identities and `passed`/`evidence` decisions. The shared +`apply-eval-grading.ps1` helper projects those decisions onto canonical +`result.json` files and verifies that every non-grading field is unchanged. +`finalize-eval-package.ps1` then validates the freeze, bridge, canonical results, +and complete grading, invokes the existing report adapter, and fails unless +`report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` +are all non-empty. A prose success message cannot substitute for its JSON +success summary. + +The delegation contract has three distinct evidence levels: + +```text +descriptor advertised harness capability +preflight locally observable readiness +terminal evidence proof for the actual delegated Eval Worker +``` + +Descriptor fields describe a possible native mechanism; they do not prove an +individual child. Preflight may prove that the installed API, plugin, or CLI +surface is present, but child-specific model, cwd, HOME/config, fresh-session, +prompt, exclusion, and result facts remain `conditional` until terminal +evidence arrives. A worker is accepted only when `evidence.delegation` proves +the requested model, exact arm identity, exact run working directory, exact +isolated home/config boundary, prompt hash/fidelity, terminal capture, paired +arm/grading exclusion, fresh worker/session identity, and exactly one model +execution. Missing or mismatched evidence makes the arm `incompatible`; it is +never a reason to invoke the parent or a different transport. + +For `dispatch_owner=orchestrator`, the harness-native transport returns a +terminal envelope with schema `codebeltnet/agentic/eval-native-worker-result/1`. +The envelope declares `capture.source = harness_native_transport`, +`capture.terminal = true`, and `capture.worker_authored = false`; the model +worker's answer is data inside the envelope, never its author. The +orchestrator preserves that envelope and invokes `record-native-result.ps1`. +For `dispatch_owner=runner`, the runner's one-arm native execution surface +produces the canonical `execution-result.json` directly; the orchestrator does +not invoke the recorder, manufacture an envelope, or copy assistant text into +transport evidence. In both modes, transport-owned timestamps, identity, +isolation observations, prompt fidelity, terminal completion, and a hashed raw +transcript/event artifact are mandatory. A parent-created summary or repaired +result is incompatible. Native bridging also checks the result's runner +identity, the descriptor's exact delegation mechanism, and the hashed artifact. +An `incompatible` arm is diagnostic-only: it is never gradeable and fails the +completion/benchmark gate. + +The descriptor's `delegation` object records the dispatch owner, native mechanism, worker role, +advertised full-capability/model-lock/working-directory/result-capture +properties, harness-authoritative capacity, and the invariant +`nested_model_execution = false`. The direct `execute` process surface is the +runner-owned native worker surface when `dispatch_owner=runner`; for +orchestrator-owned runners it remains a compatibility/conformance surface and +must not be invoked inside the native subagent. + +Native delegation mechanisms: + +- GitHub Copilot: runner-owned behavioral transport. The runner starts one + fresh Copilot CLI session per eval execution (`copilot -C + --model --output-format json`, prompt on stdin) and captures that + session's own JSONL events as terminal evidence for its model, cwd, isolated + `COPILOT_HOME`, fresh session, prompt hash, and transcript. Copilot's native + `task` tool with a full-capability `general-purpose` child remains an + advertised harness capability but is not the benchmark transport. When an + `interaction.json` sidecar is present, the runner first proves from the + installed help that an explicit session-id continuation flag is available, + captures the first session id from structured events, and adds only that + exact id on later turns; it never resumes the most recent session. +- Codex: the installed CLI's runner-owned app-server child-session surface, + `thread/start` followed by `turn/start`, with supplemental post-completion + `thread/read` when available, and the arm's `cwd`, selected model, and + ephemeral/fresh session settings. The schema/feature probe is preflight + readiness only; terminal evidence must prove the actual thread. Subscription + auth uses a temporary auth-only `CODEX_HOME` containing only `auth.json`; the + runner physically projects only the arm's `repo/`, `home/`, and candidate + `skill/` outside the source-repository ancestor chain, does not copy ambient + config, skills, agents, sessions, memories, plugins, MCP configuration, or + AGENTS.md, and removes the projection/home in `finally`. `model/rerouted` + and instruction sources outside that physical arm boundary are incompatible. + Do not wrap a native Codex app-server worker in another Codex subagent. +- OpenCode: runner-owned behavioral transport. The runner starts one fresh + OpenCode session per eval execution (`opencode run --format json --auto + --model `, prompt on stdin) and captures that session's structured + events as terminal evidence. Scripted interactions use only help-proven + `--session ` continuation after turn 1; `--continue` is + never used. Parallelism comes from the deterministic + foreground internal process fan-out (`invoke-runner-owned-arms.ps1`), not from an + orchestrator emitting sibling Task calls in one assistant turn. OpenCode's + native Task/General subagent (and read-only `Explore`/`Scout`) remain + advertised harness capabilities but are not the benchmark transport. +This directory contains the package-local implementation of the v0.9.1 Eval +Runner protocol. It is copied into prepared packages so the external Eval +Orchestrator can use the same runner implementation that was validated with the +package. It is not a model executor used by repository automation. + +The boundary is owner-dependent: + +```text +dispatch_owner=orchestrator: run.json + execution-profile.json -> native worker envelope -> record-native-result.ps1 -> execution-result.json +dispatch_owner=runner: run.json + execution-profile.json -> runner-owned native execute -> execution-result.json +``` + +`run.json` is the existing portable one-arm contract. It owns the prompt, +working directory, isolated home, staged candidate skill, and required +experimental controls. Its `filesystemIsolationRequired` and +`mustNotReadOutsideSandbox` fields describe the staged worker-facing package +boundary; they do not claim that the host has a hard OS filesystem sandbox. +`execution-profile.json` selects the runner, runner-native model selector, and +execution configuration. The model string is opaque to the portable layer: a +runner may pass it through unchanged or split it internally when its native CLI +requires separate provider/model arguments. The profile contains no credentials, +secrets, or portable provider field. +`execution-result.json` normalizes one blind execution and keeps grading +separate from raw evidence. Its `exit.status` is a numeric process exit code or +`null`, never a textual lifecycle label such as `completed`. Ordinary runs are +single-turn; an optional package-local `interaction.json` sidecar can request +scripted user turns, but only a runner that advertises and preflights +`scripted_multi_turn_same_session` may continue one fresh session. The runner +captures ordered user/assistant turns, the shared session/thread identity, +timestamps when available, and the complete final transcript/event artifact. + +Every runner exposes the same process surface: + +```text +runner.ps1 describe +runner.ps1 preflight -Run -Profile +runner.ps1 execute -Run -Profile +``` + +The native handoff additionally uses: + +```text +record-native-result.ps1 -Runner -Run -Profile -NativeResult -Output +``` + +`record-native-result.ps1` is deterministic and never starts a harness or a +model; it is used only for orchestrator-owned native envelopes. The direct +`execute` command runs exactly one arm and is the runner-owned native transport +when the descriptor says `dispatch_owner=runner`. It must not be nested inside +an outer model worker. A scripted interaction still uses one `execute` process +and one exact native session identity; later turns are runner-owned +continuation invocations, never fresh sessions or implicit last-session resumes. + +The commands emit one JSON document. `describe` and `preflight` do not consume +model tokens. `execute` runs exactly one arm, never grades or retries for answer +quality, and returns a normalized result even for refusals, timeouts, failures, +and incompatibility. Single-turn execution always starts a fresh session; +scripted continuation is an explicit capability-gated exception inside that +same runner-owned execution. + +The package resolver selects a named child directory under this directory. It +does not guess a runner and does not fall back to an improvised worker. A +selected runner that cannot satisfy the required contract returns +`incompatible`. Hard OS-level filesystem confinement is a confidence signal, +not a universal prerequisite: a run with all mandatory experimental controls +proven reports `strict` isolation when hard confinement is proven and +`pragmatic` isolation when it is not. A missing fresh context, controlled skill +boundary, prompt fidelity, result capture, or other mandatory control remains +incompatible. + +The fake runner is deterministic and is the conformance reference. It has no +harness-native delegation surface and its compatibility output is never proof +for a real harness. GitHub +Copilot, Codex, and OpenCode are thin harness-specific adapters. Their +native CLI flags, environment setup, event parsing, authentication injection, +and isolation checks stay inside their own directories. Windows is supported in +pragmatic mode when the native CLI satisfies the mandatory controls. + +`github-copilot` with `claude-haiku-4.5` is the Codebelt Reference evaluation +configuration: a stable, economical pairing for routine skill comparison. It is +a repository convention, not an Anthropic default, and preparation verifies that +the model still appears in the current Copilot catalog before selecting it. +Cross-runner and cross-model numbers are never blended into one score; a paired +`with_skill` versus `without_skill` comparison is only meaningful within one +identical runner, model, and configuration stratum. + +The following CLI details describe compatibility `execute` behavior where a +runner owns a different native surface; they are not a second model layer for +native worker orchestration. The native worker mechanisms above are +authoritative for the external handoff. A runner-owned runner may use its +`execute` command as that native surface; an orchestrator-owned runner must +keep `execute` out of the native subagent. + +For a single-turn run, GitHub Copilot uses `copilot -C --model +--output-format json --allow-all-tools --no-ask-user --disable-builtin-mcps +--no-color --log-level none --no-auto-update +--secret-env-vars=COPILOT_GITHUB_TOKEN,GH_TOKEN,GITHUB_TOKEN` with the exact +prepared prompt bytes delivered once through stdin. It passes no `--prompt`/`-p`, +`--resume`, `--continue`, `--session-id`, or `--connect` on this fresh +single-turn invocation, and it does not use +the blanket `--yolo`, `--allow-all`, `--allow-all-paths`, or `--allow-all-urls` +switches. `--allow-all-tools` is a broad tool-approval grant required for +noninteractive execution; it does not disable path or URL verification. +Repository-owned custom instructions remain enabled and are staged identically +in both paired arms. Personal Copilot configuration is excluded by run-local +`COPILOT_HOME`, `COPILOT_CACHE_HOME`, `HOME`, `USERPROFILE`, and XDG roots; +the runner does not copy the normal `.copilot` directory. Authentication prefers +explicit `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN`; when none is +present, the trusted runner may resolve `gh auth token` outside the worker and +inject only that token as a protected environment variable. Host `GH_CONFIG_DIR` +is never forwarded into the evaluated worker. `--secret-env-vars` removes every +listed token variable from shell and MCP child environments. Preflight does not +make a model request and therefore reports native keychain/service readiness as +conditional rather than claiming successful remote authentication. Codex's +compatibility API-key path uses `--ask-for-approval never` with `exec --sandbox +workspace-write`; subscription eval arms use the runner-owned app-server path +described above. It does not combine explicit sandbox selection with +`--approve-for-me`. OpenCode single-turn execution uses `run --format json --auto --model +` with isolated global/config roots and preserves +repository-owned project configuration; it does not depend on +`OPENCODE_DISABLE_PROJECT_CONFIG` or use `--pure`. For a scripted interaction, +turn 1 uses those same arguments and later turns add the exact captured session +id using the installed-help-proven `--session` form; `--continue` is rejected. + +Each captures an exact observable +CLI version and passes only documented environment credentials when the selected +runner supports them. None copies a global skill directory, memory store, plugin +set, or normal agent profile into a run. + +Model discovery lives in `scripts/Get-HarnessModels.ps1`. It uses the current local harness catalog where available: Copilot through the installed CLI SDK help-visible model list, Codex through `codex debug models`, OpenCode through `opencode models opencode --verbose`. OpenCode discovery returns only models with current metadata proving free availability; zero free models is a clear local failure, not a fallback to paid models. + +Freebuff is currently documented as planned/blocked. Its supported CLI remains +TUI-oriented and does not provide the required one-prompt, noninteractive, +machine-readable fresh-session transport, so no Freebuff runner is advertised. diff --git a/scripts/eval-runners/apply-eval-grading.ps1 b/scripts/eval-runners/apply-eval-grading.ps1 new file mode 100644 index 0000000..961133e --- /dev/null +++ b/scripts/eval-runners/apply-eval-grading.ps1 @@ -0,0 +1,199 @@ +<#! +.SYNOPSIS + Applies the external Grader's grading-only artifact to canonical results. + +.DESCRIPTION + The Grader is allowed to author only package-root grading.json. This + deterministic command validates its exact assertion identities, verifies + the immutable Phase 1 freeze, and changes only canonical grading fields. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [string]$GradingPath = 'grading.json' +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +. (Join-Path $PSScriptRoot 'runner-common.ps1') +. (Join-Path $PSScriptRoot 'manifest-paths.ps1') +. (Join-Path $PSScriptRoot 'execution-freeze.ps1') + +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) + +function Write-GradingResultJson { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) + + [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) +} + +function Get-GradingEntryKey { + param([Parameter(Mandatory = $true)][object]$Entry) + + return "$(Get-JsonProperty -Object $Entry -Name 'eval_id' -Default 0)|$(Get-JsonProperty -Object $Entry -Name 'configuration' -Default '')|$(Get-JsonProperty -Object $Entry -Name 'assertion_index' -Default -1)" +} + +function Get-MetadataAssertions { + param([Parameter(Mandatory = $true)][object]$Record) + + $metadata = Read-RunnerJson -Path $Record.MetadataPath + $assertions = @(Get-JsonProperty -Object $metadata -Name 'assertions' -Default @()) + if ($assertions.Count -eq 0) { throw "Metadata for '$($Record.EvalName)' declares no assertions." } + return @($assertions | ForEach-Object { [string]$_ }) +} + +function Assert-GradingEntryShape { + param([Parameter(Mandatory = $true)][object]$Entry) + + $allowed = @('eval_id', 'eval_name', 'configuration', 'assertion_index', 'assertion', 'passed', 'evidence') + foreach ($name in @(Get-JsonPropertyNames -Object $Entry)) { + if ($allowed -notcontains $name) { throw "grading.json entry contains unsupported field '$name'." } + } + foreach ($name in $allowed) { + if (-not (Test-JsonProperty -Object $Entry -Name $name)) { throw "grading.json entry is missing '$name'." } + } + $evalId = 0 + try { $evalId = [int]$Entry.eval_id } catch { throw 'grading.json eval_id must be an integer.' } + if ($evalId -lt 1) { throw 'grading.json eval_id must be positive.' } + $index = 0 + try { $index = [int]$Entry.assertion_index } catch { throw 'grading.json assertion_index must be an integer.' } + if ($index -lt 0 -or [double]$Entry.assertion_index -ne $index) { throw 'grading.json assertion_index must be a non-negative integer.' } + if ([string]$Entry.configuration -notin @('with_skill', 'without_skill')) { throw "grading.json configuration '$($Entry.configuration)' is unsupported." } + if ([string]::IsNullOrWhiteSpace([string]$Entry.eval_name) -or [string]::IsNullOrWhiteSpace([string]$Entry.assertion)) { throw 'grading.json eval_name and assertion must be non-empty strings.' } + if ($Entry.passed -isnot [bool]) { throw 'grading.json passed must be a boolean; incomplete grading is not finalizable.' } + if ($Entry.evidence -isnot [string]) { throw 'grading.json evidence must be a string.' } +} + +try { + $iteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + Assert-SafeRelativePath -RelativePath $GradingPath -FieldName 'grading path' + $gradingFullPath = Resolve-ContainedPath -BasePath $iteration -RelativePath $GradingPath -FieldName 'grading path' -Kind File + if (-not (Test-Path -LiteralPath $gradingFullPath -PathType Leaf)) { + throw "Grading is incomplete: grading-only artifact '$GradingPath' is missing." + } + + $gradingDocument = Read-RunnerJson -Path $gradingFullPath + $schemas = Get-RunnerSchemaNames + if ([string](Get-JsonProperty -Object $gradingDocument -Name 'schema' -Default '') -ne $schemas.Grading) { + throw "grading.json must declare '$($schemas.Grading)'." + } + $topLevelAllowed = @('schema', 'grading') + foreach ($name in @(Get-JsonPropertyNames -Object $gradingDocument)) { + if ($topLevelAllowed -notcontains $name) { throw "grading.json contains unsupported field '$name'; the Grader may author only grading entries." } + } + $submitted = @(Get-JsonProperty -Object $gradingDocument -Name 'grading' -Default @()) + foreach ($entry in $submitted) { + Assert-GradingEntryShape -Entry $entry + } + + $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState + $manifest = $freezeValidation.Manifest + $declaredGradingPath = [string](Get-JsonProperty -Object $manifest -Name 'grading' -Default '') + if ([string]::IsNullOrWhiteSpace($declaredGradingPath) -or $declaredGradingPath -ne $GradingPath) { + throw "grading path '$GradingPath' does not match manifest.grading '$declaredGradingPath'." + } + $bridgeScript = Join-Path $PSScriptRoot 'bridge-manifest-results.ps1' + if (-not (Test-Path -LiteralPath $bridgeScript -PathType Leaf)) { + throw "Package-local manifest bridge is missing at '$bridgeScript'." + } + $bridgeOutput = & pwsh -NoProfile -File $bridgeScript -IterationDirectory $iteration -RequireComplete 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Canonical result validation failed before grading application: $([string]::Join(' ', @($bridgeOutput | ForEach-Object { [string]$_ })))" + } + + $records = @(Get-ManifestRunRecords -IterationDirectory $iteration -Manifest $manifest | Sort-Object EvalId, Configuration) + $expected = @{} + $canonicalByKey = @{} + foreach ($record in $records) { + $assertions = @(Get-MetadataAssertions -Record $record) + $canonical = Read-RunnerJson -Path $record.ResultPath + if ([int]$canonical.eval_id -ne [int]$record.EvalId -or [string]$canonical.eval_name -ne [string]$record.EvalName -or [string]$canonical.configuration -ne [string]$record.Configuration) { + throw "Canonical result '$($record.ResultRelative)' does not match its exact manifest identity." + } + if ([string]$canonical.execution_status -ne 'completed') { + throw "Grading is incomplete: '$($record.EvalName)/$($record.Configuration)' is not a completed execution." + } + $canonicalByKey["$($record.EvalId)|$($record.Configuration)"] = $canonical + for ($index = 0; $index -lt $assertions.Count; $index++) { + $key = "$($record.EvalId)|$($record.Configuration)|$index" + $expected[$key] = [ordered]@{ + eval_id = [int]$record.EvalId + eval_name = [string]$record.EvalName + configuration = [string]$record.Configuration + assertion_index = $index + assertion = [string]$assertions[$index] + } + } + } + if ($submitted.Count -ne $expected.Count) { + throw "grading.json assertion cardinality $($submitted.Count) does not match the required $($expected.Count)." + } + + $validated = @{} + foreach ($entry in $submitted) { + Assert-GradingEntryShape -Entry $entry + $key = Get-GradingEntryKey -Entry $entry + if (-not $expected.ContainsKey($key)) { throw "grading.json identifies an unknown eval/configuration/assertion '$key'." } + if ($validated.ContainsKey($key)) { throw "grading.json contains duplicate grading entry '$key'." } + $target = $expected[$key] + if ([string]$entry.eval_name -ne [string]$target.eval_name -or [string]$entry.assertion -ne [string]$target.assertion) { + throw "grading.json assertion identity '$key' does not match eval-metadata.json exactly." + } + $validated[$key] = $entry + } + foreach ($key in $expected.Keys) { + if (-not $validated.ContainsKey($key)) { throw "grading.json is missing required grading entry '$key'." } + } + + $updates = [System.Collections.Generic.List[object]]::new() + foreach ($record in $records) { + $canonical = $canonicalByKey["$($record.EvalId)|$($record.Configuration)"] + if (-not (Test-JsonProperty -Object $canonical -Name 'grading')) { throw "Canonical result '$($record.ResultRelative)' is missing its grading array." } + $newGrading = [System.Collections.Generic.List[object]]::new() + $assertions = @(Get-MetadataAssertions -Record $record) + for ($index = 0; $index -lt $assertions.Count; $index++) { + $entry = $validated["$($record.EvalId)|$($record.Configuration)|$index"] + $newGrading.Add([ordered]@{ text = [string]$entry.assertion; passed = [bool]$entry.passed; evidence = [string]$entry.evidence }) + } + $beforeNonGrading = Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $canonical -PropertyName 'grading') + # Keep the parsed canonical values as-is. ConvertFrom-Json reparses + # ISO timestamp strings into DateTime values and ConvertTo-Json then + # changes their lexical precision (for example .650Z -> .65Z), which + # would make a grading-only update look like non-grading tampering. + $candidate = [ordered]@{} + foreach ($property in @($canonical.PSObject.Properties)) { + $candidate[[string]$property.Name] = $property.Value + } + $candidate.grading = @($newGrading.ToArray()) + $afterNonGrading = Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $candidate -PropertyName 'grading') + if ($beforeNonGrading -ne $afterNonGrading) { + throw "Canonical non-grading field changed while applying '$($record.ResultRelative)'." + } + $updates.Add([pscustomobject]@{ Path = $record.ResultPath; Before = $canonical; Candidate = $candidate; Changed = (Get-JsonFingerprint -Object $canonical) -ne (Get-JsonFingerprint -Object $candidate) }) + } + + foreach ($update in $updates) { + if ($update.Changed) { Write-GradingResultJson -Path $update.Path -Value $update.Candidate } + } + [void](Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState) + foreach ($update in $updates) { + $after = Read-RunnerJson -Path $update.Path + $beforeFingerprint = Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $update.Before -PropertyName 'grading') + $afterFingerprint = Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $after -PropertyName 'grading') + if ($beforeFingerprint -ne $afterFingerprint) { throw "Canonical non-grading field changed after applying '$($update.Path)'." } + } + + Write-RunnerJson -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-grading-application/1' + status = 'applied' + grading_file = $gradingFullPath + canonical_results = $updates.Count + graded_assertions = $expected.Count + changed_results = @($updates | Where-Object { $_.Changed }).Count + execution_freeze = $freezeValidation.Path + }) -AsOutput +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/bridge-execution-result.ps1 b/scripts/eval-runners/bridge-execution-result.ps1 new file mode 100644 index 0000000..61df502 --- /dev/null +++ b/scripts/eval-runners/bridge-execution-result.ps1 @@ -0,0 +1,293 @@ +<#! +.SYNOPSIS + Bridges one normalized execution-result.json into the existing eval-result/2 shape. + +.DESCRIPTION + This bridge runs after execution and before grading. It reads no expected + output or assertions, preserves any existing grading array, validates raw + artifact provenance, and deliberately leaves unavailable telemetry null. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] + [string]$Run, + + [Parameter(Mandatory = $true)] + [string]$ExecutionResult, + + [Parameter(Mandatory = $true)] + [string]$Result, + + [switch]$RequireNativeDelegation +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot 'runner-common.ps1') +. (Join-Path $PSScriptRoot 'execution-freeze.ps1') + +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) + +function Write-BridgeJson { + param( + [Parameter(Mandatory = $true)][string]$Path, + [Parameter(Mandatory = $true)][object]$Value + ) + + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + $serializable = if ($Value -is [System.Collections.IDictionary]) { + $copy = [ordered]@{} + foreach ($key in $Value.Keys) { $copy[[string]$key] = $Value[$key] } + [pscustomobject]$copy + } else { + $Value + } + [System.IO.File]::WriteAllText($Path, ((ConvertTo-Json -InputObject $serializable -Depth 100) + [Environment]::NewLine), $utf8NoBom) +} + +function Get-CapabilityBoolean { + param([object]$Value) + + if ([string]$Value -in @('supported', 'verified', 'true')) { return $true } + if ([string]$Value -in @('unsupported', 'excluded', 'false')) { return $false } + return $null +} + +function Get-MetricValue { + param( + [Parameter(Mandatory = $true)][object]$Result, + [Parameter(Mandatory = $true)][string]$Name + ) + + $metric = Get-JsonProperty -Object $Result.telemetry -Name $Name -Default $null + if ($null -eq $metric -or [string](Get-JsonProperty -Object $metric -Name 'status' -Default '') -ne 'available') { + return $null + } + return Get-JsonProperty -Object $metric -Name 'value' -Default $null +} + +function Get-ArtifactPath { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Artifact + ) + + $path = [string](Get-JsonProperty -Object $Artifact -Name 'path' -Default '') + $scope = [string](Get-JsonProperty -Object $Artifact -Name 'scope' -Default '') + if ($scope -eq 'run') { + return Resolve-ContainedPath -BasePath $RunData.RunRoot -RelativePath $path -FieldName 'execution-result artifact.path' -Kind File + } + if ($scope -eq 'package') { + return Resolve-ContainedPath -BasePath $IterationDirectory -RelativePath $path -FieldName 'execution-result package artifact.path' -Kind File + } + throw "Unsupported execution-result artifact scope '$scope'." +} + +function Get-ResultRelativeArtifactPath { + param( + [Parameter(Mandatory = $true)][string]$EvalDirectory, + [Parameter(Mandatory = $true)][string]$FullPath + ) + + $relative = [System.IO.Path]::GetRelativePath($EvalDirectory, $FullPath).Replace('\', '/') + Assert-SafeRelativePath -RelativePath $relative -FieldName 'result.output_files' + return $relative +} + +function Get-ExistingGrading { + param( + [Parameter(Mandatory = $true)][string]$ResultPath + ) + + if (-not (Test-Path -LiteralPath $ResultPath -PathType Leaf)) { + throw "Manifest-declared result stub '$ResultPath' does not exist; the bridge will not create a new grading-less result file." + } + + $existing = Read-RunnerJson -Path $ResultPath + if (-not (Test-JsonProperty -Object $existing -Name 'grading')) { + throw "Manifest-declared result stub '$ResultPath' is missing its grading array." + } + return @(Get-JsonProperty -Object $existing -Name 'grading' -Default @()) +} + +try { + $runData = Resolve-RunContract -RunPath $Run + $runPath = $runData.RunPath + $runDirectory = $runData.RunRoot + $evalDirectory = Split-Path -Parent $runDirectory + $iterationDirectory = Split-Path -Parent $evalDirectory + # The bridge is a validator of frozen evidence, never an authority that + # can bless a new raw hash. This call intentionally covers every manifest + # arm before this one-arm operation can write a canonical result. + [void](Assert-ExecutionFreeze -IterationDirectory $iterationDirectory -RequireOrchestrationState) + $executionPath = (Resolve-Path -LiteralPath $ExecutionResult -ErrorAction Stop).Path + $executionResultHash = Get-Sha256HexFromFile -Path $executionPath + $resultPath = [System.IO.Path]::GetFullPath($Result, (Get-Location).Path) + if (-not (Test-PathInside -BasePath $iterationDirectory -CandidatePath $executionPath)) { + throw 'execution-result.json must remain inside the prepared iteration package.' + } + if (-not (Test-PathInside -BasePath $iterationDirectory -CandidatePath $resultPath)) { + throw 'eval-result output must remain inside the prepared iteration package.' + } + + $raw = Read-RunnerJson -Path $executionPath + [void](Assert-ExecutionResult -Result $raw) + if ([string]$raw.input.prompt_sha256 -ne $runData.PromptHash) { + throw 'execution-result input.prompt_sha256 does not match prompt.md.' + } + if ([string]$raw.input.run_json_sha256 -ne (Get-Sha256HexFromFile -Path $runPath)) { + throw 'execution-result input.run_json_sha256 does not match run.json.' + } + $profilePath = Join-Path $iterationDirectory 'execution-profile.json' + if (-not (Test-Path -LiteralPath $profilePath -PathType Leaf)) { + throw 'Runner-aware package is missing execution-profile.json.' + } + $profile = Resolve-ExecutionProfile -ProfilePath $profilePath + if ([string]$raw.input.profile_sha256 -ne $profile.Hash) { + throw 'execution-result input.profile_sha256 does not match execution-profile.json.' + } + $rawRunnerName = [string](Get-JsonProperty -Object $raw.runner -Name 'name' -Default '') + if ($rawRunnerName -ne [string]$profile.Profile.runner) { + throw "execution-result runner '$rawRunnerName' does not match selected runner '$($profile.Profile.runner)'." + } + if ([int]$raw.run.eval_id -ne $runData.EvalId -or [string]$raw.run.configuration -ne $runData.Mode) { + throw 'execution-result run identity does not match run.json.' + } + + $artifactPaths = [System.Collections.Generic.List[string]]::new() + foreach ($artifact in @($raw.artifacts)) { + $full = Get-ArtifactPath -RunData $runData -IterationDirectory $iterationDirectory -Artifact $artifact + $expectedHash = [string](Get-JsonProperty -Object $artifact -Name 'sha256' -Default '') + if ((Get-Sha256HexFromFile -Path $full) -ne $expectedHash) { + throw "Artifact '$($artifact.path)' has a hash that does not match the recorded execution evidence." + } + $expectedSize = [int64](Get-JsonProperty -Object $artifact -Name 'size' -Default -1) + if ((Get-Item -LiteralPath $full).Length -ne $expectedSize) { + throw "Artifact '$($artifact.path)' has a size that does not match the recorded execution evidence." + } + $artifactPaths.Add((Get-ResultRelativeArtifactPath -EvalDirectory $evalDirectory -FullPath $full)) + } + + if ($RequireNativeDelegation) { + if ([string]$raw.status -eq 'incompatible') { + throw 'An incompatible native-worker arm is diagnostic only and cannot be bridged into a gradeable canonical result.' + } + $runnerDescriptor = Get-PackageRunnerDescriptor -RunnerName ([string]$profile.Profile.runner) + [void](Assert-NativeWorkerTerminalEvidence ` + -ExecutionEvidence $raw ` + -Run $runData ` + -RequestedModel ([string]$profile.Profile.Model) ` + -ExpectedRunner ([string]$profile.Profile.runner) ` + -ExpectedMechanism ([string]$runnerDescriptor.delegation.mechanism)) + Assert-NativeTerminalCaptureArtifact -ExecutionResult $raw + } + + $finalStatus = [string]$raw.final_response.status + $output = if ($finalStatus -eq 'available') { [string]$raw.final_response.text } else { '' } + $transcript = '' + $transcriptMetricObject = Get-JsonProperty -Object $raw.telemetry -Name 'transcript' -Default $null + $transcriptMetric = Get-MetricValue -Result $raw -Name 'transcript' + if ($null -ne $transcriptMetric) { + $transcriptArtifact = [string](Get-JsonProperty -Object $transcriptMetric -Name 'artifact' -Default '') + if (-not [string]::IsNullOrWhiteSpace($transcriptArtifact)) { + $transcript = "artifact: $transcriptArtifact" + } + } + $toolCallsValue = Get-MetricValue -Result $raw -Name 'tool_calls' + $costValue = Get-MetricValue -Result $raw -Name 'cost' + $tokenValue = Get-MetricValue -Result $raw -Name 'tokens' + $evidence = Get-JsonProperty -Object $raw -Name 'evidence' -Default ([ordered]@{}) + $commands = @(Get-JsonProperty -Object $evidence -Name 'commands' -Default @()) + $files = @(Get-JsonProperty -Object $evidence -Name 'files' -Default @()) + $warnings = @((Get-JsonProperty -Object $raw -Name 'warnings' -Default @()) + (Get-JsonProperty -Object $raw -Name 'compatibility_deviations' -Default @())) + $notes = [System.Collections.Generic.List[string]]::new() + $notes.Add("execution_status=$($raw.status)") + if ($finalStatus -eq 'unavailable') { $notes.Add("final_response_unavailable=$($raw.final_response.reason)") } + foreach ($warning in $warnings) { if (-not [string]::IsNullOrWhiteSpace([string]$warning)) { $notes.Add([string]$warning) } } + + $existingCanonical = Read-RunnerJson -Path $resultPath + $existingGrading = @(Get-ExistingGrading -ResultPath $resultPath) + $caps = Get-JsonProperty -Object $raw.isolation -Name 'capabilities' -Default ([ordered]@{}) + $requestedModel = [string](Get-JsonProperty -Object $raw.requested -Name 'model' -Default '') + $resolvedModelValue = Get-JsonProperty -Object $raw.resolved -Name 'model' -Default $null + $resolvedModel = if ($null -eq $resolvedModelValue) { '' } else { [string]$resolvedModelValue } + $resolutionStatus = [string](Get-JsonProperty -Object $raw.resolved -Name 'status' -Default 'unavailable') + $resolutionReason = [string](Get-JsonProperty -Object $raw.resolved -Name 'reason' -Default '') + $notes.Add("configuration_resolution=$resolutionStatus") + $portableResult = [ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + skill_name = if ($runData.Mode -eq 'with_skill') { [string](Get-JsonProperty -Object $runData.Contract -Name 'skillName' -Default '') } else { '' } + iteration = [int](Get-JsonProperty -Object $runData.Contract -Name 'iteration' -Default 0) + eval_id = $runData.EvalId + eval_name = $runData.EvalName + configuration = $runData.Mode + model = if ([string]::IsNullOrWhiteSpace($resolvedModel)) { $requestedModel } else { $resolvedModel } + requested_model = $requestedModel + resolved_model = $resolvedModel + configuration_resolution_status = $resolutionStatus + configuration_resolution_reason = $resolutionReason + harness = "$(Get-JsonProperty -Object $raw.harness -Name 'name' -Default 'unknown') $(Get-JsonProperty -Object $raw.harness -Name 'version' -Default '')".Trim() + executed_utc = [string]$raw.finished_utc + output = $output + output_files = @($artifactPaths | Sort-Object -Unique) + transcript = $transcript + shell_commands = @($commands) + files_read = @(Get-JsonProperty -Object $evidence -Name 'files_read' -Default @()) + files_written = @(Get-JsonProperty -Object $evidence -Name 'files_written' -Default @()) + stdout = if (@($artifactPaths | Where-Object { $_ -match 'events\.jsonl$' }).Count -gt 0) { 'artifact: events.jsonl' } else { '' } + stderr = if (@($artifactPaths | Where-Object { $_ -match 'stderr\.txt$' }).Count -gt 0) { 'artifact: stderr.txt' } else { '' } + exit_status = Get-JsonProperty -Object $raw.exit -Name 'status' -Default $null + duration_seconds = [double]$raw.duration_seconds + total_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'total_tokens' -Default $null } else { $null } + tool_calls = $toolCallsValue + turns = Get-JsonProperty -Object $evidence -Name 'turns' -Default $null + base_input_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'input_tokens' -Default (Get-JsonProperty -Object $tokenValue -Name 'input' -Default $null) } else { $null } + output_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'output_tokens' -Default (Get-JsonProperty -Object $tokenValue -Name 'output' -Default $null) } else { $null } + cache_read_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'cached_input_tokens' -Default (Get-JsonProperty -Object $tokenValue -Name 'cache_read' -Default $null) } else { $null } + cache_write_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'cache_write_tokens' -Default (Get-JsonProperty -Object $tokenValue -Name 'cache_write' -Default $null) } else { $null } + cache_write_1h_tokens = $null + estimated_cost_usd = $costValue + model_effort = [string](Get-JsonProperty -Object $raw.resolved -Name 'reasoning_effort' -Default '') + isolation = [ordered]@{ + level = Get-JsonProperty -Object $raw.isolation -Name 'level' -Default 'unsupported' + status = Get-JsonProperty -Object $raw.isolation -Name 'status' -Default 'unverified' + hard_filesystem_confinement = Get-JsonProperty -Object $raw.isolation -Name 'hard_filesystem_confinement' -Default $false + mechanisms = @(Get-JsonProperty -Object $raw.isolation -Name 'mechanisms' -Default @()) + fresh_context = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'fresh_context' -Default $null) + isolated_home = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'isolated_home_config' -Default $null) + isolated_cwd = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'isolated_working_directory' -Default $null) + filesystem_sandbox = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'filesystem_confinement' -Default $null) + candidate_skill_exposed = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'candidate_skill_exposure' -Default $null) + transcript_captured = if ($null -ne $transcriptMetricObject) { [string](Get-JsonProperty -Object $transcriptMetricObject -Name 'status' -Default '') -eq 'available' } else { $null } + } + execution_status = [string]$raw.status + execution_run_id = [string]$raw.run_id + execution_result_file = [System.IO.Path]::GetRelativePath($evalDirectory, $executionPath).Replace('\', '/') + execution_result_sha256 = $executionResultHash + grading = @($existingGrading) + notes = [string]::Join("`n", @($notes)) + } + + # Re-bridging is idempotent only for the exact frozen raw result. Existing + # grading is preserved, but canonical non-grading fields are never repaired + # or accepted from an external writer. + $existingRawHash = [string](Get-JsonProperty -Object $existingCanonical -Name 'execution_result_sha256' -Default '') + if (-not [string]::IsNullOrWhiteSpace($existingRawHash) -and $existingRawHash -ne $executionResultHash) { + throw "Execution integrity failure: canonical result '$Result' refers to a different raw execution hash; refusing repair." + } + if (-not [string]::IsNullOrWhiteSpace($existingRawHash)) { + $expectedFingerprint = Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $portableResult -PropertyName 'grading') + $actualFingerprint = Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $existingCanonical -PropertyName 'grading') + if ($actualFingerprint -ne $expectedFingerprint) { + throw "Execution integrity failure: canonical non-grading fields changed for '$Result'; refusing repair." + } + } elseif ([string](Get-JsonProperty -Object $existingCanonical -Name 'execution_status' -Default '') -ne 'unrun') { + throw "Execution integrity failure: canonical result '$Result' is neither a prepared stub nor the frozen bridged result; refusing repair." + } + Write-BridgeJson -Path $resultPath -Value $portableResult + Write-RunnerJson -Value ([ordered]@{ schema = 'codebeltnet/agentic/eval-result-bridge/1'; result = [System.IO.Path]::GetRelativePath($iterationDirectory, $resultPath).Replace('\', '/'); execution_status = $raw.status }) -AsOutput +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/bridge-manifest-results.ps1 b/scripts/eval-runners/bridge-manifest-results.ps1 new file mode 100644 index 0000000..14653cc --- /dev/null +++ b/scripts/eval-runners/bridge-manifest-results.ps1 @@ -0,0 +1,124 @@ +<#! +.SYNOPSIS + Bridges every available execution result using only manifest-declared arm paths. + +.DESCRIPTION + Reads manifest.json, validates the exact run_manifest, execution_result, and result + paths for every arm, rejects result-like shadow files, and invokes the existing + one-arm bridge without reconstructing any filename from an arm or mode name. + No model, runner, or grader is started by this helper. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] + [string]$IterationDirectory, + + [switch]$RequireComplete, + + [switch]$RequireNativeDelegation, + + [switch]$RequireParallelDispatch, + + [string]$OrchestrationStatePath = 'orchestration-state.json' +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +. (Join-Path $PSScriptRoot 'manifest-paths.ps1') +. (Join-Path $PSScriptRoot 'orchestration.ps1') +. (Join-Path $PSScriptRoot 'execution-freeze.ps1') +. (Join-Path $PSScriptRoot 'package-integrity.ps1') + +try { + $iterationPath = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + $manifestPath = Join-Path $iterationPath 'manifest.json' + if (-not (Test-Path -LiteralPath $manifestPath -PathType Leaf)) { + throw "Prepared iteration is missing manifest.json at '$iterationPath'." + } + + $manifest = Read-RunnerJson -Path $manifestPath + [void](Assert-PackageRunnerToolsIntegrity -IterationDirectory $iterationPath -Manifest $manifest) + $records = @(Get-ManifestRunRecords -IterationDirectory $iterationPath -Manifest $manifest) + # Validate the complete immutable Phase 1 ledger before any one-arm bridge + # can write a canonical result. This is deliberately read-only: a changed + # raw result or artifact is a failed evaluation, never a new blessing. + $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iterationPath -RequireOrchestrationState + if ($RequireComplete) { + [void](Assert-FanoutPhase1Success -Aggregate $freezeValidation.Aggregate -MessagePrefix 'Manifest bridge Phase 1') + } + $profileData = Resolve-ExecutionProfile -ProfilePath (Join-Path $iterationPath 'execution-profile.json') + $runnerDescriptor = Get-PackageRunnerDescriptor -RunnerName ([string]$profileData.Runner) + $effectiveRequireNativeDelegation = [bool]$RequireNativeDelegation -or [string](Get-JsonProperty -Object $runnerDescriptor.delegation -Name 'dispatch_owner' -Default '') -eq 'runner' + $parallelDispatch = $null + if ($RequireParallelDispatch) { + Assert-SafeRelativePath -RelativePath $OrchestrationStatePath -FieldName 'orchestration state path' + $statePath = Join-Path $iterationPath ($OrchestrationStatePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if (-not (Test-Path -LiteralPath $statePath -PathType Leaf)) { + throw "Parallel native-worker orchestration requires '$OrchestrationStatePath' at the iteration root." + } + + $plan = New-EvalOrchestrationPlan -IterationDirectory $iterationPath -Manifest $manifest -Profile $profileData.Profile -Descriptor $runnerDescriptor + $state = Read-RunnerJson -Path $statePath + $parallelDispatch = Assert-OrchestrationConcurrency -Plan $plan -State $state + } + $shadows = @(Get-ManifestShadowResultFiles -Records $records) + if ($shadows.Count -gt 0) { + $messages = @($shadows | ForEach-Object { + "$($_.EvalName)/$($_.Configuration) has unreferenced result-like sibling '$($_.Path)'; use the exact manifest result '$($_.CanonicalPath)'." + }) + throw ($messages -join [Environment]::NewLine) + } + + $oneArmBridge = Join-Path $PSScriptRoot 'bridge-execution-result.ps1' + if (-not (Test-Path -LiteralPath $oneArmBridge -PathType Leaf)) { + throw "Package-local one-arm bridge is missing at '$oneArmBridge'." + } + + $bridged = [System.Collections.Generic.List[string]]::new() + $missing = [System.Collections.Generic.List[string]]::new() + foreach ($record in $records) { + if (-not (Test-Path -LiteralPath $record.ExecutionResultPath -PathType Leaf)) { + $missing.Add("$($record.EvalName)/$($record.Configuration)") + continue + } + + # Always invoke the one-arm bridge. The raw result is the authoritative + # terminal evidence; a status/path match alone cannot prove that the + # canonical result reflects the current raw file. The one-arm bridge + # preserves existing grading while revalidating hashes and provenance. + $bridgeOutput = & pwsh -NoProfile -File $oneArmBridge ` + -Run $record.RunManifestPath ` + -ExecutionResult $record.ExecutionResultPath ` + -Result $record.ResultPath ` + -RequireNativeDelegation:$effectiveRequireNativeDelegation 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "$($record.EvalName)/$($record.Configuration) bridge failed for manifest paths run='$($record.RunManifestRelative)', execution='$($record.ExecutionResultRelative)', result='$($record.ResultRelative)': $([string]::Join(' ', @($bridgeOutput)))" + } + $bridged.Add("$($record.EvalName)/$($record.Configuration)") + } + + $validation = Test-ManifestResults ` + -IterationDirectory $iterationPath ` + -Manifest $manifest ` + -Records $records ` + -RequireComplete:$RequireComplete + if (-not $validation.Success) { + throw ([string]::Join([Environment]::NewLine, @($validation.Errors))) + } + + Write-RunnerJson -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-manifest-bridge/1' + iteration = $iterationPath + expected_arms = $validation.ExpectedArmCount + bridged_arms = $validation.BridgedResults + terminal_execution_results = $validation.TerminalExecutionResults + missing_execution_results = @($missing) + complete = $validation.Complete + parallel_dispatch = $parallelDispatch + warnings = @($validation.Warnings) + }) -AsOutput +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 new file mode 100644 index 0000000..59d1e49 --- /dev/null +++ b/scripts/eval-runners/codex/runner.ps1 @@ -0,0 +1,1847 @@ +<#! +.SYNOPSIS + Codex Eval Runner adapter. + +.DESCRIPTION + This is the only place where Codex CLI flags, CODEX_HOME handling, JSONL + event parsing, and Codex isolation limitations are defined. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'codex' + version = '0.9.1' + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = 'unavailable' } + capabilities = [ordered]@{ + single_turn = 'supported' + scripted_multi_turn_same_session = 'conditional' + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'conditional' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'conditional' + cache_token_telemetry = 'conditional' + tool_call_telemetry = 'supported' + command_evidence = 'conditional' + file_evidence = 'conditional' + cost_telemetry = 'conditional' + credential_child_filtering = 'supported' + native_skill_activation_evidence = 'unsupported' + # The app-server schema proves that a native child surface exists, not + # what the child actually resolved or inherited. Terminal evidence is + # required for every delegated-worker control. + native_worker_delegation = 'conditional' + delegated_worker_full_capability = 'conditional' + delegated_worker_model_lock = 'conditional' + delegated_worker_working_directory = 'conditional' + delegated_worker_result_capture = 'conditional' + delegated_worker_capacity_signal = 'conditional' + } + delegation = [ordered]@{ + dispatch_owner = 'runner' + mode = 'native_worker' + mechanism = 'Codex app-server native child session via thread/start and turn/start with per-worker cwd, model, and ephemeral context' + worker_role = 'native-codex-child-session' + full_capability = 'conditional' + model_lock = 'conditional' + working_directory = 'conditional' + result_capture = 'conditional' + capacity = 'harness_authoritative' + nested_model_execution = $false + } + supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Resolve-CodexInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + return [pscustomobject]@{ + Run = Resolve-RunContract -RunPath $Run + Profile = Resolve-ExecutionProfile -ProfilePath $Profile + } +} + +function Get-CodexAuthSource { + $authVariables = @(Get-ProviderAuthenticationVariables -Provider 'openai') + foreach ($name in $authVariables) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { + return [pscustomobject]@{ Kind = 'environment'; Name = $name; Path = $null } + } + } + + $configuredHome = [Environment]::GetEnvironmentVariable('CODEX_HOME') + $codexHome = if ([string]::IsNullOrWhiteSpace($configuredHome)) { + Join-Path ([Environment]::GetFolderPath('UserProfile')) '.codex' + } else { + $configuredHome + } + $authPath = Join-Path $codexHome 'auth.json' + if (Test-Path -LiteralPath $authPath -PathType Leaf) { + return [pscustomobject]@{ Kind = 'subscription_file'; Name = 'auth.json'; Path = (Resolve-Path -LiteralPath $authPath).Path } + } + + return [pscustomobject]@{ Kind = 'missing'; Name = $null; Path = $null } +} + +function Invoke-CodexCli { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][object]$Inputs, + [System.Collections.IDictionary]$Environment, + [byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 60 + ) + + $allArguments = @($CommandInfo.Prefix) + @($Arguments) + return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList $allArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function New-CodexAuthOnlyHome { + param([Parameter(Mandatory = $true)][object]$Auth) + + if ($Auth.Kind -ne 'subscription_file' -or [string]::IsNullOrWhiteSpace([string]$Auth.Path)) { + throw 'Codex auth-only home requires a resolved subscription auth.json source.' + } + $homePath = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-codex-auth-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $homePath -Force | Out-Null + $authDestination = Join-Path $homePath 'auth.json' + try { + # The temporary home is intentionally created outside the prepared + # package. It contains exactly one copied file and is removed in the + # app-server finally block, including start/timeout failures. + Copy-Item -LiteralPath $Auth.Path -Destination $authDestination -Force -ErrorAction Stop + $entries = @(Get-ChildItem -LiteralPath $homePath -Force -ErrorAction Stop) + if ($entries.Count -ne 1 -or [string]$entries[0].Name -ne 'auth.json' -or -not (Test-Path -LiteralPath $authDestination -PathType Leaf)) { + throw 'Codex temporary subscription home was not auth-only.' + } + return [pscustomobject]@{ + Path = $homePath + AuthPath = $authDestination + AuthOnly = $true + } + } catch { + if (Test-Path -LiteralPath $homePath) { + Remove-Item -LiteralPath $homePath -Recurse -Force -ErrorAction SilentlyContinue + } + throw + } +} + +function Assert-CodexProjectionSource { + param([Parameter(Mandatory = $true)][string]$Path) + + if (-not (Test-Path -LiteralPath $Path -PathType Container)) { return } + $reparsePoint = [System.IO.FileAttributes]::ReparsePoint + $links = @(Get-ChildItem -LiteralPath $Path -Recurse -Force -ErrorAction Stop | Where-Object { ($_.Attributes -band $reparsePoint) -ne 0 }) + if ($links.Count -gt 0) { + throw "Codex physical projection refuses reparse-point input '$($links[0].FullName)'." + } +} + +function Copy-CodexProjectionDirectory { + param( + [Parameter(Mandatory = $true)][string]$Source, + [Parameter(Mandatory = $true)][string]$Destination + ) + + New-Item -ItemType Directory -Path $Destination -Force | Out-Null + if (-not (Test-Path -LiteralPath $Source -PathType Container)) { return } + Assert-CodexProjectionSource -Path $Source + foreach ($item in @(Get-ChildItem -LiteralPath $Source -Force -ErrorAction Stop)) { + Copy-Item -LiteralPath $item.FullName -Destination $Destination -Recurse -Force + } +} + +function Get-CodexProjectionFileSet { + param([Parameter(Mandatory = $true)][string]$Root) + + if (-not (Test-Path -LiteralPath $Root -PathType Container)) { return @() } + return @(Get-ChildItem -LiteralPath $Root -Recurse -File -Force | ForEach-Object { + [System.IO.Path]::GetRelativePath($Root, $_.FullName).Replace('\', '/') + } | Sort-Object) +} + +function New-CodexExecutionProjection { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $projectionRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-codex-projection-' + [Guid]::NewGuid().ToString('N')) + $logicalRunRoot = [System.IO.Path]::GetFullPath([string]$Inputs.Run.RunRoot) + $physicalRunRoot = [System.IO.Path]::GetFullPath($projectionRoot) + if (Test-PathInside -BasePath $logicalRunRoot -CandidatePath $physicalRunRoot) { + throw 'Codex physical projection unexpectedly resolved under the logical arm root.' + } + New-Item -ItemType Directory -Path $physicalRunRoot -Force | Out-Null + $physicalPrompt = Join-Path $physicalRunRoot 'prompt.md' + [System.IO.File]::WriteAllBytes($physicalPrompt, [byte[]]$Inputs.Run.PromptBytes) + $physicalRepo = Join-Path $physicalRunRoot 'repo' + $physicalHome = Join-Path $physicalRunRoot 'home' + Copy-CodexProjectionDirectory -Source $Inputs.Run.WorkingDirectoryPath -Destination $physicalRepo + Copy-CodexProjectionDirectory -Source $Inputs.Run.HomeDirectoryPath -Destination $physicalHome + + $physicalSkill = $null + if ($Inputs.Run.CandidateSkillExposed) { + $skillRelative = [System.IO.Path]::GetRelativePath($Inputs.Run.RunRoot, $Inputs.Run.SkillDirectoryPath) + $physicalSkill = Join-Path $physicalRunRoot ($skillRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + Copy-CodexProjectionDirectory -Source $Inputs.Run.SkillDirectoryPath -Destination $physicalSkill + } + + $physicalInteraction = $null + if ($null -ne $Inputs.Run.InteractionPath) { + $interactionRelative = [System.IO.Path]::GetRelativePath($Inputs.Run.RunRoot, $Inputs.Run.InteractionPath) + $physicalInteraction = Join-Path $physicalRunRoot ($interactionRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $physicalInteraction) -Force | Out-Null + Copy-Item -LiteralPath $Inputs.Run.InteractionPath -Destination $physicalInteraction -Force + foreach ($turn in @($Inputs.Run.Interaction.turns)) { + $source = [string](Get-JsonProperty -Object $turn -Name 'source' -Default '') + if ([string]::IsNullOrWhiteSpace($source)) { continue } + $logicalSource = Resolve-ContainedPath -BasePath $Inputs.Run.RunRoot -RelativePath $source -FieldName 'interaction turn source' -Kind File + $sourceRelative = [System.IO.Path]::GetRelativePath($Inputs.Run.RunRoot, $logicalSource) + $physicalSource = Join-Path $physicalRunRoot ($sourceRelative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $physicalSource) -Force | Out-Null + Copy-Item -LiteralPath $logicalSource -Destination $physicalSource -Force + } + } + + $physicalRun = [pscustomobject]@{ + RunPath = $Inputs.Run.RunPath + RunRoot = $physicalRunRoot + Contract = $Inputs.Run.Contract + EvalId = $Inputs.Run.EvalId + EvalName = $Inputs.Run.EvalName + Mode = $Inputs.Run.Mode + PromptPath = $physicalPrompt + PromptBytes = $Inputs.Run.PromptBytes + PromptHash = $Inputs.Run.PromptHash + WorkingDirectoryPath = $physicalRepo + HomeDirectoryPath = $physicalHome + SkillDirectoryPath = $physicalSkill + CandidateSkillExposed = $Inputs.Run.CandidateSkillExposed + FixtureHash = $Inputs.Run.FixtureHash + SkillHash = $Inputs.Run.SkillHash + InteractionPath = $physicalInteraction + InteractionHash = $Inputs.Run.InteractionHash + Interaction = $Inputs.Run.Interaction + } + return [pscustomobject]@{ + Root = $physicalRunRoot + Run = $physicalRun + LogicalRun = $Inputs.Run + LogicalWorkingDirectory = $Inputs.Run.WorkingDirectoryPath + LogicalHomeDirectory = $Inputs.Run.HomeDirectoryPath + PhysicalWorkingDirectory = $physicalRepo + PhysicalHomeDirectory = $physicalHome + InitialRepositoryFiles = @(Get-CodexProjectionFileSet -Root $physicalRepo) + Proven = $true + } +} + +function Sync-CodexProjectedRepository { + param([Parameter(Mandatory = $true)][object]$Projection) + + $logicalRepo = [string]$Projection.LogicalWorkingDirectory + $physicalRepo = [string]$Projection.PhysicalWorkingDirectory + $initialFiles = @($Projection.InitialRepositoryFiles) + foreach ($relative in $initialFiles) { + $physicalPath = Join-Path $physicalRepo ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if (-not (Test-Path -LiteralPath $physicalPath -PathType Leaf)) { + $logicalPath = Join-Path $logicalRepo ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if (Test-Path -LiteralPath $logicalPath -PathType Leaf) { + Remove-Item -LiteralPath $logicalPath -Force + } + } + } + foreach ($item in @(Get-ChildItem -LiteralPath $physicalRepo -Force -ErrorAction Stop)) { + Copy-Item -LiteralPath $item.FullName -Destination $logicalRepo -Recurse -Force + } +} + +function Remove-CodexExecutionProjection { + param([Parameter(Mandatory = $true)][object]$Projection) + + $root = [System.IO.Path]::GetFullPath([string]$Projection.Root) + $tempRoot = [System.IO.Path]::GetFullPath([System.IO.Path]::GetTempPath()) + if (-not (Test-PathInside -BasePath $tempRoot -CandidatePath $root)) { + throw "Refusing to remove Codex projection outside the temporary directory: '$root'." + } + if (Test-Path -LiteralPath $root) { Remove-Item -LiteralPath $root -Recurse -Force } +} + +function Invoke-CodexAppServer { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$Auth, + [bool]$SupportsProviderModelFallback = $false, + [int]$TimeoutSeconds = 900 + ) + + if ($Auth.Kind -ne 'subscription_file') { + throw 'Codex app-server subscription transport requires auth.json authentication.' + } + + $start = [DateTime]::UtcNow + $deadline = $start.AddSeconds($TimeoutSeconds) + $psi = [System.Diagnostics.ProcessStartInfo]::new() + $psi.FileName = $CommandInfo.FileName + $psi.WorkingDirectory = $Inputs.Run.WorkingDirectoryPath + $psi.UseShellExecute = $false + $psi.CreateNoWindow = $true + $psi.RedirectStandardInput = $true + $psi.RedirectStandardOutput = $true + $psi.RedirectStandardError = $true + foreach ($argument in @($CommandInfo.Prefix) + @('app-server', '--stdio', '-c', 'shell_environment_policy.inherit=none')) { [void]$psi.ArgumentList.Add([string]$argument) } + + $authHome = $null + $authOnlyHomeRemoved = $false + $parentEnvironment = $null + $process = [System.Diagnostics.Process]::new() + $writer = $null + $reader = $null + $stderrTask = $null + $events = [System.Collections.Generic.List[string]]::new() + $normalized = [System.Collections.Generic.List[string]]::new() + $threadId = $null + $threadSessionId = $null + $turnId = $null + $finalText = $null + $latestUsage = $null + $timedOut = $false + $transportFailure = $null + $threadReadFailure = $null + $turnCompleted = $false + $terminalTurn = $null + $stderr = '' + $actualExitCode = $null + $processStarted = $false + $threadStartRequest = $null + $threadStartResponse = $null + $turnStartRequest = $null + $turnStartResponse = $null + $turnStartRequests = [System.Collections.Generic.List[object]]::new() + $turnStartResponses = [System.Collections.Generic.List[object]]::new() + $turnRecords = [System.Collections.Generic.List[object]]::new() + $requestedInteractionTurns = [System.Collections.Generic.List[object]]::new() + if ($null -eq $Inputs.Run.Interaction) { + $requestedInteractionTurns.Add([ordered]@{ role = 'user'; source = 'prompt.md'; content = $null }) + } else { + foreach ($interactionTurn in @($Inputs.Run.Interaction.turns)) { $requestedInteractionTurns.Add($interactionTurn) } + } + $allTurnsCompleted = $true + $threadReadResponse = $null + $instructionSources = @() + $instructionSourcesObserved = $false + $modelReroutes = [System.Collections.Generic.List[object]]::new() + + try { + $authHome = New-CodexAuthOnlyHome -Auth $Auth + $parentEnvironment = New-RunnerEnvironment -Run $Inputs.Run -Additional @{ CODEX_HOME = $authHome.Path } + $psi.Environment.Clear() + foreach ($name in @($parentEnvironment.Keys)) { $psi.Environment[$name] = [string]$parentEnvironment[$name] } + $process.StartInfo = $psi + + if (-not $process.Start()) { throw 'Could not start Codex app-server.' } + $processStarted = $true + $writer = $process.StandardInput + $reader = $process.StandardOutput + $stderrTask = $process.StandardError.ReadToEndAsync() + + $writeMessage = { + param([Parameter(Mandatory = $true)][object]$Value) + $writer.WriteLine(($Value | ConvertTo-Json -Depth 50 -Compress)) + $writer.Flush() + } + $readMessage = { + $remaining = $deadline - [DateTime]::UtcNow + if ($remaining.TotalMilliseconds -le 0) { throw [TimeoutException]::new('Codex app-server timed out.') } + $readTask = $reader.ReadLineAsync() + $waitMilliseconds = [int][Math]::Min([int]::MaxValue, [Math]::Ceiling($remaining.TotalMilliseconds)) + if (-not $readTask.Wait($waitMilliseconds)) { throw [TimeoutException]::new('Codex app-server timed out.') } + $line = $readTask.GetAwaiter().GetResult() + if ($null -eq $line) { throw [EndOfStreamException]::new('Codex app-server closed stdout before the expected response.') } + $events.Add($line) + try { return ($line | ConvertFrom-Json -Depth 50) } catch { throw [FormatException]::new("Codex app-server emitted malformed JSON: $($_.Exception.Message)") } + } + $recordModelReroute = { + param([Parameter(Mandatory = $true)][object]$Message) + $reroute = Get-JsonProperty -Object $Message -Name 'params' -Default ([ordered]@{}) + $modelReroutes.Add($reroute) + $normalized.Add(([ordered]@{ type = 'model.rerouted'; from_model = Get-JsonProperty -Object $reroute -Name 'fromModel' -Default $null; to_model = Get-JsonProperty -Object $reroute -Name 'toModel' -Default $null; reason = Get-JsonProperty -Object $reroute -Name 'reason' -Default $null } | ConvertTo-Json -Compress)) + } + $waitForResponse = { + param([Parameter(Mandatory = $true)][int]$ExpectedId, [Parameter(Mandatory = $true)][string]$Operation) + while ($true) { + $message = & $readMessage + $messageId = Get-JsonProperty -Object $message -Name 'id' -Default $null + $method = [string](Get-JsonProperty -Object $message -Name 'method' -Default '') + if (-not [string]::IsNullOrWhiteSpace($method) -and $null -ne $messageId) { + throw "Codex app-server requested unsupported interactive method '$method'." + } + if ($method -eq 'model/rerouted') { + & $recordModelReroute $message + continue + } + if ($null -eq $messageId -or [int]$messageId -ne $ExpectedId) { continue } + $error = Get-JsonProperty -Object $message -Name 'error' -Default $null + if ($null -ne $error) { + $errorMessage = [string](Get-JsonProperty -Object $error -Name 'message' -Default ($error | ConvertTo-Json -Depth 20 -Compress)) + throw "Codex app-server $Operation failed: $errorMessage" + } + return $message + } + } + + $initializeRequest = 1 + & $writeMessage ([ordered]@{ + jsonrpc = '2.0' + id = $initializeRequest + method = 'initialize' + params = [ordered]@{ + clientInfo = [ordered]@{ name = 'codebelt-agentic-eval-runner'; title = 'Codebelt Eval Runner'; version = '0.9.1' } + capabilities = [ordered]@{ experimentalApi = $true } + } + }) + $null = & $waitForResponse $initializeRequest 'initialize' + & $writeMessage ([ordered]@{ jsonrpc = '2.0'; method = 'initialized' }) + + $threadRequest = 2 + $threadStartParams = [ordered]@{ + model = $Inputs.Profile.Model + cwd = $Inputs.Run.WorkingDirectoryPath + approvalPolicy = 'never' + # thread/start can persist project trust when it begins in a + # writable sandbox. Keep the ephemeral thread read-only and + # apply the intended workspace-write policy to the turn only. + sandbox = 'read-only' + ephemeral = $true + } + if ($SupportsProviderModelFallback) { $threadStartParams.allowProviderModelFallback = $false } + $threadStartRequest = [ordered]@{ jsonrpc = '2.0'; id = $threadRequest; method = 'thread/start'; params = $threadStartParams } + & $writeMessage $threadStartRequest + $threadStartResponse = & $waitForResponse $threadRequest 'thread/start' + $threadStartResult = Get-JsonProperty -Object $threadStartResponse -Name 'result' -Default $null + $threadMetadata = Get-JsonProperty -Object $threadStartResult -Name 'thread' -Default $null + $threadId = [string](Get-JsonProperty -Object $threadMetadata -Name 'id' -Default '') + $threadSessionId = [string](Get-JsonProperty -Object $threadMetadata -Name 'sessionId' -Default '') + if ([string]::IsNullOrWhiteSpace($threadId)) { throw 'Codex app-server thread/start returned no thread id.' } + $instructionSourcesObserved = Test-JsonProperty -Object $threadStartResult -Name 'instructionSources' + if ($instructionSourcesObserved) { $instructionSources = @(Get-JsonProperty -Object $threadStartResult -Name 'instructionSources' -Default @()) } + + for ($scriptedTurnIndex = 0; $scriptedTurnIndex -lt $requestedInteractionTurns.Count; $scriptedTurnIndex++) { + $turnCompleted = $false + $finalText = $null + $latestUsage = $null + $terminalTurn = $null + $turnStartedUtc = [DateTime]::UtcNow + $turnRequest = 3 + $scriptedTurnIndex + $promptText = Get-InteractionTurnText -Turn $requestedInteractionTurns[$scriptedTurnIndex] -RunData $Inputs.Run + $turnStartParams = [ordered]@{ + threadId = $threadId + input = @([ordered]@{ type = 'text'; text = $promptText }) + cwd = $Inputs.Run.WorkingDirectoryPath + model = $Inputs.Profile.Model + effort = $Inputs.Profile.ReasoningEffort + approvalPolicy = 'never' + sandboxPolicy = [ordered]@{ + type = 'workspaceWrite' + writableRoots = @($Inputs.Run.WorkingDirectoryPath) + networkAccess = $true + } + } + $turnStartRequest = [ordered]@{ jsonrpc = '2.0'; id = $turnRequest; method = 'turn/start'; params = $turnStartParams } + $turnStartRequests.Add($turnStartRequest) + & $writeMessage $turnStartRequest + $turnStartResponse = & $waitForResponse $turnRequest 'turn/start' + $turnStartResponses.Add($turnStartResponse) + $turnStartResult = Get-JsonProperty -Object $turnStartResponse -Name 'result' -Default $null + $turnId = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $turnStartResult -Name 'turn' -Default $null) -Name 'id' -Default '') + if ([string]::IsNullOrWhiteSpace($turnId)) { throw 'Codex app-server turn/start returned no turn id.' } + + while (-not $turnCompleted) { + $message = & $readMessage + $messageId = Get-JsonProperty -Object $message -Name 'id' -Default $null + $method = [string](Get-JsonProperty -Object $message -Name 'method' -Default '') + if (-not [string]::IsNullOrWhiteSpace($method) -and $null -ne $messageId) { + throw "Codex app-server requested unsupported interactive method '$method'." + } + switch ($method) { + 'thread/started' { + $normalized.Add(([ordered]@{ type = 'thread.started'; thread_id = Get-JsonProperty -Object (Get-JsonProperty -Object $message.params -Name 'thread' -Default $null) -Name 'id' -Default $null } | ConvertTo-Json -Compress)) + } + 'model/rerouted' { + & $recordModelReroute $message + } + 'item/completed' { + $item = $message.params.item + $itemType = [string]$item.type + $normalizedType = switch ($itemType) { + 'agentMessage' { 'agent_message' } + 'commandExecution' { 'command_execution' } + 'fileChange' { 'file_change' } + 'mcpToolCall' { 'mcp_tool_call' } + default { $itemType } + } + $normalizedItem = [ordered]@{ type = $normalizedType; id = Get-JsonProperty -Object $item -Name 'id' -Default $null } + if ($itemType -eq 'agentMessage') { + $normalizedItem.text = [string]$item.text + $finalText = [string]$item.text + } elseif ($itemType -eq 'commandExecution') { + $normalizedItem.command = Get-JsonProperty -Object $item -Name 'command' -Default $null + $normalizedItem.exit_code = Get-JsonProperty -Object $item -Name 'exitCode' -Default $null + $normalizedItem.aggregated_output = Get-JsonProperty -Object $item -Name 'aggregatedOutput' -Default $null + } elseif ($itemType -eq 'fileChange') { + $normalizedItem.changes = Get-JsonProperty -Object $item -Name 'changes' -Default @() + } else { + $normalizedItem.raw = $item + } + $normalized.Add(([ordered]@{ type = 'item.completed'; item = $normalizedItem } | ConvertTo-Json -Depth 40 -Compress)) + } + 'thread/tokenUsage/updated' { + $latestUsage = Get-JsonProperty -Object (Get-JsonProperty -Object $message.params -Name 'tokenUsage' -Default $null) -Name 'last' -Default $null + } + 'turn/completed' { + $completionParams = Get-JsonProperty -Object $message -Name 'params' -Default ([ordered]@{}) + $terminalTurn = Get-JsonProperty -Object $completionParams -Name 'turn' -Default $null + $completionThreadId = [string](Get-JsonProperty -Object $completionParams -Name 'threadId' -Default '') + $completedTurnId = [string](Get-JsonProperty -Object $terminalTurn -Name 'id' -Default '') + if ($completionThreadId -ne $threadId -or $completedTurnId -ne $turnId) { + throw 'Codex app-server turn/completed identified an unexpected thread or turn.' + } + if ([string]::IsNullOrWhiteSpace($finalText)) { + $turnItems = @($terminalTurn.items) + for ($itemIndex = $turnItems.Count - 1; $itemIndex -ge 0; $itemIndex--) { + if ([string]$turnItems[$itemIndex].type -eq 'agentMessage' -and -not [string]::IsNullOrWhiteSpace([string]$turnItems[$itemIndex].text)) { + $finalText = [string]$turnItems[$itemIndex].text + break + } + } + } + if ([string]$terminalTurn.status -eq 'failed') { + $errorMessage = [string](Get-JsonProperty -Object $terminalTurn.error -Name 'message' -Default 'Codex turn failed.') + $normalized.Add(([ordered]@{ type = 'turn.failed'; error = $errorMessage } | ConvertTo-Json -Compress)) + } elseif ([string]$terminalTurn.status -eq 'interrupted') { + $normalized.Add(([ordered]@{ type = 'turn.failed'; error = 'Codex turn was interrupted.' } | ConvertTo-Json -Compress)) + } else { + $usage = $null + if ($null -ne $latestUsage) { + $usage = [ordered]@{ + input_tokens = Get-JsonProperty -Object $latestUsage -Name 'inputTokens' -Default $null + cached_input_tokens = Get-JsonProperty -Object $latestUsage -Name 'cachedInputTokens' -Default $null + output_tokens = Get-JsonProperty -Object $latestUsage -Name 'outputTokens' -Default $null + reasoning_output_tokens = Get-JsonProperty -Object $latestUsage -Name 'reasoningOutputTokens' -Default $null + } + } + $normalized.Add(([ordered]@{ type = 'turn.completed'; usage = $usage } | ConvertTo-Json -Depth 20 -Compress)) + } + $turnCompleted = $true + } + 'error' { + $errorMessage = [string](Get-JsonProperty -Object $message.params -Name 'message' -Default 'Codex app-server emitted an error.') + throw $errorMessage + } + } + } + + $turnRecords.Add([ordered]@{ sequence = ($scriptedTurnIndex * 2) + 1; role = 'user'; content_sha256 = Get-Sha256HexFromBytes -Bytes ([System.Text.UTF8Encoding]::new($false).GetBytes($promptText)); session_id = $threadId; timestamp_utc = Format-UtcTimestamp -Value $turnStartedUtc }) + $turnRecords.Add([ordered]@{ sequence = ($scriptedTurnIndex * 2) + 2; role = 'assistant'; text = if ($null -eq $finalText) { '' } else { [string]$finalText }; session_id = $threadId; timestamp_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow) }) + if (-not $turnCompleted -or [string](Get-JsonProperty -Object $terminalTurn -Name 'status' -Default '') -ne 'completed') { + $allTurnsCompleted = $false + break + } + } + + # The installed schema exposes thread/read after completion. Use it as + # a second observation of ephemeral identity, cwd, and session metadata + # when the server provides the response; never reconstruct it locally. + $threadReadRequest = 3 + $requestedInteractionTurns.Count + 1 + & $writeMessage ([ordered]@{ jsonrpc = '2.0'; id = $threadReadRequest; method = 'thread/read'; params = [ordered]@{ threadId = $threadId; includeTurns = $true } }) + try { + $threadReadResponse = & $waitForResponse $threadReadRequest 'thread/read' + } catch { + $threadReadFailure = $_.Exception.Message + $normalized.Add(([ordered]@{ type = 'thread.read.unavailable'; message = $threadReadFailure } | ConvertTo-Json -Compress)) + } + } catch [TimeoutException] { + $timedOut = $true + } catch { + $transportFailure = $_.Exception.Message + $normalized.Add(([ordered]@{ type = 'error'; message = $transportFailure } | ConvertTo-Json -Compress)) + } finally { + if ($null -ne $writer) { try { $writer.Close() } catch { } } + if ($processStarted) { + try { + if (-not $process.HasExited -and -not $process.WaitForExit(2000)) { + $process.Kill($true) + # Never use an unbounded wait in cleanup. A broken child + # or inherited pipe must not hold the runner forever. + if (-not $process.HasExited) { [void]$process.WaitForExit(5000) } + } + if ($process.HasExited) { $actualExitCode = $process.ExitCode } + } catch { } + } + if ($null -ne $stderrTask) { + try { + # A descendant that inherits stderr can keep this task open + # after the app-server has been terminated. Drain only within + # the shared finite cleanup grace; never turn cleanup into an + # unbounded GetResult() wait. + if (Wait-RunnerTaskBounded -Task $stderrTask -TimeoutMilliseconds 5000) { + $stderr = [string]$stderrTask.GetAwaiter().GetResult() + } else { + $stderr = 'Codex app-server stderr drain exceeded bounded cleanup grace.' + } + } catch { $stderr = $_.Exception.Message } + } + $process.Dispose() + if ($null -ne $authHome -and (Test-Path -LiteralPath $authHome.Path)) { + Remove-Item -LiteralPath $authHome.Path -Recurse -Force -ErrorAction SilentlyContinue + } + $authOnlyHomeRemoved = $null -eq $authHome -or -not (Test-Path -LiteralPath $authHome.Path) + } + + $finish = [DateTime]::UtcNow + $exitCode = if ($timedOut) { $null } elseif ($turnCompleted -and $null -eq $transportFailure) { 0 } elseif ($null -ne $actualExitCode -and $actualExitCode -ne 0) { $actualExitCode } else { 1 } + return [pscustomobject]@{ + Stdout = [string]::Join("`n", $normalized) + RawStdout = [string]::Join("`n", $events) + Stderr = $stderr + ExitCode = $exitCode + TimedOut = $timedOut + StartedUtc = $start + FinishedUtc = $finish + DurationSeconds = [Math]::Round(($finish - $start).TotalSeconds, 3) + FinalText = $finalText + ThreadId = $threadId + ThreadSessionId = $threadSessionId + TurnId = $turnId + TurnCompleted = $allTurnsCompleted + LastTurnCompleted = $turnCompleted + TerminalTurn = $terminalTurn + ThreadStartRequest = $threadStartRequest + ThreadStartResponse = $threadStartResponse + TurnStartRequest = $turnStartRequest + TurnStartResponse = $turnStartResponse + TurnStartRequests = @($turnStartRequests.ToArray()) + TurnStartResponses = @($turnStartResponses.ToArray()) + TurnRecords = @($turnRecords.ToArray()) + RequestedTurnCount = $requestedInteractionTurns.Count + AllTurnsCompleted = $allTurnsCompleted + ThreadReadResponse = $threadReadResponse + ThreadReadFailure = $threadReadFailure + InstructionSources = @($instructionSources) + InstructionSourcesObserved = $instructionSourcesObserved + ModelReroutes = @($modelReroutes.ToArray()) + PromptInputSha256 = if ($turnStartRequests.Count -gt 0) { Get-Sha256HexFromBytes -Bytes ([System.Text.Encoding]::UTF8.GetBytes([string]$turnStartRequests[0].params.input[0].text)) } else { $null } + ObservedModel = if ($null -ne $threadStartResponse) { [string](Get-JsonProperty -Object (Get-JsonProperty -Object $threadStartResponse -Name 'result' -Default $null) -Name 'model' -Default '') } else { '' } + ObservedWorkingDirectory = if ($null -ne $threadStartResponse) { [string](Get-JsonProperty -Object (Get-JsonProperty -Object $threadStartResponse -Name 'result' -Default $null) -Name 'cwd' -Default '') } else { '' } + ObservedEphemeral = if ($null -ne $threadStartResponse) { [bool](Get-JsonProperty -Object (Get-JsonProperty -Object (Get-JsonProperty -Object $threadStartResponse -Name 'result' -Default $null) -Name 'thread' -Default $null) -Name 'ephemeral' -Default $false) } else { $false } + AuthOnlyHome = $null -ne $authHome -and [bool]$authHome.AuthOnly + AuthOnlyHomeRemoved = $authOnlyHomeRemoved + WorkerHome = if ($null -ne $parentEnvironment) { [string]$parentEnvironment.HOME } else { '' } + TransportFailure = $transportFailure + } +} + +function Get-CodexHelpResult { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs, + [string[]]$Arguments = @('--ask-for-approval', 'never', 'exec', '--help') + ) + + $environment = New-RunnerEnvironment -Run $Inputs.Run + return Invoke-CodexCli -CommandInfo $CommandInfo -Arguments $Arguments -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 +} + +function Get-CodexSchemaProperty { + param( + [AllowNull()][object]$Schema, + [Parameter(Mandatory = $true)][string]$PropertyName + ) + + if ($null -eq $Schema) { return $null } + return Get-JsonProperty -Object (Get-JsonProperty -Object $Schema -Name 'properties' -Default $null) -Name $PropertyName -Default $null +} + +function Get-CodexSchemaReference { + param([AllowNull()][object]$Schema) + + $direct = [string](Get-JsonProperty -Object $Schema -Name '$ref' -Default '') + if (-not [string]::IsNullOrWhiteSpace($direct)) { return $direct } + foreach ($alternative in @(Get-JsonProperty -Object $Schema -Name 'anyOf' -Default @())) { + $reference = [string](Get-JsonProperty -Object $alternative -Name '$ref' -Default '') + if (-not [string]::IsNullOrWhiteSpace($reference)) { return $reference } + } + foreach ($alternative in @(Get-JsonProperty -Object $Schema -Name 'allOf' -Default @())) { + $reference = [string](Get-JsonProperty -Object $alternative -Name '$ref' -Default '') + if (-not [string]::IsNullOrWhiteSpace($reference)) { return $reference } + } + return '' +} + +function Test-CodexSchemaRequiredProperty { + param( + [AllowNull()][object]$Schema, + [Parameter(Mandatory = $true)][string]$PropertyName, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[string]]$Errors, + [Parameter(Mandatory = $true)][string]$SchemaName + ) + + if ($null -eq $Schema) { + [void]$Errors.Add("$SchemaName schema is missing.") + return $null + } + $property = Get-CodexSchemaProperty -Schema $Schema -PropertyName $PropertyName + if ($null -eq $property) { + [void]$Errors.Add("$SchemaName.properties.$PropertyName is missing.") + return $null + } + $required = @(Get-JsonProperty -Object $Schema -Name 'required' -Default @()) | ForEach-Object { [string]$_ } + if ($required -notcontains $PropertyName) { + [void]$Errors.Add("$SchemaName.required does not contain '$PropertyName'.") + } + return $property +} + +function Get-CodexSchemaDefinition { + param( + [AllowNull()][object]$Schema, + [Parameter(Mandatory = $true)][string]$DefinitionName, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[string]]$Errors, + [Parameter(Mandatory = $true)][string]$SchemaName, + [AllowNull()][object]$Definitions = $null + ) + + if ($null -ne $Definitions) { + $definitions = $Definitions + } elseif ($null -ne $Schema) { + $definitions = Get-JsonProperty -Object $Schema -Name 'definitions' -Default $null + } else { + $definitions = $null + } + $definition = Get-JsonProperty -Object $definitions -Name $DefinitionName -Default $null + if ($null -eq $definition) { [void]$Errors.Add("$SchemaName.definitions.$DefinitionName is missing.") } + return $definition +} + +function Get-CodexSchemaTypeNames { + param([AllowNull()][object]$Schema) + + $type = Get-JsonProperty -Object $Schema -Name 'type' -Default $null + if ($null -eq $type) { return @() } + return @($type | ForEach-Object { [string]$_ }) +} + +function Test-CodexSchemaType { + param( + [AllowNull()][object]$Schema, + [Parameter(Mandatory = $true)][string]$TypeName + ) + + return @(Get-CodexSchemaTypeNames -Schema $Schema) -contains $TypeName +} + +function Resolve-CodexSchemaSource { + param( + [Parameter(Mandatory = $true)][string]$SchemaDirectory, + [Parameter(Mandatory = $true)][string[]]$RequiredNames, + [AllowEmptyCollection()][string[]]$OptionalNames = @() + ) + + $files = @(Get-ChildItem -LiteralPath $SchemaDirectory -Recurse -File -ErrorAction Stop | Sort-Object FullName) + $bundleCandidates = @($files | Where-Object { $_.Name -ceq 'codex_app_server_protocol.v2.schemas.json' }) + $schemaCache = @{} + $definitions = $null + $errors = [System.Collections.Generic.List[string]]::new() + $missingRequired = [System.Collections.Generic.List[string]]::new() + $missingOptional = [System.Collections.Generic.List[string]]::new() + $sourcePath = $null + $sourceKind = $null + + if ($bundleCandidates.Count -gt 1) { + [void]$errors.Add("Installed Codex app-server has multiple v2 schema bundles: $([string]::Join(', ', @($bundleCandidates | ForEach-Object { $_.FullName }))).") + } elseif ($bundleCandidates.Count -eq 1) { + $sourcePath = [string]$bundleCandidates[0].FullName + $sourceKind = 'aggregate_v2_bundle' + try { + $bundle = [System.IO.File]::ReadAllText($sourcePath, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 + $definitions = Get-JsonProperty -Object $bundle -Name 'definitions' -Default $null + if ($null -eq $definitions) { + [void]$errors.Add("Installed Codex v2 schema bundle '$sourcePath' has no definitions object.") + foreach ($schemaName in @($RequiredNames)) { [void]$missingRequired.Add($schemaName) } + foreach ($schemaName in @($OptionalNames)) { [void]$missingOptional.Add($schemaName) } + } else { + foreach ($schemaName in @($RequiredNames) + @($OptionalNames)) { + $definition = Get-JsonProperty -Object $definitions -Name $schemaName -Default $null + if ($null -eq $definition) { + if ($RequiredNames -contains $schemaName) { [void]$missingRequired.Add($schemaName) } else { [void]$missingOptional.Add($schemaName) } + } else { + $schemaCache[$schemaName] = $definition + } + } + } + } catch { + [void]$errors.Add("Installed Codex v2 schema bundle '$sourcePath' is not valid JSON: $($_.Exception.Message)") + } + } else { + $sourceKind = 'recursive_individual_files' + foreach ($schemaName in @($RequiredNames) + @($OptionalNames)) { + $matches = @($files | Where-Object { $_.Name -ceq ("{0}.json" -f $schemaName) }) + if ($matches.Count -eq 0) { + if ($RequiredNames -contains $schemaName) { [void]$missingRequired.Add($schemaName) } else { [void]$missingOptional.Add($schemaName) } + continue + } + if ($matches.Count -gt 1) { + [void]$errors.Add("Installed Codex app-server has multiple unambiguous schema files for '$schemaName': $([string]::Join(', ', @($matches | ForEach-Object { $_.FullName }))).") + continue + } + try { + $schemaCache[$schemaName] = [System.IO.File]::ReadAllText($matches[0].FullName, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 + if ($null -eq $schemaCache[$schemaName]) { [void]$errors.Add("Installed app-server schema file '$($matches[0].FullName)' is empty.") } + } catch { + [void]$errors.Add("Installed app-server schema file '$($matches[0].FullName)' is not valid JSON: $($_.Exception.Message)") + } + } + } + + if ($missingRequired.Count -gt 0) { + $missingText = if ($missingRequired.Count -eq 1) { + "Installed Codex app-server schema is missing required v2 schema: $($missingRequired[0])." + } else { + "Installed Codex app-server schemas are missing required v2 schemas: $([string]::Join(', ', @($missingRequired)))." + } + [void]$errors.Insert(0, $missingText) + } + if ($missingOptional.Count -eq 1) { + [void]$errors.Add("Installed Codex app-server schema is missing one member of the supplemental thread/read v2 schema pair: $($missingOptional[0]).") + } + + return [pscustomobject]@{ + Available = $errors.Count -eq 0 -and $missingRequired.Count -eq 0 + Detail = [string]::Join(' ', @($errors)) + Schemas = $schemaCache + Definitions = $definitions + SourcePath = $sourcePath + SourceKind = $sourceKind + Missing = @($missingRequired) + SupplementalMissing = @($missingOptional) + SupplementalAvailable = $missingOptional.Count -eq 0 + } +} + +function Get-CodexNativeWorkerProbe { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs + ) + + $environment = New-RunnerEnvironment -Run $Inputs.Run + $help = Invoke-CodexCli -CommandInfo $CommandInfo -Arguments @('app-server', '--help') -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 + if ($help.TimedOut -or $help.ExitCode -ne 0) { + return [pscustomobject]@{ Available = $false; Detail = "codex app-server --help failed with exit status $($help.ExitCode)." } + } + $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) + if ($helpText -notmatch 'generate-json-schema') { + return [pscustomobject]@{ Available = $false; Detail = 'The installed Codex CLI does not advertise app-server schema generation.' } + } + $features = Invoke-CodexCli -CommandInfo $CommandInfo -Arguments @('features', 'list') -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 + if ($features.TimedOut -or $features.ExitCode -ne 0 -or ([string]::Join("`n", @($features.Stdout, $features.Stderr)) -notmatch '(?im)multi_agent\s+stable\s+true')) { + return [pscustomobject]@{ Available = $false; Detail = 'The installed Codex CLI did not report the stable multi_agent feature required for native child workers.' } + } + + $schemaRelativeDirectory = Join-Path ([System.IO.Path]::GetRelativePath($Inputs.Run.WorkingDirectoryPath, $Inputs.Run.HomeDirectoryPath)) 'evidence/codex-app-server-schema' + $schemaDirectory = [System.IO.Path]::GetFullPath((Join-Path $Inputs.Run.WorkingDirectoryPath $schemaRelativeDirectory)) + New-Item -ItemType Directory -Path $schemaDirectory -Force | Out-Null + $schemaProcess = Invoke-CodexCli -CommandInfo $CommandInfo -Arguments @('app-server', 'generate-json-schema', "--out=$schemaRelativeDirectory") -Inputs $Inputs -Environment $environment -TimeoutSeconds 60 + if ($schemaProcess.TimedOut -or $schemaProcess.ExitCode -ne 0) { + return [pscustomobject]@{ Available = $false; Detail = "Codex app-server schema generation failed with exit status $($schemaProcess.ExitCode): $([string]::Join(' ', @($schemaProcess.Stdout, $schemaProcess.Stderr)))." } + } + $requiredSchemaNames = @('ThreadStartParams', 'ThreadStartResponse', 'TurnStartParams', 'TurnStartResponse', 'ModelReroutedNotification') + $supplementalSchemaNames = @('ThreadReadParams', 'ThreadReadResponse') + $schemaResolution = Resolve-CodexSchemaSource -SchemaDirectory $schemaDirectory -RequiredNames $requiredSchemaNames -OptionalNames $supplementalSchemaNames + if (-not $schemaResolution.Available) { + return [pscustomobject]@{ + Available = $false + Detail = [string]$schemaResolution.Detail + SupportsProviderModelFallback = $false + SchemaDirectory = $schemaDirectory + SchemaSource = [string]$schemaResolution.SourcePath + SchemaSourceKind = [string]$schemaResolution.SourceKind + MissingSchemas = @($schemaResolution.Missing) + SupplementalMissingSchemas = @($schemaResolution.SupplementalMissing) + } + } + + $errors = [System.Collections.Generic.List[string]]::new() + $schemaCache = $schemaResolution.Schemas + $schemaDefinitions = $schemaResolution.Definitions + $missingResolved = @($requiredSchemaNames | Where-Object { $null -eq (Get-JsonProperty -Object $schemaCache -Name $_ -Default $null) }) + if ($missingResolved.Count -gt 0) { + return [pscustomobject]@{ + Available = $false + Detail = if ($missingResolved.Count -eq 1) { "Installed Codex app-server schema is missing required v2 schema: $($missingResolved[0])." } else { "Installed Codex app-server schemas are missing required v2 schemas: $([string]::Join(', ', @($missingResolved)))." } + SupportsProviderModelFallback = $false + SchemaDirectory = $schemaDirectory + SchemaSource = [string]$schemaResolution.SourcePath + SchemaSourceKind = [string]$schemaResolution.SourceKind + MissingSchemas = @($missingResolved) + SupplementalMissingSchemas = @($schemaResolution.SupplementalMissing) + } + } + + $threadStartParams = $schemaCache['ThreadStartParams'] + $threadStartResponse = $schemaCache['ThreadStartResponse'] + $turnStartParams = $schemaCache['TurnStartParams'] + $turnStartResponse = $schemaCache['TurnStartResponse'] + $threadReadParams = $schemaCache['ThreadReadParams'] + $threadReadResponse = $schemaCache['ThreadReadResponse'] + $modelRerouted = $schemaCache['ModelReroutedNotification'] + $threadReadSchemaAvailable = [bool]$schemaResolution.SupplementalAvailable + + foreach ($field in @('model', 'cwd', 'approvalPolicy', 'sandbox', 'ephemeral')) { + if ($null -eq (Get-CodexSchemaProperty -Schema $threadStartParams -PropertyName $field)) { [void]$errors.Add("ThreadStartParams.properties.$field is missing.") } + } + foreach ($field in @('model', 'cwd')) { + if (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $threadStartParams -PropertyName $field) -TypeName 'string')) { [void]$errors.Add("ThreadStartParams.properties.$field must include type string.") } + } + if (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $threadStartParams -PropertyName 'ephemeral') -TypeName 'boolean')) { [void]$errors.Add('ThreadStartParams.properties.ephemeral must include type boolean.') } + if ((Get-CodexSchemaReference -Schema (Get-CodexSchemaProperty -Schema $threadStartParams -PropertyName 'approvalPolicy')) -ne '#/definitions/AskForApproval') { [void]$errors.Add('ThreadStartParams.approvalPolicy must reference definitions.AskForApproval.') } + $sandboxProperty = Get-CodexSchemaProperty -Schema $threadStartParams -PropertyName 'sandbox' + $sandboxReference = Get-CodexSchemaReference -Schema $sandboxProperty + $sandboxDefinition = if ($sandboxReference -eq '#/definitions/SandboxMode') { Get-CodexSchemaDefinition -Schema $threadStartParams -DefinitionName 'SandboxMode' -Definitions $schemaDefinitions -Errors $errors -SchemaName 'ThreadStartParams' } else { $null } + if ($sandboxReference -ne '#/definitions/SandboxMode' -or $null -eq $sandboxDefinition) { [void]$errors.Add('ThreadStartParams.sandbox must reference definitions.SandboxMode.') } + $sandboxEnum = @((Get-JsonProperty -Object $sandboxDefinition -Name 'enum' -Default @()) | ForEach-Object { [string]$_ }) + $requiredSandboxModes = @('read-only', 'workspace-write', 'danger-full-access') + foreach ($mode in $requiredSandboxModes) { + if ($sandboxEnum -notcontains $mode) { [void]$errors.Add("ThreadStartParams.definitions.SandboxMode.enum is missing '$mode'.") } + } + $fallbackProperty = Get-CodexSchemaProperty -Schema $threadStartParams -PropertyName 'allowProviderModelFallback' + $supportsProviderModelFallback = $null -ne $fallbackProperty + if ($supportsProviderModelFallback -and -not (Test-CodexSchemaType -Schema $fallbackProperty -TypeName 'boolean')) { + [void]$errors.Add('ThreadStartParams.allowProviderModelFallback must include type boolean when installed.') + } + + foreach ($field in @('model', 'cwd', 'thread')) { [void](Test-CodexSchemaRequiredProperty -Schema $threadStartResponse -PropertyName $field -Errors $errors -SchemaName 'ThreadStartResponse') } + $instructionSourcesProperty = Get-CodexSchemaProperty -Schema $threadStartResponse -PropertyName 'instructionSources' + if ($null -eq $instructionSourcesProperty) { + [void]$errors.Add('ThreadStartResponse.properties.instructionSources is missing.') + } elseif (-not (Test-CodexSchemaType -Schema $instructionSourcesProperty -TypeName 'array')) { + [void]$errors.Add('ThreadStartResponse.instructionSources must be an array when present.') + } elseif ((Get-CodexSchemaReference -Schema (Get-JsonProperty -Object $instructionSourcesProperty -Name 'items' -Default $null)) -ne '#/definitions/LegacyAppPathString') { + [void]$errors.Add('ThreadStartResponse.instructionSources.items must reference definitions.LegacyAppPathString.') + } + $threadReference = Get-CodexSchemaReference -Schema (Get-CodexSchemaProperty -Schema $threadStartResponse -PropertyName 'thread') + if ($threadReference -ne '#/definitions/Thread') { [void]$errors.Add('ThreadStartResponse.thread must reference definitions.Thread.') } + if (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $threadStartResponse -PropertyName 'model') -TypeName 'string')) { [void]$errors.Add('ThreadStartResponse.model must include type string.') } + if ((Get-CodexSchemaReference -Schema (Get-CodexSchemaProperty -Schema $threadStartResponse -PropertyName 'cwd')) -ne '#/definitions/AbsolutePathBuf') { [void]$errors.Add('ThreadStartResponse.cwd must reference definitions.AbsolutePathBuf.') } + $threadDefinition = Get-CodexSchemaDefinition -Schema $threadStartResponse -DefinitionName 'Thread' -Definitions $schemaDefinitions -Errors $errors -SchemaName 'ThreadStartResponse' + foreach ($field in @('id', 'cwd', 'ephemeral', 'sessionId')) { [void](Test-CodexSchemaRequiredProperty -Schema $threadDefinition -PropertyName $field -Errors $errors -SchemaName 'ThreadStartResponse.definitions.Thread') } + if (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $threadDefinition -PropertyName 'id') -TypeName 'string')) { [void]$errors.Add('ThreadStartResponse.definitions.Thread.id must include type string.') } + if ((Get-CodexSchemaReference -Schema (Get-CodexSchemaProperty -Schema $threadDefinition -PropertyName 'cwd')) -ne '#/definitions/AbsolutePathBuf') { [void]$errors.Add('ThreadStartResponse.definitions.Thread.cwd must reference definitions.AbsolutePathBuf.') } + if (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $threadDefinition -PropertyName 'ephemeral') -TypeName 'boolean')) { [void]$errors.Add('ThreadStartResponse.definitions.Thread.ephemeral must include type boolean.') } + if (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $threadDefinition -PropertyName 'sessionId') -TypeName 'string')) { [void]$errors.Add('ThreadStartResponse.definitions.Thread.sessionId must include type string.') } + + foreach ($field in @('input', 'threadId')) { [void](Test-CodexSchemaRequiredProperty -Schema $turnStartParams -PropertyName $field -Errors $errors -SchemaName 'TurnStartParams') } + if (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $turnStartParams -PropertyName 'input') -TypeName 'array')) { [void]$errors.Add('TurnStartParams.input must be an array.') } + if ((Get-CodexSchemaReference -Schema (Get-JsonProperty -Object (Get-CodexSchemaProperty -Schema $turnStartParams -PropertyName 'input') -Name 'items' -Default $null)) -ne '#/definitions/UserInput') { [void]$errors.Add('TurnStartParams.input.items must reference definitions.UserInput.') } + if (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $turnStartParams -PropertyName 'threadId') -TypeName 'string')) { [void]$errors.Add('TurnStartParams.threadId must include type string.') } + foreach ($field in @('cwd', 'model')) { + if ($null -eq (Get-CodexSchemaProperty -Schema $turnStartParams -PropertyName $field)) { [void]$errors.Add("TurnStartParams.properties.$field is missing.") } + elseif (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $turnStartParams -PropertyName $field) -TypeName 'string')) { [void]$errors.Add("TurnStartParams.properties.$field must include type string.") } + } + foreach ($field in @( + [pscustomobject]@{ Name = 'effort'; Reference = '#/definitions/ReasoningEffort' } + [pscustomobject]@{ Name = 'approvalPolicy'; Reference = '#/definitions/AskForApproval' } + [pscustomobject]@{ Name = 'sandboxPolicy'; Reference = '#/definitions/SandboxPolicy' } + )) { + if ($null -eq (Get-CodexSchemaProperty -Schema $turnStartParams -PropertyName $field.Name)) { [void]$errors.Add("TurnStartParams.properties.$($field.Name) is missing.") } + elseif ((Get-CodexSchemaReference -Schema (Get-CodexSchemaProperty -Schema $turnStartParams -PropertyName $field.Name)) -ne $field.Reference) { [void]$errors.Add("TurnStartParams.$($field.Name) must reference $($field.Reference.Replace('#/definitions/', 'definitions.')).") } + } + $sandboxPolicyDefinition = Get-CodexSchemaDefinition -Schema $turnStartParams -DefinitionName 'SandboxPolicy' -Definitions $schemaDefinitions -Errors $errors -SchemaName 'TurnStartParams' + $workspaceWritePolicy = @((Get-JsonProperty -Object $sandboxPolicyDefinition -Name 'oneOf' -Default @()) | Where-Object { + $typeProperty = Get-JsonProperty -Object (Get-JsonProperty -Object $_ -Name 'properties' -Default $null) -Name 'type' -Default $null + @((Get-JsonProperty -Object $typeProperty -Name 'enum' -Default @())) -contains 'workspaceWrite' + }) | Select-Object -First 1 + if ($null -eq $workspaceWritePolicy) { + [void]$errors.Add('TurnStartParams.definitions.SandboxPolicy must advertise the workspaceWrite policy used by the runner.') + } else { + $writableRoots = Get-JsonProperty -Object (Get-JsonProperty -Object $workspaceWritePolicy -Name 'properties' -Default $null) -Name 'writableRoots' -Default $null + $networkAccess = Get-JsonProperty -Object (Get-JsonProperty -Object $workspaceWritePolicy -Name 'properties' -Default $null) -Name 'networkAccess' -Default $null + if ($null -eq $writableRoots -or -not (Test-CodexSchemaType -Schema $writableRoots -TypeName 'array')) { [void]$errors.Add('TurnStartParams.definitions.SandboxPolicy.workspaceWrite.writableRoots must be an array.') } + if ($null -eq $networkAccess -or -not (Test-CodexSchemaType -Schema $networkAccess -TypeName 'boolean')) { [void]$errors.Add('TurnStartParams.definitions.SandboxPolicy.workspaceWrite.networkAccess must be boolean.') } + } + $turnProperty = Test-CodexSchemaRequiredProperty -Schema $turnStartResponse -PropertyName 'turn' -Errors $errors -SchemaName 'TurnStartResponse' + $turnReference = Get-CodexSchemaReference -Schema $turnProperty + if ($turnReference -ne '#/definitions/Turn') { [void]$errors.Add('TurnStartResponse.turn must reference definitions.Turn.') } + $turnDefinition = Get-CodexSchemaDefinition -Schema $turnStartResponse -DefinitionName 'Turn' -Definitions $schemaDefinitions -Errors $errors -SchemaName 'TurnStartResponse' + foreach ($field in @('id', 'items', 'status')) { [void](Test-CodexSchemaRequiredProperty -Schema $turnDefinition -PropertyName $field -Errors $errors -SchemaName 'TurnStartResponse.definitions.Turn') } + if (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $turnDefinition -PropertyName 'id') -TypeName 'string')) { [void]$errors.Add('TurnStartResponse.definitions.Turn.id must include type string.') } + if (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $turnDefinition -PropertyName 'items') -TypeName 'array')) { [void]$errors.Add('TurnStartResponse.definitions.Turn.items must include type array.') } + if ((Get-CodexSchemaReference -Schema (Get-CodexSchemaProperty -Schema $turnDefinition -PropertyName 'status')) -ne '#/definitions/TurnStatus') { [void]$errors.Add('TurnStartResponse.definitions.Turn.status must reference definitions.TurnStatus.') } + + if ($threadReadSchemaAvailable) { + [void](Test-CodexSchemaRequiredProperty -Schema $threadReadParams -PropertyName 'threadId' -Errors $errors -SchemaName 'ThreadReadParams') + if (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $threadReadParams -PropertyName 'threadId') -TypeName 'string')) { [void]$errors.Add('ThreadReadParams.threadId must include type string.') } + $includeTurnsProperty = Get-CodexSchemaProperty -Schema $threadReadParams -PropertyName 'includeTurns' + if ($null -eq $includeTurnsProperty -or -not (Test-CodexSchemaType -Schema $includeTurnsProperty -TypeName 'boolean')) { [void]$errors.Add('ThreadReadParams.includeTurns must be an optional boolean.') } + $threadReadProperty = Test-CodexSchemaRequiredProperty -Schema $threadReadResponse -PropertyName 'thread' -Errors $errors -SchemaName 'ThreadReadResponse' + if ((Get-CodexSchemaReference -Schema $threadReadProperty) -ne '#/definitions/Thread') { [void]$errors.Add('ThreadReadResponse.thread must reference definitions.Thread.') } + } + foreach ($field in @('threadId', 'turnId', 'fromModel', 'toModel', 'reason')) { [void](Test-CodexSchemaRequiredProperty -Schema $modelRerouted -PropertyName $field -Errors $errors -SchemaName 'ModelReroutedNotification') } + foreach ($field in @('threadId', 'turnId', 'fromModel', 'toModel')) { + if (-not (Test-CodexSchemaType -Schema (Get-CodexSchemaProperty -Schema $modelRerouted -PropertyName $field) -TypeName 'string')) { [void]$errors.Add("ModelReroutedNotification.$field must include type string.") } + } + if ((Get-CodexSchemaReference -Schema (Get-CodexSchemaProperty -Schema $modelRerouted -PropertyName 'reason')) -ne '#/definitions/ModelRerouteReason') { [void]$errors.Add('ModelReroutedNotification.reason must reference definitions.ModelRerouteReason.') } + + if ($errors.Count -gt 0) { + return [pscustomobject]@{ + Available = $false + Detail = 'Installed Codex app-server schema failed structural validation: ' + [string]::Join(' ', @($errors)) + SupportsProviderModelFallback = $supportsProviderModelFallback + SchemaDirectory = $schemaDirectory + SchemaSource = [string]$schemaResolution.SourcePath + SchemaSourceKind = [string]$schemaResolution.SourceKind + SandboxModes = @($sandboxEnum) + ThreadReadSchemaAvailable = $threadReadSchemaAvailable + } + } + $schemaDetail = if ($threadReadSchemaAvailable) { + 'Codex multi_agent is stable and the installed v2 app-server schema structurally proves the consumed thread/start, turn/start, thread/read, and model/rerouted fields.' + } else { + 'Codex multi_agent is stable and the installed v2 app-server schema structurally proves the consumed thread/start, turn/start, and model/rerouted fields; thread/read is supplemental and not advertised.' + } + if ($supportsProviderModelFallback) { + $schemaDetail += ' allowProviderModelFallback is supported and will be sent as false; reroute notifications remain fail-closed.' + } else { + $schemaDetail += ' allowProviderModelFallback is not exposed by the installed protocol; reroute notifications remain fail-closed.' + } + return [pscustomobject]@{ + Available = $true + Detail = $schemaDetail + SupportsProviderModelFallback = $supportsProviderModelFallback + SchemaDirectory = $schemaDirectory + SchemaSource = [string]$schemaResolution.SourcePath + SchemaSourceKind = [string]$schemaResolution.SourceKind + SandboxModes = @($sandboxEnum) + ThreadReadSchemaAvailable = $threadReadSchemaAvailable + } +} + +function Resolve-SandboxCommand { + param([Parameter(Mandatory = $true)][string]$Name) + + return Resolve-ExternalCommand -Name $Name +} + +function Get-CodexDescriptor { + $copy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $copy[$key] = $descriptor[$key] } + $commandInfo = Resolve-ExternalCommand -Name 'codex' + $version = 'unavailable' + if ($null -ne $commandInfo) { + $observation = Get-ExternalCommandVersion -CommandInfo $commandInfo + $version = [string]$observation.Version + } + $copy.harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = $version } + return $copy +} + +function New-CodexCliArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][string]$LastResponsePath, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform = (Get-PlatformName) + ) + + $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $outputArgument = Get-SandboxVisiblePath -HostPath $LastResponsePath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $arguments = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--ask-for-approval', 'never', 'exec', '--ephemeral', '--ignore-user-config', '--ignore-rules', '--skip-git-repo-check', '--json', '--color', 'never', '--cd', $directoryArgument, '--model', $Inputs.Profile.Model, '--sandbox', 'workspace-write', '--config', 'shell_environment_policy.inherit=none', '--output-last-message', $outputArgument)) { + $arguments.Add([string]$argument) + } + if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { + $arguments.Add('-c') + $arguments.Add("model_reasoning_effort=$($Inputs.Profile.ReasoningEffort)") + } + $arguments.Add('-') + return @($arguments) +} + +function Get-CodexCapabilityMap { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [bool]$HardFilesystemConfinement = $false, + [bool]$NativeWorkerAvailable = $true, + [string]$AuthKind = '' + ) + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) + } + $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } + $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + foreach ($name in @('native_worker_delegation', 'delegated_worker_full_capability', 'delegated_worker_model_lock', 'delegated_worker_working_directory', 'delegated_worker_result_capture', 'delegated_worker_capacity_signal')) { + # The app-server probe proves that the native surface is available; + # only the child terminal evidence can prove the selected controls. + $capabilities[$name] = if ($NativeWorkerAvailable) { 'conditional' } else { 'unsupported' } + } + $capabilities['scripted_multi_turn_same_session'] = if ($Inputs.Run.Interaction -eq $null) { + 'conditional' + } elseif ($AuthKind -ne 'subscription_file' -or -not $NativeWorkerAvailable) { + 'unsupported' + } else { + 'supported' + } + return $capabilities +} + +function Get-CodexPreflight { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + $commandInfo = Resolve-ExternalCommand -Name 'codex' + $platform = Get-PlatformName + $sandboxName = switch ($platform) { + 'linux' { 'bwrap' } + 'macos' { 'sandbox-exec' } + default { $null } + } + $sandboxInfo = if ([string]::IsNullOrWhiteSpace([string]$sandboxName)) { $null } else { Resolve-SandboxCommand -Name $sandboxName } + $versionObservation = $null + $nativeWorkerObservation = $null + + if ($profile.Runner -ne 'codex') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than codex.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is codex.')) + } + if ([string]::IsNullOrWhiteSpace($profile.Model)) { + $reasons.Add('Codex requires a model in execution-profile.json.') + } else { + $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) + } + if ($profile.ConfigurationProfile -ne 'isolated-default') { + $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by codex.") + } + if ($profile.ToolProfile -ne 'default') { + $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by codex.") + } + + if ($null -eq $commandInfo) { + $reasons.Add('The Codex CLI executable is not available on PATH.') + } else { + $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) + try { + $versionObservation = Get-ExternalCommandVersion -CommandInfo $commandInfo -WorkingDirectory $run.WorkingDirectoryPath -Environment (New-RunnerEnvironment -Run $run) -TimeoutSeconds 30 + if (-not $versionObservation.Available) { + $reasons.Add('The Codex CLI did not expose an exact observable version through --version.') + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status unavailable -Detail 'codex --version did not return a usable version string.')) + } else { + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status passed -Detail ([string]$versionObservation.Version))) + } + + $globalHelp = Get-CodexHelpResult -CommandInfo $commandInfo -Inputs $Inputs -Arguments @('--help') + $help = Get-CodexHelpResult -CommandInfo $commandInfo -Inputs $Inputs + if ($globalHelp.TimedOut -or $globalHelp.ExitCode -ne 0) { + $reasons.Add("Codex --help failed with exit status $($globalHelp.ExitCode).") + } + if ($help.TimedOut -or $help.ExitCode -ne 0) { + $reasons.Add("Codex --ask-for-approval never exec --help failed with exit status $($help.ExitCode).") + } else { + $helpText = [string]::Join("`n", @($globalHelp.Stdout, $globalHelp.Stderr, $help.Stdout, $help.Stderr)) + foreach ($flag in @('--ask-for-approval', '--ephemeral', '--ignore-user-config', '--ignore-rules', '--json', '--output-last-message', '--sandbox', '--cd', '--model', '--config')) { + if ($helpText -notmatch [regex]::Escape($flag)) { + $reasons.Add("The installed Codex CLI does not advertise required flag '$flag'.") + } + } + $visiblePlatform = if ($platform -eq 'linux' -and $null -ne $sandboxInfo) { 'linux' } else { $platform } + $constructed = New-CodexCliArguments -Inputs $Inputs -LastResponsePath (Join-Path $run.RunRoot 'evidence/codex-final.txt') -VisiblePlatform $visiblePlatform + if (@($constructed) -contains '--approve-for-me') { + $reasons.Add('The constructed Codex invocation must not combine --approve-for-me with explicit --sandbox selection.') + } + $sandboxIndex = [Array]::IndexOf([string[]]$constructed, '--sandbox') + $approvalIndex = [Array]::IndexOf([string[]]$constructed, '--ask-for-approval') + $execIndex = [Array]::IndexOf([string[]]$constructed, 'exec') + if ($approvalIndex -lt 0 -or $execIndex -lt 0 -or $approvalIndex -gt $execIndex -or $sandboxIndex -lt 0) { + $reasons.Add('The constructed Codex invocation must set --ask-for-approval never before exec and retain --sandbox workspace-write.') + } + if ($reasons.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Codex accepts the constructed noninteractive invocation: --ask-for-approval never, exec, --sandbox workspace-write, ephemeral JSON output, and isolated configuration controls.')) + } + } + $nativeWorkerObservation = Get-CodexNativeWorkerProbe -CommandInfo $commandInfo -Inputs $Inputs + if ($nativeWorkerObservation.Available) { + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status passed -Detail ($nativeWorkerObservation.Detail + ' This proves API readiness only; the actual child remains conditional until terminal evidence.'))) + $warnings.Add('Codex app-server native-worker controls remain conditional until terminal evidence proves the actual child model, cwd, HOME/config, fresh identity, prompt, exclusions, and terminal capture.') + } else { + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status unavailable -Detail $nativeWorkerObservation.Detail)) + $reasons.Add($nativeWorkerObservation.Detail) + } + } catch { + $reasons.Add("Could not inspect Codex CLI capabilities: $($_.Exception.Message)") + } + } + + $auth = Get-CodexAuthSource + if ($auth.Kind -eq 'missing') { + $reasons.Add('Neither a narrow Codex provider API-key environment variable nor subscription auth.json is available.') + } elseif ($auth.Kind -eq 'subscription_file') { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail 'Codex app-server uses a fresh temporary auth-only CODEX_HOME containing only a copied auth.json; the source home and all ambient Codex configuration remain outside the worker.')) + } else { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the narrow $($auth.Name) environment variable; the child shell policy is set to inherit=none.")) + } + + if ($null -ne $run.Interaction) { + if ($auth.Kind -ne 'subscription_file') { + $checks.Add((New-PreflightCheck -Name 'scripted_multi_turn_same_session' -Status failed -Detail 'The Codex API-key compatibility transport is one-shot and cannot continue the same app-server thread.')) + $reasons.Add('scripted_multi_turn_same_session is incompatible for the Codex API-key compatibility transport; same-session scripted turns require the native app-server thread/start + repeated turn/start surface.') + } elseif ($null -eq $nativeWorkerObservation -or -not $nativeWorkerObservation.Available) { + $checks.Add((New-PreflightCheck -Name 'scripted_multi_turn_same_session' -Status failed -Detail 'The installed Codex app-server schema did not prove thread/start plus repeatable turn/start on one thread.')) + $reasons.Add('scripted_multi_turn_same_session is incompatible: model-free Codex app-server schema probing did not prove same-thread continuation before execution.') + } else { + $checks.Add((New-PreflightCheck -Name 'scripted_multi_turn_same_session' -Status passed -Detail 'Codex app-server reuses the fresh thread/start identity for every deterministic user turn and dispatches the next turn/start only after the prior turn reaches terminal state.')) + } + } + + if ($auth.Kind -eq 'subscription_file') { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail 'The subscription app-server transport uses a temporary auth-only home but is not wrapped by the external run-only sandbox. Codex workspace-write remains enabled for the turn.')) + $warnings.Add('Subscription execution uses pragmatic isolation. The adapter does not claim that an external filesystem sandbox protects the app-server transport.') + } elseif ($null -eq $sandboxName) { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) + $warnings.Add("Platform '$platform' has no external hard filesystem confinement in this adapter; execution will report pragmatic isolation.") + } elseif ($null -eq $sandboxInfo) { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail "External '$sandboxName' is unavailable; pragmatic isolation remains available.")) + $warnings.Add("External '$sandboxName' was unavailable; execution will report pragmatic isolation.") + } else { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $sandboxName confines Codex to the staged run and required system runtime paths; Codex sandbox=workspace-write remains enabled inside it.")) + } + + $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The selected transport starts an ephemeral thread and never supplies a resume, continue, or existing session identifier.')) + if ($auth.Kind -eq 'subscription_file') { + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The app-server parent receives a filtered environment plus a temporary auth-only CODEX_HOME. Child shell inheritance is disabled with shell_environment_policy.inherit=none, and the runner validates instructionSources against the staged arm root.')) + $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "thread/start and turn/start set cwd to $($run.WorkingDirectoryPath); HOME and USERPROFILE remain staged under $($run.HomeDirectoryPath).")) + $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only auth.json is copied into a temporary auth-only CODEX_HOME and it is removed in finally; config.toml, skills, agents, sessions, memories, plugins, MCP configuration, and AGENTS.md are not copied. This does not claim hard filesystem confinement where none is available.')) + } else { + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The compatibility transport uses an isolated CODEX_HOME plus --ignore-user-config and --ignore-rules; unrelated inherited environment variables are removed.')) + $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "--cd $($run.WorkingDirectoryPath); CODEX_HOME under $($run.HomeDirectoryPath)")) + $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only the selected provider API-key variable is passed to Codex; auth files are not copied into the worker HOME.')) + } + + $hardConfinement = $auth.Kind -eq 'environment' -and $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $capabilities = Get-CodexCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement -NativeWorkerAvailable ($null -ne $nativeWorkerObservation -and $nativeWorkerObservation.Available) -AuthKind $auth.Kind + $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } + $descriptorCopy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } + $descriptorCopy.harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = $harnessVersion } + $mechanisms = [System.Collections.Generic.List[string]]::new() + if ($auth.Kind -eq 'subscription_file') { + foreach ($mechanism in @('native app-server initialize + thread/start + turn/start', 'temporary auth-only subscription CODEX_HOME', 'ephemeral thread', 'thread/read after turn completion', 'instructionSources validation', 'model/rerouted fail-closed', 'approvalPolicy=never', 'sandboxPolicy=workspaceWrite', 'shell_environment_policy.inherit=none', 'filtered parent process environment', 'prompt in turn/start input')) { $mechanisms.Add($mechanism) } + if ($null -ne $run.Interaction) { $mechanisms.Add('same-thread repeated turn/start for scripted interaction') } else { $mechanisms.Add('no session continuation') } + } else { + foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral compatibility transport', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + } + if ($hardConfinement) { $mechanisms.Add("external $sandboxName filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + $document = New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) + if ($null -ne $nativeWorkerObservation -and $nativeWorkerObservation.Available) { + $document.protocol_observations = [ordered]@{ + schema_directory = [string]$nativeWorkerObservation.SchemaDirectory + schema_source = [string]$nativeWorkerObservation.SchemaSource + schema_source_kind = [string]$nativeWorkerObservation.SchemaSourceKind + sandbox_modes = @($nativeWorkerObservation.SandboxModes) + allow_provider_model_fallback = [bool]$nativeWorkerObservation.SupportsProviderModelFallback + thread_read_schema_available = [bool]$nativeWorkerObservation.ThreadReadSchemaAvailable + } + } + return $document +} + +function New-CodexEnvironment { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$Auth + ) + + $codexHome = Join-Path $Inputs.Run.HomeDirectoryPath '.codex' + New-Item -ItemType Directory -Path $codexHome -Force | Out-Null + $environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables @(Get-ProviderAuthenticationVariables -Provider 'openai') -Additional @{ CODEX_HOME = $codexHome } + if ($Auth.Kind -eq 'environment') { return $environment } + return $environment +} + +function Get-LinuxCodexSandboxArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + $args = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--die-with-parent', '--new-session', '--unshare-pid')) { $args.Add($argument) } + foreach ($path in @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/lib64', '/libexec', '/etc', '/opt')) { + if (Test-Path -LiteralPath $path) { + $args.Add('--ro-bind'); $args.Add($path); $args.Add($path) + } + } + $args.Add('--proc'); $args.Add('/proc') + $args.Add('--dev'); $args.Add('/dev') + $args.Add('--tmpfs'); $args.Add('/tmp') + $args.Add('--bind'); $args.Add($Inputs.Run.RunRoot); $args.Add('/run') + $commandSource = [string]$CommandInfo.Source + $commandDirectory = Split-Path -Parent $commandSource + if (-not ($commandSource.StartsWith('/usr/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/bin/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/opt/', [System.StringComparison]::Ordinal))) { + if (Test-Path -LiteralPath $commandDirectory -PathType Container) { + $args.Add('--ro-bind'); $args.Add($commandDirectory); $args.Add($commandDirectory) + } + } + $args.Add('--chdir'); $args.Add('/run/repo') + $insideEnvironment = [ordered]@{ + HOME = '/run/home' + USERPROFILE = '/run/home' + XDG_CONFIG_HOME = '/run/home/.config' + XDG_DATA_HOME = '/run/home/.local/share' + XDG_CACHE_HOME = '/run/home/.cache' + TEMP = '/run/home/tmp' + TMP = '/run/home/tmp' + CODEX_HOME = '/run/home/.codex' + PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + CI = '1' + NO_COLOR = '1' + } + foreach ($authName in @(Get-ProviderAuthenticationVariables -Provider 'openai')) { + if ($Environment.Contains($authName) -and -not [string]::IsNullOrWhiteSpace([string]$Environment[$authName])) { + $insideEnvironment[$authName] = [string]$Environment[$authName] + } + } + foreach ($key in @($insideEnvironment.Keys)) { + $args.Add('--setenv'); $args.Add($key); $args.Add([string]$insideEnvironment[$key]) + } + $args.Add('--') + $args.Add($CommandInfo.FileName) + foreach ($prefix in @($CommandInfo.Prefix)) { $args.Add($prefix) } + return @($args) +} + +function New-CodexMacosSandboxProfile { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo + ) + + $profilePath = Join-Path $Inputs.Run.HomeDirectoryPath 'codex-sandbox.sb' + $runRoot = $Inputs.Run.RunRoot.Replace('\', '/') + $commandDirectory = (Split-Path -Parent ([string]$CommandInfo.Source)).Replace('\', '/') + $readRoots = @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/libexec', '/System', '/Library', '/opt', '/private/var/db', $commandDirectory) + $lines = [System.Collections.Generic.List[string]]::new() + $lines.Add('(version 1)') + $lines.Add('(deny default)') + $lines.Add('(allow process*)') + $lines.Add('(allow network*)') + foreach ($root in $readRoots | Sort-Object -Unique) { + if (-not [string]::IsNullOrWhiteSpace($root) -and (Test-Path -LiteralPath $root -PathType Container)) { + $escapedRoot = $root.Replace('\', '/').Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRoot)) + } + } + $escapedRunRoot = $runRoot.Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add(('(allow file-write* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add('(allow file-read* (subpath "/dev"))') + $lines.Add('(allow file-write* (subpath "/dev/null"))') + [System.IO.File]::WriteAllText($profilePath, ([string]::Join("`n", $lines) + "`n"), [System.Text.UTF8Encoding]::new($false)) + return $profilePath +} + +function Write-CodexCapture { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text + ) + + $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $parent = Split-Path -Parent $path + New-Item -ItemType Directory -Path $parent -Force | Out-Null + [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) + return New-ArtifactReference -Run $RunData.Run -Path $RelativePath -Scope run -MediaType (Get-MediaType -Path $RelativePath) +} + +function Invoke-CodexProjectedTransport { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$Auth, + [Parameter(Mandatory = $true)][object]$Platform, + [object]$SandboxInfo = $null, + [Parameter(Mandatory = $true)][bool]$HardFilesystem, + [Parameter(Mandatory = $true)][ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform, + [Parameter(Mandatory = $true)][string]$LastResponseRelativePath, + [bool]$SupportsProviderModelFallback = $false + ) + + $projection = New-CodexExecutionProjection -Inputs $Inputs + $executionInputs = [pscustomobject]@{ Run = $projection.Run; Profile = $Inputs.Profile } + $process = $null + $projectedFinalResponse = $null + try { + $physicalLastResponsePath = Join-Path $projection.Root ($LastResponseRelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $physicalLastResponsePath) -Force | Out-Null + $environment = if ($Auth.Kind -eq 'environment') { New-CodexEnvironment -Inputs $executionInputs -Auth $Auth } else { $null } + $arguments = New-CodexCliArguments -Inputs $executionInputs -LastResponsePath $physicalLastResponsePath -VisiblePlatform $VisiblePlatform + if ($Auth.Kind -eq 'subscription_file') { + $process = Invoke-CodexAppServer -CommandInfo $CommandInfo -Inputs $executionInputs -Auth $Auth -SupportsProviderModelFallback $SupportsProviderModelFallback -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } elseif ($Platform -eq 'linux' -and $HardFilesystem) { + $sandboxArguments = Get-LinuxCodexSandboxArguments -Inputs $executionInputs -CommandInfo $CommandInfo -Environment $environment + $process = Invoke-RunnerProcess -FileName $SandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $executionInputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } elseif ($Platform -eq 'macos' -and $HardFilesystem) { + $sandboxProfile = New-CodexMacosSandboxProfile -Inputs $executionInputs -CommandInfo $CommandInfo + $sandboxArguments = @('-f', $sandboxProfile, '--', $CommandInfo.FileName) + @($CommandInfo.Prefix) + @($arguments) + $process = Invoke-RunnerProcess -FileName $SandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $executionInputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $process = Invoke-CodexCli -CommandInfo $CommandInfo -Arguments $arguments -Inputs $executionInputs -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } + if (Test-Path -LiteralPath $physicalLastResponsePath -PathType Leaf) { + $projectedFinalResponse = [System.IO.File]::ReadAllText($physicalLastResponsePath, [System.Text.UTF8Encoding]::new($false)) + $logicalLastResponsePath = Join-Path $Inputs.Run.RunRoot ($LastResponseRelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $logicalLastResponsePath) -Force | Out-Null + [System.IO.File]::WriteAllText($logicalLastResponsePath, $projectedFinalResponse, [System.Text.UTF8Encoding]::new($false)) + } + return [pscustomobject]@{ + Process = $process + ProjectedFinalResponse = $projectedFinalResponse + ExecutionPaths = [ordered]@{ + projection = 'physical_temp_outside_logical_package' + logical_run_root = [string]$Inputs.Run.RunRoot + logical_working_directory = [string]$Inputs.Run.WorkingDirectoryPath + logical_home_directory = [string]$Inputs.Run.HomeDirectoryPath + physical_run_root = [string]$projection.Root + physical_working_directory = [string]$projection.PhysicalWorkingDirectory + physical_home_directory = [string]$projection.PhysicalHomeDirectory + } + PhysicalProjectionProven = [bool]$projection.Proven + } + } finally { + try { Sync-CodexProjectedRepository -Projection $projection } finally { Remove-CodexExecutionProjection -Projection $projection } + } +} + +function Invoke-CodexExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $preflight = Get-CodexPreflight -Inputs $Inputs + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + $executionDescriptor = [ordered]@{} + foreach ($key in $descriptor.Keys) { $executionDescriptor[$key] = $descriptor[$key] } + $executionDescriptor.harness = $preflight.harness + if ($preflight.status -ne 'compatible') { + $finished = [DateTime]::UtcNow + $failureText = [string]::Join('; ', @($preflight.reasons)) + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message $failureText) -SessionId $sessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight; resume = $false }) -AttemptCount 1 + } + + $commandInfo = Resolve-ExternalCommand -Name 'codex' + $auth = Get-CodexAuthSource + $lastResponsePath = 'evidence/codex-final.txt' + New-Item -ItemType Directory -Path (Join-Path $Inputs.Run.RunRoot 'evidence') -Force | Out-Null + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $auth.Kind -eq 'environment' -and $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } + $protocolObservations = Get-JsonProperty -Object $preflight -Name 'protocol_observations' -Default $null + $supportsProviderModelFallback = [bool](Get-JsonProperty -Object $protocolObservations -Name 'allow_provider_model_fallback' -Default $false) + $transport = Invoke-CodexProjectedTransport -CommandInfo $commandInfo -Inputs $Inputs -Auth $auth -Platform $platform -SandboxInfo $sandboxInfo -HardFilesystem $hardFilesystem -VisiblePlatform $visiblePlatform -LastResponseRelativePath $lastResponsePath -SupportsProviderModelFallback $supportsProviderModelFallback + $process = $transport.Process + $stdoutArtifact = Write-CodexCapture -RunData $Inputs -RelativePath 'evidence/codex-events.jsonl' -Text $process.Stdout + $stderrArtifact = Write-CodexCapture -RunData $Inputs -RelativePath 'evidence/codex-stderr.txt' -Text $process.Stderr + $artifacts = [System.Collections.Generic.List[object]]::new() + $artifacts.Add($stdoutArtifact) + $artifacts.Add($stderrArtifact) + $transcriptArtifactPath = 'evidence/codex-events.jsonl' + if ($auth.Kind -eq 'subscription_file') { + $rawStdoutArtifact = Write-CodexCapture -RunData $Inputs -RelativePath 'evidence/codex-app-server-events.jsonl' -Text $process.RawStdout + $artifacts.Add($rawStdoutArtifact) + $transcriptArtifactPath = 'evidence/codex-app-server-events.jsonl' + } + + $parsed = ConvertFrom-JsonLines -Text $process.Stdout + $warnings = [System.Collections.Generic.List[string]]::new() + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("Codex event parse error: $parseError") } + $finalText = if ($auth.Kind -eq 'subscription_file') { $process.FinalText } else { $transport.ProjectedFinalResponse } + if ([string]::IsNullOrWhiteSpace($finalText) -and $null -ne $transport.ProjectedFinalResponse) { $finalText = $transport.ProjectedFinalResponse } + $threadId = if ($auth.Kind -eq 'subscription_file') { $process.ThreadId } else { $null } + $turnId = if ($auth.Kind -eq 'subscription_file') { $process.TurnId } else { $null } + $turnFailure = $null + $usage = $null + $toolCalls = 0 + $commands = [System.Collections.Generic.List[object]]::new() + $files = [System.Collections.Generic.List[object]]::new() + $eventCounts = @{} + foreach ($event in @($parsed.Events)) { + $eventType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ([string]::IsNullOrWhiteSpace($eventType)) { + $warnings.Add('Codex emitted an event without a type; it was ignored.') + continue + } + if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } + switch ($eventType) { + 'thread.started' { $threadId = [string](Get-JsonProperty -Object $event -Name 'thread_id' -Default '') } + 'item.completed' { + $item = Get-JsonProperty -Object $event -Name 'item' -Default $null + $itemType = [string](Get-JsonProperty -Object $item -Name 'type' -Default '') + if ($itemType -eq 'agent_message') { + $candidate = [string](Get-JsonProperty -Object $item -Name 'text' -Default '') + if (-not [string]::IsNullOrWhiteSpace($candidate)) { $finalText = $candidate } + } elseif ($itemType -in @('command_execution', 'mcp_tool_call', 'file_change')) { + $toolCalls++ + if ($itemType -eq 'command_execution') { + $commands.Add([ordered]@{ type = $itemType; command = Get-JsonProperty -Object $item -Name 'command'; exit_code = Get-JsonProperty -Object $item -Name 'exit_code' }) + } else { + $files.Add([ordered]@{ type = $itemType; item = $item }) + } + } + } + 'turn.completed' { + $usage = Get-JsonProperty -Object $event -Name 'usage' -Default $null + } + 'turn.failed' { $turnFailure = Get-JsonProperty -Object $event -Name 'error' -Default 'Codex turn failed.' } + 'error' { $turnFailure = Get-JsonProperty -Object $event -Name 'message' -Default 'Codex emitted an error.' } + { $_ -in @('turn.started', 'item.started', 'item.updated') } { } + default { $warnings.Add("Unknown Codex event '$eventType' was preserved as a warning.") } + } + } + if (Test-Path -LiteralPath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -PathType Leaf) { + $lastArtifact = New-ArtifactReference -Run $Inputs.Run -Path $lastResponsePath -Scope run -MediaType 'text/plain; charset=utf-8' + $artifacts.Add($lastArtifact) + if ([string]::IsNullOrWhiteSpace($finalText)) { + $finalText = [System.IO.File]::ReadAllText((Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)), [System.Text.UTF8Encoding]::new($false)) + } + } + $terminalCaptureComplete = if ($auth.Kind -eq 'subscription_file') { + [bool]$process.TurnCompleted + } else { + @($parsed.Events | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'type' -Default '') -eq 'turn.completed' }).Count -gt 0 + } + + $nativeEvidenceFailures = [System.Collections.Generic.List[string]]::new() + $observedModel = if ($auth.Kind -eq 'subscription_file') { [string]$process.ObservedModel } else { '' } + $observedWorkingDirectory = if ($auth.Kind -eq 'subscription_file') { [string]$process.ObservedWorkingDirectory } else { '' } + $promptFidelity = $auth.Kind -eq 'subscription_file' -and [string]$process.PromptInputSha256 -eq [string]$Inputs.Run.PromptHash + $unexpectedInstructionSources = [System.Collections.Generic.List[string]]::new() + $invalidInstructionSources = [System.Collections.Generic.List[string]]::new() + $threadReadObservation = 'not_applicable' + $instructionSourceProof = if ($auth.Kind -eq 'subscription_file') { 'physical_projection_boundary' } else { 'compatibility_transport' } + $instructionSourcesUnobserved = $false + $authHomeProven = $true + $terminalTurnProven = $true + $modelRerouteObserved = $false + $threadReadMetadataFailure = $false + if ($auth.Kind -eq 'subscription_file') { + # instructionSources is an optional response observation. A missing + # array is acceptable here because the physical projection is outside + # the source-repository ancestor chain and is the independent proof of + # the ambient instruction boundary. Without that proof, absence stays + # fail-closed. + $instructionSourcesUnobserved = -not [bool]$process.InstructionSourcesObserved -and -not [bool]$transport.PhysicalProjectionProven + foreach ($source in @($process.InstructionSources)) { + $sourcePath = [string]$source + if ([string]::IsNullOrWhiteSpace($sourcePath)) { + $invalidInstructionSources.Add($sourcePath) + } elseif (-not (Test-PathInside -BasePath ([string]$transport.ExecutionPaths.physical_run_root) -CandidatePath $sourcePath)) { + $unexpectedInstructionSources.Add($sourcePath) + } + } + $authHomeProven = [bool]$process.AuthOnlyHome -and [bool]$process.AuthOnlyHomeRemoved + $terminalTurnProven = [string](Get-JsonProperty -Object $process.TerminalTurn -Name 'status' -Default '') -eq 'completed' + $modelRerouteObserved = @($process.ModelReroutes).Count -gt 0 + $threadReadThread = if ($null -ne $process.ThreadReadResponse) { Get-JsonProperty -Object (Get-JsonProperty -Object $process.ThreadReadResponse -Name 'result' -Default $null) -Name 'thread' -Default $null } else { $null } + if ($null -ne $threadReadThread) { + $threadReadObservation = 'observed' + if ([string](Get-JsonProperty -Object $threadReadThread -Name 'id' -Default '') -ne [string]$process.ThreadId -or + -not [bool](Get-JsonProperty -Object $threadReadThread -Name 'ephemeral' -Default $false) -or + -not (Test-ExactObservedPath -Expected ([string]$transport.ExecutionPaths.physical_working_directory) -Observed ([string](Get-JsonProperty -Object $threadReadThread -Name 'cwd' -Default '')))) { + $threadReadMetadataFailure = $true + } + } elseif ($null -ne $process.ThreadReadFailure -or $null -eq $process.ThreadReadResponse) { + $threadReadObservation = 'unavailable_optional' + $warnings.Add('Codex thread/read is supplemental and was unavailable; thread/start and turn/completed remain the mandatory terminal proof.') + } else { + # A response was returned but did not satisfy the installed + # ThreadReadResponse shape. The thread/read observation is + # supplemental, but a present response with a missing required + # thread object is a protocol violation rather than an optional + # absence. + $threadReadObservation = 'malformed' + $threadReadMetadataFailure = $true + } + } + + $status = 'completed' + $reason = $null + $failure = $null + $exitStatus = if ($process.TimedOut) { $null } else { [Nullable[int]]$process.ExitCode } + if ($process.TimedOut) { + $status = 'timed_out' + $reason = 'codex_timeout' + $failure = New-ExecutionFailure -Code 'timed_out' -Message 'Codex did not finish before timeout_seconds.' + } elseif ($process.ExitCode -ne 0 -or $null -ne $turnFailure) { + $status = 'failed' + $reason = 'codex_failure' + $failureMessage = if ($null -ne $turnFailure) { [string]$turnFailure } elseif (-not [string]::IsNullOrWhiteSpace($process.Stderr)) { $process.Stderr.Trim() } else { "Codex exited with status $($process.ExitCode)." } + $failure = New-ExecutionFailure -Code 'codex_failure' -Message $failureMessage + } elseif ([string]::IsNullOrWhiteSpace($finalText)) { + $warnings.Add('Codex exited successfully without a final agent message.') + $reason = 'codex_did_not_return_final_response' + } + $tokenMetric = if ($null -eq $usage) { + New-UnavailableMetric -Reason 'codex_did_not_expose_turn_usage' + } else { + $usageValue = [ordered]@{} + foreach ($name in @('input_tokens', 'cached_input_tokens', 'output_tokens', 'reasoning_output_tokens')) { + $value = Get-JsonProperty -Object $usage -Name $name -Default $null + if ($null -ne $value) { $usageValue[$name] = $value } + } + if ($usageValue.Count -eq 0) { New-UnavailableMetric -Reason 'codex_usage_event_had_no_supported_buckets' } else { New-AvailableMetric -Value $usageValue } + } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = $transcriptArtifactPath; complete = $terminalCaptureComplete }) + tokens = $tokenMetric + tool_calls = New-AvailableMetric -Value $toolCalls + cost = New-UnavailableMetric -Reason 'codex_runner_does_not_estimate_cost' + } + $finished = [DateTime]::UtcNow + $sessionResultId = if ([string]::IsNullOrWhiteSpace($threadId)) { $sessionId } else { $threadId } + $capabilities = Get-CodexCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem -NativeWorkerAvailable ($auth.Kind -eq 'subscription_file') -AuthKind $auth.Kind + $mechanisms = [System.Collections.Generic.List[string]]::new() + if ($auth.Kind -eq 'subscription_file') { + foreach ($mechanism in @('native app-server initialize + thread/start + turn/start', 'temporary auth-only subscription CODEX_HOME', 'ephemeral thread', 'thread/read after turn completion', 'instructionSources validation', 'model/rerouted fail-closed', 'approvalPolicy=never', 'sandboxPolicy=workspaceWrite', 'shell_environment_policy.inherit=none', 'filtered parent process environment', 'prompt in turn/start input')) { $mechanisms.Add($mechanism) } + $continuationMechanism = if ($null -ne $Inputs.Run.Interaction) { 'same-thread repeated turn/start for scripted interaction' } else { 'no session continuation' } + $mechanisms.Add($continuationMechanism) + } else { + foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + } + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $credentialEvidence = [ordered]@{ + source = $auth.Kind + provider_environment_variable = $auth.Name + unrelated_environment_excluded = $true + child_tool_visibility = 'codex_shell_environment_policy_inherit_none' + value_observed = $false + auth_only_home = if ($auth.Kind -eq 'subscription_file') { [bool]$process.AuthOnlyHome } else { $false } + auth_only_home_removed = if ($auth.Kind -eq 'subscription_file') { [bool]$process.AuthOnlyHomeRemoved } else { $true } + ambient_codex_configuration_copied = $false + } + $outputLastMessageArgument = if ($auth.Kind -eq 'subscription_file') { $null } else { Get-SandboxVisiblePath -HostPath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -RunRoot $Inputs.Run.RunRoot -Platform $visiblePlatform } + $evidence = [ordered]@{ + thread_id = $threadId + thread_session_id = if ($auth.Kind -eq 'subscription_file') { $process.ThreadSessionId } else { $null } + turn_id = $turnId + execution_paths = $transport.ExecutionPaths + event_counts = $eventCounts + commands = @($commands) + files = @($files) + prompt_first_input = if ($auth.Kind -eq 'subscription_file') { $promptFidelity } else { $true } + resume = $false + stdout_exit_code = $process.ExitCode + sandbox = $sandboxEvidence + output_last_message_argument = $outputLastMessageArgument + credential = $credentialEvidence + } + if ($null -ne $Inputs.Run.Interaction) { $evidence.turns = @($process.TurnRecords) } + if ($auth.Kind -eq 'subscription_file') { + $rawArtifact = @($artifacts | Where-Object { [string]$_.path -eq $transcriptArtifactPath } | Select-Object -First 1) + $evidence.capture = [ordered]@{ + source = 'harness_native_transport' + terminal = [bool]$process.TurnCompleted + worker_authored = $false + artifact = $transcriptArtifactPath + sha256 = if ($rawArtifact.Count -eq 1) { [string]$rawArtifact[0].sha256 } else { $null } + } + $evidence.delegation = [ordered]@{ + dispatch_owner = 'runner' + mechanism = [string]$descriptor.delegation.mechanism + worker_session_id = $sessionResultId + observed_model = $observedModel + observed_working_directory = $observedWorkingDirectory + observed_home = [string]$process.WorkerHome + fresh_worker = [bool]$process.ObservedEphemeral -and -not [string]::IsNullOrWhiteSpace([string]$process.ThreadId) + home_config_isolated = [bool]$process.AuthOnlyHome -and [bool]$process.AuthOnlyHomeRemoved + prompt_fidelity = $promptFidelity + prompt_sha256 = $Inputs.Run.PromptHash + terminal_result_capture = [bool]$process.TurnCompleted -and -not [string]::IsNullOrWhiteSpace([string]$process.RawStdout) + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + thread_id = $threadId + thread_session_id = $process.ThreadSessionId + turn_id = $turnId + instruction_sources_observed = [bool]$process.InstructionSourcesObserved + instruction_source_proof = $instructionSourceProof + instruction_sources = @($process.InstructionSources) + invalid_instruction_sources = @($invalidInstructionSources.ToArray()) + unexpected_instruction_sources = @($unexpectedInstructionSources.ToArray()) + requested_runtime_workspace_roots = @($transport.ExecutionPaths.physical_working_directory) + logical_runtime_workspace_roots = @($Inputs.Run.WorkingDirectoryPath) + thread_read_observed = $null -ne $process.ThreadReadResponse + thread_read_observation = $threadReadObservation + model_reroutes = @($process.ModelReroutes) + same_session_continuation = if ($null -ne $Inputs.Run.Interaction) { [bool]$process.AllTurnsCompleted } else { $null } + } + if ($null -ne $Inputs.Run.Interaction) { + $evidence.interaction = [ordered]@{ + schema = (Get-RunnerSchemaNames).Interaction + mode = 'scripted' + same_session = [bool]$process.AllTurnsCompleted + session_id = $sessionResultId + turns = @($process.TurnRecords) + final_response_sequence = @($process.TurnRecords).Count + turn_start_requests = @($process.TurnStartRequests) + turn_start_responses = @($process.TurnStartResponses) + } + } + $threadStartResultEvidence = Get-JsonProperty -Object $process.ThreadStartResponse -Name 'result' -Default ([ordered]@{}) + $threadReadThreadEvidence = if ($null -ne $process.ThreadReadResponse) { Get-JsonProperty -Object (Get-JsonProperty -Object $process.ThreadReadResponse -Name 'result' -Default $null) -Name 'thread' -Default $null } else { $null } + $turnCompletionEvidence = if ($null -ne $process.TerminalTurn) { [ordered]@{ thread_id = $process.ThreadId; turn_id = $process.TurnId; status = Get-JsonProperty -Object $process.TerminalTurn -Name 'status' -Default $null } } else { $null } + $evidence.app_server = [ordered]@{ + thread_start_request = $process.ThreadStartRequest + thread_start_response = $process.ThreadStartResponse + turn_start_request = $process.TurnStartRequest + turn_start_response = $process.TurnStartResponse + thread_start = [ordered]@{ + requested_model = $Inputs.Profile.Model + requested_cwd = $transport.ExecutionPaths.physical_working_directory + requested_ephemeral = $true + requested_sandbox = 'read-only' + requested_allow_provider_model_fallback = if ($supportsProviderModelFallback) { $false } else { $null } + observed_model = Get-JsonProperty -Object $threadStartResultEvidence -Name 'model' -Default $null + observed_cwd = Get-JsonProperty -Object $threadStartResultEvidence -Name 'cwd' -Default $null + observed_ephemeral = Get-JsonProperty -Object (Get-JsonProperty -Object $threadStartResultEvidence -Name 'thread' -Default $null) -Name 'ephemeral' -Default $null + observed_sandbox = Get-JsonProperty -Object $threadStartResultEvidence -Name 'sandbox' -Default $null + instruction_sources = @($process.InstructionSources) + } + turn_start = [ordered]@{ + thread_id = $process.ThreadId + requested_model = $Inputs.Profile.Model + requested_cwd = $transport.ExecutionPaths.physical_working_directory + requested_effort = $Inputs.Profile.ReasoningEffort + requested_sandbox_policy = Get-JsonProperty -Object (Get-JsonProperty -Object $process.TurnStartRequest -Name 'params' -Default $null) -Name 'sandboxPolicy' -Default $null + prompt_sha256 = $process.PromptInputSha256 + } + turn_starts = @($process.TurnStartRequests | ForEach-Object { + [ordered]@{ + thread_id = Get-JsonProperty -Object $_.params -Name 'threadId' -Default $null + requested_model = Get-JsonProperty -Object $_.params -Name 'model' -Default $null + requested_cwd = Get-JsonProperty -Object $_.params -Name 'cwd' -Default $null + prompt_sha256 = Get-Sha256HexFromBytes -Bytes ([System.Text.UTF8Encoding]::new($false).GetBytes([string]$_.params.input[0].text)) + } + }) + terminal_turn = $turnCompletionEvidence + thread_read = [ordered]@{ + request = [ordered]@{ threadId = $process.ThreadId; includeTurns = $true } + response = if ($null -eq $threadReadThreadEvidence) { $null } else { [ordered]@{ id = Get-JsonProperty -Object $threadReadThreadEvidence -Name 'id' -Default $null; session_id = Get-JsonProperty -Object $threadReadThreadEvidence -Name 'sessionId' -Default $null; cwd = Get-JsonProperty -Object $threadReadThreadEvidence -Name 'cwd' -Default $null; ephemeral = Get-JsonProperty -Object $threadReadThreadEvidence -Name 'ephemeral' -Default $null } } + failure = $process.ThreadReadFailure + observation = $threadReadObservation + } + model_rerouted = @($process.ModelReroutes) + } + } + + if ($auth.Kind -eq 'subscription_file') { + # Run the common portable validator over the same evidence object that + # the Codex-specific checks use. This is the single terminal decision: + # additional Codex failures are merged with, never hidden from, the + # portable result and orchestration state. + $commonPreview = [ordered]@{ + status = $status + session = [ordered]@{ id = $sessionResultId; fresh = [bool]$process.ObservedEphemeral -and -not [string]::IsNullOrWhiteSpace([string]$process.ThreadId); resumed = $false } + run = [ordered]@{ eval_id = $Inputs.Run.EvalId; eval_name = $Inputs.Run.EvalName; configuration = $Inputs.Run.Mode } + requested = [ordered]@{ model = $Inputs.Profile.Model } + runner = [ordered]@{ name = 'codex' } + evidence = $evidence + } + $commonValidation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $commonPreview -Run $Inputs.Run -RequestedModel ([string]$Inputs.Profile.Model) -ExpectedWorkerSessionId $sessionResultId -ExpectedRunner 'codex' -ExpectedMechanism ([string]$descriptor.delegation.mechanism) + foreach ($failureName in @($commonValidation.Failures)) { + if ($nativeEvidenceFailures -notcontains [string]$failureName) { $nativeEvidenceFailures.Add([string]$failureName) } + } + # These checks are genuinely Codex-specific. The portable validator + # above owns model/cwd/home/freshness/prompt/terminal acceptance; this + # layer contributes only app-server protocol and transport invariants. + if ($instructionSourcesUnobserved) { $nativeEvidenceFailures.Add('instruction_sources_unobserved') } + if ($invalidInstructionSources.Count -gt 0) { $nativeEvidenceFailures.Add('invalid_instruction_sources') } + if ($unexpectedInstructionSources.Count -gt 0) { $nativeEvidenceFailures.Add('unexpected_instruction_sources') } + if (-not $authHomeProven) { $nativeEvidenceFailures.Add('isolated_auth_home') } + if (-not $terminalTurnProven) { $nativeEvidenceFailures.Add('terminal_turn_status') } + if ($modelRerouteObserved) { $nativeEvidenceFailures.Add('model_rerouted') } + if ($threadReadMetadataFailure) { $nativeEvidenceFailures.Add('thread_read_metadata') } + $uniqueNativeEvidenceFailures = @($nativeEvidenceFailures | Select-Object -Unique) + $nativeEvidenceFailures = [System.Collections.Generic.List[string]]::new() + foreach ($failureName in $uniqueNativeEvidenceFailures) { $nativeEvidenceFailures.Add([string]$failureName) } + if ($nativeEvidenceFailures.Count -gt 0) { + $status = 'incompatible' + $reason = 'codex_native_evidence_incompatible' + $baseFailureMessage = if ($null -ne $failure) { [string]$failure.message } elseif (-not [string]::IsNullOrWhiteSpace([string]$process.TransportFailure)) { [string]$process.TransportFailure } else { 'Codex app-server terminal evidence was not accepted.' } + $failure = New-ExecutionFailure -Code 'native_evidence_incompatible' -Message ("Codex app-server evidence failed closed: {0}. Transport detail: {1}" -f ([string]::Join(', ', @($nativeEvidenceFailures)), $baseFailureMessage)) + $exitStatus = $null + } + $evidence.native_worker_evidence_failures = @($nativeEvidenceFailures.ToArray()) + } + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Codex accepted the requested model and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence $evidence -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { Write-RunnerJson -Value (Get-CodexDescriptor) -AsOutput } + 'preflight' { + $inputs = Resolve-CodexInputs + Write-RunnerJson -Value (Get-CodexPreflight -Inputs $inputs) -AsOutput + } + 'execute' { + $inputs = Resolve-CodexInputs + [void](Assert-PhaseOneEvidenceWritable -Run $inputs.Run) + $result = Invoke-CodexExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/contracts/execution-freeze.schema.json b/scripts/eval-runners/contracts/execution-freeze.schema.json new file mode 100644 index 0000000..ab2e638 --- /dev/null +++ b/scripts/eval-runners/contracts/execution-freeze.schema.json @@ -0,0 +1,76 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-execution-freeze/1", + "title": "Codebelt Agentic Eval Execution Freeze", + "type": "object", + "additionalProperties": false, + "required": ["schema", "version", "generated_utc", "manifest_schema", "profile_sha256", "runner", "model", "orchestration_state_sha256", "executions"], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-execution-freeze/1" }, + "version": { "const": 1 }, + "generated_utc": { "type": "string", "minLength": 1 }, + "manifest_schema": { "type": "string", "minLength": 1 }, + "profile_sha256": { "type": "string", "pattern": "^[0-9a-fA-F]{64}$" }, + "runner": { "type": "string", "minLength": 1 }, + "model": { "type": "string", "minLength": 1 }, + "orchestration_state_sha256": { "type": ["string", "null"], "pattern": "^[0-9a-fA-F]{64}$" }, + "executions": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["worker_id", "eval_id", "eval_name", "configuration", "run_id", "run_manifest", "execution_result", "execution_result_sha256", "raw_artifacts", "runner", "harness", "requested_model", "observed_model", "resolved_model", "session_id", "thread_id", "terminal_status", "finished_utc"], + "properties": { + "worker_id": { "type": "string", "minLength": 1 }, + "eval_id": { "type": "integer", "minimum": 1 }, + "eval_name": { "type": "string", "minLength": 1 }, + "configuration": { "enum": ["with_skill", "without_skill"] }, + "run_id": { "type": "string", "minLength": 1 }, + "run_manifest": { "type": "string", "minLength": 1 }, + "execution_result": { "type": "string", "minLength": 1 }, + "execution_result_sha256": { "type": "string", "pattern": "^[0-9a-fA-F]{64}$" }, + "raw_artifacts": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["scope", "path", "sha256", "size", "media_type"], + "properties": { + "scope": { "enum": ["run", "package"] }, + "path": { "type": "string", "minLength": 1 }, + "sha256": { "type": "string", "pattern": "^[0-9a-fA-F]{64}$" }, + "size": { "type": "integer", "minimum": 0 }, + "media_type": { "type": "string" } + } + } + }, + "runner": { + "type": "object", + "additionalProperties": false, + "required": ["name", "version"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 } + } + }, + "harness": { + "type": "object", + "additionalProperties": false, + "required": ["name", "version"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 } + } + }, + "requested_model": { "type": "string" }, + "observed_model": { "type": ["string", "null"] }, + "resolved_model": { "type": ["string", "null"] }, + "session_id": { "type": "string", "minLength": 1 }, + "thread_id": { "type": ["string", "null"] }, + "terminal_status": { "enum": ["completed", "failed", "timed_out", "cancelled", "incompatible"] }, + "finished_utc": { "type": "string", "minLength": 1 } + } + } + } + } +} diff --git a/scripts/eval-runners/contracts/execution-profile.schema.json b/scripts/eval-runners/contracts/execution-profile.schema.json new file mode 100644 index 0000000..5fd745d --- /dev/null +++ b/scripts/eval-runners/contracts/execution-profile.schema.json @@ -0,0 +1,27 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-execution-profile/1", + "title": "Codebelt Agentic Eval Execution Profile", + "type": "object", + "additionalProperties": false, + "required": [ + "schema", + "runner", + "model", + "reasoning_effort", + "configuration_profile", + "tool_profile", + "timeout_seconds", + "concurrency" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-execution-profile/1" }, + "runner": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]*$" }, + "model": { "type": "string", "minLength": 1 }, + "reasoning_effort": { "type": ["string", "null"], "minLength": 1 }, + "configuration_profile": { "type": "string", "minLength": 1 }, + "tool_profile": { "type": "string", "minLength": 1 }, + "timeout_seconds": { "type": "integer", "minimum": 1, "maximum": 86400 }, + "concurrency": { "type": "integer", "minimum": 1 } + } +} diff --git a/scripts/eval-runners/contracts/execution-result.schema.json b/scripts/eval-runners/contracts/execution-result.schema.json new file mode 100644 index 0000000..f0bf233 --- /dev/null +++ b/scripts/eval-runners/contracts/execution-result.schema.json @@ -0,0 +1,201 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-execution-result/1", + "title": "Codebelt Agentic Eval Execution Result", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "protocol_version", + "run_id", + "session", + "status", + "run", + "final_response", + "runner", + "harness", + "requested", + "resolved", + "started_utc", + "finished_utc", + "duration_seconds", + "exit", + "input", + "isolation", + "telemetry", + "evidence", + "artifacts", + "warnings", + "compatibility_deviations", + "attempt_count" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-execution-result/1" }, + "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, + "run_id": { "type": "string", "minLength": 1 }, + "session": { + "type": "object", + "required": ["id", "fresh", "resumed"], + "properties": { + "id": { "type": "string", "minLength": 1 }, + "fresh": { "const": true }, + "resumed": { "const": false } + } + }, + "status": { "enum": ["completed", "failed", "timed_out", "cancelled", "incompatible"] }, + "final_response": { + "type": "object", + "required": ["status"], + "properties": { + "status": { "enum": ["available", "unavailable"] }, + "text": { "type": "string" }, + "reason": { "type": "string" } + } + }, + "runner": { + "type": "object", + "required": ["name", "version"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 } + } + }, + "harness": { + "type": "object", + "required": ["name", "version"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 } + } + }, + "requested": { + "type": "object", + "additionalProperties": false, + "required": [ + "model", + "reasoning_effort", + "configuration_profile", + "tool_profile", + "timeout_seconds" + ], + "properties": { + "model": { "type": ["string", "null"] }, + "reasoning_effort": { "type": ["string", "null"] }, + "configuration_profile": { "type": "string" }, + "tool_profile": { "type": "string" }, + "timeout_seconds": { "type": "integer" } + } + }, + "resolved": { + "type": "object", + "properties": { + "model": { "type": ["string", "null"] }, + "reasoning_effort": { "type": ["string", "null"] }, + "configuration_profile": { "type": ["string", "null"] }, + "tool_profile": { "type": ["string", "null"] }, + "status": { "enum": ["unavailable", "accepted_request", "resolved"] }, + "accepted": { + "type": "object", + "additionalProperties": false, + "required": [ + "model", + "reasoning_effort", + "configuration_profile", + "tool_profile" + ], + "properties": { + "model": { "type": ["string", "null"] }, + "reasoning_effort": { "type": ["string", "null"] }, + "configuration_profile": { "type": "string" }, + "tool_profile": { "type": "string" } + } + } + } + }, + "duration_seconds": { "type": "number", "minimum": 0 }, + "exit": { + "type": "object", + "required": ["status"], + "properties": { + "status": { "type": ["integer", "null"] }, + "failure": {} + } + }, + "input": { + "type": "object", + "required": ["prompt_sha256", "run_json_sha256", "profile_sha256"] + }, + "isolation": { + "type": "object", + "required": ["status", "level", "hard_filesystem_confinement", "capabilities", "mechanisms"], + "properties": { + "status": { "enum": ["verified", "unverified"] }, + "level": { "enum": ["strict", "pragmatic", "unsupported"] }, + "hard_filesystem_confinement": { "type": "boolean" }, + "capabilities": { "type": "object" }, + "mechanisms": { "type": "array", "items": { "type": "string" } }, + "required_controls": { "type": "array", "items": { "type": "string" } }, + "unproven_controls": { "type": "array", "items": { "type": "string" } } + } + }, + "telemetry": { "type": "object" }, + "evidence": { + "type": "object", + "properties": { + "interaction": { + "type": "object", + "additionalProperties": true, + "required": ["schema", "mode", "same_session", "session_id", "turns", "final_response_sequence"], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-interaction/1" }, + "mode": { "const": "scripted" }, + "same_session": { "const": true }, + "session_id": { "type": "string", "minLength": 1 }, + "turns": { "type": "array", "minItems": 2 }, + "final_response_sequence": { "type": "integer", "minimum": 2 } + } + }, + "delegation": { + "type": "object", + "additionalProperties": true, + "required": [ + "mechanism", + "worker_session_id", + "observed_model", + "observed_working_directory", + "observed_home", + "fresh_worker", + "home_config_isolated", + "prompt_fidelity", + "prompt_sha256", + "terminal_result_capture", + "paired_arm_visible", + "grading_material_visible", + "nested_model_execution", + "model_execution_count" + ], + "properties": { + "mechanism": { "type": "string", "minLength": 1 }, + "worker_session_id": { "type": "string", "minLength": 1 }, + "observed_model": { "type": "string", "minLength": 1 }, + "observed_working_directory": { "type": "string", "minLength": 1 }, + "observed_home": { "type": "string", "minLength": 1 }, + "fresh_worker": { "const": true }, + "home_config_isolated": { "const": true }, + "prompt_fidelity": { "const": true }, + "prompt_sha256": { "type": "string", "pattern": "^[0-9a-fA-F]{64}$" }, + "terminal_result_capture": { "const": true }, + "paired_arm_visible": { "const": false }, + "grading_material_visible": { "const": false }, + "nested_model_execution": { "const": false }, + "model_execution_count": { "const": 1 } + } + } + } + }, + "artifacts": { "type": "array" }, + "warnings": { "type": "array", "items": { "type": "string" } }, + "compatibility_deviations": { "type": "array", "items": { "type": "string" } }, + "attempt_count": { "const": 1 } + } +} diff --git a/scripts/eval-runners/contracts/grading.schema.json b/scripts/eval-runners/contracts/grading.schema.json new file mode 100644 index 0000000..23b4aac --- /dev/null +++ b/scripts/eval-runners/contracts/grading.schema.json @@ -0,0 +1,28 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-grading/1", + "title": "Codebelt Agentic Eval Grading", + "type": "object", + "additionalProperties": false, + "required": ["schema", "grading"], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-grading/1" }, + "grading": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["eval_id", "eval_name", "configuration", "assertion_index", "assertion", "passed", "evidence"], + "properties": { + "eval_id": { "type": "integer", "minimum": 1 }, + "eval_name": { "type": "string", "minLength": 1 }, + "configuration": { "enum": ["with_skill", "without_skill"] }, + "assertion_index": { "type": "integer", "minimum": 0 }, + "assertion": { "type": "string", "minLength": 1 }, + "passed": { "type": "boolean" }, + "evidence": { "type": "string" } + } + } + } + } +} diff --git a/scripts/eval-runners/contracts/interaction.schema.json b/scripts/eval-runners/contracts/interaction.schema.json new file mode 100644 index 0000000..5626d61 --- /dev/null +++ b/scripts/eval-runners/contracts/interaction.schema.json @@ -0,0 +1,30 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-interaction/1", + "title": "Codebelt Agentic Scripted Eval Interaction", + "type": "object", + "additionalProperties": false, + "required": ["schema", "mode", "turns"], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-interaction/1" }, + "mode": { "const": "scripted" }, + "turns": { + "type": "array", + "minItems": 2, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["role"], + "properties": { + "role": { "const": "user" }, + "source": { "type": "string", "minLength": 1 }, + "content": { "type": "string", "minLength": 1 } + }, + "oneOf": [ + { "required": ["source"], "not": { "required": ["content"] } }, + { "required": ["content"], "not": { "required": ["source"] } } + ] + } + } + } +} diff --git a/scripts/eval-runners/contracts/native-worker-result.schema.json b/scripts/eval-runners/contracts/native-worker-result.schema.json new file mode 100644 index 0000000..48df39f --- /dev/null +++ b/scripts/eval-runners/contracts/native-worker-result.schema.json @@ -0,0 +1,109 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-native-worker-result/1", + "title": "Codebelt Agentic Native Worker Terminal Result", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "run_id", + "session", + "status", + "run", + "final_response", + "timing", + "exit", + "isolation", + "telemetry", + "evidence", + "capture", + "artifacts", + "warnings", + "compatibility_deviations", + "attempt_count" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-native-worker-result/1" }, + "run_id": { "type": "string", "minLength": 1 }, + "session": { + "type": "object", + "required": ["id", "fresh", "resumed"], + "properties": { + "id": { "type": "string", "minLength": 1 }, + "fresh": { "const": true }, + "resumed": { "const": false } + } + }, + "status": { "enum": ["completed", "failed", "timed_out", "cancelled", "incompatible"] }, + "run": { + "type": "object", + "required": ["eval_id", "eval_name", "configuration"], + "properties": { + "eval_id": { "type": "integer", "minimum": 1 }, + "eval_name": { "type": "string", "minLength": 1 }, + "configuration": { "enum": ["with_skill", "without_skill"] } + } + }, + "final_response": { + "type": "object", + "required": ["status"], + "properties": { + "status": { "enum": ["available", "unavailable"] }, + "text": { "type": "string" }, + "reason": { "type": "string" } + } + }, + "timing": { + "type": "object", + "required": ["started_utc", "finished_utc", "duration_seconds"], + "properties": { + "started_utc": { "type": "string", "format": "date-time" }, + "finished_utc": { "type": "string", "format": "date-time" }, + "duration_seconds": { "type": "number", "minimum": 0 } + } + }, + "exit": { + "type": "object", + "required": ["status"], + "properties": { + "status": { "type": ["integer", "null"] }, + "failure": {} + } + }, + "isolation": { + "type": "object", + "required": ["capabilities", "mechanisms"], + "properties": { + "capabilities": { "type": "object" }, + "mechanisms": { "type": "array", "items": { "type": "string" } } + } + }, + "telemetry": { "type": "object" }, + "evidence": { + "type": "object", + "required": ["delegation"], + "properties": { + "delegation": { "type": "object" } + } + }, + "capture": { + "type": "object", + "required": ["source", "terminal", "worker_authored"], + "properties": { + "source": { "const": "harness_native_transport" }, + "terminal": { "const": true }, + "worker_authored": { "const": false } + } + }, + "artifacts": { + "type": "array", + "items": { + "type": "object", + "required": ["path", "scope", "sha256", "size", "media_type"] + } + }, + "warnings": { "type": "array", "items": { "type": "string" } }, + "compatibility_deviations": { "type": "array", "items": { "type": "string" } }, + "attempt_count": { "const": 1 } + } +} diff --git a/scripts/eval-runners/contracts/orchestration-plan.schema.json b/scripts/eval-runners/contracts/orchestration-plan.schema.json new file mode 100644 index 0000000..8d028d4 --- /dev/null +++ b/scripts/eval-runners/contracts/orchestration-plan.schema.json @@ -0,0 +1,83 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-orchestration-plan/1", + "title": "Codebelt Agentic Eval Native Worker Orchestration Plan", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "protocol_version", + "runner", + "model", + "dispatch_owner", + "requested_concurrency", + "parallel_dispatch_required", + "minimum_parallel_workers", + "native_worker_required", + "parent_executes_arms", + "nested_model_execution", + "arms" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-orchestration-plan/1" }, + "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, + "runner": { "type": "string", "minLength": 1 }, + "model": { "type": "string", "minLength": 1 }, + "dispatch_owner": { "enum": ["orchestrator", "runner"] }, + "requested_concurrency": { "type": "integer", "minimum": 1 }, + "parallel_dispatch_required": { "type": "boolean" }, + "minimum_parallel_workers": { "type": "integer", "minimum": 1 }, + "native_worker_required": { "const": true }, + "parent_executes_arms": { "const": false }, + "nested_model_execution": { "const": false }, + "arms": { + "type": "array", + "items": { + "type": "object", + "required": ["worker_id", "eval_id", "eval_name", "configuration", "depends_on", "parent_paths", "dispatch_owner", "worker"], + "properties": { + "worker_id": { "type": "string", "minLength": 1 }, + "eval_id": { "type": "integer", "minimum": 1 }, + "eval_name": { "type": "string", "minLength": 1 }, + "configuration": { "enum": ["with_skill", "without_skill"] }, + "dispatch_owner": { "enum": ["orchestrator", "runner"] }, + "depends_on": { "type": "array", "maxItems": 0 }, + "parent_paths": { + "type": "object", + "required": ["run_manifest", "execution_result", "result"], + "properties": { + "run_manifest": { "type": "string", "minLength": 1 }, + "execution_result": { "type": "string", "minLength": 1 }, + "result": { "type": "string", "minLength": 1 } + } + }, + "worker": { + "type": "object", + "required": [ + "worker_id", "eval_id", "eval_name", "configuration", "run_manifest", "run_manifest_path", + "model", "one_arm_only", "paired_arm_visible", "grading_material_visible", "parent_executes_arm", + "dispatch_owner", "runner_execute_invocation", "nested_model_execution", "model_execution_count" + ], + "properties": { + "worker_id": { "type": "string", "minLength": 1 }, + "eval_id": { "type": "integer", "minimum": 1 }, + "eval_name": { "type": "string", "minLength": 1 }, + "configuration": { "enum": ["with_skill", "without_skill"] }, + "run_manifest": { "type": "string", "minLength": 1 }, + "run_manifest_path": { "type": "string", "minLength": 1 }, + "model": { "type": "string", "minLength": 1 }, + "one_arm_only": { "const": true }, + "paired_arm_visible": { "const": false }, + "grading_material_visible": { "const": false }, + "parent_executes_arm": { "const": false }, + "dispatch_owner": { "enum": ["orchestrator", "runner"] }, + "runner_execute_invocation": { "enum": ["forbidden", "required"] }, + "nested_model_execution": { "const": false }, + "model_execution_count": { "const": 1 } + } + } + } + } + } + } +} diff --git a/scripts/eval-runners/contracts/preflight-result.schema.json b/scripts/eval-runners/contracts/preflight-result.schema.json new file mode 100644 index 0000000..c95636b --- /dev/null +++ b/scripts/eval-runners/contracts/preflight-result.schema.json @@ -0,0 +1,87 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-runner-preflight/1", + "title": "Codebelt Agentic Eval Runner Preflight Result", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "protocol_version", + "status", + "runner", + "harness", + "run", + "requested", + "checks", + "resolved_capabilities", + "delegation", + "isolation", + "mechanisms", + "warnings", + "reasons" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-runner-preflight/1" }, + "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, + "status": { "enum": ["compatible", "incompatible"] }, + "requested": { + "type": "object", + "additionalProperties": false, + "required": [ + "model", + "reasoning_effort", + "configuration_profile", + "tool_profile", + "timeout_seconds" + ], + "properties": { + "model": { "type": ["string", "null"] }, + "reasoning_effort": { "type": ["string", "null"] }, + "configuration_profile": { "type": "string" }, + "tool_profile": { "type": "string" }, + "timeout_seconds": { "type": "integer" } + } + }, + "checks": { "type": "array" }, + "resolved_capabilities": { "type": "object" }, + "delegation": { + "type": "object", + "additionalProperties": false, + "required": [ + "dispatch_owner", + "status", + "mode", + "mechanism", + "worker_role", + "nested_model_execution", + "required_controls", + "unproven_controls", + "terminal_evidence_required" + ], + "properties": { + "dispatch_owner": { "enum": ["orchestrator", "runner"] }, + "status": { "enum": ["supported", "conditional", "unsupported"] }, + "mode": { "enum": ["native_worker", "conditional", "unsupported"] }, + "mechanism": { "type": "string" }, + "worker_role": { "type": "string" }, + "nested_model_execution": { "type": "boolean" }, + "required_controls": { "type": "array", "items": { "type": "string" } }, + "unproven_controls": { "type": "array", "items": { "type": "string" } }, + "terminal_evidence_required": { "type": "boolean" } + } + }, + "isolation": { + "type": "object", + "required": ["level", "status", "hard_filesystem_confinement", "unproven_controls"], + "properties": { + "level": { "enum": ["strict", "pragmatic", "unsupported"] }, + "status": { "enum": ["verified", "unverified"] }, + "hard_filesystem_confinement": { "type": "boolean" }, + "unproven_controls": { "type": "array", "items": { "type": "string" } } + } + }, + "mechanisms": { "type": "array", "items": { "type": "string" } }, + "warnings": { "type": "array", "items": { "type": "string" } }, + "reasons": { "type": "array", "items": { "type": "string" } } + } +} diff --git a/scripts/eval-runners/contracts/runner-descriptor.schema.json b/scripts/eval-runners/contracts/runner-descriptor.schema.json new file mode 100644 index 0000000..68e8f3d --- /dev/null +++ b/scripts/eval-runners/contracts/runner-descriptor.schema.json @@ -0,0 +1,71 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-runner-descriptor/1", + "title": "Codebelt Agentic Eval Runner Descriptor", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "protocol_version", + "name", + "version", + "platforms", + "harness", + "capabilities", + "delegation", + "supported_telemetry", + "configuration_profiles", + "tool_profiles" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-runner-descriptor/1" }, + "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, + "name": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 }, + "platforms": { "type": "array", "items": { "type": "string" } }, + "harness": { + "type": "object", + "required": ["name", "version"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 } + } + }, + "capabilities": { + "type": "object", + "additionalProperties": { "enum": ["supported", "conditional", "unsupported"] }, + "required": ["single_turn", "scripted_multi_turn_same_session"] + }, + "delegation": { + "type": "object", + "additionalProperties": false, + "required": [ + "dispatch_owner", + "mode", + "mechanism", + "worker_role", + "full_capability", + "model_lock", + "working_directory", + "result_capture", + "capacity", + "nested_model_execution" + ], + "properties": { + "dispatch_owner": { "enum": ["orchestrator", "runner"] }, + "mode": { "enum": ["native_worker", "conditional", "unsupported"] }, + "mechanism": { "type": "string", "minLength": 1 }, + "worker_role": { "type": "string", "minLength": 1 }, + "full_capability": { "enum": ["supported", "conditional", "unsupported"] }, + "model_lock": { "enum": ["supported", "conditional", "unsupported"] }, + "working_directory": { "enum": ["supported", "conditional", "unsupported"] }, + "result_capture": { "enum": ["supported", "conditional", "unsupported"] }, + "capacity": { "enum": ["supported", "conditional", "unsupported", "harness_authoritative"] }, + "nested_model_execution": { "const": false } + } + }, + "supported_telemetry": { "type": "array", "items": { "type": "string" } }, + "configuration_profiles": { "type": "array", "items": { "type": "string" } }, + "tool_profiles": { "type": "array", "items": { "type": "string" } } + } +} diff --git a/scripts/eval-runners/execution-freeze.ps1 b/scripts/eval-runners/execution-freeze.ps1 new file mode 100644 index 0000000..67adc7e --- /dev/null +++ b/scripts/eval-runners/execution-freeze.ps1 @@ -0,0 +1,494 @@ +<#! +.SYNOPSIS + Shared immutable raw-evidence freeze and validation helpers. + +.DESCRIPTION + The freeze is created only after every Phase 1 arm has a terminal, + runner-produced result. It records the exact manifest destinations and + hashes of those results and every raw artifact they reference. Later + bridge, grading, and reporting operations validate this ledger; none of + them can replace it when a file changes. +#> +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +if (-not (Get-Command Get-RunnerSchemaNames -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'runner-common.ps1') +} +if (-not (Get-Command Get-ManifestRunRecords -ErrorAction SilentlyContinue)) { + . (Join-Path $PSScriptRoot 'manifest-paths.ps1') +} + +function Get-ExecutionFreezePath { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [string]$RelativePath = 'execution-freeze.json' + ) + + Assert-SafeRelativePath -RelativePath $RelativePath -FieldName 'execution freeze path' + return [System.IO.Path]::GetFullPath((Join-Path ((Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path) ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar))) +} + +function Get-FreezeArmKey { + param([Parameter(Mandatory = $true)][object]$Record) + + return ('arm-{0}-{1}' -f [int]$Record.EvalId, [string]$Record.Configuration) +} + +function Resolve-FreezeArtifactPath { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Artifact + ) + + $path = [string](Get-JsonProperty -Object $Artifact -Name 'path' -Default '') + $scope = [string](Get-JsonProperty -Object $Artifact -Name 'scope' -Default '') + if ($scope -eq 'run') { + return Resolve-ContainedPath -BasePath $RunData.RunRoot -RelativePath $path -FieldName 'execution-result artifact.path' -Kind File + } + if ($scope -eq 'package') { + return Resolve-ContainedPath -BasePath $IterationDirectory -RelativePath $path -FieldName 'execution-result package artifact.path' -Kind File + } + throw "Unsupported execution-result artifact scope '$scope'." +} + +function Get-FreezeObservedModel { + param([Parameter(Mandatory = $true)][object]$ExecutionResult) + + $delegation = Get-JsonProperty -Object $ExecutionResult.evidence -Name 'delegation' -Default $null + $observed = Get-JsonProperty -Object $delegation -Name 'observed_model' -Default $null + if ($null -ne $observed -and -not [string]::IsNullOrWhiteSpace([string]$observed)) { + return [string]$observed + } + $resolved = Get-JsonProperty -Object $ExecutionResult.resolved -Name 'model' -Default $null + if ($null -ne $resolved -and -not [string]::IsNullOrWhiteSpace([string]$resolved)) { + return [string]$resolved + } + return $null +} + +function Get-FreezeThreadId { + param([Parameter(Mandatory = $true)][object]$ExecutionResult) + + $delegation = Get-JsonProperty -Object $ExecutionResult.evidence -Name 'delegation' -Default $null + foreach ($name in @('thread_id', 'thread_session_id', 'session_id')) { + $value = Get-JsonProperty -Object $delegation -Name $name -Default $null + if ($null -ne $value -and -not [string]::IsNullOrWhiteSpace([string]$value)) { return [string]$value } + } + return $null +} + +function New-FreezeArtifactEntry { + param( + [Parameter(Mandatory = $true)][object]$Artifact, + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$IterationDirectory + ) + + $path = [string](Get-JsonProperty -Object $Artifact -Name 'path' -Default '') + $scope = [string](Get-JsonProperty -Object $Artifact -Name 'scope' -Default '') + $fullPath = Resolve-FreezeArtifactPath -RunData $RunData -IterationDirectory $IterationDirectory -Artifact $Artifact + $actualHash = Get-Sha256HexFromFile -Path $fullPath + $recordedHash = [string](Get-JsonProperty -Object $Artifact -Name 'sha256' -Default '') + if ($actualHash -ne $recordedHash) { + throw "Execution integrity failure: raw artifact '${scope}:${path}' does not match the hash in execution-result.json." + } + $actualSize = [int64](Get-Item -LiteralPath $fullPath).Length + $recordedSize = [int64](Get-JsonProperty -Object $Artifact -Name 'size' -Default -1) + if ($actualSize -ne $recordedSize) { + throw "Execution integrity failure: raw artifact '${scope}:${path}' does not match the size in execution-result.json." + } + + return [ordered]@{ + scope = $scope + path = $path + sha256 = $actualHash + size = $actualSize + media_type = [string](Get-JsonProperty -Object $Artifact -Name 'media_type' -Default '') + } +} + +function Assert-FreezeTerminalLedgerEntry { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][object]$Raw, + [AllowNull()][object]$State + ) + + if ($null -eq $State) { return $true } + $completed = Get-JsonProperty -Object $State -Name 'completed' -Default $null + if ($null -eq $completed) { return $true } + $workerId = Get-FreezeArmKey -Record $Record + if (-not (Test-JsonProperty -Object $completed -Name $workerId)) { + throw "Execution integrity failure: orchestration terminal ledger is missing '$workerId'." + } + $terminal = Get-JsonProperty -Object $completed -Name $workerId -Default $null + if ([string](Get-JsonProperty -Object $terminal -Name 'worker_id' -Default '') -ne $workerId -or + [int](Get-JsonProperty -Object $terminal -Name 'eval_id' -Default 0) -ne [int]$Record.EvalId -or + [string](Get-JsonProperty -Object $terminal -Name 'configuration' -Default '') -ne [string]$Record.Configuration) { + throw "Execution integrity failure: orchestration terminal ledger identity for '$workerId' does not match the manifest arm." + } + if ([string](Get-JsonProperty -Object $terminal -Name 'status' -Default '') -ne [string]$Raw.status) { + throw "Execution integrity failure: orchestration terminal ledger status for '$workerId' does not match its frozen raw result." + } + $ledgerSession = [string](Get-JsonProperty -Object $terminal -Name 'worker_session_id' -Default '') + if (-not [string]::IsNullOrWhiteSpace($ledgerSession) -and $ledgerSession -ne [string]$Raw.session.id) { + throw "Execution integrity failure: orchestration terminal ledger session for '$workerId' does not match its frozen raw result." + } + return $true +} + +function Get-OrchestrationCompletedEntries { + param([AllowNull()][object]$State) + + if ($null -eq $State) { return @() } + $completed = Get-JsonProperty -Object $State -Name 'completed' -Default $null + if ($null -eq $completed) { return @() } + + return @((Get-JsonPropertyNames -Object $completed | Sort-Object) | ForEach-Object { + $entry = Get-JsonProperty -Object $completed -Name ([string]$_) -Default $null + if ($null -ne $entry) { $entry } + }) +} + +function Test-FanoutPhase1Success { + param([Parameter(Mandatory = $true)][object]$Aggregate) + + return ( + [int](Get-JsonProperty -Object $Aggregate -Name 'terminal_count' -Default 0) -eq [int](Get-JsonProperty -Object $Aggregate -Name 'expected_count' -Default 0) -and + [int](Get-JsonProperty -Object $Aggregate -Name 'completed_count' -Default 0) -eq [int](Get-JsonProperty -Object $Aggregate -Name 'expected_count' -Default 0) -and + [int](Get-JsonProperty -Object $Aggregate -Name 'failed_count' -Default 0) -eq 0 -and + [int](Get-JsonProperty -Object $Aggregate -Name 'timed_out_count' -Default 0) -eq 0 -and + [int](Get-JsonProperty -Object $Aggregate -Name 'cancelled_count' -Default 0) -eq 0 -and + [int](Get-JsonProperty -Object $Aggregate -Name 'incompatible_count' -Default 0) -eq 0 -and + [int](Get-JsonProperty -Object $Aggregate -Name 'evidence_validation_failed_count' -Default 0) -eq 0 + ) +} + +function Get-FanoutPhase1Aggregate { + param( + [Parameter(Mandatory = $true)][int]$ExpectedCount, + [AllowNull()][object]$State + ) + + $entries = @(Get-OrchestrationCompletedEntries -State $State) + $aggregate = [ordered]@{ + expected_count = $ExpectedCount + terminal_count = $entries.Count + completed_count = @($entries | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'status' -Default '') -eq 'completed' }).Count + failed_count = @($entries | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'status' -Default '') -eq 'failed' }).Count + timed_out_count = @($entries | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'status' -Default '') -eq 'timed_out' }).Count + cancelled_count = @($entries | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'status' -Default '') -eq 'cancelled' }).Count + incompatible_count = @($entries | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'status' -Default '') -eq 'incompatible' }).Count + evidence_validation_failed_count = @($entries | Where-Object { + [string](Get-JsonProperty -Object (Get-JsonProperty -Object $_ -Name 'evidence_validation' -Default $null) -Name 'status' -Default '') -ne 'passed' + }).Count + } + $aggregate.status = if (Test-FanoutPhase1Success -Aggregate $aggregate) { 'completed' } else { 'failed' } + return $aggregate +} + +function Format-FanoutPhase1Aggregate { + param([Parameter(Mandatory = $true)][object]$Aggregate) + + $orderedFields = @( + 'expected_count', + 'terminal_count', + 'completed_count', + 'failed_count', + 'timed_out_count', + 'cancelled_count', + 'incompatible_count', + 'evidence_validation_failed_count' + ) + return [string]::Join(', ', @($orderedFields | ForEach-Object { + "$_=$(Get-JsonProperty -Object $Aggregate -Name $_ -Default 0)" + })) +} + +function Assert-FanoutPhase1Success { + param( + [Parameter(Mandatory = $true)][object]$Aggregate, + [string]$MessagePrefix = 'Phase 1' + ) + + if (-not (Test-FanoutPhase1Success -Aggregate $Aggregate)) { + throw "$MessagePrefix completion gate failed: $(Format-FanoutPhase1Aggregate -Aggregate $Aggregate)." + } + return $true +} + +function New-ExecutionFreezeDocument { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest, + [Parameter(Mandatory = $true)][object[]]$Records, + [Parameter(Mandatory = $true)][object]$Profile + ) + + $iterationPath = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + $manifestRecords = @(Get-ManifestRunRecords -IterationDirectory $iterationPath -Manifest $Manifest) + if ($manifestRecords.Count -ne $Records.Count) { + throw "Execution freeze received $($Records.Count) records, but manifest.json declares $($manifestRecords.Count) arms." + } + + $statePath = Join-Path $iterationPath 'orchestration-state.json' + $state = if (Test-Path -LiteralPath $statePath -PathType Leaf) { Read-RunnerJson -Path $statePath } else { $null } + $orchestrationStateHash = if ($null -eq $state) { + $null + } else { + # The state receives the freeze reference only after this document is + # written. Hash every other state field now so the terminal ledger and + # concurrency evidence cannot be hand-edited after Phase 1. + Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $state -PropertyName 'execution_freeze') + } + $entries = [System.Collections.Generic.List[object]]::new() + foreach ($record in @($manifestRecords | Sort-Object EvalId, Configuration)) { + $executionPath = [string]$record.ExecutionResultPath + if (-not (Test-Path -LiteralPath $executionPath -PathType Leaf)) { + throw "Execution freeze cannot be created because '$($record.ExecutionResultRelative)' is missing." + } + $runData = Resolve-RunContract -RunPath $record.RunManifestPath + $raw = Read-RunnerJson -Path $executionPath + [void](Assert-ExecutionResult -Result $raw) + if ([int]$raw.run.eval_id -ne [int]$record.EvalId -or + [string]$raw.run.eval_name -ne [string]$record.EvalName -or + [string]$raw.run.configuration -ne [string]$record.Configuration) { + throw "Execution integrity failure: '$($record.ExecutionResultRelative)' does not identify its exact manifest arm." + } + if ([string]$raw.input.prompt_sha256 -ne [string]$runData.PromptHash -or + [string]$raw.input.run_json_sha256 -ne (Get-Sha256HexFromFile -Path $runData.RunPath) -or + [string]$raw.input.profile_sha256 -ne [string]$Profile.Hash) { + throw "Execution integrity failure: '$($record.ExecutionResultRelative)' has an input hash that does not match its prepared run or profile." + } + [void](Assert-InteractionResultEvidence -ExecutionResult $raw -RunData $runData) + [void](Assert-FreezeTerminalLedgerEntry -Record $record -Raw $raw -State $state) + + $artifacts = [System.Collections.Generic.List[object]]::new() + foreach ($artifact in @($raw.artifacts | Sort-Object @{ Expression = { [string]$_.scope } }, @{ Expression = { [string]$_.path } })) { + $artifacts.Add((New-FreezeArtifactEntry -Artifact $artifact -RunData $runData -IterationDirectory $iterationPath)) + } + $runner = Get-JsonProperty -Object $raw -Name 'runner' -Default $null + $harness = Get-JsonProperty -Object $raw -Name 'harness' -Default $null + $entries.Add([ordered]@{ + worker_id = Get-FreezeArmKey -Record $record + eval_id = [int]$record.EvalId + eval_name = [string]$record.EvalName + configuration = [string]$record.Configuration + run_id = [string]$raw.run_id + run_manifest = [string]$record.RunManifestRelative + execution_result = [string]$record.ExecutionResultRelative + execution_result_sha256 = Get-Sha256HexFromFile -Path $executionPath + raw_artifacts = @($artifacts.ToArray()) + runner = [ordered]@{ name = [string]$runner.name; version = [string]$runner.version } + harness = [ordered]@{ name = [string]$harness.name; version = [string]$harness.version } + requested_model = [string](Get-JsonProperty -Object $raw.requested -Name 'model' -Default '') + observed_model = Get-FreezeObservedModel -ExecutionResult $raw + resolved_model = Get-JsonProperty -Object $raw.resolved -Name 'model' -Default $null + session_id = [string]$raw.session.id + thread_id = Get-FreezeThreadId -ExecutionResult $raw + terminal_status = [string]$raw.status + finished_utc = [string]$raw.finished_utc + }) + } + + $schemas = Get-RunnerSchemaNames + return [ordered]@{ + schema = $schemas.ExecutionFreeze + version = 1 + generated_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow) + manifest_schema = [string](Get-JsonProperty -Object $Manifest -Name 'schema' -Default '') + profile_sha256 = [string]$Profile.Hash + runner = [string]$Profile.Runner + model = [string]$Profile.Model + orchestration_state_sha256 = $orchestrationStateHash + executions = @($entries.ToArray()) + } +} + +function Write-ExecutionFreezeDocument { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Freeze, + [string]$RelativePath = 'execution-freeze.json' + ) + + $freezePath = Get-ExecutionFreezePath -IterationDirectory $IterationDirectory -RelativePath $RelativePath + if (Test-Path -LiteralPath $freezePath -PathType Leaf) { + throw "Execution freeze already exists at '$freezePath'; a second freeze would re-bless a new raw-evidence state." + } + $schemas = Get-RunnerSchemaNames + if ([string]$Freeze.schema -ne $schemas.ExecutionFreeze -or [int]$Freeze.version -ne 1) { + throw 'Execution freeze has an unsupported schema or version.' + } + [System.IO.File]::WriteAllText($freezePath, (($Freeze | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + return $freezePath +} + +function Get-FreezeEntryMap { + param([Parameter(Mandatory = $true)][object]$Freeze) + + $map = @{} + $previousEvalId = -1 + $previousConfiguration = '' + foreach ($entry in @($Freeze.executions)) { + $evalId = [int](Get-JsonProperty -Object $entry -Name 'eval_id' -Default 0) + $configuration = [string](Get-JsonProperty -Object $entry -Name 'configuration' -Default '') + $key = "$evalId|$configuration" + if ($map.ContainsKey($key)) { throw "Execution freeze contains duplicate arm '$key'." } + if ($evalId -lt $previousEvalId -or ($evalId -eq $previousEvalId -and [string]::CompareOrdinal($previousConfiguration, $configuration) -gt 0)) { + throw 'Execution freeze executions must be in deterministic eval_id/configuration order.' + } + $previousEvalId = $evalId + $previousConfiguration = $configuration + $map[$key] = $entry + } + return $map +} + +function Assert-ExecutionFreeze { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [string]$RelativePath = 'execution-freeze.json', + [switch]$RequireOrchestrationState + ) + + $iterationPath = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + $manifestPath = Join-Path $iterationPath 'manifest.json' + $profilePath = Join-Path $iterationPath 'execution-profile.json' + if (-not (Test-Path -LiteralPath $manifestPath -PathType Leaf)) { throw 'Execution integrity failure: manifest.json is missing.' } + if (-not (Test-Path -LiteralPath $profilePath -PathType Leaf)) { throw 'Execution integrity failure: execution-profile.json is missing.' } + $manifest = Read-RunnerJson -Path $manifestPath + $declaredFreezePath = [string](Get-JsonProperty -Object $manifest -Name 'execution_freeze' -Default '') + if ([string]::IsNullOrWhiteSpace($declaredFreezePath)) { throw 'Execution integrity failure: manifest.json does not declare execution_freeze.' } + if ($declaredFreezePath -ne $RelativePath) { + throw "Execution integrity failure: requested freeze path '$RelativePath' does not match manifest.execution_freeze '$declaredFreezePath'." + } + $profile = Resolve-ExecutionProfile -ProfilePath $profilePath + $freezePath = Get-ExecutionFreezePath -IterationDirectory $iterationPath -RelativePath $RelativePath + if (-not (Test-Path -LiteralPath $freezePath -PathType Leaf)) { + throw "Execution integrity failure: frozen raw evidence is missing at '$RelativePath'; Phase 1 must be rerun." + } + $freeze = Read-RunnerJson -Path $freezePath + $schemas = Get-RunnerSchemaNames + if ([string]$freeze.schema -ne $schemas.ExecutionFreeze -or [int]$freeze.version -ne 1) { + throw 'Execution integrity failure: execution-freeze.json has an unsupported schema or version.' + } + if ([string]$freeze.manifest_schema -ne [string](Get-JsonProperty -Object $manifest -Name 'schema' -Default '') -or + [string]$freeze.profile_sha256 -ne [string]$profile.Hash -or [string]$freeze.runner -ne [string]$profile.Runner -or [string]$freeze.model -ne [string]$profile.Model) { + throw 'Execution integrity failure: execution-freeze.json does not match the selected execution profile.' + } + + $statePath = Join-Path $iterationPath 'orchestration-state.json' + $state = $null + if ($RequireOrchestrationState -and -not (Test-Path -LiteralPath $statePath -PathType Leaf)) { + throw 'Execution integrity failure: orchestration-state.json is missing; the Phase 1 terminal ledger is incomplete.' + } + if (Test-Path -LiteralPath $statePath -PathType Leaf) { + $state = Read-RunnerJson -Path $statePath + $freezeRef = Get-JsonProperty -Object $state -Name 'execution_freeze' -Default $null + if ($null -eq $freezeRef) { + throw 'Execution integrity failure: orchestration-state.json does not reference execution-freeze.json.' + } + $declaredPath = [string](Get-JsonProperty -Object $freezeRef -Name 'path' -Default '') + $declaredHash = [string](Get-JsonProperty -Object $freezeRef -Name 'sha256' -Default '') + if ($declaredPath -ne $RelativePath -or $declaredHash -ne (Get-Sha256HexFromFile -Path $freezePath)) { + throw 'Execution integrity failure: execution-freeze.json does not match the immutable orchestration-state reference.' + } + $frozenStateHash = [string](Get-JsonProperty -Object $freeze -Name 'orchestration_state_sha256' -Default '') + if (-not (Test-Sha256 -Value $frozenStateHash)) { + throw 'Execution integrity failure: execution-freeze.json is missing the orchestration-state integrity hash.' + } + $currentStateHash = Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $state -PropertyName 'execution_freeze') + if ($currentStateHash -ne $frozenStateHash) { + throw 'Execution integrity failure: orchestration-state.json changed after the Phase 1 freeze; requires fresh Phase 1 execution.' + } + } + + $records = @(Get-ManifestRunRecords -IterationDirectory $iterationPath -Manifest $manifest | Sort-Object EvalId, Configuration) + $entryMap = Get-FreezeEntryMap -Freeze $freeze + if ($entryMap.Count -ne $records.Count) { + throw "Execution integrity failure: frozen arm count $($entryMap.Count) does not match manifest arm count $($records.Count)." + } + foreach ($record in $records) { + $key = "$($record.EvalId)|$($record.Configuration)" + if (-not $entryMap.ContainsKey($key)) { + throw "Execution integrity failure: execution-freeze.json is missing manifest arm '$($record.EvalName)/$($record.Configuration)'." + } + $entry = $entryMap[$key] + $workerId = Get-FreezeArmKey -Record $record + foreach ($field in @('worker_id', 'eval_id', 'eval_name', 'configuration', 'run_manifest', 'execution_result', 'execution_result_sha256', 'runner', 'harness', 'requested_model', 'session_id', 'terminal_status', 'finished_utc')) { + if (-not (Test-JsonProperty -Object $entry -Name $field)) { throw "Execution integrity failure: frozen arm '$workerId' is missing '$field'." } + } + if ([string]$entry.worker_id -ne $workerId -or [int]$entry.eval_id -ne [int]$record.EvalId -or + [string]$entry.eval_name -ne [string]$record.EvalName -or [string]$entry.configuration -ne [string]$record.Configuration -or + [string]$entry.run_manifest -ne [string]$record.RunManifestRelative -or [string]$entry.execution_result -ne [string]$record.ExecutionResultRelative) { + throw "Execution integrity failure: frozen worker identity for '$workerId' does not match manifest-declared paths." + } + + $runData = Resolve-RunContract -RunPath $record.RunManifestPath + $executionPath = Resolve-ManifestDeclaredPath -IterationDirectory $iterationPath -RelativePath $record.ExecutionResultRelative -FieldName "$workerId.execution_result" -Kind File -RequireExists + $currentHash = Get-Sha256HexFromFile -Path $executionPath + if ($currentHash -ne [string]$entry.execution_result_sha256) { + throw "Execution integrity failure: frozen raw execution result changed for $workerId (expected $($entry.execution_result_sha256), found $currentHash); requires fresh Phase 1 execution." + } + $raw = Read-RunnerJson -Path $executionPath + [void](Assert-ExecutionResult -Result $raw) + [void](Assert-InteractionResultEvidence -ExecutionResult $raw -RunData $runData) + [void](Assert-FreezeTerminalLedgerEntry -Record $record -Raw $raw -State $state) + if ([string](Get-JsonProperty -Object $entry.runner -Name 'name' -Default '') -ne [string]$raw.runner.name -or + [string](Get-JsonProperty -Object $entry.runner -Name 'version' -Default '') -ne [string]$raw.runner.version -or + [string](Get-JsonProperty -Object $entry.harness -Name 'name' -Default '') -ne [string]$raw.harness.name -or + [string](Get-JsonProperty -Object $entry.harness -Name 'version' -Default '') -ne [string]$raw.harness.version -or + [string]$entry.requested_model -ne [string]$raw.requested.model -or + [string]$entry.observed_model -ne [string](Get-FreezeObservedModel -ExecutionResult $raw) -or + [string]$entry.resolved_model -ne [string](Get-JsonProperty -Object $raw.resolved -Name 'model' -Default $null) -or + [string]$entry.thread_id -ne [string](Get-FreezeThreadId -ExecutionResult $raw)) { + throw "Execution integrity failure: frozen transport identity changed for $workerId; requires fresh Phase 1 execution." + } + if ([string]$raw.run_id -ne [string]$entry.run_id -or [string]$raw.session.id -ne [string]$entry.session_id -or + [string]$raw.finished_utc -ne [string]$entry.finished_utc -or [string]$raw.status -ne [string]$entry.terminal_status) { + throw "Execution integrity failure: frozen terminal identity changed for $workerId; requires fresh Phase 1 execution." + } + $artifactMap = @{} + foreach ($artifact in @($entry.raw_artifacts)) { + $artifactKey = "$(Get-JsonProperty -Object $artifact -Name 'scope' -Default '')|$(Get-JsonProperty -Object $artifact -Name 'path' -Default '')" + if ($artifactMap.ContainsKey($artifactKey)) { throw "Execution integrity failure: frozen arm '$workerId' contains duplicate raw artifact '$artifactKey'." } + $artifactMap[$artifactKey] = $artifact + } + $currentArtifacts = @($raw.artifacts) + if ($artifactMap.Count -ne $currentArtifacts.Count) { + throw "Execution integrity failure: raw artifact references changed for $workerId; requires fresh Phase 1 execution." + } + foreach ($artifact in $currentArtifacts) { + $artifactKey = "$(Get-JsonProperty -Object $artifact -Name 'scope' -Default '')|$(Get-JsonProperty -Object $artifact -Name 'path' -Default '')" + if (-not $artifactMap.ContainsKey($artifactKey)) { throw "Execution integrity failure: raw artifact references changed for $workerId; requires fresh Phase 1 execution." } + $frozenArtifact = $artifactMap[$artifactKey] + $fullPath = Resolve-FreezeArtifactPath -RunData $runData -IterationDirectory $iterationPath -Artifact $artifact + $currentArtifactHash = Get-Sha256HexFromFile -Path $fullPath + if ($currentArtifactHash -ne [string]$frozenArtifact.sha256) { + throw "Execution integrity failure: frozen raw artifact changed for $workerId at '$($artifact.path)' (expected $($frozenArtifact.sha256), found $currentArtifactHash); requires fresh Phase 1 execution." + } + if ($currentArtifactHash -ne [string]$artifact.sha256 -or [int64](Get-Item -LiteralPath $fullPath).Length -ne [int64]$frozenArtifact.size) { + throw "Execution integrity failure: raw artifact record changed for $workerId at '$($artifact.path)'; requires fresh Phase 1 execution." + } + } + } + $aggregate = if ($null -eq $state) { + $null + } else { + Get-FanoutPhase1Aggregate -ExpectedCount $records.Count -State $state + } + + return [pscustomobject]@{ + Path = $freezePath + Freeze = $freeze + Manifest = $manifest + Profile = $profile + Records = $records + State = $state + Aggregate = $aggregate + PhaseOneSuccess = if ($null -eq $aggregate) { $null } else { Test-FanoutPhase1Success -Aggregate $aggregate } + } +} diff --git a/scripts/eval-runners/fake/runner.ps1 b/scripts/eval-runners/fake/runner.ps1 new file mode 100644 index 0000000..924bdf7 --- /dev/null +++ b/scripts/eval-runners/fake/runner.ps1 @@ -0,0 +1,306 @@ +<#! +.SYNOPSIS + Deterministic Eval Runner used for protocol conformance tests. + +.DESCRIPTION + This runner never invokes a model or a provider. It records the same + boundaries as a real runner and emits deterministic outcomes selected by + -Scenario or AGENTIC_FAKE_SCENARIO. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile, + + [ValidateSet('normal', 'refusal', 'timeout', 'failure', 'incompatible', 'escape', 'unknown-event')] + [string]$Scenario +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'fake' + version = '0.9.1-test' + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'deterministic-fake'; version = '1' } + capabilities = [ordered]@{ + single_turn = 'supported' + scripted_multi_turn_same_session = 'unsupported' + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'supported' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'unsupported' + cache_token_telemetry = 'unsupported' + tool_call_telemetry = 'supported' + command_evidence = 'supported' + file_evidence = 'supported' + cost_telemetry = 'unsupported' + native_skill_activation_evidence = 'unsupported' + # This runner is a deterministic compatibility fixture. It has no + # harness-native worker surface and cannot prove a delegated child. + native_worker_delegation = 'unsupported' + delegated_worker_full_capability = 'unsupported' + delegated_worker_model_lock = 'unsupported' + delegated_worker_working_directory = 'unsupported' + delegated_worker_result_capture = 'unsupported' + delegated_worker_capacity_signal = 'unsupported' + } + delegation = [ordered]@{ + dispatch_owner = 'orchestrator' + mode = 'unsupported' + mechanism = 'deterministic compatibility execute fixture; no harness-native worker surface' + worker_role = 'compatibility-fixture' + full_capability = 'unsupported' + model_lock = 'unsupported' + working_directory = 'unsupported' + result_capture = 'unsupported' + capacity = 'unsupported' + nested_model_execution = $false + } + supported_telemetry = @('transcript_event_capture', 'tool_call_telemetry', 'command_evidence', 'file_evidence') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Get-FakeScenario { + if (-not [string]::IsNullOrWhiteSpace($Scenario)) { + return $Scenario + } + $fromEnvironment = [Environment]::GetEnvironmentVariable('AGENTIC_FAKE_SCENARIO') + if ([string]::IsNullOrWhiteSpace($fromEnvironment)) { + return 'normal' + } + return $fromEnvironment.ToLowerInvariant() +} + +function Resolve-FakeInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + $resolvedRun = Resolve-RunContract -RunPath $Run + $resolvedProfile = Resolve-ExecutionProfile -ProfilePath $Profile + return [pscustomobject]@{ Run = $resolvedRun; Profile = $resolvedProfile } +} + +function Get-FakePreflight { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [string]$ScenarioValue = 'normal' + ) + + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + + if ($profile.Runner -ne 'fake') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than fake.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is fake.')) + } + + if ($profile.ConfigurationProfile -notin @($descriptor.configuration_profiles)) { + $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is not supported by fake.") + } else { + $checks.Add((New-PreflightCheck -Name 'configuration_profile' -Status passed -Detail $profile.ConfigurationProfile)) + } + if ($profile.ToolProfile -notin @($descriptor.tool_profiles)) { + $reasons.Add("tool_profile '$($profile.ToolProfile)' is not supported by fake.") + } else { + $checks.Add((New-PreflightCheck -Name 'tool_profile' -Status passed -Detail $profile.ToolProfile)) + } + + $checks.Add((New-PreflightCheck -Name 'fresh_process' -Status passed -Detail 'Each fake execute command creates a new process and session id.')) + $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The prompt bytes are read once and recorded without transformation.')) + $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "repo=$($run.WorkingDirectoryPath); home=$($run.HomeDirectoryPath)")) + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail 'Fake escape probes are rejected by the contained-path guard.')) + $checks.Add((New-PreflightCheck -Name 'candidate_skill_boundary' -Status passed -Detail "candidate_skill_exposed=$($run.CandidateSkillExposed)")) + if ($ScenarioValue -eq 'incompatible') { + $reasons.Add('The deterministic incompatible scenario was requested.') + } + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName + } + $capabilities['candidate_skill_exposure'] = if ($run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + if ($reasons.Count -gt 0) { + $warnings.Add('No execute process is started for an incompatible preflight.') + } + + return New-PreflightDocument -Descriptor $descriptor -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @('pwsh-process', 'run-directory-contained-path-guard', 'isolated-home-directory') -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) +} + +function Write-FakeEvidence { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][string]$SessionId, + [Parameter(Mandatory = $true)][string]$ScenarioValue + ) + + $evidenceDirectory = Join-Path $Inputs.Run.RunRoot 'evidence' + New-Item -ItemType Directory -Path $evidenceDirectory -Force | Out-Null + $eventsPath = Join-Path $evidenceDirectory 'fake-events.jsonl' + $promptEvidencePath = Join-Path $evidenceDirectory 'prompt-delivery.json' + $boundaryEvidencePath = Join-Path $evidenceDirectory 'boundary-probes.json' + + $events = [System.Collections.Generic.List[string]]::new() + $events.Add((([ordered]@{ type = 'session.started'; session_id = $SessionId } | ConvertTo-Json -Compress))) + $events.Add((([ordered]@{ + type = 'task.input' + ordinal = 1 + prompt_sha256 = $Inputs.Run.PromptHash + byte_length = $Inputs.Run.PromptBytes.Length + first_task_input = $true + } | ConvertTo-Json -Compress))) + $events.Add((([ordered]@{ type = 'response.completed'; status = if ($ScenarioValue -eq 'refusal') { 'refusal' } else { 'completed' } } | ConvertTo-Json -Compress))) + if ($ScenarioValue -eq 'unknown-event') { + $events.Add((([ordered]@{ type = 'future.event.v99'; payload = 'ignored-by-conformance-adapter' } | ConvertTo-Json -Compress))) + } + [System.IO.File]::WriteAllText($eventsPath, ([string]::Join("`n", $events) + "`n"), [System.Text.UTF8Encoding]::new($false)) + + [ordered]@{ + prompt_sha256 = $Inputs.Run.PromptHash + first_task_input_sha256 = $Inputs.Run.PromptHash + first_task_input_bytes = $Inputs.Run.PromptBytes.Length + byte_exact = $true + candidate_skill_exposed = $Inputs.Run.CandidateSkillExposed + working_directory = $Inputs.Run.WorkingDirectoryPath + home_directory = $Inputs.Run.HomeDirectoryPath + global_rules_visible = $false + global_memory_visible = $false + global_plugins_visible = $false + global_same_name_skill_visible = $false + } | ConvertTo-Json -Depth 20 | Set-Content -LiteralPath $promptEvidencePath -Encoding utf8NoBOM + + $boundary = [ordered]@{ + read_outside_run = [ordered]@{ attempted = $false; blocked = $true; path = '../eval-metadata.json' } + write_outside_run = [ordered]@{ attempted = $false; blocked = $true; path = '../escape-write.txt' } + } + if ($ScenarioValue -eq 'escape') { + foreach ($probe in @('read_outside_run', 'write_outside_run')) { + $boundary[$probe].attempted = $true + try { + [void](Resolve-ContainedPath -BasePath $Inputs.Run.RunRoot -RelativePath ([string]$boundary[$probe].path) -FieldName $probe) + $boundary[$probe].blocked = $false + } catch { + $boundary[$probe].blocked = $true + $boundary[$probe].error = $_.Exception.Message + } + } + } + $boundary | ConvertTo-Json -Depth 20 | Set-Content -LiteralPath $boundaryEvidencePath -Encoding utf8NoBOM + + return @( + (New-ArtifactReference -Run $Inputs.Run -Path 'evidence/fake-events.jsonl' -Scope run -MediaType 'application/x-ndjson'), + (New-ArtifactReference -Run $Inputs.Run -Path 'evidence/prompt-delivery.json' -Scope run -MediaType 'application/json'), + (New-ArtifactReference -Run $Inputs.Run -Path 'evidence/boundary-probes.json' -Scope run -MediaType 'application/json') + ) +} + +function Invoke-FakeExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $scenarioValue = Get-FakeScenario + if ($scenarioValue -notin @('normal', 'refusal', 'timeout', 'failure', 'incompatible', 'escape', 'unknown-event')) { + throw "Unsupported fake scenario '$scenarioValue'." + } + + $preflight = Get-FakePreflight -Inputs $Inputs -ScenarioValue $scenarioValue + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + $warnings = [System.Collections.Generic.List[string]]::new() + $deviations = [System.Collections.Generic.List[string]]::new() + + if ($preflight.status -eq 'incompatible') { + $finished = [DateTime]::UtcNow + $incompatibilityMessage = [string]::Join('; ', @($preflight.reasons)) + if ([string]::IsNullOrWhiteSpace($incompatibilityMessage)) { + $incompatibilityMessage = "Fake preflight reported incompatible without a reason (profile_runner=$($Inputs.Profile.Runner); configuration_profile=$($Inputs.Profile.ConfigurationProfile); tool_profile=$($Inputs.Profile.ToolProfile); scenario=$scenarioValue)." + } + return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message $incompatibilityMessage) -SessionId $sessionId -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'supported'; candidate_skill_exposure = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } }) -IsolationMechanisms @('pwsh-process', 'run-directory-contained-path-guard', 'isolated-home-directory') -CompatibilityDeviations @($deviations) -Evidence ([ordered]@{ preflight = $preflight }) -AttemptCount 1 + } + + $artifacts = @(Write-FakeEvidence -Inputs $Inputs -SessionId $sessionId -ScenarioValue $scenarioValue) + if ($scenarioValue -eq 'unknown-event') { + $warnings.Add('Unknown fake event future.event.v99 was preserved as an explicit warning.') + } + + $status = 'completed' + $finalResponse = 'Deterministic fake runner completed the blind eval arm.' + $finalReason = $null + $exitStatus = [Nullable[int]]0 + $failure = $null + if ($scenarioValue -eq 'refusal') { + $finalResponse = 'I cannot complete this request.' + $warnings.Add('The harness returned a refusal; it was normalized as a completed response, not retried.') + } elseif ($scenarioValue -eq 'timeout') { + $status = 'timed_out' + $finalResponse = $null + $finalReason = 'fake_timeout' + $exitStatus = $null + $failure = New-ExecutionFailure -Code 'timed_out' -Message 'The deterministic fake exceeded its configured execution window.' + } elseif ($scenarioValue -eq 'failure') { + $status = 'failed' + $finalResponse = $null + $finalReason = 'harness_failure' + $exitStatus = [Nullable[int]]17 + $failure = New-ExecutionFailure -Code 'fake_harness_failure' -Message 'The deterministic fake reported a harness failure.' + } + + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/fake-events.jsonl'; complete = $true }) + tokens = New-UnavailableMetric -Reason 'fake_harness_does_not_expose_usage' + tool_calls = New-AvailableMetric -Value 0 + cost = New-UnavailableMetric -Reason 'fake_harness_does_not_expose_cost' + } + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalResponse -FinalResponseReason $finalReason -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'supported'; ambient_candidate_skill_exclusion = 'supported'; candidate_skill_exposure = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' }; prompt_fidelity = 'supported'; model_configuration_lock = 'supported'; response_capture = 'supported' }) -IsolationMechanisms @('pwsh-process', 'run-directory-contained-path-guard', 'isolated-home-directory') -Telemetry $telemetry -Artifacts $artifacts -Warnings @($warnings) -CompatibilityDeviations @($deviations) -Evidence ([ordered]@{ scenario = $scenarioValue; prompt_first_input = $true; resume = $false; preflight = $preflight }) -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { + Write-RunnerJson -Value $descriptor -AsOutput + } + 'preflight' { + $inputs = Resolve-FakeInputs + $preflight = Get-FakePreflight -Inputs $inputs -ScenarioValue (Get-FakeScenario) + Write-RunnerJson -Value $preflight -AsOutput + } + 'execute' { + $inputs = Resolve-FakeInputs + [void](Assert-PhaseOneEvidenceWritable -Run $inputs.Run) + $result = Invoke-FakeExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + [Console]::Error.WriteLine($_.Exception.ToString()) + exit 2 +} diff --git a/scripts/eval-runners/fanout-process.ps1 b/scripts/eval-runners/fanout-process.ps1 new file mode 100644 index 0000000..6d87c40 --- /dev/null +++ b/scripts/eval-runners/fanout-process.ps1 @@ -0,0 +1,224 @@ +<#! +.SYNOPSIS + Headless child-process helpers for the runner-owned behavioral fan-out. + +.DESCRIPTION + The runner-owned fan-out starts one fresh child pwsh process per preflight + and per eval execution. These helpers keep those children as real process + isolation boundaries while making them headless on Windows: no visible + console window flashes for each child, and stdout/stderr are streamed to the + declared files as raw bytes. They also provide a wait-any primitive so a + completed child frees its concurrency slot immediately, regardless of the + order children were started. + + This file is dot-sourced by invoke-runner-owned-arms.ps1 and by the + orchestration tests. It never runs during preparation, validation, CI, + hooks, or reporting, and it never invokes a model. +#> +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +function New-RunnerChildProcessStartInfo { + <# + Builds the headless child-process configuration used for every + runner-owned preflight and execution child. UseShellExecute=false with + CreateNoWindow=true suppresses the per-child console window on Windows, + while redirected stdout/stderr let the parent capture the child's exact + output. This is a deterministic, testable configuration probe. + #> + param( + [Parameter(Mandatory = $true)][string]$FilePath, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][string[]]$ArgumentList, + [Parameter(Mandatory = $true)][string]$WorkingDirectory + ) + + $startInfo = [System.Diagnostics.ProcessStartInfo]::new() + $startInfo.FileName = $FilePath + foreach ($argument in $ArgumentList) { $startInfo.ArgumentList.Add([string]$argument) } + $startInfo.WorkingDirectory = $WorkingDirectory + $startInfo.UseShellExecute = $false + $startInfo.CreateNoWindow = $true + $startInfo.RedirectStandardOutput = $true + $startInfo.RedirectStandardError = $true + $startInfo.StandardOutputEncoding = [System.Text.UTF8Encoding]::new($false) + $startInfo.StandardErrorEncoding = [System.Text.UTF8Encoding]::new($false) + return $startInfo +} + +function Start-RunnerChildProcess { + <# + Starts a headless child process and streams its stdout/stderr to the + declared files as raw bytes. The returned record exposes the live process + and the asynchronous copy tasks so the caller can wait for completion and + flush the destination files deterministically. + #> + param( + [Parameter(Mandatory = $true)][string]$FilePath, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][string[]]$ArgumentList, + [Parameter(Mandatory = $true)][string]$WorkingDirectory, + [Parameter(Mandatory = $true)][string]$StdoutPath, + [Parameter(Mandatory = $true)][string]$StderrPath, + [int]$TimeoutSeconds = 900 + ) + + New-Item -ItemType Directory -Path (Split-Path -Parent $StdoutPath) -Force | Out-Null + New-Item -ItemType Directory -Path (Split-Path -Parent $StderrPath) -Force | Out-Null + $startInfo = New-RunnerChildProcessStartInfo -FilePath $FilePath -ArgumentList $ArgumentList -WorkingDirectory $WorkingDirectory + $process = [System.Diagnostics.Process]::new() + $process.StartInfo = $startInfo + $stdoutStream = [System.IO.File]::Open($StdoutPath, [System.IO.FileMode]::Create, [System.IO.FileAccess]::Write, [System.IO.FileShare]::Read) + $stderrStream = [System.IO.File]::Open($StderrPath, [System.IO.FileMode]::Create, [System.IO.FileAccess]::Write, [System.IO.FileShare]::Read) + try { + if (-not $process.Start()) { throw "Failed to start runner child process '$FilePath'." } + } catch { + $stdoutStream.Dispose() + $stderrStream.Dispose() + $process.Dispose() + throw + } + $stdoutTask = $process.StandardOutput.BaseStream.CopyToAsync($stdoutStream) + $stderrTask = $process.StandardError.BaseStream.CopyToAsync($stderrStream) + $startedUtc = [DateTime]::UtcNow + return [pscustomobject]@{ + Process = $process + StdoutPath = $StdoutPath + StderrPath = $StderrPath + StdoutStream = $stdoutStream + StderrStream = $stderrStream + StdoutTask = $stdoutTask + StderrTask = $stderrTask + StartedUtc = $startedUtc + TimeoutSeconds = [Math]::Max(1, $TimeoutSeconds) + DeadlineUtc = $startedUtc.AddSeconds([Math]::Max(1, $TimeoutSeconds)) + WatchdogExpired = $false + TimedOut = $false + TerminationObserved = $false + OutputDrainCompleted = $false + FinishedUtc = $null + } +} + +function Wait-RunnerChildTaskBounded { + param( + [Parameter(Mandatory = $true)][System.Threading.Tasks.Task]$Task, + [Parameter(Mandatory = $true)][int]$TimeoutMilliseconds + ) + + if ($Task.IsCompleted) { return $true } + $bounded = [Math]::Max(1, [Math]::Min($TimeoutMilliseconds, 5000)) + try { return [bool]$Task.Wait($bounded) } catch { return [bool]$Task.IsCompleted } +} + +function Complete-RunnerChildProcess { + <# + Waits for the child to exit, drains and flushes both stdout/stderr copy + tasks, closes the destination streams, and returns the observed exit code. + Safe to call once per started child. + #> + param( + [Parameter(Mandatory = $true)][object]$Child, + [int]$TimeoutSeconds = 0 + ) + + # An explicit timeout is a cleanup override (used by the supervisor's + # error path); otherwise honor the absolute deadline captured at start. + $deadline = if ($TimeoutSeconds -gt 0) { [DateTime]::UtcNow.AddSeconds([Math]::Max(1, $TimeoutSeconds)) } elseif ($null -ne $Child.DeadlineUtc) { [DateTime]$Child.DeadlineUtc } else { [DateTime]::UtcNow.AddSeconds(1) } + $timedOut = [bool]$Child.WatchdogExpired + try { + if (-not $Child.Process.HasExited -and -not $timedOut) { + while (-not $Child.Process.HasExited) { + $remainingTotal = ($deadline - [DateTime]::UtcNow).TotalMilliseconds + if ($remainingTotal -le 0) { + $timedOut = $true + break + } + $remaining = [int][Math]::Max(1, [Math]::Min(5000, $remainingTotal)) + if ($Child.Process.WaitForExit($remaining)) { break } + } + } + } catch { $timedOut = $true } + + if ($timedOut) { + $Child.TimedOut = $true + try { $Child.Process.Kill($true) } catch { } + try { if (-not $Child.Process.HasExited) { [void]$Child.Process.WaitForExit(5000) } } catch { } + } + try { $Child.TerminationObserved = [bool]$Child.Process.HasExited } catch { $Child.TerminationObserved = $false } + + # Use one finite drain deadline. A descendant holding an inherited pipe + # open must not make the fan-out supervisor wait forever. + $drainDeadline = [DateTime]::UtcNow.AddSeconds(5) + $stdoutDone = $false + $stderrDone = $false + foreach ($entry in @( + [pscustomobject]@{ Task = $Child.StdoutTask; Name = 'stdout' }, + [pscustomobject]@{ Task = $Child.StderrTask; Name = 'stderr' } + )) { + try { + $remaining = [int][Math]::Max(1, [Math]::Min(5000, ($drainDeadline - [DateTime]::UtcNow).TotalMilliseconds)) + $done = Wait-RunnerChildTaskBounded -Task $entry.Task -TimeoutMilliseconds $remaining + if ($entry.Name -eq 'stdout') { $stdoutDone = $done } else { $stderrDone = $done } + } catch { } + } + $Child.OutputDrainCompleted = $stdoutDone -and $stderrDone + foreach ($stream in @($Child.StdoutStream, $Child.StderrStream)) { + try { $stream.Flush() } catch { } + try { $stream.Dispose() } catch { } + } + $exitCode = $null + if (-not $timedOut -and $Child.TerminationObserved) { + try { $exitCode = [int]$Child.Process.ExitCode } catch { $exitCode = $null } + } + try { $Child.Process.Dispose() } catch { } + $Child.FinishedUtc = [DateTime]::UtcNow + return $exitCode +} + +function Wait-AnyRunnerChild { + <# + Returns the list index of the first child that has exited, polling until + at least one has. Capacity is released when ANY child completes, not only + the oldest one in the list, so a slow child never blocks refilling the + slot that a faster sibling has already freed. + #> + param( + [Parameter(Mandatory = $true)][System.Collections.IList]$Running, + [int]$PollMilliseconds = 25 + ) + + if ($Running.Count -eq 0) { return -1 } + while ($true) { + for ($index = 0; $index -lt $Running.Count; $index++) { + $child = $Running[$index] + if ($child.Process.HasExited) { return $index } + # The public queue record wraps the process helper as `.child`, + # while the focused helper tests may pass the process record + # directly. Accept both shapes without losing the deadline. + $childDeadline = $null + if ($null -ne $child.PSObject.Properties['DeadlineUtc']) { + $childDeadline = $child.DeadlineUtc + } elseif ($null -ne $child.PSObject.Properties['child'] -and $null -ne $child.child -and $null -ne $child.child.PSObject.Properties['DeadlineUtc']) { + $childDeadline = $child.child.DeadlineUtc + } + if ($null -ne $childDeadline -and [DateTime]::UtcNow -ge [DateTime]$childDeadline) { + if ($null -ne $child.PSObject.Properties['WatchdogExpired']) { $child.WatchdogExpired = $true } + if ($null -ne $child.PSObject.Properties['child']) { $child.child.WatchdogExpired = $true } + return $index + } + } + $sleepMilliseconds = [Math]::Max(1, [Math]::Min($PollMilliseconds, 250)) + $nextDeadline = @($Running | ForEach-Object { + if ($null -ne $_.PSObject.Properties['DeadlineUtc']) { + [DateTime]$_.DeadlineUtc + } elseif ($null -ne $_.PSObject.Properties['child'] -and $null -ne $_.child -and $null -ne $_.child.PSObject.Properties['DeadlineUtc']) { + [DateTime]$_.child.DeadlineUtc + } + } | Sort-Object | Select-Object -First 1) + if ($nextDeadline.Count -eq 1) { + $untilDeadline = [int][Math]::Max(1, [Math]::Min($sleepMilliseconds, ($nextDeadline[0] - [DateTime]::UtcNow).TotalMilliseconds)) + Start-Sleep -Milliseconds $untilDeadline + } else { + Start-Sleep -Milliseconds $sleepMilliseconds + } + } +} diff --git a/scripts/eval-runners/finalize-eval-package.ps1 b/scripts/eval-runners/finalize-eval-package.ps1 new file mode 100644 index 0000000..bbcdc1a --- /dev/null +++ b/scripts/eval-runners/finalize-eval-package.ps1 @@ -0,0 +1,167 @@ +<#! +.SYNOPSIS + Deterministically finalizes a prepared evaluation package. + +.DESCRIPTION + This is the only normal post-grading completion boundary. It validates + Phase 1 state and the immutable raw-evidence freeze, runs the idempotent + bridge, applies the Grader's grading-only artifact, invokes the existing + Anthropic-compatible report adapter, and returns one JSON summary. Any + failure is incomplete; this command has no best-effort mode. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [string]$GradingPath = 'grading.json' +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +. (Join-Path $PSScriptRoot 'runner-common.ps1') +. (Join-Path $PSScriptRoot 'manifest-paths.ps1') +. (Join-Path $PSScriptRoot 'orchestration.ps1') +. (Join-Path $PSScriptRoot 'execution-freeze.ps1') +. (Join-Path $PSScriptRoot 'package-integrity.ps1') + +function Invoke-FinalizerCommand { + param( + [Parameter(Mandatory = $true)][string]$ScriptPath, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][string]$Description + ) + + $output = & pwsh -NoProfile -File $ScriptPath @Arguments 2>&1 + if ($LASTEXITCODE -ne 0) { + $detail = [string]::Join(' ', @($output | ForEach-Object { [string]$_ })) + throw "$Description failed: $detail" + } + return @($output) +} + +function Assert-FinalizerState { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest, + [Parameter(Mandatory = $true)][object]$Profile + ) + + $statePath = Join-Path $IterationDirectory 'orchestration-state.json' + if (-not (Test-Path -LiteralPath $statePath -PathType Leaf)) { + throw 'Finalization failed: orchestration-state.json is missing.' + } + $descriptor = Get-PackageRunnerDescriptor -RunnerName $Profile.Runner + $plan = New-EvalOrchestrationPlan -IterationDirectory $IterationDirectory -Manifest $Manifest -Profile $Profile.Profile -Descriptor $descriptor + [void](Assert-OrchestrationPlanContract -Plan $plan) + $state = Read-RunnerJson -Path $statePath + if ([string](Get-JsonProperty -Object $state -Name 'plan_schema' -Default '') -ne [string]$plan.schema -or + [string](Get-JsonProperty -Object $state -Name 'dispatch_owner' -Default '') -ne [string]$plan.dispatch_owner) { + throw 'Finalization failed: orchestration-state.json does not match the manifest execution plan.' + } + $activeWorkers = Get-JsonProperty -Object $state -Name 'active' -Default ([ordered]@{}) + if (@($state.pending_worker_ids).Count -ne 0 -or @(Get-JsonPropertyNames -Object $activeWorkers).Count -ne 0) { + throw 'Finalization failed: orchestration still has pending or active workers.' + } + $completed = Get-JsonProperty -Object $state -Name 'completed' -Default ([ordered]@{}) + if (@(Get-JsonPropertyNames -Object $completed).Count -ne @($plan.arms).Count) { + throw 'Finalization failed: orchestration terminal count does not match the manifest arm count.' + } + foreach ($arm in @($plan.arms)) { + $workerId = [string]$arm.worker_id + if (-not (Test-JsonProperty -Object $completed -Name $workerId)) { throw "Finalization failed: orchestration is missing terminal worker '$workerId'." } + $terminal = Get-JsonProperty -Object $completed -Name $workerId -Default $null + if ([string](Get-JsonProperty -Object $terminal -Name 'worker_id' -Default '') -ne $workerId -or + [int](Get-JsonProperty -Object $terminal -Name 'eval_id' -Default 0) -ne [int]$arm.eval_id -or + [string](Get-JsonProperty -Object $terminal -Name 'configuration' -Default '') -ne [string]$arm.configuration) { + throw "Finalization failed: terminal worker '$workerId' does not match its exact plan identity." + } + if ([string](Get-JsonProperty -Object $terminal -Name 'status' -Default '') -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { + throw "Finalization failed: worker '$workerId' is not terminal." + } + } + $preflight = Get-JsonProperty -Object $state -Name 'preflight' -Default $null + if ([string](Get-JsonProperty -Object $preflight -Name 'status' -Default '') -ne 'passed') { + throw 'Finalization failed: Phase 1 preflight gate did not pass.' + } + return [pscustomobject]@{ Plan = $plan; State = $state; Descriptor = $descriptor; Concurrency = (Assert-OrchestrationConcurrency -Plan $plan -State $state) } +} + +function Assert-FinalizerGrading { + param( + [Parameter(Mandatory = $true)][object[]]$Records, + [Parameter(Mandatory = $true)][string]$IterationDirectory + ) + + $graded = 0 + foreach ($record in $Records) { + $result = Read-RunnerJson -Path $record.ResultPath + if ([string]$result.execution_status -ne 'completed') { throw "Finalization failed: '$($record.ResultRelative)' is not a completed execution." } + $assertions = @(Get-JsonProperty -Object (Read-RunnerJson -Path $record.MetadataPath) -Name 'assertions' -Default @()) + $grading = @(Get-JsonProperty -Object $result -Name 'grading' -Default @()) + if ($grading.Count -ne $assertions.Count) { throw "Finalization failed: '$($record.ResultRelative)' has incomplete grading cardinality." } + foreach ($grade in $grading) { + if ((Get-JsonProperty -Object $grade -Name 'passed' -Default $null) -isnot [bool]) { + throw "Finalization failed: '$($record.ResultRelative)' contains an ungraded assertion." + } + $graded++ + } + } + return $graded +} + +try { + $iteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + $manifestPath = Join-Path $iteration 'manifest.json' + if (-not (Test-Path -LiteralPath $manifestPath -PathType Leaf)) { throw 'Finalization failed: manifest.json is missing.' } + $manifest = Read-RunnerJson -Path $manifestPath + [void](Assert-PackageRunnerToolsIntegrity -IterationDirectory $iteration -Manifest $manifest) + $declaredGradingPath = [string](Get-JsonProperty -Object $manifest -Name 'grading' -Default '') + if ([string]::IsNullOrWhiteSpace($declaredGradingPath) -or $declaredGradingPath -ne $GradingPath) { + throw "Finalization failed: grading path '$GradingPath' does not match manifest.grading '$declaredGradingPath'." + } + $profile = Resolve-ExecutionProfile -ProfilePath (Join-Path $iteration 'execution-profile.json') + $stateValidation = Assert-FinalizerState -IterationDirectory $iteration -Manifest $manifest -Profile $profile + $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState + $records = @(Get-ManifestRunRecords -IterationDirectory $iteration -Manifest $manifest) + + $manifestBridge = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath ([string]$manifest.runner_tools + '/bridge-manifest-results.ps1') -FieldName 'manifest bridge' -Kind File -RequireExists + $bridgeArgs = @('-IterationDirectory', $iteration, '-RequireComplete', '-RequireParallelDispatch') + if ([string]$stateValidation.Plan.dispatch_owner -eq 'runner') { + $bridgeArgs += '-RequireNativeDelegation' + } + [void](Invoke-FinalizerCommand -ScriptPath $manifestBridge -Arguments $bridgeArgs -Description 'Manifest bridge') + [void](Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState) + + $gradingScript = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath ([string]$manifest.runner_tools + '/apply-eval-grading.ps1') -FieldName 'grading application helper' -Kind File -RequireExists + [void](Invoke-FinalizerCommand -ScriptPath $gradingScript -Arguments @('-IterationDirectory', $iteration, '-GradingPath', $GradingPath) -Description 'Grading application') + $gradedAssertions = Assert-FinalizerGrading -Records $records -IterationDirectory $iteration + + $reportRelative = [string](Get-JsonProperty -Object $manifest.report -Name 'tool' -Default 'tools/generate-eval-report.ps1') + $reportScript = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $reportRelative -FieldName 'report adapter' -Kind File -RequireExists + [void](Invoke-FinalizerCommand -ScriptPath $reportScript -Arguments @('-IterationDirectory', $iteration, '-RequireComplete') -Description 'Report generation') + + $artifactRelatives = @('report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md') + $artifacts = [System.Collections.Generic.List[object]]::new() + foreach ($relative in $artifactRelatives) { + $path = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $relative -FieldName "final artifact '$relative'" -Kind File -RequireExists + if ((Get-Item -LiteralPath $path).Length -le 0) { throw "Finalization failed: mandatory artifact '$relative' is empty." } + $artifacts.Add([ordered]@{ path = $path; bytes = [int64](Get-Item -LiteralPath $path).Length }) + } + + Write-RunnerJson -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-finalization/1' + status = 'completed' + iteration = $iteration + runner = $profile.Runner + model = $profile.Model + expected_arms = $records.Count + completed_arms = $records.Count + graded_assertions = $gradedAssertions + execution_freeze = $freezeValidation.Path + concurrency = $stateValidation.Concurrency + artifacts = @($artifacts.ToArray()) + }) -AsOutput +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/freebuff-readiness.md b/scripts/eval-runners/freebuff-readiness.md new file mode 100644 index 0000000..e836068 --- /dev/null +++ b/scripts/eval-runners/freebuff-readiness.md @@ -0,0 +1,34 @@ +# Freebuff runner readiness + +Status: planned and blocked. No Freebuff Eval Runner is shipped or advertised +until the official Freebuff CLI provides a supported noninteractive transport. + +The current upstream [Freebuff README](https://github.com/CodebuffAI/freebuff/blob/main/README.md) +documents the `freebuff` TUI. The current upstream +[headless CLI request](https://github.com/CodebuffAI/freebuff/issues/947) asks +for a print/headless mode with machine-readable output; it is evidence that +the required transport is not currently part of the supported CLI contract. +The locally installed CLI was also inspected without a model request: +`freebuff --version` reported `0.0.150`, and its help exposed `login`, +`--continue`, `--cwd`, and `--version`, but no prompt argument, print mode, +JSON/NDJSON output mode, or supported noninteractive session protocol. + +The missing capability is therefore the complete one-prompt-in, +machine-readable-result-out, fresh-session transport. TUI keystroke +automation, screen scraping, PTY emulation, private transport use, and the +paid Codebuff SDK are not substitutes for that capability. + +Freebuff becomes ready only when the supported official CLI can demonstrate all +of the following without a human TTY: + +1. Accept exactly one task/prompt deterministically. +2. Start a fresh independent session without resume or reuse. +3. Select or identify the requested model and configuration. +4. Return the complete final result and any available structured events. +5. Expose a timeout and an enforceable run-local HOME/config/workspace setup. +6. Permit the common Eval Runner contract to keep grading material and paired + arm data out of the worker. + +At that point a runner may be added under `freebuff/runner.ps1` using the +unchanged `describe`, `preflight`, and `execute` protocol. Until then, the +absence of a runner is intentional. diff --git a/scripts/eval-runners/freeze-execution-evidence.ps1 b/scripts/eval-runners/freeze-execution-evidence.ps1 new file mode 100644 index 0000000..8915e53 --- /dev/null +++ b/scripts/eval-runners/freeze-execution-evidence.ps1 @@ -0,0 +1,87 @@ +<#! +.SYNOPSIS + Closes Phase 1 by writing the package execution-freeze.json ledger. + +.DESCRIPTION + This is the shared deterministic boundary for orchestrator-owned packages. + Runner-owned fan-out calls the same library directly after its queue is + terminal. It never overwrites an existing freeze and never repairs raw + evidence. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [string]$OrchestrationStatePath = 'orchestration-state.json' +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +. (Join-Path $PSScriptRoot 'runner-common.ps1') +. (Join-Path $PSScriptRoot 'manifest-paths.ps1') +. (Join-Path $PSScriptRoot 'orchestration.ps1') +. (Join-Path $PSScriptRoot 'execution-freeze.ps1') + +function Save-FreezeOrchestrationState { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$State) + + [System.IO.File]::WriteAllText($Path, (($State | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +try { + $iteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + Assert-SafeRelativePath -RelativePath $OrchestrationStatePath -FieldName 'orchestration state path' + $statePath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $OrchestrationStatePath -FieldName 'orchestration state' -Kind File -RequireExists + $manifest = Read-RunnerJson -Path (Join-Path $iteration 'manifest.json') + $freezeRelativePath = [string](Get-JsonProperty -Object $manifest -Name 'execution_freeze' -Default '') + if ([string]::IsNullOrWhiteSpace($freezeRelativePath)) { throw 'manifest.json must declare execution_freeze.' } + $freezePath = Get-ExecutionFreezePath -IterationDirectory $iteration -RelativePath $freezeRelativePath + if (Test-Path -LiteralPath $freezePath) { + throw "Execution integrity failure: execution-freeze.json already exists at '$freezePath'; refusing to re-freeze raw evidence. Requires fresh Phase 1 execution." + } + $profile = Resolve-ExecutionProfile -ProfilePath (Join-Path $iteration 'execution-profile.json') + $descriptor = Get-PackageRunnerDescriptor -RunnerName $profile.Runner + $plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile.Profile -Descriptor $descriptor + [void](Assert-OrchestrationPlanContract -Plan $plan) + $state = Read-RunnerJson -Path $statePath + if ([string](Get-JsonProperty -Object $state -Name 'schema' -Default '') -ne 'codebeltnet/agentic/eval-orchestration-state/1') { + throw 'Execution freeze requires a valid orchestration-state.json.' + } + $activeWorkers = Get-JsonProperty -Object $state -Name 'active' -Default ([ordered]@{}) + if (@($state.pending_worker_ids).Count -ne 0 -or @(Get-JsonPropertyNames -Object $activeWorkers).Count -ne 0) { + throw 'Execution freeze requires every orchestration worker to be terminal.' + } + $records = @(Get-ManifestRunRecords -IterationDirectory $iteration -Manifest $manifest) + $completed = Get-JsonProperty -Object $state -Name 'completed' -Default ([ordered]@{}) + if (@(Get-JsonPropertyNames -Object $completed).Count -ne $records.Count) { + throw 'Execution freeze requires one terminal orchestration record per manifest arm.' + } + foreach ($record in $records) { + $workerId = Get-FreezeArmKey -Record $record + if (-not (Test-JsonProperty -Object $completed -Name $workerId)) { throw "Execution freeze is missing terminal worker '$workerId'." } + $terminal = Get-JsonProperty -Object $completed -Name $workerId -Default $null + if ([string](Get-JsonProperty -Object $terminal -Name 'status' -Default '') -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { + throw "Execution freeze cannot close non-terminal worker '$workerId'." + } + } + $concurrency = Assert-OrchestrationConcurrency -Plan $plan -State $state + $freeze = New-ExecutionFreezeDocument -IterationDirectory $iteration -Manifest $manifest -Records $records -Profile $profile + $freezePath = Write-ExecutionFreezeDocument -IterationDirectory $iteration -Freeze $freeze -RelativePath $freezeRelativePath + $state.execution_freeze = [ordered]@{ + schema = (Get-RunnerSchemaNames).ExecutionFreeze + path = [System.IO.Path]::GetRelativePath($iteration, $freezePath).Replace('\', '/') + sha256 = Get-Sha256HexFromFile -Path $freezePath + } + Save-FreezeOrchestrationState -Path $statePath -State $state + Write-RunnerJson -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-execution-freeze-summary/1' + status = 'frozen' + execution_freeze = $freezePath + execution_freeze_sha256 = [string]$state.execution_freeze.sha256 + executions = $records.Count + concurrency = $concurrency + }) -AsOutput +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 new file mode 100644 index 0000000..d63704e --- /dev/null +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -0,0 +1,1272 @@ +<#! +.SYNOPSIS + GitHub Copilot CLI Eval Runner adapter. + +.DESCRIPTION + This is the only place where GitHub Copilot CLI flags, COPILOT_HOME + handling, non-interactive JSONL event parsing, Copilot authentication, + and Copilot isolation limitations are defined. It implements the unchanged + describe/preflight/execute process contract shared by every runner. + + Copilot with claude-haiku-4.5 is the Codebelt reference evaluation + configuration. The reference is a repository convention for economical, + stable comparison; it is not an Anthropic default. The model stays fully + configurable through execution-profile.json, so any Copilot-served model can + be selected. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +# GitHub Copilot checks these token variables before its OS credential store and +# GitHub CLI fallback. The values are forwarded only to the Copilot process; +# --secret-env-vars removes them from shell and MCP child environments. +$copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'github-copilot' + version = '0.9.1' + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = 'unavailable' } + capabilities = [ordered]@{ + single_turn = 'supported' + scripted_multi_turn_same_session = 'conditional' + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'conditional' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'conditional' + cache_token_telemetry = 'conditional' + tool_call_telemetry = 'conditional' + command_evidence = 'conditional' + file_evidence = 'conditional' + cost_telemetry = 'unsupported' + credential_child_filtering = 'supported' + native_skill_activation_evidence = 'unsupported' + # Behavioral evaluation transport is runner-owned: the runner starts one + # fresh Copilot CLI session per eval execution and captures the session's + # own terminal evidence. Copilot's native task/general-purpose subagent + # remains an advertised harness capability but is NOT the benchmark + # transport. These controls stay conditional because the runner attests + # the session it locked; the captured terminal evidence proves them. + native_worker_delegation = 'conditional' + delegated_worker_full_capability = 'conditional' + delegated_worker_model_lock = 'conditional' + delegated_worker_working_directory = 'conditional' + delegated_worker_result_capture = 'conditional' + delegated_worker_capacity_signal = 'conditional' + } + delegation = [ordered]@{ + dispatch_owner = 'runner' + mode = 'native_worker' + mechanism = 'Runner-owned GitHub Copilot CLI session (copilot --output-format json): the runner starts one fresh process, captures its exact session identity, and uses only an installed-help-proven explicit session continuation for scripted turns' + worker_role = 'primary-session' + full_capability = 'conditional' + model_lock = 'conditional' + working_directory = 'conditional' + result_capture = 'conditional' + capacity = 'harness_authoritative' + nested_model_execution = $false + } + supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Resolve-CopilotInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + return [pscustomobject]@{ + Run = Resolve-RunContract -RunPath $Run + Profile = Resolve-ExecutionProfile -ProfilePath $Profile + } +} + +function Get-CopilotTokenVariable { + foreach ($name in $copilotAuthVariables) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { + return $name + } + } + return $null +} + +function Get-CopilotGhConfigDirectory { + # GH_CONFIG_DIR is an authentication-state exception to the isolated + # Copilot configuration roots. Resolve it from GitHub CLI's documented + # precedence without reading or logging any credential file. + $configured = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') + if ([string]::IsNullOrWhiteSpace($configured)) { + $xdgConfig = [Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME') + if (-not [string]::IsNullOrWhiteSpace($xdgConfig)) { + $configured = Join-Path $xdgConfig 'gh' + } elseif ((Get-PlatformName) -eq 'windows') { + $applicationData = [Environment]::GetFolderPath([Environment+SpecialFolder]::ApplicationData) + if (-not [string]::IsNullOrWhiteSpace($applicationData)) { + $configured = Join-Path $applicationData 'GitHub CLI' + } + } else { + $userHome = [Environment]::GetEnvironmentVariable('HOME') + if ([string]::IsNullOrWhiteSpace($userHome)) { + $userHome = [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile) + } + if (-not [string]::IsNullOrWhiteSpace($userHome)) { + $configured = Join-Path (Join-Path $userHome '.config') 'gh' + } + } + } + + if ([string]::IsNullOrWhiteSpace($configured) -or -not (Test-Path -LiteralPath $configured -PathType Container)) { + return $null + } + return [System.IO.Path]::GetFullPath($configured) +} + +function Get-CopilotGitHubCliToken { + $gh = Resolve-ExternalCommand -Name 'gh' + if ($null -eq $gh) { + return $null + } + + $environment = New-RunnerProbeEnvironment + foreach ($name in @('HOME', 'USERPROFILE', 'APPDATA', 'LOCALAPPDATA', 'XDG_CONFIG_HOME', 'GH_CONFIG_DIR')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + + $probeDirectory = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $probeDirectory -Force | Out-Null + try { + $process = Invoke-RunnerProcess -FileName $gh.FileName -ArgumentList (@($gh.Prefix) + @('auth', 'token')) -WorkingDirectory $probeDirectory -Environment $environment -TimeoutSeconds 30 + if ($process.TimedOut -or $process.ExitCode -ne 0) { + return $null + } + $token = ([string]$process.Stdout).Trim() + if ([string]::IsNullOrWhiteSpace($token)) { + return $null + } + return $token + } finally { + if (Test-Path -LiteralPath $probeDirectory) { + Remove-Item -LiteralPath $probeDirectory -Recurse -Force -ErrorAction SilentlyContinue + } + } +} + +function Resolve-CopilotAuthentication { + $tokenVariable = Get-CopilotTokenVariable + if (-not [string]::IsNullOrWhiteSpace($tokenVariable)) { + return [pscustomobject]@{ + Source = 'environment' + TokenVariable = $tokenVariable + TokenValue = $null + GitHubCliTokenResolved = $false + } + } + + $githubCliToken = Get-CopilotGitHubCliToken + if (-not [string]::IsNullOrWhiteSpace($githubCliToken)) { + return [pscustomobject]@{ + Source = 'github_cli_token' + TokenVariable = 'GH_TOKEN' + TokenValue = $githubCliToken + GitHubCliTokenResolved = $true + } + } + + return [pscustomobject]@{ + Source = 'copilot_os_keychain_or_github_cli_unverified' + TokenVariable = $null + TokenValue = $null + GitHubCliTokenResolved = $false + } +} + +function Invoke-CopilotCli { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][object]$Inputs, + [System.Collections.IDictionary]$Environment, + [byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 60 + ) + + $allArguments = @($CommandInfo.Prefix) + @($Arguments) + return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList $allArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function Get-CopilotHelpResult { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs, + [string[]]$Arguments = @('--help') + ) + + $environment = New-RunnerEnvironment -Run $Inputs.Run + return Invoke-CopilotCli -CommandInfo $CommandInfo -Arguments $Arguments -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 +} + +function Remove-CopilotAnsiSequences { + param([AllowEmptyString()][string]$Text) + + return [regex]::Replace($Text, "`e\[[0-?]*[ -/]*[@-~]", '') +} + +function Get-CopilotContinuationCapability { + param([Parameter(Mandatory = $true)][AllowEmptyString()][string]$HelpText) + + $cleanText = Remove-CopilotAnsiSequences -Text $HelpText + $lines = @($cleanText -split "`r?`n") + foreach ($flag in @('--resume', '--session-id')) { + $flagPattern = [regex]::Escape($flag) + for ($lineIndex = 0; $lineIndex -lt $lines.Count; $lineIndex++) { + $line = [string]$lines[$lineIndex] + $match = [regex]::Match($line, "(?=|\s+)(?<[^>\r\n]+>|\[[^\]\r\n]+\]|\{[^\}\r\n]+\})", [Text.RegularExpressions.RegexOptions]::IgnoreCase) + if (-not $match.Success) { continue } + + $contextStart = [Math]::Max(0, $lineIndex - 1) + $contextEnd = [Math]::Min($lines.Count - 1, $lineIndex + 2) + $contextLines = [System.Collections.Generic.List[string]]::new() + foreach ($contextLine in @($lines[$contextStart..$contextEnd])) { + if ($contextLine -match '(? continuation selected from installed help" -f $continuationCapability.Flag)) + $mechanisms.Add('no implicit last-session continuation') + } else { + $mechanisms.Add('no session continuation') + } + if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + $document = New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) + $document.protocol_observations = [ordered]@{ + scripted_multi_turn_same_session = [ordered]@{ + available = [bool]$continuationCapability.Available + flag = $continuationCapability.Flag + argument_style = $continuationCapability.ArgumentStyle + parameter = $continuationCapability.Parameter + help_evidence = $continuationCapability.HelpEvidence + reason = $continuationCapability.Reason + structured_output = 'copilot --output-format json' + session_identity_source = 'runtime structured session event' + exact_session_required = $true + implicit_continuation = $false + } + } + return $document +} + +function New-CopilotEnvironment { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $copilotHome = Join-Path $Inputs.Run.HomeDirectoryPath '.copilot' + $copilotCacheHome = Join-Path $Inputs.Run.HomeDirectoryPath '.copilot-cache' + New-Item -ItemType Directory -Path $copilotHome -Force | Out-Null + New-Item -ItemType Directory -Path $copilotCacheHome -Force | Out-Null + $additional = @{ + COPILOT_HOME = $copilotHome + COPILOT_CACHE_HOME = $copilotCacheHome + COPILOT_AUTO_UPDATE = 'false' + } + $tokenVariable = Get-CopilotTokenVariable + $authState = Resolve-CopilotAuthentication + if ($authState.Source -eq 'github_cli_token') { + $additional[$authState.TokenVariable] = $authState.TokenValue + } + return New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables $copilotAuthVariables -Additional $additional +} + +function New-CopilotInsideEnvironment { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + $insideEnvironment = [ordered]@{ + HOME = '/run/home' + USERPROFILE = '/run/home' + XDG_CONFIG_HOME = '/run/home/.config' + XDG_DATA_HOME = '/run/home/.local/share' + XDG_CACHE_HOME = '/run/home/.cache' + TEMP = '/run/home/tmp' + TMP = '/run/home/tmp' + COPILOT_HOME = '/run/home/.copilot' + COPILOT_CACHE_HOME = '/run/home/.copilot-cache' + COPILOT_AUTO_UPDATE = 'false' + PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + CI = '1' + NO_COLOR = '1' + } + foreach ($authName in $copilotAuthVariables) { + if ($Environment.Contains($authName) -and -not [string]::IsNullOrWhiteSpace([string]$Environment[$authName])) { + $insideEnvironment[$authName] = [string]$Environment[$authName] + } + } + return $insideEnvironment +} + +function Write-CopilotCapture { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text + ) + + $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $path) -Force | Out-Null + [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) + return New-ArtifactReference -Run $RunData.Run -Path $RelativePath -Scope run -MediaType (Get-MediaType -Path $RelativePath) +} + +function Add-NullableInt64 { + param([object]$Current, [object]$Value) + + if ($null -eq $Value) { return $Current } + if ($null -eq $Current) { return [int64]$Value } + return ([int64]$Current + [int64]$Value) +} + +function Read-CopilotEvents { + param( + [Parameter(Mandatory = $true)][object]$Parsed, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[string]]$Warnings + ) + + $assistantContents = [System.Collections.Generic.List[string]]::new() + $finalText = $null + $observedModel = $null + $usageInput = $null + $usageOutput = $null + $usageCacheRead = $null + $usageCacheWrite = $null + $usageNumToolCalls = 0 + $usageSeen = $false + $toolStarts = 0 + $sessionError = $null + $eventCounts = @{} + $sessionIds = [System.Collections.Generic.List[string]]::new() + $observedModels = [System.Collections.Generic.List[string]]::new() + $terminalEventObserved = $false + $assistantMessageObserved = $false + $eventTimestamps = [System.Collections.Generic.List[string]]::new() + + foreach ($event in @($Parsed.Events)) { + $eventType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ([string]::IsNullOrWhiteSpace($eventType)) { + $Warnings.Add('Copilot emitted an event without a type; it was ignored.') + continue + } + if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } + $data = Get-JsonProperty -Object $event -Name 'data' -Default $null + foreach ($eventSessionId in @(Get-CopilotEventSessionIds -Event $event)) { + if ($sessionIds -notcontains $eventSessionId) { $sessionIds.Add($eventSessionId) } + } + foreach ($timestamp in @( + (Get-JsonProperty -Object $event -Name 'timestamp' -Default $null), + (Get-JsonProperty -Object $event -Name 'timestamp_utc' -Default $null) + )) { + if (-not [string]::IsNullOrWhiteSpace([string]$timestamp) -and $eventTimestamps -notcontains [string]$timestamp) { $eventTimestamps.Add([string]$timestamp) } + } + switch ($eventType) { + 'assistant.message' { + $assistantMessageObserved = $true + $content = [string](Get-JsonProperty -Object $data -Name 'content' -Default '') + if (-not [string]::IsNullOrWhiteSpace($content)) { + $assistantContents.Add($content) + $finalText = $content + } + $model = [string](Get-JsonProperty -Object $data -Name 'model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($model)) { + $observedModel = $model + if ($observedModels -notcontains $model) { $observedModels.Add($model) } + } + } + 'assistant.usage' { + $usageSeen = $true + $usageInput = Add-NullableInt64 -Current $usageInput -Value (Get-JsonProperty -Object $data -Name 'inputTokens' -Default $null) + $usageOutput = Add-NullableInt64 -Current $usageOutput -Value (Get-JsonProperty -Object $data -Name 'outputTokens' -Default $null) + $usageCacheRead = Add-NullableInt64 -Current $usageCacheRead -Value (Get-JsonProperty -Object $data -Name 'cacheReadTokens' -Default $null) + $usageCacheWrite = Add-NullableInt64 -Current $usageCacheWrite -Value (Get-JsonProperty -Object $data -Name 'cacheWriteTokens' -Default $null) + $numToolCalls = Get-JsonProperty -Object $data -Name 'numToolCalls' -Default $null + if ($null -ne $numToolCalls) { $usageNumToolCalls += [int]$numToolCalls } + $model = [string](Get-JsonProperty -Object $data -Name 'model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($model)) { + $observedModel = $model + if ($observedModels -notcontains $model) { $observedModels.Add($model) } + } + } + 'tool.execution_start' { $toolStarts++ } + 'session.error' { $sessionError = [string](Get-JsonProperty -Object $data -Name 'message' -Default 'Copilot reported a session error.') } + { $_ -in @('session.start', 'session.info', 'session.idle', 'session.shutdown', 'session.task_complete', 'user.message', 'assistant.message_start', 'assistant.message_delta', 'assistant.turn_start', 'assistant.turn_end', 'assistant.reasoning', 'assistant.tool_call_delta', 'tool.execution_progress', 'tool.execution_partial_result', 'tool.execution_complete', 'command.execute', 'command.completed', 'session.usage_info', 'session.usage_checkpoint') } { } + default { $Warnings.Add("Unknown Copilot event '$eventType' was preserved as a warning.") } + } + if ($eventType -in @('session.task_complete', 'session.idle', 'session.shutdown', 'assistant.turn_end')) { $terminalEventObserved = $true } + } + + if ([string]::IsNullOrWhiteSpace($finalText) -and $assistantContents.Count -gt 0) { + $finalText = [string]::Join("`n", $assistantContents) + } + $toolCalls = if ($toolStarts -gt 0) { $toolStarts } else { $usageNumToolCalls } + + return [pscustomobject]@{ + FinalText = $finalText + ObservedModel = $observedModel + UsageSeen = $usageSeen + UsageInput = $usageInput + UsageOutput = $usageOutput + UsageCacheRead = $usageCacheRead + UsageCacheWrite = $usageCacheWrite + ToolCalls = $toolCalls + SessionError = $sessionError + EventCounts = $eventCounts + SessionIds = @($sessionIds.ToArray()) + ObservedModels = @($observedModels.ToArray()) + TerminalEventObserved = $terminalEventObserved + AssistantMessageObserved = $assistantMessageObserved + EventTimestamps = @($eventTimestamps.ToArray()) + StructuredEventCount = @($Parsed.Events).Count + ParseErrorCount = @($Parsed.Errors).Count + } +} + +function Invoke-CopilotTurnProcess { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment, + [Parameter(Mandatory = $true)][string]$Platform, + [object]$SandboxInfo, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][byte[]]$InputBytes, + [Parameter(Mandatory = $true)][int]$TimeoutSeconds + ) + + $hardFilesystem = $null -ne $SandboxInfo -and $Platform -in @('linux', 'macos') + if ($Platform -eq 'linux' -and $hardFilesystem) { + $insideEnvironment = New-CopilotInsideEnvironment -Inputs $Inputs -Environment $Environment + $sandboxArguments = Get-LinuxEvalSandboxArguments -Inputs $Inputs -CommandInfo $CommandInfo -InsideEnvironment $insideEnvironment + return Invoke-RunnerProcess -FileName $SandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($Arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds + } + if ($Platform -eq 'macos' -and $hardFilesystem) { + $sandboxProfile = New-MacosEvalSandboxProfile -Inputs $Inputs -CommandInfo $CommandInfo + $sandboxArguments = @('-f', $sandboxProfile, '--', $CommandInfo.FileName) + @($CommandInfo.Prefix) + @($Arguments) + return Invoke-RunnerProcess -FileName $SandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds + } + return Invoke-CopilotCli -CommandInfo $CommandInfo -Arguments $Arguments -Inputs $Inputs -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function Invoke-CopilotScriptedExecute { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$Preflight, + [Parameter(Mandatory = $true)][object]$ExecutionDescriptor + ) + + $started = [DateTime]::UtcNow + $commandInfo = Resolve-ExternalCommand -Name 'copilot' + $environment = New-CopilotEnvironment -Inputs $Inputs + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } + $protocol = Get-JsonProperty -Object $Preflight -Name 'protocol_observations' -Default $null + $continuationObservation = Get-JsonProperty -Object $protocol -Name 'scripted_multi_turn_same_session' -Default $null + $continuationCapability = [pscustomobject]@{ + Available = [bool](Get-JsonProperty -Object $continuationObservation -Name 'available' -Default $false) + Flag = Get-JsonProperty -Object $continuationObservation -Name 'flag' -Default $null + ArgumentStyle = Get-JsonProperty -Object $continuationObservation -Name 'argument_style' -Default $null + Parameter = Get-JsonProperty -Object $continuationObservation -Name 'parameter' -Default $null + } + if ($null -eq $commandInfo -or -not [bool]$continuationCapability.Available) { + $finished = [DateTime]::UtcNow + $fallbackSessionId = [Guid]::NewGuid().ToString('D') + $failureMessage = [string](Get-JsonProperty -Object $continuationObservation -Name 'reason' -Default 'Copilot exact-session continuation was not proven by installed help.') + return New-ExecutionResult -Descriptor $ExecutionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message $failureMessage) -SessionId $fallbackSessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $Preflight; resume = $false }) -AttemptCount 1 + } + + $requestedTurns = @($Inputs.Run.Interaction.turns) + $baseArguments = New-CopilotCliArguments -Inputs $Inputs -VisiblePlatform $visiblePlatform + $turnRecords = [System.Collections.Generic.List[object]]::new() + $nativeTurns = [System.Collections.Generic.List[object]]::new() + $rawStdout = [System.Collections.Generic.List[string]]::new() + $rawStderr = [System.Collections.Generic.List[string]]::new() + $artifacts = [System.Collections.Generic.List[object]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $nativeFailures = [System.Collections.Generic.List[string]]::new() + $eventCounts = @{} + $observedModels = [System.Collections.Generic.List[string]]::new() + $usageInput = $null + $usageOutput = $null + $usageCacheRead = $null + $usageCacheWrite = $null + $usageSeen = $false + $toolCalls = 0 + $capturedSessionId = $null + $finalText = $null + $firstProcess = $null + $lastProcess = $null + $status = 'completed' + $failureCode = $null + $failureMessage = $null + + for ($turnIndex = 0; $turnIndex -lt $requestedTurns.Count; $turnIndex++) { + $turnText = Get-InteractionTurnText -Turn $requestedTurns[$turnIndex] -RunData $Inputs.Run + $arguments = @($baseArguments) + $targetSessionId = $null + if ($turnIndex -gt 0) { + $targetSessionId = $capturedSessionId + if ([string]::IsNullOrWhiteSpace($targetSessionId)) { + $nativeFailures.Add('session_id_unobservable') + $status = 'incompatible' + $failureCode = 'native_interaction_incompatible' + $failureMessage = 'Copilot turn 1 did not expose an exact session id, so no continuation invocation was started.' + break + } + $arguments = @($arguments) + @(New-CopilotContinuationArguments -Capability $continuationCapability -SessionId $targetSessionId) + } + + $turnStarted = [DateTime]::UtcNow + try { + $turnInputBytes = [System.Text.UTF8Encoding]::new($false).GetBytes($turnText) + $process = Invoke-CopilotTurnProcess -Inputs $Inputs -CommandInfo $commandInfo -Arguments $arguments -Environment $environment -Platform $platform -SandboxInfo $sandboxInfo -InputBytes $turnInputBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } catch { + $nativeFailures.Add('transport_failure') + $status = 'failed' + $failureCode = 'copilot_failure' + $failureMessage = $_.Exception.Message + break + } + if ($null -eq $firstProcess) { $firstProcess = $process } + $lastProcess = $process + $rawStdout.Add([string]$process.Stdout) + $rawStderr.Add([string]$process.Stderr) + $turnNumber = $turnIndex + 1 + $turnArtifact = Write-CopilotCapture -RunData $Inputs -RelativePath ("evidence/copilot-turn-{0}-events.jsonl" -f $turnNumber) -Text ([string]$process.Stdout) + $turnStderrArtifact = Write-CopilotCapture -RunData $Inputs -RelativePath ("evidence/copilot-turn-{0}-stderr.txt" -f $turnNumber) -Text ([string]$process.Stderr) + $artifacts.Add($turnArtifact) + $artifacts.Add($turnStderrArtifact) + + $parsed = if ([string]::IsNullOrEmpty([string]$process.Stdout)) { [pscustomobject]@{ Events = @(); Errors = @() } } else { ConvertFrom-JsonLines -Text $process.Stdout } + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("Copilot turn $turnNumber event parse error: $parseError") } + $parsedEvents = Read-CopilotEvents -Parsed $parsed -Warnings $warnings + foreach ($eventName in $parsedEvents.EventCounts.Keys) { + if ($eventCounts.ContainsKey($eventName)) { $eventCounts[$eventName] += [int]$parsedEvents.EventCounts[$eventName] } else { $eventCounts[$eventName] = [int]$parsedEvents.EventCounts[$eventName] } + } + if ($parsedEvents.UsageSeen) { $usageSeen = $true } + $usageInput = Add-NullableInt64 -Current $usageInput -Value $parsedEvents.UsageInput + $usageOutput = Add-NullableInt64 -Current $usageOutput -Value $parsedEvents.UsageOutput + $usageCacheRead = Add-NullableInt64 -Current $usageCacheRead -Value $parsedEvents.UsageCacheRead + $usageCacheWrite = Add-NullableInt64 -Current $usageCacheWrite -Value $parsedEvents.UsageCacheWrite + $toolCalls += [int]$parsedEvents.ToolCalls + foreach ($modelName in @($parsedEvents.ObservedModels)) { + if ($observedModels -notcontains $modelName) { $observedModels.Add($modelName) } + } + + $sessionIds = @($parsedEvents.SessionIds) + $observedSessionId = if ($sessionIds.Count -eq 1) { [string]$sessionIds[0] } else { $null } + $nativeTurn = [ordered]@{ + turn = $turnNumber + invocation = if ($turnIndex -eq 0) { 'fresh' } else { 'explicit_session_resume' } + arguments = @($arguments) + requested_model = [string]$Inputs.Profile.Model + observed_models = @($parsedEvents.ObservedModels) + model_source = if (@($parsedEvents.ObservedModels).Count -gt 0) { 'structured_event' } else { 'cli_argument' } + session_ids_observed = @($sessionIds) + session_id = $observedSessionId + target_session_id = $targetSessionId + target_session_match = if ($turnIndex -eq 0) { $null } else { $observedSessionId -eq $targetSessionId } + terminal_assistant_response = -not [string]::IsNullOrWhiteSpace([string]$parsedEvents.FinalText) + terminal_event_observed = [bool]$parsedEvents.TerminalEventObserved + structured_event_count = [int]$parsedEvents.StructuredEventCount + structured_parse_errors = [int]$parsedEvents.ParseErrorCount + structured_output = 'json' + working_directory = [string]$Inputs.Run.WorkingDirectoryPath + home = [string]$Inputs.Run.HomeDirectoryPath + copilot_home = [string]$environment['COPILOT_HOME'] + started_utc = Format-UtcTimestamp -Value $process.StartedUtc + finished_utc = Format-UtcTimestamp -Value $process.FinishedUtc + event_timestamps = @($parsedEvents.EventTimestamps) + exit_code = $process.ExitCode + timed_out = [bool]$process.TimedOut + terminal = -not $process.TimedOut -and $process.ExitCode -eq 0 + } + $nativeTurns.Add($nativeTurn) + + $turnProblem = $null + if ($process.TimedOut) { $turnProblem = 'turn_timeout'; $status = 'timed_out'; $failureCode = 'timed_out'; $failureMessage = 'Copilot did not finish before timeout_seconds.' } + elseif ($process.ExitCode -ne 0 -or $null -ne $parsedEvents.SessionError) { $turnProblem = 'turn_failed'; $status = 'failed'; $failureCode = 'copilot_failure'; $failureMessage = if ($null -ne $parsedEvents.SessionError) { [string]$parsedEvents.SessionError } else { "Copilot exited with status $($process.ExitCode)." } } + elseif ($parsedEvents.ParseErrorCount -gt 0) { $turnProblem = 'structured_event_parse'; $status = 'incompatible'; $failureCode = 'native_interaction_incompatible'; $failureMessage = "Copilot turn $turnNumber did not produce a complete structured event stream." } + elseif ($sessionIds.Count -ne 1) { $turnProblem = 'session_id_unobservable'; $status = 'incompatible'; $failureCode = 'native_interaction_incompatible'; $failureMessage = "Copilot turn $turnNumber did not expose exactly one session id in structured events." } + elseif ($turnIndex -gt 0 -and $observedSessionId -ne $targetSessionId) { $turnProblem = 'session_identity_mismatch'; $status = 'incompatible'; $failureCode = 'native_interaction_incompatible'; $failureMessage = "Copilot turn $turnNumber returned session '$observedSessionId' instead of the exact resumed session '$targetSessionId'." } + elseif (-not $parsedEvents.AssistantMessageObserved -or [string]::IsNullOrWhiteSpace([string]$parsedEvents.FinalText) -or -not [bool]$parsedEvents.TerminalEventObserved) { $turnProblem = 'terminal_turn_status'; $status = 'incompatible'; $failureCode = 'native_interaction_incompatible'; $failureMessage = "Copilot turn $turnNumber did not provide a terminal assistant response and terminal event before continuation." } + elseif (@($parsedEvents.ObservedModels | Where-Object { [string]$_ -ne [string]$Inputs.Profile.Model }).Count -gt 0) { $turnProblem = 'requested_model'; $status = 'incompatible'; $failureCode = 'native_interaction_incompatible'; $failureMessage = "Copilot turn $turnNumber reported a model different from the requested model '$($Inputs.Profile.Model)'." } + + if ($null -ne $turnProblem) { + $nativeFailures.Add($turnProblem) + break + } + if ($turnIndex -eq 0) { $capturedSessionId = $observedSessionId } + $finalText = [string]$parsedEvents.FinalText + $turnRecords.Add([ordered]@{ sequence = ($turnIndex * 2) + 1; role = 'user'; content_sha256 = Get-Sha256HexFromBytes -Bytes ([System.Text.UTF8Encoding]::new($false).GetBytes($turnText)); session_id = $capturedSessionId; timestamp_utc = Format-UtcTimestamp -Value $process.StartedUtc }) + $turnRecords.Add([ordered]@{ sequence = ($turnIndex * 2) + 2; role = 'assistant'; text = $finalText; session_id = $capturedSessionId; timestamp_utc = Format-UtcTimestamp -Value $process.FinishedUtc }) + } + + if ($null -eq $firstProcess) { + $firstProcess = [pscustomobject]@{ StartedUtc = $started; FinishedUtc = [DateTime]::UtcNow; DurationSeconds = ([DateTime]::UtcNow - $started).TotalSeconds; ExitCode = $null; TimedOut = $false } + } + if ($null -eq $lastProcess) { $lastProcess = $firstProcess } + $combinedStdout = [string]::Join('', @($rawStdout.ToArray())) + $combinedStderr = [string]::Join('', @($rawStderr.ToArray())) + if (-not [string]::IsNullOrEmpty($combinedStdout) -and -not $combinedStdout.EndsWith("`n", [StringComparison]::Ordinal)) { $combinedStdout += [Environment]::NewLine } + if (-not [string]::IsNullOrEmpty($combinedStderr) -and -not $combinedStderr.EndsWith("`n", [StringComparison]::Ordinal)) { $combinedStderr += [Environment]::NewLine } + $stdoutArtifact = Write-CopilotCapture -RunData $Inputs -RelativePath 'evidence/copilot-events.jsonl' -Text $combinedStdout + $stderrArtifact = Write-CopilotCapture -RunData $Inputs -RelativePath 'evidence/copilot-stderr.txt' -Text $combinedStderr + $artifacts.Add($stdoutArtifact) + $artifacts.Add($stderrArtifact) + + if ($nativeFailures.Count -eq 0 -and $turnRecords.Count -ne ($requestedTurns.Count * 2)) { + $nativeFailures.Add('turn_order') + $status = 'incompatible' + $failureCode = 'native_interaction_incompatible' + $failureMessage = 'Copilot scripted interaction did not complete every ordered user/assistant turn.' + } + if ($status -eq 'completed' -and $nativeFailures.Count -gt 0) { $status = 'incompatible' } + if ([string]::IsNullOrWhiteSpace($capturedSessionId)) { $capturedSessionId = [Guid]::NewGuid().ToString('D') } + $finished = $lastProcess.FinishedUtc + $durationSeconds = [Math]::Round(($finished - $firstProcess.StartedUtc).TotalSeconds, 3) + $tokenMetric = if (-not $usageSeen) { + New-UnavailableMetric -Reason 'copilot_did_not_expose_usage_events' + } else { + $usageValue = [ordered]@{} + if ($null -ne $usageInput) { $usageValue['input_tokens'] = [int64]$usageInput } + if ($null -ne $usageOutput) { $usageValue['output_tokens'] = [int64]$usageOutput } + if ($null -ne $usageCacheRead) { $usageValue['cache_read_tokens'] = [int64]$usageCacheRead } + if ($null -ne $usageCacheWrite) { $usageValue['cache_write_tokens'] = [int64]$usageCacheWrite } + if ($usageValue.Count -eq 0) { New-UnavailableMetric -Reason 'copilot_usage_event_had_no_supported_buckets' } else { New-AvailableMetric -Value $usageValue } + } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/copilot-events.jsonl'; complete = $nativeFailures.Count -eq 0 }) + tokens = $tokenMetric + tool_calls = New-AvailableMetric -Value $toolCalls + cost = New-UnavailableMetric -Reason 'copilot_exposes_a_billing_multiplier_not_a_currency_cost' + } + $authState = Resolve-CopilotAuthentication + $credentialEvidence = [ordered]@{ + source = $authState.Source + github_token_variable = $authState.TokenVariable + secret_env_vars = @($copilotAuthVariables) + secret_env_var_scope = @('shell', 'mcp') + github_cli_token_resolved = [bool]$authState.GitHubCliTokenResolved + github_cli_config_forwarded = $false + login_profile_copied = $false + auth_file_copied = $false + value_observed = $false + } + $observedModel = if ($observedModels.Count -eq 0) { [string]$Inputs.Profile.Model } else { [string]$observedModels[$observedModels.Count - 1] } + $transcriptArtifactPath = 'evidence/copilot-events.jsonl' + $transcriptArtifact = @($artifacts | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'path' -Default '') -eq $transcriptArtifactPath } | Select-Object -First 1) + $terminalCapture = $nativeFailures.Count -eq 0 -and $turnRecords.Count -eq ($requestedTurns.Count * 2) -and $rawStdout.Count -eq $requestedTurns.Count + $executionPaths = [ordered]@{ + logical_working_directory = [string]$Inputs.Run.WorkingDirectoryPath + logical_home_directory = [string]$Inputs.Run.HomeDirectoryPath + physical_working_directory = [string]$Inputs.Run.WorkingDirectoryPath + physical_home_directory = [string]$Inputs.Run.HomeDirectoryPath + physical_run_root = [string]$Inputs.Run.RunRoot + } + $interactionEvidence = [ordered]@{ + schema = (Get-RunnerSchemaNames).Interaction + mode = 'scripted' + same_session = [bool]$terminalCapture + session_id = $capturedSessionId + turns = @($turnRecords.ToArray()) + final_response_sequence = @($turnRecords).Count + transport = 'copilot-cli-explicit-session-continuation' + exact_session_flag = [string]$continuationCapability.Flag + implicit_continuation = $false + native_turns = @($nativeTurns.ToArray()) + structured_transcript_complete = [bool]$terminalCapture + working_directory = [string]$Inputs.Run.WorkingDirectoryPath + isolated_home = [string]$Inputs.Run.HomeDirectoryPath + model = [string]$Inputs.Profile.Model + } + $evidence = [ordered]@{ + execution_paths = $executionPaths + event_counts = $eventCounts + observed_model = $observedModel + observed_models = @($observedModels.ToArray()) + prompt_delivery = 'stdin' + prompt_first_input = $true + resume = $true + exact_session_continuation = [ordered]@{ + flag = [string]$continuationCapability.Flag + argument_style = [string]$continuationCapability.ArgumentStyle + exact_session_id = $capturedSessionId + implicit_last_session = $false + turns_started_after_prior_terminal = $true + } + stdout_exit_codes = @($nativeTurns.ToArray() | ForEach-Object { Get-JsonProperty -Object $_ -Name 'exit_code' -Default $null }) + sandbox = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + credential = $credentialEvidence + interaction = $interactionEvidence + capture = [ordered]@{ + source = 'harness_native_transport' + terminal = [bool]$terminalCapture + worker_authored = $false + artifact = $transcriptArtifactPath + sha256 = if ($transcriptArtifact.Count -eq 1) { [string](Get-JsonProperty -Object $transcriptArtifact[0] -Name 'sha256' -Default $null) } else { $null } + complete_structured_transcript = [bool]$terminalCapture + turn_artifacts = @($nativeTurns.ToArray() | ForEach-Object { "evidence/copilot-turn-$(Get-JsonProperty -Object $_ -Name 'turn' -Default 0)-events.jsonl" }) + } + delegation = [ordered]@{ + dispatch_owner = 'runner' + mechanism = [string]$descriptor.delegation.mechanism + worker_session_id = $capturedSessionId + observed_model = $observedModel + observed_working_directory = [string]$Inputs.Run.WorkingDirectoryPath + observed_home = [string]$Inputs.Run.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$Inputs.Run.PromptHash + terminal_result_capture = [bool]$terminalCapture + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + same_session_continuation = [bool]$terminalCapture + continuation_flag = [string]$continuationCapability.Flag + continuation_session_id = $capturedSessionId + } + native_worker_evidence_failures = @($nativeFailures | Select-Object -Unique) + } + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('runner-owned fresh Copilot CLI process for turn 1', 'copilot --output-format json structured terminal event capture', 'prompt on stdin', '--model on every turn', '-C on every turn', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'same isolated environment on every turn', 'no implicit last-session continuation')) { $mechanisms.Add($mechanism) } + $mechanisms.Add(("explicit Copilot {0} continuation selected from installed help" -f $continuationCapability.Flag)) + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + $exitStatus = if ($status -eq 'completed') { [Nullable[int]]0 } elseif ($status -eq 'timed_out') { $null } else { $null } + $failureCodeValue = if ([string]::IsNullOrWhiteSpace($failureCode)) { 'native_interaction_incompatible' } else { $failureCode } + $failureMessageValue = if ([string]::IsNullOrWhiteSpace($failureMessage)) { 'Copilot scripted interaction failed closed.' } else { $failureMessage } + $failure = if ($nativeFailures.Count -eq 0) { $null } else { New-ExecutionFailure -Code $failureCodeValue -Message $failureMessageValue } + $resultFinalResponse = if ($status -eq 'completed') { $finalText } else { $null } + $resultFinalResponseReason = if ($status -eq 'completed') { $null } else { 'native_interaction_incompatible' } + $result = New-ExecutionResult -Descriptor $ExecutionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $resultFinalResponse -FinalResponseReason $resultFinalResponseReason -StartedUtc $firstProcess.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $durationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $capturedSessionId -IsolationCapabilities (Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem -ContinuationCapability $continuationCapability) -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Copilot accepted the requested model alias and configuration; scripted turns retained the exact requested model on every invocation.'; observations = [ordered]@{ model = $Inputs.Profile.Model; observed_models = @($observedModels.ToArray()); continuation_flag = $continuationCapability.Flag } }) -Telemetry $telemetry -Artifacts @($artifacts.ToArray()) -Warnings @($warnings.ToArray()) -Evidence $evidence -AttemptCount 1 + if ($status -eq 'completed') { [void](Assert-InteractionResultEvidence -ExecutionResult $result -RunData $Inputs.Run) } + return $result +} + +function Invoke-CopilotExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $preflight = Get-CopilotPreflight -Inputs $Inputs + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + $executionDescriptor = [ordered]@{} + foreach ($key in $descriptor.Keys) { $executionDescriptor[$key] = $descriptor[$key] } + $executionDescriptor.harness = $preflight.harness + if ($preflight.status -ne 'compatible') { + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight; resume = $false }) -AttemptCount 1 + } + + if ($null -ne $Inputs.Run.Interaction) { + return Invoke-CopilotScriptedExecute -Inputs $Inputs -Preflight $preflight -ExecutionDescriptor $executionDescriptor + } + + $commandInfo = Resolve-ExternalCommand -Name 'copilot' + $environment = New-CopilotEnvironment -Inputs $Inputs + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } + $arguments = New-CopilotCliArguments -Inputs $Inputs -VisiblePlatform $visiblePlatform + + if ($platform -eq 'linux' -and $hardFilesystem) { + $insideEnvironment = New-CopilotInsideEnvironment -Inputs $Inputs -Environment $environment + $sandboxArguments = Get-LinuxEvalSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -InsideEnvironment $insideEnvironment + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } elseif ($platform -eq 'macos' -and $hardFilesystem) { + $sandboxProfile = New-MacosEvalSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo + $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + @($arguments) + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $process = Invoke-CopilotCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $Inputs -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } + + $stdoutArtifact = Write-CopilotCapture -RunData $Inputs -RelativePath 'evidence/copilot-events.jsonl' -Text $process.Stdout + $stderrArtifact = Write-CopilotCapture -RunData $Inputs -RelativePath 'evidence/copilot-stderr.txt' -Text $process.Stderr + $artifacts = [System.Collections.Generic.List[object]]::new() + $artifacts.Add($stdoutArtifact) + $artifacts.Add($stderrArtifact) + + $warnings = [System.Collections.Generic.List[string]]::new() + $parsed = if ([string]::IsNullOrEmpty([string]$process.Stdout)) { + [pscustomobject]@{ Events = @(); Errors = @() } + } else { + ConvertFrom-JsonLines -Text $process.Stdout + } + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("Copilot event parse error: $parseError") } + $parsedEvents = Read-CopilotEvents -Parsed $parsed -Warnings $warnings + + $finalText = $parsedEvents.FinalText + $status = 'completed' + $reason = $null + $failure = $null + $exitStatus = if ($process.TimedOut) { $null } else { [Nullable[int]]$process.ExitCode } + if ($process.TimedOut) { + $status = 'timed_out' + $reason = 'copilot_timeout' + $failure = New-ExecutionFailure -Code 'timed_out' -Message 'Copilot did not finish before timeout_seconds.' + } elseif ($process.ExitCode -ne 0 -or $null -ne $parsedEvents.SessionError) { + $status = 'failed' + $reason = 'copilot_failure' + $message = if ($null -ne $parsedEvents.SessionError) { [string]$parsedEvents.SessionError } else { "Copilot exited with status $($process.ExitCode)." } + $failure = New-ExecutionFailure -Code 'copilot_failure' -Message $message + } elseif ([string]::IsNullOrWhiteSpace($finalText)) { + $warnings.Add('Copilot exited successfully without an assistant message; the final response is unavailable.') + $reason = 'copilot_did_not_return_final_response' + } + + $tokenMetric = if (-not $parsedEvents.UsageSeen) { + New-UnavailableMetric -Reason 'copilot_did_not_expose_usage_events' + } else { + $usageValue = [ordered]@{} + if ($null -ne $parsedEvents.UsageInput) { $usageValue['input_tokens'] = [int64]$parsedEvents.UsageInput } + if ($null -ne $parsedEvents.UsageOutput) { $usageValue['output_tokens'] = [int64]$parsedEvents.UsageOutput } + if ($null -ne $parsedEvents.UsageCacheRead) { $usageValue['cache_read_tokens'] = [int64]$parsedEvents.UsageCacheRead } + if ($null -ne $parsedEvents.UsageCacheWrite) { $usageValue['cache_write_tokens'] = [int64]$parsedEvents.UsageCacheWrite } + if ($usageValue.Count -eq 0) { New-UnavailableMetric -Reason 'copilot_usage_event_had_no_supported_buckets' } else { New-AvailableMetric -Value $usageValue } + } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/copilot-events.jsonl'; complete = $true }) + tokens = $tokenMetric + tool_calls = New-AvailableMetric -Value ([int]$parsedEvents.ToolCalls) + cost = New-UnavailableMetric -Reason 'copilot_exposes_a_billing_multiplier_not_a_currency_cost' + } + + $capabilities = Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all-tools broad tool approval', 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + + $authState = Resolve-CopilotAuthentication + $credentialEvidence = [ordered]@{ + source = $authState.Source + github_token_variable = $authState.TokenVariable + secret_env_vars = @($copilotAuthVariables) + secret_env_var_scope = @('shell', 'mcp') + github_cli_token_resolved = [bool]$authState.GitHubCliTokenResolved + github_cli_config_forwarded = $false + login_profile_copied = $false + auth_file_copied = $false + value_observed = $false + } + $observedModel = if ([string]::IsNullOrWhiteSpace([string]$parsedEvents.ObservedModel)) { $null } else { [string]$parsedEvents.ObservedModel } + $resolvedConfiguration = [ordered]@{ + status = 'accepted_request' + reason = 'Copilot accepted the requested model alias and configuration; it does not expose a distinct backend model snapshot beyond the model it reports in usage events.' + observations = [ordered]@{ + model = $Inputs.Profile.Model + reasoning_effort = $Inputs.Profile.ReasoningEffort + observed_model = $observedModel + } + } + + $finished = [DateTime]::UtcNow + $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + # Runner-owned terminal evidence for the direct Copilot session. The runner + # controlled the fresh session, its model lock, working directory, isolated + # COPILOT_HOME, and stdin prompt, and captured the session's own JSONL + # transcript. This is transport-owned evidence, never orchestrator-authored. + $transcriptArtifactPath = 'evidence/copilot-events.jsonl' + $transcriptArtifact = @($artifacts | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'path' -Default '') -eq $transcriptArtifactPath } | Select-Object -First 1) + $terminalCapture = (-not $process.TimedOut) -and (-not [string]::IsNullOrWhiteSpace([string]$process.Stdout)) + $delegationObservedModel = if ([string]::IsNullOrWhiteSpace([string]$parsedEvents.ObservedModel)) { [string]$Inputs.Profile.Model } else { [string]$parsedEvents.ObservedModel } + $evidence = [ordered]@{ + event_counts = $parsedEvents.EventCounts + observed_model = $observedModel + prompt_delivery = 'stdin' + prompt_first_input = $true + resume = $false + stdout_exit_code = $process.ExitCode + sandbox = $sandboxEvidence + credential = $credentialEvidence + capture = [ordered]@{ + source = 'harness_native_transport' + terminal = [bool]$terminalCapture + worker_authored = $false + artifact = $transcriptArtifactPath + sha256 = if ($transcriptArtifact.Count -eq 1) { [string](Get-JsonProperty -Object $transcriptArtifact[0] -Name 'sha256' -Default $null) } else { $null } + } + delegation = [ordered]@{ + dispatch_owner = 'runner' + mechanism = [string]$descriptor.delegation.mechanism + worker_session_id = $sessionId + observed_model = $delegationObservedModel + observed_working_directory = [string]$Inputs.Run.WorkingDirectoryPath + observed_home = [string]$Inputs.Run.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$Inputs.Run.PromptHash + terminal_result_capture = [bool]$terminalCapture + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + } + } + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration $resolvedConfiguration -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence $evidence -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { Write-RunnerJson -Value (Get-CopilotDescriptor) -AsOutput } + 'preflight' { + $inputs = Resolve-CopilotInputs + Write-RunnerJson -Value (Get-CopilotPreflight -Inputs $inputs) -AsOutput + } + 'execute' { + $inputs = Resolve-CopilotInputs + [void](Assert-PhaseOneEvidenceWritable -Run $inputs.Run) + $result = Invoke-CopilotExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/invoke-runner-owned-arms.ps1 b/scripts/eval-runners/invoke-runner-owned-arms.ps1 new file mode 100644 index 0000000..93b1ede --- /dev/null +++ b/scripts/eval-runners/invoke-runner-owned-arms.ps1 @@ -0,0 +1,496 @@ +<#! +.SYNOPSIS + Deterministically fans out runner-owned native Eval Worker arms. + +.DESCRIPTION + This helper is the runner-owned Phase 1 external-handoff surface. It is + invoked exactly once by the external Eval Orchestrator, runs in the + foreground, and returns one terminal JSON summary. It does not run during + preparation, validation, CI, hooks, or reporting. It starts the selected + package-local runner once per manifest arm, redirects each runner's sole + JSON stdout directly to the manifest-declared execution_result path, and + owns acceptance/terminal registration and orchestration state. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)][string]$IterationDirectory +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$iteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path +. (Join-Path $PSScriptRoot 'runner-common.ps1') +. (Join-Path $PSScriptRoot 'manifest-paths.ps1') +. (Join-Path $PSScriptRoot 'orchestration.ps1') +. (Join-Path $PSScriptRoot 'fanout-process.ps1') +. (Join-Path $PSScriptRoot 'execution-freeze.ps1') +. (Join-Path $PSScriptRoot 'package-integrity.ps1') + +function Write-FanoutSummary { + param( + [Parameter(Mandatory = $true)][object]$Summary, + [int]$ExitCode = 0 + ) + + [Console]::Out.WriteLine(($Summary | ConvertTo-Json -Depth 100 -Compress)) + if ($ExitCode -ne 0) { exit $ExitCode } +} + +function Save-OrchestrationState { + param( + [Parameter(Mandatory = $true)][string]$Path, + [Parameter(Mandatory = $true)][object]$State + ) + + [System.IO.File]::WriteAllText($Path, (($State | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Invoke-RunnerPreflight { + param( + [Parameter(Mandatory = $true)][string]$RunnerPath, + [Parameter(Mandatory = $true)][string]$RunPath, + [Parameter(Mandatory = $true)][string]$ProfilePath, + [int]$TimeoutSeconds = 120 + ) + + $stdoutPath = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-runner-preflight-' + [Guid]::NewGuid().ToString('N') + '.stdout') + $stderrPath = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-runner-preflight-' + [Guid]::NewGuid().ToString('N') + '.stderr') + $child = $null + try { + $pwshPath = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) + # ProcessStartInfo.ArgumentList escapes each argument natively, so raw + # paths are passed without manual quoting. The child preflight runs + # headless (no visible console window on Windows). + $arguments = @( + '-NoProfile' + '-File' + $RunnerPath + 'preflight' + '-Run' + $RunPath + '-Profile' + $ProfilePath + ) + $child = Start-RunnerChildProcess -FilePath $pwshPath -ArgumentList $arguments -WorkingDirectory (Split-Path -Parent $RunPath) -StdoutPath $stdoutPath -StderrPath $stderrPath -TimeoutSeconds $TimeoutSeconds + $exitCode = Complete-RunnerChildProcess -Child $child + $childTimedOut = [bool]$child.TimedOut + $childTerminationObserved = [bool]$child.TerminationObserved + $child = $null + $stdout = if (Test-Path -LiteralPath $stdoutPath -PathType Leaf) { [System.IO.File]::ReadAllText($stdoutPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + $stderr = if (Test-Path -LiteralPath $stderrPath -PathType Leaf) { [System.IO.File]::ReadAllText($stderrPath, [System.Text.UTF8Encoding]::new($false)) } else { '' } + $result = $null + $parseError = '' + if (-not [string]::IsNullOrWhiteSpace($stdout)) { + try { + $result = $stdout | ConvertFrom-Json -Depth 100 + } catch { + $parseError = $_.Exception.Message + } + } + return [pscustomobject]@{ + Result = $result + ExitCode = $exitCode + Stdout = $stdout + Stderr = $stderr + ParseError = $parseError + TimedOut = $childTimedOut + TerminationObserved = $childTerminationObserved + } + } finally { + if ($null -ne $child) { try { [void](Complete-RunnerChildProcess -Child $child) } catch { } } + foreach ($path in @($stdoutPath, $stderrPath)) { + if (Test-Path -LiteralPath $path -PathType Leaf) { Remove-Item -LiteralPath $path -Force -ErrorAction SilentlyContinue } + } + } +} + +function Get-RunnerGraceSeconds { + # This is internal runner-child watchdog grace. The external controller + # never waits for this model-execution allowance. Process termination and + # stream drains have their own smaller, finite bounds in the helpers. + return 30 +} + +function Get-RunnerPreflightTimeoutSeconds { + # Preflight is a model-free capability probe. Its watchdog is deliberately + # independent of the model execution allowance in execution-profile.json. + return 120 +} + +function Get-RunnerRunTurnCount { + param([Parameter(Mandatory = $true)][string]$RunPath) + + $run = Read-RunnerJson -Path $RunPath + $interactionFile = [string](Get-JsonProperty -Object $run -Name 'interactionFile' -Default '') + if ([string]::IsNullOrWhiteSpace($interactionFile)) { return 1 } + $runRoot = Split-Path -Parent $RunPath + $interactionPath = Resolve-ContainedPath -BasePath $runRoot -RelativePath $interactionFile -FieldName 'interactionFile' -Kind File + $interaction = Read-RunnerJson -Path $interactionPath + $turns = @(Get-JsonProperty -Object $interaction -Name 'turns' -Default @()) + return [Math]::Max(1, $turns.Count) +} + +function Get-RunnerChildTimeoutSeconds { + param( + [Parameter(Mandatory = $true)][string]$RunPath, + [Parameter(Mandatory = $true)][int]$ProfileTimeoutSeconds + ) + + $turnCount = Get-RunnerRunTurnCount -RunPath $RunPath + return [pscustomobject]@{ + TurnCount = $turnCount + RunnerGraceSeconds = Get-RunnerGraceSeconds + TimeoutSeconds = ($turnCount * [Math]::Max(1, $ProfileTimeoutSeconds)) + (Get-RunnerGraceSeconds) + } +} + +function New-PreflightWorkerSummary { + param( + [Parameter(Mandatory = $true)][object]$Record, + [Parameter(Mandatory = $true)][object]$Invocation, + [Parameter(Mandatory = $true)][object]$Descriptor + ) + + $preflight = $Invocation.Result + $invocationExitCode = Get-JsonProperty -Object $Invocation -Name 'ExitCode' -Default $null + $invocationTimedOut = [bool](Get-JsonProperty -Object $Invocation -Name 'TimedOut' -Default $false) + $invocationTerminated = [bool](Get-JsonProperty -Object $Invocation -Name 'TerminationObserved' -Default $false) + $reasons = [System.Collections.Generic.List[string]]::new() + if ($null -ne $preflight) { + foreach ($reason in @(Get-JsonProperty -Object $preflight -Name 'reasons' -Default @())) { + if (-not [string]::IsNullOrWhiteSpace([string]$reason)) { [void]$reasons.Add([string]$reason) } + } + } else { + if (-not [string]::IsNullOrWhiteSpace([string]$Invocation.ParseError)) { [void]$reasons.Add("runner preflight returned invalid JSON: $($Invocation.ParseError)") } + if ([string]::IsNullOrWhiteSpace([string]$Invocation.Stdout)) { [void]$reasons.Add('runner preflight returned no JSON result.') } + } + if ($null -eq $invocationExitCode -or [int]$invocationExitCode -ne 0) { + $diagnostic = [string]::Join(' ', @([string]$Invocation.Stderr, [string]$Invocation.Stdout).Where({ -not [string]::IsNullOrWhiteSpace($_) })) + if ([string]::IsNullOrWhiteSpace($diagnostic)) { $diagnostic = 'no diagnostic output' } + $reportedExitCode = if ($null -eq $invocationExitCode) { 'unknown' } else { [string]$invocationExitCode } + [void]$reasons.Add("runner preflight exited with status ${reportedExitCode}: $diagnostic") + } + if ($invocationTimedOut) { + [void]$reasons.Add('runner preflight watchdog timed out; no execution was started.') + } + if (-not $invocationTerminated) { [void]$reasons.Add('runner preflight process termination was not observed; no execution was started.') } + + $effectivePreflight = if ($null -ne $preflight) { + $preflight + } else { + [ordered]@{ + status = 'incompatible' + delegation = [ordered]@{} + resolved_capabilities = [ordered]@{} + } + } + $status = [string](Get-JsonProperty -Object $effectivePreflight -Name 'status' -Default 'incompatible') + if ($status -ne 'compatible' -and $reasons.Count -eq 0) { [void]$reasons.Add("runner preflight returned status '$status'.") } + $delegationAssertion = 'passed' + $delegationError = '' + try { + [void](Assert-NativeWorkerDelegation -Descriptor $Descriptor -Preflight $effectivePreflight) + } catch { + $delegationAssertion = 'failed' + $delegationError = $_.Exception.Message + [void]$reasons.Add($delegationError) + } + + return [ordered]@{ + worker_id = 'arm-{0}-{1}' -f $Record.EvalId, $Record.Configuration + eval_id = [int]$Record.EvalId + eval_name = [string]$Record.EvalName + configuration = [string]$Record.Configuration + run_manifest = [string]$Record.RunManifestRelative + execution_result = [string]$Record.ExecutionResultRelative + status = if ($status -eq 'compatible' -and $delegationAssertion -eq 'passed' -and $null -ne $invocationExitCode -and [int]$invocationExitCode -eq 0 -and -not $invocationTimedOut -and $invocationTerminated) { 'compatible' } else { 'incompatible' } + reasons = @($reasons.ToArray()) + native_delegation_assertion = [ordered]@{ + status = $delegationAssertion + error = $delegationError + } + runner_exit_code = if ($null -eq $invocationExitCode) { $null } else { [int]$invocationExitCode } + } +} + +function Get-PreflightGateSummary { + param( + [Parameter(Mandatory = $true)][object]$Profile, + [Parameter(Mandatory = $true)][object[]]$Preflights, + [string]$Status = 'preflight_incompatible', + [bool]$ExecutionStarted = $false, + [int]$ExecutionCount = 0, + [string]$Error = '' + ) + + $incompatible = @($Preflights | Where-Object { [string]$_.status -ne 'compatible' }) + $summary = [ordered]@{ + schema = 'codebeltnet/agentic/runner-owned-fanout-summary/1' + phase = 'preflight' + status = $Status + runner = [string](Get-JsonProperty -Object $Profile -Name 'runner' -Default '') + model = [string](Get-JsonProperty -Object $Profile -Name 'model' -Default '') + dispatch_owner = 'runner' + requested_concurrency = [int](Get-JsonProperty -Object $Profile -Name 'concurrency' -Default 0) + preflight_count = @($Preflights).Count + incompatible_count = $incompatible.Count + execution_started = $ExecutionStarted + execution_count = $ExecutionCount + preflights = @($Preflights) + } + if (-not [string]::IsNullOrWhiteSpace($Error)) { $summary.error = $Error } + return $summary +} + +function New-ArmSummary { + param([Parameter(Mandatory = $true)][object]$State) + + return @(Get-OrchestrationCompletedEntries -State $State | Sort-Object { [string](Get-JsonProperty -Object $_ -Name 'worker_id' -Default '') } | ForEach-Object { + $evidenceValidation = Get-JsonProperty -Object $_ -Name 'evidence_validation' -Default $null + [ordered]@{ + worker_id = [string](Get-JsonProperty -Object $_ -Name 'worker_id' -Default '') + eval_id = [int](Get-JsonProperty -Object $_ -Name 'eval_id' -Default 0) + eval_name = [string](Get-JsonProperty -Object $_ -Name 'eval_name' -Default '') + configuration = [string](Get-JsonProperty -Object $_ -Name 'configuration' -Default '') + status = [string](Get-JsonProperty -Object $_ -Name 'status' -Default '') + worker_session_id = Get-JsonProperty -Object $_ -Name 'worker_session_id' -Default $null + evidence_validation = [ordered]@{ + status = [string](Get-JsonProperty -Object $evidenceValidation -Name 'status' -Default '') + reasons = @((Get-JsonProperty -Object $evidenceValidation -Name 'reasons' -Default @())) + } + native_worker_evidence = [string](Get-JsonProperty -Object $_ -Name 'native_worker_evidence' -Default '') + native_worker_evidence_failures = @((Get-JsonProperty -Object $_ -Name 'native_worker_evidence_failures' -Default @())) + } + }) +} + +function Get-FanoutSummary { + param( + [Parameter(Mandatory = $true)][object]$Profile, + [Parameter(Mandatory = $true)][object]$Plan, + [Parameter(Mandatory = $true)][object]$State, + [object]$Concurrency = $null, + [object[]]$Preflights = @(), + [string]$Status = 'failed', + [string]$Error = '' + ) + + $expectedCount = @((Get-JsonProperty -Object $Plan -Name 'arms' -Default @())).Count + $aggregate = Get-FanoutPhase1Aggregate -ExpectedCount $expectedCount -State $State + $preflight = Get-JsonProperty -Object $State -Name 'preflight' -Default $null + $executionStarted = [bool](Get-JsonProperty -Object $preflight -Name 'execution_started' -Default ($aggregate.terminal_count -gt 0)) + if ([string]::IsNullOrWhiteSpace($Status)) { $Status = [string]$aggregate.status } + $summary = [ordered]@{ + schema = 'codebeltnet/agentic/runner-owned-fanout-summary/1' + phase = 'phase1' + status = $Status + runner = [string](Get-JsonProperty -Object $Profile -Name 'runner' -Default '') + model = [string](Get-JsonProperty -Object $Profile -Name 'model' -Default '') + dispatch_owner = [string](Get-JsonProperty -Object $Plan -Name 'dispatch_owner' -Default 'runner') + requested_concurrency = [int](Get-JsonProperty -Object $Plan -Name 'requested_concurrency' -Default 0) + expected_count = [int]$aggregate.expected_count + terminal_count = [int]$aggregate.terminal_count + completed_count = [int]$aggregate.completed_count + failed_count = [int]$aggregate.failed_count + timed_out_count = [int]$aggregate.timed_out_count + cancelled_count = [int]$aggregate.cancelled_count + incompatible_count = [int]$aggregate.incompatible_count + evidence_validation_failed_count = [int]$aggregate.evidence_validation_failed_count + max_observed_active = [int](Get-JsonProperty -Object $State -Name 'max_observed_active' -Default 0) + orchestration_state = 'orchestration-state.json' + preflight_count = @($Preflights).Count + execution_started = $executionStarted + execution_count = [int]$aggregate.terminal_count + arms = @(New-ArmSummary -State $State) + } + if (@($Preflights).Count -gt 0) { $summary.preflights = @($Preflights) } + if ($null -ne $Concurrency) { $summary.concurrency = $Concurrency } + if (-not [string]::IsNullOrWhiteSpace($Error)) { $summary.error = $Error } + return $summary +} + +$running = $null +try { + $manifestPath = Join-Path $iteration 'manifest.json' + $manifest = Read-RunnerJson -Path $manifestPath + $freezeRelativePath = [string](Get-JsonProperty -Object $manifest -Name 'execution_freeze' -Default '') + if ([string]::IsNullOrWhiteSpace($freezeRelativePath)) { throw 'manifest.json must declare execution_freeze.' } + $freezePath = Get-ExecutionFreezePath -IterationDirectory $iteration -RelativePath $freezeRelativePath + if (Test-Path -LiteralPath $freezePath) { + throw "Execution integrity failure: Phase 1 is already frozen at '$freezePath'; refusing a second runner-owned execution. Requires fresh Phase 1 execution." + } + $profileRelativePath = [string](Get-JsonProperty -Object $manifest -Name 'execution_profile' -Default '') + if ([string]::IsNullOrWhiteSpace($profileRelativePath)) { throw 'manifest.json must declare execution_profile.' } + $profilePath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $profileRelativePath -FieldName 'execution_profile' -Kind File -RequireExists + $profile = Resolve-ExecutionProfile -ProfilePath $profilePath + $runnerName = [string]$profile.Runner + + $resolverPath = Join-Path $PSScriptRoot 'resolve-runner.ps1' + $resolutionOutput = & pwsh -NoProfile -File $resolverPath $runnerName 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Selected runner '$runnerName' could not be resolved: $([string]::Join(' ', @($resolutionOutput)))" } + $resolution = ([string]::Join([Environment]::NewLine, @($resolutionOutput)) | ConvertFrom-Json) + $runnerPath = [System.IO.Path]::GetFullPath((Join-Path $PSScriptRoot ([string]$resolution.path -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + if (-not (Test-Path -LiteralPath $runnerPath -PathType Leaf)) { throw "Resolved runner '$runnerName' is missing its runner.ps1." } + $descriptor = Get-PackageRunnerDescriptor -RunnerName $runnerName + $delegation = Get-JsonProperty -Object $descriptor -Name 'delegation' -Default $null + if ([string](Get-JsonProperty -Object $delegation -Name 'dispatch_owner' -Default '') -ne 'runner') { + throw "Selected runner '$runnerName' does not declare delegation.dispatch_owner=runner." + } + + $statePath = Join-Path $iteration 'orchestration-state.json' + if (Test-Path -LiteralPath $statePath -PathType Leaf) { + throw "Runner-owned fan-out refuses to replace an existing orchestration state at '$statePath'." + } + $manifestRecords = @(Get-ManifestRunRecords -IterationDirectory $iteration -Manifest $manifest) + $preflightRecords = [System.Collections.Generic.List[object]]::new() + foreach ($record in $manifestRecords) { + $invocation = Invoke-RunnerPreflight -RunnerPath $runnerPath -RunPath $record.RunManifestPath -ProfilePath ([string]$profile.Path) -TimeoutSeconds (Get-RunnerPreflightTimeoutSeconds) + $preflightRecords.Add((New-PreflightWorkerSummary -Record $record -Invocation $invocation -Descriptor $descriptor)) + } + $failedPreflights = @($preflightRecords | Where-Object { [string]$_.status -ne 'compatible' }) + if ($failedPreflights.Count -gt 0) { + Write-FanoutSummary -Summary (Get-PreflightGateSummary -Profile $profile.Profile -Preflights @($preflightRecords.ToArray())) -ExitCode 2 + } + + $plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile -Descriptor $descriptor + [void](Assert-OrchestrationPlanContract -Plan $plan) + if ([string]$plan.dispatch_owner -ne 'runner') { throw 'Runner-owned fan-out requires a runner-owned orchestration plan.' } + $state = New-OrchestrationState -Plan $plan + $state.preflight = [ordered]@{ + status = 'passed' + count = $preflightRecords.Count + workers = @($preflightRecords.ToArray()) + execution_started = $false + } + Save-OrchestrationState -Path $statePath -State $state + + $running = [System.Collections.Generic.List[object]]::new() + while (@($state.pending_worker_ids).Count -gt 0 -or $running.Count -gt 0) { + foreach ($dispatch in @(Get-NextWorkerDispatches -Plan $plan -State $state)) { + $workerId = [string]$dispatch.worker_id + $arm = Get-OrchestrationArmByWorkerId -Plan $plan -WorkerId $workerId + $runPath = [string]$arm.worker.run_manifest_path + $executionResultRelativePath = [string]$arm.parent_paths.execution_result + $executionResultPath = Resolve-ManifestDeclaredPath -IterationDirectory $iteration -RelativePath $executionResultRelativePath -FieldName "$workerId.execution_result" -Kind File + if (Test-Path -LiteralPath $executionResultPath) { + throw "$workerId has an existing manifest-declared execution result; refusing to overwrite a prior attempt." + } + $stderrPath = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-runner-owned-' + [Guid]::NewGuid().ToString('N') + '.stderr') + # ProcessStartInfo.ArgumentList escapes each argument natively, so raw + # paths are passed without manual quoting. + $arguments = @( + '-NoProfile' + '-File' + $runnerPath + 'execute' + '-Run' + $runPath + '-Profile' + ([string]$profile.Path) + ) + $pwshPath = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) + $childBudget = Get-RunnerChildTimeoutSeconds -RunPath $runPath -ProfileTimeoutSeconds ([int]$profile.TimeoutSeconds) + # Headless child: CreateNoWindow suppresses the per-child console + # window on Windows while the process stays a real isolation + # boundary. The child's sole stdout is streamed to the exact + # manifest-declared execution result; stderr is captured separately. + $child = Start-RunnerChildProcess -FilePath $pwshPath -ArgumentList $arguments -WorkingDirectory (Split-Path -Parent $runPath) -StdoutPath $executionResultPath -StderrPath $stderrPath -TimeoutSeconds ([int]$childBudget.TimeoutSeconds) + if (-not [bool]$state.preflight.execution_started) { + $state.preflight.execution_started = $true + Save-OrchestrationState -Path $statePath -State $state + } + [void](Register-DelegationAccepted -State $state -WorkerId $workerId) + Save-OrchestrationState -Path $statePath -State $state + $running.Add([pscustomobject]@{ worker_id = $workerId; child = $child; Process = $child.Process; result_path = $executionResultPath; stderr_path = $stderrPath; run_path = $runPath; timeout_seconds = [int]$childBudget.TimeoutSeconds; turn_count = [int]$childBudget.TurnCount }) + } + + if ($running.Count -eq 0) { throw 'Runner-owned fan-out has pending arms but no active native process.' } + # Release a slot as soon as ANY child completes, not only the oldest in + # the list, so a slow eval execution never blocks refilling the slot a + # faster sibling already freed. + $completedIndex = Wait-AnyRunnerChild -Running $running + $activeRun = $running[$completedIndex] + $exitCode = Complete-RunnerChildProcess -Child $activeRun.child + if ([bool]$activeRun.child.TimedOut) { + throw "$($activeRun.worker_id) runner child watchdog timed out after $($activeRun.timeout_seconds) seconds (turns=$($activeRun.turn_count)); no retry or redispatch was performed." + } + if (-not (Test-Path -LiteralPath $activeRun.result_path -PathType Leaf) -or (Get-Item -LiteralPath $activeRun.result_path).Length -eq 0) { + $stderr = if (Test-Path -LiteralPath $activeRun.stderr_path -PathType Leaf) { [System.IO.File]::ReadAllText($activeRun.stderr_path, [System.Text.UTF8Encoding]::new($false)).Trim() } else { '' } + throw "$($activeRun.worker_id) runner exited with status $exitCode without writing its manifest-declared execution result. $stderr" + } + $executionResult = Read-RunnerJson -Path $activeRun.result_path + [void](Assert-ExecutionResult -Result $executionResult) + $sessionId = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $executionResult -Name 'session' -Default $null) -Name 'id' -Default '') + [void](Register-DelegationSession -State $state -WorkerId ([string]$activeRun.worker_id) -WorkerSessionId $sessionId) + [void](Register-WorkerTerminal -Plan $plan -State $state -WorkerId ([string]$activeRun.worker_id) -ExecutionEvidence $executionResult) + Save-OrchestrationState -Path $statePath -State $state + if (Test-Path -LiteralPath $activeRun.stderr_path -PathType Leaf) { Remove-Item -LiteralPath $activeRun.stderr_path -Force -ErrorAction SilentlyContinue } + $running.RemoveAt($completedIndex) + } + + $concurrency = Assert-OrchestrationConcurrency -Plan $plan -State $state + Save-OrchestrationState -Path $statePath -State $state + # Phase 1 ends here. Freeze the exact runner-produced bytes and all raw + # artifacts before any bridge, grader, or report process can run. The + # freeze is intentionally written once; later phases can validate it but + # cannot replace it after evidence changes. + $freeze = New-ExecutionFreezeDocument -IterationDirectory $iteration -Manifest $manifest -Records $manifestRecords -Profile $profile + $freezePath = Write-ExecutionFreezeDocument -IterationDirectory $iteration -Freeze $freeze -RelativePath $freezeRelativePath + $state.execution_freeze = [ordered]@{ + schema = (Get-RunnerSchemaNames).ExecutionFreeze + path = [System.IO.Path]::GetRelativePath($iteration, $freezePath).Replace('\', '/') + sha256 = Get-Sha256HexFromFile -Path $freezePath + } + Save-OrchestrationState -Path $statePath -State $state + $aggregate = Get-FanoutPhase1Aggregate -ExpectedCount @($plan.arms).Count -State $state + $summary = Get-FanoutSummary -Profile $profile -Plan $plan -State $state -Concurrency $concurrency -Preflights @($preflightRecords.ToArray()) -Status ([string]$aggregate.status) + $summary.execution_freeze = [ordered]@{ + path = $freezePath + sha256 = [string]$state.execution_freeze.sha256 + schema = [string]$state.execution_freeze.schema + } + $phaseOneExitCode = if (Test-FanoutPhase1Success -Aggregate $aggregate) { 0 } else { 2 } + Write-FanoutSummary -Summary $summary -ExitCode $phaseOneExitCode +} catch { + $errorMessage = $_.Exception.Message + if ($null -ne $running) { + foreach ($active in @($running.ToArray())) { + try { [void](Complete-RunnerChildProcess -Child $active.child -TimeoutSeconds 1) } catch { } + } + } + $fallbackProfile = [ordered]@{ runner = ''; model = '' } + $fallbackPlan = [ordered]@{ dispatch_owner = 'runner'; requested_concurrency = 0 } + $fallbackState = [ordered]@{ max_observed_active = 0; completed = [ordered]@{} } + try { + if (Test-Path -LiteralPath (Join-Path $iteration 'orchestration-state.json') -PathType Leaf) { $fallbackState = Read-RunnerJson -Path (Join-Path $iteration 'orchestration-state.json') } + } catch { } + try { + Write-FanoutSummary -Summary (Get-FanoutSummary -Profile $fallbackProfile -Plan $fallbackPlan -State $fallbackState -Status 'failed' -Error $errorMessage) -ExitCode 2 + } catch { + # A persisted JSON state is intentionally deserialized and therefore no + # longer exposes the mutable dictionaries used by the live queue. Keep + # the failure machine-readable without masking its original reason. + Write-FanoutSummary -Summary ([ordered]@{ + schema = 'codebeltnet/agentic/runner-owned-fanout-summary/1' + phase = 'phase1' + status = 'failed' + runner = '' + model = '' + dispatch_owner = 'runner' + requested_concurrency = 0 + expected_count = 0 + terminal_count = 0 + completed_count = 0 + failed_count = 0 + timed_out_count = 0 + cancelled_count = 0 + incompatible_count = 0 + evidence_validation_failed_count = 0 + max_observed_active = 0 + orchestration_state = 'orchestration-state.json' + arms = @() + error = $errorMessage + }) -ExitCode 2 + } +} diff --git a/scripts/eval-runners/manifest-paths.ps1 b/scripts/eval-runners/manifest-paths.ps1 new file mode 100644 index 0000000..4f257c7 --- /dev/null +++ b/scripts/eval-runners/manifest-paths.ps1 @@ -0,0 +1,373 @@ +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +. (Join-Path $PSScriptRoot 'runner-common.ps1') + +function Resolve-ManifestDeclaredPath { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][string]$FieldName, + [ValidateSet('Any', 'File', 'Directory')][string]$Kind = 'Any', + [switch]$RequireExists + ) + + Assert-SafeRelativePath -RelativePath $RelativePath -FieldName $FieldName + $resolvedIteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + $candidate = [System.IO.Path]::GetFullPath((Join-Path $resolvedIteration ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + if (-not (Test-PathInside -BasePath $resolvedIteration -CandidatePath $candidate)) { + throw "$FieldName resolves outside the prepared iteration package." + } + + $exists = switch ($Kind) { + 'File' { Test-Path -LiteralPath $candidate -PathType Leaf } + 'Directory' { Test-Path -LiteralPath $candidate -PathType Container } + default { Test-Path -LiteralPath $candidate } + } + if ($RequireExists -and -not $exists) { + throw "$FieldName '$RelativePath' does not exist in the prepared iteration package." + } + + if ($exists) { + $resolvedCandidate = (Resolve-Path -LiteralPath $candidate -ErrorAction Stop).Path + if (-not (Test-PathInside -BasePath $resolvedIteration -CandidatePath $resolvedCandidate)) { + throw "$FieldName resolves through a link outside the prepared iteration package." + } + return $resolvedCandidate + } + + return $candidate +} + +function Get-ManifestConfigurations { + param([Parameter(Mandatory = $true)][object]$Manifest) + + $configurations = @(Get-JsonProperty -Object $Manifest -Name 'configurations' -Default @()) + if ($configurations.Count -eq 0) { + throw 'manifest.json must declare at least one configuration arm.' + } + + $seen = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) + $result = [System.Collections.Generic.List[string]]::new() + foreach ($value in $configurations) { + $configuration = [string]$value + if ([string]::IsNullOrWhiteSpace($configuration)) { + throw 'manifest.json contains an empty configuration arm.' + } + if (-not $seen.Add($configuration)) { + throw "manifest.json declares configuration arm '$configuration' more than once." + } + $result.Add($configuration) + } + + return @($result) +} + +function Get-ManifestRunRecords { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest + ) + + $records = [System.Collections.Generic.List[object]]::new() + $seenPaths = @{} + $configurations = @(Get-ManifestConfigurations -Manifest $Manifest) + $evals = @(Get-JsonProperty -Object $Manifest -Name 'evals' -Default @()) + if ($evals.Count -eq 0) { + throw 'manifest.json must declare at least one eval case.' + } + + foreach ($entry in $evals) { + $evalName = [string](Get-JsonProperty -Object $entry -Name 'eval_name' -Default '') + $evalId = [int](Get-JsonProperty -Object $entry -Name 'eval_id' -Default 0) + $directoryRelative = [string](Get-JsonProperty -Object $entry -Name 'directory' -Default '') + $metadataRelative = [string](Get-JsonProperty -Object $entry -Name 'metadata' -Default '') + if ([string]::IsNullOrWhiteSpace($evalName) -or $evalId -lt 1) { + throw 'Each manifest eval entry must declare a non-empty eval_name and positive eval_id.' + } + + $evalDirectory = Resolve-ManifestDeclaredPath ` + -IterationDirectory $IterationDirectory ` + -RelativePath $directoryRelative ` + -FieldName "$evalName.directory" ` + -Kind Directory ` + -RequireExists + $metadataPath = Resolve-ManifestDeclaredPath ` + -IterationDirectory $IterationDirectory ` + -RelativePath $metadataRelative ` + -FieldName "$evalName.metadata" ` + -Kind File ` + -RequireExists + $runContainer = Get-JsonProperty -Object $entry -Name 'runs' -Default $null + if ($null -eq $runContainer) { + throw "$evalName manifest entry must declare runs." + } + + foreach ($configuration in $configurations) { + $runEntry = Get-JsonProperty -Object $runContainer -Name $configuration -Default $null + if ($null -eq $runEntry) { + throw "$evalName manifest entry is missing runs.$configuration." + } + + $declaredPaths = [ordered]@{} + foreach ($field in @('run_manifest', 'execution_result', 'result')) { + if (-not (Test-JsonProperty -Object $runEntry -Name $field)) { + throw "$evalName/$configuration manifest entry must declare '$field'." + } + $relative = [string](Get-JsonProperty -Object $runEntry -Name $field -Default '') + if ([string]::IsNullOrWhiteSpace($relative)) { + throw "$evalName/$configuration manifest field '$field' must be non-empty." + } + $declaredPaths[$field] = $relative + } + + $runManifestPath = Resolve-ManifestDeclaredPath ` + -IterationDirectory $IterationDirectory ` + -RelativePath $declaredPaths.run_manifest ` + -FieldName "$evalName/$configuration.run_manifest" ` + -Kind File ` + -RequireExists + $runManifest = Read-RunnerJson -Path $runManifestPath + $runEvalId = [int](Get-JsonProperty -Object $runManifest -Name 'evalId' -Default 0) + $runEvalName = [string](Get-JsonProperty -Object $runManifest -Name 'evalName' -Default '') + $runMode = [string](Get-JsonProperty -Object $runManifest -Name 'mode' -Default '') + if ($runEvalId -ne $evalId -or $runEvalName -ne $evalName -or $runMode -ne $configuration) { + throw "$evalName/$configuration run_manifest '$($declaredPaths.run_manifest)' identifies evalId='$runEvalId', evalName='$runEvalName', mode='$runMode'; it does not match the manifest arm." + } + $executionResultPath = Resolve-ManifestDeclaredPath ` + -IterationDirectory $IterationDirectory ` + -RelativePath $declaredPaths.execution_result ` + -FieldName "$evalName/$configuration.execution_result" ` + -Kind File + $resultPath = Resolve-ManifestDeclaredPath ` + -IterationDirectory $IterationDirectory ` + -RelativePath $declaredPaths.result ` + -FieldName "$evalName/$configuration.result" ` + -Kind File ` + -RequireExists + + $mode = [string](Get-JsonProperty -Object $runEntry -Name 'mode' -Default '') + if (-not [string]::IsNullOrWhiteSpace($mode) -and $mode -ne $configuration) { + throw "$evalName/$configuration manifest mode '$mode' does not match its arm key." + } + + foreach ($path in @( + [pscustomobject]@{ Field = 'run_manifest'; FullPath = $runManifestPath } + [pscustomobject]@{ Field = 'execution_result'; FullPath = $executionResultPath } + [pscustomobject]@{ Field = 'result'; FullPath = $resultPath } + )) { + $pathKey = [System.IO.Path]::GetFullPath($path.FullPath) + if ($seenPaths.ContainsKey($pathKey)) { + $previous = $seenPaths[$pathKey] + throw "$evalName/$configuration.$($path.Field) duplicates $($previous.EvalName)/$($previous.Configuration).$($previous.Field)." + } + $seenPaths[$pathKey] = [pscustomobject]@{ + EvalName = $evalName + Configuration = $configuration + Field = $path.Field + } + } + + $records.Add([pscustomobject]@{ + EvalId = $evalId + EvalName = $evalName + Configuration = $configuration + EvalDirectory = $evalDirectory + MetadataPath = $metadataPath + RunEntry = $runEntry + RunManifestRelative = $declaredPaths.run_manifest + ExecutionResultRelative = $declaredPaths.execution_result + ResultRelative = $declaredPaths.result + RunManifestPath = $runManifestPath + ExecutionResultPath = $executionResultPath + ResultPath = $resultPath + }) + } + } + + return @($records) +} + +function Get-ManifestShadowResultFiles { + param([Parameter(Mandatory = $true)][object[]]$Records) + + $shadows = [System.Collections.Generic.List[object]]::new() + $groups = @($Records | Group-Object { Split-Path -Parent ([string]$_.ResultPath) }) + foreach ($group in $groups) { + $canonical = @($group.Group) + $canonicalPaths = @{} + $canonicalNames = @{} + foreach ($record in $canonical) { + $fullPath = [System.IO.Path]::GetFullPath([string]$record.ResultPath) + $canonicalPaths[$fullPath] = $record + $normalizedName = ([System.IO.Path]::GetFileName($fullPath)).ToLowerInvariant() -replace '[-_]', '' + $canonicalNames[$normalizedName] = $record + } + + $resultDirectory = [string]$group.Name + if (-not (Test-Path -LiteralPath $resultDirectory -PathType Container)) { + continue + } + foreach ($candidate in @(Get-ChildItem -LiteralPath $resultDirectory -File -Filter '*.result.json' -Force)) { + $candidatePath = [System.IO.Path]::GetFullPath($candidate.FullName) + if ($canonicalPaths.ContainsKey($candidatePath)) { + continue + } + $normalizedName = $candidate.Name.ToLowerInvariant() -replace '[-_]', '' + if ($canonicalNames.ContainsKey($normalizedName)) { + $record = $canonicalNames[$normalizedName] + $shadows.Add([pscustomobject]@{ + Path = $candidatePath + CanonicalPath = [string]$record.ResultPath + EvalName = [string]$record.EvalName + Configuration = [string]$record.Configuration + }) + } + } + } + + return @($shadows) +} + +function Test-ManifestResults { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest, + [object[]]$Records, + [switch]$RequireComplete + ) + + $manifestRecords = if ($null -eq $Records -or $Records.Count -eq 0) { + @(Get-ManifestRunRecords -IterationDirectory $IterationDirectory -Manifest $Manifest) + } else { + @($Records) + } + $errors = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $terminalStatuses = @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible') + $terminalExecutionResults = 0 + $bridgedResults = 0 + + foreach ($shadow in @(Get-ManifestShadowResultFiles -Records $manifestRecords)) { + $errors.Add("$($shadow.EvalName)/$($shadow.Configuration) has an unreferenced result-like sibling '$($shadow.Path)'; the manifest canonical result is '$($shadow.CanonicalPath)'.") + } + + foreach ($record in $manifestRecords) { + $rawExists = Test-Path -LiteralPath $record.ExecutionResultPath -PathType Leaf + $canonicalExists = Test-Path -LiteralPath $record.ResultPath -PathType Leaf + if (-not $canonicalExists) { + $errors.Add("$($record.EvalName)/$($record.Configuration) is missing its manifest-declared result '$($record.ResultRelative)'.") + continue + } + + $raw = $null + $rawStatus = '' + if ($rawExists) { + try { + $raw = Read-RunnerJson -Path $record.ExecutionResultPath + $rawStatus = [string](Get-JsonProperty -Object $raw -Name 'status' -Default '') + } catch { + $errors.Add("$($record.EvalName)/$($record.Configuration) manifest-declared execution result '$($record.ExecutionResultRelative)' is invalid: $($_.Exception.Message)") + continue + } + + if ($terminalStatuses -notcontains $rawStatus) { + $errors.Add("$($record.EvalName)/$($record.Configuration) execution result has non-terminal status '$rawStatus'.") + } else { + $terminalExecutionResults++ + if ($RequireComplete -and $rawStatus -ne 'completed') { + if ($rawStatus -eq 'incompatible') { + $errors.Add("$($record.EvalName)/$($record.Configuration) is incompatible; incompatible execution evidence is diagnostic only and cannot be graded or benchmarked.") + } else { + $errors.Add("$($record.EvalName)/$($record.Configuration) execution status '$rawStatus' is terminal but incomplete; only completed execution evidence can be graded or benchmarked.") + } + } + } + } elseif ($RequireComplete) { + $errors.Add("$($record.EvalName)/$($record.Configuration) is missing its manifest-declared execution result '$($record.ExecutionResultRelative)'.") + } else { + $warnings.Add("$($record.EvalName)/$($record.Configuration) has no execution result at the exact manifest path '$($record.ExecutionResultRelative)'.") + } + + $canonical = $null + try { + $canonical = Read-RunnerJson -Path $record.ResultPath + } catch { + $errors.Add("$($record.EvalName)/$($record.Configuration) manifest-declared result '$($record.ResultRelative)' is invalid: $($_.Exception.Message)") + continue + } + + if ([string]$canonical.configuration -ne $record.Configuration) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical result declares configuration '$($canonical.configuration)'.") + } + if ([int]$canonical.eval_id -ne $record.EvalId) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical result declares eval_id '$($canonical.eval_id)'.") + } + + $metadata = $null + try { + $metadata = Read-RunnerJson -Path $record.MetadataPath + } catch { + $errors.Add("$($record.EvalName) manifest-declared metadata '$($record.MetadataPath)' is invalid: $($_.Exception.Message)") + } + $assertionCount = if ($null -eq $metadata) { -1 } else { @((Get-JsonProperty -Object $metadata -Name 'assertions' -Default @())).Count } + $grading = @(Get-JsonProperty -Object $canonical -Name 'grading' -Default @()) + if ($assertionCount -ge 0 -and $grading.Count -ne $assertionCount) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical result grading count $($grading.Count) does not match the $assertionCount manifest assertions.") + } + + $canonicalStatus = [string](Get-JsonProperty -Object $canonical -Name 'execution_status' -Default '') + if ($rawExists -and $terminalStatuses -contains $rawStatus) { + if ($canonicalStatus -eq 'unrun') { + $errors.Add("$($record.EvalName)/$($record.Configuration) has a terminal execution result but its canonical manifest result remains unrun.") + } elseif ($canonicalStatus -ne $rawStatus) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical execution_status '$canonicalStatus' does not match raw status '$rawStatus'.") + } + + foreach ($field in @('model', 'harness', 'execution_status', 'execution_run_id', 'execution_result_file', 'execution_result_sha256')) { + if (-not (Test-JsonProperty -Object $canonical -Name $field) -or [string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $canonical -Name $field -Default ''))) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical result is missing populated '$field'.") + } + } + $expectedExecutionFile = [System.IO.Path]::GetRelativePath($record.EvalDirectory, $record.ExecutionResultPath).Replace('\', '/') + $actualExecutionFile = [string](Get-JsonProperty -Object $canonical -Name 'execution_result_file' -Default '') + if ($actualExecutionFile -ne $expectedExecutionFile) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical execution_result_file '$actualExecutionFile' does not match the manifest execution_result path '$($record.ExecutionResultRelative)'.") + } + $expectedExecutionHash = Get-Sha256HexFromFile -Path $record.ExecutionResultPath + $actualExecutionHash = [string](Get-JsonProperty -Object $canonical -Name 'execution_result_sha256' -Default '') + if ($actualExecutionHash -ne $expectedExecutionHash) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical execution_result_sha256 does not match the current manifest execution result.") + } + if ($rawStatus -eq 'incompatible' -and $RequireComplete) { + $gradedEntries = @($grading | Where-Object { $null -ne (Get-JsonProperty -Object $_ -Name 'passed' -Default $null) }) + if ($gradedEntries.Count -gt 0) { + $errors.Add("$($record.EvalName)/$($record.Configuration) has grading for an incompatible execution; diagnostic arms must not contribute grading evidence.") + } + } + if ($canonicalStatus -eq $rawStatus -and $grading.Count -eq $assertionCount -and $rawStatus -eq 'completed') { + $bridgedResults++ + } + } elseif (-not $rawExists -and $RequireComplete -and $canonicalStatus -ne 'unrun') { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical result is populated without a manifest-declared execution result proving the bridge.") + } + } + + if ($RequireComplete -and $terminalExecutionResults -ne $manifestRecords.Count) { + $errors.Add("Completion gate expected $($manifestRecords.Count) terminal execution results but found $terminalExecutionResults.") + } + if ($RequireComplete -and $bridgedResults -ne $manifestRecords.Count) { + $errors.Add("Completion gate expected $($manifestRecords.Count) bridged canonical results but found $bridgedResults.") + } + + return [pscustomobject]@{ + Records = @($manifestRecords) + Errors = @($errors) + Warnings = @($warnings) + ExpectedArmCount = $manifestRecords.Count + TerminalExecutionResults = $terminalExecutionResults + BridgedResults = $bridgedResults + Complete = $errors.Count -eq 0 -and $terminalExecutionResults -eq $manifestRecords.Count -and $bridgedResults -eq $manifestRecords.Count + Success = $errors.Count -eq 0 + } +} diff --git a/scripts/eval-runners/opencode/runner.ps1 b/scripts/eval-runners/opencode/runner.ps1 new file mode 100644 index 0000000..7c71aa1 --- /dev/null +++ b/scripts/eval-runners/opencode/runner.ps1 @@ -0,0 +1,2623 @@ +<#! +.SYNOPSIS + OpenCode Eval Runner adapter. + +.DESCRIPTION + This is the only place where OpenCode CLI flags, project/global + configuration handling, sandbox process setup, and JSON event parsing are + defined. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'opencode' + version = '0.9.1' + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'OpenCode CLI'; version = 'unavailable' } + capabilities = [ordered]@{ + single_turn = 'supported' + scripted_multi_turn_same_session = 'conditional' + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'conditional' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'conditional' + cache_token_telemetry = 'conditional' + tool_call_telemetry = 'supported' + command_evidence = 'conditional' + file_evidence = 'conditional' + cost_telemetry = 'conditional' + credential_child_filtering = 'conditional' + native_skill_activation_evidence = 'unsupported' + # Behavioral evaluation transport is runner-owned: the runner starts + # foreground opencode run --format json processes, captures the exact + # fresh session id from turn 1 structured events, and uses only a + # model-free help-proven explicit --session continuation for + # scripted turns. OpenCode's native Task/General subagent remains an + # advertised harness capability but is NOT the benchmark transport. + # These controls stay conditional because terminal evidence proves the + # concrete session, model, prompt, cwd, and isolated home/config. + native_worker_delegation = 'conditional' + delegated_worker_full_capability = 'conditional' + delegated_worker_model_lock = 'conditional' + delegated_worker_working_directory = 'conditional' + delegated_worker_result_capture = 'conditional' + delegated_worker_capacity_signal = 'conditional' + } + delegation = [ordered]@{ + dispatch_owner = 'runner' + mode = 'native_worker' + mechanism = 'Runner-owned OpenCode CLI session (opencode run --format json): the runner starts one fresh foreground process, captures its exact non-empty session identity from structured JSON events, and uses only installed-help-proven explicit --session continuation for scripted turns' + worker_role = 'primary-session' + full_capability = 'conditional' + model_lock = 'conditional' + working_directory = 'conditional' + result_capture = 'conditional' + capacity = 'harness_authoritative' + nested_model_execution = $false + } + supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Resolve-OpenCodeInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + return [pscustomobject]@{ + Run = Resolve-RunContract -RunPath $Run + Profile = Resolve-ExecutionProfile -ProfilePath $Profile + } +} + +function Invoke-OpenCodeCli { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][object]$Inputs, + [System.Collections.IDictionary]$Environment, + [byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 60 + ) + + $allArguments = @($CommandInfo.Prefix) + @($Arguments) + return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList $allArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function Get-OpenCodeHelpResult { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs, + [System.Collections.IDictionary]$Environment = $null + ) + + $environment = if ($null -eq $Environment) { New-OpenCodeEnvironment -Inputs $Inputs } else { $Environment } + return Invoke-OpenCodeCli -CommandInfo $CommandInfo -Arguments @('run', '--help') -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 +} + +function Get-OpenCodeDebugHelpResult { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + return Invoke-OpenCodeCli -CommandInfo $CommandInfo -Arguments @('debug', '--help') -Inputs $Inputs -Environment $Environment -TimeoutSeconds 30 +} + +function Get-OpenCodeDebugConfigResult { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + return Invoke-OpenCodeCli -CommandInfo $CommandInfo -Arguments @('debug', 'config') -Inputs $Inputs -Environment $Environment -TimeoutSeconds 30 +} + +function Get-OpenCodeCandidateSkillName { + param([Parameter(Mandatory = $true)][object]$Run) + + if (-not [bool]$Run.CandidateSkillExposed) { return $null } + $skillPath = [System.IO.Path]::GetFullPath([string]$Run.SkillDirectoryPath).TrimEnd([char[]]@('\', '/')) + $skillName = [System.IO.Path]::GetFileName($skillPath) + if ([string]::IsNullOrWhiteSpace($skillName) -or $skillName -notmatch '^[A-Za-z0-9][A-Za-z0-9._-]*$') { + throw "OpenCode prepared candidate skill directory has an unsupported native skill name: '$skillName'." + } + $declaredName = [string](Get-JsonProperty -Object $Run.Contract -Name 'skillName' -Default '') + if (-not [string]::IsNullOrWhiteSpace($declaredName) -and $declaredName -ne $skillName) { + throw "OpenCode prepared candidate skill name '$skillName' does not match run.json skillName '$declaredName'." + } + return $skillName +} + +function Get-OpenCodeSkillPermissionPolicy { + param([Parameter(Mandatory = $true)][object]$Inputs) + + if (-not [bool]$Inputs.Run.CandidateSkillExposed) { return 'deny' } + $skillName = Get-OpenCodeCandidateSkillName -Run $Inputs.Run + $permission = [ordered]@{} + $permission['*'] = 'deny' + $permission[$skillName] = 'allow' + return $permission +} + +function Test-OpenCodePathEqual { + param( + [AllowEmptyString()][string]$Expected, + [AllowEmptyString()][string]$Observed + ) + + if ([string]::IsNullOrWhiteSpace($Expected) -or [string]::IsNullOrWhiteSpace($Observed)) { return $false } + try { + $expectedFull = [System.IO.Path]::GetFullPath($Expected) + $observedFull = [System.IO.Path]::GetFullPath($Observed) + $comparison = if ((Get-PlatformName) -eq 'windows') { [System.StringComparison]::OrdinalIgnoreCase } else { [System.StringComparison]::Ordinal } + return [string]::Equals($expectedFull.TrimEnd([char[]]@('\', '/')), $observedFull.TrimEnd([char[]]@('\', '/')), $comparison) + } catch { + return $false + } +} + +function Get-OpenCodeHostHomeCandidates { + $candidates = [System.Collections.Generic.List[string]]::new() + foreach ($name in @('USERPROFILE', 'HOME')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value) -and $candidates -notcontains $value) { $candidates.Add($value) } + } + if ((Get-PlatformName) -eq 'windows') { + $drive = [Environment]::GetEnvironmentVariable('HOMEDRIVE') + $path = [Environment]::GetEnvironmentVariable('HOMEPATH') + if (-not [string]::IsNullOrWhiteSpace($drive) -and -not [string]::IsNullOrWhiteSpace($path)) { + $combined = $drive + $path + if ($candidates -notcontains $combined) { $candidates.Add($combined) } + } + } + return @($candidates.ToArray()) +} + +function Get-OpenCodeRuntimeHomeObservation { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + $nodeInfo = Resolve-ExternalCommand -Name 'node' + if ($null -ne $nodeInfo) { + try { + $probe = Invoke-RunnerProcess -FileName $nodeInfo.FileName -ArgumentList (@($nodeInfo.Prefix) + @('-p', 'require("os").homedir()')) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -TimeoutSeconds 30 + $runtimeHomeLines = @($probe.Stdout -split "`r?`n" | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) } | Select-Object -First 1) + if ($probe.ExitCode -eq 0 -and -not $probe.TimedOut -and $runtimeHomeLines.Count -eq 1 -and -not [string]::IsNullOrWhiteSpace([string]$runtimeHomeLines[0])) { + return [pscustomobject]@{ + Available = $true + Home = ([string]$runtimeHomeLines[0]).Trim() + Source = 'node.os.homedir' + Process = $probe + Fallback = $false + Reason = $null + } + } + $nodeReason = "node homedir probe exited with status $($probe.ExitCode) or returned no usable path." + } catch { + $nodeReason = "node homedir probe failed: $($_.Exception.Message)" + } + } else { + $nodeReason = 'node is unavailable on PATH.' + } + + # OpenCode is Node-based, but an installed distribution can theoretically + # omit a separately discoverable node executable. Its model-free debug + # paths command is the deterministic fallback for that case. + try { + $paths = Invoke-OpenCodeCli -CommandInfo $CommandInfo -Arguments @('debug', 'paths') -Inputs $Inputs -Environment $Environment -TimeoutSeconds 30 + $pathLine = @(([string]::Join("`n", @($paths.Stdout, $paths.Stderr))) -split "`r?`n" | Where-Object { $_ -match '(?i)^\s*home\s*:?[ \t]+(?.+?)\s*$' } | Select-Object -First 1) + if ($paths.ExitCode -eq 0 -and -not $paths.TimedOut -and $pathLine.Count -eq 1) { + $match = [regex]::Match([string]$pathLine[0], '(?i)^\s*home\s*:?[ \t]+(?.+?)\s*$') + if ($match.Success -and -not [string]::IsNullOrWhiteSpace($match.Groups['path'].Value)) { + return [pscustomobject]@{ + Available = $true + Home = $match.Groups['path'].Value.Trim() + Source = 'opencode.debug.paths' + Process = $paths + Fallback = $true + Reason = $null + } + } + } + $debugReason = "opencode debug paths did not return a parseable home path (exit=$($paths.ExitCode))." + } catch { + $debugReason = "opencode debug paths failed: $($_.Exception.Message)" + } + return [pscustomobject]@{ + Available = $false + Home = $null + Source = 'unavailable' + Process = $null + Fallback = $false + Reason = "$nodeReason $debugReason" + } +} + +function Get-OpenCodeHomeIsolationObservation { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment, + [Parameter(Mandatory = $true)][object]$RuntimeHome + ) + + $expectedHome = [System.IO.Path]::GetFullPath([string]$Inputs.Run.HomeDirectoryPath) + $hostHomes = @(Get-OpenCodeHostHomeCandidates) + $runtimeMatchesExpected = [bool]$RuntimeHome.Available -and (Test-OpenCodePathEqual -Expected $expectedHome -Observed ([string]$RuntimeHome.Home)) + $runtimeMatchesHost = @($hostHomes | Where-Object { Test-OpenCodePathEqual -Expected ([string]$_) -Observed ([string]$RuntimeHome.Home) }).Count -gt 0 + $windowsProfilePartsCoherent = $true + if ((Get-PlatformName) -eq 'windows') { + $drive = [string](Get-JsonProperty -Object $Environment -Name 'HOMEDRIVE' -Default '') + $path = [string](Get-JsonProperty -Object $Environment -Name 'HOMEPATH' -Default '') + $windowsProfilePartsCoherent = -not [string]::IsNullOrWhiteSpace($drive) -and -not [string]::IsNullOrWhiteSpace($path) -and (Test-OpenCodePathEqual -Expected $expectedHome -Observed ($drive + $path)) + } + $pathValues = @('HOME', 'USERPROFILE', 'APPDATA', 'LOCALAPPDATA', 'XDG_CONFIG_HOME', 'XDG_DATA_HOME', 'XDG_CACHE_HOME', 'OPENCODE_CONFIG_DIR', 'OPENCODE_CONFIG') + $pathValuesInsideHome = $true + foreach ($name in $pathValues) { + $value = [string](Get-JsonProperty -Object $Environment -Name $name -Default '') + if ([string]::IsNullOrWhiteSpace($value) -or -not (Test-PathInside -BasePath $expectedHome -CandidatePath $value) -and -not (Test-OpenCodePathEqual -Expected $expectedHome -Observed $value)) { + $pathValuesInsideHome = $false + break + } + } + $nodePathAbsent = -not $Environment.Contains('NODE_PATH') + $valid = [bool]$RuntimeHome.Available -and $runtimeMatchesExpected -and -not $runtimeMatchesHost -and $windowsProfilePartsCoherent -and $pathValuesInsideHome -and $nodePathAbsent + $reasonParts = [System.Collections.Generic.List[string]]::new() + if (-not [bool]$RuntimeHome.Available) { $reasonParts.Add([string]$RuntimeHome.Reason) } + if (-not $runtimeMatchesExpected) { $reasonParts.Add("effective runtime home '$($RuntimeHome.Home)' does not match isolated home '$expectedHome'.") } + if ($runtimeMatchesHost) { $reasonParts.Add("effective runtime home '$($RuntimeHome.Home)' matches a parent user profile/home.") } + if (-not $windowsProfilePartsCoherent) { $reasonParts.Add('HOMEDRIVE/HOMEPATH do not resolve to the isolated home.') } + if (-not $pathValuesInsideHome) { $reasonParts.Add('one or more OpenCode home/config environment paths escape the isolated home.') } + if (-not $nodePathAbsent) { $reasonParts.Add('NODE_PATH is present in the OpenCode child environment.') } + return [pscustomobject]@{ + Valid = $valid + ExpectedHome = $expectedHome + RuntimeHome = [string]$RuntimeHome.Home + RuntimeHomeAvailable = [bool]$RuntimeHome.Available + RuntimeHomeSource = [string]$RuntimeHome.Source + RuntimeHomeMatchesExpected = $runtimeMatchesExpected + RuntimeHomeMatchesHost = $runtimeMatchesHost + HostHomeCandidates = $hostHomes + WindowsProfilePartsCoherent = $windowsProfilePartsCoherent + PathValuesInsideHome = $pathValuesInsideHome + NodePathAbsent = $nodePathAbsent + Environment = [ordered]@{ + HOME = [string](Get-JsonProperty -Object $Environment -Name 'HOME' -Default '') + USERPROFILE = [string](Get-JsonProperty -Object $Environment -Name 'USERPROFILE' -Default '') + HOMEDRIVE = [string](Get-JsonProperty -Object $Environment -Name 'HOMEDRIVE' -Default '') + HOMEPATH = [string](Get-JsonProperty -Object $Environment -Name 'HOMEPATH' -Default '') + APPDATA = [string](Get-JsonProperty -Object $Environment -Name 'APPDATA' -Default '') + LOCALAPPDATA = [string](Get-JsonProperty -Object $Environment -Name 'LOCALAPPDATA' -Default '') + XDG_CONFIG_HOME = [string](Get-JsonProperty -Object $Environment -Name 'XDG_CONFIG_HOME' -Default '') + OPENCODE_CONFIG_DIR = [string](Get-JsonProperty -Object $Environment -Name 'OPENCODE_CONFIG_DIR' -Default '') + OPENCODE_CONFIG = [string](Get-JsonProperty -Object $Environment -Name 'OPENCODE_CONFIG' -Default '') + NODE_PATH_present = -not $nodePathAbsent + } + Reason = [string]::Join(' ', @($reasonParts.ToArray())) + } +} + +function Get-OpenCodeSkillPolicyObservation { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + $configPath = [string](Get-JsonProperty -Object $Environment -Name 'OPENCODE_CONFIG' -Default '') + $expectedPermission = Get-OpenCodeSkillPermissionPolicy -Inputs $Inputs + $observation = [ordered]@{ + available = $false + config_path = $configPath + configured_permission_skill = $null + expected_permission_skill = $expectedPermission + permission_match = $false + external_skill_scans_disabled = [string](Get-JsonProperty -Object $Environment -Name 'OPENCODE_DISABLE_EXTERNAL_SKILLS' -Default '') -eq '1' + claude_code_skill_scans_disabled = [string](Get-JsonProperty -Object $Environment -Name 'OPENCODE_DISABLE_CLAUDE_CODE_SKILLS' -Default '') -eq '1' + candidate_skill_exposed = [bool]$Inputs.Run.CandidateSkillExposed + candidate_skill_name = Get-OpenCodeCandidateSkillName -Run $Inputs.Run + mechanism = 'isolated home/config plus OpenCode external-skill disable flags and permission.skill' + reason = $null + } + if ([string]::IsNullOrWhiteSpace($configPath) -or -not (Test-Path -LiteralPath $configPath -PathType Leaf)) { + $observation.reason = 'OpenCode policy config file is missing.' + return [pscustomobject]$observation + } + try { + $config = [IO.File]::ReadAllText($configPath, [Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 50 + $permission = Get-JsonProperty -Object $config -Name 'permission' -Default $null + $actualPermission = Get-JsonProperty -Object $permission -Name 'skill' -Default $null + $observation.available = $true + $observation.configured_permission_skill = $actualPermission + $observation.permission_match = ($actualPermission | ConvertTo-Json -Depth 20 -Compress) -eq ($expectedPermission | ConvertTo-Json -Depth 20 -Compress) + if (-not [bool]$observation.permission_match) { $observation.reason = 'OpenCode policy config permission.skill does not match the requested arm policy.' } + elseif (-not [bool]$observation.external_skill_scans_disabled -or -not [bool]$observation.claude_code_skill_scans_disabled) { $observation.reason = 'OpenCode external skill discovery disable flags are not both enabled.' } + } catch { + $observation.reason = "OpenCode policy config could not be parsed: $($_.Exception.Message)" + } + return [pscustomobject]$observation +} + +function Get-OpenCodeDebugConfigObservation { + param( + [Parameter(Mandatory = $true)][object]$DebugResult, + [Parameter(Mandatory = $true)][object]$Inputs + ) + + $expectedPermission = Get-OpenCodeSkillPermissionPolicy -Inputs $Inputs + $observation = [ordered]@{ + available = $false + permission_skill = $null + permission_match = $false + output_sha256 = $null + reason = $null + } + $text = [string]::Join("`n", @($DebugResult.Stdout, $DebugResult.Stderr)).Trim() + if ($DebugResult.TimedOut -or $DebugResult.ExitCode -ne 0 -or [string]::IsNullOrWhiteSpace($text)) { + $observation.reason = "opencode debug config failed (exit=$($DebugResult.ExitCode), timed_out=$($DebugResult.TimedOut))." + return [pscustomobject]$observation + } + try { + $config = $text | ConvertFrom-Json -Depth 50 + $permission = Get-JsonProperty -Object $config -Name 'permission' -Default $null + $actualPermission = Get-JsonProperty -Object $permission -Name 'skill' -Default $null + $observation.available = $true + $observation.permission_skill = $actualPermission + $observation.permission_match = ($actualPermission | ConvertTo-Json -Depth 20 -Compress) -eq ($expectedPermission | ConvertTo-Json -Depth 20 -Compress) + $observation.output_sha256 = Get-Sha256HexFromBytes -Bytes ([Text.UTF8Encoding]::new($false).GetBytes($text)) + if (-not [bool]$observation.permission_match) { $observation.reason = 'opencode debug config did not report the exact requested permission.skill policy.' } + } catch { + $observation.reason = "opencode debug config returned non-JSON output: $($_.Exception.Message)" + } + return [pscustomobject]$observation +} + +function Get-OpenCodeSkillRootObservation { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $repo = [string]$Inputs.Run.WorkingDirectoryPath + $isolatedHome = [string]$Inputs.Run.HomeDirectoryPath + $projectSkillRoot = Join-Path (Join-Path $repo '.opencode') 'skills' + $projectSingularSkillRoot = Join-Path (Join-Path $repo '.opencode') 'skill' + $globalOpenCodeSkillRoot = Join-Path (Join-Path (Join-Path $isolatedHome '.config') 'opencode') 'skills' + $globalOpenCodeSingularSkillRoot = Join-Path (Join-Path (Join-Path $isolatedHome '.config') 'opencode') 'skill' + $agentsSkillRoot = Join-Path (Join-Path $isolatedHome '.agents') 'skills' + $claudeSkillRoot = Join-Path (Join-Path $isolatedHome '.claude') 'skills' + $roots = @( + [ordered]@{ kind = 'project'; path = $projectSkillRoot }, + [ordered]@{ kind = 'project'; path = $projectSingularSkillRoot }, + [ordered]@{ kind = 'global_opencode'; path = $globalOpenCodeSkillRoot }, + [ordered]@{ kind = 'global_opencode'; path = $globalOpenCodeSingularSkillRoot }, + [ordered]@{ kind = 'external_agents'; path = $agentsSkillRoot }, + [ordered]@{ kind = 'external_claude'; path = $claudeSkillRoot } + ) + $skills = [System.Collections.Generic.List[object]]::new() + $scanErrors = [System.Collections.Generic.List[string]]::new() + foreach ($root in $roots) { + $rootPath = [string]$root.path + try { + if (-not (Test-Path -LiteralPath $rootPath -PathType Container -ErrorAction Stop)) { continue } + foreach ($directory in @(Get-ChildItem -LiteralPath $rootPath -Directory -Force -ErrorAction Stop)) { + $skillFile = Join-Path $directory.FullName 'SKILL.md' + if (Test-Path -LiteralPath $skillFile -PathType Leaf -ErrorAction Stop) { + $skills.Add([ordered]@{ kind = [string]$root.kind; name = [string]$directory.Name; path = [string]$directory.FullName }) + } + } + } catch { + $scanErrors.Add("$($root.kind) skill root '$rootPath' could not be inspected: $($_.Exception.Message)") + } + } + $candidatePath = if ([bool]$Inputs.Run.CandidateSkillExposed) { [string]$Inputs.Run.SkillDirectoryPath } else { $null } + $candidateSkills = @($skills | Where-Object { -not [string]::IsNullOrWhiteSpace($candidatePath) -and (Test-OpenCodePathEqual -Expected $candidatePath -Observed ([string]$_.path)) }) + $ambientSkills = @($skills | ForEach-Object { + $skill = $_ + if (@($candidateSkills | Where-Object { Test-OpenCodePathEqual -Expected ([string]$_.path) -Observed ([string]$skill.path) }).Count -eq 0) { $skill } + }) + $candidateHash = if ($candidateSkills.Count -eq 1) { Get-OpenCodeTreeHash -Root ([string]$candidateSkills[0].path) } else { $null } + $preparedHash = if ([bool]$Inputs.Run.CandidateSkillExposed) { [string]$Inputs.Run.SkillHash } else { $null } + $valid = if ([bool]$Inputs.Run.CandidateSkillExposed) { + $scanErrors.Count -eq 0 -and $candidateSkills.Count -eq 1 -and $ambientSkills.Count -eq 0 -and $candidateHash -eq $preparedHash + } else { + $scanErrors.Count -eq 0 -and $skills.Count -eq 0 + } + $reason = if ($valid) { $null } elseif ($scanErrors.Count -gt 0) { [string]::Join(' ', @($scanErrors.ToArray())) } elseif ([bool]$Inputs.Run.CandidateSkillExposed) { 'physical OpenCode skill roots do not contain exactly the prepared candidate and no ambient skills.' } else { 'without_skill physical OpenCode skill roots are not empty.' } + return [pscustomobject]@{ + Valid = $valid + CandidateSkillExposed = [bool]$Inputs.Run.CandidateSkillExposed + CandidateSkillPath = $candidatePath + CandidateSkillCount = $candidateSkills.Count + CandidateSkillHash = $candidateHash + PreparedSkillHash = $preparedHash + AmbientSkillCount = $ambientSkills.Count + AmbientSkills = @($ambientSkills) + DiscoveredSkills = @($skills.ToArray()) + ScanErrors = @($scanErrors.ToArray()) + Reason = $reason + } +} + +function New-OpenCodeSkillIsolationEvidence { + param([Parameter(Mandatory = $true)][object]$Observation) + + return [ordered]@{ + valid = [bool]$Observation.Valid + candidate_skill_exposed = [bool]$Observation.CandidateSkillExposed + candidate_skill_path = [string]$Observation.CandidateSkillPath + candidate_skill_count = [int]$Observation.CandidateSkillCount + candidate_skill_hash = [string]$Observation.CandidateSkillHash + prepared_skill_hash = [string]$Observation.PreparedSkillHash + ambient_skill_count = [int]$Observation.AmbientSkillCount + ambient_skills = @($Observation.AmbientSkills) + discovered_skills = @($Observation.DiscoveredSkills) + scan_errors = @($Observation.ScanErrors) + reason = [string]$Observation.Reason + } +} + +function New-OpenCodeHomeIsolationEvidence { + param([Parameter(Mandatory = $true)][object]$Observation) + + return [ordered]@{ + valid = [bool]$Observation.Valid + expected_home = [string]$Observation.ExpectedHome + effective_runtime_home = [string]$Observation.RuntimeHome + effective_runtime_home_available = [bool]$Observation.RuntimeHomeAvailable + effective_runtime_home_source = [string]$Observation.RuntimeHomeSource + effective_runtime_home_matches_expected = [bool]$Observation.RuntimeHomeMatchesExpected + effective_runtime_home_matches_real_user_profile = [bool]$Observation.RuntimeHomeMatchesHost + host_home_candidates = @($Observation.HostHomeCandidates) + windows_profile_parts_coherent = [bool]$Observation.WindowsProfilePartsCoherent + path_values_inside_home = [bool]$Observation.PathValuesInsideHome + node_path_absent = [bool]$Observation.NodePathAbsent + environment = $Observation.Environment + reason = [string]$Observation.Reason + } +} + +function New-OpenCodeExecutionPaths { + param( + [Parameter(Mandatory = $true)][object]$LogicalInputs, + [Parameter(Mandatory = $true)][object]$ExecutionInputs, + [Parameter(Mandatory = $true)][object]$Projection, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment, + [object]$HomeIsolation + ) + + $physicalCwdOutsideSource = $null -eq $Projection.SourceRepositoryRoot -or -not (Test-PathInside -BasePath $Projection.SourceRepositoryRoot -CandidatePath $Projection.PhysicalWorkingDirectory) + $physicalHomeOutsideSource = $null -eq $Projection.SourceRepositoryRoot -or -not (Test-PathInside -BasePath $Projection.SourceRepositoryRoot -CandidatePath $Projection.PhysicalHomeDirectory) + return [ordered]@{ + logical_working_directory = [string]$LogicalInputs.Run.WorkingDirectoryPath + logical_home_directory = [string]$LogicalInputs.Run.HomeDirectoryPath + physical_working_directory = [string]$Projection.PhysicalWorkingDirectory + physical_home_directory = [string]$Projection.PhysicalHomeDirectory + physical_isolated_home = [string]$Projection.PhysicalHomeDirectory + effective_runtime_home = if ($null -eq $HomeIsolation) { $null } else { [string]$HomeIsolation.RuntimeHome } + effective_runtime_home_source = if ($null -eq $HomeIsolation) { 'unavailable' } else { [string]$HomeIsolation.RuntimeHomeSource } + effective_opencode_config_root = [string]$Environment['OPENCODE_CONFIG_DIR'] + physical_config_directory = [string]$Environment['OPENCODE_CONFIG_DIR'] + physical_config_file = [string]$Environment['OPENCODE_CONFIG'] + physical_run_root = [string]$Projection.Root + physical_projection_root = [string]$Projection.Root + physical_cwd_outside_source_repository = [bool]$physicalCwdOutsideSource + physical_home_outside_source_repository = [bool]$physicalHomeOutsideSource + projection_cleanup = 'pending' + } +} + +function New-OpenCodeCandidateSkillExposure { + param( + [Parameter(Mandatory = $true)][object]$LogicalInputs, + [Parameter(Mandatory = $true)][object]$Projection, + [object]$SkillIsolation + ) + + if ([bool]$LogicalInputs.Run.CandidateSkillExposed) { + return [ordered]@{ + status = 'supported' + logical_staged = $true + native_discovery_root = '.opencode/skills//SKILL.md' + candidate_skill_name = Get-OpenCodeCandidateSkillName -Run $LogicalInputs.Run + physical_path = [string]$Projection.PhysicalSkillDirectory + physical_tree_hash = [string]$Projection.PhysicalSkillHash + prepared_tree_hash = [string]$LogicalInputs.Run.SkillHash + hash_match = [string]$Projection.PhysicalSkillHash -eq [string]$LogicalInputs.Run.SkillHash + ambient_skill_roots_hidden = $null -eq $SkillIsolation -or [int]$SkillIsolation.AmbientSkillCount -eq 0 + } + } + return [ordered]@{ + status = 'excluded' + logical_staged = $false + native_discovery_root = $null + candidate_skill_name = $null + physical_path = $null + physical_tree_hash = $null + prepared_tree_hash = $null + hash_match = $null + ambient_skill_roots_hidden = $null -eq $SkillIsolation -or [int]$SkillIsolation.AmbientSkillCount -eq 0 + } +} + +function New-OpenCodeIsolationFailureResult { + param( + [Parameter(Mandatory = $true)][object]$LogicalInputs, + [Parameter(Mandatory = $true)][object]$ExecutionDescriptor, + [Parameter(Mandatory = $true)][object]$Preflight, + [Parameter(Mandatory = $true)][object]$Projection, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment, + [object]$HomeIsolation, + [object]$PolicyObservation, + [object]$SkillIsolation, + [string]$PreflightSource = 'fresh_preflight', + [string[]]$Reasons = @(), + [Parameter(Mandatory = $true)][string]$SessionId, + [Parameter(Mandatory = $true)][datetime]$StartedUtc, + [bool]$Resume = $false + ) + + $finished = [DateTime]::UtcNow + $message = if ($Reasons.Count -gt 0) { [string]::Join('; ', @($Reasons)) } else { 'OpenCode isolation could not be proven before model execution.' } + $executionPaths = New-OpenCodeExecutionPaths -LogicalInputs $LogicalInputs -ExecutionInputs $Projection.Inputs -Projection $Projection -Environment $Environment -HomeIsolation $HomeIsolation + $evidence = [ordered]@{ + preflight = $Preflight + preflight_source = $PreflightSource + execution_paths = $executionPaths + effective_home = if ($null -eq $HomeIsolation) { $null } else { New-OpenCodeHomeIsolationEvidence -Observation $HomeIsolation } + skill_policy = $PolicyObservation + skill_isolation = if ($null -eq $SkillIsolation) { $null } else { New-OpenCodeSkillIsolationEvidence -Observation $SkillIsolation } + candidate_skill_exposure = New-OpenCodeCandidateSkillExposure -LogicalInputs $LogicalInputs -Projection $Projection -SkillIsolation $SkillIsolation + resume = $Resume + native_execution_started = $false + delegation = [ordered]@{ + dispatch_owner = 'runner' + worker_session_id = $SessionId + fresh_worker = $true + home_config_isolated = $false + prompt_fidelity = $false + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 0 + } + } + return New-ExecutionResult -Descriptor $ExecutionDescriptor -Profile $LogicalInputs.Profile -Run $LogicalInputs.Run -Status incompatible -FinalResponseReason 'isolation_incompatible' -StartedUtc $StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $StartedUtc).TotalSeconds -Failure (New-ExecutionFailure -Code 'opencode_isolation_incompatible' -Message $message) -SessionId $SessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('physical home/config/skill isolation was not proven; model execution was not started') -Warnings @('OpenCode model execution was not started because the physical home/config/skill isolation contract failed closed.') -Evidence $evidence -AttemptCount 1 +} + +function Remove-OpenCodeAnsiSequences { + param([AllowEmptyString()][string]$Text) + + return [regex]::Replace($Text, "`e\[[0-?]*[ -/]*[@-~]", '') +} + +function Get-OpenCodeContinuationCapability { + param([Parameter(Mandatory = $true)][AllowEmptyString()][string]$HelpText) + + $cleanText = Remove-OpenCodeAnsiSequences -Text $HelpText + $lines = @($cleanText -split "`r?`n") + $flag = '--session' + $flagPattern = [regex]::Escape($flag) + for ($lineIndex = 0; $lineIndex -lt $lines.Count; $lineIndex++) { + $line = [string]$lines[$lineIndex] + $match = [regex]::Match($line, "(?, --session SESSION_ID, --session=id, + # and the compact `-s, --session session id to continue` form. + # Parse the option token separately from its prose instead of making + # angle/bracket syntax the compatibility contract. + $remainder = $line.Substring($match.Index + $match.Length) + $argumentStyle = 'separate' + $parameter = $null + $equalsMatch = [regex]::Match($remainder, '^\s*=\s*(?[^\s,;]+)') + if ($equalsMatch.Success) { + $argumentStyle = 'equals' + $parameter = [string]$equalsMatch.Groups['value'].Value + } else { + $separateMatch = [regex]::Match($remainder, '^\s+(?[^\s,;]+)') + if ($separateMatch.Success) { + $candidateParameter = [string]$separateMatch.Groups['value'].Value + if ($candidateParameter -notmatch '(?i)^(continue|resume|resumes|the|a|an|existing|previous|specific|target|session)$') { + $parameter = $candidateParameter + } + } + } + + # Keep only the current option entry and nearby wrapped description + # lines. A neighbouring --continue entry is a different option and + # must not accidentally prove exact-session support for --session. + $contextLines = [System.Collections.Generic.List[string]]::new() + if ($lineIndex -gt 0 -and [string]$lines[$lineIndex - 1] -notmatch '(?\]\})]+$', '') } + $parameterIdentifiesSession = $normalizedParameter -match '(?i)^(?:session[-_ ]?id|id|identifier)$' + if (-not $parameterIdentifiesSession -and $identityDescription -and $contextWithoutFlag -match '(?i)\bsession\s*(?:id|identifier)\b') { + # The installed-style entry has prose rather than a separate + # placeholder token: `session id to continue`. + $parameter = 'session-id' + $parameterIdentifiesSession = $true + } + $explicitIdentityDescription = $continuationDescription -and ($identityDescription -or $parameterIdentifiesSession) + if (-not $explicitIdentityDescription) { continue } + return [pscustomobject]@{ + Available = $true + Flag = $flag + ArgumentStyle = $argumentStyle + Parameter = $parameter + HelpEvidence = $context.Trim() + Reason = $null + } + } + return [pscustomobject]@{ + Available = $false + Flag = $null + ArgumentStyle = $null + Parameter = $null + HelpEvidence = $null + Reason = 'The installed OpenCode run help did not prove --session with an explicit session id; --continue or another implicit last-session mode is not permitted.' + } +} + +function New-OpenCodeContinuationArguments { + param( + [Parameter(Mandatory = $true)][object]$Capability, + [Parameter(Mandatory = $true)][string]$SessionId + ) + + if (-not [bool]$Capability.Available -or [string]::IsNullOrWhiteSpace([string]$Capability.Flag)) { + throw 'OpenCode exact-session continuation was requested without a proven continuation capability.' + } + if ([string]$Capability.ArgumentStyle -eq 'equals') { + return @(([string]$Capability.Flag + '=' + $SessionId)) + } + return @([string]$Capability.Flag, $SessionId) +} + +function Invoke-OpenCodeTurnProcess { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment, + [Parameter(Mandatory = $true)][string]$Platform, + [object]$SandboxInfo, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][byte[]]$InputBytes, + [Parameter(Mandatory = $true)][int]$TimeoutSeconds + ) + + $hardFilesystem = $null -ne $SandboxInfo -and $Platform -in @('linux', 'macos') + if ($Platform -eq 'linux' -and $hardFilesystem) { + $sandboxArguments = Get-LinuxSandboxArguments -Inputs $Inputs -CommandInfo $CommandInfo -Environment $Environment + return Invoke-RunnerProcess -FileName $SandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($Arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds + } + if ($Platform -eq 'macos' -and $hardFilesystem) { + $sandboxProfile = New-MacosSandboxProfile -Inputs $Inputs -CommandInfo $CommandInfo + $sandboxArguments = @('-f', $sandboxProfile, '--', $CommandInfo.FileName) + @($CommandInfo.Prefix) + $Arguments + return Invoke-RunnerProcess -FileName $SandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds + } + return Invoke-OpenCodeCli -CommandInfo $CommandInfo -Arguments $Arguments -Inputs $Inputs -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function Invoke-OpenCodeScriptedExecute { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$Preflight, + [Parameter(Mandatory = $true)][object]$ExecutionDescriptor, + [string]$PreflightSource = 'fresh_preflight' + ) + + $started = [DateTime]::UtcNow + $commandInfo = Resolve-ExternalCommand -Name 'opencode' + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } + $protocol = Get-JsonProperty -Object $Preflight -Name 'protocol_observations' -Default $null + $continuationObservation = Get-JsonProperty -Object $protocol -Name 'scripted_multi_turn_same_session' -Default $null + $continuationCapability = [pscustomobject]@{ + Available = [bool](Get-JsonProperty -Object $continuationObservation -Name 'available' -Default $false) + Flag = Get-JsonProperty -Object $continuationObservation -Name 'flag' -Default $null + ArgumentStyle = Get-JsonProperty -Object $continuationObservation -Name 'argument_style' -Default $null + Parameter = Get-JsonProperty -Object $continuationObservation -Name 'parameter' -Default $null + } + if ($null -eq $commandInfo -or -not [bool]$continuationCapability.Available) { + $finished = [DateTime]::UtcNow + $fallbackSessionId = [Guid]::NewGuid().ToString('D') + $failureMessage = [string](Get-JsonProperty -Object $continuationObservation -Name 'reason' -Default 'OpenCode exact-session continuation was not proven by installed help.') + return New-ExecutionResult -Descriptor $ExecutionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message $failureMessage) -SessionId $fallbackSessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $Preflight; resume = $false }) -AttemptCount 1 + } + + $projection = $null + $result = $null + try { + $projection = New-OpenCodeExecutionProjection -Inputs $Inputs + $executionInputs = $projection.Inputs + $environment = New-OpenCodeEnvironment -Inputs $executionInputs + $runtimeHomeObservation = Get-OpenCodeRuntimeHomeObservation -CommandInfo $commandInfo -Inputs $executionInputs -Environment $environment + $homeIsolationObservation = Get-OpenCodeHomeIsolationObservation -Inputs $executionInputs -Environment $environment -RuntimeHome $runtimeHomeObservation + $policyObservation = Get-OpenCodeSkillPolicyObservation -Inputs $executionInputs -Environment $environment + $skillIsolationObservation = Get-OpenCodeSkillRootObservation -Inputs $executionInputs + $isolationReasons = [System.Collections.Generic.List[string]]::new() + if (-not [bool]$homeIsolationObservation.Valid) { $isolationReasons.Add([string]$homeIsolationObservation.Reason) } + if (-not [bool]$policyObservation.permission_match -or -not [bool]$policyObservation.external_skill_scans_disabled -or -not [bool]$policyObservation.claude_code_skill_scans_disabled) { $isolationReasons.Add([string]$policyObservation.reason) } + if (-not [bool]$skillIsolationObservation.Valid) { $isolationReasons.Add([string]$skillIsolationObservation.Reason) } + if ($isolationReasons.Count -gt 0) { + $result = New-OpenCodeIsolationFailureResult -LogicalInputs $Inputs -ExecutionDescriptor $ExecutionDescriptor -Preflight $Preflight -Projection $projection -Environment $environment -HomeIsolation $homeIsolationObservation -PolicyObservation $policyObservation -SkillIsolation $skillIsolationObservation -PreflightSource $PreflightSource -Reasons @($isolationReasons.ToArray()) -SessionId ([Guid]::NewGuid().ToString('D')) -StartedUtc $started -Resume $false + return $result + } + $requestedTurns = @($Inputs.Run.Interaction.turns) + $baseArguments = New-OpenCodeCliArguments -Inputs $executionInputs -VisiblePlatform $visiblePlatform + $turnRecords = [System.Collections.Generic.List[object]]::new() + $nativeTurns = [System.Collections.Generic.List[object]]::new() + $rawStdout = [System.Collections.Generic.List[string]]::new() + $rawStderr = [System.Collections.Generic.List[string]]::new() + $artifacts = [System.Collections.Generic.List[object]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $nativeFailures = [System.Collections.Generic.List[string]]::new() + $eventCounts = @{} + $observedModels = [System.Collections.Generic.List[string]]::new() + $usageBuckets = [ordered]@{} + $toolCalls = 0 + $capturedSessionId = $null + $finalText = $null + $firstProcess = $null + $lastProcess = $null + $status = 'completed' + $failureCode = $null + $failureMessage = $null + $turnTimingRecords = [System.Collections.Generic.List[object]]::new() + $futureCanary = Get-OpenCodeFutureTurnCanary -Run $Inputs.Run + $interactionSourcePhysicalPaths = @(Get-OpenCodeInteractionSourcePhysicalPaths -Inputs $Inputs -Projection $projection) + $interactionJsonPhysicalPaths = @(Get-OpenCodeInteractionJsonPhysicalPaths -Inputs $Inputs -Projection $projection) + $interactionPhysicalPresent = @($interactionJsonPhysicalPaths | Where-Object { Test-Path -LiteralPath $_ -PathType Leaf }).Count -gt 0 + $futureSourcePhysicalPresent = @($interactionSourcePhysicalPaths | Where-Object { Test-Path -LiteralPath $_ }).Count -gt 0 + $canaryInProjectionBeforeTurn1 = Test-OpenCodeCanaryInTree -Root $projection.Root -Canary $futureCanary + $canaryInEnvironment = @($environment.GetEnumerator() | Where-Object { [string]$_.Value -like "*$futureCanary*" }).Count -gt 0 + $turnOneInputContainsFutureCanary = $false + $turnOneArgumentsContainFutureCanary = $false + $turnTwoSentAfterTurnOneTerminal = $false + + for ($turnIndex = 0; $turnIndex -lt $requestedTurns.Count; $turnIndex++) { + $turnText = Get-InteractionTurnText -Turn $requestedTurns[$turnIndex] -RunData $Inputs.Run + $arguments = @($baseArguments) + $targetSessionId = $null + if ($turnIndex -gt 0) { + $targetSessionId = $capturedSessionId + if ([string]::IsNullOrWhiteSpace($targetSessionId)) { + $nativeFailures.Add('session_id_unobservable') + $status = 'incompatible' + $failureCode = 'native_interaction_incompatible' + $failureMessage = 'OpenCode turn 1 did not expose an exact session id, so no continuation invocation was started.' + break + } + $arguments = @($arguments) + @(New-OpenCodeContinuationArguments -Capability $continuationCapability -SessionId $targetSessionId) + } + if ($turnIndex -eq 0) { + $turnOneInputContainsFutureCanary = [string]$turnText -like "*$futureCanary*" + $turnOneArgumentsContainFutureCanary = @($arguments | Where-Object { [string]$_ -like "*$futureCanary*" }).Count -gt 0 + } + + try { + $turnInputBytes = [System.Text.UTF8Encoding]::new($false).GetBytes($turnText) + $process = Invoke-OpenCodeTurnProcess -Inputs $executionInputs -CommandInfo $commandInfo -Arguments $arguments -Environment $environment -Platform $platform -SandboxInfo $sandboxInfo -InputBytes $turnInputBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } catch { + $nativeFailures.Add('transport_failure') + $status = 'failed' + $failureCode = 'opencode_failure' + $failureMessage = $_.Exception.Message + break + } + if ($null -eq $firstProcess) { $firstProcess = $process } + $lastProcess = $process + $rawStdout.Add([string]$process.Stdout) + $rawStderr.Add([string]$process.Stderr) + $turnNumber = $turnIndex + 1 + $turnArtifact = Write-OpenCodeCapture -RunData $Inputs -RelativePath ("evidence/opencode-turn-{0}-events.jsonl" -f $turnNumber) -Text ([string]$process.Stdout) + $turnStderrArtifact = Write-OpenCodeCapture -RunData $Inputs -RelativePath ("evidence/opencode-turn-{0}-stderr.txt" -f $turnNumber) -Text ([string]$process.Stderr) + $artifacts.Add($turnArtifact) + $artifacts.Add($turnStderrArtifact) + $parsed = if ([string]::IsNullOrEmpty([string]$process.Stdout)) { [pscustomobject]@{ Events = @(); Errors = @() } } else { ConvertFrom-JsonLines -Text $process.Stdout } + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("OpenCode turn $turnNumber event parse error: $parseError") } + $parsedEvents = Read-OpenCodeScriptedTurn -Parsed $parsed -Warnings $warnings + $turnTiming = [ordered]@{ + turn = $turnNumber + invocation = if ($turnIndex -eq 0) { 'fresh' } else { 'explicit_session_resume' } + process_duration_seconds = [double]$process.DurationSeconds + cli_startup_and_execution_duration_seconds = [double]$process.DurationSeconds + } + if ($null -ne $parsedEvents.EventTiming) { + $turnTiming.event_timing = $parsedEvents.EventTiming + } + $turnTimingRecords.Add($turnTiming) + foreach ($eventName in $parsedEvents.EventCounts.Keys) { + if ($eventCounts.ContainsKey($eventName)) { $eventCounts[$eventName] += [int]$parsedEvents.EventCounts[$eventName] } else { $eventCounts[$eventName] = [int]$parsedEvents.EventCounts[$eventName] } + } + $toolCalls += [int]$parsedEvents.ToolCalls + foreach ($modelName in @($parsedEvents.ObservedModels)) { + if ($observedModels -notcontains $modelName) { $observedModels.Add($modelName) } + } + foreach ($usageName in $parsedEvents.UsageBuckets.Keys) { + $usageBuckets[$usageName] = Add-OpenCodeNullableInt64 -Current (Get-JsonProperty -Object $usageBuckets -Name $usageName -Default $null) -Value $parsedEvents.UsageBuckets[$usageName] + } + + $sessionIds = @($parsedEvents.SessionIds) + $observedSessionId = if ($sessionIds.Count -eq 1) { [string]$sessionIds[0] } else { $null } + $nativeTurn = [ordered]@{ + turn = $turnNumber + invocation = if ($turnIndex -eq 0) { 'fresh' } else { 'explicit_session_resume' } + arguments = @($arguments) + requested_model = [string]$Inputs.Profile.Model + observed_models = @($parsedEvents.ObservedModels) + model_source = if (@($parsedEvents.ObservedModels).Count -gt 0) { 'structured_event' } else { 'cli_argument' } + session_ids_observed = @($sessionIds) + session_id = $observedSessionId + target_session_id = $targetSessionId + target_session_match = if ($turnIndex -eq 0) { $null } else { $observedSessionId -eq $targetSessionId } + terminal_assistant_response = -not [string]::IsNullOrWhiteSpace([string]$parsedEvents.FinalText) + terminal_event_observed = [bool]$parsedEvents.TerminalEventObserved + structured_event_count = [int]$parsedEvents.StructuredEventCount + structured_parse_errors = [int]$parsedEvents.ParseErrorCount + structured_output = 'json' + working_directory = [string]$executionInputs.Run.WorkingDirectoryPath + home = [string]$executionInputs.Run.HomeDirectoryPath + effective_runtime_home = [string]$homeIsolationObservation.RuntimeHome + effective_runtime_home_source = [string]$homeIsolationObservation.RuntimeHomeSource + config_directory = [string]$environment['OPENCODE_CONFIG_DIR'] + config_file = [string]$environment['OPENCODE_CONFIG'] + skill_policy = $policyObservation.configured_permission_skill + candidate_skill_exposed = [bool]$executionInputs.Run.CandidateSkillExposed + process_duration_seconds = [double]$process.DurationSeconds + cli_startup_and_execution_duration_seconds = [double]$process.DurationSeconds + started_utc = Format-UtcTimestamp -Value $process.StartedUtc + finished_utc = Format-UtcTimestamp -Value $process.FinishedUtc + event_timestamps = @($parsedEvents.EventTimestamps) + exit_code = $process.ExitCode + terminal = -not $process.TimedOut -and $process.ExitCode -eq 0 + } + if ($null -ne $parsedEvents.EventTiming) { + $nativeTurn.event_timing = $parsedEvents.EventTiming + } + $nativeTurns.Add($nativeTurn) + if ($turnIndex -eq 1 -and $nativeTurns.Count -ge 2) { + $turnTwoSentAfterTurnOneTerminal = [DateTime]::Compare([DateTime]$nativeTurns[0].finished_utc, [DateTime]$nativeTurn.started_utc) -le 0 + } + + $turnProblem = $null + if ($process.TimedOut) { $turnProblem = 'turn_timeout'; $status = 'timed_out'; $failureCode = 'timed_out'; $failureMessage = 'OpenCode did not finish before timeout_seconds.' } + elseif ($process.ExitCode -ne 0 -or $null -ne $parsedEvents.FailureMessage) { $turnProblem = 'turn_failed'; $status = 'failed'; $failureCode = 'opencode_failure'; $failureMessage = if ($null -ne $parsedEvents.FailureMessage) { [string]$parsedEvents.FailureMessage } else { "OpenCode exited with status $($process.ExitCode)." } } + elseif ($parsedEvents.ParseErrorCount -gt 0) { $turnProblem = 'structured_event_parse'; $status = 'incompatible'; $failureCode = 'native_interaction_incompatible'; $failureMessage = "OpenCode turn $turnNumber did not produce a complete structured event stream." } + elseif ($sessionIds.Count -ne 1) { $turnProblem = 'session_id_unobservable'; $status = 'incompatible'; $failureCode = 'native_interaction_incompatible'; $failureMessage = "OpenCode turn $turnNumber did not expose exactly one session id in structured events." } + elseif ($turnIndex -gt 0 -and $observedSessionId -ne $targetSessionId) { $turnProblem = 'session_identity_mismatch'; $status = 'incompatible'; $failureCode = 'native_interaction_incompatible'; $failureMessage = "OpenCode turn $turnNumber returned session '$observedSessionId' instead of the exact resumed session '$targetSessionId'." } + elseif ([string]::IsNullOrWhiteSpace([string]$parsedEvents.FinalText) -or -not [bool]$parsedEvents.TerminalEventObserved) { $turnProblem = 'terminal_turn_status'; $status = 'incompatible'; $failureCode = 'native_interaction_incompatible'; $failureMessage = "OpenCode turn $turnNumber did not provide a terminal structured assistant response before continuation." } + elseif (@($parsedEvents.ObservedModels | Where-Object { [string]$_ -ne [string]$Inputs.Profile.Model }).Count -gt 0) { $turnProblem = 'requested_model'; $status = 'incompatible'; $failureCode = 'native_interaction_incompatible'; $failureMessage = "OpenCode turn $turnNumber reported a model different from the requested model '$($Inputs.Profile.Model)'." } + if ($null -ne $turnProblem) { + $nativeFailures.Add($turnProblem) + break + } + if ($turnIndex -eq 0) { $capturedSessionId = $observedSessionId } + $finalText = [string]$parsedEvents.FinalText + $turnRecords.Add([ordered]@{ sequence = ($turnIndex * 2) + 1; role = 'user'; content_sha256 = Get-Sha256HexFromBytes -Bytes ([System.Text.UTF8Encoding]::new($false).GetBytes($turnText)); session_id = $capturedSessionId; timestamp_utc = Format-UtcTimestamp -Value $process.StartedUtc }) + $turnRecords.Add([ordered]@{ sequence = ($turnIndex * 2) + 2; role = 'assistant'; text = $finalText; session_id = $capturedSessionId; timestamp_utc = Format-UtcTimestamp -Value $process.FinishedUtc }) + } + + if ($null -eq $firstProcess) { $firstProcess = [pscustomobject]@{ StartedUtc = $started; FinishedUtc = [DateTime]::UtcNow; DurationSeconds = ([DateTime]::UtcNow - $started).TotalSeconds; ExitCode = $null; TimedOut = $false } } + if ($null -eq $lastProcess) { $lastProcess = $firstProcess } + $combinedStdout = [string]::Join('', @($rawStdout.ToArray())) + $combinedStderr = [string]::Join('', @($rawStderr.ToArray())) + if (-not [string]::IsNullOrEmpty($combinedStdout) -and -not $combinedStdout.EndsWith("`n", [StringComparison]::Ordinal)) { $combinedStdout += [Environment]::NewLine } + if (-not [string]::IsNullOrEmpty($combinedStderr) -and -not $combinedStderr.EndsWith("`n", [StringComparison]::Ordinal)) { $combinedStderr += [Environment]::NewLine } + $stdoutArtifact = Write-OpenCodeCapture -RunData $Inputs -RelativePath 'evidence/opencode-events.jsonl' -Text $combinedStdout + $stderrArtifact = Write-OpenCodeCapture -RunData $Inputs -RelativePath 'evidence/opencode-stderr.txt' -Text $combinedStderr + $artifacts.Add($stdoutArtifact) + $artifacts.Add($stderrArtifact) + if ($nativeFailures.Count -eq 0 -and $turnRecords.Count -ne ($requestedTurns.Count * 2)) { + $nativeFailures.Add('turn_order') + $status = 'incompatible' + $failureCode = 'native_interaction_incompatible' + $failureMessage = 'OpenCode scripted interaction did not complete every ordered user/assistant turn.' + } + if ($status -eq 'completed' -and $nativeFailures.Count -gt 0) { $status = 'incompatible' } + if ([string]::IsNullOrWhiteSpace($capturedSessionId)) { $capturedSessionId = [Guid]::NewGuid().ToString('D') } + $finished = $lastProcess.FinishedUtc + $durationSeconds = [Math]::Round(($finished - $firstProcess.StartedUtc).TotalSeconds, 3) + $nativeCliProcessTotalSeconds = [Math]::Round(([double](@($turnTimingRecords | ForEach-Object { [double]$_.process_duration_seconds } | Measure-Object -Sum).Sum)), 3) + $tokenMetric = if ($usageBuckets.Count -eq 0) { New-UnavailableMetric -Reason 'opencode_did_not_expose_usage' } else { New-AvailableMetric -Value $usageBuckets } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/opencode-events.jsonl'; complete = $nativeFailures.Count -eq 0 }) + tokens = $tokenMetric + tool_calls = New-AvailableMetric -Value $toolCalls + cost = if ($usageBuckets.Contains('cost')) { New-AvailableMetric -Value $usageBuckets['cost'] } else { New-UnavailableMetric -Reason 'opencode_did_not_expose_cost' } + } + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$Inputs.Profile.Model) + $credentialNames = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + $credentialEvidence = [ordered]@{ + model_provider = $modelProvider + provider_environment_variables = $credentialNames + unrelated_environment_excluded = $true + child_tool_visibility = 'provider_credential_may_be_visible_to_native_child_tools; no supported child filter is exposed' + value_observed = $false + } + $observedModel = if ($observedModels.Count -eq 0) { [string]$Inputs.Profile.Model } else { [string]$observedModels[$observedModels.Count - 1] } + $transcriptArtifactPath = 'evidence/opencode-events.jsonl' + $transcriptArtifact = @($artifacts | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'path' -Default '') -eq $transcriptArtifactPath } | Select-Object -First 1) + $terminalCapture = $nativeFailures.Count -eq 0 -and $turnRecords.Count -eq ($requestedTurns.Count * 2) -and $rawStdout.Count -eq $requestedTurns.Count + $executionPaths = New-OpenCodeExecutionPaths -LogicalInputs $Inputs -ExecutionInputs $executionInputs -Projection $projection -Environment $environment -HomeIsolation $homeIsolationObservation + $candidateSkillExposure = New-OpenCodeCandidateSkillExposure -LogicalInputs $Inputs -Projection $projection -SkillIsolation $skillIsolationObservation + $interactionEvidence = [ordered]@{ + schema = (Get-RunnerSchemaNames).Interaction + mode = 'scripted' + same_session = [bool]$terminalCapture + session_id = $capturedSessionId + turns = @($turnRecords.ToArray()) + final_response_sequence = @($turnRecords).Count + transport = 'opencode-run-explicit-session-continuation' + exact_session_flag = [string]$continuationCapability.Flag + implicit_continuation = $false + native_turns = @($nativeTurns.ToArray()) + structured_transcript_complete = [bool]$terminalCapture + working_directory = [string]$projection.PhysicalWorkingDirectory + isolated_home = [string]$projection.PhysicalHomeDirectory + logical_working_directory = [string]$Inputs.Run.WorkingDirectoryPath + logical_isolated_home = [string]$Inputs.Run.HomeDirectoryPath + config_directory = [string]$environment['OPENCODE_CONFIG_DIR'] + config_file = [string]$environment['OPENCODE_CONFIG'] + model = [string]$Inputs.Profile.Model + } + $evidence = [ordered]@{ + execution_paths = $executionPaths + event_counts = $eventCounts + observed_model = $observedModel + observed_models = @($observedModels.ToArray()) + prompt_delivery = 'stdin' + prompt_first_input = $true + resume = $true + exact_session_continuation = [ordered]@{ + flag = [string]$continuationCapability.Flag + argument_style = [string]$continuationCapability.ArgumentStyle + exact_session_id = $capturedSessionId + implicit_last_session = $false + turns_started_after_prior_terminal = [bool]$turnTwoSentAfterTurnOneTerminal + } + stdout_exit_codes = @($nativeTurns.ToArray() | ForEach-Object { Get-JsonProperty -Object $_ -Name 'exit_code' -Default $null }) + model_argument = [string]$Inputs.Profile.Model + sandbox = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + project_configuration = 'repository_owned_project_config_preserved' + disable_project_config_environment = $false + credential = $credentialEvidence + interaction = $interactionEvidence + future_turn_secrecy = [ordered]@{ + stable_canary = $futureCanary + interaction_json_projected = [bool]$interactionPhysicalPresent + future_source_files_projected = [bool]$futureSourcePhysicalPresent + canary_in_physical_projection = [bool]$canaryInProjectionBeforeTurn1 + canary_in_environment = [bool]$canaryInEnvironment + canary_in_arguments = [bool]$turnOneArgumentsContainFutureCanary + turn_1_input_contains_future_canary = [bool]$turnOneInputContainsFutureCanary + turn_2_sent_only_after_turn_1_terminal_completed = [bool]$turnTwoSentAfterTurnOneTerminal + } + candidate_skill_exposure = $candidateSkillExposure + effective_home = New-OpenCodeHomeIsolationEvidence -Observation $homeIsolationObservation + skill_policy = $policyObservation + skill_isolation = New-OpenCodeSkillIsolationEvidence -Observation $skillIsolationObservation + ambient_skill_policy = [ordered]@{ + mechanism = [string]$policyObservation.mechanism + permission_skill = $policyObservation.configured_permission_skill + external_skill_scans_disabled = [bool]$policyObservation.external_skill_scans_disabled + claude_code_skill_scans_disabled = [bool]$policyObservation.claude_code_skill_scans_disabled + ambient_skill_roots_hidden = [int]$skillIsolationObservation.AmbientSkillCount -eq 0 + candidate_skill_exposed = [bool]$executionInputs.Run.CandidateSkillExposed + candidate_skill_physical_path = if ([bool]$executionInputs.Run.CandidateSkillExposed) { [string]$projection.PhysicalSkillDirectory } else { $null } + } + timing = [ordered]@{ + preflight = Get-JsonProperty -Object $Preflight -Name 'timing' -Default $null + preflight_source = $PreflightSource + projection_setup_duration_seconds = [double]$projection.SetupDurationSeconds + native_cli_process_total_seconds = $nativeCliProcessTotalSeconds + turns = @($turnTimingRecords.ToArray()) + total_runner_execution_seconds = [double]$durationSeconds + } + capture = [ordered]@{ + source = 'harness_native_transport' + terminal = [bool]$terminalCapture + worker_authored = $false + artifact = $transcriptArtifactPath + sha256 = if ($transcriptArtifact.Count -eq 1) { [string](Get-JsonProperty -Object $transcriptArtifact[0] -Name 'sha256' -Default $null) } else { $null } + complete_structured_transcript = [bool]$terminalCapture + turn_artifacts = @($nativeTurns.ToArray() | ForEach-Object { "evidence/opencode-turn-$(Get-JsonProperty -Object $_ -Name 'turn' -Default 0)-events.jsonl" }) + } + delegation = [ordered]@{ + dispatch_owner = 'runner' + mechanism = [string]$descriptor.delegation.mechanism + worker_session_id = $capturedSessionId + observed_model = $observedModel + observed_working_directory = [string]$projection.PhysicalWorkingDirectory + observed_home = [string]$projection.PhysicalHomeDirectory + effective_runtime_home = [string]$homeIsolationObservation.RuntimeHome + effective_opencode_config_root = [string]$environment['OPENCODE_CONFIG_DIR'] + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$Inputs.Run.PromptHash + terminal_result_capture = [bool]$terminalCapture + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + same_session_continuation = [bool]$terminalCapture + continuation_flag = [string]$continuationCapability.Flag + continuation_session_id = $capturedSessionId + } + native_worker_evidence_failures = @($nativeFailures | Select-Object -Unique) + } + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('runner-owned fresh OpenCode process for turn 1', 'opencode run --format json structured event capture', 'prompt on stdin', '--model on every turn', '--dir on every turn', 'isolated OPENCODE_CONFIG_DIR and OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'coherent Windows HOME/USERPROFILE/HOMEDRIVE/HOMEPATH', 'OPENCODE_DISABLE_EXTERNAL_SKILLS=1', 'OPENCODE_DISABLE_CLAUDE_CODE_SKILLS=1', 'permission.skill arm policy', 'same isolated environment on every turn', 'repository-owned project configuration preserved', 'no implicit last-session continuation')) { $mechanisms.Add($mechanism) } + $mechanisms.Add(("explicit OpenCode {0} continuation selected from installed help" -f $continuationCapability.Flag)) + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + $failureCodeValue = if ([string]::IsNullOrWhiteSpace($failureCode)) { 'native_interaction_incompatible' } else { $failureCode } + $failureMessageValue = if ([string]::IsNullOrWhiteSpace($failureMessage)) { 'OpenCode scripted interaction failed closed.' } else { $failureMessage } + $failure = if ($nativeFailures.Count -eq 0) { $null } else { New-ExecutionFailure -Code $failureCodeValue -Message $failureMessageValue } + $exitStatus = if ($status -eq 'completed') { [Nullable[int]]0 } else { $null } + $resultFinalResponse = if ($status -eq 'completed') { $finalText } else { $null } + $resultFinalResponseReason = if ($status -eq 'completed') { $null } else { 'native_interaction_incompatible' } + $result = New-ExecutionResult -Descriptor $ExecutionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $resultFinalResponse -FinalResponseReason $resultFinalResponseReason -StartedUtc $firstProcess.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $durationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $capturedSessionId -IsolationCapabilities (Get-OpenCodeCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem -ContinuationCapability $continuationCapability) -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'OpenCode accepted the requested model selector and configuration; scripted turns retained the exact requested model on every invocation.'; observations = [ordered]@{ model = $Inputs.Profile.Model; observed_models = @($observedModels.ToArray()); continuation_flag = $continuationCapability.Flag } }) -Telemetry $telemetry -Artifacts @($artifacts.ToArray()) -Warnings @($warnings.ToArray()) -Evidence $evidence -AttemptCount 1 + if ($status -eq 'completed') { [void](Assert-InteractionResultEvidence -ExecutionResult $result -RunData $Inputs.Run) } + return $result + } finally { + if ($null -ne $projection) { + try { + Remove-OpenCodeProjectedCandidateSkill -Projection $projection + Sync-OpenCodeProjectedRepository -Projection $projection + } finally { + Remove-OpenCodeExecutionProjection -Projection $projection + if ($null -ne $result -and $null -ne $result.evidence -and $null -ne $result.evidence.execution_paths) { + $result.evidence.execution_paths.projection_cleanup = 'removed' + } + if ($null -ne $result -and $null -ne $result.evidence -and $null -ne $result.evidence.timing) { + $cleanupFinished = [DateTime]::UtcNow + $result.evidence.timing.projection_cleanup_duration_seconds = [Math]::Round(($cleanupFinished - $finished).TotalSeconds, 3) + $result.evidence.timing.total_runner_execution_seconds = [Math]::Round(($cleanupFinished - $started).TotalSeconds, 3) + } + } + } + } +} + +function Get-OpenCodeEventSessionIds { + param([Parameter(Mandatory = $true)][object]$Event) + + $part = Get-JsonProperty -Object $Event -Name 'part' -Default $null + $values = [System.Collections.Generic.List[string]]::new() + foreach ($value in @( + (Get-JsonProperty -Object $Event -Name 'sessionID' -Default $null), + (Get-JsonProperty -Object $Event -Name 'sessionId' -Default $null), + (Get-JsonProperty -Object $Event -Name 'session_id' -Default $null), + (Get-JsonProperty -Object $part -Name 'sessionID' -Default $null), + (Get-JsonProperty -Object $part -Name 'sessionId' -Default $null), + (Get-JsonProperty -Object $part -Name 'session_id' -Default $null) + )) { + if (-not [string]::IsNullOrWhiteSpace([string]$value) -and $values -notcontains [string]$value) { $values.Add([string]$value) } + } + return @($values.ToArray()) +} + +function Get-OpenCodeEventModels { + param([Parameter(Mandatory = $true)][object]$Event) + + $part = Get-JsonProperty -Object $Event -Name 'part' -Default $null + $values = [System.Collections.Generic.List[string]]::new() + foreach ($source in @($Event, $part)) { + $provider = [string](Get-JsonProperty -Object $source -Name 'providerID' -Default (Get-JsonProperty -Object $source -Name 'providerId' -Default (Get-JsonProperty -Object $source -Name 'provider_id' -Default ''))) + $modelId = [string](Get-JsonProperty -Object $source -Name 'modelID' -Default (Get-JsonProperty -Object $source -Name 'modelId' -Default (Get-JsonProperty -Object $source -Name 'model_id' -Default ''))) + if (-not [string]::IsNullOrWhiteSpace($provider) -and -not [string]::IsNullOrWhiteSpace($modelId)) { + $combined = "$provider/$modelId" + if ($values -notcontains $combined) { $values.Add($combined) } + } + } + foreach ($value in @( + (Get-JsonProperty -Object $Event -Name 'model' -Default $null), + (Get-JsonProperty -Object $Event -Name 'modelID' -Default $null), + (Get-JsonProperty -Object $Event -Name 'modelId' -Default $null), + (Get-JsonProperty -Object $Event -Name 'model_id' -Default $null), + (Get-JsonProperty -Object $part -Name 'model' -Default $null), + (Get-JsonProperty -Object $part -Name 'modelID' -Default $null), + (Get-JsonProperty -Object $part -Name 'modelId' -Default $null), + (Get-JsonProperty -Object $part -Name 'model_id' -Default $null) + )) { + $modelValue = [string]$value + if (-not [string]::IsNullOrWhiteSpace($modelValue) -and $values -notcontains $modelValue -and @($values | Where-Object { [string]$_ -like "*/$modelValue" }).Count -eq 0) { $values.Add($modelValue) } + } + return @($values.ToArray()) +} + +function Get-OpenCodeAuthVariable { + param([Parameter(Mandatory = $true)][string]$Provider) + + $variables = @(Get-ProviderAuthenticationVariables -Provider $Provider) + foreach ($name in $variables) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { + return $name + } + } + return $null +} + +function Get-OpenCodeModelProvider { + param([string]$Model) + + if ([string]::IsNullOrWhiteSpace($Model) -or $Model -notmatch '/') { + return $null + } + $parts = $Model.Split([char[]]@('/'), 2, [System.StringSplitOptions]::None) + if ($parts.Count -lt 2 -or [string]::IsNullOrWhiteSpace($parts[0]) -or [string]::IsNullOrWhiteSpace($parts[1])) { + return $null + } + return $parts[0] +} + +function Resolve-SandboxCommand { + param([Parameter(Mandatory = $true)][string]$Name) + + return Resolve-ExternalCommand -Name $Name +} + +function Get-OpenCodeDescriptor { + $copy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $copy[$key] = $descriptor[$key] } + $commandInfo = Resolve-ExternalCommand -Name 'opencode' + $version = 'unavailable' + if ($null -ne $commandInfo) { + $observation = Get-ExternalCommandVersion -CommandInfo $commandInfo + $version = [string]$observation.Version + } + $copy.harness = [ordered]@{ name = 'OpenCode CLI'; version = $version } + return $copy +} + +function New-OpenCodeCliArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform = (Get-PlatformName) + ) + $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $arguments = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('run', '--format', 'json', '--dir', $directoryArgument, '--model', $Inputs.Profile.Model, '--auto')) { + $arguments.Add([string]$argument) + } + if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { + $arguments.Add('--variant') + $arguments.Add([string]$Inputs.Profile.ReasoningEffort) + } + return @($arguments) +} + +function Get-OpenCodeSourceRepositoryRoot { + param([Parameter(Mandatory = $true)][string]$RunRoot) + + # Prepared packages normally live below the source checkout, so discover + # that checkout from the logical run first. The adapter fallback protects + # the same invariant when a package is relocated outside the checkout: + # the runner itself still identifies the repository whose ancestry must not + # become an OpenCode execution root. + foreach ($startPath in @($RunRoot, $PSScriptRoot)) { + if ([string]::IsNullOrWhiteSpace([string]$startPath)) { continue } + $current = [System.IO.Path]::GetFullPath([string]$startPath) + while ($true) { + $gitMarker = Join-Path $current '.git' + if (Test-Path -LiteralPath $gitMarker) { + return $current + } + $parent = Split-Path -Parent $current + if ([string]::IsNullOrWhiteSpace($parent) -or [string]::Equals($parent, $current, [System.StringComparison]::OrdinalIgnoreCase)) { + break + } + $current = $parent + } + } + return $null +} + +function Get-OpenCodeProjectionBaseDirectory { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $sourceRepositoryRoot = Get-OpenCodeSourceRepositoryRoot -RunRoot $Inputs.Run.RunRoot + $candidates = [System.Collections.Generic.List[string]]::new() + $candidates.Add([System.IO.Path]::GetFullPath([System.IO.Path]::GetTempPath())) + foreach ($specialFolder in @([System.Environment+SpecialFolder]::LocalApplicationData, [System.Environment+SpecialFolder]::CommonApplicationData)) { + $path = [Environment]::GetFolderPath($specialFolder) + if (-not [string]::IsNullOrWhiteSpace($path)) { $candidates.Add([System.IO.Path]::GetFullPath($path)) } + } + + foreach ($candidate in @($candidates | Select-Object -Unique)) { + if ($null -ne $sourceRepositoryRoot -and (Test-PathInside -BasePath $sourceRepositoryRoot -CandidatePath $candidate)) { + continue + } + if (Test-PathInside -BasePath $Inputs.Run.RunRoot -CandidatePath $candidate) { + continue + } + return [pscustomobject]@{ + Path = $candidate + SourceRepositoryRoot = $sourceRepositoryRoot + } + } + + throw 'OpenCode could not select a physical projection parent outside the prepared run and its source-repository ancestry.' +} + +function Get-OpenCodeProjectionPlan { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $base = Get-OpenCodeProjectionBaseDirectory -Inputs $Inputs + $root = [System.IO.Path]::GetFullPath((Join-Path $base.Path ('agentic-opencode-projection-' + [Guid]::NewGuid().ToString('N')))) + if ($null -ne $base.SourceRepositoryRoot -and (Test-PathInside -BasePath $base.SourceRepositoryRoot -CandidatePath $root)) { + throw 'OpenCode physical projection unexpectedly resolved under the source repository ancestry.' + } + if (Test-PathInside -BasePath $Inputs.Run.RunRoot -CandidatePath $root) { + throw 'OpenCode physical projection unexpectedly resolved under the logical arm root.' + } + return [pscustomobject]@{ + Root = $root + Parent = $base.Path + SourceRepositoryRoot = $base.SourceRepositoryRoot + } +} + +function Assert-OpenCodeProjectionSource { + param([Parameter(Mandatory = $true)][string]$Path) + + if (-not (Test-Path -LiteralPath $Path -PathType Container)) { return } + $reparsePoint = [System.IO.FileAttributes]::ReparsePoint + $rootItem = Get-Item -LiteralPath $Path -Force -ErrorAction Stop + if (($rootItem.Attributes -band $reparsePoint) -ne 0) { + throw "OpenCode physical projection refuses reparse-point input '$Path'." + } + $links = @(Get-ChildItem -LiteralPath $Path -Recurse -Force -ErrorAction Stop | Where-Object { ($_.Attributes -band $reparsePoint) -ne 0 }) + if ($links.Count -gt 0) { + throw "OpenCode physical projection refuses reparse-point input '$($links[0].FullName)'." + } +} + +function Copy-OpenCodeProjectionDirectory { + param( + [Parameter(Mandatory = $true)][string]$Source, + [Parameter(Mandatory = $true)][string]$Destination, + [string[]]$ExcludeRelativePaths = @() + ) + + New-Item -ItemType Directory -Path $Destination -Force | Out-Null + if (-not (Test-Path -LiteralPath $Source -PathType Container)) { return } + Assert-OpenCodeProjectionSource -Path $Source + $excluded = @($ExcludeRelativePaths | ForEach-Object { ([string]$_).Replace('\', '/').Trim('/') } | Where-Object { $_ }) + foreach ($item in @(Get-ChildItem -LiteralPath $Source -Force -ErrorAction Stop)) { + $relative = [System.IO.Path]::GetRelativePath($Source, $item.FullName).Replace('\', '/') + $isExcluded = @($excluded | Where-Object { + $_ -eq $relative -or $relative.StartsWith($_ + '/', [System.StringComparison]::OrdinalIgnoreCase) + }).Count -gt 0 + if ($isExcluded) { continue } + $destinationPath = Join-Path $Destination ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if ($item.PSIsContainer) { + Copy-OpenCodeProjectionDirectory -Source $item.FullName -Destination $destinationPath -ExcludeRelativePaths @($excluded | ForEach-Object { + if ($_.StartsWith($relative + '/', [System.StringComparison]::OrdinalIgnoreCase)) { $_.Substring($relative.Length + 1) } + }) + } else { + New-Item -ItemType Directory -Path (Split-Path -Parent $destinationPath) -Force | Out-Null + Copy-Item -LiteralPath $item.FullName -Destination $destinationPath -Force + } + } +} + +function Get-OpenCodeProjectionFileSet { + param([Parameter(Mandatory = $true)][string]$Root) + + if (-not (Test-Path -LiteralPath $Root -PathType Container)) { return @() } + return @(Get-ChildItem -LiteralPath $Root -Recurse -File -Force | ForEach-Object { + [System.IO.Path]::GetRelativePath($Root, $_.FullName).Replace('\', '/') + } | Sort-Object) +} + +function Get-OpenCodeTreeHash { + param([Parameter(Mandatory = $true)][string]$Root) + + if (-not (Test-Path -LiteralPath $Root -PathType Container)) { return $null } + $entries = [System.Collections.Generic.List[string]]::new() + foreach ($file in @(Get-ChildItem -LiteralPath $Root -Recurse -File -Force | Sort-Object FullName)) { + $relative = [System.IO.Path]::GetRelativePath($Root, $file.FullName).Replace('\', '/') + $segments = $relative.Split('/') + if ($relative.StartsWith('evals/', [System.StringComparison]::OrdinalIgnoreCase) -or + @($segments | Where-Object { $_ -in @('bin', 'obj', '__pycache__') }).Count -gt 0) { + continue + } + $entries.Add("$relative`:$((Get-Sha256HexFromFile -Path $file.FullName))") + } + $joined = [string]::Join("`n", @($entries | Sort-Object)) + return Get-Sha256HexFromBytes -Bytes ([System.Text.UTF8Encoding]::new($false).GetBytes($joined)) +} + +function New-OpenCodeExecutionProjection { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $projectionStarted = [DateTime]::UtcNow + $plan = Get-OpenCodeProjectionPlan -Inputs $Inputs + New-Item -ItemType Directory -Path $plan.Root -Force | Out-Null + try { + $physicalPrompt = Join-Path $plan.Root 'prompt.md' + [System.IO.File]::WriteAllBytes($physicalPrompt, [byte[]]$Inputs.Run.PromptBytes) + $physicalRepo = Join-Path $plan.Root 'repo' + $physicalHome = Join-Path $plan.Root 'home' + + # Scripted interaction inputs are parent-owned data. They are read into + # memory by the runner immediately before execution and are never made + # part of the physical OpenCode filesystem projection. This prevents a + # later turn (or its source file) from being discovered early. + $excludedRepositoryFiles = [System.Collections.Generic.List[string]]::new() + $excludedHomeFiles = [System.Collections.Generic.List[string]]::new() + $excludedLogicalInteractionFiles = [System.Collections.Generic.List[string]]::new() + if ($null -ne $Inputs.Run.InteractionPath) { + $excludedLogicalInteractionFiles.Add([string]$Inputs.Run.InteractionPath) + } + if ($null -ne $Inputs.Run.Interaction) { + foreach ($turn in @($Inputs.Run.Interaction.turns)) { + $source = [string](Get-JsonProperty -Object $turn -Name 'source' -Default '') + if ([string]::IsNullOrWhiteSpace($source)) { continue } + $excludedLogicalInteractionFiles.Add((Resolve-ContainedPath -BasePath $Inputs.Run.RunRoot -RelativePath $source -FieldName 'interaction turn source' -Kind File)) + } + } + foreach ($logicalInteractionFile in @($excludedLogicalInteractionFiles.ToArray() | Select-Object -Unique)) { + if (Test-PathInside -BasePath $Inputs.Run.WorkingDirectoryPath -CandidatePath $logicalInteractionFile) { + $excludedRepositoryFiles.Add([System.IO.Path]::GetRelativePath($Inputs.Run.WorkingDirectoryPath, $logicalInteractionFile).Replace('\', '/')) + } elseif (Test-PathInside -BasePath $Inputs.Run.HomeDirectoryPath -CandidatePath $logicalInteractionFile) { + $excludedHomeFiles.Add([System.IO.Path]::GetRelativePath($Inputs.Run.HomeDirectoryPath, $logicalInteractionFile).Replace('\', '/')) + } + } + Copy-OpenCodeProjectionDirectory -Source $Inputs.Run.WorkingDirectoryPath -Destination $physicalRepo -ExcludeRelativePaths @($excludedRepositoryFiles.ToArray()) + Copy-OpenCodeProjectionDirectory -Source $Inputs.Run.HomeDirectoryPath -Destination $physicalHome -ExcludeRelativePaths @($excludedHomeFiles.ToArray()) + + $physicalSkill = $null + $physicalSkillHash = $null + $runnerInjectedRepositoryFiles = [System.Collections.Generic.List[string]]::new() + if ($Inputs.Run.CandidateSkillExposed) { + $skillName = Get-OpenCodeCandidateSkillName -Run $Inputs.Run + # OpenCode's installed native discovery surface is the project-local + # .opencode/skills//SKILL.md root. The source is still the + # prepared run's staged skill directory; only the projection target + # changes so the candidate is discoverable by OpenCode itself. + $physicalSkill = Join-Path (Join-Path (Join-Path $physicalRepo '.opencode') 'skills') $skillName + if (Test-Path -LiteralPath $physicalSkill) { + throw "OpenCode physical repository already contains a native candidate skill at '$physicalSkill'." + } + Copy-OpenCodeProjectionDirectory -Source $Inputs.Run.SkillDirectoryPath -Destination $physicalSkill + $physicalSkillHash = Get-OpenCodeTreeHash -Root $physicalSkill + if ($physicalSkillHash -ne [string]$Inputs.Run.SkillHash) { + throw 'OpenCode physical candidate skill hash does not match the prepared run manifest.' + } + foreach ($relative in @(Get-OpenCodeProjectionFileSet -Root $physicalSkill)) { + $runnerInjectedRepositoryFiles.Add(('.opencode/skills/{0}' -f $skillName) + '/' + $relative) + } + } + + $physicalRun = [pscustomobject]@{ + RunPath = $Inputs.Run.RunPath + RunRoot = $plan.Root + Contract = $Inputs.Run.Contract + EvalId = $Inputs.Run.EvalId + EvalName = $Inputs.Run.EvalName + Mode = $Inputs.Run.Mode + PromptPath = $physicalPrompt + PromptBytes = $Inputs.Run.PromptBytes + PromptHash = $Inputs.Run.PromptHash + WorkingDirectoryPath = $physicalRepo + HomeDirectoryPath = $physicalHome + SkillDirectoryPath = $physicalSkill + CandidateSkillExposed = $Inputs.Run.CandidateSkillExposed + FixtureHash = $Inputs.Run.FixtureHash + SkillHash = $Inputs.Run.SkillHash + InteractionPath = $null + InteractionHash = $Inputs.Run.InteractionHash + Interaction = $Inputs.Run.Interaction + } + $projectionFinished = [DateTime]::UtcNow + return [pscustomobject]@{ + Root = $plan.Root + Parent = $plan.Parent + SourceRepositoryRoot = $plan.SourceRepositoryRoot + Run = $physicalRun + Inputs = [pscustomobject]@{ Run = $physicalRun; Profile = $Inputs.Profile } + LogicalRun = $Inputs.Run + LogicalWorkingDirectory = $Inputs.Run.WorkingDirectoryPath + LogicalHomeDirectory = $Inputs.Run.HomeDirectoryPath + PhysicalWorkingDirectory = $physicalRepo + PhysicalHomeDirectory = $physicalHome + PhysicalSkillDirectory = $physicalSkill + PhysicalSkillHash = $physicalSkillHash + LogicalRepositoryFiles = @(Get-OpenCodeProjectionFileSet -Root $Inputs.Run.WorkingDirectoryPath) + LogicalRepositoryHash = Get-OpenCodeTreeHash -Root $Inputs.Run.WorkingDirectoryPath + ExcludedRepositoryFiles = @($excludedRepositoryFiles.ToArray()) + ExcludedHomeFiles = @($excludedHomeFiles.ToArray()) + InitialRepositoryFiles = @(Get-OpenCodeProjectionFileSet -Root $physicalRepo) + InitialRepositoryHash = Get-OpenCodeTreeHash -Root $physicalRepo + RunnerInjectedRepositoryFiles = @($runnerInjectedRepositoryFiles.ToArray()) + RuntimeCreatedRepositoryFiles = @() + SetupStartedUtc = $projectionStarted + SetupFinishedUtc = $projectionFinished + SetupDurationSeconds = [Math]::Round(($projectionFinished - $projectionStarted).TotalSeconds, 3) + Proven = $true + } + } catch { + if (Test-Path -LiteralPath $plan.Root) { Remove-Item -LiteralPath $plan.Root -Recurse -Force -ErrorAction SilentlyContinue } + throw + } +} + +function Sync-OpenCodeProjectedRepository { + param([Parameter(Mandatory = $true)][object]$Projection) + + $logicalRepo = [string]$Projection.LogicalWorkingDirectory + $physicalRepo = [string]$Projection.PhysicalWorkingDirectory + Assert-OpenCodeProjectionSource -Path $physicalRepo + $initialLogical = @($Projection.LogicalRepositoryFiles | ForEach-Object { ([string]$_).Replace('\', '/') }) + $excluded = @($Projection.ExcludedRepositoryFiles | ForEach-Object { ([string]$_).Replace('\', '/') }) + $injected = @($Projection.RunnerInjectedRepositoryFiles | ForEach-Object { ([string]$_).Replace('\', '/') }) + + $isExcluded = { + param([string]$Relative) + @($excluded | Where-Object { $_ -eq $Relative }).Count -gt 0 + } + $isInjected = { + param([string]$Relative) + @($injected | Where-Object { $_ -eq $Relative -or $Relative.StartsWith($_ + '/', [System.StringComparison]::OrdinalIgnoreCase) }).Count -gt 0 + } + $isKnownRuntimeOnly = { + param([string]$Relative) + if ($Relative.StartsWith('.opencode/node_modules/', [System.StringComparison]::OrdinalIgnoreCase) -and $initialLogical -notcontains $Relative) { return $true } + if ($Relative -eq '.opencode/.gitignore' -and $initialLogical -notcontains $Relative) { return $true } + return $false + } + $copyFile = { + param([string]$Relative) + $normalized = $Relative.Replace('\', '/') + if ((& $isExcluded $normalized) -or (& $isInjected $normalized) -or (& $isKnownRuntimeOnly $normalized)) { return } + $physicalPath = Join-Path $physicalRepo ($normalized -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if (-not (Test-Path -LiteralPath $physicalPath -PathType Leaf)) { return } + $logicalPath = Join-Path $logicalRepo ($normalized -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $logicalPath) -Force | Out-Null + Copy-Item -LiteralPath $physicalPath -Destination $logicalPath -Force + } + + # Deletes are limited to the original logical repository set. Interaction + # source files are deliberately excluded from the physical projection and + # therefore can never be treated as task deletions. + foreach ($relative in $initialLogical) { + if (& $isExcluded $relative) { continue } + $physicalPath = Join-Path $physicalRepo ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if (-not (Test-Path -LiteralPath $physicalPath -PathType Leaf)) { + $logicalPath = Join-Path $logicalRepo ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if (Test-Path -LiteralPath $logicalPath -PathType Leaf) { Remove-Item -LiteralPath $logicalPath -Force } + } + } + + # Copy both changed original files and newly created task outputs. Candidate + # skill files and known OpenCode runtime-only files are never copied back. + foreach ($relative in @(Get-OpenCodeProjectionFileSet -Root $physicalRepo)) { + & $copyFile $relative + } +} + +function Remove-OpenCodeProjectedCandidateSkill { + param([Parameter(Mandatory = $true)][object]$Projection) + + $candidatePath = [string]$Projection.PhysicalSkillDirectory + if ([string]::IsNullOrWhiteSpace($candidatePath)) { return } + $physicalRepo = [System.IO.Path]::GetFullPath([string]$Projection.PhysicalWorkingDirectory) + $candidateFull = [System.IO.Path]::GetFullPath($candidatePath) + if (-not (Test-PathInside -BasePath $physicalRepo -CandidatePath $candidateFull)) { + throw "Refusing to remove an OpenCode projected candidate skill outside the physical repository: '$candidateFull'." + } + $relative = [System.IO.Path]::GetRelativePath($physicalRepo, $candidateFull).Replace('\', '/') + if ($relative -notmatch '(?i)^\.opencode/skills/[^/]+$') { + throw "Refusing to remove an OpenCode projected candidate skill outside the native project skill root: '$relative'." + } + if (Test-Path -LiteralPath $candidateFull) { Remove-Item -LiteralPath $candidateFull -Recurse -Force } + $nativeSkillsRoot = Split-Path -Parent $candidateFull + $nativeProjectRoot = Split-Path -Parent $nativeSkillsRoot + foreach ($emptyRoot in @($nativeSkillsRoot, $nativeProjectRoot)) { + if ((Test-Path -LiteralPath $emptyRoot -PathType Container) -and @((Get-ChildItem -LiteralPath $emptyRoot -Force -ErrorAction SilentlyContinue)).Count -eq 0) { + Remove-Item -LiteralPath $emptyRoot -Force + } + } +} + +function Remove-OpenCodeExecutionProjection { + param([Parameter(Mandatory = $true)][object]$Projection) + + $root = [System.IO.Path]::GetFullPath([string]$Projection.Root) + $parent = [System.IO.Path]::GetFullPath([string]$Projection.Parent) + if (-not (Test-PathInside -BasePath $parent -CandidatePath $root) -or + [System.IO.Path]::GetFileName($root) -notmatch '^agentic-opencode-projection-[0-9a-f-]+$') { + throw "Refusing to remove an OpenCode projection outside its validated temporary parent: '$root'." + } + if (Test-Path -LiteralPath $root) { Remove-Item -LiteralPath $root -Recurse -Force } +} + +function Get-OpenCodeCommandFingerprint { + param([Parameter(Mandatory = $true)][object]$CommandInfo) + + try { + $source = (Resolve-Path -LiteralPath ([string]$CommandInfo.Source) -ErrorAction Stop).Path + if (-not (Test-Path -LiteralPath $source -PathType Leaf)) { return $null } + return [ordered]@{ + source = $source + sha256 = Get-Sha256HexFromFile -Path $source + } + } catch { + return $null + } +} + +function Get-OpenCodePreflightCacheIdentity { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo + ) + + $commandFingerprint = Get-OpenCodeCommandFingerprint -CommandInfo $CommandInfo + if ($null -eq $commandFingerprint) { return $null } + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$Inputs.Profile.Model) + $authVariables = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + $adapterPath = Join-Path $PSScriptRoot 'runner.ps1' + if (-not (Test-Path -LiteralPath $adapterPath -PathType Leaf)) { return $null } + $identity = [ordered]@{ + run_path = [System.IO.Path]::GetFullPath($Inputs.Run.RunPath) + run_json_sha256 = Get-Sha256HexFromFile -Path $Inputs.Run.RunPath + profile_path = [System.IO.Path]::GetFullPath($Inputs.Profile.Path) + profile_sha256 = [string]$Inputs.Profile.Hash + command_source = [string]$commandFingerprint.source + command_sha256 = [string]$commandFingerprint.sha256 + adapter_sha256 = Get-Sha256HexFromFile -Path $adapterPath + auth_variables = @($authVariables | Sort-Object -Unique) + auth_present = @($authVariables | Where-Object { -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($_)) } | Sort-Object -Unique) + } + $keyText = $identity | ConvertTo-Json -Depth 20 -Compress + $key = Get-Sha256HexFromBytes -Bytes ([System.Text.UTF8Encoding]::new($false).GetBytes($keyText)) + $base = Get-OpenCodeProjectionBaseDirectory -Inputs $Inputs + $cacheRoot = Join-Path $base.Path 'agentic-opencode-preflight-cache' + return [pscustomobject]@{ + Identity = $identity + Key = $key + Path = Join-Path $cacheRoot ($key + '.json') + Root = $cacheRoot + } +} + +function Save-OpenCodePreflightObservation { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$Preflight + ) + + if ([string]$Preflight.status -ne 'compatible') { return $null } + $commandInfo = Resolve-ExternalCommand -Name 'opencode' + if ($null -eq $commandInfo) { return $null } + $identity = Get-OpenCodePreflightCacheIdentity -Inputs $Inputs -CommandInfo $commandInfo + if ($null -eq $identity) { return $null } + New-Item -ItemType Directory -Path $identity.Root -Force | Out-Null + $record = [ordered]@{ + schema = 'codebeltnet/agentic/opencode-preflight-observation/1' + created_utc = [DateTime]::UtcNow.ToString('o') + identity = $identity.Identity + harness_version = [string](Get-JsonProperty -Object $Preflight.harness -Name 'version' -Default 'unavailable') + preflight = $Preflight + } + [System.IO.File]::WriteAllText($identity.Path, (($record | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + return $identity.Path +} + +function Get-OpenCodeCachedPreflight { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $commandInfo = Resolve-ExternalCommand -Name 'opencode' + if ($null -eq $commandInfo) { return [pscustomobject]@{ Hit = $false; Preflight = $null; CachePath = $null; Source = 'fresh_preflight' } } + $identity = Get-OpenCodePreflightCacheIdentity -Inputs $Inputs -CommandInfo $commandInfo + if ($null -eq $identity -or -not (Test-Path -LiteralPath $identity.Path -PathType Leaf)) { + return [pscustomobject]@{ Hit = $false; Preflight = $null; CachePath = if ($null -eq $identity) { $null } else { $identity.Path }; Source = 'fresh_preflight' } + } + try { + $record = Read-RunnerJson -Path $identity.Path + $created = [DateTime]::Parse([string]$record.created_utc).ToUniversalTime() + if (([DateTime]::UtcNow - $created).TotalSeconds -gt 1800) { throw 'cached OpenCode preflight observation expired.' } + foreach ($name in @($identity.Identity.Keys)) { + $expected = $identity.Identity[$name] + $actual = Get-JsonProperty -Object $record.identity -Name $name -Default $null + if (($expected | ConvertTo-Json -Depth 20 -Compress) -ne ($actual | ConvertTo-Json -Depth 20 -Compress)) { + throw "cached OpenCode preflight identity mismatch for '$name'." + } + } + $preflight = Get-JsonProperty -Object $record -Name 'preflight' -Default $null + if ($null -eq $preflight -or [string]$preflight.status -ne 'compatible') { throw 'cached OpenCode preflight was not compatible.' } + return [pscustomobject]@{ Hit = $true; Preflight = $preflight; CachePath = $identity.Path; Source = 'authoritative_cached_preflight' } + } catch { + return [pscustomobject]@{ Hit = $false; Preflight = $null; CachePath = $identity.Path; Source = 'fresh_preflight' } + } +} + +function Remove-OpenCodePreflightCache { + param([string]$Path) + + if ([string]::IsNullOrWhiteSpace($Path)) { return } + if (Test-Path -LiteralPath $Path -PathType Leaf) { Remove-Item -LiteralPath $Path -Force -ErrorAction SilentlyContinue } +} + +function Get-OpenCodeEventTiming { + param([AllowEmptyCollection()][string[]]$Timestamps = @()) + + $parsed = [System.Collections.Generic.List[DateTimeOffset]]::new() + foreach ($timestamp in @($Timestamps)) { + $value = [DateTimeOffset]::MinValue + if ([DateTimeOffset]::TryParse([string]$timestamp, [Globalization.CultureInfo]::InvariantCulture, [Globalization.DateTimeStyles]::RoundtripKind, [ref]$value)) { + $parsed.Add($value) + } + } + if ($parsed.Count -eq 0) { return $null } + $first = ($parsed | Measure-Object -Property UtcDateTime -Minimum).Minimum + $last = ($parsed | Measure-Object -Property UtcDateTime -Maximum).Maximum + return [ordered]@{ + first_event_utc = ([DateTime]$first).ToUniversalTime().ToString('o') + last_event_utc = ([DateTime]$last).ToUniversalTime().ToString('o') + structured_event_span_seconds = [Math]::Max(0, [Math]::Round(([DateTime]$last - [DateTime]$first).TotalSeconds, 3)) + } +} + +function Get-OpenCodeCapabilityMap { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [bool]$HardFilesystemConfinement = $false, + [object]$ContinuationCapability = $null + ) + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) + } + $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } + $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + $capabilities['scripted_multi_turn_same_session'] = if ($null -eq $Inputs.Run.Interaction) { + 'conditional' + } elseif ($null -ne $ContinuationCapability -and [bool]$ContinuationCapability.Available) { + 'supported' + } else { + 'unsupported' + } + return $capabilities +} + +function Get-OpenCodePreflight { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $preflightStarted = [DateTime]::UtcNow + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + $platform = Get-PlatformName + $commandInfo = Resolve-ExternalCommand -Name 'opencode' + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $versionObservation = $null + $help = $null + $debugHelp = $null + $debugConfig = $null + $runtimeHomeObservation = $null + $homeIsolationObservation = $null + $policyObservation = $null + $debugConfigObservation = $null + $openCodeEnvironment = $null + $continuationCapability = [pscustomobject]@{ + Available = $false + Flag = $null + ArgumentStyle = $null + Parameter = $null + HelpEvidence = $null + Reason = 'OpenCode exact-session continuation was not probed because no scripted interaction is present.' + } + $projectionPlan = $null + try { + $projectionPlan = Get-OpenCodeProjectionPlan -Inputs $Inputs + $checks.Add((New-PreflightCheck -Name 'physical_projection' -Status passed -Detail 'Behavioral OpenCode execution will use one physical projection outside the prepared run and its source-repository ancestry.')) + } catch { + $reasons.Add("OpenCode physical projection is unavailable: $($_.Exception.Message)") + $checks.Add((New-PreflightCheck -Name 'physical_projection' -Status failed -Detail 'The adapter could not prove a physical execution root outside the source-repository ancestry.')) + } + + if ($profile.Runner -ne 'opencode') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than opencode.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is opencode.')) + } + if ([string]::IsNullOrWhiteSpace($profile.Model)) { + $reasons.Add('OpenCode requires a model in execution-profile.json.') + } else { + $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) + } + if ([int]$profile.Concurrency -lt 2) { + $checks.Add((New-PreflightCheck -Name 'parallel_dispatch' -Status failed -Detail 'OpenCode native-worker evaluations require at least two concurrent worker slots; a serial execution profile is not supported.')) + $reasons.Add('OpenCode native-worker evaluations require execution-profile.json concurrency >= 2. Sequential dispatch is incompatible unless the external harness reports a capacity limit during orchestration.') + } else { + $checks.Add((New-PreflightCheck -Name 'parallel_dispatch' -Status passed -Detail "OpenCode native-worker evaluations require bounded concurrent dispatch; requested slots=$($profile.Concurrency).")) + } + if ($profile.ConfigurationProfile -ne 'isolated-default') { + $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by opencode.") + } + if ($profile.ToolProfile -ne 'default') { + $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by opencode.") + } + if ($null -eq $commandInfo) { + $reasons.Add('The OpenCode CLI executable is not available on PATH.') + if ($null -ne $run.Interaction) { + $checks.Add((New-PreflightCheck -Name 'scripted_multi_turn_same_session' -Status failed -Detail 'The OpenCode executable is unavailable, so exact-session continuation cannot be proven before execution.')) + $reasons.Add('scripted_multi_turn_same_session is incompatible: the OpenCode executable is unavailable and no model-free run --help probe can run.') + } + } else { + $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) + try { + # Build the exact child environment once. Every model-free probe + # and every later OpenCode turn uses this same isolation policy. + $openCodeEnvironment = New-OpenCodeEnvironment -Inputs $Inputs + $versionObservation = Get-ExternalCommandVersion -CommandInfo $commandInfo -WorkingDirectory $run.WorkingDirectoryPath -Environment $openCodeEnvironment -TimeoutSeconds 30 + if (-not $versionObservation.Available) { + $reasons.Add('The OpenCode CLI did not expose an exact observable version through --version.') + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status unavailable -Detail 'opencode --version did not return a usable version string.')) + } else { + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status passed -Detail ([string]$versionObservation.Version))) + } + $runtimeHomeObservation = Get-OpenCodeRuntimeHomeObservation -CommandInfo $commandInfo -Inputs $Inputs -Environment $openCodeEnvironment + $homeIsolationObservation = Get-OpenCodeHomeIsolationObservation -Inputs $Inputs -Environment $openCodeEnvironment -RuntimeHome $runtimeHomeObservation + if ([bool]$homeIsolationObservation.Valid) { + $checks.Add((New-PreflightCheck -Name 'effective_home' -Status passed -Detail ("Node/OpenCode resolved home '{0}' from the isolated child environment." -f $homeIsolationObservation.RuntimeHome))) + } else { + $checks.Add((New-PreflightCheck -Name 'effective_home' -Status failed -Detail $homeIsolationObservation.Reason)) + $reasons.Add('OpenCode effective-home isolation is incompatible: ' + $homeIsolationObservation.Reason) + } + $policyObservation = Get-OpenCodeSkillPolicyObservation -Inputs $Inputs -Environment $openCodeEnvironment + if ([bool]$policyObservation.permission_match -and [bool]$policyObservation.external_skill_scans_disabled -and [bool]$policyObservation.claude_code_skill_scans_disabled) { + $checks.Add((New-PreflightCheck -Name 'skill_isolation_policy' -Status passed -Detail 'OpenCode permission.skill and external-skill disable flags match the requested arm.')) + } else { + $checks.Add((New-PreflightCheck -Name 'skill_isolation_policy' -Status failed -Detail ([string]$policyObservation.reason))) + $reasons.Add('OpenCode skill-isolation policy is incompatible: ' + [string]$policyObservation.reason) + } + $help = Get-OpenCodeHelpResult -CommandInfo $commandInfo -Inputs $Inputs -Environment $openCodeEnvironment + if ($help.TimedOut -or $help.ExitCode -ne 0) { + $helpFailureReason = "OpenCode run --help failed with exit status $($help.ExitCode); single-turn CLI controls cannot be proven." + $reasons.Add($helpFailureReason) + if ($null -ne $run.Interaction) { + $checks.Add((New-PreflightCheck -Name 'scripted_multi_turn_same_session' -Status failed -Detail $helpFailureReason)) + $reasons.Add('scripted_multi_turn_same_session is incompatible: exact-session continuation cannot be proven without run --help.') + } + } else { + $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) + foreach ($flag in @('--format', '--dir', '--model', '--auto')) { + if ($helpText -notmatch [regex]::Escape($flag)) { + $reasons.Add("The installed OpenCode CLI does not advertise required flag '$flag'.") + } + } + if (-not [string]::IsNullOrWhiteSpace([string]$profile.ReasoningEffort) -and $helpText -notmatch [regex]::Escape('--variant')) { + $reasons.Add("execution-profile.json requests reasoning_effort '$($profile.ReasoningEffort)', but the installed OpenCode CLI does not advertise --variant.") + } + $visiblePlatform = if ($platform -eq 'linux' -and $null -ne $sandboxInfo) { 'linux' } else { $platform } + $constructed = New-OpenCodeCliArguments -Inputs $Inputs -VisiblePlatform $visiblePlatform + foreach ($forbidden in @('--pure', '--continue', '--session')) { + if (@($constructed) -contains $forbidden) { $reasons.Add("The constructed OpenCode invocation must not use session or project-suppression option '$forbidden'.") } + } + $debugHelp = Get-OpenCodeDebugHelpResult -CommandInfo $commandInfo -Inputs $Inputs -Environment $openCodeEnvironment + $debugHelpText = [string]::Join("`n", @($debugHelp.Stdout, $debugHelp.Stderr)) + $debugConfigAdvertised = -not $debugHelp.TimedOut -and $debugHelp.ExitCode -eq 0 -and $debugHelpText -match '(?i)(^|\s)config(\s|$)' + if ($debugConfigAdvertised) { + $debugConfig = Get-OpenCodeDebugConfigResult -CommandInfo $commandInfo -Inputs $Inputs -Environment $openCodeEnvironment + $debugConfigObservation = Get-OpenCodeDebugConfigObservation -DebugResult $debugConfig -Inputs $Inputs + if (-not [bool]$debugConfigObservation.available -or -not [bool]$debugConfigObservation.permission_match) { + $checks.Add((New-PreflightCheck -Name 'skill_permission_debug' -Status failed -Detail ([string]$debugConfigObservation.reason))) + $reasons.Add('Installed OpenCode advertises debug config, but its effective permission.skill policy was not proven: ' + [string]$debugConfigObservation.reason) + } else { + $checks.Add((New-PreflightCheck -Name 'skill_permission_debug' -Status passed -Detail 'Installed OpenCode debug config reports the exact permission.skill policy for this arm.')) + } + } else { + $checks.Add((New-PreflightCheck -Name 'skill_permission_debug' -Status not_applicable -Detail 'Installed OpenCode does not advertise a model-free debug config command; filesystem/home isolation remains authoritative.')) + } + if ($reasons.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'OpenCode run advertises noninteractive, model, directory, and structured-output controls; the adapter intentionally does not use --pure.')) + } + if ($null -ne $run.Interaction) { + $continuationProbe = Get-OpenCodeContinuationCapability -HelpText $helpText + $continuationCapability = $continuationProbe + if ([bool]$continuationProbe.Available) { + $checks.Add((New-PreflightCheck -Name 'scripted_multi_turn_same_session' -Status passed -Detail ("OpenCode run --help proves explicit exact-session continuation through {0} {1}; resumed invocations retain --format json, --model, --dir, --auto, and the isolated environment." -f $continuationProbe.Flag, $continuationProbe.Parameter))) + } else { + $checks.Add((New-PreflightCheck -Name 'scripted_multi_turn_same_session' -Status failed -Detail $continuationCapability.Reason)) + $reasons.Add('scripted_multi_turn_same_session is incompatible: ' + $continuationCapability.Reason) + } + } + } + } catch { + $reasons.Add("Could not inspect OpenCode CLI capabilities: $($_.Exception.Message)") + if ($null -ne $run.Interaction -and @($checks | Where-Object { $_.name -eq 'scripted_multi_turn_same_session' }).Count -eq 0) { + $continuationCapability.Reason = 'OpenCode capability inspection failed before exact-session continuation could be proven.' + $checks.Add((New-PreflightCheck -Name 'scripted_multi_turn_same_session' -Status failed -Detail $continuationCapability.Reason)) + } + } + } + + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$profile.Model) + $authVariable = if ([string]::IsNullOrWhiteSpace($modelProvider)) { $null } else { Get-OpenCodeAuthVariable -Provider $modelProvider } + $knownAuthVariables = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + if ($knownAuthVariables.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status not_applicable -Detail 'No runner-known provider API-key environment variable is required for this OpenCode model selector.')) + } elseif ([string]::IsNullOrWhiteSpace($authVariable)) { + $reasons.Add("No narrow provider authentication environment variable is available for model provider '$modelProvider'. OpenCode global auth profiles are not copied into an eval run.") + } else { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Provider credential will be passed only as $authVariable.")) + } + + if ($platform -notin @('linux', 'macos')) { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) + $warnings.Add("Platform '$platform' has no external hard filesystem confinement in this adapter; execution will report pragmatic isolation.") + } elseif ($null -eq $sandboxInfo) { + $sandboxName = if ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail "External '$sandboxName' is unavailable; pragmatic isolation remains available.")) + $warnings.Add("External '$sandboxName' was unavailable; execution will report pragmatic isolation.") + } else { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $($sandboxInfo.Source) sandbox confines the process to the staged run and required system runtime paths.")) + } + $freshSessionDetail = if ($null -eq $run.Interaction) { + 'The adapter starts one new opencode run process and supplies no resume, continue, or session id.' + } else { + 'The adapter starts one new opencode run process for turn 1, captures its exact session id from structured events, and resumes scripted turns only with the explicit --session flag proven by installed help.' + } + $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail $freshSessionDetail)) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter isolates global/user configuration roots, disables external skill discovery, and deliberately preserves repository-owned project configuration; OPENCODE_DISABLE_PROJECT_CONFIG is not used.')) + $promptFidelityDetail = if ($null -eq $run.Interaction) { + 'The exact prompt bytes are sent on stdin as the first and only task input.' + } else { + 'The parent runner reads scripted inputs outside the worker-visible projection, projects none of the interaction sidecar/source files, and sends only the current UTF-8 user turn through stdin for each opencode run invocation.' + } + $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail $promptFidelityDetail)) + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status unavailable -Detail 'Behavioral eval transport is the runner-owned direct OpenCode session; preflight cannot yet observe that session''s resolved model, cwd, HOME/config, fresh identity, prompt, exclusions, or terminal capture, so those controls stay conditional until execute captures the session''s terminal evidence. OpenCode''s native Task/General subagent (and read-only Explore/Scout) remain separate advertised capabilities and are not the transport.')) + $warnings.Add('OpenCode runner-owned session controls are conditional. Execution must capture the session''s own terminal evidence (model, cwd, isolated OPENCODE_CONFIG_DIR/HOME, fresh session, prompt hash, transcript); the native Task/General subagent is a harness capability, not the benchmark transport.') + $warnings.Add('OpenCode does not expose a supported child-tool environment filter in this CLI contract; the runner removes unrelated inherited variables but cannot independently prove that the selected provider credential is hidden from every OpenCode-launched tool.') + + $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $capabilities = Get-OpenCodeCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement -ContinuationCapability $continuationCapability + if ($platform -eq 'macos') { + $warnings.Add('macOS sandbox-exec is deprecated by Apple but is used only when present; a future runner revision may replace it with an equivalent supported mechanism.') + } + $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } + $descriptorCopy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } + $descriptorCopy.harness = [ordered]@{ name = 'OpenCode CLI'; version = $harnessVersion } + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('runner-owned fresh OpenCode CLI session per eval execution', 'opencode run --format json terminal event capture', 'native Task/General subagent available as a separate harness capability, not the transport', 'deterministic runner-owned concurrent fan-out', '--auto', 'isolated OPENCODE_CONFIG_DIR', 'isolated OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'coherent Windows HOME/USERPROFILE/HOMEDRIVE/HOMEPATH', 'OPENCODE_DISABLE_EXTERNAL_SKILLS=1', 'OPENCODE_DISABLE_CLAUDE_CODE_SKILLS=1', 'permission.skill arm policy', 'repository-owned project configuration preserved', 'stdin scripted turn inputs')) { $mechanisms.Add($mechanism) } + if ($null -ne $run.Interaction -and $continuationCapability.Available) { + $mechanisms.Add('exact session id captured from turn 1 structured events') + $mechanisms.Add('explicit --session continuation selected from installed help') + $mechanisms.Add('exact requested provider/model supplied on every turn') + $mechanisms.Add('no --continue, implicit last-session, daemon, SSE idle, or session.status dependency') + } else { + $mechanisms.Add('no scripted continuation') + } + if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + $document = New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) + $preflightFinished = [DateTime]::UtcNow + $preflightTiming = [ordered]@{ + total_duration_seconds = [Math]::Round(($preflightFinished - $preflightStarted).TotalSeconds, 3) + probe_count = 0 + } + if ($null -ne $versionObservation -and $null -ne $versionObservation.Process) { + $preflightTiming.version_probe_duration_seconds = [double]$versionObservation.Process.DurationSeconds + $preflightTiming.probe_count++ + } + if ($null -ne $help) { + $preflightTiming.help_probe_duration_seconds = [double]$help.DurationSeconds + $preflightTiming.probe_count++ + } + if ($null -ne $runtimeHomeObservation -and $null -ne $runtimeHomeObservation.Process) { + $preflightTiming.effective_home_probe_duration_seconds = [double]$runtimeHomeObservation.Process.DurationSeconds + $preflightTiming.probe_count++ + } + if ($null -ne $debugHelp) { + $preflightTiming.debug_help_probe_duration_seconds = [double]$debugHelp.DurationSeconds + $preflightTiming.probe_count++ + } + if ($null -ne $debugConfig) { + $preflightTiming.debug_config_probe_duration_seconds = [double]$debugConfig.DurationSeconds + $preflightTiming.probe_count++ + } + $document.timing = $preflightTiming + $document.protocol_observations = [ordered]@{ + effective_home = [ordered]@{ + logical_home = [string]$run.HomeDirectoryPath + physical_isolated_home = [string]$run.HomeDirectoryPath + effective_runtime_home = if ($null -eq $homeIsolationObservation) { $null } else { [string]$homeIsolationObservation.RuntimeHome } + effective_runtime_home_source = if ($null -eq $homeIsolationObservation) { 'unavailable' } else { [string]$homeIsolationObservation.RuntimeHomeSource } + expected_isolated_home = if ($null -eq $homeIsolationObservation) { [string]$run.HomeDirectoryPath } else { [string]$homeIsolationObservation.ExpectedHome } + matches_isolated_home = if ($null -eq $homeIsolationObservation) { $false } else { [bool]$homeIsolationObservation.RuntimeHomeMatchesExpected } + matches_real_user_profile = if ($null -eq $homeIsolationObservation) { $false } else { [bool]$homeIsolationObservation.RuntimeHomeMatchesHost } + windows_profile_parts_coherent = if ($null -eq $homeIsolationObservation) { $false } else { [bool]$homeIsolationObservation.WindowsProfilePartsCoherent } + host_home_candidates = if ($null -eq $homeIsolationObservation) { @() } else { @($homeIsolationObservation.HostHomeCandidates) } + environment = if ($null -eq $homeIsolationObservation) { $null } else { $homeIsolationObservation.Environment } + valid = if ($null -eq $homeIsolationObservation) { $false } else { [bool]$homeIsolationObservation.Valid } + reason = if ($null -eq $homeIsolationObservation) { 'OpenCode effective-home probe did not run.' } else { [string]$homeIsolationObservation.Reason } + } + skill_isolation = [ordered]@{ + candidate_skill_exposed = [bool]$run.CandidateSkillExposed + candidate_skill_name = if ([bool]$run.CandidateSkillExposed) { Get-OpenCodeCandidateSkillName -Run $run } else { $null } + candidate_skill_physical_path = $null + ambient_skill_roots_hidden = $true + permission_skill = if ($null -eq $policyObservation) { $null } else { $policyObservation.configured_permission_skill } + expected_permission_skill = if ($null -eq $policyObservation) { Get-OpenCodeSkillPermissionPolicy -Inputs $Inputs } else { $policyObservation.expected_permission_skill } + permission_match = if ($null -eq $policyObservation) { $false } else { [bool]$policyObservation.permission_match } + permission_layer = if ($null -eq $debugConfigObservation) { 'unavailable' } elseif ([bool]$debugConfigObservation.available -and [bool]$debugConfigObservation.permission_match) { 'supported_and_verified' } else { 'unavailable_or_unverified' } + external_skill_scans_disabled = if ($null -eq $policyObservation) { $false } else { [bool]$policyObservation.external_skill_scans_disabled } + claude_code_skill_scans_disabled = if ($null -eq $policyObservation) { $false } else { [bool]$policyObservation.claude_code_skill_scans_disabled } + mechanism = if ($null -eq $policyObservation) { 'isolated physical home/config boundary' } else { [string]$policyObservation.mechanism } + debug_config = $debugConfigObservation + } + scripted_multi_turn_same_session = [ordered]@{ + available = [bool]$continuationCapability.Available + transport = 'opencode-run-explicit-session-continuation' + flag = [string]$continuationCapability.Flag + argument_style = [string]$continuationCapability.ArgumentStyle + parameter = [string]$continuationCapability.Parameter + help_evidence = [string]$continuationCapability.HelpEvidence + reason = $continuationCapability.Reason + structured_output = 'opencode run --format json' + session_identity_source = 'turn 1 structured JSON events' + exact_session_required = $true + implicit_continuation = $false + sse_dependency = $false + session_status_dependency = $false + } + } + return $document +} + +function New-OpenCodeEnvironment { + param([Parameter(Mandatory = $true)][object]$Inputs) + + # OpenCode 1.18.x resolves its native global config and skills below + # XDG_CONFIG_HOME/opencode. Keep that root inside the physical per-run + # home so HOME, USERPROFILE, and the Windows profile-part variables all + # identify the same boundary. + $configDirectory = Join-Path (Join-Path $Inputs.Run.HomeDirectoryPath '.config') 'opencode' + $appDataDirectory = Join-Path $Inputs.Run.HomeDirectoryPath 'appdata' + $localAppDataDirectory = Join-Path $Inputs.Run.HomeDirectoryPath 'localappdata' + foreach ($directory in @($appDataDirectory, $localAppDataDirectory)) { + New-Item -ItemType Directory -Path $directory -Force | Out-Null + } + New-Item -ItemType Directory -Path $configDirectory -Force | Out-Null + $configPath = Join-Path $configDirectory 'opencode.json' + $skillPermission = Get-OpenCodeSkillPermissionPolicy -Inputs $Inputs + $config = [ordered]@{ + '$schema' = 'https://opencode.ai/config.json' + permission = [ordered]@{ skill = $skillPermission } + } + [System.IO.File]::WriteAllText($configPath, (($config | ConvertTo-Json -Depth 20) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$Inputs.Profile.Model) + $authVariables = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + $environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables $authVariables -Additional @{ + APPDATA = $appDataDirectory + LOCALAPPDATA = $localAppDataDirectory + OPENCODE_CONFIG_DIR = $configDirectory + OPENCODE_CONFIG = $configPath + OPENCODE_DISABLE_AUTOUPDATE = '1' + OPENCODE_DISABLE_EXTERNAL_SKILLS = '1' + OPENCODE_DISABLE_CLAUDE_CODE_SKILLS = '1' + } + # New-RunnerEnvironment intentionally has a broad cross-runner whitelist + # for compatibility. OpenCode must not inherit NODE_PATH: it can redirect + # the Node module graph into the user's ambient installation. + [void]$environment.Remove('NODE_PATH') + if ((Get-PlatformName) -eq 'windows') { + $isolatedHomeFull = [System.IO.Path]::GetFullPath([string]$Inputs.Run.HomeDirectoryPath) + $root = [System.IO.Path]::GetPathRoot($isolatedHomeFull) + if ([string]::IsNullOrWhiteSpace($root)) { throw "OpenCode isolated home '$isolatedHomeFull' has no Windows path root." } + $homePart = $isolatedHomeFull.Substring($root.Length).TrimStart([char[]]@('\', '/')) + $environment['HOMEDRIVE'] = $root.TrimEnd([char[]]@('\', '/')) + $environment['HOMEPATH'] = '\' + $homePart + } + return $environment +} + +function Get-LinuxSandboxArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + $args = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--die-with-parent', '--new-session', '--unshare-pid')) { $args.Add($argument) } + foreach ($path in @('/usr', '/bin', '/lib', '/lib64', '/etc', '/opt')) { + if (Test-Path -LiteralPath $path) { + $args.Add('--ro-bind'); $args.Add($path); $args.Add($path) + } + } + $args.Add('--proc'); $args.Add('/proc') + $args.Add('--dev'); $args.Add('/dev') + $args.Add('--tmpfs'); $args.Add('/tmp') + $args.Add('--bind'); $args.Add($Inputs.Run.RunRoot); $args.Add('/run') + $commandSource = [string]$CommandInfo.Source + $commandDirectory = Split-Path -Parent $commandSource + if (-not ($commandSource.StartsWith('/usr/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/bin/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/opt/', [System.StringComparison]::Ordinal))) { + if (Test-Path -LiteralPath $commandDirectory -PathType Container) { + $args.Add('--ro-bind'); $args.Add($commandDirectory); $args.Add($commandDirectory) + } + } + $args.Add('--chdir'); $args.Add('/run/repo') + $insideEnvironment = [ordered]@{ + HOME = '/run/home' + USERPROFILE = '/run/home' + XDG_CONFIG_HOME = '/run/home/.config' + XDG_DATA_HOME = '/run/home/.local/share' + XDG_CACHE_HOME = '/run/home/.cache' + TEMP = '/run/home/tmp' + TMP = '/run/home/tmp' + APPDATA = '/run/home/appdata' + LOCALAPPDATA = '/run/home/localappdata' + OPENCODE_CONFIG_DIR = '/run/home/.config/opencode' + OPENCODE_CONFIG = '/run/home/.config/opencode/opencode.json' + OPENCODE_DISABLE_AUTOUPDATE = [string]$Environment['OPENCODE_DISABLE_AUTOUPDATE'] + OPENCODE_DISABLE_EXTERNAL_SKILLS = [string]$Environment['OPENCODE_DISABLE_EXTERNAL_SKILLS'] + OPENCODE_DISABLE_CLAUDE_CODE_SKILLS = [string]$Environment['OPENCODE_DISABLE_CLAUDE_CODE_SKILLS'] + PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + CI = '1' + NO_COLOR = '1' + } + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$Inputs.Profile.Model) + $authVariables = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + foreach ($authName in $authVariables) { + if ($Environment.Contains($authName) -and -not [string]::IsNullOrWhiteSpace([string]$Environment[$authName])) { + $insideEnvironment[$authName] = [string]$Environment[$authName] + } + } + foreach ($key in @($insideEnvironment.Keys)) { + $args.Add('--setenv'); $args.Add($key); $args.Add([string]$insideEnvironment[$key]) + } + $args.Add('--') + $args.Add($CommandInfo.FileName) + foreach ($prefix in @($CommandInfo.Prefix)) { $args.Add($prefix) } + return @($args) +} + +function New-MacosSandboxProfile { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo + ) + + $profilePath = Join-Path $Inputs.Run.HomeDirectoryPath 'opencode-sandbox.sb' + $runRoot = $Inputs.Run.RunRoot.Replace('\', '/') + $commandDirectory = (Split-Path -Parent ([string]$CommandInfo.Source)).Replace('\', '/') + $systemReadRoots = @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/libexec', '/System', '/Library', '/opt', '/private/var/db', $commandDirectory) + $lines = [System.Collections.Generic.List[string]]::new() + $lines.Add('(version 1)') + $lines.Add('(deny default)') + $lines.Add('(allow process*)') + $lines.Add('(allow network*)') + foreach ($root in $systemReadRoots | Sort-Object -Unique) { + if (-not [string]::IsNullOrWhiteSpace($root) -and (Test-Path -LiteralPath $root -PathType Container)) { + $escapedRoot = $root.Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRoot)) + } + } + $escapedRunRoot = $runRoot.Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add(('(allow file-write* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add('(allow file-read* (subpath "/dev"))') + $lines.Add('(allow file-write* (subpath "/dev/null"))') + [System.IO.File]::WriteAllText($profilePath, ([string]::Join("`n", $lines) + "`n"), [System.Text.UTF8Encoding]::new($false)) + return $profilePath +} + +function Write-OpenCodeCapture { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text + ) + + $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $path) -Force | Out-Null + [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) + return New-ArtifactReference -Run $RunData.Run -Path $RelativePath -Scope run -MediaType (Get-MediaType -Path $RelativePath) +} + +function Add-OpenCodeNullableInt64 { + param([object]$Current, [object]$Value) + + if ($null -eq $Value) { return $Current } + if ($null -eq $Current) { return [int64]$Value } + return ([int64]$Current + [int64]$Value) +} + +function Read-OpenCodeScriptedTurn { + param( + [Parameter(Mandatory = $true)][object]$Parsed, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[string]]$Warnings + ) + + $finalTextParts = [System.Collections.Generic.List[string]]::new() + $sessionIds = [System.Collections.Generic.List[string]]::new() + $observedModels = [System.Collections.Generic.List[string]]::new() + $eventTimestamps = [System.Collections.Generic.List[string]]::new() + $eventCounts = @{} + $usageBuckets = [ordered]@{} + $toolCalls = 0 + $failureMessage = $null + $terminalEventObserved = $false + foreach ($event in @($Parsed.Events)) { + $eventType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ([string]::IsNullOrWhiteSpace($eventType)) { + $Warnings.Add('OpenCode emitted an event without a type; it was ignored.') + continue + } + if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } + foreach ($sessionId in @(Get-OpenCodeEventSessionIds -Event $event)) { + if ($sessionIds -notcontains $sessionId) { $sessionIds.Add($sessionId) } + } + foreach ($modelName in @(Get-OpenCodeEventModels -Event $event)) { + if ($observedModels -notcontains $modelName) { $observedModels.Add($modelName) } + } + foreach ($timestamp in @( + (Get-JsonProperty -Object $event -Name 'timestamp' -Default $null), + (Get-JsonProperty -Object $event -Name 'timestamp_utc' -Default $null) + )) { + if (-not [string]::IsNullOrWhiteSpace([string]$timestamp) -and $eventTimestamps -notcontains [string]$timestamp) { $eventTimestamps.Add([string]$timestamp) } + } + $part = Get-JsonProperty -Object $event -Name 'part' -Default $null + switch ($eventType) { + 'text' { + $text = Get-JsonProperty -Object $event -Name 'text' -Default (Get-JsonProperty -Object $part -Name 'text' -Default '') + if (-not [string]::IsNullOrWhiteSpace([string]$text)) { $finalTextParts.Add([string]$text) } + } + 'step_finish' { + $terminalEventObserved = $true + $tokens = Get-JsonProperty -Object $part -Name 'tokens' -Default (Get-JsonProperty -Object $event -Name 'tokens' -Default $null) + if ($null -ne $tokens) { + foreach ($name in @('input', 'output', 'reasoning', 'cache_read', 'cache_write')) { + $value = Get-JsonProperty -Object $tokens -Name $name -Default $null + if ($null -ne $value) { $usageBuckets[$name] = Add-OpenCodeNullableInt64 -Current (Get-JsonProperty -Object $usageBuckets -Name $name -Default $null) -Value $value } + } + } + $costValue = Get-JsonProperty -Object $part -Name 'cost' -Default (Get-JsonProperty -Object $event -Name 'cost' -Default $null) + if ($null -ne $costValue) { $usageBuckets['cost'] = Add-OpenCodeNullableInt64 -Current (Get-JsonProperty -Object $usageBuckets -Name 'cost' -Default $null) -Value $costValue } + } + 'tool_use' { $toolCalls++ } + 'error' { $failureMessage = [string](Get-JsonProperty -Object $event -Name 'message' -Default (Get-JsonProperty -Object $part -Name 'message' -Default 'OpenCode emitted an error.')) } + { $_ -in @('session.completed', 'run.completed', 'done') } { $terminalEventObserved = $true } + 'step_start' { } + 'reasoning' { } + default { $Warnings.Add("Unknown OpenCode event '$eventType' was preserved as a warning.") } + } + } + return [pscustomobject]@{ + FinalText = if ($finalTextParts.Count -gt 0) { [string]::Join('', $finalTextParts) } else { $null } + SessionIds = @($sessionIds.ToArray()) + ObservedModels = @($observedModels.ToArray()) + EventTimestamps = @($eventTimestamps.ToArray()) + EventTiming = Get-OpenCodeEventTiming -Timestamps @($eventTimestamps.ToArray()) + EventCounts = $eventCounts + UsageBuckets = $usageBuckets + ToolCalls = $toolCalls + FailureMessage = $failureMessage + TerminalEventObserved = $terminalEventObserved + StructuredEventCount = @($Parsed.Events).Count + ParseErrorCount = @($Parsed.Errors).Count + } +} + +function Get-OpenCodeInteractionSourcePhysicalPaths { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$Projection + ) + + $paths = [System.Collections.Generic.List[string]]::new() + foreach ($turn in @($Inputs.Run.Interaction.turns)) { + $source = [string](Get-JsonProperty -Object $turn -Name 'source' -Default '') + if ([string]::IsNullOrWhiteSpace($source)) { continue } + $logicalSource = Resolve-ContainedPath -BasePath $Inputs.Run.RunRoot -RelativePath $source -FieldName 'interaction turn source' -Kind File + if (Test-PathInside -BasePath $Inputs.Run.WorkingDirectoryPath -CandidatePath $logicalSource) { + $relative = [System.IO.Path]::GetRelativePath($Inputs.Run.WorkingDirectoryPath, $logicalSource) + $paths.Add((Join-Path $Projection.PhysicalWorkingDirectory ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + } elseif (Test-PathInside -BasePath $Inputs.Run.HomeDirectoryPath -CandidatePath $logicalSource) { + $relative = [System.IO.Path]::GetRelativePath($Inputs.Run.HomeDirectoryPath, $logicalSource) + $paths.Add((Join-Path $Projection.PhysicalHomeDirectory ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + } + } + return @($paths.ToArray() | Select-Object -Unique) +} + +function Get-OpenCodeInteractionJsonPhysicalPaths { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$Projection + ) + + if ($null -eq $Inputs.Run.InteractionPath) { return @() } + $paths = [System.Collections.Generic.List[string]]::new() + if (Test-PathInside -BasePath $Inputs.Run.WorkingDirectoryPath -CandidatePath $Inputs.Run.InteractionPath) { + $relative = [System.IO.Path]::GetRelativePath($Inputs.Run.WorkingDirectoryPath, $Inputs.Run.InteractionPath) + $paths.Add((Join-Path $Projection.PhysicalWorkingDirectory ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + } elseif (Test-PathInside -BasePath $Inputs.Run.HomeDirectoryPath -CandidatePath $Inputs.Run.InteractionPath) { + $relative = [System.IO.Path]::GetRelativePath($Inputs.Run.HomeDirectoryPath, $Inputs.Run.InteractionPath) + $paths.Add((Join-Path $Projection.PhysicalHomeDirectory ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + } + return @($paths.ToArray() | Select-Object -Unique) +} + +function Get-OpenCodeFutureTurnCanary { + param([Parameter(Mandatory = $true)][object]$Run) + + $seed = [string]$Run.InteractionHash + if ([string]::IsNullOrWhiteSpace($seed)) { $seed = Get-Sha256HexFromFile -Path $Run.InteractionPath } + return 'CODEBELT_FUTURE_TURN_CANARY_' + $seed.Substring(0, [Math]::Min(16, $seed.Length)).ToUpperInvariant() +} + +function Test-OpenCodeCanaryInTree { + param( + [Parameter(Mandatory = $true)][string]$Root, + [Parameter(Mandatory = $true)][string]$Canary + ) + + if ([string]::IsNullOrWhiteSpace($Canary) -or -not (Test-Path -LiteralPath $Root -PathType Container)) { return $false } + foreach ($file in @(Get-ChildItem -LiteralPath $Root -Recurse -Force -File -ErrorAction SilentlyContinue)) { + if ($file.Name -like "*$Canary*") { return $true } + try { + if ([System.IO.File]::ReadAllText($file.FullName, [System.Text.UTF8Encoding]::new($false)).Contains($Canary)) { return $true } + } catch { } + } + return $false +} + +function Invoke-OpenCodeExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $preflightCache = $null + try { + $preflightCache = Get-OpenCodeCachedPreflight -Inputs $Inputs + $preflightSource = [string]$preflightCache.Source + $preflight = if ([bool]$preflightCache.Hit) { $preflightCache.Preflight } else { Get-OpenCodePreflight -Inputs $Inputs } + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + $executionDescriptor = [ordered]@{} + foreach ($key in $descriptor.Keys) { $executionDescriptor[$key] = $descriptor[$key] } + $executionDescriptor.harness = $preflight.harness + if ($preflight.status -ne 'compatible') { + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight; preflight_source = $preflightSource; resume = $false }) -AttemptCount 1 + } + + if ($null -ne $Inputs.Run.Interaction) { + return Invoke-OpenCodeScriptedExecute -Inputs $Inputs -Preflight $preflight -ExecutionDescriptor $executionDescriptor -PreflightSource $preflightSource + } + + $projection = $null + $singleResult = $null + try { + $commandInfo = Resolve-ExternalCommand -Name 'opencode' + $projection = New-OpenCodeExecutionProjection -Inputs $Inputs + $executionInputs = $projection.Inputs + $environment = New-OpenCodeEnvironment -Inputs $executionInputs + $runtimeHomeObservation = Get-OpenCodeRuntimeHomeObservation -CommandInfo $commandInfo -Inputs $executionInputs -Environment $environment + $homeIsolationObservation = Get-OpenCodeHomeIsolationObservation -Inputs $executionInputs -Environment $environment -RuntimeHome $runtimeHomeObservation + $policyObservation = Get-OpenCodeSkillPolicyObservation -Inputs $executionInputs -Environment $environment + $skillIsolationObservation = Get-OpenCodeSkillRootObservation -Inputs $executionInputs + $isolationReasons = [System.Collections.Generic.List[string]]::new() + if (-not [bool]$homeIsolationObservation.Valid) { $isolationReasons.Add([string]$homeIsolationObservation.Reason) } + if (-not [bool]$policyObservation.permission_match -or -not [bool]$policyObservation.external_skill_scans_disabled -or -not [bool]$policyObservation.claude_code_skill_scans_disabled) { $isolationReasons.Add([string]$policyObservation.reason) } + if (-not [bool]$skillIsolationObservation.Valid) { $isolationReasons.Add([string]$skillIsolationObservation.Reason) } + if ($isolationReasons.Count -gt 0) { + $singleResult = New-OpenCodeIsolationFailureResult -LogicalInputs $Inputs -ExecutionDescriptor $executionDescriptor -Preflight $preflight -Projection $projection -Environment $environment -HomeIsolation $homeIsolationObservation -PolicyObservation $policyObservation -SkillIsolation $skillIsolationObservation -PreflightSource $preflightSource -Reasons @($isolationReasons.ToArray()) -SessionId $sessionId -StartedUtc $started -Resume $false + return $singleResult + } + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } + $model = [string]$Inputs.Profile.Model + $arguments = New-OpenCodeCliArguments -Inputs $executionInputs -VisiblePlatform $visiblePlatform + + if ($platform -eq 'linux' -and $hardFilesystem) { + $sandboxArguments = Get-LinuxSandboxArguments -Inputs $executionInputs -CommandInfo $commandInfo -Environment $environment + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $executionInputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $executionInputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } elseif ($platform -eq 'macos' -and $hardFilesystem) { + $sandboxProfile = New-MacosSandboxProfile -Inputs $executionInputs -CommandInfo $commandInfo + $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + $arguments + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $executionInputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $executionInputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $process = Invoke-OpenCodeCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $executionInputs -Environment $environment -InputBytes $executionInputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } + + $stdoutArtifact = Write-OpenCodeCapture -RunData $Inputs -RelativePath 'evidence/opencode-events.jsonl' -Text $process.Stdout + $stderrArtifact = Write-OpenCodeCapture -RunData $Inputs -RelativePath 'evidence/opencode-stderr.txt' -Text $process.Stderr + $artifacts = [System.Collections.Generic.List[object]]::new() + $artifacts.Add($stdoutArtifact); $artifacts.Add($stderrArtifact) + $parsed = ConvertFrom-JsonLines -Text $process.Stdout + $warnings = [System.Collections.Generic.List[string]]::new() + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("OpenCode event parse error: $parseError") } + $finalTextParts = [System.Collections.Generic.List[string]]::new() + $sessionIds = [System.Collections.Generic.List[string]]::new() + $observedModels = [System.Collections.Generic.List[string]]::new() + $eventTimestamps = [System.Collections.Generic.List[string]]::new() + $eventCounts = @{} + $toolCalls = 0 + $commands = [System.Collections.Generic.List[object]]::new() + $usageBuckets = [ordered]@{} + $failureMessage = $null + $terminalEventObserved = $false + foreach ($event in @($parsed.Events)) { + $eventType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ([string]::IsNullOrWhiteSpace($eventType)) { + $warnings.Add('OpenCode emitted an event without a type; it was ignored.') + continue + } + if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } + foreach ($eventSessionId in @(Get-OpenCodeEventSessionIds -Event $event)) { + if ($sessionIds -notcontains $eventSessionId) { $sessionIds.Add($eventSessionId) } + } + foreach ($eventModel in @(Get-OpenCodeEventModels -Event $event)) { + if ($observedModels -notcontains $eventModel) { $observedModels.Add($eventModel) } + } + foreach ($timestamp in @( + (Get-JsonProperty -Object $event -Name 'timestamp' -Default $null), + (Get-JsonProperty -Object $event -Name 'timestamp_utc' -Default $null) + )) { + if (-not [string]::IsNullOrWhiteSpace([string]$timestamp) -and $eventTimestamps -notcontains [string]$timestamp) { $eventTimestamps.Add([string]$timestamp) } + } + $part = Get-JsonProperty -Object $event -Name 'part' -Default $null + switch ($eventType) { + 'text' { + $text = Get-JsonProperty -Object $event -Name 'text' -Default (Get-JsonProperty -Object $part -Name 'text' -Default '') + if (-not [string]::IsNullOrWhiteSpace([string]$text)) { $finalTextParts.Add([string]$text) } + } + 'step_finish' { + $terminalEventObserved = $true + $tokens = Get-JsonProperty -Object $part -Name 'tokens' -Default (Get-JsonProperty -Object $event -Name 'tokens' -Default $null) + if ($null -ne $tokens) { + foreach ($name in @('input', 'output', 'reasoning', 'cache_read', 'cache_write')) { + $value = Get-JsonProperty -Object $tokens -Name $name -Default $null + if ($null -ne $value) { $usageBuckets[$name] = $value } + } + } + $costValue = Get-JsonProperty -Object $part -Name 'cost' -Default (Get-JsonProperty -Object $event -Name 'cost' -Default $null) + if ($null -ne $costValue) { $usageBuckets['cost'] = $costValue } + } + 'tool_use' { + $toolCalls++ + $toolName = Get-JsonProperty -Object $part -Name 'tool' -Default (Get-JsonProperty -Object $event -Name 'tool' -Default '') + $commands.Add([ordered]@{ tool = [string]$toolName }) + } + 'error' { + $failureMessage = [string](Get-JsonProperty -Object $event -Name 'message' -Default (Get-JsonProperty -Object $part -Name 'message' -Default 'OpenCode emitted an error.')) + } + { $_ -in @('session.completed', 'run.completed', 'done') } { $terminalEventObserved = $true } + 'step_start' { } + 'reasoning' { } + default { $warnings.Add("Unknown OpenCode event '$eventType' was preserved as a warning.") } + } + } + $finalText = if ($finalTextParts.Count -gt 0) { [string]::Join('', $finalTextParts) } else { $null } + $observedSessionIds = @($sessionIds.ToArray()) + $exactSessionId = if ($observedSessionIds.Count -eq 1) { [string]$observedSessionIds[0] } else { $null } + if (-not [string]::IsNullOrWhiteSpace($exactSessionId)) { $sessionId = $exactSessionId } + $eventTiming = Get-OpenCodeEventTiming -Timestamps @($eventTimestamps.ToArray()) + $status = 'completed' + $reason = $null + $failure = $null + $exitStatus = if ($process.TimedOut) { $null } else { [Nullable[int]]$process.ExitCode } + if ($process.TimedOut) { + $status = 'timed_out'; $reason = 'opencode_timeout'; $failure = New-ExecutionFailure -Code 'timed_out' -Message 'OpenCode did not finish before timeout_seconds.' + } elseif ($process.ExitCode -ne 0 -or $null -ne $failureMessage) { + $status = 'failed'; $reason = 'opencode_failure'; $failure = New-ExecutionFailure -Code 'opencode_failure' -Message ([string]$failureMessage) + } elseif (@($parsed.Errors).Count -gt 0) { + $status = 'incompatible'; $reason = 'native_interaction_incompatible'; $failure = New-ExecutionFailure -Code 'native_interaction_incompatible' -Message 'OpenCode single-turn execution did not produce a complete structured JSON event stream.' + } elseif ($observedSessionIds.Count -ne 1) { + $status = 'incompatible'; $reason = 'native_interaction_incompatible'; $failure = New-ExecutionFailure -Code 'native_interaction_incompatible' -Message 'OpenCode single-turn execution did not expose exactly one non-empty session id in structured events.' + } elseif (@($observedModels | Where-Object { [string]$_ -ne [string]$Inputs.Profile.Model }).Count -gt 0) { + $status = 'incompatible'; $reason = 'native_interaction_incompatible'; $failure = New-ExecutionFailure -Code 'native_interaction_incompatible' -Message "OpenCode single-turn execution reported a model different from the requested model '$($Inputs.Profile.Model)'." + } elseif (-not [bool]$terminalEventObserved) { + $status = 'incompatible'; $reason = 'native_interaction_incompatible'; $failure = New-ExecutionFailure -Code 'native_interaction_incompatible' -Message 'OpenCode single-turn execution did not emit a terminal structured event.' + } elseif ([string]::IsNullOrWhiteSpace($finalText)) { + $reason = 'opencode_did_not_return_final_response'; $warnings.Add('OpenCode exited successfully without a text response.') + } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/opencode-events.jsonl'; complete = $true }) + tokens = if ($usageBuckets.Count -eq 0) { New-UnavailableMetric -Reason 'opencode_did_not_expose_usage' } else { New-AvailableMetric -Value $usageBuckets } + tool_calls = New-AvailableMetric -Value $toolCalls + cost = if ($usageBuckets.Contains('cost')) { New-AvailableMetric -Value $usageBuckets['cost'] } else { New-UnavailableMetric -Reason 'opencode_did_not_expose_cost' } + } + $capabilities = Get-OpenCodeCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('opencode run --format json', '--auto', 'isolated OPENCODE_CONFIG_DIR', 'isolated OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'repository-owned project configuration preserved', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement'); $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$Inputs.Profile.Model) + $credentialNames = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + $credentialEvidence = [ordered]@{ + model_provider = $modelProvider + provider_environment_variables = $credentialNames + unrelated_environment_excluded = $true + child_tool_visibility = 'provider_credential_may_be_visible_to_native_child_tools; no supported child filter is exposed' + value_observed = $false + } + # Runner-owned terminal evidence for the direct OpenCode session. The runner + # controlled the fresh session, its model lock, working directory, isolated + # OPENCODE_CONFIG_DIR/HOME, and stdin prompt, and captured the session's own + # structured event transcript. This is transport-owned evidence, never + # orchestrator-authored. + $transcriptArtifactPath = 'evidence/opencode-events.jsonl' + $transcriptArtifact = @($artifacts | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'path' -Default '') -eq $transcriptArtifactPath } | Select-Object -First 1) + $terminalCapture = $status -eq 'completed' -and $observedSessionIds.Count -eq 1 -and (-not $process.TimedOut) -and (-not [string]::IsNullOrWhiteSpace([string]$process.Stdout)) -and [bool]$terminalEventObserved + $executionPaths = New-OpenCodeExecutionPaths -LogicalInputs $Inputs -ExecutionInputs $executionInputs -Projection $projection -Environment $environment -HomeIsolation $homeIsolationObservation + $candidateSkillExposure = New-OpenCodeCandidateSkillExposure -LogicalInputs $Inputs -Projection $projection -SkillIsolation $skillIsolationObservation + $evidence = [ordered]@{ + execution_paths = $executionPaths + event_counts = $eventCounts + commands = @($commands) + observed_session_ids = @($observedSessionIds) + prompt_first_input = $true + resume = $false + model_argument = $model + observed_model = if ($observedModels.Count -eq 0) { [string]$model } else { [string]$observedModels[$observedModels.Count - 1] } + observed_models = @($observedModels.ToArray()) + sandbox = $sandboxEvidence + project_configuration = 'repository_owned_project_config_preserved' + disable_project_config_environment = $false + credential = $credentialEvidence + candidate_skill_exposure = $candidateSkillExposure + effective_home = New-OpenCodeHomeIsolationEvidence -Observation $homeIsolationObservation + skill_policy = $policyObservation + skill_isolation = New-OpenCodeSkillIsolationEvidence -Observation $skillIsolationObservation + ambient_skill_policy = [ordered]@{ + mechanism = [string]$policyObservation.mechanism + permission_skill = $policyObservation.configured_permission_skill + external_skill_scans_disabled = [bool]$policyObservation.external_skill_scans_disabled + claude_code_skill_scans_disabled = [bool]$policyObservation.claude_code_skill_scans_disabled + ambient_skill_roots_hidden = [int]$skillIsolationObservation.AmbientSkillCount -eq 0 + candidate_skill_exposed = [bool]$executionInputs.Run.CandidateSkillExposed + candidate_skill_physical_path = if ([bool]$executionInputs.Run.CandidateSkillExposed) { [string]$projection.PhysicalSkillDirectory } else { $null } + } + timing = [ordered]@{ + preflight = Get-JsonProperty -Object $preflight -Name 'timing' -Default $null + preflight_source = $preflightSource + projection_setup_duration_seconds = [double]$projection.SetupDurationSeconds + native_cli_process_total_seconds = [double]$process.DurationSeconds + turns = @([ordered]@{ + turn = 1 + invocation = 'fresh' + process_duration_seconds = [double]$process.DurationSeconds + cli_startup_and_execution_duration_seconds = [double]$process.DurationSeconds + }) + total_runner_execution_seconds = [double]$process.DurationSeconds + } + capture = [ordered]@{ + source = 'harness_native_transport' + terminal = [bool]$terminalCapture + worker_authored = $false + artifact = $transcriptArtifactPath + sha256 = if ($transcriptArtifact.Count -eq 1) { [string](Get-JsonProperty -Object $transcriptArtifact[0] -Name 'sha256' -Default $null) } else { $null } + } + delegation = [ordered]@{ + dispatch_owner = 'runner' + mechanism = [string]$descriptor.delegation.mechanism + worker_session_id = $sessionId + observed_model = if ($observedModels.Count -eq 0) { [string]$model } else { [string]$observedModels[$observedModels.Count - 1] } + observed_working_directory = [string]$projection.PhysicalWorkingDirectory + observed_home = [string]$projection.PhysicalHomeDirectory + effective_runtime_home = [string]$homeIsolationObservation.RuntimeHome + effective_opencode_config_root = [string]$environment['OPENCODE_CONFIG_DIR'] + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$Inputs.Run.PromptHash + terminal_result_capture = [bool]$terminalCapture + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + } + } + if ($null -ne $eventTiming) { + $evidence.timing.turns[0].event_timing = $eventTiming + } + $singleResult = New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $process.FinishedUtc.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'OpenCode accepted the requested runner-native model selector and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence $evidence -AttemptCount 1 + return $singleResult + } finally { + if ($null -ne $projection) { + try { + Remove-OpenCodeProjectedCandidateSkill -Projection $projection + Sync-OpenCodeProjectedRepository -Projection $projection + } finally { + Remove-OpenCodeExecutionProjection -Projection $projection + if ($null -ne $singleResult -and $null -ne $singleResult.evidence -and $null -ne $singleResult.evidence.execution_paths) { + $singleResult.evidence.execution_paths.projection_cleanup = 'removed' + } + if ($null -ne $singleResult -and $null -ne $singleResult.evidence -and $null -ne $singleResult.evidence.timing) { + $cleanupFinished = [DateTime]::UtcNow + $singleResult.evidence.timing.projection_cleanup_duration_seconds = [Math]::Round(($cleanupFinished - $process.FinishedUtc).TotalSeconds, 3) + $singleResult.evidence.timing.total_runner_execution_seconds = [Math]::Round(($cleanupFinished - $started).TotalSeconds, 3) + } + } + } + } + } finally { + $cachePath = if ($null -eq $preflightCache) { $null } else { [string]$preflightCache.CachePath } + Remove-OpenCodePreflightCache -Path $cachePath + } +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { Write-RunnerJson -Value (Get-OpenCodeDescriptor) -AsOutput } + 'preflight' { + $inputs = Resolve-OpenCodeInputs + $preflight = Get-OpenCodePreflight -Inputs $inputs + try { [void](Save-OpenCodePreflightObservation -Inputs $inputs -Preflight $preflight) } catch { } + Write-RunnerJson -Value $preflight -AsOutput + } + 'execute' { + $inputs = Resolve-OpenCodeInputs + [void](Assert-PhaseOneEvidenceWritable -Run $inputs.Run) + $result = Invoke-OpenCodeExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/orchestration.ps1 b/scripts/eval-runners/orchestration.ps1 new file mode 100644 index 0000000..6ed0797 --- /dev/null +++ b/scripts/eval-runners/orchestration.ps1 @@ -0,0 +1,498 @@ +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +. (Join-Path $PSScriptRoot 'runner-common.ps1') +. (Join-Path $PSScriptRoot 'manifest-paths.ps1') + +function Get-OrchestrationProfileValue { + param( + [Parameter(Mandatory = $true)][object]$Profile, + [Parameter(Mandatory = $true)][string]$Name, + [object]$Default = $null + ) + + if ($Profile -is [System.Collections.IDictionary] -and $Profile.Contains($Name)) { + return $Profile[$Name] + } + if ($Profile.PSObject.Properties.Name -contains $Name) { + return $Profile.$Name + } + return $Default +} + +function New-EvalOrchestrationPlan { + <# + Deterministic parent-side planning only. This function never starts a + harness process, calls a model, reads grading material, or exposes a + paired arm to a worker envelope. + #> + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest, + [Parameter(Mandatory = $true)][object]$Profile, + [object]$Descriptor = $null + ) + + $records = @(Get-ManifestRunRecords -IterationDirectory $IterationDirectory -Manifest $Manifest) + $requestedConcurrency = [int](Get-OrchestrationProfileValue -Profile $Profile -Name 'concurrency' -Default 0) + if ($requestedConcurrency -lt 1) { + throw 'execution-profile.json concurrency must be at least 1 for native worker orchestration.' + } + + $runner = [string](Get-OrchestrationProfileValue -Profile $Profile -Name 'runner' -Default '') + $model = [string](Get-OrchestrationProfileValue -Profile $Profile -Name 'model' -Default '') + if ([string]::IsNullOrWhiteSpace($runner) -or [string]::IsNullOrWhiteSpace($model)) { + throw 'Native worker orchestration requires a selected runner and model.' + } + + # Older deterministic callers can omit the descriptor and retain the + # original orchestrator-owned contract. Package handoffs always provide + # the selected descriptor so the native dispatch owner is explicit. + $dispatchOwner = 'orchestrator' + $dispatchMechanism = '' + if ($null -ne $Descriptor) { + $dispatchOwner = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $Descriptor -Name 'delegation' -Default $null) -Name 'dispatch_owner' -Default '') + $dispatchMechanism = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $Descriptor -Name 'delegation' -Default $null) -Name 'mechanism' -Default '') + } + if ($dispatchOwner -notin @('orchestrator', 'runner')) { + throw "Native worker orchestration dispatch_owner '$dispatchOwner' is unsupported." + } + + $arms = [System.Collections.Generic.List[object]]::new() + $seenWorkers = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) + foreach ($record in $records) { + $workerId = ('arm-{0}-{1}' -f $record.EvalId, $record.Configuration) + if (-not $seenWorkers.Add($workerId)) { + throw "Manifest produced duplicate native worker id '$workerId'." + } + + # The parent retains the exact manifest-declared destinations. They are + # deliberately kept outside the worker envelope below. + $arms.Add([ordered]@{ + worker_id = $workerId + eval_id = $record.EvalId + eval_name = $record.EvalName + configuration = $record.Configuration + dispatch_owner = $dispatchOwner + depends_on = @() + parent_paths = [ordered]@{ + run_manifest = $record.RunManifestRelative + execution_result = $record.ExecutionResultRelative + result = $record.ResultRelative + } + worker = [ordered]@{ + worker_id = $workerId + eval_id = $record.EvalId + eval_name = $record.EvalName + configuration = $record.Configuration + run_manifest = $record.RunManifestRelative + run_manifest_path = $record.RunManifestPath + model = $model + reasoning_effort = Get-OrchestrationProfileValue -Profile $Profile -Name 'reasoning_effort' + configuration_profile = Get-OrchestrationProfileValue -Profile $Profile -Name 'configuration_profile' + tool_profile = Get-OrchestrationProfileValue -Profile $Profile -Name 'tool_profile' + timeout_seconds = [int](Get-OrchestrationProfileValue -Profile $Profile -Name 'timeout_seconds' -Default 0) + one_arm_only = $true + paired_arm_visible = $false + grading_material_visible = $false + parent_executes_arm = $false + dispatch_owner = $dispatchOwner + dispatch_mechanism = $dispatchMechanism + runner_execute_invocation = if ($dispatchOwner -eq 'runner') { 'required' } else { 'forbidden' } + nested_model_execution = $false + model_execution_count = 1 + } + }) + } + + $schemas = Get-RunnerSchemaNames + $parallelDispatchRequired = @($arms).Count -gt 1 -and $requestedConcurrency -gt 1 + return [ordered]@{ + schema = $schemas.OrchestrationPlan + protocol_version = $schemas.Protocol + runner = $runner + model = $model + dispatch_owner = $dispatchOwner + requested_concurrency = $requestedConcurrency + parallel_dispatch_required = $parallelDispatchRequired + minimum_parallel_workers = if ($parallelDispatchRequired) { 2 } else { 1 } + native_worker_required = $true + parent_executes_arms = $false + nested_model_execution = $false + dispatch_policy = if ($dispatchOwner -eq 'runner') { 'one fresh runner-owned native worker transport per arm; independent transports must run concurrently up to requested_concurrency when capacity permits' } else { 'one fresh orchestrator-owned harness-native worker per arm; independent workers must run concurrently up to requested_concurrency when capacity permits' } + capacity_policy = 'harness_authoritative; a rejected delegation that did not start remains queued and is not an eval attempt' + arms = $arms.ToArray() + } +} + +function Get-OrchestrationArmByWorkerId { + param( + [Parameter(Mandatory = $true)][object]$Plan, + [Parameter(Mandatory = $true)][string]$WorkerId + ) + + foreach ($arm in @($Plan.arms)) { + if ([string]$arm.worker_id -eq $WorkerId) { return $arm } + } + throw "Unknown orchestration worker '$WorkerId'." +} + +function New-OrchestrationState { + param([Parameter(Mandatory = $true)][object]$Plan) + + $pending = @($Plan.arms | ForEach-Object { [string]$_.worker_id }) + return [ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestration-state/1' + plan_schema = [string]$Plan.schema + dispatch_owner = [string](Get-JsonProperty -Object $Plan -Name 'dispatch_owner' -Default 'orchestrator') + requested_concurrency = [int]$Plan.requested_concurrency + parallel_dispatch_required = [bool](Get-JsonProperty -Object $Plan -Name 'parallel_dispatch_required' -Default $false) + minimum_parallel_workers = [int](Get-JsonProperty -Object $Plan -Name 'minimum_parallel_workers' -Default 1) + pending_worker_ids = @($pending) + active = [ordered]@{} + completed = [ordered]@{} + delegation_rejections = [ordered]@{} + capacity_limit_reported = $false + eval_attempts = [ordered]@{} + max_observed_active = 0 + execution_freeze = $null + } +} + +function Get-OrchestrationActiveCount { + param([Parameter(Mandatory = $true)][object]$State) + + return @((Get-OrchestrationDictionary -Object $State -Name 'active').Keys).Count +} + +function Get-OrchestrationDictionary { + param( + [Parameter(Mandatory = $true)][object]$Object, + [Parameter(Mandatory = $true)][string]$Name + ) + + $value = Get-JsonProperty -Object $Object -Name $Name -Default $null + if ($null -eq $value -or -not ($value -is [System.Collections.IDictionary])) { + throw "Orchestration state field '$Name' must be a dictionary." + } + return $value +} + +function Get-NextWorkerDispatches { + <# + Returns dispatch envelopes without consuming pending work. An external + harness decides whether each native delegation request was accepted. That + is what lets a harness-owned capacity limit reject a request without + turning it into an eval attempt. + #> + param( + [Parameter(Mandatory = $true)][object]$Plan, + [Parameter(Mandatory = $true)][object]$State + ) + + $requested = [int]$Plan.requested_concurrency + $active = Get-OrchestrationActiveCount -State $State + $slots = [Math]::Max(0, $requested - $active) + if ($slots -eq 0) { return @() } + + $pending = @($State.pending_worker_ids) + $dispatches = [System.Collections.Generic.List[object]]::new() + foreach ($workerId in $pending | Select-Object -First $slots) { + $arm = Get-OrchestrationArmByWorkerId -Plan $Plan -WorkerId ([string]$workerId) + $dispatches.Add((New-WorkerDispatchEnvelope -Arm $arm)) + } + return @($dispatches) +} + +function Register-DelegationAccepted { + param( + [Parameter(Mandatory = $true)][object]$State, + [Parameter(Mandatory = $true)][string]$WorkerId, + [string]$WorkerSessionId = '' + ) + + $active = Get-OrchestrationDictionary -Object $State -Name 'active' + $completed = Get-OrchestrationDictionary -Object $State -Name 'completed' + if ($active.Contains($WorkerId) -or $completed.Contains($WorkerId)) { + throw "Worker '$WorkerId' was already accepted or completed; delegation acceptance is exactly-once and must not be retried." + } + $pending = [System.Collections.Generic.List[string]]::new() + foreach ($id in @($State.pending_worker_ids)) { [void]$pending.Add([string]$id) } + if (-not $pending.Contains($WorkerId)) { + throw "Worker '$WorkerId' cannot be accepted because it is not pending; preserve the orchestration state and do not register a second attempt." + } + + [void]$pending.Remove($WorkerId) + $State.pending_worker_ids = @($pending) + $attempts = Get-OrchestrationDictionary -Object $State -Name 'eval_attempts' + $attempts[$WorkerId] = 1 + $active[$WorkerId] = [ordered]@{ + worker_id = $WorkerId + worker_session_id = if ([string]::IsNullOrWhiteSpace($WorkerSessionId)) { $null } else { $WorkerSessionId } + accepted_utc = [DateTime]::UtcNow.ToString('o') + attempt_count = 1 + } + $activeCount = Get-OrchestrationActiveCount -State $State + if ($activeCount -gt [int]$State.max_observed_active) { $State.max_observed_active = $activeCount } + return $true +} + +function Register-DelegationRejected { + param( + [Parameter(Mandatory = $true)][object]$State, + [Parameter(Mandatory = $true)][string]$WorkerId, + [Parameter(Mandatory = $true)][string]$Reason, + [switch]$CapacityLimited + ) + + $pending = @($State.pending_worker_ids) + if ($pending -notcontains $WorkerId) { + throw "Delegation rejection for '$WorkerId' is invalid because the arm is not pending." + } + $rejections = Get-OrchestrationDictionary -Object $State -Name 'delegation_rejections' + $count = if ($rejections.Contains($WorkerId)) { [int]$rejections[$WorkerId].count } else { 0 } + $rejections[$WorkerId] = [ordered]@{ + count = $count + 1 + last_reason = $Reason + last_rejected_utc = [DateTime]::UtcNow.ToString('o') + capacity_limited = [bool]$CapacityLimited + eval_attempt_started = $false + } + if ($CapacityLimited) { + $State.capacity_limit_reported = $true + } + return $true +} + +function Register-DelegationSession { + param( + [Parameter(Mandatory = $true)][object]$State, + [Parameter(Mandatory = $true)][string]$WorkerId, + [Parameter(Mandatory = $true)][string]$WorkerSessionId + ) + + if ([string]::IsNullOrWhiteSpace($WorkerSessionId)) { + throw "Runner-produced session id for '$WorkerId' is empty." + } + $active = Get-OrchestrationDictionary -Object $State -Name 'active' + if (-not $active.Contains($WorkerId)) { + throw "Worker '$WorkerId' cannot register a session because it is not active." + } + $existing = [string](Get-JsonProperty -Object $active[$WorkerId] -Name 'worker_session_id' -Default '') + if (-not [string]::IsNullOrWhiteSpace($existing) -and $existing -ne $WorkerSessionId) { + throw "Worker '$WorkerId' attempted to replace its runner-produced session id." + } + $active[$WorkerId].worker_session_id = $WorkerSessionId + return $true +} + +function Assert-OrchestrationConcurrency { + <# + The queue exposes concurrent slots; this completion gate proves that + the external orchestrator used them. A serial run is valid only when the + harness explicitly rejected additional workers for its own capacity. + #> + param( + [Parameter(Mandatory = $true)][object]$Plan, + [Parameter(Mandatory = $true)][object]$State + ) + + $requestedConcurrency = [int](Get-JsonProperty -Object $Plan -Name 'requested_concurrency' -Default 0) + $armCount = @((Get-JsonProperty -Object $Plan -Name 'arms' -Default @())).Count + $defaultRequired = if ($armCount -gt 1 -and $requestedConcurrency -gt 1) { 2 } else { 1 } + $required = [int](Get-JsonProperty -Object $Plan -Name 'minimum_parallel_workers' -Default $defaultRequired) + $maxObserved = [int](Get-JsonProperty -Object $State -Name 'max_observed_active' -Default 0) + $capacityReported = [bool](Get-JsonProperty -Object $State -Name 'capacity_limit_reported' -Default $false) + + if ($required -gt 1 -and $maxObserved -lt $required -and -not $capacityReported) { + throw "Native worker orchestration was serial: max_observed_active=$maxObserved, required_parallel_workers=$required, requested_concurrency=$requestedConcurrency. A serial dispatch is incompatible unless the harness records an explicit capacity rejection." + } + + return [ordered]@{ + status = if ($maxObserved -ge $required) { 'verified' } else { 'capacity_limited' } + requested_concurrency = $requestedConcurrency + required_parallel_workers = $required + max_observed_active = $maxObserved + capacity_limit_reported = $capacityReported + } +} + +function Register-WorkerTerminal { + param( + [Parameter(Mandatory = $true)][object]$Plan, + [Parameter(Mandatory = $true)][object]$State, + [Parameter(Mandatory = $true)][string]$WorkerId, + [Parameter(Mandatory = $true)][object]$ExecutionEvidence + ) + + $active = Get-OrchestrationDictionary -Object $State -Name 'active' + if (-not $active.Contains($WorkerId)) { + $completed = Get-OrchestrationDictionary -Object $State -Name 'completed' + $stateDescription = if ($completed.Contains($WorkerId)) { 'already terminal' } else { 'not accepted' } + throw "Worker '$WorkerId' cannot become terminal because it is $stateDescription; terminal registration is exactly-once and must not be retried." + } + $status = [string](Get-JsonProperty -Object $ExecutionEvidence -Name 'status' -Default '') + if ($status -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { + throw "Worker '$WorkerId' returned non-terminal status '$status'." + } + $arm = Get-OrchestrationArmByWorkerId -Plan $Plan -WorkerId $WorkerId + $activeWorker = $active[$WorkerId] + $expectedSessionId = [string](Get-JsonProperty -Object $activeWorker -Name 'worker_session_id' -Default '') + $terminalEvidenceFailures = [System.Collections.Generic.List[string]]::new() + + # Preserve runner-owned failure codes before the common validator runs. + foreach ($reportedFailure in @(Get-NativeWorkerReportedFailures -ExecutionEvidence $ExecutionEvidence)) { + $terminalEvidenceFailures.Add([string]$reportedFailure) + } + + # The plan stores the exact manifest-declared run path. Resolve only that + # arm here; do not infer a path from configuration or inspect grading data. + try { + $runData = Resolve-RunContract -RunPath ([string]$arm.worker.run_manifest_path) + $validation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $ExecutionEvidence -Run $runData -RequestedModel ([string]$arm.worker.model) -ExpectedWorkerSessionId $expectedSessionId -ExpectedMechanism ([string](Get-JsonProperty -Object $arm.worker -Name 'dispatch_mechanism' -Default '')) + foreach ($failure in @($validation.Failures)) { + if ($terminalEvidenceFailures -notcontains [string]$failure) { $terminalEvidenceFailures.Add([string]$failure) } + } + $evidenceSessionId = [string](Get-JsonProperty -Object $validation.Delegation -Name 'worker_session_id' -Default '') + if ($terminalEvidenceFailures.Count -eq 0 -and -not [string]::IsNullOrWhiteSpace($evidenceSessionId)) { + $completedWorkers = Get-OrchestrationDictionary -Object $State -Name 'completed' + foreach ($completedWorker in @($completedWorkers.Values)) { + if ([string](Get-JsonProperty -Object $completedWorker -Name 'worker_session_id' -Default '') -eq $evidenceSessionId) { + $terminalEvidenceFailures.Add('fresh_worker') + break + } + } + } + } catch { + $terminalEvidenceFailures.Add('terminal_evidence_unresolvable') + $terminalEvidenceFailures.Add($_.Exception.Message) + } + + # Do NOT rewrite the runner-produced terminal status. Keep it authoritative. + # Record evidence-validation separately so the execution status remains honest. + $evidenceValidationStatus = if ($terminalEvidenceFailures.Count -gt 0) { 'failed' } else { 'passed' } + + if ($status -eq 'incompatible' -and $terminalEvidenceFailures.Count -eq 0) { + # Runner explicitly reported incompatible; surface that as evidence failure + $terminalEvidenceFailures.Add('runner_reported_incompatible') + $evidenceValidationStatus = 'failed' + } + + # Persist reported failures into the execution-evidence object without + # mutating the runner-reported terminal status field. + try { [void](Set-NativeWorkerReportedFailures -ExecutionEvidence $ExecutionEvidence -Failures @($terminalEvidenceFailures.ToArray())) } catch { } + + $active.Remove($WorkerId) + $completed = Get-OrchestrationDictionary -Object $State -Name 'completed' + $completed[$WorkerId] = [ordered]@{ + worker_id = $WorkerId + eval_id = [int]$arm.eval_id + eval_name = [string]$arm.eval_name + configuration = [string]$arm.configuration + # ledger status mirrors the raw runner status exactly (authority retained) + status = $status + terminal_utc = [DateTime]::UtcNow.ToString('o') + worker_session_id = if ([string]::IsNullOrWhiteSpace($expectedSessionId)) { + Get-JsonProperty -Object (Get-JsonProperty -Object (Get-JsonProperty -Object $ExecutionEvidence -Name 'evidence' -Default $null) -Name 'delegation' -Default $null) -Name 'worker_session_id' -Default $null + } else { $expectedSessionId } + # Preserve existing compatibility field for backward consumers but do not + # let it mutate the authoritative execution status. Mark evidence level. + native_worker_evidence = if ($status -eq 'incompatible' -or $terminalEvidenceFailures.Count -gt 0) { 'incompatible' } else { 'verified' } + native_worker_evidence_failures = @($terminalEvidenceFailures.ToArray()) + evidence_validation = [ordered]@{ + status = $evidenceValidationStatus + reasons = @($terminalEvidenceFailures.ToArray()) + } + } + return $true +} + +function New-WorkerDispatchEnvelope { + param([Parameter(Mandatory = $true)][object]$Arm) + + $worker = $Arm.worker + return [ordered]@{ + type = 'eval_worker_dispatch' + worker_id = [string]$worker.worker_id + arm = [ordered]@{ + eval_id = [int]$worker.eval_id + eval_name = [string]$worker.eval_name + configuration = [string]$worker.configuration + run_manifest = [string]$worker.run_manifest + run_manifest_path = [string]$worker.run_manifest_path + } + requested = [ordered]@{ + model = [string]$worker.model + reasoning_effort = $worker.reasoning_effort + configuration_profile = $worker.configuration_profile + tool_profile = $worker.tool_profile + timeout_seconds = [int]$worker.timeout_seconds + } + worker_contract = [ordered]@{ + one_arm_only = $true + paired_arm_visible = $false + grading_material_visible = $false + parent_executes_arm = $false + dispatch_owner = [string](Get-JsonProperty -Object $worker -Name 'dispatch_owner' -Default 'orchestrator') + dispatch_mechanism = [string](Get-JsonProperty -Object $worker -Name 'dispatch_mechanism' -Default '') + runner_execute_invocation = [string](Get-JsonProperty -Object $worker -Name 'runner_execute_invocation' -Default 'forbidden') + nested_model_execution = $false + model_execution_count = 1 + fresh_worker_required = $true + } + } +} + +function Assert-OrchestrationPlanContract { + param([Parameter(Mandatory = $true)][object]$Plan) + + if ([string]$Plan.schema -ne (Get-RunnerSchemaNames).OrchestrationPlan) { + throw 'Orchestration plan has an unsupported schema.' + } + $planDispatchOwner = [string](Get-JsonProperty -Object $Plan -Name 'dispatch_owner' -Default '') + if ($planDispatchOwner -notin @('orchestrator', 'runner')) { + throw "Orchestration plan dispatch_owner '$planDispatchOwner' is unsupported." + } + if (-not [bool]$Plan.native_worker_required -or [bool]$Plan.parent_executes_arms -or [bool]$Plan.nested_model_execution) { + throw 'Orchestration plan must require native workers and forbid parent or nested model execution.' + } + $armCount = @($Plan.arms).Count + $requestedConcurrency = [int]$Plan.requested_concurrency + $expectedParallel = $armCount -gt 1 -and $requestedConcurrency -gt 1 + if ([bool](Get-JsonProperty -Object $Plan -Name 'parallel_dispatch_required' -Default $false) -ne $expectedParallel) { + throw 'Orchestration plan parallel_dispatch_required does not match its independent arm count and requested concurrency.' + } + $minimumParallelWorkers = [int](Get-JsonProperty -Object $Plan -Name 'minimum_parallel_workers' -Default 1) + $expectedMinimumParallelWorkers = if ($expectedParallel) { 2 } else { 1 } + if ($minimumParallelWorkers -ne $expectedMinimumParallelWorkers) { + throw 'Orchestration plan minimum_parallel_workers must require two workers whenever independent concurrent dispatch is requested.' + } + $workerIds = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) + foreach ($arm in @($Plan.arms)) { + if (-not $workerIds.Add([string]$arm.worker_id)) { throw "Orchestration plan duplicates worker '$($arm.worker_id)'." } + if (@($arm.depends_on).Count -ne 0) { throw "Worker '$($arm.worker_id)' has an unrelated dependency." } + $worker = $arm.worker + foreach ($property in @('one_arm_only', 'paired_arm_visible', 'grading_material_visible', 'parent_executes_arm', 'dispatch_owner', 'runner_execute_invocation', 'nested_model_execution', 'model_execution_count')) { + if (-not (Test-JsonProperty -Object $worker -Name $property)) { throw "Worker '$($arm.worker_id)' is missing '$property'." } + } + if (-not [bool]$worker.one_arm_only -or [bool]$worker.paired_arm_visible -or [bool]$worker.grading_material_visible -or [bool]$worker.parent_executes_arm -or [bool]$worker.nested_model_execution -or [int]$worker.model_execution_count -ne 1) { + throw "Worker '$($arm.worker_id)' violates the one-arm/one-model contract." + } + if ([string]$worker.worker_id -ne [string]$arm.worker_id -or [int]$worker.eval_id -ne [int]$arm.eval_id -or [string]$worker.configuration -ne [string]$arm.configuration) { + throw "Worker '$($arm.worker_id)' does not identify exactly its manifest arm." + } + if ([string]$arm.dispatch_owner -ne $planDispatchOwner -or [string]$worker.dispatch_owner -ne $planDispatchOwner) { + throw "Worker '$($arm.worker_id)' dispatch ownership does not match the plan." + } + $expectedRunnerExecute = if ($planDispatchOwner -eq 'runner') { 'required' } else { 'forbidden' } + if ([string]$worker.runner_execute_invocation -ne $expectedRunnerExecute) { + throw "Worker '$($arm.worker_id)' has runner_execute_invocation '$($worker.runner_execute_invocation)'; expected '$expectedRunnerExecute' for dispatch owner '$planDispatchOwner'." + } + foreach ($forbiddenProperty in @('paired_arm', 'grading', 'expected_output', 'assertions', 'eval_metadata', 'execution_result', 'result')) { + if (Test-JsonProperty -Object $worker -Name $forbiddenProperty) { + throw "Worker '$($arm.worker_id)' exposes forbidden parent or grading field '$forbiddenProperty'." + } + } + } + return $true +} diff --git a/scripts/eval-runners/package-integrity.ps1 b/scripts/eval-runners/package-integrity.ps1 new file mode 100644 index 0000000..51ff8c2 --- /dev/null +++ b/scripts/eval-runners/package-integrity.ps1 @@ -0,0 +1,44 @@ +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +function Get-PackageTreeIntegrity { + param([Parameter(Mandatory = $true)][string]$Root) + + $resolvedRoot = (Resolve-Path -LiteralPath $Root -ErrorAction Stop).Path + $entries = [System.Collections.Generic.List[string]]::new() + foreach ($file in (Get-ChildItem -LiteralPath $resolvedRoot -Recurse -File -Force | Sort-Object FullName)) { + $relative = [System.IO.Path]::GetRelativePath($resolvedRoot, $file.FullName).Replace('\', '/') + $entries.Add("$relative`:$(Get-Sha256HexFromFile -Path $file.FullName)") + } + $joined = [string]::Join("`n", @($entries | Sort-Object)) + return [pscustomobject]@{ + Path = $resolvedRoot + FileCount = $entries.Count + Sha256 = Get-Sha256HexFromBytes -Bytes ([System.Text.UTF8Encoding]::new($false).GetBytes($joined)) + } +} + +function Assert-PackageRunnerToolsIntegrity { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest + ) + + $declared = Get-JsonProperty -Object $Manifest -Name 'runner_tools_integrity' -Default $null + if ($null -eq $declared -or [string](Get-JsonProperty -Object $declared -Name 'schema' -Default '') -ne 'codebeltnet/agentic/package-tree-integrity/1') { + throw 'manifest.json does not declare the versioned package-local Eval Runner tool integrity record.' + } + $runnerToolsRelative = [string](Get-JsonProperty -Object $Manifest -Name 'runner_tools' -Default '') + $declaredPath = [string](Get-JsonProperty -Object $declared -Name 'path' -Default '') + if ([string]::IsNullOrWhiteSpace($runnerToolsRelative) -or $declaredPath -ne $runnerToolsRelative) { + throw 'manifest.runner_tools_integrity.path does not match manifest.runner_tools.' + } + $runnerToolsPath = Resolve-ContainedPath -BasePath $IterationDirectory -RelativePath $runnerToolsRelative -FieldName 'runner_tools' -Kind Directory + $actual = Get-PackageTreeIntegrity -Root $runnerToolsPath + $declaredHash = [string](Get-JsonProperty -Object $declared -Name 'sha256' -Default '') + $declaredCount = [int](Get-JsonProperty -Object $declared -Name 'file_count' -Default -1) + if ($declaredHash -ne [string]$actual.Sha256 -or $declaredCount -ne [int]$actual.FileCount) { + throw "Package-local Eval Runner tools changed after preparation (expected hash $declaredHash/$declaredCount files, found $($actual.Sha256)/$($actual.FileCount) files). Requires a fresh package." + } + return $actual +} diff --git a/scripts/eval-runners/record-native-result.ps1 b/scripts/eval-runners/record-native-result.ps1 new file mode 100644 index 0000000..8047ae1 --- /dev/null +++ b/scripts/eval-runners/record-native-result.ps1 @@ -0,0 +1,221 @@ +<#! +.SYNOPSIS + Records a native worker terminal envelope as the runner-owned raw result. + +.DESCRIPTION + This command is deterministic. It never starts a harness or a model. The + external orchestrator uses it after a harness-native worker has finished so + that the selected package runner, rather than the orchestrator, owns the + eval-execution-result/1 serialization boundary. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)][string]$Runner, + [Parameter(Mandatory = $true)][string]$Run, + [Parameter(Mandatory = $true)][string]$Profile, + [Parameter(Mandatory = $true)][string]$NativeResult, + [Parameter(Mandatory = $true)][string]$Output +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot 'runner-common.ps1') + +function ConvertTo-StringDictionary { + param([Parameter(Mandatory = $true)][object]$Value) + + $result = [ordered]@{} + foreach ($name in @(Get-JsonPropertyNames -Object $Value)) { + $result[[string]$name] = [string](Get-JsonProperty -Object $Value -Name ([string]$name) -Default '') + } + return $result +} + +function Assert-NativeResultEnvelope { + param( + [Parameter(Mandatory = $true)][object]$Native, + [Parameter(Mandatory = $true)][object]$RunData + ) + + if ([string]$Native.schema -ne 'codebeltnet/agentic/eval-native-worker-result/1') { + throw "Native worker result must declare 'codebeltnet/agentic/eval-native-worker-result/1'." + } + + foreach ($field in @('run_id', 'session', 'status', 'run', 'final_response', 'timing', 'exit', 'isolation', 'telemetry', 'evidence', 'capture', 'artifacts', 'warnings', 'compatibility_deviations', 'attempt_count')) { + if (-not (Test-JsonProperty -Object $Native -Name $field)) { + throw "Native worker result is missing '$field'." + } + } + if ([string]::IsNullOrWhiteSpace([string]$Native.run_id)) { + throw 'Native worker result run_id must be non-empty.' + } + if ([string]$Native.status -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { + throw "Native worker result status '$($Native.status)' is unsupported." + } + if ([int]$Native.attempt_count -ne 1) { + throw 'Native worker result attempt_count must be exactly 1.' + } + + $runIdentity = $Native.run + if ([int]$runIdentity.eval_id -ne [int]$RunData.EvalId -or + [string]$runIdentity.eval_name -ne [string]$RunData.EvalName -or + [string]$runIdentity.configuration -ne [string]$RunData.Mode) { + throw 'Native worker result run identity does not match run.json.' + } + + $session = $Native.session + if ([string]::IsNullOrWhiteSpace([string]$session.id) -or + -not [bool]$session.fresh -or [bool]$session.resumed) { + throw 'Native worker result must identify a fresh, non-resumed session.' + } + + $response = $Native.final_response + if ([string]$response.status -eq 'available') { + if (-not (Test-JsonProperty -Object $response -Name 'text')) { + throw 'Available native worker responses must contain text.' + } + } elseif ([string]$response.status -eq 'unavailable') { + if ([string]::IsNullOrWhiteSpace([string]$response.reason)) { + throw 'Unavailable native worker responses must contain a reason.' + } + } else { + throw "Native worker final_response status '$($response.status)' is unsupported." + } + + $timing = $Native.timing + foreach ($field in @('started_utc', 'finished_utc', 'duration_seconds')) { + if (-not (Test-JsonProperty -Object $timing -Name $field)) { + throw "Native worker result timing.$field must be present." + } + } + try { + $started = [DateTime]::Parse([string]$timing.started_utc).ToUniversalTime() + $finished = [DateTime]::Parse([string]$timing.finished_utc).ToUniversalTime() + } catch { + throw "Native worker result timing timestamps are invalid: $($_.Exception.Message)" + } + if ($finished -lt $started -or [double]$timing.duration_seconds -lt 0) { + throw 'Native worker result timing must be ordered and non-negative.' + } + + $exit = $Native.exit + if (-not (Test-JsonProperty -Object $exit -Name 'status')) { + throw 'Native worker result exit.status must be present and numeric or null.' + } + $exitStatus = Get-JsonProperty -Object $exit -Name 'status' -Default $null + if ($null -ne $exitStatus -and -not ($exitStatus -is [byte] -or $exitStatus -is [sbyte] -or $exitStatus -is [int16] -or $exitStatus -is [uint16] -or $exitStatus -is [int32] -or $exitStatus -is [uint32] -or $exitStatus -is [int64] -or $exitStatus -is [uint64])) { + throw 'Native worker result exit.status must be a JSON number or null.' + } + + $isolation = $Native.isolation + if (-not (Test-JsonProperty -Object $isolation -Name 'capabilities') -or + -not (Test-JsonProperty -Object $isolation -Name 'mechanisms')) { + throw 'Native worker result isolation must declare capabilities and mechanisms.' + } + if (@(Get-JsonProperty -Object $isolation -Name 'mechanisms' -Default @()).Count -eq 0) { + throw 'Native worker result isolation.mechanisms must not be empty.' + } + if (-not (Test-JsonProperty -Object $Native.evidence -Name 'delegation')) { + throw 'Native worker result evidence.delegation must be present.' + } + $capture = $Native.capture + if ([string]$capture.source -ne 'harness_native_transport' -or + -not [bool]$capture.terminal -or [bool]$capture.worker_authored) { + throw 'Native worker result capture must come from the terminal harness-native transport; the worker may not author the envelope.' + } + if (@($Native.artifacts).Count -lt 1) { + throw 'Native worker result must record at least one artifact, including terminal evidence.' + } +} + +try { + $runData = Resolve-RunContract -RunPath $Run + [void](Assert-PhaseOneEvidenceWritable -Run $runData) + $profileData = Resolve-ExecutionProfile -ProfilePath $Profile + if ([string]$profileData.Runner -ne $Runner) { + throw "Selected runner '$Runner' does not match execution-profile.json runner '$($profileData.Runner)'." + } + + $iterationDirectory = Split-Path -Parent (Split-Path -Parent $runData.RunRoot) + $nativePath = (Resolve-Path -LiteralPath $NativeResult -ErrorAction Stop).Path + $outputPath = [System.IO.Path]::GetFullPath($Output, (Get-Location).Path) + if (-not (Test-PathInside -BasePath $iterationDirectory -CandidatePath $nativePath)) { + throw 'Native worker result input must remain inside the prepared iteration package.' + } + if (-not (Test-PathInside -BasePath $iterationDirectory -CandidatePath $outputPath)) { + throw 'Recorded execution result output must remain inside the prepared iteration package.' + } + $outputDirectory = Split-Path -Parent $outputPath + if (-not (Test-Path -LiteralPath $outputDirectory -PathType Container)) { + throw "Recorded execution result directory '$outputDirectory' does not exist." + } + + $native = Read-RunnerJson -Path $nativePath + Assert-NativeResultEnvelope -Native $native -RunData $runData + $descriptor = Get-PackageRunnerDescriptor -RunnerName $Runner + + $capabilities = ConvertTo-StringDictionary -Value $native.isolation.capabilities + $response = $native.final_response + $finalText = if ([string]$response.status -eq 'available') { [string]$response.text } else { $null } + $finalReason = if ([string]$response.status -eq 'unavailable') { [string]$response.reason } else { $null } + $exitStatusValue = Get-JsonProperty -Object $native.exit -Name 'status' -Default $null + $exitStatus = if ($null -eq $exitStatusValue) { $null } else { [int]$exitStatusValue } + $resolvedConfiguration = Get-JsonProperty -Object $native -Name 'resolved' -Default $null + $evidence = Get-JsonProperty -Object $native -Name 'evidence' -Default ([ordered]@{}) + if ($evidence -is [System.Collections.IDictionary]) { + $evidence['capture'] = $native.capture + } else { + Add-Member -InputObject $evidence -MemberType NoteProperty -Name capture -Value $native.capture -Force + } + $result = New-ExecutionResult ` + -Descriptor $descriptor ` + -Profile $profileData ` + -Run $runData ` + -Status ([string]$native.status) ` + -FinalResponse $finalText ` + -FinalResponseReason $finalReason ` + -StartedUtc ([string]$native.timing.started_utc) ` + -FinishedUtc ([string]$native.timing.finished_utc) ` + -DurationSeconds ([double]$native.timing.duration_seconds) ` + -ExitStatus $exitStatus ` + -Failure (Get-JsonProperty -Object $native.exit -Name 'failure' -Default $null) ` + -SessionId ([string]$native.session.id) ` + -IsolationCapabilities $capabilities ` + -IsolationMechanisms @((Get-JsonProperty -Object $native.isolation -Name 'mechanisms' -Default @())) ` + -ResolvedConfiguration $resolvedConfiguration ` + -Telemetry (Get-JsonProperty -Object $native -Name 'telemetry' -Default $null) ` + -Artifacts @((Get-JsonProperty -Object $native -Name 'artifacts' -Default @())) ` + -Warnings @((Get-JsonProperty -Object $native -Name 'warnings' -Default @())) ` + -CompatibilityDeviations @((Get-JsonProperty -Object $native -Name 'compatibility_deviations' -Default @())) ` + -Evidence $evidence ` + -AttemptCount 1 + $result.run_id = [string]$native.run_id + + [void](Assert-ExecutionResult -Result $result) + if ([string]$result.status -ne 'incompatible') { + $terminalValidation = Test-NativeWorkerTerminalEvidence ` + -ExecutionEvidence $result ` + -Run $runData ` + -RequestedModel ([string]$profileData.Model) ` + -ExpectedWorkerSessionId ([string]$native.session.id) ` + -ExpectedRunner $Runner ` + -ExpectedMechanism ([string]$descriptor.delegation.mechanism) + if (-not $terminalValidation.Valid) { + throw "Native worker terminal evidence is incompatible: $([string]::Join(', ', @($terminalValidation.Failures)))." + } + Assert-NativeTerminalCaptureArtifact -ExecutionResult $result + } + + $utf8NoBom = [System.Text.UTF8Encoding]::new($false) + [System.IO.File]::WriteAllText($outputPath, ((ConvertTo-Json -InputObject $result -Depth 100) + [Environment]::NewLine), $utf8NoBom) + $relativeOutput = [System.IO.Path]::GetRelativePath($iterationDirectory, $outputPath).Replace('\', '/') + Write-RunnerJson -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-native-worker-record/1' + runner = $Runner + execution_result = $relativeOutput + execution_status = $result.status + }) -AsOutput +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/resolve-runner.ps1 b/scripts/eval-runners/resolve-runner.ps1 new file mode 100644 index 0000000..9517dac --- /dev/null +++ b/scripts/eval-runners/resolve-runner.ps1 @@ -0,0 +1,33 @@ +<#! +.SYNOPSIS + Resolves one package-local Eval Runner without guessing or falling back. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [string]$Runner +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$protocol = 'codebeltnet/agentic/eval-runner-protocol/1' +if ($Runner -notmatch '^[a-z0-9][a-z0-9-]*$') { + throw "Runner name '$Runner' is not a safe package-local runner name." +} + +$runnerPath = Join-Path (Join-Path $PSScriptRoot $Runner) 'runner.ps1' +if (-not (Test-Path -LiteralPath $runnerPath -PathType Leaf)) { + throw "Selected Eval Runner '$Runner' is unavailable in this package." +} + +$resolved = (Resolve-Path -LiteralPath $runnerPath).Path +$root = (Resolve-Path -LiteralPath $PSScriptRoot).Path +$relative = [System.IO.Path]::GetRelativePath($root, $resolved).Replace('\', '/') + +[ordered]@{ + schema = 'codebeltnet/agentic/eval-runner-resolution/1' + protocol_version = $protocol + runner = $Runner + path = $relative +} | ConvertTo-Json -Depth 10 -Compress diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 new file mode 100644 index 0000000..a729279 --- /dev/null +++ b/scripts/eval-runners/runner-common.ps1 @@ -0,0 +1,2117 @@ +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +function Format-UtcTimestamp { + param([Parameter(Mandatory = $true)][DateTime]$Value) + + return $Value.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ss.fffZ', [Globalization.CultureInfo]::InvariantCulture) +} + +function Get-RunnerSchemaNames { + return [ordered]@{ + Protocol = 'codebeltnet/agentic/eval-runner-protocol/1' + Descriptor = 'codebeltnet/agentic/eval-runner-descriptor/1' + Preflight = 'codebeltnet/agentic/eval-runner-preflight/1' + Profile = 'codebeltnet/agentic/eval-execution-profile/1' + Result = 'codebeltnet/agentic/eval-execution-result/1' + PortableResult = 'codebeltnet/agentic/eval-result/2' + Run = 'codebeltnet/agentic/eval-run/1' + OrchestrationPlan = 'codebeltnet/agentic/eval-orchestration-plan/1' + Interaction = 'codebeltnet/agentic/eval-interaction/1' + ExecutionFreeze = 'codebeltnet/agentic/eval-execution-freeze/1' + Grading = 'codebeltnet/agentic/eval-grading/1' + } +} + +function Get-PackageRunnerDescriptor { + param([Parameter(Mandatory = $true)][string]$RunnerName) + + if ($RunnerName -notmatch '^[a-z0-9-]+$') { + throw "execution-profile.json runner '$RunnerName' is not a valid package runner name." + } + + $runnerPath = Join-Path (Join-Path $PSScriptRoot $RunnerName) 'runner.ps1' + if (-not (Test-Path -LiteralPath $runnerPath -PathType Leaf)) { + throw "Package-local runner '$RunnerName' is missing its runner.ps1 descriptor." + } + + $pwshPath = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) + $descriptorProcess = Invoke-RunnerProcess -FileName $pwshPath -ArgumentList @('-NoProfile', '-NonInteractive', '-File', $runnerPath, 'describe') -WorkingDirectory $PSScriptRoot -Environment (New-RunnerProbeEnvironment) -TimeoutSeconds 30 + if ($descriptorProcess.TimedOut) { + throw "Package-local runner '$RunnerName' descriptor exceeded the 30-second model-free probe timeout." + } + if ($descriptorProcess.ExitCode -ne 0) { + throw "Package-local runner '$RunnerName' descriptor failed: $([string]::Join(' ', @($descriptorProcess.Stdout, $descriptorProcess.Stderr)))" + } + + try { + $descriptor = [string]$descriptorProcess.Stdout | ConvertFrom-Json + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + } catch { + throw "Package-local runner '$RunnerName' returned an invalid descriptor: $($_.Exception.Message)" + } + + if ([string]$descriptor.name -ne $RunnerName) { + throw "Package-local runner descriptor name '$($descriptor.name)' does not match selected runner '$RunnerName'." + } + + return $descriptor +} + +function Get-JsonProperty { + param( + [object]$Object, + [string]$Name, + [object]$Default = $null + ) + + if ($null -ne $Object -and $Object -is [System.Collections.IDictionary] -and $Object.Contains($Name)) { + if ($null -ne $Object[$Name]) { + return $Object[$Name] + } + return $Default + } + + if ($null -ne $Object -and @($Object.PSObject.Properties | ForEach-Object { [string]$_.Name }) -contains $Name -and $null -ne $Object.$Name) { + return $Object.$Name + } + + return $Default +} + +function Get-JsonPropertyNames { + param([object]$Object) + + if ($null -eq $Object) { + return @() + } + if ($Object -is [System.Collections.IDictionary]) { + return @($Object.Keys | ForEach-Object { [string]$_ }) + } + return @($Object.PSObject.Properties | ForEach-Object { [string]$_.Name }) +} + +function Get-JsonWithoutProperty { + param( + [Parameter(Mandatory = $true)][object]$Object, + [Parameter(Mandatory = $true)][string]$PropertyName + ) + + if ($Object -is [System.Collections.IDictionary]) { + $copy = [ordered]@{} + foreach ($key in $Object.Keys) { + if ([string]$key -ne $PropertyName) { $copy[[string]$key] = $Object[$key] } + } + return $copy + } + $copyObject = [ordered]@{} + foreach ($property in @($Object.PSObject.Properties)) { + if ([string]$property.Name -ne $PropertyName) { $copyObject[[string]$property.Name] = $property.Value } + } + return $copyObject +} + +function ConvertTo-CanonicalJsonValue { + param([AllowNull()][object]$Value) + + if ($null -eq $Value -or $Value -is [string] -or $Value -is [ValueType]) { return $Value } + if ($Value -is [System.Collections.IDictionary]) { + $canonical = [ordered]@{} + foreach ($key in @($Value.Keys | ForEach-Object { [string]$_ } | Sort-Object)) { + $canonical[$key] = ConvertTo-CanonicalJsonValue -Value $Value[$key] + } + return $canonical + } + if ($Value -is [System.Collections.IEnumerable]) { + $canonical = [System.Collections.Generic.List[object]]::new() + foreach ($item in $Value) { $canonical.Add((ConvertTo-CanonicalJsonValue -Value $item)) } + return @($canonical.ToArray()) + } + $canonicalObject = [ordered]@{} + foreach ($property in @($Value.PSObject.Properties | Sort-Object Name)) { + $canonicalObject[[string]$property.Name] = ConvertTo-CanonicalJsonValue -Value $property.Value + } + return $canonicalObject +} + +function Get-JsonFingerprint { + param([Parameter(Mandatory = $true)][object]$Object) + + $json = ConvertTo-CanonicalJsonValue -Value $Object | ConvertTo-Json -Depth 100 -Compress + return Get-Sha256HexFromBytes -Bytes ([System.Text.UTF8Encoding]::new($false).GetBytes($json)) +} + +function Test-JsonProperty { + param( + [object]$Object, + [Parameter(Mandatory = $true)][string]$Name + ) + + return (Get-JsonPropertyNames -Object $Object) -contains $Name +} + +function Read-RunnerJson { + param([Parameter(Mandatory = $true)][string]$Path) + + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { + throw "JSON file '$Path' does not exist." + } + + $json = [System.IO.File]::ReadAllText((Resolve-Path -LiteralPath $Path).Path, [System.Text.UTF8Encoding]::new($false)) + $convertFromJson = Get-Command ConvertFrom-Json -ErrorAction Stop + if ($convertFromJson.Parameters.ContainsKey('DateKind')) { + return $json | ConvertFrom-Json -DateKind String + } + + # DateKind was added after the oldest supported PowerShell 7 releases. + # Keep those versions usable; the canonical writers still emit UTC strings. + return $json | ConvertFrom-Json +} + +function Write-RunnerJson { + param( + [Parameter(Mandatory = $true)][object]$Value, + [switch]$AsOutput + ) + + $json = ((ConvertTo-Json -InputObject $Value -Depth 100) + [Environment]::NewLine) + if ($AsOutput) { + [Console]::Out.Write($json) + return + } + + return $json +} + +function Get-Sha256HexFromBytes { + param([Parameter(Mandatory = $true)][AllowEmptyCollection()][byte[]]$Bytes) + + $sha = [System.Security.Cryptography.SHA256]::Create() + try { + return ([Convert]::ToHexString($sha.ComputeHash($Bytes))).ToLowerInvariant() + } finally { + $sha.Dispose() + } +} + +function Get-Sha256HexFromFile { + param([Parameter(Mandatory = $true)][string]$Path) + + $resolved = (Resolve-Path -LiteralPath $Path -ErrorAction Stop).Path + return Get-Sha256HexFromBytes -Bytes ([System.IO.File]::ReadAllBytes($resolved)) +} + +function Test-Sha256 { + param([string]$Value) + + return -not [string]::IsNullOrWhiteSpace($Value) -and $Value -match '^[0-9a-fA-F]{64}$' +} + +function Expand-WindowsShortPath { + param([Parameter(Mandatory = $true)][string]$Path) + + if (-not $IsWindows -or [string]::IsNullOrWhiteSpace($Path)) { return $Path } + try { + # Windows APIs and .NET can return either the long or 8.3 spelling of + # the same existing path. Normalize that spelling before comparing + # runner-observed paths; this does not resolve symlinks or authorize a + # path outside the requested boundary. + $fullPath = [System.IO.Path]::GetFullPath($Path) + $root = [System.IO.Path]::GetPathRoot($fullPath) + if (-not [string]::IsNullOrWhiteSpace($root)) { + # DirectoryInfo expands an 8.3 component only when that component + # is the final component. Walk existing components so a short + # parent such as ADMINI~1 is expanded before comparing a nested + # runner path. + $current = $root + $remaining = $fullPath.Substring($root.Length) -split '[\\/]' + for ($componentIndex = 0; $componentIndex -lt $remaining.Count; $componentIndex++) { + $component = [string]$remaining[$componentIndex] + if ([string]::IsNullOrWhiteSpace($component)) { continue } + $next = Join-Path -Path $current -ChildPath $component + if (Test-Path -LiteralPath $next -PathType Container) { + $current = ([System.IO.DirectoryInfo]::new($next)).FullName + } elseif (Test-Path -LiteralPath $next -PathType Leaf) { + $current = ([System.IO.FileInfo]::new($next)).FullName + } else { + $current = Join-Path -Path $current -ChildPath $component + if ($componentIndex + 1 -lt $remaining.Count) { + $current = Join-Path -Path $current -ChildPath ([string]::Join([System.IO.Path]::DirectorySeparatorChar, @($remaining[($componentIndex + 1)..($remaining.Count - 1)] | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) }))) + } + break + } + } + if (-not [string]::IsNullOrWhiteSpace($current)) { return $current } + } + if ($null -eq ([System.Management.Automation.PSTypeName]'CodebeltAgenticWin32Path').Type) { + Add-Type -TypeDefinition @' +using System; +using System.Text; +using System.Runtime.InteropServices; + +public static class CodebeltAgenticWin32Path +{ + [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)] + private static extern uint GetLongPathName(string shortPath, StringBuilder longPath, uint longPathLength); + + public static string Expand(string path) + { + uint required = GetLongPathName(path, null, 0); + if (required == 0) return path; + + var buffer = new StringBuilder((int)required + 1); + uint written = GetLongPathName(path, buffer, (uint)buffer.Capacity); + return written == 0 ? path : buffer.ToString(); + } +} +'@ -ErrorAction Stop | Out-Null + } + return [CodebeltAgenticWin32Path]::Expand($Path) + } catch { + return $Path + } +} + +function Test-PathInside { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$CandidatePath + ) + + $base = ConvertTo-ComparablePath -Path $BasePath + $candidate = ConvertTo-ComparablePath -Path $CandidatePath + if ($null -eq $base -or $null -eq $candidate) { return $false } + return $candidate -eq $base -or $candidate.StartsWith($base + [System.IO.Path]::DirectorySeparatorChar, [System.StringComparison]::OrdinalIgnoreCase) +} + +function Assert-SafeRelativePath { + param( + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][string]$FieldName + ) + + if ([string]::IsNullOrWhiteSpace($RelativePath) -or [System.IO.Path]::IsPathRooted($RelativePath) -or $RelativePath -match '^[A-Za-z]:') { + throw "$FieldName must be a non-empty relative path." + } + + $normalized = $RelativePath.Replace('\', '/') + if (($normalized -split '/') -contains '..') { + throw "$FieldName must not contain a parent-directory segment." + } +} + +function Resolve-ContainedPath { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][string]$FieldName, + [ValidateSet('Any', 'File', 'Directory')][string]$Kind = 'Any' + ) + + Assert-SafeRelativePath -RelativePath $RelativePath -FieldName $FieldName + $resolvedBase = (Resolve-Path -LiteralPath $BasePath -ErrorAction Stop).Path + $candidate = [System.IO.Path]::GetFullPath((Join-Path $resolvedBase ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + if (-not (Test-PathInside -BasePath $resolvedBase -CandidatePath $candidate)) { + throw "$FieldName resolves outside the run directory." + } + + $exists = switch ($Kind) { + 'File' { Test-Path -LiteralPath $candidate -PathType Leaf } + 'Directory' { Test-Path -LiteralPath $candidate -PathType Container } + default { Test-Path -LiteralPath $candidate } + } + if (-not $exists) { + throw "$FieldName '$RelativePath' does not exist under '$resolvedBase'." + } + + $resolvedCandidate = (Resolve-Path -LiteralPath $candidate -ErrorAction Stop).Path + if (-not (Test-PathInside -BasePath $resolvedBase -CandidatePath $resolvedCandidate)) { + throw "$FieldName resolves through a link outside the run directory." + } + return $resolvedCandidate +} + +function Get-PlatformName { + if ($IsWindows) { return 'windows' } + if ($IsMacOS) { return 'macos' } + if ($IsLinux) { return 'linux' } + return 'unknown' +} + +function Get-SandboxVisiblePath { + param( + [Parameter(Mandatory = $true)][string]$HostPath, + [Parameter(Mandatory = $true)][string]$RunRoot, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName), + [string]$MountRoot = '/run' + ) + + $fullHostPath = [System.IO.Path]::GetFullPath($HostPath) + if ($Platform -ne 'linux' -or -not (Test-PathInside -BasePath $RunRoot -CandidatePath $fullHostPath)) { + return $fullHostPath + } + + $relative = [System.IO.Path]::GetRelativePath(([System.IO.Path]::GetFullPath($RunRoot)), $fullHostPath).Replace('\', '/') + if ($relative -eq '.') { + return $MountRoot.TrimEnd('/') + } + return $MountRoot.TrimEnd('/') + '/' + $relative.TrimStart('/') +} + +function Get-ObservableVersionFromText { + param([string]$Text) + + foreach ($line in ($Text -split "`r?`n")) { + $trimmed = $line.Trim() + if (-not [string]::IsNullOrWhiteSpace($trimmed)) { + return $trimmed + } + } + return $null +} + +function Get-RunnerSystemDirectorySet { + # System/OS directories that must never be used as writable scratch, even by + # an elevated process. A model-free probe that resolves its temp here is the + # iteration-11 failure (GetTempPath() -> %WINDIR% when TEMP/TMP/USERPROFILE + # are absent), so any temp candidate resolving to one of these is rejected. + $set = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) + $folders = @( + [Environment]::GetEnvironmentVariable('WINDIR'), + [Environment]::GetEnvironmentVariable('SystemRoot'), + [Environment]::GetFolderPath([Environment+SpecialFolder]::Windows), + [Environment]::GetFolderPath([Environment+SpecialFolder]::System), + [Environment]::GetFolderPath([Environment+SpecialFolder]::SystemX86) + ) + foreach ($value in $folders) { + if ([string]::IsNullOrWhiteSpace($value)) { continue } + try { [void]$set.Add(([System.IO.Path]::GetFullPath($value)).TrimEnd('\', '/')) } catch { } + } + return $set +} + +function Test-RunnerDirectoryWritable { + param([Parameter(Mandatory = $true)][string]$Path) + + try { + if (-not (Test-Path -LiteralPath $Path -PathType Container)) { + New-Item -ItemType Directory -Path $Path -Force -ErrorAction Stop | Out-Null + } + $probe = Join-Path $Path ('.agentic-writable-probe-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $probe -Force -ErrorAction Stop | Out-Null + Remove-Item -LiteralPath $probe -Recurse -Force -ErrorAction SilentlyContinue + return $true + } catch { + return $false + } +} + +function Resolve-RunnerProbeTempRoot { + <# + Resolves one explicit, writable scratch directory for a MODEL-FREE harness + probe (a --version/--help or `describe` capability check). A model-free + probe is an ordinary, possibly non-elevated OS process: it needs a writable + temporary directory and has nothing to do with eval skill isolation. This + resolver deliberately avoids the implicit [System.IO.Path]::GetTempPath() + fallback that resolves to %WINDIR% when TEMP/TMP/USERPROFILE are stripped, + and it never returns the Windows/system directory even for an elevated user. + #> + $systemDirectories = Get-RunnerSystemDirectorySet + $candidates = [System.Collections.Generic.List[string]]::new() + foreach ($name in @('TEMP', 'TMP')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { $candidates.Add($value) } + } + $localAppData = [Environment]::GetEnvironmentVariable('LOCALAPPDATA') + if (-not [string]::IsNullOrWhiteSpace($localAppData)) { $candidates.Add((Join-Path $localAppData 'Temp')) } + try { + $frameworkTemp = [System.IO.Path]::GetTempPath() + if (-not [string]::IsNullOrWhiteSpace($frameworkTemp)) { $candidates.Add($frameworkTemp) } + } catch { } + + foreach ($candidate in $candidates) { + if ([string]::IsNullOrWhiteSpace($candidate)) { continue } + $full = $null + try { $full = [System.IO.Path]::GetFullPath($candidate) } catch { continue } + if ($systemDirectories.Contains($full.TrimEnd('\', '/'))) { continue } + if (Test-RunnerDirectoryWritable -Path $full) { return $full } + } + + throw 'Unable to resolve a writable model-free probe temporary directory from TEMP, TMP, or LOCALAPPDATA. A model-free harness probe must not require write access to the Windows system directory.' +} + +function New-RunnerProbeEnvironment { + <# + Builds the environment for a MODEL-FREE harness probe (a --version/--help or + `describe` capability check). This is intentionally NOT the model-backed eval + isolation environment; New-RunnerEnvironment builds that. The distinction is + load-bearing and must not be merged: a probe is an ordinary OS process, so it + is given an explicit writable TEMP/TMP, but it carries no HOME/USERPROFILE and + no ambient skill/config roots, so it can never establish or stand in for the + eval isolation environment. TEMP/TMP are OS scratch infrastructure resolved + explicitly here so neither this process nor a child launched with this + environment falls back to %WINDIR% via GetTempPath() (the iteration-11 bug). + #> + $environment = [ordered]@{} + foreach ($name in @('PATH', 'SystemRoot', 'WINDIR', 'ComSpec', 'PATHEXT', 'LANG', 'LC_ALL', 'TZ', 'SSL_CERT_FILE', 'NODE_PATH')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + $probeTempRoot = Resolve-RunnerProbeTempRoot + $environment['TEMP'] = $probeTempRoot + $environment['TMP'] = $probeTempRoot + $environment['CI'] = '1' + $environment['NO_COLOR'] = '1' + return $environment +} + +function Get-ExternalCommandVersion { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [string]$WorkingDirectory = '', + [System.Collections.IDictionary]$Environment = (New-RunnerProbeEnvironment), + [int]$TimeoutSeconds = 30 + ) + + $probeDirectory = $WorkingDirectory + $ownsProbeDirectory = $false + if ([string]::IsNullOrWhiteSpace($probeDirectory)) { + # Use the explicit writable probe temp carried by the probe environment + # rather than GetTempPath(), which resolves to %WINDIR% inside a stripped + # probe child. Fall back to an explicit resolution only if the caller + # supplied an environment without TEMP. + $probeTempRoot = if ($null -ne $Environment -and $Environment.Contains('TEMP') -and -not [string]::IsNullOrWhiteSpace([string]$Environment['TEMP'])) { + [string]$Environment['TEMP'] + } else { + Resolve-RunnerProbeTempRoot + } + $probeDirectory = Join-Path $probeTempRoot ('agentic-version-probe-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $probeDirectory -Force | Out-Null + $ownsProbeDirectory = $true + } + try { + $process = Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList (@($CommandInfo.Prefix) + @('--version')) -WorkingDirectory $probeDirectory -Environment $Environment -TimeoutSeconds $TimeoutSeconds + $text = [string]::Join("`n", @($process.Stdout, $process.Stderr)) + $version = Get-ObservableVersionFromText -Text $text + return [pscustomobject]@{ + Version = if ($process.TimedOut -or $process.ExitCode -ne 0 -or [string]::IsNullOrWhiteSpace($version)) { 'unavailable' } else { $version } + Available = (-not $process.TimedOut -and $process.ExitCode -eq 0 -and -not [string]::IsNullOrWhiteSpace($version)) + Process = $process + } + } catch { + return [pscustomobject]@{ Version = 'unavailable'; Available = $false; Process = $null; Error = $_.Exception.Message } + } finally { + if ($ownsProbeDirectory -and (Test-Path -LiteralPath $probeDirectory)) { + Remove-Item -LiteralPath $probeDirectory -Recurse -Force -ErrorAction SilentlyContinue + } + } +} + +function Get-IsolationCapabilityAssessment { + param([System.Collections.IDictionary]$Capabilities) + + $required = @( + 'fresh_context', + 'isolated_home_config', + 'isolated_working_directory', + 'ambient_candidate_skill_exclusion', + 'candidate_skill_exposure', + 'prompt_fidelity', + 'model_configuration_lock', + 'response_capture' + ) + $unproven = [System.Collections.Generic.List[string]]::new() + foreach ($name in $required) { + $value = if ($null -ne $Capabilities -and $Capabilities.Contains($name)) { [string]$Capabilities[$name] } else { 'unavailable' } + $valid = if ($name -eq 'candidate_skill_exposure') { $value -in @('supported', 'excluded') } else { $value -eq 'supported' } + if (-not $valid) { $unproven.Add($name) } + } + + $filesystemValue = if ($null -ne $Capabilities -and $Capabilities.Contains('filesystem_confinement')) { [string]$Capabilities['filesystem_confinement'] } else { 'unavailable' } + $hardFilesystem = $filesystemValue -eq 'supported' + $mandatoryProven = $unproven.Count -eq 0 + return [pscustomobject]@{ + MandatoryProven = $mandatoryProven + HardFilesystemConfinement = $hardFilesystem + Level = if (-not $mandatoryProven) { 'unsupported' } elseif ($hardFilesystem) { 'strict' } else { 'pragmatic' } + Unproven = $unproven.ToArray() + Required = @($required) + } +} + +function Get-DelegationCapabilityAssessment { + param( + [Parameter(Mandatory = $true)][object]$Descriptor, + [System.Collections.IDictionary]$Capabilities + ) + + $required = @( + 'native_worker_delegation', + 'delegated_worker_full_capability', + 'delegated_worker_model_lock', + 'delegated_worker_working_directory', + 'delegated_worker_result_capture', + 'delegated_worker_capacity_signal' + ) + $unproven = [System.Collections.Generic.List[string]]::new() + $conditional = [System.Collections.Generic.List[string]]::new() + $unsupported = [System.Collections.Generic.List[string]]::new() + foreach ($name in $required) { + $value = if ($null -ne $Capabilities -and $Capabilities.Contains($name)) { [string]$Capabilities[$name] } else { 'unavailable' } + if ($value -ne 'supported') { + $unproven.Add($name) + if ($value -eq 'conditional') { $conditional.Add($name) } else { $unsupported.Add($name) } + } + } + + $delegation = Get-JsonProperty -Object $Descriptor -Name 'delegation' -Default $null + $dispatchOwner = [string](Get-JsonProperty -Object $delegation -Name 'dispatch_owner' -Default 'unsupported') + if ($dispatchOwner -notin @('orchestrator', 'runner')) { + $unproven.Add('delegation.dispatch_owner') + $unsupported.Add('delegation.dispatch_owner') + } + $mode = [string](Get-JsonProperty -Object $delegation -Name 'mode' -Default 'unsupported') + $nestedModelExecution = [bool](Get-JsonProperty -Object $delegation -Name 'nested_model_execution' -Default $true) + $mechanism = [string](Get-JsonProperty -Object $delegation -Name 'mechanism' -Default '') + $workerRole = [string](Get-JsonProperty -Object $delegation -Name 'worker_role' -Default '') + $modeIsNative = $mode -eq 'native_worker' + if (-not $modeIsNative) { + $unproven.Add('delegation.mode') + if ($mode -eq 'conditional') { $conditional.Add('delegation.mode') } else { $unsupported.Add('delegation.mode') } + } + if ($nestedModelExecution) { + $unproven.Add('delegation.nested_model_execution') + $unsupported.Add('delegation.nested_model_execution') + } + $delegationFields = [ordered]@{ + full_capability = 'delegated_worker_full_capability' + model_lock = 'delegated_worker_model_lock' + working_directory = 'delegated_worker_working_directory' + result_capture = 'delegated_worker_result_capture' + capacity = 'delegated_worker_capacity_signal' + } + foreach ($field in $delegationFields.Keys) { + $value = [string](Get-JsonProperty -Object $delegation -Name $field -Default 'unsupported') + $valid = if ($field -eq 'capacity') { $value -in @('supported', 'harness_authoritative') } else { $value -eq 'supported' } + if (-not $valid) { + $unproven.Add("delegation.$field") + if ($value -eq 'conditional') { $conditional.Add("delegation.$field") } else { $unsupported.Add("delegation.$field") } + } + } + + $status = if ($unsupported.Count -gt 0) { + 'unsupported' + } elseif ($conditional.Count -gt 0) { + 'conditional' + } else { + 'supported' + } + + return [pscustomobject]@{ + MandatoryProven = $status -eq 'supported' + Status = $status + DispatchOwner = $dispatchOwner + Mode = $mode + Mechanism = $mechanism + WorkerRole = $workerRole + NestedModelExecution = $nestedModelExecution + Unproven = @($unproven) + Conditional = @($conditional) + Unsupported = @($unsupported) + Required = @($required) + } +} + +function Get-NativeWorkerTerminalEvidenceRequirements { + return @( + 'mechanism', + 'worker_session_id', + 'observed_model', + 'observed_working_directory', + 'observed_home', + 'fresh_worker', + 'home_config_isolated', + 'prompt_fidelity', + 'prompt_sha256', + 'terminal_result_capture', + 'paired_arm_visible', + 'grading_material_visible', + 'nested_model_execution', + 'model_execution_count' + ) +} + +function ConvertTo-ComparablePath { + param([Parameter(Mandatory = $true)][AllowEmptyString()][string]$Path) + + if ([string]::IsNullOrWhiteSpace($Path)) { return $null } + try { + $full = [System.IO.Path]::GetFullPath($Path) + $full = Expand-WindowsShortPath -Path $full + $root = [System.IO.Path]::GetPathRoot($full) + if (-not [string]::IsNullOrWhiteSpace($root) -and $full.Length -gt $root.Length) { + $full = $full.TrimEnd([char[]]@('\', '/')) + } + return $full + } catch { + return $null + } +} + +function Test-ExactObservedPath { + param( + [Parameter(Mandatory = $true)][string]$Expected, + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Observed + ) + + $expectedComparable = ConvertTo-ComparablePath -Path $Expected + $observedComparable = ConvertTo-ComparablePath -Path $Observed + if ($null -eq $expectedComparable -or $null -eq $observedComparable) { return $false } + $comparison = if ($IsWindows) { [System.StringComparison]::OrdinalIgnoreCase } else { [System.StringComparison]::Ordinal } + return [string]::Equals($expectedComparable, $observedComparable, $comparison) +} + +function Get-NativeWorkerReportedFailures { + param([Parameter(Mandatory = $true)][object]$ExecutionEvidence) + + $evidence = Get-JsonProperty -Object $ExecutionEvidence -Name 'evidence' -Default $null + $reported = @(Get-JsonProperty -Object $evidence -Name 'native_worker_evidence_failures' -Default @()) + if ($reported.Count -eq 0) { + $reported = @(Get-JsonProperty -Object $ExecutionEvidence -Name 'native_worker_evidence_failures' -Default @()) + } + return @($reported | ForEach-Object { [string]$_ } | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | Select-Object -Unique) +} + +function Set-NativeWorkerReportedFailures { + param( + [Parameter(Mandatory = $true)][object]$ExecutionEvidence, + [Parameter(Mandatory = $true)][string[]]$Failures + ) + + $uniqueFailures = @($Failures | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) } | Select-Object -Unique) + $evidence = Get-JsonProperty -Object $ExecutionEvidence -Name 'evidence' -Default $null + if ($null -ne $evidence) { + if ($evidence -is [System.Collections.IDictionary]) { + $evidence['native_worker_evidence_failures'] = $uniqueFailures + } elseif (Test-JsonProperty -Object $evidence -Name 'native_worker_evidence_failures') { + $evidence.native_worker_evidence_failures = $uniqueFailures + } else { + Add-Member -InputObject $evidence -MemberType NoteProperty -Name native_worker_evidence_failures -Value $uniqueFailures -Force + } + } + return $uniqueFailures +} + +function Test-NativeWorkerTerminalEvidence { + <# + Descriptor fields describe what a harness advertises. This validator is + deliberately separate: it accepts only observations from the actual + delegated worker for this exact arm. A direct compatibility-run result + without evidence.delegation is therefore never native-worker evidence. + #> + param( + [Parameter(Mandatory = $true)][object]$ExecutionEvidence, + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][string]$RequestedModel, + [string]$ExpectedWorkerSessionId = '', + [string]$ExpectedRunner = '', + [string]$ExpectedMechanism = '' + ) + + $failures = [System.Collections.Generic.List[string]]::new() + $status = [string](Get-JsonProperty -Object $ExecutionEvidence -Name 'status' -Default '') + if ($status -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { + $failures.Add('terminal_result_capture') + } + $runEvidence = Get-JsonProperty -Object $ExecutionEvidence -Name 'run' -Default $null + if ($null -eq $runEvidence) { + $failures.Add('arm_identity') + } else { + if ([int](Get-JsonProperty -Object $runEvidence -Name 'eval_id' -Default 0) -ne [int]$Run.EvalId -or + [string](Get-JsonProperty -Object $runEvidence -Name 'eval_name' -Default '') -ne [string]$Run.EvalName -or + [string](Get-JsonProperty -Object $runEvidence -Name 'configuration' -Default '') -ne [string]$Run.Mode) { + $failures.Add('arm_identity') + } + } + + $requestedEvidence = Get-JsonProperty -Object $ExecutionEvidence -Name 'requested' -Default $null + if ($null -eq $requestedEvidence -or [string](Get-JsonProperty -Object $requestedEvidence -Name 'model' -Default '') -ne $RequestedModel) { + $failures.Add('requested_model') + } + + if (-not [string]::IsNullOrWhiteSpace($ExpectedRunner)) { + $runnerEvidence = Get-JsonProperty -Object $ExecutionEvidence -Name 'runner' -Default $null + if ($null -eq $runnerEvidence -or [string](Get-JsonProperty -Object $runnerEvidence -Name 'name' -Default '') -ne $ExpectedRunner) { + $failures.Add('runner_identity') + } + } + + # Preserve runner-specific terminal codes even when the result has no + # portable delegation object. The common validator must not erase the + # transport's exact failure reason while reporting the missing common proof. + $reportedFailures = @(Get-NativeWorkerReportedFailures -ExecutionEvidence $ExecutionEvidence) + if ($status -eq 'incompatible') { + if ($reportedFailures.Count -eq 0) { + $failures.Add('runner_reported_incompatible') + } else { + foreach ($reportedFailure in $reportedFailures) { $failures.Add($reportedFailure) } + } + } elseif ($reportedFailures.Count -gt 0) { + $failures.Add('runner_evidence_status_mismatch') + foreach ($reportedFailure in $reportedFailures) { $failures.Add($reportedFailure) } + } + + $delegation = Get-JsonProperty -Object (Get-JsonProperty -Object $ExecutionEvidence -Name 'evidence' -Default $null) -Name 'delegation' -Default $null + if ($null -eq $delegation) { + $failures.Add('delegation_terminal_evidence') + return [pscustomobject]@{ Valid = $false; Failures = @($failures); Delegation = $null } + } + + # Different evidence requirements depending on terminal status. A + # completed scripted run must provide full same-session interaction proof. + # Non-success terminal statuses (timed_out/failed/cancelled) have weaker but + # truthful evidence expectations and must not be forced to fabricate a final + # assistant response. + $isSuccessTerminal = ($status -eq 'completed') + + if ($isSuccessTerminal) { + foreach ($name in @(Get-NativeWorkerTerminalEvidenceRequirements)) { + if (-not (Test-JsonProperty -Object $delegation -Name $name)) { + $failures.Add($name) + } + } + } else { + # Minimal truthful delegation fields for non-success terminals. + foreach ($name in @('mechanism','worker_session_id','observed_model','observed_working_directory','observed_home')) { + if (-not (Test-JsonProperty -Object $delegation -Name $name)) { + $failures.Add($name) + } + } + } + + if ([string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $delegation -Name 'mechanism' -Default ''))) { + $failures.Add('mechanism') + } + if (-not [string]::IsNullOrWhiteSpace($ExpectedMechanism) -and + [string](Get-JsonProperty -Object $delegation -Name 'mechanism' -Default '') -ne $ExpectedMechanism) { + $failures.Add('native_mechanism') + } + + $workerSessionId = [string](Get-JsonProperty -Object $delegation -Name 'worker_session_id' -Default '') + if ([string]::IsNullOrWhiteSpace($workerSessionId)) { + if ($failures -notcontains 'worker_session_id') { $failures.Add('worker_session_id') } + } elseif (-not [string]::IsNullOrWhiteSpace($ExpectedWorkerSessionId) -and $workerSessionId -ne $ExpectedWorkerSessionId) { + $failures.Add('worker_session_id') + } + + if ([string](Get-JsonProperty -Object $delegation -Name 'observed_model' -Default '') -ne $RequestedModel) { + $failures.Add('requested_model') + } + + if ($isSuccessTerminal) { + if (-not [bool](Get-JsonProperty -Object $delegation -Name 'fresh_worker' -Default $false)) { + $failures.Add('fresh_worker') + } + if (-not [bool](Get-JsonProperty -Object $delegation -Name 'home_config_isolated' -Default $false)) { + $failures.Add('isolated_home_config') + } + if (-not [bool](Get-JsonProperty -Object $delegation -Name 'prompt_fidelity' -Default $false) -or + [string](Get-JsonProperty -Object $delegation -Name 'prompt_sha256' -Default '') -ne [string]$Run.PromptHash) { + $failures.Add('prompt_fidelity') + } + if (-not [bool](Get-JsonProperty -Object $delegation -Name 'terminal_result_capture' -Default $false)) { + $failures.Add('terminal_result_capture') + } + if ([bool](Get-JsonProperty -Object $delegation -Name 'paired_arm_visible' -Default $true) -or + [bool](Get-JsonProperty -Object $delegation -Name 'grading_material_visible' -Default $true)) { + $failures.Add('paired_arm_and_grading_exclusion') + } + if ([bool](Get-JsonProperty -Object $delegation -Name 'nested_model_execution' -Default $true) -or + [int](Get-JsonProperty -Object $delegation -Name 'model_execution_count' -Default 0) -ne 1) { + $failures.Add('nested_model_execution') + } + } else { + # For non-success terminals, skip strict prompt/terminal-capture checks + # but still validate working/home path alignment when provided. + if ([bool](Get-JsonProperty -Object $delegation -Name 'paired_arm_visible' -Default $true) -or + [bool](Get-JsonProperty -Object $delegation -Name 'grading_material_visible' -Default $true)) { + $failures.Add('paired_arm_and_grading_exclusion') + } + } + + $executionPaths = Get-JsonProperty -Object (Get-JsonProperty -Object $ExecutionEvidence -Name 'evidence' -Default $null) -Name 'execution_paths' -Default $null + $expectedWorkingDirectory = [string]$Run.WorkingDirectoryPath + $expectedHomeDirectory = [string]$Run.HomeDirectoryPath + if ($null -ne $executionPaths) { + $logicalWorkingDirectory = [string](Get-JsonProperty -Object $executionPaths -Name 'logical_working_directory' -Default '') + $logicalHomeDirectory = [string](Get-JsonProperty -Object $executionPaths -Name 'logical_home_directory' -Default '') + if (-not (Test-ExactObservedPath -Expected ([string]$Run.WorkingDirectoryPath) -Observed $logicalWorkingDirectory)) { + $failures.Add('working_directory') + } + if (-not (Test-ExactObservedPath -Expected ([string]$Run.HomeDirectoryPath) -Observed $logicalHomeDirectory)) { + $failures.Add('isolated_home_config') + } + $physicalWorkingDirectory = [string](Get-JsonProperty -Object $executionPaths -Name 'physical_working_directory' -Default '') + $physicalHomeDirectory = [string](Get-JsonProperty -Object $executionPaths -Name 'physical_home_directory' -Default '') + if (-not [string]::IsNullOrWhiteSpace($physicalWorkingDirectory)) { $expectedWorkingDirectory = $physicalWorkingDirectory } + if (-not [string]::IsNullOrWhiteSpace($physicalHomeDirectory)) { $expectedHomeDirectory = $physicalHomeDirectory } + } + if (-not (Test-ExactObservedPath -Expected $expectedWorkingDirectory -Observed ([string](Get-JsonProperty -Object $delegation -Name 'observed_working_directory' -Default '')))) { + $failures.Add('working_directory') + } + if (-not (Test-ExactObservedPath -Expected $expectedHomeDirectory -Observed ([string](Get-JsonProperty -Object $delegation -Name 'observed_home' -Default '')))) { + $failures.Add('isolated_home_config') + } + + $session = Get-JsonProperty -Object $ExecutionEvidence -Name 'session' -Default $null + if ($isSuccessTerminal) { + if ($null -eq $session -or + -not [bool](Get-JsonProperty -Object $session -Name 'fresh' -Default $false) -or + [bool](Get-JsonProperty -Object $session -Name 'resumed' -Default $true) -or + [string](Get-JsonProperty -Object $session -Name 'id' -Default '') -ne $workerSessionId) { + $failures.Add('fresh_worker') + } + } else { + # If a session object exists for a non-success terminal, require the + # recorded session id to match the delegation worker_session_id when + # present; otherwise do not force fresh/resumed semantics. + if ($null -ne $session -and -not [string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $session -Name 'id' -Default '')) -and + -not [string]::IsNullOrWhiteSpace($workerSessionId) -and + [string](Get-JsonProperty -Object $session -Name 'id' -Default '') -ne $workerSessionId) { + $failures.Add('worker_session_id') + } + } + + return [pscustomobject]@{ + Valid = $failures.Count -eq 0 + Failures = @($failures | Select-Object -Unique) + Delegation = $delegation + } +} + +function Assert-NativeWorkerTerminalEvidence { + param( + [Parameter(Mandatory = $true)][object]$ExecutionEvidence, + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][string]$RequestedModel, + [string]$ExpectedWorkerSessionId = '', + [string]$ExpectedRunner = '', + [string]$ExpectedMechanism = '' + ) + + $validation = Test-NativeWorkerTerminalEvidence ` + -ExecutionEvidence $ExecutionEvidence ` + -Run $Run ` + -RequestedModel $RequestedModel ` + -ExpectedWorkerSessionId $ExpectedWorkerSessionId ` + -ExpectedRunner $ExpectedRunner ` + -ExpectedMechanism $ExpectedMechanism + if (-not $validation.Valid) { + throw "Native worker terminal evidence is incompatible: $([string]::Join(', ', @($validation.Failures)))." + } + return $true +} + +function Assert-NativeTerminalCaptureArtifact { + param([Parameter(Mandatory = $true)][object]$ExecutionResult) + + $capture = Get-JsonProperty -Object $ExecutionResult.evidence -Name 'capture' -Default $null + if ([string](Get-JsonProperty -Object $capture -Name 'source' -Default '') -ne 'harness_native_transport' -or + -not [bool](Get-JsonProperty -Object $capture -Name 'terminal' -Default $false) -or + [bool](Get-JsonProperty -Object $capture -Name 'worker_authored' -Default $true)) { + throw 'Native worker execution must preserve harness-native terminal capture provenance.' + } + + $transcriptMetric = Get-JsonProperty -Object $ExecutionResult.telemetry -Name 'transcript' -Default $null + $transcriptStatus = [string](Get-JsonProperty -Object $transcriptMetric -Name 'status' -Default '') + $transcriptArtifact = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $transcriptMetric -Name 'value' -Default $null) -Name 'artifact' -Default '') + if ($transcriptStatus -ne 'available' -or [string]::IsNullOrWhiteSpace($transcriptArtifact)) { + throw 'Native worker execution must provide an available terminal transcript artifact.' + } + $matchingArtifacts = @($ExecutionResult.artifacts | Where-Object { + [string](Get-JsonProperty -Object $_ -Name 'path' -Default '') -eq $transcriptArtifact + }) + if ($matchingArtifacts.Count -ne 1) { + throw "Native worker transcript artifact '$transcriptArtifact' is not recorded exactly once in execution-result.json." + } +} + +function Resolve-RunContract { + param([Parameter(Mandatory = $true)][string]$RunPath) + + $resolvedRunPath = (Resolve-Path -LiteralPath $RunPath -ErrorAction Stop).Path + $runRoot = Split-Path -Parent $resolvedRunPath + $run = Read-RunnerJson -Path $resolvedRunPath + $schemas = Get-RunnerSchemaNames + + if ([string]$run.schema -ne $schemas.Run) { + throw "run.json must declare '$($schemas.Run)'." + } + if (-not [bool]$run.freshContextRequired -or -not [bool]$run.filesystemIsolationRequired -or -not [bool]$run.isolatedHomeRequired) { + throw 'run.json must require a fresh context, a staged filesystem/workspace boundary, and an isolated home.' + } + + $mode = [string]$run.mode + if ($mode -notin @('with_skill', 'without_skill')) { + throw "run.json mode '$mode' is not with_skill or without_skill." + } + + $promptPath = Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$run.promptFile) -FieldName 'promptFile' -Kind File + $workingPath = Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$run.workingDirectory) -FieldName 'workingDirectory' -Kind Directory + $homePath = Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$run.homeDirectory) -FieldName 'homeDirectory' -Kind Directory + + $interactionPath = $null + $interaction = $null + $interactionHash = $null + $interactionFile = [string](Get-JsonProperty -Object $run -Name 'interactionFile' -Default '') + if (-not [string]::IsNullOrWhiteSpace($interactionFile)) { + $interactionPath = Resolve-ContainedPath -BasePath $runRoot -RelativePath $interactionFile -FieldName 'interactionFile' -Kind File + $interaction = Read-RunnerJson -Path $interactionPath + $interactionSchemas = Get-RunnerSchemaNames + if ([string]$interaction.schema -ne $interactionSchemas.Interaction -or [string]$interaction.mode -ne 'scripted') { + throw "interaction.json must declare '$($interactionSchemas.Interaction)' and mode 'scripted'." + } + $interactionTurns = @(Get-JsonProperty -Object $interaction -Name 'turns' -Default @()) + if ($interactionTurns.Count -lt 2) { throw 'Scripted interaction must contain at least two user turns.' } + $allowedInteractionFields = @('schema', 'mode', 'turns') + foreach ($name in @(Get-JsonPropertyNames -Object $interaction)) { + if ($allowedInteractionFields -notcontains $name) { throw "interaction.json contains unsupported field '$name'." } + } + for ($turnIndex = 0; $turnIndex -lt $interactionTurns.Count; $turnIndex++) { + $turn = $interactionTurns[$turnIndex] + foreach ($name in @(Get-JsonPropertyNames -Object $turn)) { + if (@('role', 'source', 'content') -notcontains $name) { throw "interaction turn $turnIndex contains unsupported field '$name'." } + } + if ([string](Get-JsonProperty -Object $turn -Name 'role' -Default '') -ne 'user') { throw "interaction turn $turnIndex must have role 'user'." } + $hasSource = (Test-JsonProperty -Object $turn -Name 'source') -and -not [string]::IsNullOrWhiteSpace([string]$turn.source) + $hasContent = (Test-JsonProperty -Object $turn -Name 'content') -and -not [string]::IsNullOrWhiteSpace([string]$turn.content) + if ($hasSource -eq $hasContent) { throw "interaction turn $turnIndex must declare exactly one non-empty source or content." } + if ($hasSource) { + [void](Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$turn.source) -FieldName "interaction.turns[$turnIndex].source" -Kind File) + } + } + if (-not (Test-JsonProperty -Object $run -Name 'interactionHash') -or -not (Test-Sha256 -Value ([string]$run.interactionHash))) { + throw 'run.json with interactionFile must declare a valid interactionHash.' + } + $interactionHash = [string]$run.interactionHash + if ($interactionHash -ne (Get-Sha256HexFromFile -Path $interactionPath)) { throw 'run.json interactionHash does not match interaction.json.' } + } + + $skillPath = $null + if ($mode -eq 'with_skill') { + if ([string]::IsNullOrWhiteSpace([string]$run.skillDirectory)) { + throw 'with_skill run.json must declare skillDirectory.' + } + $skillPath = Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$run.skillDirectory) -FieldName 'skillDirectory' -Kind Directory + if (-not (Test-Path -LiteralPath (Join-Path $skillPath 'SKILL.md') -PathType Leaf)) { + throw 'with_skill skillDirectory must contain SKILL.md.' + } + } else { + if ($null -ne $run.skillDirectory -and -not [string]::IsNullOrWhiteSpace([string]$run.skillDirectory)) { + throw 'without_skill run.json must not declare skillDirectory.' + } + $skillRoot = Join-Path $runRoot 'skill' + if (Test-Path -LiteralPath $skillRoot) { + throw 'without_skill run must not contain a skill directory.' + } + } + + $promptBytes = [System.IO.File]::ReadAllBytes($promptPath) + $fixtureHash = [string](Get-JsonProperty -Object $run -Name 'fixtureHash' -Default '') + if (-not (Test-Sha256 -Value $fixtureHash)) { + throw 'run.json fixtureHash must be a SHA-256 value.' + } + if ($mode -eq 'with_skill' -and -not (Test-Sha256 -Value ([string]$run.skillHash))) { + throw 'with_skill run.json skillHash must be a SHA-256 value.' + } + + return [pscustomobject]@{ + RunPath = $resolvedRunPath + RunRoot = $runRoot + Contract = $run + EvalId = [int]$run.evalId + EvalName = [string]$run.evalName + Mode = $mode + PromptPath = $promptPath + PromptBytes = $promptBytes + PromptHash = Get-Sha256HexFromBytes -Bytes $promptBytes + WorkingDirectoryPath = $workingPath + HomeDirectoryPath = $homePath + SkillDirectoryPath = $skillPath + CandidateSkillExposed = $mode -eq 'with_skill' + FixtureHash = $fixtureHash + SkillHash = if ($mode -eq 'with_skill') { [string]$run.skillHash } else { $null } + InteractionPath = $interactionPath + InteractionHash = $interactionHash + Interaction = $interaction + } +} + +function Get-InteractionTurnText { + param( + [Parameter(Mandatory = $true)][object]$Turn, + [Parameter(Mandatory = $true)][object]$RunData + ) + + $content = Get-JsonProperty -Object $Turn -Name 'content' -Default $null + if ($null -ne $content) { return [string]$content } + $sourcePath = Resolve-ContainedPath -BasePath $RunData.RunRoot -RelativePath ([string]$Turn.source) -FieldName 'interaction turn source' -Kind File + return [System.IO.File]::ReadAllText($sourcePath, [System.Text.UTF8Encoding]::new($false)) +} + +function Assert-InteractionResultEvidence { + param( + [Parameter(Mandatory = $true)][object]$ExecutionResult, + [Parameter(Mandatory = $true)][object]$RunData + ) + + if ($null -eq $RunData.Interaction) { return $true } + if ([string]$ExecutionResult.status -ne 'completed') { return $true } + $interaction = Get-JsonProperty -Object $ExecutionResult.evidence -Name 'interaction' -Default $null + if ($null -eq $interaction) { throw 'Scripted interaction execution must provide evidence.interaction.' } + $schemas = Get-RunnerSchemaNames + if ([string]$interaction.schema -ne $schemas.Interaction -or [string]$interaction.mode -ne 'scripted' -or -not [bool]$interaction.same_session) { + throw 'Scripted interaction evidence must prove the eval stayed in one same-session scripted interaction.' + } + $sessionId = [string]$ExecutionResult.session.id + if ([string]$interaction.session_id -ne $sessionId) { throw 'Scripted interaction evidence session_id does not match execution-result session.id.' } + $requestedTurns = @(Get-JsonProperty -Object $RunData.Interaction -Name 'turns' -Default @()) + $observedTurns = @(Get-JsonProperty -Object $interaction -Name 'turns' -Default @()) + if ($observedTurns.Count -ne ($requestedTurns.Count * 2)) { + throw "Scripted interaction evidence has $($observedTurns.Count) turns; expected $($requestedTurns.Count * 2) ordered user/assistant turns." + } + for ($index = 0; $index -lt $observedTurns.Count; $index++) { + $observed = $observedTurns[$index] + $expectedRole = if (($index % 2) -eq 0) { 'user' } else { 'assistant' } + if ([int](Get-JsonProperty -Object $observed -Name 'sequence' -Default -1) -ne ($index + 1) -or [string]$observed.role -ne $expectedRole) { + throw 'Scripted interaction evidence does not preserve ordered user/assistant turns.' + } + $observedSession = [string](Get-JsonProperty -Object $observed -Name 'session_id' -Default $sessionId) + if ($observedSession -ne $sessionId) { throw 'Scripted interaction evidence changed session identity between turns.' } + if ($expectedRole -eq 'user') { + $requestedTurn = $requestedTurns[[int]($index / 2)] + $expectedText = Get-InteractionTurnText -Turn $requestedTurn -RunData $RunData + $expectedHash = Get-Sha256HexFromBytes -Bytes ([System.Text.UTF8Encoding]::new($false).GetBytes($expectedText)) + if ([string]$observed.content_sha256 -ne $expectedHash) { throw "Scripted interaction user turn $($index / 2) does not match its deterministic package input." } + } elseif (-not (Test-JsonProperty -Object $observed -Name 'text')) { + throw "Scripted interaction assistant turn $($index / 2) is missing its terminal response text." + } + } + $finalSequence = [int](Get-JsonProperty -Object $interaction -Name 'final_response_sequence' -Default 0) + if ($finalSequence -ne $observedTurns.Count) { throw 'Scripted interaction evidence final response does not identify the last assistant turn.' } + if ([string]$ExecutionResult.final_response.status -eq 'available' -and + [string]$ExecutionResult.final_response.text -ne [string]$observedTurns[$observedTurns.Count - 1].text) { + throw 'Scripted interaction final_response does not match the last assistant turn.' + } + return $true +} + +function Assert-ProfileHasNoSecrets { + param([Parameter(Mandatory = $true)][object]$Profile) + + foreach ($property in @($Profile.PSObject.Properties)) { + if ([string]$property.Name -match '(?i)(secret|token|password|credential|api[_-]?key|private[_-]?key)') { + throw "execution-profile.json must not contain secret-bearing field '$($property.Name)'." + } + } +} + +function Resolve-ExecutionProfile { + param([Parameter(Mandatory = $true)][string]$ProfilePath) + + $resolvedProfilePath = (Resolve-Path -LiteralPath $ProfilePath -ErrorAction Stop).Path + $profile = Read-RunnerJson -Path $resolvedProfilePath + $schemas = Get-RunnerSchemaNames + if ([string]$profile.schema -ne $schemas.Profile) { + throw "execution-profile.json must declare '$($schemas.Profile)'." + } + Assert-ProfileHasNoSecrets -Profile $profile + + $allowedProperties = @('schema', 'runner', 'model', 'reasoning_effort', 'configuration_profile', 'tool_profile', 'timeout_seconds', 'concurrency') + foreach ($propertyName in @(Get-JsonPropertyNames -Object $profile)) { + if ($allowedProperties -notcontains $propertyName) { + throw "execution-profile.json contains unsupported field '$propertyName'." + } + } + + $timeout = [int](Get-JsonProperty -Object $profile -Name 'timeout_seconds' -Default 0) + $concurrency = [int](Get-JsonProperty -Object $profile -Name 'concurrency' -Default 0) + if ($timeout -lt 1 -or $timeout -gt 86400) { + throw 'execution-profile.json timeout_seconds must be between 1 and 86400.' + } + if ($concurrency -lt 1) { + throw 'execution-profile.json concurrency must be at least 1.' + } + if ([string]::IsNullOrWhiteSpace([string]$profile.configuration_profile) -or [string]::IsNullOrWhiteSpace([string]$profile.tool_profile)) { + throw 'execution-profile.json must declare configuration_profile and tool_profile.' + } + + $runnerValue = [string](Get-JsonProperty -Object $profile -Name 'runner' -Default '') + $modelValue = [string](Get-JsonProperty -Object $profile -Name 'model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($runnerValue) -and $runnerValue -notmatch '^[a-z0-9][a-z0-9-]*$') { + throw 'execution-profile.json runner must be a safe lowercase runner name.' + } + if ([string]::IsNullOrWhiteSpace($runnerValue) -or [string]::IsNullOrWhiteSpace($modelValue)) { + throw 'execution-profile.json must declare non-empty runner and model before a runner can execute.' + } + return [pscustomobject]@{ + Path = $resolvedProfilePath + Profile = $profile + Hash = Get-Sha256HexFromFile -Path $resolvedProfilePath + Runner = if ([string]::IsNullOrWhiteSpace($runnerValue)) { $null } else { $runnerValue } + Model = if ([string]::IsNullOrWhiteSpace($modelValue)) { $null } else { $modelValue } + ReasoningEffort = if ([string]::IsNullOrWhiteSpace([string]$profile.reasoning_effort)) { $null } else { [string]$profile.reasoning_effort } + ConfigurationProfile = [string]$profile.configuration_profile + ToolProfile = [string]$profile.tool_profile + TimeoutSeconds = $timeout + Concurrency = $concurrency + } +} + +function Assert-RunnerDescriptor { + param([Parameter(Mandatory = $true)][object]$Descriptor) + + $schemas = Get-RunnerSchemaNames + if ([string]$Descriptor.schema -ne $schemas.Descriptor) { + throw "Runner descriptor must declare '$($schemas.Descriptor)'." + } + if ([string]$Descriptor.protocol_version -ne $schemas.Protocol) { + throw "Runner descriptor protocol_version must be '$($schemas.Protocol)'." + } + foreach ($field in @('name', 'version', 'platforms', 'harness', 'capabilities', 'delegation', 'configuration_profiles', 'tool_profiles')) { + if (-not (Test-JsonProperty -Object $Descriptor -Name $field)) { + throw "Runner descriptor is missing '$field'." + } + } + if ([string]::IsNullOrWhiteSpace([string]$Descriptor.name) -or [string]::IsNullOrWhiteSpace([string]$Descriptor.version)) { + throw 'Runner descriptor name and version must be non-empty.' + } + if (-not (Test-JsonProperty -Object $Descriptor.harness -Name 'name') -or -not (Test-JsonProperty -Object $Descriptor.harness -Name 'version')) { + throw 'Runner descriptor harness must declare name and version.' + } + + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $Descriptor.capabilities)) { + $capabilityValue = Get-JsonProperty -Object $Descriptor.capabilities -Name $capabilityName + if ([string]$capabilityValue -notin @('supported', 'conditional', 'unsupported')) { + throw "Runner capability '$capabilityName' must be supported, conditional, or unsupported." + } + } + + $required = @( + 'single_turn', + 'scripted_multi_turn_same_session', + 'fresh_context', + 'isolated_home_config', + 'isolated_working_directory', + 'filesystem_confinement', + 'ambient_candidate_skill_exclusion', + 'candidate_skill_exposure', + 'prompt_fidelity', + 'model_configuration_lock', + 'response_capture' + ) + foreach ($name in $required) { + if (-not (Test-JsonProperty -Object $Descriptor.capabilities -Name $name)) { + throw "Runner descriptor is missing required capability '$name'." + } + } + + $delegationRequired = @( + 'native_worker_delegation', + 'delegated_worker_full_capability', + 'delegated_worker_model_lock', + 'delegated_worker_working_directory', + 'delegated_worker_result_capture', + 'delegated_worker_capacity_signal' + ) + foreach ($name in $delegationRequired) { + if (-not (Test-JsonProperty -Object $Descriptor.capabilities -Name $name)) { + throw "Runner descriptor is missing required delegation capability '$name'." + } + } + $delegation = $Descriptor.delegation + foreach ($field in @('dispatch_owner', 'mode', 'mechanism', 'worker_role', 'full_capability', 'model_lock', 'working_directory', 'result_capture', 'capacity', 'nested_model_execution')) { + if (-not (Test-JsonProperty -Object $delegation -Name $field)) { + throw "Runner descriptor delegation is missing '$field'." + } + } + if ([string]$delegation.dispatch_owner -notin @('orchestrator', 'runner')) { + throw "Runner delegation dispatch_owner '$($delegation.dispatch_owner)' must be orchestrator or runner." + } + if ([string]$delegation.mode -notin @('native_worker', 'conditional', 'unsupported')) { + throw "Runner delegation mode '$($delegation.mode)' is unsupported." + } + foreach ($field in @('full_capability', 'model_lock', 'working_directory', 'result_capture', 'capacity')) { + if ([string]$delegation.$field -notin @('supported', 'conditional', 'unsupported', 'harness_authoritative')) { + throw "Runner delegation '$field' must be supported, conditional, unsupported, or harness_authoritative." + } + } + if ([bool]$delegation.nested_model_execution) { + throw 'Runner delegation must not describe nested model execution.' + } + + return $true +} + +function New-PreflightCheck { + param( + [Parameter(Mandatory = $true)][string]$Name, + [Parameter(Mandatory = $true)][ValidateSet('passed', 'failed', 'unavailable', 'not_applicable')][string]$Status, + [Parameter(Mandatory = $true)][string]$Detail + ) + + return [ordered]@{ name = $Name; status = $Status; detail = $Detail } +} + +function New-PreflightDocument { + param( + [Parameter(Mandatory = $true)][object]$Descriptor, + [Parameter(Mandatory = $true)][object]$Profile, + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][bool]$Compatible, + [object[]]$Checks = @(), + [string[]]$Mechanisms = @(), + [object]$ResolvedCapabilities = $null, + [string[]]$Warnings = @(), + [string[]]$Reasons = @() + ) + + $schemas = Get-RunnerSchemaNames + $capabilitiesForAssessment = if ($null -eq $ResolvedCapabilities) { [ordered]@{} } else { $ResolvedCapabilities } + $assessment = Get-IsolationCapabilityAssessment -Capabilities $capabilitiesForAssessment + $delegationAssessment = Get-DelegationCapabilityAssessment -Descriptor $Descriptor -Capabilities $capabilitiesForAssessment + # Isolation is the compatibility transport's local readiness gate. Native + # delegation has a separate gate: conditional controls may proceed to a + # delegated worker, while an unavailable/unsupported mechanism cannot. + $effectiveCompatible = $Compatible -and $assessment.MandatoryProven + $unprovenControls = [string[]]$assessment.Unproven + if (-not $effectiveCompatible) { $unprovenControls = [string[]](@($assessment.Unproven) + @('preflight')) } + return [ordered]@{ + schema = $schemas.Preflight + protocol_version = $schemas.Protocol + status = if ($effectiveCompatible) { 'compatible' } else { 'incompatible' } + runner = [ordered]@{ name = [string]$Descriptor.name; version = [string]$Descriptor.version } + harness = $Descriptor.harness + run = [ordered]@{ eval_id = $Run.EvalId; eval_name = $Run.EvalName; configuration = $Run.Mode } + requested = [ordered]@{ + model = $Profile.Model + reasoning_effort = $Profile.ReasoningEffort + configuration_profile = $Profile.ConfigurationProfile + tool_profile = $Profile.ToolProfile + timeout_seconds = $Profile.TimeoutSeconds + } + checks = @($Checks) + resolved_capabilities = if ($null -eq $ResolvedCapabilities) { [ordered]@{} } else { $ResolvedCapabilities } + delegation = [ordered]@{ + dispatch_owner = $delegationAssessment.DispatchOwner + status = $delegationAssessment.Status + mode = $delegationAssessment.Mode + mechanism = $delegationAssessment.Mechanism + worker_role = $delegationAssessment.WorkerRole + nested_model_execution = $delegationAssessment.NestedModelExecution + required_controls = @($delegationAssessment.Required) + unproven_controls = [string[]]$delegationAssessment.Unproven + terminal_evidence_required = $delegationAssessment.Status -eq 'conditional' + } + isolation = [ordered]@{ + level = if ($effectiveCompatible) { $assessment.Level } else { 'unsupported' } + status = if ($effectiveCompatible) { 'verified' } else { 'unverified' } + hard_filesystem_confinement = if ($effectiveCompatible) { $assessment.HardFilesystemConfinement } else { $false } + unproven_controls = $unprovenControls + } + mechanisms = @($Mechanisms) + warnings = @($Warnings) + reasons = @($Reasons) + } +} + +function Assert-NativeWorkerDelegation { + param( + [Parameter(Mandatory = $true)][object]$Descriptor, + [Parameter(Mandatory = $true)][object]$Preflight + ) + + $delegation = Get-JsonProperty -Object $Descriptor -Name 'delegation' -Default $null + $dispatchOwner = [string](Get-JsonProperty -Object $delegation -Name 'dispatch_owner' -Default '') + if ($dispatchOwner -notin @('orchestrator', 'runner')) { + throw "Runner '$($Descriptor.name)' has no valid native dispatch owner." + } + $mode = [string](Get-JsonProperty -Object $delegation -Name 'mode' -Default 'unsupported') + if ($mode -notin @('native_worker', 'conditional')) { + throw "Runner '$($Descriptor.name)' cannot satisfy the mandatory native Eval Worker contract: delegation mode is '$mode'." + } + if ([bool](Get-JsonProperty -Object $delegation -Name 'nested_model_execution' -Default $true)) { + throw "Runner '$($Descriptor.name)' describes nested model execution; one eval arm must have exactly one model-backed worker." + } + if ([string](Get-JsonProperty -Object $Preflight -Name 'status' -Default 'incompatible') -ne 'compatible') { + throw "Runner '$($Descriptor.name)' preflight is incompatible; the orchestrator must not execute an arm in the parent context." + } + $delegationPreflight = Get-JsonProperty -Object $Preflight -Name 'delegation' -Default $null + $delegationStatus = [string](Get-JsonProperty -Object $delegationPreflight -Name 'status' -Default 'unsupported') + if ($delegationStatus -notin @('supported', 'conditional')) { + $unproven = @((Get-JsonProperty -Object $delegationPreflight -Name 'unproven_controls' -Default @())) + throw "Runner '$($Descriptor.name)' native worker delegation is unavailable during preflight: $([string]::Join(', ', $unproven)). No parent or compatibility-execute fallback is permitted." + } + if ($delegationStatus -eq 'conditional' -and -not [bool](Get-JsonProperty -Object $delegationPreflight -Name 'terminal_evidence_required' -Default $false)) { + throw "Runner '$($Descriptor.name)' reports conditional native worker controls without requiring terminal evidence." + } + $capabilities = Get-JsonProperty -Object $Preflight -Name 'resolved_capabilities' -Default $null + $required = @( + 'native_worker_delegation', + 'delegated_worker_full_capability', + 'delegated_worker_model_lock', + 'delegated_worker_working_directory', + 'delegated_worker_result_capture', + 'delegated_worker_capacity_signal' + ) + foreach ($name in $required) { + $value = [string](Get-JsonProperty -Object $capabilities -Name $name -Default 'unsupported') + if ($value -notin @('supported', 'conditional')) { + throw "Runner '$($Descriptor.name)' native worker capability '$name' is unavailable; the orchestrator must not fall back to parent or compatibility execution." + } + } + return $true +} + +function New-UnavailableMetric { + param([Parameter(Mandatory = $true)][string]$Reason) + + return [ordered]@{ status = 'unavailable'; reason = $Reason } +} + +function New-AvailableMetric { + param([Parameter(Mandatory = $true)][object]$Value) + + return [ordered]@{ status = 'available'; value = $Value } +} + +function New-ExecutionFailure { + param( + [Parameter(Mandatory = $true)][string]$Code, + [Parameter(Mandatory = $true)][string]$Message + ) + + return [ordered]@{ code = $Code; message = $Message } +} + +function Assert-PhaseOneEvidenceWritable { + param([Parameter(Mandatory = $true)][object]$Run) + + # Every runner-owned transport reaches this shared result builder. Once the + # package-level freeze exists, refusing to build another result prevents a + # direct runner invocation (or the orchestrator-owned recorder) from + # truncating or replacing frozen raw evidence. + $runRoot = [System.IO.Path]::GetFullPath([string]$Run.RunRoot) + $iterationDirectory = Split-Path -Parent (Split-Path -Parent $runRoot) + $freezeRelativePath = 'execution-freeze.json' + $manifestPath = Join-Path $iterationDirectory 'manifest.json' + if (Test-Path -LiteralPath $manifestPath -PathType Leaf) { + $manifest = Read-RunnerJson -Path $manifestPath + $declaredFreezePath = [string](Get-JsonProperty -Object $manifest -Name 'execution_freeze' -Default '') + if (-not [string]::IsNullOrWhiteSpace($declaredFreezePath)) { $freezeRelativePath = $declaredFreezePath } + } + Assert-SafeRelativePath -RelativePath $freezeRelativePath -FieldName 'execution freeze path' + $freezePath = Join-Path $iterationDirectory ($freezeRelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if (Test-Path -LiteralPath $freezePath) { + throw "Execution integrity failure: Phase 1 raw evidence is already frozen at '$freezePath'; requires fresh Phase 1 execution." + } + return $true +} + +function New-ExecutionResult { + param( + [Parameter(Mandatory = $true)][object]$Descriptor, + [Parameter(Mandatory = $true)][object]$Profile, + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][ValidateSet('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')][string]$Status, + [string]$FinalResponse, + [string]$FinalResponseReason, + [string]$StartedUtc, + [string]$FinishedUtc, + [double]$DurationSeconds = 0, + [Nullable[int]]$ExitStatus, + [object]$Failure, + [string]$SessionId, + [System.Collections.IDictionary]$IsolationCapabilities, + [string[]]$IsolationMechanisms = @(), + [object]$ResolvedConfiguration = $null, + [object]$Telemetry = $null, + [object[]]$Artifacts = @(), + [string[]]$Warnings = @(), + [string[]]$CompatibilityDeviations = @(), + [object]$Evidence = $null, + [int]$AttemptCount = 1 + ) + + [void](Assert-PhaseOneEvidenceWritable -Run $Run) + $schemas = Get-RunnerSchemaNames + $assessment = Get-IsolationCapabilityAssessment -Capabilities $IsolationCapabilities + $effectiveStatus = $Status + $effectiveFinalResponse = $FinalResponse + $effectiveFinalResponseReason = $FinalResponseReason + $effectiveExitStatus = $ExitStatus + $effectiveFailure = $Failure + $effectiveDeviations = [System.Collections.Generic.List[string]]::new() + foreach ($deviation in @($CompatibilityDeviations)) { $effectiveDeviations.Add([string]$deviation) } + if ($Status -ne 'incompatible' -and -not $assessment.MandatoryProven) { + $effectiveStatus = 'incompatible' + $effectiveFinalResponse = $null + $effectiveFinalResponseReason = 'isolation_controls_unproven' + $effectiveExitStatus = $null + $effectiveFailure = New-ExecutionFailure -Code 'isolation_unproven' -Message ("Mandatory isolation controls were not proven: {0}." -f ([string]::Join(', ', @($assessment.Unproven)))) + $effectiveDeviations.Add('execution_rejected_because_mandatory_isolation_controls_were_unproven') + } + $hasResponse = -not [string]::IsNullOrWhiteSpace($effectiveFinalResponse) + $started = if ([string]::IsNullOrWhiteSpace($StartedUtc)) { [DateTime]::UtcNow } else { [DateTime]::Parse($StartedUtc).ToUniversalTime() } + $finished = if ([string]::IsNullOrWhiteSpace($FinishedUtc)) { [DateTime]::UtcNow } else { [DateTime]::Parse($FinishedUtc).ToUniversalTime() } + $isolation = [ordered]@{ + status = if ($effectiveStatus -ne 'incompatible' -and $assessment.MandatoryProven) { 'verified' } else { 'unverified' } + level = if ($effectiveStatus -eq 'incompatible') { 'unsupported' } else { $assessment.Level } + hard_filesystem_confinement = if ($effectiveStatus -eq 'incompatible') { $false } else { $assessment.HardFilesystemConfinement } + capabilities = if ($null -eq $IsolationCapabilities) { [ordered]@{} } else { $IsolationCapabilities } + mechanisms = @($IsolationMechanisms) + required_controls = @($assessment.Required) + unproven_controls = [string[]]$assessment.Unproven + } + + $resolved = [ordered]@{ + model = $null + reasoning_effort = $null + configuration_profile = $null + tool_profile = $null + status = 'unavailable' + reason = 'harness_only_confirmed_the_requested_configuration' + accepted = [ordered]@{ + model = $Profile.Model + reasoning_effort = $Profile.ReasoningEffort + configuration_profile = $Profile.ConfigurationProfile + tool_profile = $Profile.ToolProfile + } + } + if ($null -ne $ResolvedConfiguration) { + $resolved.status = [string](Get-JsonProperty -Object $ResolvedConfiguration -Name 'status' -Default 'resolved') + $resolved.reason = Get-JsonProperty -Object $ResolvedConfiguration -Name 'reason' -Default $null + foreach ($name in @('model', 'reasoning_effort', 'configuration_profile', 'tool_profile')) { + $value = Get-JsonProperty -Object $ResolvedConfiguration -Name $name -Default $null + if ($null -ne $value) { $resolved[$name] = $value } + } + $observations = Get-JsonProperty -Object $ResolvedConfiguration -Name 'observations' -Default $null + if ($null -ne $observations) { $resolved.observations = $observations } + } + + return [ordered]@{ + schema = $schemas.Result + protocol_version = $schemas.Protocol + run_id = [Guid]::NewGuid().ToString('D') + session = [ordered]@{ + id = if ([string]::IsNullOrWhiteSpace($SessionId)) { [Guid]::NewGuid().ToString('D') } else { $SessionId } + fresh = $true + resumed = $false + } + status = $effectiveStatus + run = [ordered]@{ + eval_id = $Run.EvalId + eval_name = $Run.EvalName + configuration = $Run.Mode + } + final_response = if ($hasResponse) { + [ordered]@{ status = 'available'; text = $effectiveFinalResponse } + } else { + [ordered]@{ status = 'unavailable'; reason = if ([string]::IsNullOrWhiteSpace($effectiveFinalResponseReason)) { 'harness_did_not_return_a_final_response' } else { $effectiveFinalResponseReason } } + } + runner = [ordered]@{ name = [string]$Descriptor.name; version = [string]$Descriptor.version } + harness = $Descriptor.harness + requested = [ordered]@{ + model = $Profile.Model + reasoning_effort = $Profile.ReasoningEffort + configuration_profile = $Profile.ConfigurationProfile + tool_profile = $Profile.ToolProfile + timeout_seconds = $Profile.TimeoutSeconds + } + resolved = $resolved + started_utc = Format-UtcTimestamp -Value $started + finished_utc = Format-UtcTimestamp -Value $finished + duration_seconds = [Math]::Max([double]0, [Math]::Round([double]$DurationSeconds, 3)) + exit = [ordered]@{ status = $effectiveExitStatus; failure = $effectiveFailure } + input = [ordered]@{ + prompt_sha256 = $Run.PromptHash + run_json_sha256 = Get-Sha256HexFromFile -Path $Run.RunPath + profile_sha256 = $Profile.Hash + } + isolation = $isolation + telemetry = if ($null -eq $Telemetry) { + [ordered]@{ + transcript = New-UnavailableMetric -Reason 'harness_did_not_expose_transcript' + tokens = New-UnavailableMetric -Reason 'harness_did_not_expose_usage' + tool_calls = New-UnavailableMetric -Reason 'harness_did_not_expose_tool_calls' + cost = New-UnavailableMetric -Reason 'harness_did_not_expose_cost' + } + } else { $Telemetry } + evidence = if ($null -eq $Evidence) { [ordered]@{} } else { $Evidence } + artifacts = @($Artifacts) + warnings = @($Warnings) + compatibility_deviations = @($effectiveDeviations) + attempt_count = $AttemptCount + } +} + +function Assert-ExecutionResult { + param([Parameter(Mandatory = $true)][object]$Result) + + $schemas = Get-RunnerSchemaNames + if ([string]$Result.schema -ne $schemas.Result) { + throw "execution-result.json must declare '$($schemas.Result)'." + } + if ([string]$Result.protocol_version -ne $schemas.Protocol) { + throw "execution-result.json protocol_version must be '$($schemas.Protocol)'." + } + if ([string]$Result.status -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { + throw "execution-result.json status '$($Result.status)' is unsupported." + } + foreach ($field in @('run_id', 'runner', 'harness', 'requested', 'resolved', 'started_utc', 'finished_utc', 'duration_seconds', 'exit', 'final_response', 'input', 'isolation', 'telemetry', 'evidence', 'artifacts', 'warnings', 'compatibility_deviations')) { + if (-not (Test-JsonProperty -Object $Result -Name $field)) { + throw "execution-result.json is missing '$field'." + } + } + $runIdentity = Get-JsonProperty -Object $Result -Name 'run' -Default $null + foreach ($field in @('eval_id', 'eval_name', 'configuration')) { + if ($null -eq $runIdentity -or -not (Test-JsonProperty -Object $runIdentity -Name $field)) { + throw "execution-result.json run.$field must be present." + } + } + if (-not (Test-JsonProperty -Object $Result.requested -Name 'timeout_seconds')) { + throw 'execution-result.json requested.timeout_seconds must be present.' + } + foreach ($identityName in @('runner', 'harness')) { + $identity = Get-JsonProperty -Object $Result -Name $identityName -Default $null + foreach ($identityField in @('name', 'version')) { + if ($null -eq $identity -or -not (Test-JsonProperty -Object $identity -Name $identityField) -or + [string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $identity -Name $identityField -Default ''))) { + throw "execution-result.json $identityName.$identityField must be non-empty." + } + } + } + $exitObject = Get-JsonProperty -Object $Result -Name 'exit' -Default $null + if ($null -eq $exitObject -or -not (Test-JsonProperty -Object $exitObject -Name 'status')) { + throw 'execution-result.json exit.status must be present and numeric or null.' + } + if ([string]::IsNullOrWhiteSpace([string]$Result.run_id)) { + throw 'execution-result.json run_id must be non-empty.' + } + if (-not (Test-JsonProperty -Object $Result.session -Name 'id') -or + -not [bool](Get-JsonProperty -Object $Result.session -Name 'fresh' -Default $false) -or + [bool](Get-JsonProperty -Object $Result.session -Name 'resumed' -Default $true)) { + throw 'execution-result.json must identify a fresh, non-resumed session.' + } + if ([int](Get-JsonProperty -Object $Result -Name 'attempt_count' -Default 0) -ne 1) { + throw 'execution-result.json attempt_count must be exactly 1; quality retries are not allowed.' + } + $exitStatus = Get-JsonProperty -Object $exitObject -Name 'status' -Default $null + if ($null -ne $exitStatus) { + $numericExitStatus = $exitStatus -is [byte] -or + $exitStatus -is [sbyte] -or + $exitStatus -is [int16] -or + $exitStatus -is [uint16] -or + $exitStatus -is [int32] -or + $exitStatus -is [uint32] -or + $exitStatus -is [int64] -or + $exitStatus -is [uint64] + if (-not $numericExitStatus) { + throw 'execution-result.json exit.status must be a JSON number or null; textual lifecycle labels are not valid exit codes.' + } + } + $isolationStatus = [string](Get-JsonProperty -Object $Result.isolation -Name 'status' -Default '') + $isolationLevel = [string](Get-JsonProperty -Object $Result.isolation -Name 'level' -Default '') + $hardFilesystem = [bool](Get-JsonProperty -Object $Result.isolation -Name 'hard_filesystem_confinement' -Default $false) + if ($isolationStatus -notin @('verified', 'unverified')) { + throw "execution-result.json isolation.status '$isolationStatus' is unsupported." + } + if ($isolationLevel -notin @('strict', 'pragmatic', 'unsupported')) { + throw "execution-result.json isolation.level '$isolationLevel' is unsupported." + } + if ($Result.status -eq 'incompatible') { + if ($isolationStatus -ne 'unverified' -or $isolationLevel -ne 'unsupported') { + throw 'An incompatible execution must report unverified, unsupported isolation.' + } + } else { + if ($isolationStatus -ne 'verified' -or $isolationLevel -eq 'unsupported') { + throw 'A non-incompatible execution must prove the mandatory experimental controls.' + } + if ($isolationLevel -eq 'strict' -and -not $hardFilesystem) { + throw 'Strict isolation must report hard filesystem confinement.' + } + if ($isolationLevel -eq 'pragmatic' -and $hardFilesystem) { + throw 'Pragmatic isolation must not claim hard filesystem confinement.' + } + $requiredControls = @('fresh_context', 'isolated_home_config', 'isolated_working_directory', 'ambient_candidate_skill_exclusion', 'candidate_skill_exposure', 'prompt_fidelity', 'model_configuration_lock', 'response_capture') + foreach ($control in $requiredControls) { + $value = [string](Get-JsonProperty -Object $Result.isolation.capabilities -Name $control -Default 'unavailable') + if ($control -eq 'candidate_skill_exposure') { + if ($value -notin @('supported', 'excluded')) { throw "Mandatory isolation capability '$control' is not proven." } + } elseif ($value -ne 'supported') { + throw "Mandatory isolation capability '$control' is not proven." + } + } + } + $resolvedStatus = [string](Get-JsonProperty -Object $Result.resolved -Name 'status' -Default '') + if ($resolvedStatus -notin @('unavailable', 'accepted_request', 'resolved')) { + throw "execution-result.json resolved.status '$resolvedStatus' is unsupported." + } + if (-not (Test-JsonProperty -Object $Result.resolved -Name 'accepted')) { + throw 'execution-result.json resolved must preserve the requested configuration as accepted evidence.' + } + $hasPortableProvider = + (Test-JsonProperty -Object $Result.requested -Name 'provider') -or + (Test-JsonProperty -Object $Result.resolved -Name 'provider') -or + (Test-JsonProperty -Object $Result.resolved.accepted -Name 'provider') + if ($hasPortableProvider) { + throw 'execution-result.json must not expose provider in portable requested/resolved configuration fields.' + } + foreach ($hashField in @('prompt_sha256', 'run_json_sha256', 'profile_sha256')) { + if (-not (Test-Sha256 -Value ([string]$Result.input.$hashField))) { + throw "execution-result.json input.$hashField must be a SHA-256 value." + } + } + if ([double]$Result.duration_seconds -lt 0) { + throw 'execution-result.json duration_seconds must not be negative.' + } + $responseStatus = [string]$Result.final_response.status + if ($responseStatus -eq 'available') { + if (-not (Test-JsonProperty -Object $Result.final_response -Name 'text')) { + throw 'Available final_response must contain text.' + } + } elseif ($responseStatus -eq 'unavailable') { + if ([string]::IsNullOrWhiteSpace([string]$Result.final_response.reason)) { + throw 'Unavailable final_response must contain a reason.' + } + } else { + throw "final_response status '$responseStatus' is unsupported." + } + + foreach ($metricName in @(Get-JsonPropertyNames -Object $Result.telemetry)) { + $metric = Get-JsonProperty -Object $Result.telemetry -Name $metricName + $status = [string](Get-JsonProperty -Object $metric -Name 'status' -Default '') + if ($status -notin @('available', 'unavailable')) { + throw "Telemetry '$metricName' must declare available or unavailable status." + } + if ($status -eq 'unavailable' -and [string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $metric -Name 'reason' -Default ''))) { + throw "Unavailable telemetry '$metricName' must declare a reason." + } + } + + foreach ($artifact in @($Result.artifacts)) { + $path = [string](Get-JsonProperty -Object $artifact -Name 'path' -Default '') + $scope = [string](Get-JsonProperty -Object $artifact -Name 'scope' -Default '') + Assert-SafeRelativePath -RelativePath $path -FieldName 'artifact.path' + if ($scope -notin @('run', 'package')) { + throw "artifact.scope '$scope' must be run or package." + } + if (-not (Test-Sha256 -Value ([string]$artifact.sha256))) { + throw 'artifact.sha256 must be a SHA-256 value.' + } + if ([int64]$artifact.size -lt 0 -or [string]::IsNullOrWhiteSpace([string]$artifact.media_type)) { + throw 'artifact must declare non-negative size and media_type.' + } + } + + return $true +} + +function New-RunnerEnvironment { + <# + Builds the MODEL-BACKED eval isolation environment. This is deliberately + NOT the model-free probe environment (New-RunnerProbeEnvironment): here + HOME, USERPROFILE, XDG_*, and TEMP/TMP are all pinned INSIDE the per-run + isolated home so the harness cannot consult the real user profile, ambient + config, or ambient skill roots. A model-free --version/--help/describe probe + must never be routed through this environment, and this environment must + never be reduced to the probe environment: the probe carries OS scratch + TEMP/TMP but no isolated home, and the eval carries an isolated home but + never the host profile. + #> + param( + [Parameter(Mandatory = $true)][object]$Run, + [string[]]$AuthenticationVariables = @(), + [hashtable]$Additional = @{} + ) + + $environment = [ordered]@{} + foreach ($name in @('PATH', 'SystemRoot', 'WINDIR', 'ComSpec', 'PATHEXT', 'LANG', 'LC_ALL', 'TZ', 'SSL_CERT_FILE', 'NODE_PATH')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + + $tempPath = Join-Path $Run.HomeDirectoryPath 'tmp' + foreach ($directory in @($Run.HomeDirectoryPath, $tempPath, (Join-Path $Run.HomeDirectoryPath '.config'), (Join-Path $Run.HomeDirectoryPath '.local/share'), (Join-Path $Run.HomeDirectoryPath '.cache'))) { + New-Item -ItemType Directory -Path $directory -Force | Out-Null + } + + $environment['HOME'] = $Run.HomeDirectoryPath + $environment['USERPROFILE'] = $Run.HomeDirectoryPath + $environment['XDG_CONFIG_HOME'] = Join-Path $Run.HomeDirectoryPath '.config' + $environment['XDG_DATA_HOME'] = Join-Path $Run.HomeDirectoryPath '.local/share' + $environment['XDG_CACHE_HOME'] = Join-Path $Run.HomeDirectoryPath '.cache' + $environment['TEMP'] = $tempPath + $environment['TMP'] = $tempPath + $environment['CI'] = '1' + $environment['NO_COLOR'] = '1' + + foreach ($name in $AuthenticationVariables | Sort-Object -Unique) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + foreach ($key in $Additional.Keys) { + $environment[$key] = [string]$Additional[$key] + } + + return $environment +} + +function Get-LinuxEvalSandboxArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$InsideEnvironment, + [string[]]$ReadOnlyRoots = @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/lib64', '/libexec', '/etc', '/opt') + ) + + $arguments = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--die-with-parent', '--new-session', '--unshare-pid')) { $arguments.Add($argument) } + foreach ($path in $ReadOnlyRoots) { + if (Test-Path -LiteralPath $path -PathType Container) { + $arguments.Add('--ro-bind'); $arguments.Add($path); $arguments.Add($path) + } + } + $arguments.Add('--proc'); $arguments.Add('/proc') + $arguments.Add('--dev'); $arguments.Add('/dev') + $arguments.Add('--tmpfs'); $arguments.Add('/tmp') + $arguments.Add('--bind'); $arguments.Add($Inputs.Run.RunRoot); $arguments.Add('/run') + $commandSource = [string]$CommandInfo.Source + $commandDirectory = Split-Path -Parent $commandSource + if (-not ($commandSource.StartsWith('/usr/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/bin/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/opt/', [System.StringComparison]::Ordinal))) { + if (Test-Path -LiteralPath $commandDirectory -PathType Container) { + $arguments.Add('--ro-bind'); $arguments.Add($commandDirectory); $arguments.Add($commandDirectory) + } + } + $arguments.Add('--chdir'); $arguments.Add('/run/repo') + foreach ($key in @($InsideEnvironment.Keys)) { + $arguments.Add('--setenv'); $arguments.Add([string]$key); $arguments.Add([string]$InsideEnvironment[$key]) + } + $arguments.Add('--') + $arguments.Add($CommandInfo.FileName) + foreach ($prefix in @($CommandInfo.Prefix)) { $arguments.Add($prefix) } + return @($arguments) +} + +function New-MacosEvalSandboxProfile { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [string[]]$ReadOnlyRoots = @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/libexec', '/System', '/Library', '/opt', '/private/var/db') + ) + + $profilePath = Join-Path $Inputs.Run.HomeDirectoryPath 'eval-sandbox.sb' + $runRoot = $Inputs.Run.RunRoot.Replace('\', '/') + $commandDirectory = (Split-Path -Parent ([string]$CommandInfo.Source)).Replace('\', '/') + $lines = [System.Collections.Generic.List[string]]::new() + $lines.Add('(version 1)') + $lines.Add('(deny default)') + $lines.Add('(allow process*)') + $lines.Add('(allow network*)') + foreach ($root in @($ReadOnlyRoots + @($commandDirectory)) | Sort-Object -Unique) { + if (-not [string]::IsNullOrWhiteSpace($root) -and (Test-Path -LiteralPath $root -PathType Container)) { + $escapedRoot = $root.Replace('\', '/').Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRoot)) + } + } + $escapedRunRoot = $runRoot.Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add(('(allow file-write* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add('(allow file-read* (subpath "/dev"))') + $lines.Add('(allow file-write* (subpath "/dev/null"))') + [System.IO.File]::WriteAllText($profilePath, ([string]::Join("`n", $lines) + "`n"), [System.Text.UTF8Encoding]::new($false)) + return $profilePath +} + +function Resolve-ExternalCommand { + param([Parameter(Mandatory = $true)][string]$Name) + + $command = Get-Command $Name -ErrorAction SilentlyContinue + if ($null -eq $command) { + foreach ($candidateName in @("$Name.ps1", "$Name.cmd", "$Name.exe")) { + $command = Get-Command $candidateName -ErrorAction SilentlyContinue + if ($null -ne $command) { break } + } + } + if ($null -eq $command) { + return $null + } + + $source = [string]$command.Source + $extension = [System.IO.Path]::GetExtension($source).ToLowerInvariant() + if ($extension -eq '.ps1') { + $pwsh = Get-Command pwsh -ErrorAction SilentlyContinue + if ($null -eq $pwsh) { + return $null + } + return [pscustomobject]@{ FileName = [string]$pwsh.Source; Prefix = @('-NoProfile', '-File', $source); Source = $source } + } + + return [pscustomobject]@{ FileName = $source; Prefix = @(); Source = $source } +} + +function Wait-RunnerTaskBounded { + param( + [Parameter(Mandatory = $true)][System.Threading.Tasks.Task]$Task, + [Parameter(Mandatory = $true)][int]$TimeoutMilliseconds + ) + + if ($Task.IsCompleted) { return $true } + $boundedMilliseconds = [Math]::Max(1, [Math]::Min($TimeoutMilliseconds, 5000)) + try { + return [bool]$Task.Wait($boundedMilliseconds) + } catch { + return [bool]$Task.IsCompleted + } +} + +function Get-RunnerTaskResultIfCompleted { + param( + [System.Threading.Tasks.Task]$Task, + [string]$Default = '' + ) + + if ($null -eq $Task -or -not $Task.IsCompleted) { return $Default } + try { + return [string]$Task.GetAwaiter().GetResult() + } catch { + return $Default + } +} + +function Invoke-RunnerProcess { + param( + [Parameter(Mandatory = $true)][string]$FileName, + [string[]]$ArgumentList = @(), + [Parameter(Mandatory = $true)][string]$WorkingDirectory, + [System.Collections.IDictionary]$Environment = @{}, + [AllowEmptyCollection()][byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 900 + ) + + $start = [DateTime]::UtcNow + $startInfo = [System.Diagnostics.ProcessStartInfo]::new() + $startInfo.FileName = $FileName + $startInfo.WorkingDirectory = $WorkingDirectory + $startInfo.UseShellExecute = $false + $startInfo.CreateNoWindow = $true + $startInfo.RedirectStandardInput = $true + $startInfo.RedirectStandardOutput = $true + $startInfo.RedirectStandardError = $true + foreach ($argument in @($ArgumentList)) { + [void]$startInfo.ArgumentList.Add([string]$argument) + } + $startInfo.Environment.Clear() + foreach ($key in $Environment.Keys) { + $startInfo.Environment[$key] = [string]$Environment[$key] + } + + $process = [System.Diagnostics.Process]::new() + $process.StartInfo = $startInfo + $processStarted = $false + $stdoutTask = $null + $stderrTask = $null + $inputTask = $null + $timedOut = $false + $killAttempted = $false + $terminationObserved = $false + $stdoutDrainCompleted = $false + $stderrDrainCompleted = $false + $timeoutValue = [Math]::Max(1, $TimeoutSeconds) + $deadline = $start.AddSeconds($timeoutValue) + try { + if (-not $process.Start()) { + throw "Could not start '$FileName'." + } + $processStarted = $true + + # Start both readers before writing stdin. A child that emits enough + # output while reading its prompt must not deadlock the runner. + $stdoutTask = $process.StandardOutput.ReadToEndAsync() + $stderrTask = $process.StandardError.ReadToEndAsync() + if ($null -ne $InputBytes -and $InputBytes.Length -gt 0) { + $inputTask = $process.StandardInput.BaseStream.WriteAsync($InputBytes, 0, $InputBytes.Length) + while (-not $inputTask.IsCompleted) { + $remainingTotal = ($deadline - [DateTime]::UtcNow).TotalMilliseconds + if ($remainingTotal -le 0) { + $timedOut = $true + break + } + $remaining = [int][Math]::Max(1, [Math]::Min(5000, $remainingTotal)) + if (Wait-RunnerTaskBounded -Task $inputTask -TimeoutMilliseconds $remaining) { break } + } + if (-not $timedOut) { + # The task is complete, so this cannot introduce an + # unbounded wait. It propagates a real pipe-write failure. + [void]$inputTask.GetAwaiter().GetResult() + } + } + try { $process.StandardInput.Close() } catch { } + + if (-not $timedOut) { + while (-not $process.HasExited) { + $remainingTotal = ($deadline - [DateTime]::UtcNow).TotalMilliseconds + if ($remainingTotal -le 0) { + $timedOut = $true + break + } + $remaining = [int][Math]::Max(1, [Math]::Min(5000, $remainingTotal)) + if ($process.WaitForExit($remaining)) { break } + } + if (-not $timedOut) { $terminationObserved = [bool]$process.HasExited } + } + + if ($timedOut) { + $killAttempted = $true + try { $process.Kill($true) } catch { } + # A forced termination is allowed a small, finite grace period. + # Never call the parameterless WaitForExit on this path: a child + # or inherited pipe can remain alive indefinitely. + try { + if (-not $process.HasExited) { [void]$process.WaitForExit(5000) } + } catch { } + try { $terminationObserved = [bool]$process.HasExited } catch { $terminationObserved = $false } + } + + # Drain stdout/stderr only until one shared finite deadline. If a + # descendant keeps a pipe open, return the bounded result instead of + # waiting forever for ReadToEndAsync(). + $drainDeadline = [DateTime]::UtcNow.AddSeconds(5) + foreach ($stream in @( + [pscustomobject]@{ Task = $stdoutTask; Name = 'stdout' }, + [pscustomobject]@{ Task = $stderrTask; Name = 'stderr' } + )) { + if ($null -eq $stream.Task) { continue } + $remainingDrain = [int][Math]::Max(1, [Math]::Min(5000, ($drainDeadline - [DateTime]::UtcNow).TotalMilliseconds)) + $completed = Wait-RunnerTaskBounded -Task $stream.Task -TimeoutMilliseconds $remainingDrain + if ($stream.Name -eq 'stdout') { $stdoutDrainCompleted = $completed } else { $stderrDrainCompleted = $completed } + } + $stdout = Get-RunnerTaskResultIfCompleted -Task $stdoutTask + $stderr = Get-RunnerTaskResultIfCompleted -Task $stderrTask + $finish = [DateTime]::UtcNow + + return [pscustomobject]@{ + ExitCode = if ($timedOut -or -not $terminationObserved) { $null } else { $process.ExitCode } + TimedOut = $timedOut + Stdout = $stdout + Stderr = $stderr + StartedUtc = $start + FinishedUtc = $finish + DurationSeconds = [Math]::Round(($finish - $start).TotalSeconds, 3) + KillAttempted = $killAttempted + TerminationObserved = $terminationObserved + StdoutDrainCompleted = $stdoutDrainCompleted + StderrDrainCompleted = $stderrDrainCompleted + } + } catch { + if ($processStarted) { + try { $process.Kill($true) } catch { } + try { if (-not $process.HasExited) { [void]$process.WaitForExit(5000) } } catch { } + } + throw + } finally { + $process.Dispose() + } +} + +function Get-ProviderAuthenticationVariables { + param([string]$Provider) + + $normalized = ([string]$Provider).ToLowerInvariant() + switch -Regex ($normalized) { + '^openai$|^chatgpt$' { return @('OPENAI_API_KEY') } + '^anthropic$' { return @('ANTHROPIC_API_KEY') } + '^google$|^google-vertex$|^gemini$' { return @('GOOGLE_API_KEY', 'GEMINI_API_KEY') } + '^openrouter$' { return @('OPENROUTER_API_KEY') } + '^xai$|^x-ai$' { return @('XAI_API_KEY') } + '^mistral$' { return @('MISTRAL_API_KEY') } + + default { return @() } + } +} + +function Test-EnvironmentVariablePresent { + param([string[]]$Names) + + foreach ($name in @($Names)) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { + return $true + } + } + return $false +} + +function New-ArtifactReference { + param( + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][string]$Path, + [ValidateSet('run', 'package')][string]$Scope = 'run', + [string]$MediaType = 'application/octet-stream' + ) + + Assert-SafeRelativePath -RelativePath $Path -FieldName 'artifact.path' + $base = if ($Scope -eq 'run') { $Run.RunRoot } else { Split-Path -Parent (Split-Path -Parent $Run.RunRoot) } + $full = Resolve-ContainedPath -BasePath $base -RelativePath $Path -FieldName 'artifact.path' -Kind File + return [ordered]@{ + path = $Path.Replace('\', '/') + scope = $Scope + sha256 = Get-Sha256HexFromFile -Path $full + size = (Get-Item -LiteralPath $full).Length + media_type = $MediaType + } +} + +function Get-MediaType { + param([string]$Path) + + switch ([System.IO.Path]::GetExtension($Path).ToLowerInvariant()) { + '.jsonl' { return 'application/x-ndjson' } + '.json' { return 'application/json' } + '.txt' { return 'text/plain; charset=utf-8' } + '.md' { return 'text/markdown; charset=utf-8' } + default { return 'application/octet-stream' } + } +} + +function ConvertFrom-JsonLines { + param([Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text) + + $events = [System.Collections.Generic.List[object]]::new() + $errors = [System.Collections.Generic.List[string]]::new() + $lineNumber = 0 + foreach ($line in ($Text -split "`r?`n")) { + $lineNumber++ + if ([string]::IsNullOrWhiteSpace($line)) { continue } + try { + $events.Add(($line | ConvertFrom-Json)) + } catch { + $errors.Add("line ${lineNumber}: $($_.Exception.Message)") + } + } + return [pscustomobject]@{ Events = @($events); Errors = @($errors) } +} + +function Get-OutputTextFromFinalResponse { + param([object]$Result) + + if ([string](Get-JsonProperty -Object $Result.final_response -Name 'status' -Default '') -eq 'available') { + return [string]$Result.final_response.text + } + return $null +} diff --git a/scripts/eval-runners/tests/fixtures/codex-events.jsonl b/scripts/eval-runners/tests/fixtures/codex-events.jsonl new file mode 100644 index 0000000..3a3399f --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/codex-events.jsonl @@ -0,0 +1,5 @@ +{"type":"thread.started","thread_id":"fixture-thread"} +{"type":"turn.started"} +{"type":"item.completed","item":{"type":"agent_message","text":"fixture response"}} +{"type":"turn.completed","usage":{"input_tokens":12,"cached_input_tokens":3,"output_tokens":4}} +{"type":"future.event.v99","payload":"unknown"} diff --git a/scripts/eval-runners/tests/fixtures/codex-thread-start-rejection.jsonl b/scripts/eval-runners/tests/fixtures/codex-thread-start-rejection.jsonl new file mode 100644 index 0000000..7db98b0 --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/codex-thread-start-rejection.jsonl @@ -0,0 +1,3 @@ +{"jsonrpc":"2.0","id":1,"result":{"userAgent":"codebelt-agentic-eval-runner/0.9.1","codexHome":"C:\\Users\\test\\AppData\\Local\\Temp\\agentic-codex-auth-home","platformFamily":"windows","platformOs":"windows"}} +{"jsonrpc":"2.0","method":"remoteControl/statusChanged","params":{"status":"enabled"}} +{"jsonrpc":"2.0","id":2,"error":{"code":-32600,"message":"Invalid request: unknown variant `readOnly`, expected one of `read-only`, `workspace-write`, `danger-full-access`"}} diff --git a/scripts/eval-runners/tests/fixtures/copilot-events.jsonl b/scripts/eval-runners/tests/fixtures/copilot-events.jsonl new file mode 100644 index 0000000..e3b5a1a --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/copilot-events.jsonl @@ -0,0 +1,9 @@ +{"type":"session.start","id":"e1","parentId":null,"timestamp":"2026-01-01T00:00:00.000Z","data":{"sessionId":"fixture-session"}} +{"type":"user.message","id":"e2","parentId":"e1","timestamp":"2026-01-01T00:00:00.100Z","data":{"content":"fixture prompt"}} +{"type":"assistant.message","id":"e3","parentId":"e2","timestamp":"2026-01-01T00:00:01.000Z","data":{"messageId":"m1","model":"claude-haiku-4.5","content":"fixture progress note"}} +{"type":"tool.execution_start","id":"e4","parentId":"e3","timestamp":"2026-01-01T00:00:01.200Z","data":{"callId":"t1","toolName":"str_replace_editor"}} +{"type":"tool.execution_complete","id":"e5","parentId":"e4","timestamp":"2026-01-01T00:00:01.800Z","data":{"callId":"t1","status":"success"}} +{"type":"assistant.message","id":"e6","parentId":"e5","timestamp":"2026-01-01T00:00:02.000Z","data":{"messageId":"m2","model":"claude-haiku-4.5","content":"fixture final response"}} +{"type":"assistant.usage","id":"e7","parentId":"e6","ephemeral":true,"timestamp":"2026-01-01T00:00:02.100Z","data":{"model":"claude-haiku-4.5","inputTokens":12,"outputTokens":4,"cacheReadTokens":3,"numToolCalls":1,"cost":0.2,"finishReason":"stop"}} +{"type":"session.task_complete","id":"e8","parentId":"e7","timestamp":"2026-01-01T00:00:02.200Z","data":{}} +{"type":"future.event.v99","payload":"unknown"} diff --git a/scripts/eval-runners/tests/fixtures/copilot-help-exact-session.txt b/scripts/eval-runners/tests/fixtures/copilot-help-exact-session.txt new file mode 100644 index 0000000..5b6a72e --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/copilot-help-exact-session.txt @@ -0,0 +1,11 @@ +Usage: copilot [options] + -C, --directory Working directory + --model Model selector + --output-format Structured output format + --allow-all-tools Allow tools + --no-ask-user Disable prompts + --disable-builtin-mcps Disable built-in MCP servers + --secret-env-vars Secret environment variables + --no-auto-update Disable updates + --resume Resume a previous Copilot session by its exact session id + --continue Continue the most recent session diff --git a/scripts/eval-runners/tests/fixtures/copilot-help-no-exact-session.txt b/scripts/eval-runners/tests/fixtures/copilot-help-no-exact-session.txt new file mode 100644 index 0000000..22e78b2 --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/copilot-help-no-exact-session.txt @@ -0,0 +1,12 @@ +Usage: copilot [options] + -C, --directory Working directory + --model Model selector + --output-format Structured output format + --allow-all-tools Allow tools + --no-ask-user Disable prompts + --disable-builtin-mcps Disable built-in MCP servers + --secret-env-vars Secret environment variables + --no-auto-update Disable updates + --resume Resume the most recent Copilot session by id + --session-id Set the session id for a new session + --continue Continue the most recent session diff --git a/scripts/eval-runners/tests/fixtures/copilot-scripted-turn-1-events.jsonl b/scripts/eval-runners/tests/fixtures/copilot-scripted-turn-1-events.jsonl new file mode 100644 index 0000000..f232def --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/copilot-scripted-turn-1-events.jsonl @@ -0,0 +1,4 @@ +{"type":"session.start","timestamp":"2026-01-01T00:00:01.000Z","data":{"sessionId":"fixture-copilot-session"}} +{"type":"assistant.message","timestamp":"2026-01-01T00:00:02.000Z","data":{"model":"claude-haiku-4.5","content":"copilot scripted turn one"}} +{"type":"assistant.usage","timestamp":"2026-01-01T00:00:03.000Z","data":{"model":"claude-haiku-4.5","inputTokens":2,"outputTokens":3}} +{"type":"session.task_complete","timestamp":"2026-01-01T00:00:04.000Z","data":{}} diff --git a/scripts/eval-runners/tests/fixtures/copilot-scripted-turn-2-events.jsonl b/scripts/eval-runners/tests/fixtures/copilot-scripted-turn-2-events.jsonl new file mode 100644 index 0000000..7efba03 --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/copilot-scripted-turn-2-events.jsonl @@ -0,0 +1,4 @@ +{"type":"session.start","timestamp":"2026-01-01T00:01:01.000Z","data":{"sessionId":"fixture-copilot-session"}} +{"type":"assistant.message","timestamp":"2026-01-01T00:01:02.000Z","data":{"model":"claude-haiku-4.5","content":"copilot scripted final turn"}} +{"type":"assistant.usage","timestamp":"2026-01-01T00:01:03.000Z","data":{"model":"claude-haiku-4.5","inputTokens":4,"outputTokens":5}} +{"type":"session.task_complete","timestamp":"2026-01-01T00:01:04.000Z","data":{}} diff --git a/scripts/eval-runners/tests/fixtures/opencode-debug-config.json b/scripts/eval-runners/tests/fixtures/opencode-debug-config.json new file mode 100644 index 0000000..ee7b0c9 --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/opencode-debug-config.json @@ -0,0 +1,7 @@ +{ + "$schema": "https://opencode.ai/config.json", + "permission": { + "skill": "deny" + }, + "username": "unknown" +} diff --git a/scripts/eval-runners/tests/fixtures/opencode-events.jsonl b/scripts/eval-runners/tests/fixtures/opencode-events.jsonl new file mode 100644 index 0000000..d19e2df --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/opencode-events.jsonl @@ -0,0 +1,5 @@ +{"type":"step_start","timestamp":"2026-01-01T00:00:01.000Z","sessionID":"fixture-opencode-session","modelID":"opencode/muse-spark-1.2-contributor-free","part":{"id":"fixture-step"}} +{"type":"text","timestamp":"2026-01-01T00:00:02.000Z","sessionID":"fixture-opencode-session","modelID":"opencode/muse-spark-1.2-contributor-free","part":{"text":"fixture response"}} +{"type":"tool_use","timestamp":"2026-01-01T00:00:03.000Z","sessionID":"fixture-opencode-session","modelID":"opencode/muse-spark-1.2-contributor-free","part":{"tool":"read"}} +{"type":"step_finish","timestamp":"2026-01-01T00:00:04.000Z","sessionID":"fixture-opencode-session","modelID":"opencode/muse-spark-1.2-contributor-free","part":{"tokens":{"input":12,"output":4},"cost":0.01}} +{"type":"future.event.v99","payload":"unknown"} diff --git a/scripts/eval-runners/tests/fixtures/opencode-scripted-turn-1-events.jsonl b/scripts/eval-runners/tests/fixtures/opencode-scripted-turn-1-events.jsonl new file mode 100644 index 0000000..3b65be1 --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/opencode-scripted-turn-1-events.jsonl @@ -0,0 +1,3 @@ +{"type":"step_start","timestamp":"2026-01-01T00:00:01.000Z","sessionID":"fixture-opencode-session","modelID":"opencode/muse-spark-1.2-contributor-free"} +{"type":"text","timestamp":"2026-01-01T00:00:02.000Z","sessionID":"fixture-opencode-session","text":"opencode scripted turn one"} +{"type":"step_finish","timestamp":"2026-01-01T00:00:03.000Z","sessionID":"fixture-opencode-session","modelID":"opencode/muse-spark-1.2-contributor-free","part":{"tokens":{"input":2,"output":3},"cost":0.01}} diff --git a/scripts/eval-runners/tests/fixtures/opencode-scripted-turn-2-events.jsonl b/scripts/eval-runners/tests/fixtures/opencode-scripted-turn-2-events.jsonl new file mode 100644 index 0000000..5521746 --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/opencode-scripted-turn-2-events.jsonl @@ -0,0 +1,3 @@ +{"type":"step_start","timestamp":"2026-01-01T00:01:01.000Z","sessionID":"fixture-opencode-session","modelID":"opencode/muse-spark-1.2-contributor-free"} +{"type":"text","timestamp":"2026-01-01T00:01:02.000Z","sessionID":"fixture-opencode-session","text":"opencode scripted final turn"} +{"type":"step_finish","timestamp":"2026-01-01T00:01:03.000Z","sessionID":"fixture-opencode-session","modelID":"opencode/muse-spark-1.2-contributor-free","part":{"tokens":{"input":4,"output":5},"cost":0.02}} diff --git a/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 b/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 new file mode 100644 index 0000000..5edb3b1 --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1 @@ -0,0 +1,266 @@ +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)][ValidateSet('describe', 'preflight', 'execute')][string]$Command, + [string]$Run, + [string]$Profile +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'fixture' + version = 'test' + platforms = @('windows') + harness = [ordered]@{ name = 'deterministic runner-owned fixture'; version = 'test' } + capabilities = [ordered]@{ + single_turn = 'supported' + scripted_multi_turn_same_session = 'supported' + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'unsupported' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + native_worker_delegation = 'supported' + delegated_worker_full_capability = 'supported' + delegated_worker_model_lock = 'supported' + delegated_worker_working_directory = 'supported' + delegated_worker_result_capture = 'supported' + delegated_worker_capacity_signal = 'supported' + } + delegation = [ordered]@{ + dispatch_owner = 'runner' + mode = 'native_worker' + mechanism = 'deterministic-runner-owned-fixture' + worker_role = 'fixture-worker' + full_capability = 'supported' + model_lock = 'supported' + working_directory = 'supported' + result_capture = 'supported' + capacity = 'supported' + nested_model_execution = $false + } + supported_telemetry = @() + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + if ($Command -eq 'describe') { Write-RunnerJson -Value $descriptor -AsOutput; exit 0 } + $inputs = [pscustomobject]@{ Run = Resolve-RunContract -RunPath $Run; Profile = Resolve-ExecutionProfile -ProfilePath $Profile } + function Write-FixtureEvent { + param([Parameter(Mandatory = $true)][string]$Kind) + + $logPath = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG') + if ([string]::IsNullOrWhiteSpace($logPath)) { return } + $mutex = [Threading.Mutex]::new($false, 'agentic-runner-owned-fixture-event-log') + try { + [void]$mutex.WaitOne() + $event = [ordered]@{ + kind = $Kind + eval_id = [int]$inputs.Run.EvalId + eval_name = [string]$inputs.Run.EvalName + configuration = [string]$inputs.Run.Mode + utc = [DateTime]::UtcNow.ToString('o') + } + [IO.File]::AppendAllText($logPath, (($event | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + } finally { + try { [void]$mutex.ReleaseMutex() } catch { } + $mutex.Dispose() + } + } + + function Get-FixtureTerminalStatus { + $statusPath = Join-Path $inputs.Run.HomeDirectoryPath 'terminal-status' + if (-not (Test-Path -LiteralPath $statusPath -PathType Leaf)) { + return 'completed' + } + + $status = ([System.IO.File]::ReadAllText($statusPath, [Text.UTF8Encoding]::new($false))).Trim() + if ($status -notin @('completed', 'failed', 'timed_out', 'cancelled')) { + throw "Unsupported fixture terminal status '$status'." + } + return $status + } + + function Test-FixtureEvidenceValidationFailure { + return (Test-Path -LiteralPath (Join-Path $inputs.Run.HomeDirectoryPath 'evidence-validation-failed') -PathType Leaf) + } + + if ($Command -eq 'preflight') { + Write-FixtureEvent -Kind 'preflight' + if (Test-Path -LiteralPath (Join-Path $inputs.Run.HomeDirectoryPath 'preflight-incompatible') -PathType Leaf) { + $preflightCapabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { $preflightCapabilities[$capabilityName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) } + $preflightReason = "fixture preflight rejected $($inputs.Run.EvalName)/$($inputs.Run.Mode)" + if ($null -ne $inputs.Run.Interaction) { + $preflightCapabilities.scripted_multi_turn_same_session = 'unsupported' + $preflightReason += ': scripted interaction capability unsupported' + } + Write-RunnerJson -Value (New-PreflightDocument -Descriptor $descriptor -Profile $inputs.Profile -Run $inputs.Run -Compatible $false -Reasons @($preflightReason) -ResolvedCapabilities $preflightCapabilities -Mechanisms @('deterministic fixture')) -AsOutput + exit 0 + } + Write-RunnerJson -Value (New-PreflightDocument -Descriptor $descriptor -Profile $inputs.Profile -Run $inputs.Run -Compatible $true -ResolvedCapabilities $descriptor.capabilities -Mechanisms @('deterministic fixture')) -AsOutput + exit 0 + } + # Refuse a second execution before any transport event or raw artifact can + # be created after Phase 1 has been frozen. + [void](Assert-PhaseOneEvidenceWritable -Run $inputs.Run) + $executeStartUtc = [DateTime]::UtcNow + Write-FixtureEvent -Kind 'execute' + # An optional per-run marker lets a test make one arm deliberately slow so + # the fan-out's capacity refill can be observed: a fast sibling must free its + # slot and let the next pending arm start without waiting for the slow arm. + $delayMs = 250 + $delayMarker = Join-Path $inputs.Run.HomeDirectoryPath 'execute-delay-ms' + if (Test-Path -LiteralPath $delayMarker -PathType Leaf) { + $parsedDelay = 0 + if ([int]::TryParse((([System.IO.File]::ReadAllText($delayMarker, [Text.UTF8Encoding]::new($false))).Trim()), [ref]$parsedDelay) -and $parsedDelay -ge 0) { $delayMs = $parsedDelay } + } + Start-Sleep -Milliseconds $delayMs + $executeFinishUtc = [DateTime]::UtcNow + $capabilities = [ordered]@{} + foreach ($propertyName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { $capabilities[$propertyName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $propertyName) } + if ([string]$inputs.Run.Mode -eq 'without_skill') { $capabilities.candidate_skill_exposure = 'excluded' } + $sessionId = ('fixture-session-' + [Guid]::NewGuid().ToString('N')) + $terminalStatus = Get-FixtureTerminalStatus + $turnRecords = [System.Collections.Generic.List[object]]::new() + $fixtureFinalResponse = 'deterministic fixture response' + if ($null -ne $inputs.Run.Interaction) { + $requestedTurns = @($inputs.Run.Interaction.turns) + for ($turnIndex = 0; $turnIndex -lt $requestedTurns.Count; $turnIndex++) { + $turnText = Get-InteractionTurnText -Turn $requestedTurns[$turnIndex] -RunData $inputs.Run + $turnRecords.Add([ordered]@{ sequence = ($turnIndex * 2) + 1; role = 'user'; content_sha256 = Get-Sha256HexFromBytes -Bytes ([Text.UTF8Encoding]::new($false).GetBytes($turnText)); session_id = $sessionId; timestamp_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow) }) + $assistantText = if ($turnIndex -eq 0) { 'fixture confirmation required' } else { 'fixture protected operation completed after confirmation' } + $turnRecords.Add([ordered]@{ sequence = ($turnIndex * 2) + 2; role = 'assistant'; text = $assistantText; session_id = $sessionId; timestamp_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow) }) + $fixtureFinalResponse = $assistantText + } + } + # Deterministic validation can request stable worker metrics without + # changing the normal fixture behavior. These values are injected before + # the Phase 1 freeze, never by grading or report code. + $finalResponseOverride = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_FINAL_RESPONSE') + if (-not [string]::IsNullOrWhiteSpace($finalResponseOverride)) { + $fixtureFinalResponse = $finalResponseOverride + if ($turnRecords.Count -gt 0) { + $lastTurn = $turnRecords[$turnRecords.Count - 1] + if ($lastTurn -is [System.Collections.IDictionary]) { + $lastTurn['text'] = $finalResponseOverride + } else { + Add-Member -InputObject $lastTurn -MemberType NoteProperty -Name text -Value $finalResponseOverride -Force + } + } + } + $durationSeconds = [Math]::Round(($executeFinishUtc - $executeStartUtc).TotalSeconds, 3) + $durationOverride = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_DURATION_SECONDS') + $parsedDuration = 0.0 + if (-not [string]::IsNullOrWhiteSpace($durationOverride) -and [double]::TryParse($durationOverride, [Globalization.NumberStyles]::Float, [Globalization.CultureInfo]::InvariantCulture, [ref]$parsedDuration) -and $parsedDuration -ge 0) { + $durationSeconds = $parsedDuration + } + $fixtureTelemetryTokens = New-UnavailableMetric -Reason 'fixture' + $fixtureTelemetryToolCalls = New-AvailableMetric -Value 0 + if ([Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_METRICS') -eq '1') { + $fixtureTelemetryTokens = New-AvailableMetric -Value ([ordered]@{ + input_tokens = 27 + output_tokens = 123 + total_tokens = 123 + cached_input_tokens = 456 + cache_write_tokens = 78 + }) + $fixtureTelemetryToolCalls = New-AvailableMetric -Value 2 + } + $eventsPath = Join-Path $inputs.Run.RunRoot 'evidence/fixture-events.jsonl' + New-Item -ItemType Directory -Path (Split-Path -Parent $eventsPath) -Force | Out-Null + $eventLines = [System.Collections.Generic.List[string]]::new() + if ($turnRecords.Count -gt 0) { + $eventSequence = 0 + foreach ($turn in @($turnRecords.ToArray())) { + if ([string]$turn.role -eq 'assistant' -and [int]$turn.sequence -eq 4) { + # The confirmation fixture models the protected operation as a + # transport event between the second user dispatch and its + # terminal response. It can therefore prove the operation did + # not happen in the first turn without consulting model text. + $eventSequence++ + $eventLines.Add(([ordered]@{ + type = 'protected.operation' + sequence = $eventSequence + turn_sequence = [int]$turn.sequence + session_id = [string]$turn.session_id + timestamp_utc = Format-UtcTimestamp -Value ([DateTime]::UtcNow) + } | ConvertTo-Json -Compress)) + } + $eventSequence++ + $eventLines.Add(([ordered]@{ + type = if ([string]$turn.role -eq 'user') { 'user.dispatched' } else { 'assistant.terminal' } + sequence = $eventSequence + turn_sequence = [int]$turn.sequence + session_id = [string]$turn.session_id + timestamp_utc = [string]$turn.timestamp_utc + } | ConvertTo-Json -Compress)) + } + } else { + $eventLines.Add(([ordered]@{ type = 'assistant.terminal'; sequence = 1; session_id = $sessionId; timestamp_utc = Format-UtcTimestamp -Value $executeFinishUtc } | ConvertTo-Json -Compress)) + } + [IO.File]::WriteAllText($eventsPath, ([string]::Join("`n", $eventLines) + "`n"), [Text.UTF8Encoding]::new($false)) + $eventsArtifact = New-ArtifactReference -Run $inputs.Run -Path 'evidence/fixture-events.jsonl' -Scope run -MediaType 'application/x-ndjson; charset=utf-8' + $evidence = [ordered]@{ + capture = [ordered]@{ source = 'harness_native_transport'; terminal = $true; worker_authored = $false } + delegation = [ordered]@{ + dispatch_owner = 'runner' + mechanism = 'deterministic-runner-owned-fixture' + worker_session_id = $sessionId + observed_model = [string]$inputs.Profile.Model + observed_working_directory = [string]$inputs.Run.WorkingDirectoryPath + observed_home = [string]$inputs.Run.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$inputs.Run.PromptHash + terminal_result_capture = $true + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + } + turns = if ($turnRecords.Count -gt 0) { @($turnRecords.ToArray()) } else { $null } + } + if (Test-FixtureEvidenceValidationFailure) { + $evidence.delegation.prompt_fidelity = $false + } + if ($turnRecords.Count -gt 0 -and $terminalStatus -eq 'completed') { + $evidence.interaction = [ordered]@{ schema = (Get-RunnerSchemaNames).Interaction; mode = 'scripted'; same_session = $true; session_id = $sessionId; turns = @($turnRecords.ToArray()); final_response_sequence = $turnRecords.Count } + } + $finalResponse = if ($terminalStatus -eq 'completed') { $fixtureFinalResponse } else { $null } + $finalResponseReason = switch ($terminalStatus) { + 'failed' { 'fixture_failure' } + 'timed_out' { 'fixture_timeout' } + 'cancelled' { 'fixture_cancelled' } + default { $null } + } + $exitStatus = switch ($terminalStatus) { + 'completed' { [Nullable[int]]0 } + 'failed' { [Nullable[int]]17 } + 'cancelled' { [Nullable[int]]130 } + default { $null } + } + $failure = switch ($terminalStatus) { + 'failed' { New-ExecutionFailure -Code 'fixture_harness_failure' -Message 'The deterministic runner-owned fixture reported a harness failure.' } + 'timed_out' { New-ExecutionFailure -Code 'timed_out' -Message 'The deterministic runner-owned fixture exceeded its execution window.' } + 'cancelled' { New-ExecutionFailure -Code 'cancelled' -Message 'The deterministic runner-owned fixture was cancelled before completion.' } + default { $null } + } + $result = New-ExecutionResult -Descriptor $descriptor -Profile $inputs.Profile -Run $inputs.Run -Status $terminalStatus -FinalResponse $finalResponse -FinalResponseReason $finalResponseReason -StartedUtc ($executeStartUtc.ToString('o')) -FinishedUtc ($executeFinishUtc.ToString('o')) -DurationSeconds $durationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @('deterministic runner-owned fixture') -Telemetry ([ordered]@{ transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/fixture-events.jsonl'; complete = $true }); tokens = $fixtureTelemetryTokens; tool_calls = $fixtureTelemetryToolCalls; cost = New-UnavailableMetric -Reason 'fixture' }) -Artifacts @($eventsArtifact) -Evidence $evidence -AttemptCount 1 + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/tests/test-integrity-finalization.ps1 b/scripts/eval-runners/tests/test-integrity-finalization.ps1 new file mode 100644 index 0000000..f288839 --- /dev/null +++ b/scripts/eval-runners/tests/test-integrity-finalization.ps1 @@ -0,0 +1,480 @@ +<#! +.SYNOPSIS + Deterministic frozen-evidence, grading-isolation, and finalization tests. + +.DESCRIPTION + Builds a package around the repository's model-free runner fixture. The + fixture produces six native terminal results, including one scripted + interaction case. This suite deliberately simulates the latest Copilot + failure by writing grading into a raw execution result after Phase 1 and + verifies that no later bridge or finalizer can bless or repair it. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +$repositoryRoot = (Resolve-Path (Join-Path $runnerRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'manifest-paths.ps1') +. (Join-Path $runnerRoot 'orchestration.ps1') +. (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'package-integrity.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -cne [string]$Actual) { + throw "ASSERT: $Message (expected '$Expected', got '$Actual')" + } +} + +function Assert-Contains { + param([string]$Text, [string]$Expected, [string]$Message) + if ($Text.IndexOf($Expected, [System.StringComparison]::OrdinalIgnoreCase) -lt 0) { + throw "ASSERT: $Message (missing '$Expected')" + } +} + +function Write-TestJson { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) + + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Read-TestJson { + param([Parameter(Mandatory = $true)][string]$Path) + return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 +} + +function Invoke-TestTool { + param( + [Parameter(Mandatory = $true)][string]$Path, + [Parameter(Mandatory = $true)][string[]]$Arguments + ) + + $output = & pwsh -NoProfile -File $Path @Arguments 2>&1 + $exitCode = $LASTEXITCODE + return [pscustomobject]@{ + ExitCode = $exitCode + Text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + } +} + +function Invoke-ForegroundPhaseOne { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][string]$IterationDirectory) + + $invocation = Invoke-TestTool -Path $Path -Arguments @('-IterationDirectory', $IterationDirectory) + $document = $invocation.Text | ConvertFrom-Json -Depth 100 + return [pscustomobject]@{ ExitCode = $invocation.ExitCode; Text = $invocation.Text; Document = $document } +} + +function Assert-ToolPasses { + param([Parameter(Mandatory = $true)][object]$Invocation, [Parameter(Mandatory = $true)][string]$Description) + if ([int]$Invocation.ExitCode -ne 0) { throw "ASSERT: $Description failed: $($Invocation.Text)" } +} + +function Assert-ToolFails { + param( + [Parameter(Mandatory = $true)][object]$Invocation, + [Parameter(Mandatory = $true)][string]$Description, + [string]$ExpectedText = '' + ) + if ([int]$Invocation.ExitCode -eq 0) { throw "ASSERT: $Description unexpectedly passed: $($Invocation.Text)" } + if (-not [string]::IsNullOrWhiteSpace($ExpectedText)) { Assert-Contains -Text $Invocation.Text -Expected $ExpectedText -Message $Description } +} + +function New-TestRun { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][int]$EvalId, + [Parameter(Mandatory = $true)][string]$EvalName, + [Parameter(Mandatory = $true)][string]$Configuration, + [object]$Interaction = $null + ) + + $evalDirectory = Join-Path $IterationDirectory $EvalName + $runDirectory = Join-Path $evalDirectory $Configuration + $repoDirectory = Join-Path $runDirectory 'repo' + $homeDirectory = Join-Path $runDirectory 'home' + New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'execute-delay-ms'), '0', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "deterministic fixture prompt for $EvalName/$Configuration`n", [System.Text.UTF8Encoding]::new($false)) + + $skillDirectory = $null + if ($Configuration -eq 'with_skill') { + $skillDirectory = Join-Path $runDirectory 'skill/test-skill' + New-Item -ItemType Directory -Path $skillDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $skillDirectory 'SKILL.md'), '# deterministic fixture skill`n', [System.Text.UTF8Encoding]::new($false)) + } + + $interactionFile = $null + $interactionHash = $null + if ($null -ne $Interaction) { + $interactionFile = Join-Path $runDirectory 'interaction.json' + Write-TestJson -Path $interactionFile -Value $Interaction + $interactionHash = Get-Sha256HexFromFile -Path $interactionFile + } + + $run = [ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $EvalId + evalName = $EvalName + skillName = if ($Configuration -eq 'with_skill') { 'test-skill' } else { $null } + iteration = 1 + mode = $Configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = if ($Configuration -eq 'with_skill') { 'skill/test-skill' } else { $null } + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + inputFiles = @() + fixtureHash = ('a' * 64) + skillHash = if ($Configuration -eq 'with_skill') { ('b' * 64) } else { $null } + contract = [ordered]@{ + sandboxRoot = '.' + workingDirectory = 'repo' + homeDirectory = 'home' + mustNotReadOutsideSandbox = $true + mustNotExposeGlobalSkillsOrConfig = $true + } + } + if ($null -ne $interactionFile) { + $run.interactionFile = 'interaction.json' + $run.interactionHash = $interactionHash + } + Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value $run + + return [pscustomobject]@{ + Directory = $runDirectory + RunPath = Join-Path $runDirectory 'run.json' + InteractionPath = $interactionFile + } +} + +function New-TestGradingDocument { + param([Parameter(Mandatory = $true)][object[]]$Records) + + $entries = [System.Collections.Generic.List[object]]::new() + foreach ($record in @($Records | Sort-Object EvalId, Configuration)) { + $metadata = Read-TestJson -Path $record.MetadataPath + $assertions = @($metadata.assertions) + for ($index = 0; $index -lt $assertions.Count; $index++) { + $entries.Add([ordered]@{ + eval_id = [int]$record.EvalId + eval_name = [string]$record.EvalName + configuration = [string]$record.Configuration + assertion_index = $index + assertion = [string]$assertions[$index] + passed = $true + evidence = 'deterministic grading-isolation fixture evidence' + }) + } + } + return [ordered]@{ schema = (Get-RunnerSchemaNames).Grading; grading = @($entries.ToArray()) } +} + +function Remove-TestReportArtifacts { + param([Parameter(Mandatory = $true)][string]$IterationDirectory) + foreach ($relative in @('report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { + $path = Join-Path $IterationDirectory $relative + if (Test-Path -LiteralPath $path -PathType Leaf) { Remove-Item -LiteralPath $path -Force } + } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-integrity-finalization-' + [Guid]::NewGuid().ToString('N')) +$oldReportMode = [Environment]::GetEnvironmentVariable('AGENTIC_TEST_REPORT_MODE') +try { + $iteration = Join-Path $testRoot 'iteration-1' + $packageTools = Join-Path $iteration 'tools/eval-runners' + New-Item -ItemType Directory -Path $packageTools -Force | Out-Null + + # The package receives the same runner tree a prepared package would carry; + # the selected fixture is deterministic and never calls a model. + foreach ($item in @(Get-ChildItem -LiteralPath $runnerRoot -Force)) { + Copy-Item -LiteralPath $item.FullName -Destination $packageTools -Recurse -Force + } + $fixtureDirectory = Join-Path $packageTools 'fixture' + New-Item -ItemType Directory -Path $fixtureDirectory -Force | Out-Null + Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests/fixtures/runner-owned-fixture.ps1') -Destination (Join-Path $fixtureDirectory 'runner.ps1') -Force + + $reportScript = Join-Path $iteration 'tools/test-report.ps1' + $reportScriptText = @' +[CmdletBinding()] +param([Parameter(Mandatory = $true)][string]$IterationDirectory, [switch]$RequireComplete) +$ErrorActionPreference = 'Stop' +$mode = [Environment]::GetEnvironmentVariable('AGENTIC_TEST_REPORT_MODE') +if ($mode -eq 'failure') { throw 'deterministic report fixture failure' } +$files = @('report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md') +$count = if ($mode -eq 'missing') { 3 } else { 4 } +for ($index = 0; $index -lt $count; $index++) { + $path = Join-Path $IterationDirectory $files[$index] + $content = if ($files[$index] -eq 'benchmark.json') { '{"schema":"deterministic-report-fixture/1","status":"completed"}' } else { "deterministic report fixture: $($files[$index])`n" } + [System.IO.File]::WriteAllText($path, $content, [System.Text.UTF8Encoding]::new($false)) +} +'@ + [System.IO.File]::WriteAllText($reportScript, $reportScriptText, [System.Text.UTF8Encoding]::new($false)) + + $interaction = [ordered]@{ + schema = (Get-RunnerSchemaNames).Interaction + mode = 'scripted' + turns = @( + [ordered]@{ role = 'user'; source = 'prompt.md' } + [ordered]@{ role = 'user'; content = 'Yes, proceed.' } + ) + } + + $manifestEvals = [System.Collections.Generic.List[object]]::new() + for ($evalId = 1; $evalId -le 3; $evalId++) { + $evalName = if ($evalId -eq 2) { 'dotnet-strong-name-signing-confirmation' } else { 'integrity-eval-{0:d2}' -f $evalId } + $evalDirectory = Join-Path $iteration $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + $assertion = if ($evalId -eq 2) { 'the protected operation is absent before confirmation and occurs only after the same-session confirmation turn' } else { 'the deterministic terminal response is captured' } + Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-metadata/1' + eval_id = $evalId + eval_name = $evalName + prompt = "fixture prompt $evalId" + expected_output = 'fixture output' + assertions = @($assertion) + }) + $runs = [ordered]@{} + foreach ($configuration in @('with_skill', 'without_skill')) { + $interactionForRun = if ($evalId -eq 2) { $interaction } else { $null } + $run = New-TestRun -IterationDirectory $iteration -EvalId $evalId -EvalName $evalName -Configuration $configuration -Interaction $interactionForRun + $resultName = "$configuration.result.json" + $executionName = "$configuration.execution-result.json" + Write-TestJson -Path (Join-Path $evalDirectory "results/$resultName") -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + eval_id = $evalId + eval_name = $evalName + configuration = $configuration + execution_status = 'unrun' + grading = @([ordered]@{ text = $assertion; passed = $null; evidence = '' }) + }) + $runs[$configuration] = [ordered]@{ + mode = $configuration + run_manifest = "$evalName/$configuration/run.json" + execution_result = "$evalName/results/$executionName" + result = "$evalName/results/$resultName" + } + } + $manifestEvals.Add([ordered]@{ + eval_id = $evalId + eval_name = $evalName + directory = $evalName + metadata = "$evalName/eval-metadata.json" + runs = $runs + }) + } + + $toolIntegrity = Get-PackageTreeIntegrity -Root $packageTools + $manifest = [ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + configurations = @('with_skill', 'without_skill') + execution_profile = 'execution-profile.json' + runner_tools = 'tools/eval-runners' + runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } + execution_freeze = 'execution-freeze.json' + grading = 'grading.json' + report = [ordered]@{ tool = 'tools/test-report.ps1' } + evals = @($manifestEvals.ToArray()) + } + $profile = [ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fixture' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 60 + concurrency = 3 + } + Write-TestJson -Path (Join-Path $iteration 'manifest.json') -Value $manifest + Write-TestJson -Path (Join-Path $iteration 'execution-profile.json') -Value $profile + + $fanoutScript = Join-Path $packageTools 'invoke-runner-owned-arms.ps1' + $fanout = Invoke-ForegroundPhaseOne -Path $fanoutScript -IterationDirectory $iteration + Assert-ToolPasses -Invocation $fanout -Description 'runner-owned fixture Phase 1' + $fanoutSummary = $fanout.Document + Assert-Equal 'completed' $fanoutSummary.status 'six deterministic fixture arms complete' + Assert-Equal 6 $fanoutSummary.execution_count 'six raw execution results are registered' + Assert-True (Test-Path -LiteralPath (Join-Path $iteration 'execution-freeze.json') -PathType Leaf) 'Phase 1 writes an execution freeze' + + $freeze = Assert-ExecutionFreeze -IterationDirectory $iteration -RequireOrchestrationState + Assert-Equal 6 @($freeze.Freeze.executions).Count 'execution freeze contains all six arms' + Assert-Equal 'arm-1-with_skill' $freeze.Freeze.executions[0].worker_id 'freeze ordering is deterministic' + Assert-Equal 'arm-3-without_skill' $freeze.Freeze.executions[5].worker_id 'freeze ordering includes the final arm' + Assert-True (Test-Sha256 -Value ([string]$freeze.Freeze.orchestration_state_sha256)) 'freeze anchors the terminal orchestration ledger' + + $records = @(Get-ManifestRunRecords -IterationDirectory $iteration -Manifest (Read-TestJson -Path (Join-Path $iteration 'manifest.json')) | Sort-Object EvalId, Configuration) + $rawBeforeSecondPhase = @{} + foreach ($record in $records) { $rawBeforeSecondPhase[$record.ExecutionResultPath] = Get-Sha256HexFromFile -Path $record.ExecutionResultPath } + $secondPhaseOne = Invoke-TestTool -Path (Join-Path $packageTools 'fixture/runner.ps1') -Arguments @('execute', '-Run', $records[0].RunManifestPath, '-Profile', (Join-Path $iteration 'execution-profile.json')) + Assert-ToolFails -Invocation $secondPhaseOne -Description 'runner refuses execution after the raw-evidence freeze' -ExpectedText 'already frozen' + foreach ($record in $records) { Assert-Equal $rawBeforeSecondPhase[$record.ExecutionResultPath] (Get-Sha256HexFromFile -Path $record.ExecutionResultPath) 'post-freeze runner refusal leaves raw evidence unchanged' } + foreach ($record in $records) { + $raw = Read-TestJson -Path $record.ExecutionResultPath + [void](Assert-ExecutionResult -Result $raw) + if ($record.EvalId -eq 2) { + [void](Assert-InteractionResultEvidence -ExecutionResult $raw -RunData (Resolve-RunContract -RunPath $record.RunManifestPath)) + $turns = @($raw.evidence.interaction.turns) + Assert-Equal 4 $turns.Count 'strong-name confirmation fixture preserves two user/assistant pairs' + Assert-Equal ([string]$turns[0].session_id) ([string]$turns[3].session_id) 'scripted fixture preserves one session identity' + $eventPath = Join-Path (Split-Path -Parent $record.RunManifestPath) 'evidence/fixture-events.jsonl' + $events = @(Get-Content -LiteralPath $eventPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 5 $events.Count 'strong-name confirmation fixture records the protected operation event' + Assert-Equal 'user.dispatched' $events[0].type 'first scripted event is the user dispatch' + Assert-Equal 'assistant.terminal' $events[1].type 'second scripted event is the first assistant terminal response' + Assert-Equal 'user.dispatched' $events[2].type 'follow-up dispatch waits for first terminal response' + Assert-Equal 'protected.operation' $events[3].type 'protected operation occurs after the confirmation dispatch' + Assert-Equal 'assistant.terminal' $events[4].type 'final scripted event is the second assistant terminal response' + Assert-True ([int]$events[3].sequence -gt [int]$events[2].sequence) 'protected operation follows the confirmation user turn' + Assert-True ((@($events | Where-Object { $_.type -eq 'protected.operation' -and [int]$_.sequence -le [int]$events[2].sequence }).Count) -eq 0) 'protected operation evidence cannot appear before confirmation' + } + } + + $bridgeScript = Join-Path $packageTools 'bridge-manifest-results.ps1' + $bridgeArguments = @('-IterationDirectory', $iteration, '-RequireComplete', '-RequireParallelDispatch') + Assert-ToolPasses -Invocation (Invoke-TestTool -Path $bridgeScript -Arguments $bridgeArguments) -Description 'initial frozen manifest bridge' + Assert-ToolPasses -Invocation (Invoke-TestTool -Path $bridgeScript -Arguments $bridgeArguments) -Description 'unchanged idempotent manifest bridge' + + $statePath = Join-Path $iteration 'orchestration-state.json' + $stateBytes = [System.IO.File]::ReadAllBytes($statePath) + $tamperedState = Read-TestJson -Path $statePath + $tamperedState.max_observed_active = 1 + Write-TestJson -Path $statePath -Value $tamperedState + $tamperedStateBridge = Invoke-TestTool -Path $bridgeScript -Arguments $bridgeArguments + Assert-ToolFails -Invocation $tamperedStateBridge -Description 'bridge rejects post-freeze orchestration-state mutation' -ExpectedText 'orchestration-state.json changed' + [System.IO.File]::WriteAllBytes($statePath, $stateBytes) + Assert-ToolPasses -Invocation (Invoke-TestTool -Path $bridgeScript -Arguments $bridgeArguments) -Description 'bridge passes after exact orchestration-state restoration' + + $selectedRecord = $records[0] + $selectedRawBytes = [System.IO.File]::ReadAllBytes($selectedRecord.ExecutionResultPath) + $selectedRawHash = Get-Sha256HexFromFile -Path $selectedRecord.ExecutionResultPath + $selectedRaw = Read-TestJson -Path $selectedRecord.ExecutionResultPath + $selectedRaw | Add-Member -NotePropertyName grading -NotePropertyValue @([ordered]@{ passed = $true }) + Write-TestJson -Path $selectedRecord.ExecutionResultPath -Value $selectedRaw + $corruptedBridge = Invoke-TestTool -Path $bridgeScript -Arguments $bridgeArguments + Assert-ToolFails -Invocation $corruptedBridge -Description 'bridge rejects raw grading mutation' -ExpectedText 'Execution integrity failure' + Assert-Contains -Text $corruptedBridge.Text -Expected 'requires fresh Phase 1 execution' -Message 'raw mutation requires fresh Phase 1 execution' + $freezeAfterRawMutation = Read-TestJson -Path (Join-Path $iteration 'execution-freeze.json') + Assert-Equal $selectedRawHash (Get-JsonProperty -Object $freezeAfterRawMutation.executions[0] -Name 'execution_result_sha256') 'freeze hash is not re-blessed after raw mutation' + [System.IO.File]::WriteAllBytes($selectedRecord.ExecutionResultPath, $selectedRawBytes) + Assert-Equal $selectedRawHash (Get-Sha256HexFromFile -Path $selectedRecord.ExecutionResultPath) 'exact raw bytes are restored' + Assert-ToolPasses -Invocation (Invoke-TestTool -Path $bridgeScript -Arguments $bridgeArguments) -Description 'bridge passes after exact raw restoration' + + $artifactPath = Join-Path (Split-Path -Parent $selectedRecord.RunManifestPath) 'evidence/fixture-events.jsonl' + $artifactBytes = [System.IO.File]::ReadAllBytes($artifactPath) + [System.IO.File]::WriteAllBytes($artifactPath, $artifactBytes + [byte[]](0x20)) + $corruptedArtifactBridge = Invoke-TestTool -Path $bridgeScript -Arguments $bridgeArguments + Assert-ToolFails -Invocation $corruptedArtifactBridge -Description 'bridge rejects referenced artifact mutation' -ExpectedText 'Execution integrity failure' + [System.IO.File]::WriteAllBytes($artifactPath, $artifactBytes) + Assert-ToolPasses -Invocation (Invoke-TestTool -Path $bridgeScript -Arguments $bridgeArguments) -Description 'bridge passes after exact artifact restoration' + + $gradingPath = Join-Path $iteration 'grading.json' + $validGrading = New-TestGradingDocument -Records $records + Write-TestJson -Path $gradingPath -Value $validGrading + $canonicalBeforeGrading = @{} + foreach ($record in $records) { $canonicalBeforeGrading[$record.ResultPath] = Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object (Read-TestJson -Path $record.ResultPath) -PropertyName 'grading') } + $applyScript = Join-Path $packageTools 'apply-eval-grading.ps1' + $applyArguments = @('-IterationDirectory', $iteration, '-GradingPath', 'grading.json') + Assert-ToolPasses -Invocation (Invoke-TestTool -Path $applyScript -Arguments $applyArguments) -Description 'allowed grading-only artifact application' + foreach ($record in $records) { + $canonical = Read-TestJson -Path $record.ResultPath + Assert-Equal $canonicalBeforeGrading[$record.ResultPath] (Get-JsonFingerprint -Object (Get-JsonWithoutProperty -Object $canonical -PropertyName 'grading')) 'grading application leaves canonical non-grading fields unchanged' + } + + $invalidGrading = [ordered]@{ schema = (Get-RunnerSchemaNames).Grading; grading = @($validGrading.grading); output = 'raw output is forbidden here' } + Write-TestJson -Path $gradingPath -Value $invalidGrading + $invalidApply = Invoke-TestTool -Path $applyScript -Arguments $applyArguments + Assert-ToolFails -Invocation $invalidApply -Description 'grading artifact with raw output is rejected' -ExpectedText 'unsupported field' + + $forbiddenGradingFields = @('model', 'harness', 'execution_result_sha256', 'session_id', 'telemetry') + foreach ($forbiddenField in $forbiddenGradingFields) { + $forbiddenEntries = @($validGrading.grading | ForEach-Object { + $copy = [ordered]@{} + foreach ($name in @('eval_id', 'eval_name', 'configuration', 'assertion_index', 'assertion', 'passed', 'evidence')) { + $copy[$name] = Get-JsonProperty -Object $_ -Name $name + } + $copy[$forbiddenField] = 'forbidden' + $copy + }) + Write-TestJson -Path $gradingPath -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Grading; grading = $forbiddenEntries }) + $forbiddenApply = Invoke-TestTool -Path $applyScript -Arguments $applyArguments + Assert-ToolFails -Invocation $forbiddenApply -Description "grading artifact with $forbiddenField is rejected" -ExpectedText 'unsupported field' + } + Write-TestJson -Path $gradingPath -Value $validGrading + + $canonicalPath = $records[0].ResultPath + $canonicalBytes = [System.IO.File]::ReadAllBytes($canonicalPath) + $tamperedCanonical = Read-TestJson -Path $canonicalPath + $tamperedCanonical.output = 'manual canonical tampering' + Write-TestJson -Path $canonicalPath -Value $tamperedCanonical + $finalizerScript = Join-Path $packageTools 'finalize-eval-package.ps1' + $finalizerArguments = @('-IterationDirectory', $iteration, '-GradingPath', 'grading.json') + $canonicalTamperFinalizer = Invoke-TestTool -Path $finalizerScript -Arguments $finalizerArguments + Assert-ToolFails -Invocation $canonicalTamperFinalizer -Description 'finalizer rejects canonical non-grading mutation' -ExpectedText 'Execution integrity failure' + Remove-TestReportArtifacts -IterationDirectory $iteration + [System.IO.File]::WriteAllBytes($canonicalPath, $canonicalBytes) + + $canonicalApplyTamperBytes = [System.IO.File]::ReadAllBytes($canonicalPath) + $canonicalApplyTamper = Read-TestJson -Path $canonicalPath + $canonicalApplyTamper.output = 'direct application tampering' + Write-TestJson -Path $canonicalPath -Value $canonicalApplyTamper + $canonicalApplyTamperResult = Invoke-TestTool -Path $applyScript -Arguments $applyArguments + Assert-ToolFails -Invocation $canonicalApplyTamperResult -Description 'grading application rejects canonical non-grading mutation before applying' -ExpectedText 'Execution integrity failure' + [System.IO.File]::WriteAllBytes($canonicalPath, $canonicalApplyTamperBytes) + + # Exact reproduction of the latest Copilot mistake: grading is written to + # execution-result.json after a valid bridge. The finalizer must fail closed + # and must not repair the bytes or create report artifacts. + $copilotMistakeBytes = [System.IO.File]::ReadAllBytes($selectedRecord.ExecutionResultPath) + $copilotMistakeRaw = Read-TestJson -Path $selectedRecord.ExecutionResultPath + $copilotMistakeRaw | Add-Member -NotePropertyName grading -NotePropertyValue @([ordered]@{ text = 'wrong location' }) + Write-TestJson -Path $selectedRecord.ExecutionResultPath -Value $copilotMistakeRaw + Remove-TestReportArtifacts -IterationDirectory $iteration + $copilotMistakeFinalizer = Invoke-TestTool -Path $finalizerScript -Arguments $finalizerArguments + Assert-ToolFails -Invocation $copilotMistakeFinalizer -Description 'finalizer rejects Copilot raw-result grading mistake' -ExpectedText 'Execution integrity failure' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $iteration 'report.html') -PathType Leaf)) 'corrupted package produces no report' + [System.IO.File]::WriteAllBytes($selectedRecord.ExecutionResultPath, $copilotMistakeBytes) + Assert-Equal $selectedRawHash (Get-Sha256HexFromFile -Path $selectedRecord.ExecutionResultPath) 'Copilot mistake restoration returns exact frozen bytes' + + [Environment]::SetEnvironmentVariable('AGENTIC_TEST_REPORT_MODE', 'missing') + Remove-TestReportArtifacts -IterationDirectory $iteration + $missingReportFinalizer = Invoke-TestTool -Path $finalizerScript -Arguments $finalizerArguments + Assert-ToolFails -Invocation $missingReportFinalizer -Description 'finalizer rejects missing report artifact' + [Environment]::SetEnvironmentVariable('AGENTIC_TEST_REPORT_MODE', 'failure') + Remove-TestReportArtifacts -IterationDirectory $iteration + $failedReportFinalizer = Invoke-TestTool -Path $finalizerScript -Arguments $finalizerArguments + Assert-ToolFails -Invocation $failedReportFinalizer -Description 'finalizer rejects report generator failure' -ExpectedText 'Report generation failed' + + [Environment]::SetEnvironmentVariable('AGENTIC_TEST_REPORT_MODE', '') + Remove-TestReportArtifacts -IterationDirectory $iteration + $successfulFinalizer = Invoke-TestTool -Path $finalizerScript -Arguments $finalizerArguments + Assert-ToolPasses -Invocation $successfulFinalizer -Description 'deterministic finalizer success' + $finalSummary = $successfulFinalizer.Text | ConvertFrom-Json -Depth 100 | Select-Object -Last 1 + Assert-Equal 'completed' $finalSummary.status 'finalizer returns machine-readable completed status' + foreach ($relative in @('report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { + $artifact = Join-Path $iteration $relative + Assert-True (Test-Path -LiteralPath $artifact -PathType Leaf) "finalizer creates $relative" + Assert-True ((Get-Item -LiteralPath $artifact).Length -gt 0) "$relative is non-empty" + } + + Write-Output 'Eval package integrity and finalization: PASS' +} finally { + [Environment]::SetEnvironmentVariable('AGENTIC_TEST_REPORT_MODE', $oldReportMode) + if (Test-Path -LiteralPath $testRoot) { + Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue + } +} diff --git a/scripts/eval-runners/tests/test-mixed-terminal-regression.ps1 b/scripts/eval-runners/tests/test-mixed-terminal-regression.ps1 new file mode 100644 index 0000000..9d0d09a --- /dev/null +++ b/scripts/eval-runners/tests/test-mixed-terminal-regression.ps1 @@ -0,0 +1,182 @@ +<# +.SYNOPSIS + Regression: mixed-terminal fan-out preserves raw statuses and records evidence_validation separately. +.DESCRIPTION + MODEL-FREE deterministic check. Does not execute any model. +#> + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'orchestration.ps1') + +function Assert-True { param([bool]$c,[string]$m) if (-not $c) { throw "ASSERT: $m" } } +function Assert-Equal { param($e,$a,$m) if ([string]$e -ne [string]$a) { throw "ASSERT: $m (expected '$e', got '$a')" } } + +function New-TestNativeTerminalEvidence { + param( + [Parameter(Mandatory = $true)][object]$Arm, + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$WorkerSessionId + ) + + return [ordered]@{ + status = 'completed' + session = [ordered]@{ id = $WorkerSessionId; fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$Arm.eval_id; eval_name = [string]$Arm.eval_name; configuration = [string]$Arm.configuration } + requested = [ordered]@{ model = [string]$Arm.worker.model } + input = [ordered]@{ prompt_sha256 = [string]$RunData.PromptHash } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = 'deterministic-fake-native-worker' + worker_session_id = $WorkerSessionId + observed_model = [string]$Arm.worker.model + observed_working_directory = [string]$RunData.WorkingDirectoryPath + observed_home = [string]$RunData.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$RunData.PromptHash + terminal_result_capture = $true + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + } + } + } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-mixed-terminal-' + [Guid]::NewGuid().ToString('N')) +$iteration = Join-Path $testRoot 'iteration-1' +New-Item -ItemType Directory -Path $iteration -Force | Out-Null + +# Create 4 eval arms +$manifestEvals = [System.Collections.Generic.List[object]]::new() +for ($evalId = 1; $evalId -le 4; $evalId++) { + $evalName = 'eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $iteration $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'with_skill') -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'without_skill') -Force | Out-Null + Write-Output "prepared $evalName" + $runs = [ordered]@{} + foreach ($configuration in @('with_skill','without_skill')) { + $runPath = Join-Path $evalDirectory $configuration + New-Item -ItemType Directory -Path $runPath -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $runPath 'repo') -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $runPath 'home') -Force | Out-Null + if ($configuration -eq 'with_skill') { + New-Item -ItemType Directory -Path (Join-Path $runPath 'skill') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path (Join-Path $runPath 'skill') 'SKILL.md'), '# deterministic terminal test skill', [System.Text.UTF8Encoding]::new($false)) + } + [System.IO.File]::WriteAllText((Join-Path $runPath 'prompt.md'), "terminal test prompt for $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + $runJson = [ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $evalId + evalName = $evalName + mode = $configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = if ($configuration -eq 'with_skill') { 'skill' } else { $null } + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + mustNotReadOutsideSandbox = $true + fixtureHash = ('a' * 64) + skillHash = if ($configuration -eq 'with_skill') { ('b' * 64) } else { $null } + } + [System.IO.File]::WriteAllText((Join-Path $runPath 'run.json'), ($runJson | ConvertTo-Json -Depth 100), [System.Text.UTF8Encoding]::new($false)) + $resultsDir = Join-Path $evalDirectory 'results' + New-Item -ItemType Directory -Path $resultsDir -Force | Out-Null + $executionResultRel = "$evalName/results/$configuration.execution-result.json" + $resultRel = "$evalName/results/$configuration.result.json" + # create a canonical (empty) result.json so manifest validation is satisfied + [System.IO.File]::WriteAllText((Join-Path $resultsDir "$configuration.result.json"), (([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun'; grading = @() }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) + $runs[$configuration] = [ordered]@{ mode = $configuration; run_manifest = "$evalName/$configuration/run.json"; execution_result = $executionResultRel; result = $resultRel } + } + [System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('assertion') }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) + $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) +} + +$manifest = [ordered]@{ schema = (Get-RunnerSchemaNames).OrchestrationPlan; configurations = @('with_skill'); execution_freeze = 'execution-freeze.json'; evals = @($manifestEvals) } +$profile = [ordered]@{ schema = (Get-RunnerSchemaNames).Profile; runner = 'fake'; model = 'fixture-model'; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 60; concurrency = 4 } + +$plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile +$state = New-OrchestrationState -Plan $plan + +# Accept all workers +$dispatches = @(Get-NextWorkerDispatches -Plan $plan -State $state) +foreach ($d in $dispatches) { [void](Register-DelegationAccepted -State $state -WorkerId $d.worker_id -WorkerSessionId ('sess-' + $d.worker_id)) } +Assert-Equal 4 (Get-OrchestrationActiveCount -State $state) 'all workers active' + +# Prepare synthetic execution evidences +$arms = @($plan.arms) +$evidences = @{} +# arm1 -> timed_out +$arm1 = $arms[0] +$evidences[$arm1.worker_id] = [ordered]@{ + status = 'timed_out' + session = [ordered]@{ id = ('sess-' + $arm1.worker_id); fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$arm1.eval_id; eval_name = [string]$arm1.eval_name; configuration = [string]$arm1.configuration } + requested = [ordered]@{ model = [string]$arm1.worker.model } + evidence = [ordered]@{ delegation = [ordered]@{ mechanism = 'fake'; worker_session_id = ('sess-' + $arm1.worker_id); observed_model = [string]$arm1.worker.model; terminal_result_capture = $false } } +} +# arm2 -> completed (valid) +$arm2 = $arms[1] +$runData2 = Resolve-RunContract -RunPath ([string]$arm2.worker.run_manifest_path) +$evidences[$arm2.worker_id] = New-TestNativeTerminalEvidence -Arm $arm2 -RunData $runData2 -WorkerSessionId ('sess-' + $arm2.worker_id) +# arm3 -> failed +$arm3 = $arms[2] +$evidences[$arm3.worker_id] = [ordered]@{ + status = 'failed' + session = [ordered]@{ id = ('sess-' + $arm3.worker_id); fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$arm3.eval_id; eval_name = [string]$arm3.eval_name; configuration = [string]$arm3.configuration } + requested = [ordered]@{ model = [string]$arm3.worker.model } + evidence = [ordered]@{ delegation = [ordered]@{ mechanism = 'fake'; worker_session_id = ('sess-' + $arm3.worker_id); observed_model = [string]$arm3.worker.model; terminal_result_capture = $false } } +} +# arm4 -> completed +$arm4 = $arms[3] +$runData4 = Resolve-RunContract -RunPath ([string]$arm4.worker.run_manifest_path) +$evidences[$arm4.worker_id] = New-TestNativeTerminalEvidence -Arm $arm4 -RunData $runData4 -WorkerSessionId ('sess-' + $arm4.worker_id) + +# Sanity: ensure evidences exist for each arm +Write-Output "ARM KEYS:" +foreach ($arm in $arms) { Write-Output " - $($arm.worker_id)" } +Write-Output "EVIDENCE KEYS:" +foreach ($k in $evidences.Keys) { Write-Output " - $k" } + +# Register terminals +foreach ($arm in $arms) { + $w = [string]$arm.worker_id + if (-not $evidences.ContainsKey($w)) { throw "Missing synthetic evidence for $w" } + $exec = $evidences[$w] + [void](Register-WorkerTerminal -Plan $plan -State $state -WorkerId $w -ExecutionEvidence $exec) +} + +# Verify ledger preserves raw statuses and records evidence_validation +foreach ($arm in $arms) { + $w = $arm.worker_id + $ledger = $state.completed[$w] + $raw = $evidences[$w] + Assert-Equal $raw.status $ledger.status "ledger.status should equal raw for $w" + $ev = Get-JsonProperty -Object $ledger -Name 'evidence_validation' -Default $null + Write-Output "ledger[$w].native_worker_evidence_failures = $([string]::Join(', ', @($ledger.native_worker_evidence_failures | Where-Object {$_} )))" + Write-Output "ledger[$w].evidence_validation.status = $($ev.status)" + if ($raw.status -eq 'completed') { Assert-Equal 'passed' $ev.status "evidence_validation should pass for $w" } else { Assert-Equal 'failed' $ev.status "evidence_validation should fail for $w" } +} + +# Negative integrity: ledger mismatch must be rejected by Assert-FreezeTerminalLedgerEntry +# craft a fake record and raw object +$fakeRecord = [ordered]@{ EvalId = 999; Configuration = 'with_skill'; } +$fakeRaw = [ordered]@{ status = 'timed_out'; session = [ordered]@{ id = 'sess-fake' } } +# craft a state with a mismatched ledger entry +$badState = [ordered]@{ completed = [ordered]@{ 'arm-999-with_skill' = [ordered]@{ worker_id = 'arm-999-with_skill'; eval_id = 999; configuration = 'with_skill'; status = 'incompatible'; worker_session_id = 'sess-fake' } } } +$threw = $false +try { [void](Assert-FreezeTerminalLedgerEntry -Record $fakeRecord -Raw $fakeRaw -State $badState) } catch { $threw = $true } +Assert-True $threw 'Assert-FreezeTerminalLedgerEntry must reject ledger/raw status mismatch' + +Write-Output 'MIXED-TERMINAL REGRESSION: PASS' \ No newline at end of file diff --git a/scripts/eval-runners/tests/test-opencode-timedout-regression.ps1 b/scripts/eval-runners/tests/test-opencode-timedout-regression.ps1 new file mode 100644 index 0000000..9112b6d --- /dev/null +++ b/scripts/eval-runners/tests/test-opencode-timedout-regression.ps1 @@ -0,0 +1,83 @@ +<# +.SYNOPSIS + Regression: OpenCode timed-out interaction evidence accepted as honest terminal result. +.DESCRIPTION + MODEL-FREE deterministic check using synthetic opencode-style execution evidence. +#> + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'orchestration.ps1') + +function Assert-True { param([bool]$c,[string]$m) if (-not $c) { throw "ASSERT: $m" } } +function Assert-Equal { param($e,$a,$m) if ([string]$e -ne [string]$a) { throw "ASSERT: $m (expected '$e', got '$a')" } } + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-opencode-tmo-' + [Guid]::NewGuid().ToString('N')) +$iteration = Join-Path $testRoot 'iteration-1' +New-Item -ItemType Directory -Path $iteration -Force | Out-Null + +# single eval/arm +$evalName = 'eval-01' +$evalDirectory = Join-Path $iteration $evalName +New-Item -ItemType Directory -Path (Join-Path $evalDirectory 'with_skill') -Force | Out-Null +# run manifest +$runPath = Join-Path $evalDirectory 'with_skill' +$runJson = [ordered]@{ schema = (Get-RunnerSchemaNames).Run; evalId = 1; evalName = $evalName; mode = 'with_skill'; promptFile = 'prompt.md'; workingDirectory = 'repo'; homeDirectory = 'home'; freshContextRequired = $true; filesystemIsolationRequired = $true; isolatedHomeRequired = $true; fixtureHash = ('a' * 64); skillHash = ('b' * 64) } +[System.IO.File]::WriteAllText((Join-Path $runPath 'run.json'), ($runJson | ConvertTo-Json -Depth 100), [System.Text.UTF8Encoding]::new($false)) +New-Item -ItemType Directory -Path (Join-Path $runPath 'repo') -Force | Out-Null +New-Item -ItemType Directory -Path (Join-Path $runPath 'home') -Force | Out-Null +# results and metadata +$resultsDir = Join-Path $evalDirectory 'results' +New-Item -ItemType Directory -Path $resultsDir -Force | Out-Null +[System.IO.File]::WriteAllText((Join-Path $resultsDir 'with_skill.result.json'), (([ordered]@{ eval_id = 1; configuration = 'with_skill'; execution_status = 'unrun'; grading = @() }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) +[System.IO.File]::WriteAllText((Join-Path $evalDirectory 'eval-metadata.json'), (([ordered]@{ eval_id = 1; eval_name = $evalName; assertions = @('assertion') }) | ConvertTo-Json -Depth 10), [System.Text.UTF8Encoding]::new($false)) + +$manifest = [ordered]@{ schema = (Get-RunnerSchemaNames).OrchestrationPlan; configurations = @('with_skill'); execution_freeze = 'execution-freeze.json'; evals = @([ordered]@{ eval_id = 1; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = [ordered]@{ with_skill = [ordered]@{ mode = 'with_skill'; run_manifest = "$evalName/with_skill/run.json"; execution_result = "$evalName/results/with_skill.execution-result.json"; result = "$evalName/results/with_skill.result.json" } } }) } + +$profile = [ordered]@{ schema = (Get-RunnerSchemaNames).Profile; runner = 'opencode'; model = 'opencode/muse-spark-1.2-contributor-free'; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 900; concurrency = 1 } + +$plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile +$state = New-OrchestrationState -Plan $plan + +# accept the single worker +$dispatches = @(Get-NextWorkerDispatches -Plan $plan -State $state) +[void](Register-DelegationAccepted -State $state -WorkerId $dispatches[0].worker_id -WorkerSessionId ('sess-' + $dispatches[0].worker_id)) + +# craft opencode-style timed_out execution evidence +$workerId = $dispatches[0].worker_id +$execEvidence = [ordered]@{ + status = 'timed_out' + session = [ordered]@{ id = ('opencode-session-' + $workerId); fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = 1; eval_name = $evalName; configuration = 'with_skill' } + requested = [ordered]@{ model = $profile.model } + input = [ordered]@{ prompt_sha256 = ('a' * 64) } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = 'opencode-native' + worker_session_id = ('opencode-session-' + $workerId) + observed_model = $profile.model + observed_working_directory = (Join-Path $runPath 'repo') + observed_home = (Join-Path $runPath 'home') + fresh_worker = $true + home_config_isolated = $true + # No terminal_result_capture because timed out before assistant response + terminal_result_capture = $false + # include HTTP timeout metadata + http = [ordered]@{ request_start_utc = (Get-Date).ToUniversalTime().ToString('o'); timeout_utc = (Get-Date).AddSeconds(30).ToUniversalTime().ToString('o'); classification = 'request_timeout' } + terminal_event = [ordered]@{ type = 'timeout'; reason = 'request_timeout' } + } + execution_paths = [ordered]@{ logical_working_directory = (Join-Path $runPath 'repo'); logical_home_directory = (Join-Path $runPath 'home') } + } +} + +# Register terminal; orchestration should preserve raw 'timed_out' and record evidence_validation failed +[void](Register-WorkerTerminal -Plan $plan -State $state -WorkerId $workerId -ExecutionEvidence $execEvidence) +$ledger = $state.completed[$workerId] +Assert-Equal 'timed_out' $ledger.status 'ledger must preserve raw timed_out status' +$ev = Get-JsonProperty -Object $ledger -Name 'evidence_validation' -Default $null +Assert-Equal 'failed' $ev.status 'evidence_validation should be recorded as failed for a timed_out (no terminal capture)' + +Write-Output 'OPENCODE TIMED-OUT REGRESSION: PASS' \ No newline at end of file diff --git a/scripts/eval-runners/tests/test-orchestration.ps1 b/scripts/eval-runners/tests/test-orchestration.ps1 new file mode 100644 index 0000000..9de39a9 --- /dev/null +++ b/scripts/eval-runners/tests/test-orchestration.ps1 @@ -0,0 +1,860 @@ +<# +.SYNOPSIS + Deterministic native-worker orchestration contract tests. + +.DESCRIPTION + Exercises only the manifest queue/state machinery. The fake capacity + harness below never starts a process or contacts a model. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'orchestration.ps1') +. (Join-Path $runnerRoot 'fanout-process.ps1') +. (Join-Path $runnerRoot 'package-integrity.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -ne [string]$Actual) { + throw "ASSERT: $Message (expected '$Expected', got '$Actual')" + } +} + +function Write-TestJson { + param([string]$Path, [object]$Value) + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Invoke-ForegroundPhaseOne { + param([Parameter(Mandatory = $true)][string]$IterationDirectory) + + $fanout = Join-Path $IterationDirectory 'tools/eval-runners/invoke-runner-owned-arms.ps1' + $output = & pwsh -NoProfile -File $fanout -IterationDirectory $IterationDirectory 2>&1 + $exitCode = $LASTEXITCODE + $text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + $document = $text | ConvertFrom-Json -Depth 100 + return [pscustomobject]@{ ExitCode = $exitCode; Text = $text; Document = $document } +} + +function Remove-PhaseOnePackageState { + param([Parameter(Mandatory = $true)][string]$IterationDirectory) + + foreach ($name in @('orchestration-state.json', 'execution-freeze.json')) { + Remove-Item -LiteralPath (Join-Path $IterationDirectory $name) -Force -ErrorAction SilentlyContinue + } +} + +function Copy-TestObject { + param([Parameter(Mandatory = $true)][object]$Value) + + return $Value | ConvertTo-Json -Depth 100 | ConvertFrom-Json +} + +function New-TestNativeTerminalEvidence { + param( + [Parameter(Mandatory = $true)][object]$Arm, + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$WorkerSessionId + ) + + return [ordered]@{ + status = 'completed' + session = [ordered]@{ id = $WorkerSessionId; fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$Arm.eval_id; eval_name = [string]$Arm.eval_name; configuration = [string]$Arm.configuration } + requested = [ordered]@{ model = [string]$Arm.worker.model } + input = [ordered]@{ prompt_sha256 = [string]$RunData.PromptHash } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = 'deterministic-fake-native-worker' + worker_session_id = $WorkerSessionId + observed_model = [string]$Arm.worker.model + observed_working_directory = [string]$RunData.WorkingDirectoryPath + observed_home = [string]$RunData.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$RunData.PromptHash + terminal_result_capture = $true + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + } + } + } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-orchestration-' + [Guid]::NewGuid().ToString('N')) +$oldFixtureLogPath = $env:AGENTIC_RUNNER_FIXTURE_LOG +try { + $iteration = Join-Path $testRoot 'iteration-1' + New-Item -ItemType Directory -Path $iteration -Force | Out-Null + + $manifestEvals = [System.Collections.Generic.List[object]]::new() + for ($evalId = 1; $evalId -le 8; $evalId++) { + $evalName = 'eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $iteration $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ + eval_id = $evalId + eval_name = $evalName + assertions = @('assertion') + }) + + $runs = [ordered]@{} + foreach ($configuration in @('with_skill', 'without_skill')) { + $runDirectory = Join-Path $evalDirectory $configuration + New-Item -ItemType Directory -Path $runDirectory -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $runDirectory 'repo') -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $runDirectory 'home') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "terminal test prompt for $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + if ($configuration -eq 'with_skill') { + New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path (Join-Path $runDirectory 'skill') 'SKILL.md'), '# deterministic terminal test skill', [System.Text.UTF8Encoding]::new($false)) + } + Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $evalId + evalName = $evalName + mode = $configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = if ($configuration -eq 'with_skill') { 'skill' } else { $null } + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + mustNotReadOutsideSandbox = $true + fixtureHash = ('a' * 64) + skillHash = if ($configuration -eq 'with_skill') { ('b' * 64) } else { $null } + }) + $resultFileName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } + Write-TestJson -Path (Join-Path $evalDirectory (Join-Path 'results' $resultFileName)) -Value ([ordered]@{ + eval_id = $evalId + configuration = $configuration + execution_status = 'unrun' + grading = @() + }) + $executionFileName = if ($configuration -eq 'with_skill') { 'with-skill.execution-result.json' } else { 'without-skill.execution-result.json' } + $runs[$configuration] = [ordered]@{ + mode = $configuration + run_manifest = "$evalName/$configuration/run.json" + execution_result = "$evalName/results/$executionFileName" + result = "$evalName/results/$resultFileName" + } + } + $manifestEvals.Add([ordered]@{ + eval_id = $evalId + eval_name = $evalName + directory = $evalName + metadata = "$evalName/eval-metadata.json" + runs = $runs + }) + } + + $manifest = [ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + configurations = @('with_skill', 'without_skill') + execution_freeze = 'execution-freeze.json' + evals = @($manifestEvals) + } + $profile = [ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fake' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 60 + concurrency = 16 + } + + $plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile + [void](Assert-OrchestrationPlanContract -Plan $plan) + $serializedPlan = $plan | ConvertTo-Json -Depth 100 | ConvertFrom-Json + [void](Assert-OrchestrationPlanContract -Plan $serializedPlan) + Assert-Equal 16 @($plan.arms).Count '8 eval cases fan out to 16 independent arms' + Assert-Equal 16 $plan.requested_concurrency 'requested concurrency is preserved in the plan' + Assert-True ([bool]$plan.parallel_dispatch_required) 'independent arms require concurrent dispatch' + Assert-Equal 2 $plan.minimum_parallel_workers 'independent arms require at least two active workers' + Assert-True ([bool]$plan.native_worker_required) 'native worker delegation is mandatory' + Assert-True (-not [bool]$plan.parent_executes_arms) 'the parent is forbidden from executing arms' + Assert-True (-not [bool]$plan.nested_model_execution) 'the plan forbids a nested model layer' + Assert-Equal 16 (@($plan.arms | ForEach-Object { [string]$_.worker_id } | Sort-Object -Unique).Count) 'every arm has a distinct worker identity' + Assert-Equal 16 (@($plan.arms | Where-Object { @($_.depends_on).Count -eq 0 }).Count) 'unrelated arms have no sequential dependencies' + + $initialState = New-OrchestrationState -Plan $plan + $initialDispatches = @(Get-NextWorkerDispatches -Plan $plan -State $initialState) + Assert-Equal 16 $initialDispatches.Count 'requested concurrency exposes all 16 pending arms' + foreach ($dispatch in $initialDispatches) { + Assert-True ([bool]$dispatch.worker_contract.one_arm_only) "$($dispatch.worker_id) receives one arm" + Assert-True (-not [bool]$dispatch.worker_contract.paired_arm_visible) "$($dispatch.worker_id) cannot see its paired arm" + Assert-True (-not [bool]$dispatch.worker_contract.grading_material_visible) "$($dispatch.worker_id) cannot see grading material" + Assert-True (-not [bool]$dispatch.worker_contract.parent_executes_arm) "$($dispatch.worker_id) cannot execute in the parent" + Assert-Equal 'forbidden' $dispatch.worker_contract.runner_execute_invocation "$($dispatch.worker_id) cannot invoke direct runner execute" + Assert-True (-not [bool]$dispatch.worker_contract.nested_model_execution) "$($dispatch.worker_id) has no nested model layer" + Assert-Equal 1 $dispatch.worker_contract.model_execution_count "$($dispatch.worker_id) has one model execution" + Assert-True ($dispatch.PSObject.Properties.Name -notcontains 'paired_arm') "$($dispatch.worker_id) has no paired-arm payload" + Assert-True ($dispatch.PSObject.Properties.Name -notcontains 'grading') "$($dispatch.worker_id) has no grading payload" + Assert-True ($dispatch.PSObject.Properties.Name -notcontains 'expected_output') "$($dispatch.worker_id) has no expected-output payload" + } + + # Codex uses runner-owned native dispatch. The runner process/thread is the + # worker, so the portable queue must not ask an outer model orchestrator to + # create a subagent first. Six recorded arms are advanced concurrently; + # this test never starts a process or model. + $runnerDescriptor = [pscustomobject]@{ + name = 'codex' + delegation = [ordered]@{ + dispatch_owner = 'runner' + mechanism = 'deterministic-fake-native-worker' + } + } + $runnerProfile = [ordered]@{ + runner = 'codex' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 60 + concurrency = 3 + } + $runnerPlan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $runnerProfile -Descriptor $runnerDescriptor + [void](Assert-OrchestrationPlanContract -Plan $runnerPlan) + Assert-Equal 'runner' $runnerPlan.dispatch_owner 'runner-owned descriptor selects runner dispatch' + $runnerState = New-OrchestrationState -Plan $runnerPlan + $outerSubagentCalls = 0 + $runnerWorkerStarts = 0 + $runnerDispatches = @(Get-NextWorkerDispatches -Plan $runnerPlan -State $runnerState) + Assert-Equal 3 $runnerDispatches.Count 'runner-owned dispatch respects requested concurrency' + foreach ($dispatch in $runnerDispatches) { + Assert-Equal 'runner' $dispatch.worker_contract.dispatch_owner "$($dispatch.worker_id) is runner-owned" + Assert-Equal 'required' $dispatch.worker_contract.runner_execute_invocation "$($dispatch.worker_id) uses the runner-owned execution surface" + $runnerWorkerStarts++ + [void](Register-DelegationAccepted -State $runnerState -WorkerId $dispatch.worker_id) + } + Assert-Equal 3 (Get-OrchestrationActiveCount -State $runnerState) 'runner-owned workers are active concurrently' + Assert-True ($runnerState.max_observed_active -gt 1) 'runner-owned dispatch observes parallel active workers' + foreach ($workerId in @($runnerState.active.Keys)) { + $arm = Get-OrchestrationArmByWorkerId -Plan $runnerPlan -WorkerId ([string]$workerId) + $runData = Resolve-RunContract -RunPath ([string]$arm.worker.run_manifest_path) + [void](Register-WorkerTerminal -Plan $runnerPlan -State $runnerState -WorkerId ([string]$workerId) -ExecutionEvidence (New-TestNativeTerminalEvidence -Arm $arm -RunData $runData -WorkerSessionId ('runner-session-' + $workerId))) + } + $runnerDispatches = @(Get-NextWorkerDispatches -Plan $runnerPlan -State $runnerState) + Assert-Equal 3 $runnerDispatches.Count 'runner-owned queue dispatches the next concurrent batch' + foreach ($dispatch in $runnerDispatches) { + $runnerWorkerStarts++ + [void](Register-DelegationAccepted -State $runnerState -WorkerId $dispatch.worker_id) + } + foreach ($workerId in @($runnerState.active.Keys)) { + $arm = Get-OrchestrationArmByWorkerId -Plan $runnerPlan -WorkerId ([string]$workerId) + $runData = Resolve-RunContract -RunPath ([string]$arm.worker.run_manifest_path) + [void](Register-WorkerTerminal -Plan $runnerPlan -State $runnerState -WorkerId ([string]$workerId) -ExecutionEvidence (New-TestNativeTerminalEvidence -Arm $arm -RunData $runData -WorkerSessionId ('runner-session-' + $workerId))) + } + Assert-Equal 6 @($runnerState.completed.Keys).Count 'runner-owned test completes six independent arms' + Assert-Equal 6 $runnerWorkerStarts 'runner-owned dispatch starts one runner surface per arm' + Assert-Equal 0 $outerSubagentCalls 'runner-owned dispatch never requests an outer model subagent' + Assert-Equal 3 $runnerState.max_observed_active 'runner-owned state records concurrent maximum' + + # A fake harness accepts only four simultaneous native workers. This limit + # belongs to the fake harness, not to the portable plan or queue. + $capacityState = New-OrchestrationState -Plan $plan + $startedWorkers = [System.Collections.Generic.List[string]]::new() + $rejectedWorkers = [System.Collections.Generic.List[string]]::new() + $attemptsAtRejection = @{} + $maxObservedByHarness = 0 + $firstDispatches = @(Get-NextWorkerDispatches -Plan $plan -State $capacityState) + foreach ($dispatch in $firstDispatches) { + if ((Get-OrchestrationActiveCount -State $capacityState) -lt 4) { + [void](Register-DelegationAccepted -State $capacityState -WorkerId $dispatch.worker_id -WorkerSessionId ('session-' + $dispatch.worker_id)) + $startedWorkers.Add([string]$dispatch.worker_id) + } else { + [void](Register-DelegationRejected -State $capacityState -WorkerId $dispatch.worker_id -Reason 'fake harness capacity is four' -CapacityLimited) + $rejectedWorkers.Add([string]$dispatch.worker_id) + $attemptsAtRejection[[string]$dispatch.worker_id] = $capacityState.eval_attempts.Contains([string]$dispatch.worker_id) + } + $maxObservedByHarness = [Math]::Max($maxObservedByHarness, (Get-OrchestrationActiveCount -State $capacityState)) + } + Assert-Equal 4 $startedWorkers.Count 'the fake harness starts four workers before rejecting capacity overflow' + Assert-True ($rejectedWorkers.Count -gt 0) 'capacity overflow creates queued rejections' + Assert-True (@($attemptsAtRejection.Values | Where-Object { $_ }).Count -eq 0) 'a rejected delegation is not an eval attempt' + Assert-Equal 16 $capacityState.requested_concurrency 'capacity handling does not lower portable requested concurrency' + + while (@($capacityState.completed.Keys).Count -lt @($plan.arms).Count) { + $dispatches = @(Get-NextWorkerDispatches -Plan $plan -State $capacityState) + foreach ($dispatch in $dispatches) { + if ((Get-OrchestrationActiveCount -State $capacityState) -lt 4) { + [void](Register-DelegationAccepted -State $capacityState -WorkerId $dispatch.worker_id -WorkerSessionId ('session-' + $dispatch.worker_id)) + if ($startedWorkers -notcontains [string]$dispatch.worker_id) { $startedWorkers.Add([string]$dispatch.worker_id) } + } else { + [void](Register-DelegationRejected -State $capacityState -WorkerId $dispatch.worker_id -Reason 'fake harness capacity is four' -CapacityLimited) + } + $maxObservedByHarness = [Math]::Max($maxObservedByHarness, (Get-OrchestrationActiveCount -State $capacityState)) + } + + $activeIds = @($capacityState.active.Keys) + if ($activeIds.Count -gt 0) { + $workerId = [string]$activeIds[0] + $arm = Get-OrchestrationArmByWorkerId -Plan $plan -WorkerId $workerId + $runData = Resolve-RunContract -RunPath ([string]$arm.worker.run_manifest_path) + $workerSessionId = [string]$capacityState.active[$workerId].worker_session_id + [void](Register-WorkerTerminal -Plan $plan -State $capacityState -WorkerId $workerId -ExecutionEvidence ([ordered]@{ + status = 'completed' + session = [ordered]@{ id = $workerSessionId; fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$arm.eval_id; eval_name = [string]$arm.eval_name; configuration = [string]$arm.configuration } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = 'deterministic-fake-native-worker' + worker_session_id = $workerSessionId + observed_model = [string]$arm.worker.model + observed_working_directory = [string]$runData.WorkingDirectoryPath + observed_home = [string]$runData.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$runData.PromptHash + terminal_result_capture = $true + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + } + } + })) + } elseif (@($capacityState.pending_worker_ids).Count -gt 0) { + throw 'capacity queue deadlocked with pending workers and no active worker.' + } + } + Assert-Equal 16 @($capacityState.completed.Keys).Count 'all 16 arms eventually become terminal' + Assert-Equal 16 $startedWorkers.Count 'all 16 arms start exactly once after capacity is released' + Assert-Equal 4 $capacityState.max_observed_active 'state records the fake harness maximum of four active workers' + Assert-Equal 4 $maxObservedByHarness 'the fake harness never exceeds four active workers' + Assert-Equal 0 @($capacityState.pending_worker_ids).Count 'no arm remains queued after capacity is released' + [void](Assert-OrchestrationConcurrency -Plan $plan -State $capacityState) + + $serialState = New-OrchestrationState -Plan $plan + $serialState.max_observed_active = 1 + $serialState.pending_worker_ids = @() + $serialState.active = [ordered]@{} + $serialState.completed = [ordered]@{} + $serialRejected = $false + try { [void](Assert-OrchestrationConcurrency -Plan $plan -State $serialState) } catch { $serialRejected = $true } + Assert-True $serialRejected 'serial dispatch without capacity evidence fails the concurrency gate' + + $capacityStateJson = $capacityState | ConvertTo-Json -Depth 100 | ConvertFrom-Json + [void](Assert-OrchestrationConcurrency -Plan $plan -State $capacityStateJson) + Assert-True ([bool]$capacityStateJson.capacity_limit_reported) 'serialized fallback is permitted only with persisted capacity evidence' + + $badDescriptor = [pscustomobject]@{ + name = 'fake-without-delegation' + delegation = [ordered]@{ mode = 'conditional'; nested_model_execution = $false } + } + $badPreflight = [ordered]@{ + status = 'compatible' + delegation = [ordered]@{ status = 'conditional'; unproven_controls = @('native_worker_delegation') } + resolved_capabilities = [ordered]@{} + } + $parentDispatches = 0 + $fallbackRejected = $false + try { + [void](Assert-NativeWorkerDelegation -Descriptor $badDescriptor -Preflight $badPreflight) + $parentDispatches++ + } catch { + $fallbackRejected = $true + } + Assert-True $fallbackRejected 'missing native delegation fails preflight' + Assert-Equal 0 $parentDispatches 'failed delegation preflight never invokes parent fallback' + + # A native mechanism may be locally ready while worker-specific controls + # remain conditional. The gate must allow that handoff only when it will + # validate terminal evidence; it must never reuse compatibility execute. + $conditionalDescriptor = [pscustomobject]@{ + name = 'conditional-native' + delegation = [ordered]@{ dispatch_owner = 'orchestrator'; mode = 'native_worker'; nested_model_execution = $false } + } + $conditionalCapabilities = [ordered]@{ + native_worker_delegation = 'conditional' + delegated_worker_full_capability = 'conditional' + delegated_worker_model_lock = 'conditional' + delegated_worker_working_directory = 'conditional' + delegated_worker_result_capture = 'conditional' + delegated_worker_capacity_signal = 'supported' + } + $conditionalPreflight = [ordered]@{ + status = 'compatible' + delegation = [ordered]@{ status = 'conditional'; unproven_controls = @('delegated_worker_model_lock'); terminal_evidence_required = $true } + resolved_capabilities = $conditionalCapabilities + } + Assert-True (Assert-NativeWorkerDelegation -Descriptor $conditionalDescriptor -Preflight $conditionalPreflight) 'conditional native preflight is accepted only for terminal validation' + $conditionalWithoutTerminalEvidence = [ordered]@{ + status = 'compatible' + delegation = [ordered]@{ status = 'conditional'; unproven_controls = @('delegated_worker_model_lock') } + resolved_capabilities = $conditionalCapabilities + } + $conditionalWithoutTerminalRejected = $false + try { [void](Assert-NativeWorkerDelegation -Descriptor $conditionalDescriptor -Preflight $conditionalWithoutTerminalEvidence) } catch { $conditionalWithoutTerminalRejected = $true } + Assert-True $conditionalWithoutTerminalRejected 'conditional native preflight without a terminal-evidence requirement is rejected' + + $terminalArm = $plan.arms[0] + $terminalRunData = Resolve-RunContract -RunPath ([string]$terminalArm.worker.run_manifest_path) + $validTerminalEvidence = New-TestNativeTerminalEvidence -Arm $terminalArm -RunData $terminalRunData -WorkerSessionId 'native-terminal-session' + Assert-True ((Test-NativeWorkerTerminalEvidence -ExecutionEvidence $validTerminalEvidence -Run $terminalRunData -RequestedModel ([string]$terminalArm.worker.model) -ExpectedWorkerSessionId 'native-terminal-session').Valid) 'valid terminal native-worker evidence is accepted' + Assert-True (Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $validTerminalEvidence -Run $terminalRunData -RequestedModel ([string]$terminalArm.worker.model) -ExpectedWorkerSessionId 'native-terminal-session') 'valid terminal evidence passes the assert gate' + + # Runner-specific checks and the portable validator must make one terminal + # decision. Preserve the exact runner codes while the common validator + # rechecks the portable evidence. + $codexFailureCodes = @('instruction_sources_unobserved', 'observed_model', 'observed_working_directory', 'fresh_worker', 'prompt_fidelity', 'terminal_result_capture', 'terminal_turn_status') + $runnerIncompatible = Copy-TestObject -Value $validTerminalEvidence + $runnerIncompatible.status = 'incompatible' + Add-Member -InputObject $runnerIncompatible.evidence -MemberType NoteProperty -Name native_worker_evidence_failures -Value $codexFailureCodes -Force + $runnerValidation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $runnerIncompatible -Run $terminalRunData -RequestedModel ([string]$terminalArm.worker.model) -ExpectedWorkerSessionId 'native-terminal-session' + Assert-True (-not [bool]$runnerValidation.Valid) 'common terminal validation rejects runner-reported incompatibility' + Assert-True (@($runnerValidation.Failures | Where-Object { $_ -eq 'terminal_turn_status' }).Count -eq 1) 'common validation preserves an exact runner-specific failure code' + $runnerState = New-OrchestrationState -Plan ([pscustomobject]@{ schema = $plan.schema; requested_concurrency = 1; arms = @($terminalArm) }) + [void](Register-DelegationAccepted -State $runnerState -WorkerId ([string]$terminalArm.worker_id) -WorkerSessionId 'native-terminal-session') + [void](Register-WorkerTerminal -Plan ([pscustomobject]@{ arms = @($terminalArm) }) -State $runnerState -WorkerId ([string]$terminalArm.worker_id) -ExecutionEvidence $runnerIncompatible) + Assert-Equal 'incompatible' $runnerState.completed[[string]$terminalArm.worker_id].native_worker_evidence 'runner-specific incompatibility cannot become common verified evidence' + foreach ($failureCode in $codexFailureCodes) { + Assert-True (@($runnerState.completed[[string]$terminalArm.worker_id].native_worker_evidence_failures | Where-Object { $_ -eq $failureCode }).Count -eq 1) "exact runner failure '$failureCode' survives orchestration state" + } + $optionalReadEvidence = Copy-TestObject -Value $validTerminalEvidence + Add-Member -InputObject $optionalReadEvidence.evidence -MemberType NoteProperty -Name app_server -Value ([ordered]@{ thread_read = [ordered]@{ observation = 'unavailable_optional' } }) -Force + Assert-True ((Test-NativeWorkerTerminalEvidence -ExecutionEvidence $optionalReadEvidence -Run $terminalRunData -RequestedModel ([string]$terminalArm.worker.model) -ExpectedWorkerSessionId 'native-terminal-session').Valid) 'optional thread/read absence does not invalidate mandatory common evidence' + + $exactOnceState = New-OrchestrationState -Plan ([pscustomobject]@{ + schema = $plan.schema + requested_concurrency = 1 + arms = @($terminalArm) + }) + [void](Register-DelegationAccepted -State $exactOnceState -WorkerId ([string]$terminalArm.worker_id) -WorkerSessionId 'native-terminal-session') + $duplicateAcceptanceRejected = $false + try { + [void](Register-DelegationAccepted -State $exactOnceState -WorkerId ([string]$terminalArm.worker_id) -WorkerSessionId 'duplicate-session') + } catch { + $duplicateAcceptanceRejected = $_.Exception.Message -match 'exactly-once' + } + Assert-True $duplicateAcceptanceRejected 'duplicate worker acceptance is rejected with an exactly-once diagnostic' + Assert-Equal 1 $exactOnceState.active[[string]$terminalArm.worker_id].attempt_count 'duplicate acceptance does not create another attempt' + [void](Register-WorkerTerminal -Plan ([pscustomobject]@{ arms = @($terminalArm) }) -State $exactOnceState -WorkerId ([string]$terminalArm.worker_id) -ExecutionEvidence (Copy-TestObject -Value $validTerminalEvidence)) + $duplicateTerminalRejected = $false + try { + [void](Register-WorkerTerminal -Plan ([pscustomobject]@{ arms = @($terminalArm) }) -State $exactOnceState -WorkerId ([string]$terminalArm.worker_id) -ExecutionEvidence (Copy-TestObject -Value $validTerminalEvidence)) + } catch { + $duplicateTerminalRejected = $_.Exception.Message -match 'already terminal|exactly-once' + } + Assert-True $duplicateTerminalRejected 'duplicate terminal registration is rejected after the worker is terminal' + Assert-Equal 1 @($exactOnceState.completed.Keys).Count 'duplicate terminal registration does not add another completion' + + function Invoke-TerminalEvidenceCase { + param( + [Parameter(Mandatory = $true)][string]$Name, + [Parameter(Mandatory = $true)][object]$Evidence, + [Parameter(Mandatory = $true)][string]$ExpectedFailure + ) + + $caseState = New-OrchestrationState -Plan ([pscustomobject]@{ + schema = $plan.schema + requested_concurrency = 1 + arms = @($terminalArm) + }) + [void](Register-DelegationAccepted -State $caseState -WorkerId ([string]$terminalArm.worker_id) -WorkerSessionId 'native-terminal-session') + [void](Register-WorkerTerminal -Plan ([pscustomobject]@{ arms = @($terminalArm) }) -State $caseState -WorkerId ([string]$terminalArm.worker_id) -ExecutionEvidence $Evidence) + # Ledger must preserve the raw runner status and record evidence validation + Assert-Equal $Evidence.status $caseState.completed[[string]$terminalArm.worker_id].status "$Name ledger.status must equal raw Evidence.status" + $ev = Get-JsonProperty -Object $caseState.completed[[string]$terminalArm.worker_id] -Name 'evidence_validation' -Default $null + Assert-Equal 'failed' $ev.status "$Name evidence_validation should be failed" + Assert-True (([string]::Join(',', @($caseState.completed[[string]$terminalArm.worker_id].native_worker_evidence_failures))) -match [regex]::Escape($ExpectedFailure)) "$Name records $ExpectedFailure" + } + + $modelMismatch = Copy-TestObject -Value $validTerminalEvidence + $modelMismatch.evidence.delegation.observed_model = 'different-model' + Invoke-TerminalEvidenceCase -Name 'model mismatch' -Evidence $modelMismatch -ExpectedFailure 'requested_model' + + $workingDirectoryMismatch = Copy-TestObject -Value $validTerminalEvidence + $workingDirectoryMismatch.evidence.delegation.observed_working_directory = (Join-Path $terminalRunData.RunRoot 'other-repo') + Invoke-TerminalEvidenceCase -Name 'working-directory mismatch' -Evidence $workingDirectoryMismatch -ExpectedFailure 'working_directory' + + $missingHomeProof = Copy-TestObject -Value $validTerminalEvidence + $missingHomeProof.evidence.delegation.home_config_isolated = $false + $missingHomeProof.evidence.delegation.observed_home = '' + Invoke-TerminalEvidenceCase -Name 'missing HOME/config proof' -Evidence $missingHomeProof -ExpectedFailure 'isolated_home_config' + + $missingDelegationEvidence = Copy-TestObject -Value $validTerminalEvidence + $missingDelegationEvidence.evidence = $null + Invoke-TerminalEvidenceCase -Name 'missing terminal delegation evidence' -Evidence $missingDelegationEvidence -ExpectedFailure 'delegation_terminal_evidence' + + $armMismatch = Copy-TestObject -Value $validTerminalEvidence + $armMismatch.run.eval_id = 999 + Invoke-TerminalEvidenceCase -Name 'arm identity mismatch' -Evidence $armMismatch -ExpectedFailure 'arm_identity' + + $nestedExecution = Copy-TestObject -Value $validTerminalEvidence + $nestedExecution.evidence.delegation.nested_model_execution = $true + $nestedExecution.evidence.delegation.model_execution_count = 2 + Invoke-TerminalEvidenceCase -Name 'nested model execution' -Evidence $nestedExecution -ExpectedFailure 'nested_model_execution' + + $freshWorkerMismatch = Copy-TestObject -Value $validTerminalEvidence + $freshWorkerMismatch.session.id = 'different-worker-session' + $freshWorkerMismatch.evidence.delegation.worker_session_id = 'different-worker-session' + Invoke-TerminalEvidenceCase -Name 'fresh worker/session mismatch' -Evidence $freshWorkerMismatch -ExpectedFailure 'worker_session_id' + + $duplicateSessionState = New-OrchestrationState -Plan ([pscustomobject]@{ + schema = $plan.schema + requested_concurrency = 1 + arms = @($terminalArm) + }) + $duplicateSessionState.completed['prior-worker'] = [ordered]@{ worker_id = 'prior-worker'; worker_session_id = 'native-terminal-session' } + [void](Register-DelegationAccepted -State $duplicateSessionState -WorkerId ([string]$terminalArm.worker_id) -WorkerSessionId 'native-terminal-session') + [void](Register-WorkerTerminal -Plan ([pscustomobject]@{ arms = @($terminalArm) }) -State $duplicateSessionState -WorkerId ([string]$terminalArm.worker_id) -ExecutionEvidence (Copy-TestObject -Value $validTerminalEvidence)) + # Ledger must preserve the raw runner status even when session reuse is suspicious + Assert-Equal $validTerminalEvidence.status $duplicateSessionState.completed[[string]$terminalArm.worker_id].status 'reused worker session ledger preserves raw status' + $dupEv = Get-JsonProperty -Object $duplicateSessionState.completed[[string]$terminalArm.worker_id] -Name 'evidence_validation' -Default $null + Assert-Equal 'failed' $dupEv.status 'reused worker session records failed evidence_validation' + Assert-True (([string]::Join(',', @($duplicateSessionState.completed[[string]$terminalArm.worker_id].native_worker_evidence_failures))) -match 'fresh_worker') 'reused worker session records fresh-worker failure' + + $fanoutPath = Join-Path $runnerRoot 'invoke-runner-owned-arms.ps1' + $fanoutText = [System.IO.File]::ReadAllText($fanoutPath, [System.Text.UTF8Encoding]::new($false)) + $fanoutProcessText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'fanout-process.ps1'), [System.Text.UTF8Encoding]::new($false)) + foreach ($needle in @('manifest.json', 'New-EvalOrchestrationPlan', 'dispatch_owner', 'Get-OrchestrationArmByWorkerId', 'arm.parent_paths.execution_result', 'Start-RunnerChildProcess', 'Wait-AnyRunnerChild', 'Register-DelegationAccepted', 'Register-DelegationSession', 'Register-WorkerTerminal', 'Assert-OrchestrationConcurrency', 'orchestration-state.json')) { + Assert-True $fanoutText.Contains($needle) "runner-owned helper contains deterministic '$needle' behavior" + } + Assert-True ($fanoutText.Contains('runner child watchdog timed out') -and $fanoutText.Contains('no retry or redispatch was performed')) 'runner-owned helper reports the exact timed-out worker and forbids retry or redispatch' + Assert-True ($fanoutProcessText.Contains('DeadlineUtc') -and $fanoutProcessText.Contains('Kill($true)') -and $fanoutProcessText.Contains('WaitForExit(5000)')) 'runner-owned process helper has a finite child deadline, tree kill, and kill grace' + Assert-True (-not $fanoutText.Contains('Start-Process')) 'runner-owned helper no longer uses window-creating Start-Process for child dispatch' + Assert-True ($fanoutText -notmatch '(?i)spawn[_ -]?agent|subagent|native-worker-result|with_skill.*worker_id|without_skill.*worker_id') 'runner-owned helper does not create outer subagents, synthetic envelopes, or derived worker IDs' + Assert-True ($fanoutText -notmatch '(?i)(with[-_]skill|without[-_]skill)\.execution-result|execution_result\s*=\s*.*configuration') 'runner-owned helper does not reconstruct result filenames' + Assert-True ($fanoutText.Contains('[Parameter(Mandatory = $true)][string]$IterationDirectory') -and -not $fanoutText.Contains('Assert-RunnerOwnedFanoutAuthorization') -and -not $fanoutText.Contains('SupervisorId') -and -not $fanoutText.Contains('phase1-control-common.ps1')) 'runner-owned fan-out is the foreground Phase 1 surface without durable supervisor authorization' + + # Exercise the helper end-to-end with a deterministic runner-owned fixture. + # The fixture is a protocol adapter only; it never calls a model or an AI + # CLI. Six short processes must be active through the helper's first batch. + $fanoutPackage = Join-Path $testRoot 'runner-owned fanout package' + $fanoutTools = Join-Path $fanoutPackage 'tools\eval-runners' + New-Item -ItemType Directory -Path $fanoutTools -Force | Out-Null + foreach ($toolItem in @(Get-ChildItem -LiteralPath $runnerRoot -Force | Where-Object { $_.Name -ne 'tests' })) { + Copy-Item -LiteralPath $toolItem.FullName -Destination $fanoutTools -Recurse -Force + } + $fixtureRunnerDirectory = Join-Path $fanoutTools 'fixture' + New-Item -ItemType Directory -Path $fixtureRunnerDirectory -Force | Out-Null + Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests\fixtures\runner-owned-fixture.ps1') -Destination (Join-Path $fixtureRunnerDirectory 'runner.ps1') -Force + $fanoutProfileRelative = 'execution-profile.json' + Write-TestJson -Path (Join-Path $fanoutPackage $fanoutProfileRelative) -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fixture' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 16 + }) + $fanoutManifestEvals = [System.Collections.Generic.List[object]]::new() + for ($evalId = 1; $evalId -le 3; $evalId++) { + $evalName = 'fanout-eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $fanoutPackage $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('fixture') }) + $runs = [ordered]@{} + foreach ($configuration in @('with_skill', 'without_skill')) { + $runDirectory = Join-Path $evalDirectory $configuration + $repoDirectory = Join-Path $runDirectory 'repo' + $homeDirectory = Join-Path $runDirectory 'home' + New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $repoDirectory 'input.txt'), "$evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "fixture prompt $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + $skillDirectory = $null + $skillHash = $null + if ($configuration -eq 'with_skill') { + $skillDirectory = 'skill/candidate' + New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill\candidate') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'skill\candidate\SKILL.md'), '# fixture', [System.Text.UTF8Encoding]::new($false)) + $skillHash = ('b' * 64) + } + Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $evalId + evalName = $evalName + skillName = if ($configuration -eq 'with_skill') { 'candidate' } else { $null } + iteration = 1 + mode = $configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $skillDirectory + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + fixtureHash = ('a' * 64) + skillHash = $skillHash + }) + $resultDirectory = Join-Path $runDirectory 'results' + New-Item -ItemType Directory -Path $resultDirectory -Force | Out-Null + $runs[$configuration] = [ordered]@{ + run_manifest = "$evalName/$configuration/run.json" + execution_result = "$evalName/$configuration/results/execution-result.json" + result = "$evalName/$configuration/results/result.json" + mode = $configuration + } + Write-TestJson -Path (Join-Path $resultDirectory 'result.json') -Value ([ordered]@{ eval_id = $evalId; configuration = $configuration }) + } + $fanoutManifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) + } + $fanoutToolIntegrity = Get-PackageTreeIntegrity -Root $fanoutTools + Write-TestJson -Path (Join-Path $fanoutPackage 'manifest.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + configurations = @('with_skill', 'without_skill') + execution_profile = $fanoutProfileRelative + runner_tools = 'tools/eval-runners' + runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $fanoutToolIntegrity.Sha256; file_count = $fanoutToolIntegrity.FileCount } + execution_freeze = 'execution-freeze.json' + evals = $fanoutManifestEvals.ToArray() + }) + $fixtureLogPath = Join-Path $testRoot 'runner-owned-events.jsonl' + $env:AGENTIC_RUNNER_FIXTURE_LOG = $fixtureLogPath + $fanoutControl = Invoke-ForegroundPhaseOne -IterationDirectory $fanoutPackage + Assert-Equal 0 $fanoutControl.ExitCode ("deterministic runner-owned Phase 1 exits successfully; output: " + $fanoutControl.Text) + $fanoutSummary = $fanoutControl.Document + Assert-Equal 'phase1' $fanoutSummary.phase 'deterministic runner-owned fan-out reports a Phase 1 summary' + Assert-Equal 'completed' $fanoutSummary.status 'deterministic runner-owned fan-out completes six fixture arms' + Assert-Equal 6 $fanoutSummary.expected_count 'runner-owned helper preserves the six-arm manifest count' + Assert-Equal 6 $fanoutSummary.terminal_count 'runner-owned helper reports all six arms as terminal' + Assert-Equal 6 $fanoutSummary.preflight_count 'runner-owned helper preflights all six manifest arms before fan-out' + Assert-True $fanoutSummary.execution_started 'runner-owned helper records that execution started after preflight' + Assert-Equal 6 $fanoutSummary.execution_count 'runner-owned helper executes exactly six compatible arms' + Assert-Equal 6 $fanoutSummary.completed_count 'runner-owned helper counts only completed arms' + Assert-Equal 0 $fanoutSummary.failed_count 'runner-owned helper reports zero failed arms in the success fixture' + Assert-Equal 0 $fanoutSummary.timed_out_count 'runner-owned helper reports zero timed out arms in the success fixture' + Assert-Equal 0 $fanoutSummary.cancelled_count 'runner-owned helper reports zero cancelled arms in the success fixture' + Assert-Equal 0 $fanoutSummary.incompatible_count 'runner-owned helper reports zero incompatible arms in the success fixture' + Assert-Equal 0 $fanoutSummary.evidence_validation_failed_count 'runner-owned helper reports zero evidence-validation failures in the success fixture' + Assert-True ([int]$fanoutSummary.max_observed_active -gt 1) 'runner-owned helper reaches parallel active execution' + $fanoutEvents = @(Get-Content -LiteralPath $fixtureLogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 12 $fanoutEvents.Count 'runner-owned fixture records six preflight and six execute events' + Assert-Equal 6 @($fanoutEvents | Where-Object { $_.kind -eq 'preflight' }).Count 'all six preflight calls complete' + Assert-Equal 6 @($fanoutEvents | Where-Object { $_.kind -eq 'execute' }).Count 'all six execute calls start after compatible preflight' + $firstExecuteIndex = -1 + $lastPreflightIndex = -1 + for ($eventIndex = 0; $eventIndex -lt $fanoutEvents.Count; $eventIndex++) { + if ($fanoutEvents[$eventIndex].kind -eq 'preflight') { $lastPreflightIndex = $eventIndex } + if ($fanoutEvents[$eventIndex].kind -eq 'execute' -and $firstExecuteIndex -lt 0) { $firstExecuteIndex = $eventIndex } + } + Assert-True ($lastPreflightIndex -ge 0 -and $firstExecuteIndex -gt $lastPreflightIndex) 'all runner-owned preflights occur before the first execute process' + $fanoutState = Read-RunnerJson -Path (Join-Path $fanoutPackage 'orchestration-state.json') + Assert-Equal 6 $fanoutState.preflight.count 'orchestration state preserves all preflight worker records' + Assert-Equal 'passed' $fanoutState.preflight.status 'orchestration state records the passed preflight gate' + Assert-Equal 'verified' ([string]$fanoutSummary.concurrency.status) 'runner-owned helper persists verified concurrency state' + Assert-Equal 16 ([int]$fanoutSummary.concurrency.requested_concurrency) 'runner-owned helper preserves requested concurrency 16' + Assert-Equal 'passed' ([string]$fanoutSummary.arms[0].evidence_validation.status) 'runner-owned helper surfaces arm-level evidence validation state' + Assert-Equal 'with_skill' ([string]$fanoutSummary.arms[0].configuration) 'runner-owned helper preserves arm-level configuration' + Assert-True (-not [string]::IsNullOrWhiteSpace([string]$fanoutSummary.arms[0].worker_session_id)) 'runner-owned helper preserves arm-level worker session ids' + $fanoutWorkerIds = @($fanoutState.completed.PSObject.Properties.Name | Sort-Object) + Assert-Equal 'arm-1-with_skill,arm-1-without_skill,arm-2-with_skill,arm-2-without_skill,arm-3-with_skill,arm-3-without_skill' ([string]::Join(',', $fanoutWorkerIds)) 'runner-owned helper preserves exact manifest plan worker IDs' + foreach ($record in @(Get-ManifestRunRecords -IterationDirectory $fanoutPackage -Manifest (Read-RunnerJson -Path (Join-Path $fanoutPackage 'manifest.json')))) { + $result = Read-RunnerJson -Path $record.ExecutionResultPath + $workerId = "arm-$($record.EvalId)-$($record.Configuration)" + $completedEntry = $fanoutState.completed.PSObject.Properties[$workerId].Value + Assert-Equal ([string]$result.session.id) ([string]$completedEntry.worker_session_id) "runner-owned helper derives $workerId session identity from its result" + } + + # The preflight gate must fail closed: one incompatible arm is reported + # with its manifest identity and no runner-owned execute process starts. + $gatePackage = Join-Path $testRoot 'runner-owned preflight gate package' + Copy-Item -LiteralPath $fanoutPackage -Destination $gatePackage -Recurse -Force + foreach ($executionResultFile in @(Get-ChildItem -LiteralPath $gatePackage -Recurse -File -Filter 'execution-result.json')) { + Remove-Item -LiteralPath $executionResultFile.FullName -Force + } + Remove-PhaseOnePackageState -IterationDirectory $gatePackage + $gateInteraction = [ordered]@{ + schema = (Get-RunnerSchemaNames).Interaction + mode = 'scripted' + turns = @( + [ordered]@{ role = 'user'; content = 'fixture confirmation request' } + [ordered]@{ role = 'user'; content = 'fixture protected operation request' } + ) + } + foreach ($gateRunFile in @(Get-ChildItem -LiteralPath $gatePackage -Recurse -File -Filter 'run.json')) { + $gateInteractionPath = Join-Path $gateRunFile.DirectoryName 'interaction.json' + Write-TestJson -Path $gateInteractionPath -Value $gateInteraction + $gateRun = Read-RunnerJson -Path $gateRunFile.FullName + Add-Member -InputObject $gateRun -MemberType NoteProperty -Name interactionFile -Value 'interaction.json' -Force + Add-Member -InputObject $gateRun -MemberType NoteProperty -Name interactionHash -Value (Get-Sha256HexFromFile -Path $gateInteractionPath) -Force + Write-TestJson -Path $gateRunFile.FullName -Value $gateRun + } + $gateMarker = Join-Path $gatePackage 'fanout-eval-02\with_skill\home\preflight-incompatible' + [IO.File]::WriteAllText($gateMarker, 'fixture', [Text.UTF8Encoding]::new($false)) + $gateLogPath = Join-Path $testRoot 'runner-owned-gate-events.jsonl' + $env:AGENTIC_RUNNER_FIXTURE_LOG = $gateLogPath + $gateControl = Invoke-ForegroundPhaseOne -IterationDirectory $gatePackage + Assert-Equal 2 $gateControl.ExitCode ("incompatible runner-owned preflight exits non-zero; output: " + $gateControl.Text) + $gateSummary = $gateControl.Document + Assert-Equal 'preflight_incompatible' $gateSummary.status 'incompatible preflight stops before fan-out' + Assert-Equal 6 $gateSummary.preflight_count 'incompatible gate still preflights every pending arm' + Assert-Equal 1 $gateSummary.incompatible_count 'incompatible gate reports one failed arm' + Assert-Equal $false $gateSummary.execution_started 'incompatible gate records zero started executions' + Assert-Equal 0 $gateSummary.execution_count 'incompatible gate reports zero executions' + $failedGateArm = @($gateSummary.preflights | Where-Object { $_.worker_id -eq 'arm-2-with_skill' }) + Assert-Equal 1 $failedGateArm.Count 'incompatible gate preserves the exact manifest worker ID' + Assert-True ([string]$failedGateArm[0].reasons -match 'fixture preflight rejected fanout-eval-02/with_skill') 'incompatible gate preserves the exact preflight reason' + Assert-True ([string]$failedGateArm[0].reasons -match 'scripted interaction capability unsupported') 'incompatible gate rejects an unsupported scripted interaction before execution' + $gateEvents = @(Get-Content -LiteralPath $gateLogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 6 $gateEvents.Count 'incompatible gate records only the six preflight calls' + Assert-Equal 0 @($gateEvents | Where-Object { $_.kind -eq 'execute' }).Count 'incompatible gate starts zero execute processes' + Assert-Equal 0 @(Get-ChildItem -LiteralPath $gatePackage -Recurse -File -Filter 'execution-result.json').Count 'incompatible gate leaves manifest execution results untouched' + + # A compatibility-transport answer has no native delegation evidence and + # is rejected rather than retried through runner.ps1 execute or parent code. + $compatibilityTransportResult = Copy-TestObject -Value $validTerminalEvidence + $compatibilityTransportResult.evidence.PSObject.Properties.Remove('delegation') + Invoke-TerminalEvidenceCase -Name 'compatibility transport result' -Evidence $compatibilityTransportResult -ExpectedFailure 'delegation_terminal_evidence' + + # Regression (Copilot forensic iteration-4): an orchestrator-authored / + # synthetic execution-result must be rejected by the schema gate with no + # fallback to a manual "success". This mirrors the real invalid Copilot run + # -- capture provenance "agent_output", a textual terminal_status instead of + # a numeric exit code, a non-fresh session, and no runner terminal + # delegation evidence -- proving the outer orchestrator cannot hand-build a + # passing result. + $syntheticCopilotResult = [ordered]@{ + schema = (Get-RunnerSchemaNames).Result + protocol_version = (Get-RunnerSchemaNames).Protocol + run_id = '1_with_skill' + session = [ordered]@{ id = ''; fresh = $false; resumed = $true } + status = 'completed' + run = [ordered]@{ eval_id = 1; eval_name = 'strong-name'; configuration = 'with_skill' } + runner = [ordered]@{ name = 'github-copilot'; version = '0.9.1' } + harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = 'unavailable' } + requested = [ordered]@{ model = 'claude-haiku-4.5'; reasoning_effort = $null; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 900 } + resolved = [ordered]@{ status = 'accepted_request'; accepted = [ordered]@{ model = 'claude-haiku-4.5' } } + started_utc = '2026-08-25T20:00:00.000Z' + finished_utc = '2026-08-25T20:00:10.000Z' + duration_seconds = 10 + exit = [ordered]@{ status = 'success'; failure = $null } + terminal_status = 'success' + output_summary = 'Successfully generated repo.snk (1024-bit RSA, 596 bytes)' + final_response = [ordered]@{ status = 'available'; text = 'done' } + input = [ordered]@{ prompt_sha256 = ('0' * 64); run_json_sha256 = ('0' * 64); profile_sha256 = ('0' * 64) } + isolation = [ordered]@{ status = 'verified'; level = 'pragmatic'; hard_filesystem_confinement = $false; capabilities = [ordered]@{}; mechanisms = @(); required_controls = @(); unproven_controls = @() } + telemetry = [ordered]@{ transcript = [ordered]@{ status = 'available'; value = [ordered]@{ transcript_excerpt = '' } } } + evidence = [ordered]@{ capture = [ordered]@{ source = 'agent_output'; terminal = $true; worker_authored = $false }; tool_call_count = 0; transcript_excerpt = '' } + artifacts = @() + warnings = @() + compatibility_deviations = @() + attempt_count = 1 + } + $syntheticResultObject = ($syntheticCopilotResult | ConvertTo-Json -Depth 100) | ConvertFrom-Json + $syntheticRejected = $false + $syntheticReason = '' + try { [void](Assert-ExecutionResult -Result $syntheticResultObject) } catch { $syntheticRejected = $true; $syntheticReason = $_.Exception.Message } + Assert-True $syntheticRejected 'orchestrator-authored synthetic Copilot execution-result is rejected by the schema gate' + Assert-True ($syntheticReason -match 'fresh, non-resumed session|exit.status must be a JSON number') "synthetic Copilot result is rejected for its non-fresh session or textual exit status: $syntheticReason" + # A synthetic result that is schema-shaped but carries agent_output capture + # with no runner delegation evidence is still incompatible at the terminal + # gate, so it can never be graded as a real eval execution. + $agentOutputResult = Copy-TestObject -Value $validTerminalEvidence + $agentOutputResult.evidence | Add-Member -NotePropertyName capture -NotePropertyValue ([pscustomobject]@{ source = 'agent_output'; terminal = $true; worker_authored = $true }) -Force + $agentOutputResult.evidence.PSObject.Properties.Remove('delegation') + Invoke-TerminalEvidenceCase -Name 'agent_output synthetic result' -Evidence $agentOutputResult -ExpectedFailure 'delegation_terminal_evidence' + + # Regression (#6): runner-owned child processes are headless on Windows. The + # start configuration keeps a real isolation boundary but must not create a + # visible console window, and stdout/stderr stay redirected for capture. + $headlessStartInfo = New-RunnerChildProcessStartInfo -FilePath 'pwsh' -ArgumentList @('-NoProfile', '-File', 'runner.ps1', 'execute') -WorkingDirectory $testRoot + Assert-True (-not $headlessStartInfo.UseShellExecute) 'runner child process does not use ShellExecute' + Assert-True ($headlessStartInfo.CreateNoWindow) 'runner child process creates no visible console window' + Assert-True ($headlessStartInfo.RedirectStandardOutput) 'runner child process redirects stdout for exact result capture' + Assert-True ($headlessStartInfo.RedirectStandardError) 'runner child process redirects stderr for diagnostics' + Assert-Equal 'runner.ps1' ([string]$headlessStartInfo.ArgumentList[2]) 'runner child process preserves its exact argument vector without manual quoting' + + # Regression (OpenCode hang): both the shared process helper and the + # runner-owned child watchdog must return after a forced termination even + # when a fake child sleeps forever. The upper bound includes the one-second + # test deadline, the finite kill grace, and the finite stream-drain grace. + $pwshPath = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) + $timeoutDirectory = Join-Path $testRoot 'bounded-timeout' + New-Item -ItemType Directory -Path $timeoutDirectory -Force | Out-Null + $timeoutClock = [Diagnostics.Stopwatch]::StartNew() + $sharedTimeoutResult = Invoke-RunnerProcess -FileName $pwshPath -ArgumentList @('-NoProfile', '-Command', 'Start-Sleep -Seconds 60') -WorkingDirectory $timeoutDirectory -TimeoutSeconds 1 + $timeoutClock.Stop() + Assert-True ([bool]$sharedTimeoutResult.TimedOut) 'shared process helper reports a sleeping child as timed_out' + Assert-True ([bool]$sharedTimeoutResult.KillAttempted) 'shared process helper attempts entire-process-tree termination at the deadline' + Assert-True ($timeoutClock.Elapsed.TotalSeconds -lt 12) ("shared process timeout path remains bounded; elapsed={0:N3}s" -f $timeoutClock.Elapsed.TotalSeconds) + Assert-True ($sharedTimeoutResult.DurationSeconds -lt 12) 'shared process timeout result reports a bounded duration' + + $watchdogStdout = Join-Path $timeoutDirectory 'watchdog.stdout' + $watchdogStderr = Join-Path $timeoutDirectory 'watchdog.stderr' + $watchdogChild = Start-RunnerChildProcess -FilePath $pwshPath -ArgumentList @('-NoProfile', '-Command', 'Start-Sleep -Seconds 60') -WorkingDirectory $timeoutDirectory -StdoutPath $watchdogStdout -StderrPath $watchdogStderr -TimeoutSeconds 1 + $watchdogRecord = [pscustomobject]@{ worker_id = 'arm-watchdog-with_skill'; child = $watchdogChild; Process = $watchdogChild.Process } + $watchdogList = [System.Collections.Generic.List[object]]::new() + $watchdogList.Add($watchdogRecord) + $watchdogClock = [Diagnostics.Stopwatch]::StartNew() + $watchdogIndex = Wait-AnyRunnerChild -Running $watchdogList + $watchdogExitCode = Complete-RunnerChildProcess -Child $watchdogChild + $watchdogClock.Stop() + Assert-Equal 0 $watchdogIndex 'runner-child watchdog selects the exact expired worker' + Assert-True ([bool]$watchdogChild.WatchdogExpired) 'runner-child watchdog marks the selected worker expired' + Assert-True ([bool]$watchdogChild.TimedOut) 'runner-child completion reports the watchdog timeout' + Assert-True ($watchdogClock.Elapsed.TotalSeconds -lt 12) ("runner-child watchdog kill and drain remain bounded; elapsed={0:N3}s" -f $watchdogClock.Elapsed.TotalSeconds) + Assert-True ($null -eq $watchdogExitCode) 'timed-out runner child has no synthesized exit success' + + # Regression (#5.8): capacity is released when ANY child completes, not only + # the oldest queued child. Wait-AnyRunnerChild must return whichever child + # exited first so a slow sibling never blocks refilling a freed slot. + $waitAnyLaterFirst = [System.Collections.Generic.List[object]]::new() + $waitAnyLaterFirst.Add([pscustomobject]@{ Process = [pscustomobject]@{ HasExited = $false } }) + $waitAnyLaterFirst.Add([pscustomobject]@{ Process = [pscustomobject]@{ HasExited = $false } }) + $waitAnyLaterFirst.Add([pscustomobject]@{ Process = [pscustomobject]@{ HasExited = $true } }) + Assert-Equal 2 (Wait-AnyRunnerChild -Running $waitAnyLaterFirst) 'capacity refill selects a later-completing child, not the oldest queued one' + $waitAnyOldestFirst = [System.Collections.Generic.List[object]]::new() + $waitAnyOldestFirst.Add([pscustomobject]@{ Process = [pscustomobject]@{ HasExited = $true } }) + $waitAnyOldestFirst.Add([pscustomobject]@{ Process = [pscustomobject]@{ HasExited = $false } }) + Assert-Equal 0 (Wait-AnyRunnerChild -Running $waitAnyOldestFirst) 'capacity refill also selects the oldest child when it completes first' + + # Regression (OpenCode forensic iteration-5): synthetic concurrency state + # cannot be injected. The runner-owned fan-out refuses to run over a + # pre-authored orchestration-state.json, so a hand-written max_observed_active + # (with no observed process/session lifecycle) is rejected fail-closed. + $syntheticStatePackage = Join-Path $testRoot 'runner-owned synthetic concurrency package' + Copy-Item -LiteralPath $fanoutPackage -Destination $syntheticStatePackage -Recurse -Force + foreach ($executionResultFile in @(Get-ChildItem -LiteralPath $syntheticStatePackage -Recurse -File -Filter 'execution-result.json')) { + Remove-Item -LiteralPath $executionResultFile.FullName -Force + } + Remove-PhaseOnePackageState -IterationDirectory $syntheticStatePackage + Write-TestJson -Path (Join-Path $syntheticStatePackage 'orchestration-state.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestration-state/1' + dispatch_owner = 'orchestrator' + requested_concurrency = 16 + parallel_dispatch_required = $true + minimum_parallel_workers = 2 + pending_worker_ids = @() + active = [ordered]@{} + completed = [ordered]@{} + delegation_rejections = [ordered]@{} + capacity_limit_reported = $false + eval_attempts = [ordered]@{} + max_observed_active = 6 + }) + $syntheticStateLog = Join-Path $testRoot 'runner-owned-synthetic-state-events.jsonl' + $env:AGENTIC_RUNNER_FIXTURE_LOG = $syntheticStateLog + $syntheticStateControl = Invoke-ForegroundPhaseOne -IterationDirectory $syntheticStatePackage + Assert-Equal 2 $syntheticStateControl.ExitCode ("foreground Phase 1 rejects a pre-authored orchestration state; output: " + $syntheticStateControl.Text) + $syntheticStateSummary = $syntheticStateControl.Document + Assert-Equal 'failed' $syntheticStateSummary.status 'synthetic concurrency state cannot drive a completed fan-out' + Assert-True ([string]$syntheticStateSummary.error -match 'Runner-owned fan-out refuses to replace an existing orchestration state') ("foreground Phase 1 fails closed on pre-authored state; error: " + [string]$syntheticStateSummary.error) + Assert-Equal 0 @(Get-ChildItem -LiteralPath $syntheticStatePackage -Recurse -File -Filter 'execution-result.json').Count 'synthetic concurrency rejection starts zero real executions' + + Write-Output 'Native worker orchestration: PASS' +} finally { + if ($null -eq $oldFixtureLogPath) { Remove-Item Env:AGENTIC_RUNNER_FIXTURE_LOG -ErrorAction SilentlyContinue } else { $env:AGENTIC_RUNNER_FIXTURE_LOG = $oldFixtureLogPath } + if (Test-Path -LiteralPath $testRoot) { Remove-Item -LiteralPath $testRoot -Recurse -Force } +} diff --git a/scripts/eval-runners/tests/test-phase1-aggregate-regressions.ps1 b/scripts/eval-runners/tests/test-phase1-aggregate-regressions.ps1 new file mode 100644 index 0000000..6a93e98 --- /dev/null +++ b/scripts/eval-runners/tests/test-phase1-aggregate-regressions.ps1 @@ -0,0 +1,435 @@ +<#! +.SYNOPSIS + Deterministic Phase 1 aggregate fail-closed regressions. + +.DESCRIPTION + Exercises the real runner-owned Phase 1 path with mixed terminal outcomes + and with completed raw results whose evidence validation fails. MODEL-FREE. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'manifest-paths.ps1') +. (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'package-integrity.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -cne [string]$Actual) { + throw "ASSERT: $Message (expected '$Expected', got '$Actual')" + } +} + +function Assert-Contains { + param([string]$Text, [string]$Expected, [string]$Message) + if ($Text.IndexOf($Expected, [System.StringComparison]::OrdinalIgnoreCase) -lt 0) { + throw "ASSERT: $Message (missing '$Expected')" + } +} + +function Write-TestJson { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) + + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Read-TestJson { + param([Parameter(Mandatory = $true)][string]$Path) + + return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 +} + +function Invoke-TestTool { + param( + [Parameter(Mandatory = $true)][string]$Path, + [Parameter(Mandatory = $true)][string[]]$Arguments + ) + + $output = & pwsh -NoProfile -File $Path @Arguments 2>&1 + return [pscustomobject]@{ + ExitCode = $LASTEXITCODE + Text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + } +} + +function Invoke-ForegroundPhaseOne { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][string]$IterationDirectory) + + $invocation = Invoke-TestTool -Path $Path -Arguments @('-IterationDirectory', $IterationDirectory) + $document = $invocation.Text | ConvertFrom-Json -Depth 100 + return [pscustomobject]@{ ExitCode = $invocation.ExitCode; Text = $invocation.Text; Document = $document } +} + +function Assert-ToolFails { + param( + [Parameter(Mandatory = $true)][object]$Invocation, + [Parameter(Mandatory = $true)][string]$Description, + [string]$ExpectedText = '' + ) + + if ([int]$Invocation.ExitCode -eq 0) { + throw "ASSERT: $Description unexpectedly passed: $($Invocation.Text)" + } + if (-not [string]::IsNullOrWhiteSpace($ExpectedText)) { + Assert-Contains -Text $Invocation.Text -Expected $ExpectedText -Message $Description + } +} + +function New-TestRun { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][int]$EvalId, + [Parameter(Mandatory = $true)][string]$EvalName + ) + + $runDirectory = Join-Path (Join-Path $IterationDirectory $EvalName) 'with_skill' + $repoDirectory = Join-Path $runDirectory 'repo' + $homeDirectory = Join-Path $runDirectory 'home' + $skillDirectory = Join-Path $runDirectory 'skill/test-skill' + New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $skillDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'execute-delay-ms'), '0', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $skillDirectory 'SKILL.md'), '# deterministic fixture skill`n', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "phase1 aggregate prompt for $EvalName/with_skill`n", [System.Text.UTF8Encoding]::new($false)) + + $run = [ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $EvalId + evalName = $EvalName + skillName = 'test-skill' + iteration = 1 + mode = 'with_skill' + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = 'skill/test-skill' + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + inputFiles = @() + fixtureHash = ('a' * 64) + skillHash = ('b' * 64) + contract = [ordered]@{ + sandboxRoot = '.' + workingDirectory = 'repo' + homeDirectory = 'home' + mustNotReadOutsideSandbox = $true + mustNotExposeGlobalSkillsOrConfig = $true + } + } + Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value $run + + return [pscustomobject]@{ + RunDirectory = $runDirectory + HomeDirectory = $homeDirectory + } +} + +function New-ReportFixtureScript { + param([Parameter(Mandatory = $true)][string]$Path) + + $scriptText = @' +[CmdletBinding()] +param([Parameter(Mandatory = $true)][string]$IterationDirectory, [switch]$RequireComplete) +$ErrorActionPreference = 'Stop' +foreach ($file in @('report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { + [System.IO.File]::WriteAllText((Join-Path $IterationDirectory $file), "unexpected report artifact: $file`n", [System.Text.UTF8Encoding]::new($false)) +} +'@ + [System.IO.File]::WriteAllText($Path, $scriptText, [System.Text.UTF8Encoding]::new($false)) +} + +function Initialize-PhaseOneFailurePackage { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][hashtable]$StatusesByEvalId, + [int[]]$EvidenceFailureEvalIds = @() + ) + + $packageTools = Join-Path $IterationDirectory 'tools/eval-runners' + New-Item -ItemType Directory -Path $packageTools -Force | Out-Null + foreach ($item in @(Get-ChildItem -LiteralPath $runnerRoot -Force)) { + Copy-Item -LiteralPath $item.FullName -Destination $packageTools -Recurse -Force + } + $fixtureDirectory = Join-Path $packageTools 'fixture' + New-Item -ItemType Directory -Path $fixtureDirectory -Force | Out-Null + Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests/fixtures/runner-owned-fixture.ps1') -Destination (Join-Path $fixtureDirectory 'runner.ps1') -Force + New-ReportFixtureScript -Path (Join-Path $IterationDirectory 'tools/test-report.ps1') + + $manifestEvals = [System.Collections.Generic.List[object]]::new() + for ($evalId = 1; $evalId -le 4; $evalId++) { + $evalName = 'phase1-eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $IterationDirectory $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-metadata/1' + eval_id = $evalId + eval_name = $evalName + prompt = "fixture prompt $evalId" + expected_output = 'fixture output' + assertions = @('deterministic fixture assertion') + }) + + $run = New-TestRun -IterationDirectory $IterationDirectory -EvalId $evalId -EvalName $evalName + $status = if ($StatusesByEvalId.ContainsKey($evalId)) { [string]$StatusesByEvalId[$evalId] } else { 'completed' } + if ($status -ne 'completed') { + [System.IO.File]::WriteAllText((Join-Path $run.HomeDirectory 'terminal-status'), $status, [System.Text.UTF8Encoding]::new($false)) + } + if ($EvidenceFailureEvalIds -contains $evalId) { + [System.IO.File]::WriteAllText((Join-Path $run.HomeDirectory 'evidence-validation-failed'), 'fixture', [System.Text.UTF8Encoding]::new($false)) + } + + $resultsDirectory = Join-Path $evalDirectory 'results' + New-Item -ItemType Directory -Path $resultsDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $resultsDirectory 'with_skill.result.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + eval_id = $evalId + eval_name = $evalName + configuration = 'with_skill' + execution_status = 'unrun' + grading = @([ordered]@{ text = 'deterministic fixture assertion'; passed = $null; evidence = '' }) + }) + + $manifestEvals.Add([ordered]@{ + eval_id = $evalId + eval_name = $evalName + directory = $evalName + metadata = "$evalName/eval-metadata.json" + runs = [ordered]@{ + with_skill = [ordered]@{ + mode = 'with_skill' + run_manifest = "$evalName/with_skill/run.json" + execution_result = "$evalName/results/with_skill.execution-result.json" + result = "$evalName/results/with_skill.result.json" + } + } + }) + } + + $toolIntegrity = Get-PackageTreeIntegrity -Root $packageTools + Write-TestJson -Path (Join-Path $IterationDirectory 'manifest.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + skill_name = 'phase1-aggregate-fixture' + iteration = 1 + configurations = @('with_skill') + execution_profile = 'execution-profile.json' + runner_tools = 'tools/eval-runners' + runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } + execution_freeze = 'execution-freeze.json' + grading = 'grading.json' + report = [ordered]@{ tool = 'tools/test-report.ps1' } + evals = @($manifestEvals.ToArray()) + }) + Write-TestJson -Path (Join-Path $IterationDirectory 'execution-profile.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fixture' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 60 + concurrency = 4 + }) + + return [pscustomobject]@{ + IterationDirectory = $IterationDirectory + FanoutScript = Join-Path $packageTools 'invoke-runner-owned-arms.ps1' + BridgeScript = Join-Path $packageTools 'bridge-manifest-results.ps1' + FinalizerScript = Join-Path $packageTools 'finalize-eval-package.ps1' + LogPath = Join-Path $IterationDirectory 'runner-events.jsonl' + Records = @(Get-ManifestRunRecords -IterationDirectory $IterationDirectory -Manifest (Read-TestJson -Path (Join-Path $IterationDirectory 'manifest.json')) | Sort-Object EvalId, Configuration) + } +} + +function Assert-Counts { + param( + [Parameter(Mandatory = $true)][object]$Source, + [Parameter(Mandatory = $true)][hashtable]$Expected, + [Parameter(Mandatory = $true)][string]$MessagePrefix + ) + + foreach ($name in @( + 'expected_count', + 'terminal_count', + 'completed_count', + 'failed_count', + 'timed_out_count', + 'cancelled_count', + 'incompatible_count', + 'evidence_validation_failed_count' + )) { + Assert-Equal $Expected[$name] (Get-JsonProperty -Object $Source -Name $name -Default $null) "$MessagePrefix $name" + } +} + +function Assert-ArmSummaryShape { + param([Parameter(Mandatory = $true)][object]$Summary, [Parameter(Mandatory = $true)][string]$ScenarioName) + + foreach ($arm in @($Summary.arms)) { + foreach ($field in @('worker_id', 'eval_id', 'configuration', 'status', 'worker_session_id', 'evidence_validation')) { + Assert-True (Test-JsonProperty -Object $arm -Name $field) "$ScenarioName arm summary contains $field" + } + $evidenceValidation = Get-JsonProperty -Object $arm -Name 'evidence_validation' -Default $null + Assert-True (Test-JsonProperty -Object $evidenceValidation -Name 'status') "$ScenarioName arm summary contains evidence_validation.status" + Assert-True (Test-JsonProperty -Object $evidenceValidation -Name 'reasons') "$ScenarioName arm summary contains evidence_validation.reasons" + } +} + +function Assert-CanonicalResultsRemainUnrun { + param([Parameter(Mandatory = $true)][object[]]$Records, [Parameter(Mandatory = $true)][string]$ScenarioName) + + foreach ($record in $Records) { + $result = Read-TestJson -Path $record.ResultPath + Assert-Equal 'unrun' ([string](Get-JsonProperty -Object $result -Name 'execution_status' -Default '')) "$ScenarioName keeps $($record.ResultRelative) unbridged" + } +} + +function Assert-NoPhaseTwoArtifacts { + param([Parameter(Mandatory = $true)][string]$IterationDirectory, [Parameter(Mandatory = $true)][string]$ScenarioName) + + foreach ($relative in @('grading.json', 'report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { + Assert-True (-not (Test-Path -LiteralPath (Join-Path $IterationDirectory $relative) -PathType Leaf)) "$ScenarioName does not produce $relative" + } +} + +function Assert-NoRetries { + param([Parameter(Mandatory = $true)][string]$LogPath, [Parameter(Mandatory = $true)][int]$ExpectedArmCount, [Parameter(Mandatory = $true)][string]$ScenarioName) + + $events = @(Get-Content -LiteralPath $LogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal ($ExpectedArmCount * 2) $events.Count "$ScenarioName records one preflight and one execute event per arm" + Assert-Equal $ExpectedArmCount @($events | Where-Object { $_.kind -eq 'preflight' }).Count "$ScenarioName records one preflight per arm" + Assert-Equal $ExpectedArmCount @($events | Where-Object { $_.kind -eq 'execute' }).Count "$ScenarioName records one execute per arm" + + foreach ($evalId in 1..$ExpectedArmCount) { + $executeEvents = @($events | Where-Object { $_.kind -eq 'execute' -and [int]$_.eval_id -eq $evalId -and [string]$_.configuration -eq 'with_skill' }) + Assert-Equal 1 $executeEvents.Count "$ScenarioName does not retry eval $evalId" + } +} + +function Assert-LedgerMatchesFrozenStatuses { + param([Parameter(Mandatory = $true)][object]$FreezeValidation, [Parameter(Mandatory = $true)][string]$ScenarioName) + + foreach ($entry in @($FreezeValidation.Freeze.executions)) { + $terminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name ([string]$entry.worker_id) -Default $null + Assert-Equal ([string]$entry.terminal_status) ([string](Get-JsonProperty -Object $terminal -Name 'status' -Default '')) "$ScenarioName preserves frozen ledger status for $($entry.worker_id)" + } +} + +function Invoke-PhaseOneFailureScenario { + param( + [Parameter(Mandatory = $true)][string]$ScenarioName, + [Parameter(Mandatory = $true)][hashtable]$StatusesByEvalId, + [int[]]$EvidenceFailureEvalIds = @(), + [Parameter(Mandatory = $true)][hashtable]$ExpectedCounts, + [Parameter(Mandatory = $true)][string[]]$ExpectedFrozenStatuses, + [scriptblock]$AdditionalAssertions = $null + ) + + $iterationDirectory = Join-Path $testRoot $ScenarioName + $package = Initialize-PhaseOneFailurePackage -IterationDirectory $iterationDirectory -StatusesByEvalId $StatusesByEvalId -EvidenceFailureEvalIds $EvidenceFailureEvalIds + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $package.LogPath) + + $fanout = Invoke-ForegroundPhaseOne -Path $package.FanoutScript -IterationDirectory $iterationDirectory + Assert-Equal 2 $fanout.ExitCode "$ScenarioName Phase 1 exits non-zero" + $summary = $fanout.Document + Assert-Equal 'phase1' ([string](Get-JsonProperty -Object $summary -Name 'phase' -Default '')) "$ScenarioName summary identifies Phase 1" + Assert-Equal 'failed' ([string](Get-JsonProperty -Object $summary -Name 'status' -Default '')) "$ScenarioName summary is non-success" + Assert-Counts -Source $summary -Expected $ExpectedCounts -MessagePrefix "$ScenarioName summary" + Assert-ArmSummaryShape -Summary $summary -ScenarioName $ScenarioName + Assert-True (Test-Path -LiteralPath (Join-Path $iterationDirectory 'execution-freeze.json') -PathType Leaf) "$ScenarioName writes execution-freeze.json before failing" + $summaryFreeze = Get-JsonProperty -Object $summary -Name 'execution_freeze' -Default $null + Assert-True (Test-JsonProperty -Object $summaryFreeze -Name 'path') "$ScenarioName summary reports execution_freeze.path" + Assert-True (Test-JsonProperty -Object $summaryFreeze -Name 'sha256') "$ScenarioName summary reports execution_freeze.sha256" + + $freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iterationDirectory -RequireOrchestrationState + Assert-True (-not [bool]$freezeValidation.PhaseOneSuccess) "$ScenarioName frozen aggregate remains non-success" + Assert-Counts -Source $freezeValidation.Aggregate -Expected $ExpectedCounts -MessagePrefix "$ScenarioName frozen aggregate" + Assert-Equal ([string]::Join(',', $ExpectedFrozenStatuses)) ([string]::Join(',', @($freezeValidation.Freeze.executions | ForEach-Object { [string]$_.terminal_status }))) "$ScenarioName freeze preserves exact raw terminal statuses" + Assert-LedgerMatchesFrozenStatuses -FreezeValidation $freezeValidation -ScenarioName $ScenarioName + Assert-CanonicalResultsRemainUnrun -Records $package.Records -ScenarioName $ScenarioName + Assert-NoPhaseTwoArtifacts -IterationDirectory $iterationDirectory -ScenarioName $ScenarioName + + $bridge = Invoke-TestTool -Path $package.BridgeScript -Arguments @('-IterationDirectory', $iterationDirectory, '-RequireComplete', '-RequireParallelDispatch', '-RequireNativeDelegation') + Assert-ToolFails -Invocation $bridge -Description "$ScenarioName complete bridge is blocked" -ExpectedText 'completion gate failed' + Assert-CanonicalResultsRemainUnrun -Records $package.Records -ScenarioName $ScenarioName + + $finalizer = Invoke-TestTool -Path $package.FinalizerScript -Arguments @('-IterationDirectory', $iterationDirectory) + Assert-ToolFails -Invocation $finalizer -Description "$ScenarioName finalizer is blocked" -ExpectedText 'Manifest bridge failed' + Assert-NoPhaseTwoArtifacts -IterationDirectory $iterationDirectory -ScenarioName $ScenarioName + + if ($null -ne $AdditionalAssertions) { + & $AdditionalAssertions $summary $freezeValidation $package + } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-phase1-aggregate-' + [Guid]::NewGuid().ToString('N')) +$oldFixtureLogPath = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG') +try { + Invoke-PhaseOneFailureScenario ` + -ScenarioName 'mixed-terminal' ` + -StatusesByEvalId @{ 1 = 'completed'; 2 = 'timed_out'; 3 = 'failed'; 4 = 'completed' } ` + -ExpectedCounts @{ + expected_count = 4 + terminal_count = 4 + completed_count = 2 + failed_count = 1 + timed_out_count = 1 + cancelled_count = 0 + incompatible_count = 0 + evidence_validation_failed_count = 0 + } ` + -ExpectedFrozenStatuses @('completed', 'timed_out', 'failed', 'completed') ` + -AdditionalAssertions { + param($Summary, $FreezeValidation, $Package) + + Assert-NoRetries -LogPath $Package.LogPath -ExpectedArmCount 4 -ScenarioName 'mixed-terminal' + foreach ($workerId in @('arm-1-with_skill', 'arm-2-with_skill', 'arm-3-with_skill', 'arm-4-with_skill')) { + $terminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name $workerId -Default $null + $evidenceValidation = Get-JsonProperty -Object $terminal -Name 'evidence_validation' -Default $null + Assert-Equal 'passed' ([string](Get-JsonProperty -Object $evidenceValidation -Name 'status' -Default '')) "mixed-terminal keeps honest evidence_validation for $workerId" + } + } + + Invoke-PhaseOneFailureScenario ` + -ScenarioName 'completed-with-evidence-failure' ` + -StatusesByEvalId @{ 1 = 'completed'; 2 = 'completed'; 3 = 'completed'; 4 = 'completed' } ` + -EvidenceFailureEvalIds @(3) ` + -ExpectedCounts @{ + expected_count = 4 + terminal_count = 4 + completed_count = 4 + failed_count = 0 + timed_out_count = 0 + cancelled_count = 0 + incompatible_count = 0 + evidence_validation_failed_count = 1 + } ` + -ExpectedFrozenStatuses @('completed', 'completed', 'completed', 'completed') ` + -AdditionalAssertions { + param($Summary, $FreezeValidation, $Package) + + $failedTerminal = Get-JsonProperty -Object (Get-JsonProperty -Object $FreezeValidation.State -Name 'completed' -Default $null) -Name 'arm-3-with_skill' -Default $null + $failedEvidence = Get-JsonProperty -Object $failedTerminal -Name 'evidence_validation' -Default $null + Assert-Equal 'completed' ([string](Get-JsonProperty -Object $failedTerminal -Name 'status' -Default '')) 'evidence-failure scenario keeps the raw completed status' + Assert-Equal 'failed' ([string](Get-JsonProperty -Object $failedEvidence -Name 'status' -Default '')) 'evidence-failure scenario records failed evidence validation' + Assert-Contains -Text ([string]::Join(', ', @((Get-JsonProperty -Object $failedEvidence -Name 'reasons' -Default @()) | ForEach-Object { [string]$_ }))) -Expected 'prompt_fidelity' -Message 'evidence-failure scenario preserves the validation reason' + } + + Write-Output 'Phase 1 aggregate regressions: PASS' +} finally { + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $oldFixtureLogPath) + if (Test-Path -LiteralPath $testRoot) { + Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue + } +} diff --git a/scripts/eval-runners/tests/test-phase1-controller-lifecycle.ps1 b/scripts/eval-runners/tests/test-phase1-controller-lifecycle.ps1 new file mode 100644 index 0000000..76cbc65 --- /dev/null +++ b/scripts/eval-runners/tests/test-phase1-controller-lifecycle.ps1 @@ -0,0 +1,229 @@ +<#! +.SYNOPSIS + Deterministic foreground runner-owned Phase 1 lifecycle tests. + +.DESCRIPTION + Exercises the restored runner-owned topology: one foreground + invoke-runner-owned-arms.ps1 invocation per iteration. The fixture runner + is model-free and never calls an AI CLI. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'manifest-paths.ps1') +. (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'package-integrity.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -ne [string]$Actual) { + throw "ASSERT: $Message (expected '$Expected', got '$Actual')" + } +} + +function Write-TestJson { + param([Parameter(Mandatory = $true)][string]$Path, [Parameter(Mandatory = $true)][object]$Value) + + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Read-TestJson { + param([Parameter(Mandatory = $true)][string]$Path) + + return [System.IO.File]::ReadAllText($Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 100 +} + +function Invoke-ForegroundPhaseOne { + param([Parameter(Mandatory = $true)][string]$IterationDirectory) + + $fanout = Join-Path $IterationDirectory 'tools/eval-runners/invoke-runner-owned-arms.ps1' + $output = & pwsh -NoProfile -File $fanout -IterationDirectory $IterationDirectory 2>&1 + $exitCode = $LASTEXITCODE + $text = [string]::Join([Environment]::NewLine, @($output | ForEach-Object { [string]$_ })) + $document = $text | ConvertFrom-Json -Depth 100 + return [pscustomobject]@{ ExitCode = $exitCode; Text = $text; Document = $document } +} + +function New-ForegroundPackage { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [int]$EvalCount = 2, + [int]$Concurrency = 2 + ) + + $tools = Join-Path $IterationDirectory 'tools\eval-runners' + New-Item -ItemType Directory -Path $tools -Force | Out-Null + foreach ($toolItem in @(Get-ChildItem -LiteralPath $runnerRoot -Force | Where-Object { $_.Name -ne 'tests' })) { + Copy-Item -LiteralPath $toolItem.FullName -Destination $tools -Recurse -Force + } + $fixtureRunnerDirectory = Join-Path $tools 'fixture' + New-Item -ItemType Directory -Path $fixtureRunnerDirectory -Force | Out-Null + Copy-Item -LiteralPath (Join-Path $runnerRoot 'tests\fixtures\runner-owned-fixture.ps1') -Destination (Join-Path $fixtureRunnerDirectory 'runner.ps1') -Force + Write-TestJson -Path (Join-Path $IterationDirectory 'execution-profile.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fixture' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = $Concurrency + }) + + $manifestEvals = [System.Collections.Generic.List[object]]::new() + for ($evalId = 1; $evalId -le $EvalCount; $evalId++) { + $evalName = 'foreground-eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $IterationDirectory $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ eval_id = $evalId; eval_name = $evalName; assertions = @('fixture') }) + $runs = [ordered]@{} + foreach ($configuration in @('with_skill', 'without_skill')) { + $runDirectory = Join-Path $evalDirectory $configuration + $repoDirectory = Join-Path $runDirectory 'repo' + $homeDirectory = Join-Path $runDirectory 'home' + $resultDirectory = Join-Path $evalDirectory 'results' + New-Item -ItemType Directory -Path $repoDirectory, $homeDirectory, $resultDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $repoDirectory 'input.txt'), "$evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "foreground prompt $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + $skillDirectory = $null + $skillHash = $null + if ($configuration -eq 'with_skill') { + $skillDirectory = 'skill/candidate' + New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill\candidate') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'skill\candidate\SKILL.md'), '# fixture', [System.Text.UTF8Encoding]::new($false)) + $skillHash = ('b' * 64) + } + Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = $evalId + evalName = $evalName + skillName = if ($configuration -eq 'with_skill') { 'candidate' } else { $null } + iteration = 1 + mode = $configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $skillDirectory + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + fixtureHash = ('a' * 64) + skillHash = $skillHash + }) + $resultName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } + $executionName = if ($configuration -eq 'with_skill') { 'with-skill.execution-result.json' } else { 'without-skill.execution-result.json' } + Write-TestJson -Path (Join-Path $resultDirectory $resultName) -Value ([ordered]@{ eval_id = $evalId; configuration = $configuration; execution_status = 'unrun' }) + $runs[$configuration] = [ordered]@{ + mode = $configuration + run_manifest = "$evalName/$configuration/run.json" + execution_result = "$evalName/results/$executionName" + result = "$evalName/results/$resultName" + } + } + $manifestEvals.Add([ordered]@{ eval_id = $evalId; eval_name = $evalName; directory = $evalName; metadata = "$evalName/eval-metadata.json"; runs = $runs }) + } + + $toolIntegrity = Get-PackageTreeIntegrity -Root $tools + Write-TestJson -Path (Join-Path $IterationDirectory 'manifest.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + configurations = @('with_skill', 'without_skill') + execution_profile = 'execution-profile.json' + runner_tools = 'tools/eval-runners' + runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $toolIntegrity.Sha256; file_count = $toolIntegrity.FileCount } + execution_freeze = 'execution-freeze.json' + evals = @($manifestEvals.ToArray()) + }) + + return [pscustomobject]@{ + IterationDirectory = $IterationDirectory + Tools = $tools + LogPath = Join-Path $IterationDirectory 'fixture-events.jsonl' + } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-foreground-phase1-' + [Guid]::NewGuid().ToString('N')) +$oldFixtureLogPath = [Environment]::GetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG') +try { + $success = New-ForegroundPackage -IterationDirectory (Join-Path $testRoot 'success') -EvalCount 2 -Concurrency 2 + foreach ($obsolete in @('phase1-control-common.ps1', 'control-runner-owned-phase1.ps1', 'supervise-runner-owned-phase1.ps1')) { + Assert-True (-not (Test-Path -LiteralPath (Join-Path $success.Tools $obsolete) -PathType Leaf)) "foreground package does not carry obsolete $obsolete" + } + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $success.LogPath) + $first = Invoke-ForegroundPhaseOne -IterationDirectory $success.IterationDirectory + Assert-Equal 0 $first.ExitCode 'foreground Phase 1 exits successfully' + Assert-Equal 'phase1' ([string]$first.Document.phase) 'foreground Phase 1 returns the fan-out summary directly' + Assert-Equal 'completed' ([string]$first.Document.status) 'foreground Phase 1 completes' + Assert-Equal 4 ([int]$first.Document.expected_count) 'foreground Phase 1 sees four paired arms' + Assert-Equal 4 ([int]$first.Document.terminal_count) 'foreground Phase 1 registers every arm terminal' + Assert-Equal 4 ([int]$first.Document.execution_count) 'foreground Phase 1 executes every compatible arm' + Assert-True ([int]$first.Document.max_observed_active -gt 1) 'foreground Phase 1 honors requested concurrency when capacity permits' + Assert-True (Test-Path -LiteralPath (Join-Path $success.IterationDirectory 'execution-freeze.json') -PathType Leaf) 'foreground Phase 1 writes execution-freeze.json only after terminal arms' + $freeze = Assert-ExecutionFreeze -IterationDirectory $success.IterationDirectory -RequireOrchestrationState + Assert-True ([bool]$freeze.PhaseOneSuccess) 'foreground Phase 1 freeze validates as successful' + Assert-Equal 4 @($freeze.Freeze.executions).Count 'foreground Phase 1 freeze contains every expected arm' + $events = @(Get-Content -LiteralPath $success.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 8 $events.Count 'foreground Phase 1 invokes each preflight and execution exactly once' + Assert-Equal 4 @($events | Where-Object { $_.kind -eq 'preflight' }).Count 'foreground Phase 1 preflights every arm' + Assert-Equal 4 @($events | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 executes every compatible arm' + $firstExecuteIndex = -1 + $lastPreflightIndex = -1 + for ($index = 0; $index -lt $events.Count; $index++) { + if ($events[$index].kind -eq 'preflight') { $lastPreflightIndex = $index } + if ($events[$index].kind -eq 'execute' -and $firstExecuteIndex -lt 0) { $firstExecuteIndex = $index } + } + Assert-True ($firstExecuteIndex -gt $lastPreflightIndex) 'foreground Phase 1 starts zero executions before all preflights pass' + + $second = Invoke-ForegroundPhaseOne -IterationDirectory $success.IterationDirectory + Assert-Equal 2 $second.ExitCode 'foreground Phase 1 refuses a second invocation after freeze' + Assert-True ([string]$second.Document.error -match 'already frozen|existing orchestration state') 'foreground Phase 1 reports why rerun is refused' + $eventsAfterSecond = @(Get-Content -LiteralPath $success.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 4 @($eventsAfterSecond | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 rerun starts zero additional executions' + + $interrupted = Join-Path $testRoot 'interrupted' + Copy-Item -LiteralPath $success.IterationDirectory -Destination $interrupted -Recurse -Force + Remove-Item -LiteralPath (Join-Path $interrupted 'execution-freeze.json') -Force + foreach ($raw in @(Get-ChildItem -LiteralPath $interrupted -Recurse -File -Filter '*.execution-result.json')) { + Remove-Item -LiteralPath $raw.FullName -Force + } + $interruptedLog = Join-Path $interrupted 'new-events.jsonl' + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $interruptedLog) + $interruptedResult = Invoke-ForegroundPhaseOne -IterationDirectory $interrupted + Assert-Equal 2 $interruptedResult.ExitCode 'foreground Phase 1 fails closed on interrupted state without a freeze' + Assert-True ([string]$interruptedResult.Document.error -match 'refuses to replace an existing orchestration state') 'foreground Phase 1 does not adopt or rerun incomplete state' + Assert-True (-not (Test-Path -LiteralPath $interruptedLog -PathType Leaf)) 'foreground Phase 1 interrupted-state refusal starts zero executions' + + $preflightGate = New-ForegroundPackage -IterationDirectory (Join-Path $testRoot 'preflight-gate') -EvalCount 2 -Concurrency 2 + [System.IO.File]::WriteAllText((Join-Path $preflightGate.IterationDirectory 'foreground-eval-02\with_skill\home\preflight-incompatible'), 'fixture', [System.Text.UTF8Encoding]::new($false)) + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $preflightGate.LogPath) + $gate = Invoke-ForegroundPhaseOne -IterationDirectory $preflightGate.IterationDirectory + Assert-Equal 2 $gate.ExitCode 'foreground Phase 1 exits non-zero for incompatible preflight' + Assert-Equal 'preflight' ([string]$gate.Document.phase) 'foreground Phase 1 reports preflight phase failure' + Assert-Equal 'preflight_incompatible' ([string]$gate.Document.status) 'foreground Phase 1 reports incompatible preflight' + Assert-Equal 4 ([int]$gate.Document.preflight_count) 'foreground Phase 1 still probes every arm' + Assert-True (-not [bool]$gate.Document.execution_started) 'foreground Phase 1 starts zero executions when any preflight is incompatible' + $gateEvents = @(Get-Content -LiteralPath $preflightGate.LogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 4 @($gateEvents | Where-Object { $_.kind -eq 'preflight' }).Count 'foreground Phase 1 preflight gate records every preflight' + Assert-Equal 0 @($gateEvents | Where-Object { $_.kind -eq 'execute' }).Count 'foreground Phase 1 preflight gate records zero executions' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $preflightGate.IterationDirectory 'execution-freeze.json') -PathType Leaf)) 'foreground Phase 1 writes no freeze before a failed preflight gate' + + $fanoutText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'invoke-runner-owned-arms.ps1'), [System.Text.UTF8Encoding]::new($false)) + Assert-True ($fanoutText -notmatch '(?i)Job Object|breakaway|process ancestry|supervisor independence|phase1-control-common|AGENTIC_PHASE1_SUPERVISOR_ID') 'foreground Phase 1 has no Windows host-security/durable-detachment requirement' + + Write-Output 'Runner-owned foreground Phase 1 lifecycle: PASS' +} finally { + [Environment]::SetEnvironmentVariable('AGENTIC_RUNNER_FIXTURE_LOG', $oldFixtureLogPath) + if (Test-Path -LiteralPath $testRoot) { + Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue + } +} diff --git a/scripts/eval-runners/tests/test-probe-environment.ps1 b/scripts/eval-runners/tests/test-probe-environment.ps1 new file mode 100644 index 0000000..daa9a13 --- /dev/null +++ b/scripts/eval-runners/tests/test-probe-environment.ps1 @@ -0,0 +1,257 @@ +<# +.SYNOPSIS + Regression: model-free harness probe temp resolution and probe/eval environment separation. +.DESCRIPTION + Two MODEL-FREE deterministic regressions: + + 1. The iteration-11 failure. The model-free descriptor/version probe must + always have a writable temporary directory and must never fall back to + %WINDIR% via [System.IO.Path]::GetTempPath() when TEMP/TMP/USERPROFILE + are stripped. A normal, non-elevated Windows user must never need write + access to C:\Windows for a --version/--help/describe probe. + + 2. The probe-vs-eval boundary. The model-free probe environment carries OS + scratch TEMP/TMP but no isolated home and no ambient skill/config policy, + while the model-backed OpenCode eval environment pins HOME/USERPROFILE/ + config/TEMP inside the isolated per-run home and disables ambient shared + skill discovery. The two must remain distinct concepts. + + No model is ever executed. The only external processes are `opencode describe` + (a version/help probe) and `pwsh --version`. +#> + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') + +function Assert-True { param([bool]$c, [string]$m) if (-not $c) { throw "ASSERT: $m" } } +function Assert-False { param([bool]$c, [string]$m) if ($c) { throw "ASSERT: $m" } } +function Assert-Equal { param($e, $a, [string]$m) if ([string]$e -ne [string]$a) { throw "ASSERT: $m (expected '$e', got '$a')" } } + +function Import-OpenCodeRunnerFunctions { + if ($null -ne (Get-Command New-OpenCodeEnvironment -CommandType Function -ErrorAction SilentlyContinue)) { return } + $tokens = $null + $errors = $null + $openCodeRunnerPath = Join-Path $runnerRoot 'opencode\runner.ps1' + $ast = [System.Management.Automation.Language.Parser]::ParseFile($openCodeRunnerPath, [ref]$tokens, [ref]$errors) + if ($errors.Count -gt 0) { throw 'probe-environment regression could not parse opencode runner.ps1.' } + $functionAsts = @($ast.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] }, $true) | Sort-Object { $_.Extent.StartOffset }) + foreach ($functionAst in $functionAsts) { + $definition = [regex]::Replace($functionAst.Extent.Text, ('(?im)^function\s+' + [regex]::Escape($functionAst.Name) + '\b'), ('function script:' + $functionAst.Name), 1) + Invoke-Expression $definition + } +} + +$systemDirectories = Get-RunnerSystemDirectorySet + +function Test-IsSystemDirectory { + param([string]$Path) + + if ([string]::IsNullOrWhiteSpace($Path)) { return $false } + try { $full = [System.IO.Path]::GetFullPath($Path) } catch { return $false } + return $systemDirectories.Contains($full.TrimEnd([char[]]@('\', '/'))) +} + +function Invoke-ProbeEnvironmentChild { + param([Parameter(Mandatory = $true)][string[]]$ArgumentList) + + # Applies New-RunnerProbeEnvironment to a child EXACTLY as Invoke-RunnerProcess + # does: a cleared process environment populated only from the probe dictionary. + # This reproduces the stripped-environment context that resolved GetTempPath() + # to C:\WINDOWS in iteration-11. + $probeEnvironment = New-RunnerProbeEnvironment + $startInfo = [System.Diagnostics.ProcessStartInfo]::new() + $startInfo.FileName = [string]((Get-Command pwsh -CommandType Application -ErrorAction Stop | Select-Object -First 1).Source) + foreach ($argument in $ArgumentList) { [void]$startInfo.ArgumentList.Add([string]$argument) } + $startInfo.WorkingDirectory = $runnerRoot + $startInfo.UseShellExecute = $false + $startInfo.CreateNoWindow = $true + $startInfo.RedirectStandardOutput = $true + $startInfo.RedirectStandardError = $true + $startInfo.Environment.Clear() + foreach ($key in $probeEnvironment.Keys) { $startInfo.Environment[[string]$key] = [string]$probeEnvironment[$key] } + $process = [System.Diagnostics.Process]::Start($startInfo) + $stdout = $process.StandardOutput.ReadToEnd() + $stderr = $process.StandardError.ReadToEnd() + if (-not $process.WaitForExit(60000)) { + try { $process.Kill($true) } catch { } + throw 'probe-environment child process exceeded its finite test wait.' + } + return [pscustomobject]@{ ExitCode = $process.ExitCode; Stdout = $stdout; Stderr = $stderr } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-probe-env-' + [Guid]::NewGuid().ToString('N')) +New-Item -ItemType Directory -Path $testRoot -Force | Out-Null +try { + # ===================================================================== + # Scenario 1 (item #7): a model-free probe always has a writable temp and + # never uses %WINDIR% - the exact iteration-11 descriptor/version failure. + # ===================================================================== + + # 1a: the shared resolver returns a writable, non-system directory. + $resolvedRoot = Resolve-RunnerProbeTempRoot + Assert-True (Test-RunnerDirectoryWritable -Path $resolvedRoot) 'resolved probe temp root must be writable' + Assert-False (Test-IsSystemDirectory -Path $resolvedRoot) 'resolved probe temp root must not be a Windows/system directory' + + # 1b: the probe environment pins a writable TEMP and TMP. + $probeEnvironment = New-RunnerProbeEnvironment + Assert-True ($probeEnvironment.Contains('TEMP') -and $probeEnvironment.Contains('TMP')) 'probe environment must set TEMP and TMP' + Assert-Equal ([string]$probeEnvironment['TEMP']) ([string]$probeEnvironment['TMP']) 'probe TEMP and TMP must point at one directory' + Assert-True (Test-RunnerDirectoryWritable -Path ([string]$probeEnvironment['TEMP'])) 'probe TEMP must be writable' + Assert-False (Test-IsSystemDirectory -Path ([string]$probeEnvironment['TEMP'])) 'probe TEMP must not be a Windows/system directory' + + # 1c: elevation-independent rejection. Even when TEMP/TMP are forced to the + # Windows directory (which an elevated user CAN write to), the resolver must + # never return it. This is the guarantee that makes elevation irrelevant. + $windowsDirectory = [Environment]::GetEnvironmentVariable('WINDIR') + if (-not [string]::IsNullOrWhiteSpace($windowsDirectory)) { + $originalTemp = [Environment]::GetEnvironmentVariable('TEMP') + $originalTmp = [Environment]::GetEnvironmentVariable('TMP') + try { + [Environment]::SetEnvironmentVariable('TEMP', $windowsDirectory) + [Environment]::SetEnvironmentVariable('TMP', $windowsDirectory) + $forcedRoot = Resolve-RunnerProbeTempRoot + Assert-False (Test-IsSystemDirectory -Path $forcedRoot) 'resolver must reject a Windows-directory TEMP even when it is writable' + $forcedEnvironment = New-RunnerProbeEnvironment + Assert-False (Test-IsSystemDirectory -Path ([string]$forcedEnvironment['TEMP'])) 'probe environment must not fall back to %WINDIR% when TEMP points at it' + } finally { + [Environment]::SetEnvironmentVariable('TEMP', $originalTemp) + [Environment]::SetEnvironmentVariable('TMP', $originalTmp) + } + } + + # 1d: the exact iteration-11 command. `opencode/runner.ps1 describe` under the + # stripped probe environment must exit 0 and return a valid descriptor. When + # the OpenCode CLI is installed this runs `opencode --version` through the + # probe temp; before the fix it failed with an access-denied to C:\WINDOWS. + $openCodeRunnerPath = Join-Path $runnerRoot 'opencode\runner.ps1' + $describe = Invoke-ProbeEnvironmentChild -ArgumentList @('-NoProfile', '-NonInteractive', '-File', $openCodeRunnerPath, 'describe') + Assert-Equal 0 $describe.ExitCode "opencode describe under the probe environment must exit 0 (stderr: $($describe.Stderr))" + $descriptor = $describe.Stdout | ConvertFrom-Json + Assert-Equal 'opencode' ([string]$descriptor.name) 'opencode describe must return the opencode descriptor' + Assert-False ([string]::IsNullOrWhiteSpace([string]$descriptor.harness.version)) 'opencode describe must report a harness version (real or "unavailable")' + + # 1e: inside a stripped probe child, a real --version probe creates its scratch + # under the writable temp and never touches %WINDIR%. Uses pwsh --version, so + # it is fully model-free. + $childScriptPath = Join-Path $testRoot 'probe-version-child.ps1' + $childScript = @' +param([Parameter(Mandatory = $true)][string]$CommonPath) +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. $CommonPath +$getTempPath = [System.IO.Path]::GetTempPath() +$resolved = Resolve-RunnerProbeTempRoot +$systemDirectories = Get-RunnerSystemDirectorySet +$pwshInfo = Resolve-ExternalCommand -Name 'pwsh' +$version = Get-ExternalCommandVersion -CommandInfo $pwshInfo +[ordered]@{ + process_temp_present = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('TEMP')) + get_temp_path_is_system = $systemDirectories.Contains(([System.IO.Path]::GetFullPath($getTempPath)).TrimEnd([char[]]@('\', '/'))) + resolved_is_system = $systemDirectories.Contains(([System.IO.Path]::GetFullPath($resolved)).TrimEnd([char[]]@('\', '/'))) + version_available = [bool]$version.Available +} | ConvertTo-Json -Compress +'@ + [System.IO.File]::WriteAllText($childScriptPath, $childScript, [System.Text.UTF8Encoding]::new($false)) + $commonPath = (Resolve-Path (Join-Path $runnerRoot 'runner-common.ps1')).Path + $childProbe = Invoke-ProbeEnvironmentChild -ArgumentList @('-NoProfile', '-NonInteractive', '-File', $childScriptPath, '-CommonPath', $commonPath) + Assert-Equal 0 $childProbe.ExitCode "stripped-child version probe must exit 0 (stderr: $($childProbe.Stderr))" + $childSummary = $childProbe.Stdout | ConvertFrom-Json + Assert-True ([bool]$childSummary.process_temp_present) 'probe child must inherit an explicit TEMP from the probe environment' + Assert-False ([bool]$childSummary.get_temp_path_is_system) 'probe child GetTempPath() must not resolve to a system directory' + Assert-False ([bool]$childSummary.resolved_is_system) 'probe child resolver must not return a system directory' + Assert-True ([bool]$childSummary.version_available) 'model-free pwsh --version probe must succeed under the probe environment' + + # ===================================================================== + # Scenario 2 (item #8): the model-free probe environment and the model-backed + # OpenCode eval isolation environment are distinct and must not be merged. + # ===================================================================== + Import-OpenCodeRunnerFunctions + + $separationRoot = Join-Path $testRoot 'separation' + $withoutRoot = Join-Path $separationRoot 'without_skill' + $isolatedHome = Join-Path $withoutRoot 'home' + $isolatedRepo = Join-Path $withoutRoot 'repo' + New-Item -ItemType Directory -Path $isolatedHome, $isolatedRepo -Force | Out-Null + + # Ambient host canary: a real user's OPENCODE_CONFIG must not leak into the + # model-backed eval environment. + $ambientConfigCanary = Join-Path $testRoot 'ambient-user-opencode.json' + [System.IO.File]::WriteAllText($ambientConfigCanary, '{"ambient_user_config":true}', [System.Text.UTF8Encoding]::new($false)) + $originalOpenCodeConfig = [Environment]::GetEnvironmentVariable('OPENCODE_CONFIG') + try { + [Environment]::SetEnvironmentVariable('OPENCODE_CONFIG', $ambientConfigCanary) + + $inputsWithout = [pscustomobject]@{ + Run = [pscustomobject]@{ + HomeDirectoryPath = $isolatedHome + WorkingDirectoryPath = $isolatedRepo + CandidateSkillExposed = $false + SkillDirectoryPath = $null + SkillHash = $null + Contract = [pscustomobject]@{} + } + Profile = [pscustomobject]@{ Model = 'fixture-model'; ReasoningEffort = 'high' } + } + $evalEnvironment = New-OpenCodeEnvironment -Inputs $inputsWithout + + # The eval environment identifies the isolated home, not the host profile. + Assert-True ($evalEnvironment.Contains('HOME')) 'eval environment must set HOME' + Assert-True (Test-PathInside -BasePath $isolatedHome -CandidatePath ([string]$evalEnvironment['HOME'])) 'eval HOME must be inside the isolated run home' + Assert-True (Test-PathInside -BasePath $isolatedHome -CandidatePath ([string]$evalEnvironment['USERPROFILE'])) 'eval USERPROFILE must be inside the isolated run home' + Assert-True (Test-PathInside -BasePath $isolatedHome -CandidatePath ([string]$evalEnvironment['OPENCODE_CONFIG'])) 'eval OPENCODE_CONFIG must be inside the isolated run home' + Assert-False (Test-OpenCodePathEqual -Expected $ambientConfigCanary -Observed ([string]$evalEnvironment['OPENCODE_CONFIG'])) 'eval OPENCODE_CONFIG must not be the ambient host config' + Assert-Equal '1' ([string]$evalEnvironment['OPENCODE_DISABLE_EXTERNAL_SKILLS']) 'eval environment must disable external skill scans' + Assert-Equal '1' ([string]$evalEnvironment['OPENCODE_DISABLE_CLAUDE_CODE_SKILLS']) 'eval environment must disable Claude-compatible skill scans' + Assert-True (Test-PathInside -BasePath $isolatedHome -CandidatePath ([string]$evalEnvironment['TEMP'])) 'eval TEMP must be inside the isolated run home' + Assert-False ($evalEnvironment.Contains('NODE_PATH')) 'eval environment must not carry NODE_PATH' + + $withoutConfig = [System.IO.File]::ReadAllText([string]$evalEnvironment['OPENCODE_CONFIG'], [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json + Assert-Equal 'deny' ([string]$withoutConfig.permission.skill) 'without_skill eval config must deny all skills' + + # The model-free probe environment is NOT the eval environment. + $probeEnvironmentForSeparation = New-RunnerProbeEnvironment + Assert-False ($probeEnvironmentForSeparation.Contains('HOME')) 'probe environment must not carry HOME' + Assert-False ($probeEnvironmentForSeparation.Contains('USERPROFILE')) 'probe environment must not carry USERPROFILE' + Assert-False ($probeEnvironmentForSeparation.Contains('XDG_CONFIG_HOME')) 'probe environment must not carry XDG_CONFIG_HOME' + Assert-False ($probeEnvironmentForSeparation.Contains('OPENCODE_CONFIG')) 'probe environment must not carry OPENCODE_CONFIG' + Assert-False ($probeEnvironmentForSeparation.Contains('OPENCODE_DISABLE_EXTERNAL_SKILLS')) 'probe environment must not carry the eval skill-scan policy' + Assert-True ($probeEnvironmentForSeparation.Contains('TEMP')) 'probe environment must carry OS scratch TEMP' + Assert-False (Test-PathInside -BasePath $isolatedHome -CandidatePath ([string]$probeEnvironmentForSeparation['TEMP'])) 'probe TEMP is OS scratch, not the isolated home' + Assert-False (Test-OpenCodePathEqual -Expected ([string]$evalEnvironment['TEMP']) -Observed ([string]$probeEnvironmentForSeparation['TEMP'])) 'probe temp and eval temp must be distinct locations' + } finally { + [Environment]::SetEnvironmentVariable('OPENCODE_CONFIG', $originalOpenCodeConfig) + } + + # with_skill exposes ONLY the prepared candidate and denies everything else. + $withRoot = Join-Path $separationRoot 'with_skill' + $withHome = Join-Path $withRoot 'home' + $withRepo = Join-Path $withRoot 'repo' + $stagedCandidate = Join-Path (Join-Path $withRoot 'skill') 'dotnet-strong-name-signing' + New-Item -ItemType Directory -Path $withHome, $withRepo, $stagedCandidate -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $stagedCandidate 'SKILL.md'), '# candidate', [System.Text.UTF8Encoding]::new($false)) + $inputsWith = [pscustomobject]@{ + Run = [pscustomobject]@{ + HomeDirectoryPath = $withHome + WorkingDirectoryPath = $withRepo + CandidateSkillExposed = $true + SkillDirectoryPath = $stagedCandidate + SkillHash = ('a' * 64) + Contract = [pscustomobject]@{ skillName = 'dotnet-strong-name-signing' } + } + Profile = [pscustomobject]@{ Model = 'fixture-model'; ReasoningEffort = 'high' } + } + $evalEnvironmentWith = New-OpenCodeEnvironment -Inputs $inputsWith + Assert-Equal 'dotnet-strong-name-signing' (Get-OpenCodeCandidateSkillName -Run $inputsWith.Run) 'candidate skill name resolves from the staged package' + $withConfig = [System.IO.File]::ReadAllText([string]$evalEnvironmentWith['OPENCODE_CONFIG'], [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json + Assert-Equal 'deny' ([string]$withConfig.permission.skill.'*') 'with_skill eval config must deny all skills by default' + Assert-Equal 'allow' ([string]$withConfig.permission.skill.'dotnet-strong-name-signing') 'with_skill eval config must allow only the prepared candidate' + + Write-Output 'Probe environment regression: PASS' +} finally { + if (Test-Path -LiteralPath $testRoot) { + Remove-Item -LiteralPath $testRoot -Recurse -Force -ErrorAction SilentlyContinue + } +} diff --git a/scripts/eval-runners/tests/test-report-utf8.ps1 b/scripts/eval-runners/tests/test-report-utf8.ps1 new file mode 100644 index 0000000..4d8580d --- /dev/null +++ b/scripts/eval-runners/tests/test-report-utf8.ps1 @@ -0,0 +1,144 @@ +<# +.SYNOPSIS + Windows UTF-8 report-generation regression (model-free). + +.DESCRIPTION + OpenCode's forensic iteration-5 exposed a real Windows encoding problem: the + packaged upstream Anthropic skill-creator viewer reads model output and + writes the static HTML report using the platform default encoding (cp1252 on + Windows), so non-ASCII evidence is silently corrupted (and, for characters + outside cp1252, generation can fail outright). + + generate-eval-report.ps1 fixes this centrally by forcing CPython UTF-8 Mode + (PYTHONUTF8=1 / PYTHONIOENCODING=utf-8) for every upstream Python invocation, + WITHOUT modifying any packaged upstream Python source. This test proves the + fix against the real, unmodified upstream generate_review.py: with UTF-8 Mode + the generated report embeds the correct code points (\u2615, \u65e5); on + Windows without it, the same run corrupts them. It never invokes a model. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +$repoScriptsRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..')).Path +$reportScriptPath = Join-Path $repoScriptsRoot 'generate-eval-report.ps1' +$reportScript = [System.IO.File]::ReadAllText($reportScriptPath, [System.Text.UTF8Encoding]::new($false)) + +# 1. The fix must be wired into the shared report invocation, not into upstream. +Assert-True ($reportScript -match "PYTHONUTF8\s*=\s*'1'") 'generate-eval-report.ps1 forces CPython UTF-8 Mode for upstream Python tooling' +Assert-True ($reportScript -match "PYTHONIOENCODING\s*=\s*'utf-8'") 'generate-eval-report.ps1 forces UTF-8 Python stdio encoding' + +function Resolve-PythonCommand { + foreach ($name in @('python', 'py')) { + $command = Get-Command $name -CommandType Application -ErrorAction SilentlyContinue | Select-Object -First 1 + if ($null -ne $command) { return [string]$command.Source } + } + return $null +} + +function Resolve-SkillCreatorPath { + $candidates = [System.Collections.Generic.List[string]]::new() + if (-not [string]::IsNullOrWhiteSpace($env:SKILL_CREATOR_PATH)) { $candidates.Add($env:SKILL_CREATOR_PATH) } + if (-not [string]::IsNullOrWhiteSpace($env:USERPROFILE)) { + $candidates.Add((Join-Path $env:USERPROFILE '.agents/skills/skill-creator')) + $candidates.Add((Join-Path $env:USERPROFILE '.claude/skills/skill-creator')) + $candidates.Add((Join-Path $env:USERPROFILE '.gemini/antigravity-cli/skills/skill-creator')) + } + if (-not [string]::IsNullOrWhiteSpace($env:HOME)) { + $candidates.Add((Join-Path $env:HOME '.agents/skills/skill-creator')) + $candidates.Add((Join-Path $env:HOME '.claude/skills/skill-creator')) + } + foreach ($candidate in $candidates) { + if (-not [string]::IsNullOrWhiteSpace($candidate) -and (Test-Path -LiteralPath (Join-Path $candidate 'eval-viewer/generate_review.py'))) { + return (Resolve-Path -LiteralPath $candidate).Path + } + } + return $null +} + +$pythonCommand = Resolve-PythonCommand +$skillCreatorPath = Resolve-SkillCreatorPath +if ($null -eq $pythonCommand -or $null -eq $skillCreatorPath) { + Write-Output 'Report UTF-8 regression: SKIP (python or upstream skill-creator viewer unavailable in this environment)' + exit 0 +} + +$viewerPath = Join-Path $skillCreatorPath 'eval-viewer/generate_review.py' +$viewerBytesBefore = [System.IO.File]::ReadAllBytes($viewerPath) +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) + +# A coffee glyph (U+2615) and a CJK ideograph (U+65E5) are both outside cp1252; +# with correct UTF-8 reading the upstream viewer embeds them as \u2615 / \u65e5. +$coffee = [char]0x2615 +$sun = [char]0x65E5 +$marker = "MARKER cafe $coffee $sun done" + +$workspaceRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-report-utf8-' + [Guid]::NewGuid().ToString('N')) +try { + $runDirectory = Join-Path $workspaceRoot 'eval-1\with_skill\run-1' + $outputsDirectory = Join-Path $runDirectory 'outputs' + New-Item -ItemType Directory -Path $outputsDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'eval_metadata.json'), '{"prompt":"probe prompt","eval_id":1}', $utf8NoBom) + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'grading.json'), '{"expectations":[{"text":"t","passed":true,"evidence":"e"}],"summary":{"passed":1,"failed":0,"total":1}}', $utf8NoBom) + [System.IO.File]::WriteAllText((Join-Path $outputsDirectory 'output.md'), $marker, $utf8NoBom) + + function Invoke-Viewer { + param([string]$Label, [bool]$Utf8Mode) + + $previousUtf8 = [Environment]::GetEnvironmentVariable('PYTHONUTF8') + $previousIo = [Environment]::GetEnvironmentVariable('PYTHONIOENCODING') + if ($Utf8Mode) { + $env:PYTHONUTF8 = '1' + $env:PYTHONIOENCODING = 'utf-8' + } else { + Remove-Item Env:PYTHONUTF8 -ErrorAction SilentlyContinue + Remove-Item Env:PYTHONIOENCODING -ErrorAction SilentlyContinue + } + $outputHtml = Join-Path $workspaceRoot "report-$Label.html" + try { + $viewerOutput = & $pythonCommand $viewerPath $workspaceRoot '--skill-name' 'report-utf8-probe' '--static' $outputHtml 2>&1 + $exit = $LASTEXITCODE + } finally { + if ($null -eq $previousUtf8) { Remove-Item Env:PYTHONUTF8 -ErrorAction SilentlyContinue } else { $env:PYTHONUTF8 = $previousUtf8 } + if ($null -eq $previousIo) { Remove-Item Env:PYTHONIOENCODING -ErrorAction SilentlyContinue } else { $env:PYTHONIOENCODING = $previousIo } + } + return [pscustomobject]@{ + ExitCode = $exit + Output = [string]::Join([Environment]::NewLine, @($viewerOutput)) + Html = if (Test-Path -LiteralPath $outputHtml -PathType Leaf) { [System.IO.File]::ReadAllText($outputHtml, $utf8NoBom) } else { $null } + } + } + + # With the fix, the unmodified upstream viewer must succeed and embed the + # correct code points for the non-ASCII evidence. + $withFix = Invoke-Viewer -Label 'withfix' -Utf8Mode $true + Assert-True ($withFix.ExitCode -eq 0) "upstream viewer succeeds under UTF-8 Mode: $($withFix.Output)" + Assert-True (-not [string]::IsNullOrWhiteSpace($withFix.Html)) 'upstream viewer writes a non-empty static report under UTF-8 Mode' + Assert-True ($withFix.Html.Contains('\u2615')) 'UTF-8 Mode report preserves the coffee code point (U+2615) as correct evidence' + Assert-True ($withFix.Html.Contains('\u65e5')) 'UTF-8 Mode report preserves the CJK code point (U+65E5) as correct evidence' + + # On Windows the default is cp1252, so the same unmodified viewer WITHOUT the + # fix corrupts the evidence: the correct code points do not appear. This + # proves the central fix is necessary. Non-Windows defaults are already + # UTF-8, so only assert the necessity where the platform default differs. + if ($IsWindows) { + $withoutFix = Invoke-Viewer -Label 'nofix' -Utf8Mode $false + Assert-True (-not ($withoutFix.Html -and $withoutFix.Html.Contains('\u2615'))) 'without UTF-8 Mode the Windows default (cp1252) corrupts non-ASCII evidence, proving the fix is required' + } + + # The packaged upstream Python source must be byte-identical: the fix is in + # the invocation, never in upstream files. + $viewerBytesAfter = [System.IO.File]::ReadAllBytes($viewerPath) + Assert-True ([System.Linq.Enumerable]::SequenceEqual([byte[]]$viewerBytesBefore, [byte[]]$viewerBytesAfter)) 'upstream generate_review.py remains byte-identical; the fix never patches upstream Python' + + Write-Output 'Report UTF-8 regression: PASS' +} finally { + if (Test-Path -LiteralPath $workspaceRoot) { Remove-Item -LiteralPath $workspaceRoot -Recurse -Force -ErrorAction SilentlyContinue } +} diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 new file mode 100644 index 0000000..c55e48e --- /dev/null +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -0,0 +1,2397 @@ +<#! +.SYNOPSIS + Deterministic conformance suite for the common Eval Runner protocol. + +.DESCRIPTION + Creates ephemeral packages under the system temp directory, invokes the + deterministic fake runner and recorded fake CLI processes, and checks the + contracts and recorded event fixtures. It never invokes a real harness or + a live model. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..\..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'manifest-paths.ps1') +. (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'package-integrity.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Get-OpenCodeRunnerAst { + $tokens = $null + $errors = $null + $runnerPath = Join-Path $runnerRoot 'opencode\runner.ps1' + $ast = [System.Management.Automation.Language.Parser]::ParseFile($runnerPath, [ref]$tokens, [ref]$errors) + if ($errors.Count -gt 0) { throw 'OpenCode runner regression fixture could not parse runner.ps1.' } + return $ast +} + +function New-TestChildEnvironment { + param([Parameter(Mandatory = $true)][string]$HomePath) + + $environment = [ordered]@{} + foreach ($variable in @(Get-ChildItem Env:)) { $environment[$variable.Name] = [string]$variable.Value } + $homeFullPath = [System.IO.Path]::GetFullPath($HomePath) + $homeDrive = [System.IO.Path]::GetPathRoot($homeFullPath).TrimEnd('\') + $homePathPart = $homeFullPath.Substring($homeDrive.Length) + $environment['HOME'] = $homeFullPath + $environment['USERPROFILE'] = $homeFullPath + $environment['HOMEDRIVE'] = $homeDrive + $environment['HOMEPATH'] = if ([string]::IsNullOrWhiteSpace($homePathPart)) { '\' } else { $homePathPart } + $environment['APPDATA'] = Join-Path $homeFullPath 'appdata' + $environment['LOCALAPPDATA'] = Join-Path $homeFullPath 'localappdata' + $environment['XDG_CONFIG_HOME'] = Join-Path $homeFullPath '.config' + $environment['XDG_DATA_HOME'] = Join-Path $homeFullPath '.local\share' + $environment['XDG_CACHE_HOME'] = Join-Path $homeFullPath '.cache' + foreach ($directory in @($environment['APPDATA'], $environment['LOCALAPPDATA'], $environment['XDG_CONFIG_HOME'], $environment['XDG_DATA_HOME'], $environment['XDG_CACHE_HOME'])) { + New-Item -ItemType Directory -Path $directory -Force | Out-Null + } + return $environment +} + +function Invoke-GeneratedRunnerPrompt { + $preparePath = Join-Path $repoRoot 'scripts\prepare-skill-evals.ps1' + $tokens = $null + $errors = $null + $ast = [System.Management.Automation.Language.Parser]::ParseFile($preparePath, [ref]$tokens, [ref]$errors) + if ($errors.Count -gt 0) { throw 'Generated handoff regression could not parse prepare-skill-evals.ps1.' } + $functionAst = @($ast.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'New-RunnerPrompt' }, $true) | Select-Object -First 1) + if ($functionAst.Count -ne 1) { throw 'Generated handoff function New-RunnerPrompt was not found.' } + $evalRunnerToolRelativePath = 'tools/eval-runners' + Invoke-Expression $functionAst[0].Extent.Text + $selection = [pscustomobject]@{ Harness = 'OpenCode CLI'; Runner = 'opencode'; Model = 'opencode/muse-spark-1.2-contributor-free'; Preset = '' } + $generatedRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-generated-handoff-' + [Guid]::NewGuid().ToString('N')) + try { + $metadataPath = Join-Path $generatedRoot 'eval-01\eval-metadata.json' + Write-TestJson -Path $metadataPath -Value ([ordered]@{ + interaction = [ordered]@{ turns = @( + [ordered]@{ role = 'user'; content = 'first' } + [ordered]@{ role = 'user'; content = 'second' } + ) } + }) + $manifestEval = [pscustomobject]@{ metadata = 'eval-01/eval-metadata.json' } + return New-RunnerPrompt -IterationDirectory $generatedRoot -IterationNumber 1 -ManifestEvals @($manifestEval) -ExecutionSelection $selection -RequestedConcurrency 2 -PerArmTimeoutSeconds 30 + } finally { + if (Test-Path -LiteralPath $generatedRoot) { Remove-Item -LiteralPath $generatedRoot -Recurse -Force -ErrorAction SilentlyContinue } + } +} + +function Invoke-AdapterJson { + param( + [Parameter(Mandatory = $true)][string]$RunnerPath, + [Parameter(Mandatory = $true)][string]$Command, + [Parameter(Mandatory = $true)][string]$RunPath, + [Parameter(Mandatory = $true)][string]$ProfilePath + ) + + $output = & pwsh -NoProfile -File $RunnerPath $Command -Run $RunPath -Profile $ProfilePath + if ($LASTEXITCODE -ne 0) { throw "Recorded runner '$Command' failed for '$RunnerPath': $([string]::Join(' ', @($output)))" } + $json = [string]::Join([Environment]::NewLine, @($output)) + if ([string]::IsNullOrWhiteSpace($json)) { throw "Recorded runner '$Command' returned no JSON for '$RunnerPath'." } + return $json | ConvertFrom-Json +} + +function Invoke-RecordedRunnerTests { + $recordedRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-recorded-runner-' + [Guid]::NewGuid().ToString('N')) +$recordedOldPath = $env:PATH +$recordedOldOpenAi = $env:OPENAI_API_KEY +$recordedOldCodexHome = $env:CODEX_HOME +$recordedOldGlobalSecret = $env:AGENTIC_GLOBAL_SECRET +$recordedOldProjectDisable = $env:OPENCODE_DISABLE_PROJECT_CONFIG +$recordedOldCopilotToken = $env:COPILOT_GITHUB_TOKEN +$recordedOldGhToken = $env:GH_TOKEN +$recordedOldGithubToken = $env:GITHUB_TOKEN +$recordedOldCopilotHome = $env:COPILOT_HOME +$recordedOldGhConfigDir = $env:GH_CONFIG_DIR +$recordedOldFixtures = $env:AGENTIC_RECORDED_FIXTURES +try { + $fakeBin = Join-Path $recordedRoot 'bin' + New-Item -ItemType Directory -Path $fakeBin -Force | Out-Null + $fakeAmbientUserRoot = Join-Path $fakeBin '.fake-user' + $fakeAmbientCandidateRoots = @( + (Join-Path $fakeAmbientUserRoot '.agents\skills\dotnet-strong-name-signing'), + (Join-Path $fakeAmbientUserRoot '.claude\skills\dotnet-strong-name-signing'), + (Join-Path $fakeAmbientUserRoot '.config\opencode\skills\dotnet-strong-name-signing') + ) + foreach ($fakeAmbientCandidateRoot in $fakeAmbientCandidateRoots) { + New-Item -ItemType Directory -Path $fakeAmbientCandidateRoot -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $fakeAmbientCandidateRoot 'SKILL.md'), 'CODEBELT_OPENCODE_GLOBAL_SKILL_LEAK_CANARY_8F43D1A7`nfake ambient fact: this value is not in the task prompt.', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $fakeAmbientCandidateRoot 'FORMS.md'), 'CODEBELT_OPENCODE_GLOBAL_SKILL_LEAK_CANARY_8F43D1A7`nfake ambient form fact.', [System.Text.UTF8Encoding]::new($false)) + } + Assert-True (Test-Path -LiteralPath (Join-Path $fakeAmbientUserRoot '.agents\skills\dotnet-strong-name-signing\SKILL.md') -PathType Leaf) 'OpenCode fake .agents ambient candidate fixture exists' + Assert-True (Test-Path -LiteralPath (Join-Path $fakeAmbientUserRoot '.claude\skills\dotnet-strong-name-signing\FORMS.md') -PathType Leaf) 'OpenCode fake .claude ambient candidate FORMS.md fixture exists' + Assert-True (Test-Path -LiteralPath (Join-Path $fakeAmbientUserRoot '.config\opencode\skills\dotnet-strong-name-signing\SKILL.md') -PathType Leaf) 'OpenCode fake native global ambient candidate fixture exists' + # The OpenCode isolation regression deliberately places a candidate skill, + # FORMS.md, and project instructions in the source-repository ancestry. + # A logical run under this directory would expose them; a physical + # projection outside it must not. + New-Item -ItemType Directory -Path (Join-Path $recordedRoot '.git'), (Join-Path $recordedRoot 'skills\dotnet-strong-name-signing'), (Join-Path $recordedRoot '.github') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $recordedRoot 'skills\dotnet-strong-name-signing\SKILL.md'), 'CODEBELT_BASELINE_LEAK_CANARY_7C9E4AF2', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $recordedRoot 'skills\dotnet-strong-name-signing\FORMS.md'), 'CODEBELT_BASELINE_FORMS_CANARY_7C9E4AF2', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $recordedRoot 'AGENTS.md'), 'CODEBELT_SOURCE_ANCESTOR_AGENTS_CANARY_7C9E4AF2', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $recordedRoot '.github\copilot-instructions.md'), 'CODEBELT_SOURCE_ANCESTOR_COPILOT_CANARY_7C9E4AF2', [System.Text.UTF8Encoding]::new($false)) + Assert-True (Test-Path -LiteralPath (Join-Path $recordedRoot 'skills\dotnet-strong-name-signing\SKILL.md') -PathType Leaf) 'OpenCode baseline canary source skill fixture exists' + Assert-True (Test-Path -LiteralPath (Join-Path $recordedRoot 'skills\dotnet-strong-name-signing\FORMS.md') -PathType Leaf) 'OpenCode baseline canary FORMS.md fixture exists' + $recordedFixtureRoot = (Resolve-Path (Join-Path $PSScriptRoot 'fixtures')).Path + $env:AGENTIC_RECORDED_FIXTURES = $recordedFixtureRoot + Copy-Item -LiteralPath $recordedFixtureRoot -Destination (Join-Path $fakeBin 'fixtures') -Recurse -Force + $recordedIteration = Join-Path $recordedRoot 'iteration-1' + New-Item -ItemType Directory -Path $recordedIteration -Force | Out-Null + $with = New-TestRun -IterationDirectory $recordedIteration -Configuration with_skill + $without = New-TestRun -IterationDirectory $recordedIteration -Configuration without_skill + [System.IO.File]::WriteAllText((Join-Path $with.Root 'repo\opencode.json'), '{"fixture_project_config":true}', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $without.Root 'repo\opencode.json'), '{"fixture_project_config":true}', [System.Text.UTF8Encoding]::new($false)) + Assert-Equal (Get-TestTreeHash -Root (Join-Path $with.Root 'repo')) (Get-TestTreeHash -Root (Join-Path $without.Root 'repo')) 'OpenCode paired logical fixture repositories remain byte-identical' + $fakeCli = @' +param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) +$harness = [System.IO.Path]::GetFileNameWithoutExtension($MyInvocation.MyCommand.Path) +$logPath = Join-Path (Get-Location).Path ("{0}-fake-cli-log.jsonl" -f $harness) +$arguments = @($RemainingArguments | ForEach-Object { [string]$_ }) +$fixtureRoot = [Environment]::GetEnvironmentVariable('AGENTIC_RECORDED_FIXTURES') +if ([string]::IsNullOrWhiteSpace($fixtureRoot)) { $fixtureRoot = Join-Path (Split-Path -Parent $MyInvocation.MyCommand.Path) 'fixtures' } +$fixtureHome = [Environment]::GetEnvironmentVariable('HOME') +$fakeAmbientUserRoot = Join-Path (Split-Path -Parent $MyInvocation.MyCommand.Path) '.fake-user' +function Test-FixtureMarker { + param([Parameter(Mandatory = $true)][string]$Name) + if ([string]::IsNullOrWhiteSpace($fixtureHome)) { return $false } + return Test-Path -LiteralPath (Join-Path $fixtureHome $Name) -PathType Leaf +} +$scriptedFixture = Test-FixtureMarker -Name 'scripted-session-fixture' +$exactSessionHelpFixture = Test-FixtureMarker -Name ("{0}-exact-session-help" -f $harness) +$noExactSessionHelpFixture = Test-FixtureMarker -Name ("{0}-no-exact-session-help" -f $harness) +$timingFixture = Test-FixtureMarker -Name ("{0}-timing-fixture" -f $harness) +$fakeHomeFixture = Test-FixtureMarker -Name 'opencode-fake-home' +$noSessionFirstFixture = Test-FixtureMarker -Name 'scripted-no-session-first' +$noTerminalFirstFixture = Test-FixtureMarker -Name 'scripted-no-terminal-first' +$mismatchSessionFixture = Test-FixtureMarker -Name 'scripted-session-mismatch' +if ($arguments -contains '--version') { + $version = switch ($harness) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-unknown 9.3' } } + if ($timingFixture) { Start-Sleep -Milliseconds 20 } + [IO.File]::AppendAllText($logPath, (([ordered]@{ invocation_kind = 'version_probe'; args = $arguments } | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output $version + exit 0 +} +if ($arguments -contains '--help' -and -not ($harness -eq 'codex' -and $arguments -contains 'app-server') -and -not ($harness -eq 'opencode' -and $arguments -contains 'debug')) { + $help = switch ($harness) { + 'codex' { '--ask-for-approval never --ephemeral --ignore-user-config --ignore-rules --json --output-last-message --sandbox --cd --model --config --approve-for-me' } + 'opencode' { + if ($noExactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { "--format json`n--dir `n--model `n--auto`n--variant `n--continue" } + else { "--format json`n--dir `n--model `n--auto`n--variant `n--session continue by session id" } + } + 'copilot' { + if ($exactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { [IO.File]::ReadAllText((Join-Path $fixtureRoot 'copilot-help-exact-session.txt'), [Text.UTF8Encoding]::new($false)) } + elseif ($noExactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { [IO.File]::ReadAllText((Join-Path $fixtureRoot 'copilot-help-no-exact-session.txt'), [Text.UTF8Encoding]::new($false)) } + else { '--prompt --output-format --model --allow-all-tools --no-ask-user --no-custom-instructions --disable-builtin-mcps --no-color --log-level --secret-env-vars --no-auto-update -C --resume --continue --session-id --connect --yolo --allow-all --allow-all-paths --allow-all-urls' } + } + default { '--json --auto-approve --cwd --config --data-dir --hooks-dir --provider --model --thinking --timeout --retries --id' } + } + if ($timingFixture) { Start-Sleep -Milliseconds 30 } + [IO.File]::AppendAllText($logPath, (([ordered]@{ invocation_kind = 'help_probe'; args = $arguments } | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output $help + exit 0 +} +$continuationFlag = $null +foreach ($candidate in @('--resume', '--session-id', '--session')) { + if ($arguments -contains $candidate -or @($arguments | Where-Object { [string]$_ -like ($candidate + '=*') }).Count -gt 0) { + $continuationFlag = $candidate + break + } +} +$continuationSessionId = $null +if (-not [string]::IsNullOrWhiteSpace([string]$continuationFlag)) { + $continuationIndex = [Array]::IndexOf([string[]]$arguments, [string]$continuationFlag) + if ($continuationIndex -ge 0 -and $continuationIndex + 1 -lt $arguments.Count -and $arguments[$continuationIndex + 1] -notmatch '^--') { + $continuationSessionId = [string]$arguments[$continuationIndex + 1] + } else { + $continuationAssignment = @($arguments | Where-Object { $_ -like (([string]$continuationFlag) + '=*') } | Select-Object -First 1) + if ($continuationAssignment.Count -eq 1) { $continuationSessionId = [string]$continuationAssignment[0].Substring(([string]$continuationFlag).Length + 1) } + } +} +$authNames = @('OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'GEMINI_API_KEY', 'OPENROUTER_API_KEY', 'XAI_API_KEY', 'MISTRAL_API_KEY') +$authPresent = @($authNames | Where-Object { -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($_)) }) +$copilotAuthNames = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') +$copilotAuthPresent = @($copilotAuthNames | Where-Object { -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($_)) }) +$copilotHome = [Environment]::GetEnvironmentVariable('COPILOT_HOME') +$repositoryAgentsPath = Join-Path (Get-Location).Path 'AGENTS.md' +$repositoryCopilotInstructionsPath = Join-Path (Get-Location).Path '.github\copilot-instructions.md' +$candidateSkillPath = if ($harness -eq 'opencode') { Join-Path (Get-Location).Path '.opencode\skills\candidate' } else { Join-Path (Split-Path -Parent (Get-Location).Path) 'skill' } +$sourceAncestorCanaryVisible = $false +$sourceFormsCanaryVisible = $false +$sourceAncestorAgentsVisible = $false +$sourceAncestorCopilotVisible = $false +$stagedCandidateSkillVisible = Test-Path -LiteralPath $candidateSkillPath -PathType Container +$stagedCandidateSkillHash = $null +$candidatePathInArguments = $false +$candidatePathInEnvironment = $false +function Test-CanaryUnder { + param([Parameter(Mandatory = $true)][string]$Root) + + if (-not (Test-Path -LiteralPath $Root -PathType Container)) { return $false } + foreach ($file in @(Get-ChildItem -LiteralPath $Root -Recurse -File -Force -ErrorAction SilentlyContinue)) { + $text = [IO.File]::ReadAllText($file.FullName, [Text.UTF8Encoding]::new($false)) + if ($text -match 'CODEBELT_BASELINE_LEAK_CANARY_|CODEBELT_BASELINE_FORMS_CANARY_|CODEBELT_OPENCODE_GLOBAL_SKILL_LEAK_CANARY_') { return $true } + } + return $false +} +$homeRoots = @([Environment]::GetEnvironmentVariable('HOME'), [Environment]::GetEnvironmentVariable('USERPROFILE')) | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) } | Select-Object -Unique +$ambientAgentsSkillVisible = @($homeRoots | Where-Object { Test-CanaryUnder -Root (Join-Path ([string]$_) '.agents\skills\dotnet-strong-name-signing') }).Count -gt 0 +$ambientClaudeSkillVisible = @($homeRoots | Where-Object { Test-CanaryUnder -Root (Join-Path ([string]$_) '.claude\skills\dotnet-strong-name-signing') }).Count -gt 0 +$configRoots = @([Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME')) + @($homeRoots | ForEach-Object { Join-Path ([string]$_) '.config' }) | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) } | Select-Object -Unique +$ambientOpenCodeSkillVisible = @($configRoots | Where-Object { Test-CanaryUnder -Root (Join-Path ([string]$_) 'opencode\skills\dotnet-strong-name-signing') }).Count -gt 0 +$fakeAmbientCandidateFixtureVisible = Test-CanaryUnder -Root $fakeAmbientUserRoot +$stagedRepoCanaryVisible = Test-CanaryUnder -Root (Get-Location).Path +$homeCanaryVisible = Test-CanaryUnder -Root $fixtureHome +$configRoot = [Environment]::GetEnvironmentVariable('OPENCODE_CONFIG_DIR') +$configCanaryVisible = if ([string]::IsNullOrWhiteSpace($configRoot)) { $false } else { Test-CanaryUnder -Root $configRoot } +$ancestor = [System.IO.Path]::GetFullPath((Get-Location).Path) +while ($true) { + $ancestorSkillRoot = Join-Path $ancestor 'skills' + if (Test-Path -LiteralPath $ancestorSkillRoot -PathType Container) { + foreach ($ancestorSkillFile in @(Get-ChildItem -LiteralPath $ancestorSkillRoot -Recurse -File -Force -ErrorAction SilentlyContinue)) { + $ancestorSkillText = [IO.File]::ReadAllText($ancestorSkillFile.FullName, [Text.UTF8Encoding]::new($false)) + if ($ancestorSkillText -match 'CODEBELT_BASELINE_LEAK_CANARY_') { $sourceAncestorCanaryVisible = $true } + if ($ancestorSkillText -match 'CODEBELT_BASELINE_FORMS_CANARY_') { $sourceFormsCanaryVisible = $true } + } + } + $ancestorAgentsPath = Join-Path $ancestor 'AGENTS.md' + if ((Test-Path -LiteralPath $ancestorAgentsPath -PathType Leaf) -and ([IO.File]::ReadAllText($ancestorAgentsPath, [Text.UTF8Encoding]::new($false)) -match 'CODEBELT_SOURCE_ANCESTOR_AGENTS_CANARY_')) { $sourceAncestorAgentsVisible = $true } + $ancestorCopilotPath = Join-Path $ancestor '.github\copilot-instructions.md' + if ((Test-Path -LiteralPath $ancestorCopilotPath -PathType Leaf) -and ([IO.File]::ReadAllText($ancestorCopilotPath, [Text.UTF8Encoding]::new($false)) -match 'CODEBELT_SOURCE_ANCESTOR_COPILOT_CANARY_')) { $sourceAncestorCopilotVisible = $true } + $parent = Split-Path -Parent $ancestor + if ([string]::IsNullOrWhiteSpace($parent) -or [string]::Equals($parent, $ancestor, [StringComparison]::OrdinalIgnoreCase)) { break } + $ancestor = $parent +} +if ($stagedCandidateSkillVisible) { + $stagedSkillFiles = @(Get-ChildItem -LiteralPath $candidateSkillPath -Recurse -File -Force -ErrorAction SilentlyContinue) + $stagedCandidateSkillHash = [string]::Join('|', @($stagedSkillFiles | Sort-Object FullName | ForEach-Object { "$( [IO.Path]::GetRelativePath($candidateSkillPath, $_.FullName) ):$( [Convert]::ToHexString(([Security.Cryptography.SHA256]::HashData([IO.File]::ReadAllBytes($_.FullName)))).ToLowerInvariant())" })) +} +$candidatePathInArguments = @($arguments | Where-Object { [string]$_ -match '(?i)CODEBELT_BASELINE_LEAK_CANARY_|CODEBELT_OPENCODE_GLOBAL_SKILL_LEAK_CANARY_|dotnet-strong-name-signing|[\\/]skill[\\/]candidate|[\\/]\.opencode[\\/]skills[\\/]candidate' }).Count -gt 0 +$candidatePathInEnvironment = @(Get-ChildItem Env: | Where-Object { [string]$_.Value -match '(?i)CODEBELT_BASELINE_LEAK_CANARY_|CODEBELT_BASELINE_FORMS_CANARY_|CODEBELT_OPENCODE_GLOBAL_SKILL_LEAK_CANARY_|dotnet-strong-name-signing|[\\/]skill[\\/]candidate|[\\/]\.opencode[\\/]skills[\\/]candidate' }).Count -gt 0 +$candidateCanaryInEnvironment = @(Get-ChildItem Env: | Where-Object { [string]$_.Value -match '(?i)CODEBELT_OPENCODE_GLOBAL_SKILL_LEAK_CANARY_' }).Count -gt 0 +$invocationKind = if ($arguments -contains 'debug' -and $arguments -contains 'config') { 'debug_config_probe' } elseif ($arguments -contains 'debug' -and $arguments -contains 'paths') { 'debug_paths_probe' } elseif ($arguments -contains 'debug' -and $arguments -contains '--help') { 'debug_help_probe' } elseif ($arguments -contains '--version') { 'version_probe' } elseif ($arguments -contains '--help') { 'help_probe' } elseif ($scriptedFixture -and -not [string]::IsNullOrWhiteSpace([string]$continuationSessionId)) { 'explicit_session_resume' } else { 'native_execution' } +$fakeDelayMilliseconds = if (-not $timingFixture) { 0 } else { switch ($invocationKind) { 'version_probe' { 20 } 'help_probe' { 30 } 'explicit_session_resume' { 50 } default { 40 } } } +$repositoryInstructionMarkerVisible = $false +if (Test-Path -LiteralPath $repositoryCopilotInstructionsPath -PathType Leaf) { + $repositoryInstructionMarkerVisible = [IO.File]::ReadAllText($repositoryCopilotInstructionsPath, [Text.UTF8Encoding]::new($false)).Contains('repo-owned-copilot-instruction') +} +$copilotAuthenticationSource = if ($copilotAuthPresent.Count -gt 0) { + 'explicit_environment' +} elseif (-not [string]::IsNullOrWhiteSpace($copilotHome) -and (Test-Path -LiteralPath (Join-Path $copilotHome 'fixture-os-keychain-available') -PathType Leaf)) { + 'os_keychain' +} elseif (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('GH_CONFIG_DIR')) -and (Test-Path -LiteralPath ([Environment]::GetEnvironmentVariable('GH_CONFIG_DIR')) -PathType Container)) { + 'github_cli' +} else { + 'unavailable' +} +$record = [ordered]@{ + args = $arguments + working_directory = (Get-Location).Path + home = [Environment]::GetEnvironmentVariable('HOME') + userprofile = [Environment]::GetEnvironmentVariable('USERPROFILE') + homedrive = [Environment]::GetEnvironmentVariable('HOMEDRIVE') + homepath = [Environment]::GetEnvironmentVariable('HOMEPATH') + appdata = [Environment]::GetEnvironmentVariable('APPDATA') + local_appdata = [Environment]::GetEnvironmentVariable('LOCALAPPDATA') + xdg_config_home = [Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME') + node_path = [Environment]::GetEnvironmentVariable('NODE_PATH') + config_directory = [Environment]::GetEnvironmentVariable('OPENCODE_CONFIG_DIR') + config_file = [Environment]::GetEnvironmentVariable('OPENCODE_CONFIG') + disable_external_skills = [Environment]::GetEnvironmentVariable('OPENCODE_DISABLE_EXTERNAL_SKILLS') + disable_claude_code_skills = [Environment]::GetEnvironmentVariable('OPENCODE_DISABLE_CLAUDE_CODE_SKILLS') + node_homedir = $null + skill_permission = $null + auth_names_present = $authPresent + copilot_auth_names_present = $copilotAuthPresent + copilot_authentication_source = $copilotAuthenticationSource + unrelated_present = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('AGENTIC_GLOBAL_SECRET')) + disable_project_config_present = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('OPENCODE_DISABLE_PROJECT_CONFIG')) + project_config_visible = Test-Path -LiteralPath (Join-Path (Get-Location).Path 'opencode.json') -PathType Leaf + stdin_received = $false + prompt_via_arg = @($arguments | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count -gt 0 + prompt_arg_count = @($arguments | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count + copilot_home = $copilotHome + copilot_cache_home = [Environment]::GetEnvironmentVariable('COPILOT_CACHE_HOME') + gh_config_dir = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') + fixture_mode = if ($scriptedFixture) { 'scripted' } else { 'single_turn' } + continuation_flag = $continuationFlag + continuation_session_id = $continuationSessionId + custom_instructions_disabled = ($arguments -contains '--no-custom-instructions') + builtin_mcps_disabled = ($arguments -contains '--disable-builtin-mcps') + repository_agents_visible = Test-Path -LiteralPath $repositoryAgentsPath -PathType Leaf + repository_copilot_instructions_visible = Test-Path -LiteralPath $repositoryCopilotInstructionsPath -PathType Leaf + repository_instruction_marker_visible = $repositoryInstructionMarkerVisible + candidate_skill_staged = Test-Path -LiteralPath $candidateSkillPath -PathType Container + source_ancestor_candidate_skill_visible = $sourceAncestorCanaryVisible + source_ancestor_forms_visible = $sourceFormsCanaryVisible + source_ancestor_agents_visible = $sourceAncestorAgentsVisible + source_ancestor_copilot_visible = $sourceAncestorCopilotVisible + staged_repo_canary_visible = $stagedRepoCanaryVisible + home_canary_visible = $homeCanaryVisible + config_canary_visible = $configCanaryVisible + fake_ambient_candidate_fixture_visible = $fakeAmbientCandidateFixtureVisible + ambient_agents_skill_visible = $ambientAgentsSkillVisible + ambient_claude_skill_visible = $ambientClaudeSkillVisible + ambient_opencode_skill_visible = $ambientOpenCodeSkillVisible + staged_candidate_skill_visible = $stagedCandidateSkillVisible + staged_candidate_skill_hash = $stagedCandidateSkillHash + candidate_skill_path_in_arguments = $candidatePathInArguments + candidate_skill_path_in_environment = $candidatePathInEnvironment + candidate_canary_in_environment = $candidateCanaryInEnvironment + candidate_skill_exposure = if ($stagedCandidateSkillVisible) { 'included' } else { 'excluded' } + invocation_kind = $invocationKind + fake_delay_milliseconds = $fakeDelayMilliseconds + ambient_copilot_instructions_visible = if ([string]::IsNullOrWhiteSpace($copilotHome)) { $false } else { Test-Path -LiteralPath (Join-Path $copilotHome 'copilot-instructions.md') -PathType Leaf } + secret_env_vars_arg = @($arguments | Where-Object { $_ -like '--secret-env-vars=*' }) +} +if ($harness -eq 'opencode') { + try { + $nodeHomeOutput = @(& node -p 'require("os").homedir()' 2>$null | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) } | Select-Object -First 1) + if ($nodeHomeOutput.Count -eq 1) { $record.node_homedir = ([string]$nodeHomeOutput[0]).Trim() } + } catch { } + if (-not [string]::IsNullOrWhiteSpace([string]$record.config_file) -and (Test-Path -LiteralPath $record.config_file -PathType Leaf)) { + try { + $recordedConfig = [IO.File]::ReadAllText($record.config_file, [Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 50 + $record.skill_permission = $recordedConfig.permission.skill + } catch { } + } +} +if ($harness -eq 'codex' -and $arguments -contains 'app-server' -and $arguments -contains 'generate-json-schema') { + $outArgument = @($arguments | Where-Object { $_ -like '--out=*' } | Select-Object -First 1) + if ($outArgument.Count -eq 0) { exit 2 } + $schemaDirectory = [IO.Path]::GetFullPath((Join-Path (Get-Location).Path ([string]$outArgument[0].Substring(6)))) + New-Item -ItemType Directory -Path $schemaDirectory -Force | Out-Null + foreach ($existingSchemaFile in @(Get-ChildItem -LiteralPath $schemaDirectory -Force -ErrorAction SilentlyContinue)) { + Remove-Item -LiteralPath $existingSchemaFile.FullName -Recurse -Force + } + $schema = 'http://json-schema.org/draft-07/schema#' + $definitions = [ordered]@{ + AbsolutePathBuf = [ordered]@{ type = 'string' } + LegacyAppPathString = [ordered]@{ type = 'string' } + SandboxMode = [ordered]@{ type = 'string'; enum = @('read-only', 'workspace-write', 'danger-full-access') } + AskForApproval = [ordered]@{ oneOf = @([ordered]@{ type = 'string'; enum = @('untrusted', 'on-request', 'never') }) } + ReasoningEffort = [ordered]@{ type = 'string'; minLength = 1 } + ModelRerouteReason = [ordered]@{ type = 'string'; enum = @('highRiskCyberActivity') } + TurnStatus = [ordered]@{ type = 'string'; enum = @('inProgress', 'completed', 'failed', 'interrupted') } + UserInput = [ordered]@{ oneOf = @([ordered]@{ type = 'object'; required = @('text', 'type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('text') }; text = [ordered]@{ type = 'string' } } }) } + SandboxPolicy = [ordered]@{ oneOf = @( + [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('dangerFullAccess') } } } + [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('readOnly') }; networkAccess = [ordered]@{ type = 'boolean' } } } + [ordered]@{ type = 'object'; required = @('type'); properties = [ordered]@{ type = [ordered]@{ type = 'string'; enum = @('workspaceWrite') }; writableRoots = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' } }; networkAccess = [ordered]@{ type = 'boolean' } } } + ) } + Thread = [ordered]@{ type = 'object'; required = @('id', 'cwd', 'ephemeral', 'sessionId', 'turns'); properties = [ordered]@{ id = [ordered]@{ type = 'string' }; cwd = [ordered]@{ allOf = @([ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' }) }; ephemeral = [ordered]@{ type = 'boolean' }; sessionId = [ordered]@{ type = 'string' }; turns = [ordered]@{ type = 'array' } } } + Turn = [ordered]@{ type = 'object'; required = @('id', 'items', 'status'); properties = [ordered]@{ id = [ordered]@{ type = 'string' }; items = [ordered]@{ type = 'array' }; status = [ordered]@{ '$ref' = '#/definitions/TurnStatus' } } } + } + $definitions.ThreadStartParams = [ordered]@{ + '$schema' = $schema + title = 'ThreadStartParams' + type = 'object' + properties = [ordered]@{ + model = [ordered]@{ type = @('string', 'null') } + cwd = [ordered]@{ type = @('string', 'null') } + approvalPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/AskForApproval' }, [ordered]@{ type = 'null' }) } + sandbox = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/SandboxMode' }, [ordered]@{ type = 'null' }) } + ephemeral = [ordered]@{ type = @('boolean', 'null') } + } + } + $definitions.ThreadStartResponse = [ordered]@{ + '$schema' = $schema + title = 'ThreadStartResponse' + type = 'object' + required = @('approvalPolicy', 'approvalsReviewer', 'cwd', 'model', 'modelProvider', 'sandbox', 'thread') + properties = [ordered]@{ + approvalPolicy = [ordered]@{ '$ref' = '#/definitions/AskForApproval' } + cwd = [ordered]@{ '$ref' = '#/definitions/AbsolutePathBuf' } + instructionSources = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/LegacyAppPathString' } } + model = [ordered]@{ type = 'string' } + sandbox = [ordered]@{ allOf = @([ordered]@{ '$ref' = '#/definitions/SandboxPolicy' }) } + thread = [ordered]@{ '$ref' = '#/definitions/Thread' } + } + } + $definitions.TurnStartParams = [ordered]@{ + '$schema' = $schema + title = 'TurnStartParams' + type = 'object' + required = @('input', 'threadId') + properties = [ordered]@{ + threadId = [ordered]@{ type = 'string' } + input = [ordered]@{ type = 'array'; items = [ordered]@{ '$ref' = '#/definitions/UserInput' } } + cwd = [ordered]@{ type = @('string', 'null') } + model = [ordered]@{ type = @('string', 'null') } + effort = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/ReasoningEffort' }, [ordered]@{ type = 'null' }) } + approvalPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/AskForApproval' }, [ordered]@{ type = 'null' }) } + sandboxPolicy = [ordered]@{ anyOf = @([ordered]@{ '$ref' = '#/definitions/SandboxPolicy' }, [ordered]@{ type = 'null' }) } + } + } + $definitions.TurnStartResponse = [ordered]@{ + '$schema' = $schema + title = 'TurnStartResponse' + type = 'object' + required = @('turn') + properties = [ordered]@{ turn = [ordered]@{ '$ref' = '#/definitions/Turn' } } + } + $definitions.ThreadReadParams = [ordered]@{ + '$schema' = $schema + title = 'ThreadReadParams' + type = 'object' + required = @('threadId') + properties = [ordered]@{ threadId = [ordered]@{ type = 'string' }; includeTurns = [ordered]@{ type = 'boolean' } } + } + $definitions.ThreadReadResponse = [ordered]@{ + '$schema' = $schema + title = 'ThreadReadResponse' + type = 'object' + required = @('thread') + properties = [ordered]@{ thread = [ordered]@{ '$ref' = '#/definitions/Thread' } } + } + $definitions.ModelReroutedNotification = [ordered]@{ + '$schema' = $schema + title = 'ModelReroutedNotification' + type = 'object' + required = @('fromModel', 'reason', 'threadId', 'toModel', 'turnId') + properties = [ordered]@{ + fromModel = [ordered]@{ type = 'string' } + reason = [ordered]@{ '$ref' = '#/definitions/ModelRerouteReason' } + threadId = [ordered]@{ type = 'string' } + toModel = [ordered]@{ type = 'string' } + turnId = [ordered]@{ type = 'string' } + } + } + $fixtureHome = [Environment]::GetEnvironmentVariable('HOME') + $schemaMode = if (-not [string]::IsNullOrWhiteSpace($fixtureHome) -and (Test-Path -LiteralPath (Join-Path $fixtureHome 'codex-schema-individual-v2') -PathType Leaf)) { 'individual-v2' } else { '' } + $missingSchema = if (-not [string]::IsNullOrWhiteSpace($fixtureHome) -and (Test-Path -LiteralPath (Join-Path $fixtureHome 'codex-schema-missing-ThreadStartParams') -PathType Leaf)) { 'ThreadStartParams' } else { '' } + $missingRequiredSchemas = -not [string]::IsNullOrWhiteSpace($fixtureHome) -and (Test-Path -LiteralPath (Join-Path $fixtureHome 'codex-schema-missing-required') -PathType Leaf) + $withoutThreadRead = -not [string]::IsNullOrWhiteSpace($fixtureHome) -and (Test-Path -LiteralPath (Join-Path $fixtureHome 'codex-schema-without-thread-read') -PathType Leaf) + if (-not [string]::IsNullOrWhiteSpace($missingSchema)) { [void]$definitions.Remove($missingSchema) } + if ($missingRequiredSchemas) { + [void]$definitions.Remove('ThreadStartParams') + [void]$definitions.Remove('TurnStartResponse') + } + if ($withoutThreadRead) { + [void]$definitions.Remove('ThreadReadParams') + [void]$definitions.Remove('ThreadReadResponse') + } + $schemaFiles = [ordered]@{ + 'codex_app_server_protocol.v2.schemas.json' = [ordered]@{ '$schema' = $schema; title = 'codex_app_server_protocol.v2.schemas'; type = 'object'; definitions = $definitions } + } + foreach ($schemaName in @('ThreadStartParams', 'ThreadStartResponse', 'TurnStartParams', 'TurnStartResponse', 'ThreadReadParams', 'ThreadReadResponse', 'ModelReroutedNotification')) { + if ($definitions.Contains($schemaName)) { + $source = $definitions[$schemaName] + $individual = [ordered]@{ '$schema' = $schema } + foreach ($propertyName in @('title', 'type', 'properties', 'required')) { + if ($source.Contains($propertyName)) { $individual[$propertyName] = $source[$propertyName] } + } + $individual.definitions = $definitions + $schemaFiles[('v2\{0}.json' -f $schemaName)] = $individual + } + } + if ($schemaMode -eq 'individual-v2') { [void]$schemaFiles.Remove('codex_app_server_protocol.v2.schemas.json') } + foreach ($schemaName in $schemaFiles.Keys) { + $schemaPath = Join-Path $schemaDirectory $schemaName + New-Item -ItemType Directory -Path (Split-Path -Parent $schemaPath) -Force | Out-Null + [IO.File]::WriteAllText($schemaPath, ([string]($schemaFiles[$schemaName] | ConvertTo-Json -Depth 100)), [Text.UTF8Encoding]::new($false)) + } + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + exit 0 +} +if ($harness -eq 'codex' -and $arguments -contains 'app-server' -and $arguments -contains '--help') { + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output 'generate-json-schema' + exit 0 +} +if ($harness -eq 'codex' -and $arguments -contains 'features' -and $arguments -contains 'list') { + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output 'multi_agent stable true' + exit 0 +} +if ($harness -eq 'codex' -and $arguments -contains 'app-server') { + function Read-AppServerMessage { + $line = [Console]::In.ReadLine() + if ($null -eq $line) { throw 'recorded app-server reached EOF before the expected request' } + return ($line | ConvertFrom-Json -Depth 50) + } + function Write-AppServerMessage { + param([Parameter(Mandatory = $true)][object]$Value) + [Console]::Out.WriteLine(($Value | ConvertTo-Json -Depth 50 -Compress)) + [Console]::Out.Flush() + } + + $initialize = Read-AppServerMessage + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; id = $initialize.id; result = [ordered]@{ serverInfo = [ordered]@{ name = 'recorded-codex'; version = '9.1' } } }) + $initialized = Read-AppServerMessage + $threadStart = Read-AppServerMessage + $fixtureReroute = Test-Path -LiteralPath (Join-Path ([Environment]::GetEnvironmentVariable('HOME')) 'codex-reroute') -PathType Leaf + $fixtureAmbientInstruction = Test-Path -LiteralPath (Join-Path ([Environment]::GetEnvironmentVariable('HOME')) 'codex-ambient-instruction') -PathType Leaf + $fixtureThreadReadUnavailable = Test-Path -LiteralPath (Join-Path ([Environment]::GetEnvironmentVariable('HOME')) 'codex-thread-read-unavailable') -PathType Leaf + $instructionSources = if ($fixtureAmbientInstruction) { @('C:\ambient\AGENTS.md') } else { @($repositoryAgentsPath) } + $threadObject = [ordered]@{ + id = 'recorded-subscription-thread' + sessionId = 'recorded-subscription-session' + ephemeral = $true + cwd = (Get-Location).Path + cliVersion = '9.1' + createdAt = 1 + updatedAt = 1 + modelProvider = 'recorded-provider' + preview = $false + projectId = $null + source = 'startup' + status = [ordered]@{ type = 'idle' } + turns = @() + } + $threadStartResult = [ordered]@{ + approvalPolicy = 'never' + approvalsReviewer = 'user' + cwd = (Get-Location).Path + model = 'gpt-5.6-luna' + modelProvider = 'recorded-provider' + sandbox = [ordered]@{ type = 'readOnly' } + instructionSources = $instructionSources + thread = $threadObject + } + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; id = $threadStart.id; result = $threadStartResult }) + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'thread/started'; params = [ordered]@{ thread = [ordered]@{ id = 'recorded-subscription-thread' } } }) + $turnStart = Read-AppServerMessage + if ($fixtureReroute) { + # This notification deliberately arrives before turn/start's response + # so the recorded transport proves reroute capture while waiting for a + # JSON-RPC response, not only in the terminal event loop. + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'model/rerouted'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turnId = 'recorded-subscription-turn'; fromModel = 'gpt-5.6-luna'; toModel = 'gpt-5.6-other'; reason = 'highRiskCyberActivity' } }) + } + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; id = $turnStart.id; result = [ordered]@{ turn = [ordered]@{ id = 'recorded-subscription-turn'; status = 'inProgress'; items = @() } } }) + + $promptText = [string]$turnStart.params.input[0].text + $promptBytes = [Text.Encoding]::UTF8.GetBytes($promptText) + $expectedPromptHashPath = Join-Path ([Environment]::GetEnvironmentVariable('HOME')) 'expected-prompt-sha256.txt' + $expectedPromptHash = if (Test-Path -LiteralPath $expectedPromptHashPath -PathType Leaf) { [IO.File]::ReadAllText($expectedPromptHashPath, [Text.UTF8Encoding]::new($false)).Trim() } else { [Convert]::ToHexString(([Security.Cryptography.SHA256]::HashData([byte[]]$promptBytes))).ToLowerInvariant() } + $record.stdin_received = $promptBytes.Length -gt 0 + $record.stdin_delivery_count = if ($promptBytes.Length -gt 0) { 1 } else { 0 } + $record.stdin_byte_length = $promptBytes.Length + $record.stdin_sha256 = [Convert]::ToHexString(([Security.Cryptography.SHA256]::HashData($promptBytes))).ToLowerInvariant() + $record.stdin_expected_sha256 = $expectedPromptHash + $record.stdin_exact = $record.stdin_sha256 -eq $expectedPromptHash + $record.stdin_utf8_round_trip = $record.stdin_exact + $record.worker_provider_visible = $false + $record.worker_copilot_token_visible = $false + $record.worker_gh_token_visible = $false + $record.worker_github_token_visible = $false + $record.worker_auth_file_visible = $false + $record.worker_global_secret_visible = $false + $record.worker_project_disable_visible = $false + $record.parent_codex_home = [Environment]::GetEnvironmentVariable('CODEX_HOME') + $record.parent_auth_file_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'auth.json') -PathType Leaf + $record.parent_config_file_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'config.toml') -PathType Leaf + $record.parent_skills_directory_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'skills') -PathType Container + $record.parent_agents_directory_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'agents') -PathType Container + $record.parent_sessions_directory_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'sessions') -PathType Container + $record.parent_memories_directory_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'memories') -PathType Container + $record.parent_plugins_directory_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'plugins') -PathType Container + $record.parent_mcp_configuration_visible = (Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'mcp.json') -PathType Leaf) -or (Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'mcp') -PathType Container) + $record.parent_agents_file_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'AGENTS.md') -PathType Leaf + $record.auth_only_home = [bool]$record.parent_auth_file_visible -and -not [bool]$record.parent_config_file_visible -and -not [bool]$record.parent_skills_directory_visible -and -not [bool]$record.parent_agents_directory_visible -and -not [bool]$record.parent_sessions_directory_visible -and -not [bool]$record.parent_memories_directory_visible -and -not [bool]$record.parent_plugins_directory_visible -and -not [bool]$record.parent_mcp_configuration_visible -and -not [bool]$record.parent_agents_file_visible + $record.rpc_methods = @($initialize.method, $initialized.method, $threadStart.method, $turnStart.method, 'thread/read') + $record.thread_params = $threadStart.params + $record.turn_params = $turnStart.params + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Depth 50 -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'item/completed'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turnId = 'recorded-subscription-turn'; completedAtMs = 1; item = [ordered]@{ type = 'commandExecution'; id = 'command-1'; command = 'recorded command'; commandActions = @(); cwd = (Get-Location).Path; status = 'completed'; exitCode = 0; aggregatedOutput = 'recorded output' } } }) + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'item/completed'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turnId = 'recorded-subscription-turn'; completedAtMs = 2; item = [ordered]@{ type = 'fileChange'; id = 'file-1'; status = 'completed'; changes = @([ordered]@{ path = 'recorded.txt'; kind = [ordered]@{ type = 'add' } }) } } }) + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'item/completed'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turnId = 'recorded-subscription-turn'; completedAtMs = 3; item = [ordered]@{ type = 'agentMessage'; id = 'message-1'; text = 'recorded subscription response' } } }) + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'thread/tokenUsage/updated'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turnId = 'recorded-subscription-turn'; tokenUsage = [ordered]@{ total = [ordered]@{ inputTokens = 2; cachedInputTokens = 1; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 }; last = [ordered]@{ inputTokens = 2; cachedInputTokens = 1; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 } } } }) + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'turn/completed'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turn = [ordered]@{ id = 'recorded-subscription-turn'; status = 'completed'; items = @() } } }) + $threadRead = Read-AppServerMessage + if (-not $fixtureThreadReadUnavailable) { Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; id = $threadRead.id; result = [ordered]@{ thread = $threadObject } }) } + exit 0 +} +if ($harness -eq 'opencode' -and $arguments -contains 'debug' -and $arguments -contains 'config') { + if ($fakeDelayMilliseconds -gt 0) { Start-Sleep -Milliseconds $fakeDelayMilliseconds } + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + if ([string]::IsNullOrWhiteSpace([string]$record.config_file) -or -not (Test-Path -LiteralPath $record.config_file -PathType Leaf)) { + [Console]::Error.WriteLine('recorded OpenCode debug config has no config file') + exit 31 + } + Write-Output ([IO.File]::ReadAllText($record.config_file, [Text.UTF8Encoding]::new($false))) + exit 0 +} +if ($harness -eq 'opencode' -and $arguments -contains 'debug' -and $arguments -contains 'paths') { + if ($fakeDelayMilliseconds -gt 0) { Start-Sleep -Milliseconds $fakeDelayMilliseconds } + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + $debugHome = if ($fakeHomeFixture) { $fakeAmbientUserRoot } else { [Environment]::GetEnvironmentVariable('HOME') } + Write-Output ('home ' + $debugHome) + Write-Output ('config ' + [Environment]::GetEnvironmentVariable('OPENCODE_CONFIG_DIR')) + exit 0 +} +if ($harness -eq 'opencode' -and $arguments -contains 'debug' -and $arguments -contains '--help') { + if ($fakeDelayMilliseconds -gt 0) { Start-Sleep -Milliseconds $fakeDelayMilliseconds } + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output 'config paths skill' + exit 0 +} +if ($arguments -contains '--version') { + $version = switch ($harness) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-unknown 9.3' } } + if ($fakeDelayMilliseconds -gt 0) { Start-Sleep -Milliseconds $fakeDelayMilliseconds } + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output $version + exit 0 +} +if ($arguments -contains '--help') { + $help = switch ($harness) { + 'codex' { '--ask-for-approval never --ephemeral --ignore-user-config --ignore-rules --json --output-last-message --sandbox --cd --model --config --approve-for-me' } + 'opencode' { + if ($noExactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { "--format json`n--dir `n--model `n--auto`n--variant `n--continue" } + else { "--format json`n--dir `n--model `n--auto`n--variant `n--session continue by session id" } + } + 'copilot' { + if ($exactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { [IO.File]::ReadAllText((Join-Path $fixtureRoot 'copilot-help-exact-session.txt'), [Text.UTF8Encoding]::new($false)) } + elseif ($noExactSessionHelpFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot)) { [IO.File]::ReadAllText((Join-Path $fixtureRoot 'copilot-help-no-exact-session.txt'), [Text.UTF8Encoding]::new($false)) } + else { '--prompt --output-format --model --allow-all-tools --no-ask-user --no-custom-instructions --disable-builtin-mcps --no-color --log-level --secret-env-vars --no-auto-update -C --resume --continue --session-id --connect --yolo --allow-all --allow-all-paths --allow-all-urls' } + } + default { '--json --auto-approve --cwd --config --data-dir --hooks-dir --provider --model --thinking --timeout --retries --id' } + } + if ($fakeDelayMilliseconds -gt 0) { Start-Sleep -Milliseconds $fakeDelayMilliseconds } + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output $help + exit 0 +} +$stdinMemory = [IO.MemoryStream]::new() +[Console]::OpenStandardInput().CopyTo($stdinMemory) +$stdinBytes = $stdinMemory.ToArray() +$stdinHash = [Convert]::ToHexString(([Security.Cryptography.SHA256]::HashData($stdinBytes))).ToLowerInvariant() +$expectedPromptHashPath = Join-Path ([Environment]::GetEnvironmentVariable('HOME')) 'expected-prompt-sha256.txt' +$expectedPromptHash = if ($scriptedFixture) { $stdinHash } elseif (Test-Path -LiteralPath $expectedPromptHashPath -PathType Leaf) { [IO.File]::ReadAllText($expectedPromptHashPath, [Text.UTF8Encoding]::new($false)).Trim() } else { $stdinHash } +$record.stdin_received = $stdinBytes.Length -gt 0 +$record.stdin_delivery_count = if ($stdinBytes.Length -gt 0) { 1 } else { 0 } +$record.stdin_byte_length = $stdinBytes.Length +$record.stdin_sha256 = $stdinHash +$record.stdin_exact = $stdinHash -eq $expectedPromptHash +$record.stdin_expected_sha256 = $expectedPromptHash +$record.stdin_utf8_round_trip = $record.stdin_sha256 -eq $expectedPromptHash +$record.candidate_canary_in_stdin = ([Text.Encoding]::UTF8.GetString($stdinBytes) -match 'CODEBELT_BASELINE_LEAK_CANARY_|CODEBELT_BASELINE_FORMS_CANARY_|CODEBELT_OPENCODE_GLOBAL_SKILL_LEAK_CANARY_') +$probeCommand = '$result = [ordered]@{ provider_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENAI_API_KEY'')); copilot_token_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''COPILOT_GITHUB_TOKEN'')); gh_token_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''GH_TOKEN'')); github_token_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''GITHUB_TOKEN'')); auth_file_visible = Test-Path -LiteralPath (Join-Path ([Environment]::GetEnvironmentVariable(''HOME'')) ''.codex/auth.json''); global_secret_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''AGENTIC_GLOBAL_SECRET'')); project_disable_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENCODE_DISABLE_PROJECT_CONFIG'')) }; $result | ConvertTo-Json -Compress' +$probeInfo = [Diagnostics.ProcessStartInfo]::new() +$probeInfo.FileName = (Get-Command pwsh).Source +$probeInfo.UseShellExecute = $false +$probeInfo.CreateNoWindow = $true +$probeInfo.RedirectStandardOutput = $true +$probeInfo.RedirectStandardError = $true +$probeInfo.WorkingDirectory = (Get-Location).Path +$probeInfo.ArgumentList.Add('-NoProfile') +$probeInfo.ArgumentList.Add('-Command') +$probeInfo.ArgumentList.Add($probeCommand) +$probeInfo.Environment.Clear() +$probeInfo.Environment['PATH'] = [Environment]::GetEnvironmentVariable('PATH') +$probeInfo.Environment['HOME'] = [Environment]::GetEnvironmentVariable('HOME') +$probeInfo.Environment['USERPROFILE'] = [Environment]::GetEnvironmentVariable('USERPROFILE') +if ($harness -ne 'codex') { $probeInfo.Environment['OPENAI_API_KEY'] = [Environment]::GetEnvironmentVariable('OPENAI_API_KEY') } +$probe = [Diagnostics.Process]::new() +$probe.StartInfo = $probeInfo +try { + [void]$probe.Start() + $probeOutput = $probe.StandardOutput.ReadToEnd() + $probeError = $probe.StandardError.ReadToEnd() + if (-not $probe.WaitForExit(5000)) { throw 'worker credential probe exceeded its finite test wait.' } + if ($probe.ExitCode -ne 0) { throw "worker credential probe failed: $probeError" } + $probeResult = $probeOutput | ConvertFrom-Json + $record.worker_provider_visible = [bool]$probeResult.provider_visible + $record.worker_copilot_token_visible = [bool]$probeResult.copilot_token_visible + $record.worker_gh_token_visible = [bool]$probeResult.gh_token_visible + $record.worker_github_token_visible = [bool]$probeResult.github_token_visible + $record.worker_auth_file_visible = [bool]$probeResult.auth_file_visible + $record.worker_global_secret_visible = [bool]$probeResult.global_secret_visible + $record.worker_project_disable_visible = [bool]$probeResult.project_disable_visible +} finally { + $probe.Dispose() +} +if ($fakeDelayMilliseconds -gt 0) { Start-Sleep -Milliseconds $fakeDelayMilliseconds } +[IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) +if ($harness -eq 'copilot' -and $copilotAuthenticationSource -eq 'unavailable') { + [Console]::Error.WriteLine('deterministic fixture: no Copilot authentication mechanism is available') + exit 17 +} +if ($scriptedFixture -and -not [string]::IsNullOrWhiteSpace($fixtureRoot) -and $harness -in @('copilot', 'opencode')) { + $turnNumber = if ([string]::IsNullOrWhiteSpace([string]$continuationSessionId)) { 1 } else { 2 } + $fixturePath = Join-Path $fixtureRoot ("{0}-scripted-turn-{1}-events.jsonl" -f $harness, $turnNumber) + if (-not (Test-Path -LiteralPath $fixturePath -PathType Leaf)) { + [Console]::Error.WriteLine("recorded scripted fixture is missing: $fixturePath") + exit 23 + } + if ($harness -eq 'opencode') { + New-Item -ItemType Directory -Path (Join-Path (Get-Location).Path '.opencode\node_modules') -Force | Out-Null + [IO.File]::WriteAllText((Join-Path (Get-Location).Path '.opencode\node_modules\runtime-only.txt'), 'runtime', [Text.UTF8Encoding]::new($false)) + [IO.File]::WriteAllText((Join-Path (Get-Location).Path '.opencode\.gitignore'), 'runtime', [Text.UTF8Encoding]::new($false)) + [IO.File]::WriteAllText((Join-Path (Get-Location).Path 'normal-task-output.snk'), 'task-output', [Text.UTF8Encoding]::new($false)) + } + $fixtureText = [IO.File]::ReadAllText($fixturePath, [Text.UTF8Encoding]::new($false)) + if ($turnNumber -eq 1 -and $noSessionFirstFixture) { + if ($harness -eq 'copilot') { $fixtureText = $fixtureText.Replace('"sessionId":"fixture-copilot-session"', '"sessionId":null') } + if ($harness -eq 'opencode') { $fixtureText = $fixtureText.Replace('"sessionID":"fixture-opencode-session"', '"sessionID":null') } + } + if ($turnNumber -eq 1 -and $noTerminalFirstFixture) { + $terminalEventPattern = if ($harness -eq 'copilot') { 'session.task_complete' } else { 'step_finish' } + $fixtureText = [string]::Join("`n", @($fixtureText -split "`r?`n" | Where-Object { $_ -notmatch [regex]::Escape($terminalEventPattern) })) + } + if ($turnNumber -gt 1 -and $mismatchSessionFixture) { + if ($harness -eq 'copilot') { $fixtureText = $fixtureText.Replace('fixture-copilot-session', 'fixture-copilot-mismatch') } + if ($harness -eq 'opencode') { $fixtureText = $fixtureText.Replace('fixture-opencode-session', 'fixture-opencode-mismatch') } + } + if ($harness -eq 'opencode' -and ($sourceAncestorCanaryVisible -or $sourceFormsCanaryVisible -or $sourceAncestorAgentsVisible -or $sourceAncestorCopilotVisible -or $ambientAgentsSkillVisible -or $ambientClaudeSkillVisible -or $ambientOpenCodeSkillVisible)) { + $leakMarker = if ($ambientAgentsSkillVisible -or $ambientClaudeSkillVisible -or $ambientOpenCodeSkillVisible) { 'CODEBELT_OPENCODE_GLOBAL_SKILL_LEAK_CANARY_8F43D1A7' } else { 'CODEBELT_BASELINE_LEAK_CANARY_7C9E4AF2' } + Write-Output ('{"type":"text","text":"' + $leakMarker + '"}') + Write-Output '{"type":"step_finish"}' + exit 0 + } + foreach ($fixtureLine in @($fixtureText -split "`r?`n")) { + if (-not [string]::IsNullOrWhiteSpace([string]$fixtureLine)) { Write-Output $fixtureLine } + } + exit 0 +} +if ($harness -eq 'codex') { + $outputIndex = [Array]::IndexOf([string[]]$arguments, '--output-last-message') + if ($outputIndex -ge 0 -and $outputIndex + 1 -lt $arguments.Count) { + $outputPath = $arguments[$outputIndex + 1] + $outputParent = Split-Path -Parent $outputPath + if (-not (Test-Path -LiteralPath $outputParent -PathType Container)) { + [Console]::Error.WriteLine("recorded Codex requires the output parent directory to exist: $outputParent") + exit 19 + } + [IO.File]::WriteAllText($outputPath, 'recorded Codex final response', [Text.UTF8Encoding]::new($false)) + } + Write-Output '{"type":"thread.started","thread_id":"recorded-thread"}' + Write-Output '{"type":"item.completed","item":{"type":"agent_message","text":"recorded Codex final response"}}' + Write-Output '{"type":"turn.completed","usage":{"input_tokens":2,"output_tokens":3}}' + Write-Output '{"type":"future.event.v99","payload":"fixture"}' +} elseif ($harness -eq 'opencode') { + Write-Output '{"type":"step_start","timestamp":"2026-01-01T00:00:01.000Z","sessionID":"recorded-opencode-session","modelID":"opencode/muse-spark-1.2-contributor-free"}' + Write-Output '{"type":"text","timestamp":"2026-01-01T00:00:02.000Z","sessionID":"recorded-opencode-session","modelID":"opencode/muse-spark-1.2-contributor-free","text":"recorded OpenCode final response"}' + Write-Output '{"type":"step_finish","timestamp":"2026-01-01T00:00:03.000Z","sessionID":"recorded-opencode-session","modelID":"opencode/muse-spark-1.2-contributor-free","part":{"tokens":{"input":2,"output":3},"cost":0.01}}' + Write-Output '{"type":"future.event.v99","payload":"fixture"}' +} elseif ($harness -eq 'copilot') { + Write-Output '{"type":"session.start","id":"e1","parentId":null,"data":{"sessionId":"recorded"}}' + Write-Output '{"type":"assistant.message","id":"e2","parentId":"e1","data":{"messageId":"m1","model":"claude-haiku-4.5","content":"recorded Copilot progress"}}' + Write-Output '{"type":"tool.execution_start","id":"e3","parentId":"e2","data":{"callId":"t1","toolName":"str_replace_editor"}}' + Write-Output '{"type":"tool.execution_complete","id":"e4","parentId":"e3","data":{"callId":"t1","status":"success"}}' + Write-Output '{"type":"assistant.message","id":"e5","parentId":"e4","data":{"messageId":"m2","model":"claude-haiku-4.5","content":"recorded Copilot final response"}}' + Write-Output '{"type":"assistant.usage","id":"e6","parentId":"e5","ephemeral":true,"data":{"model":"claude-haiku-4.5","inputTokens":2,"outputTokens":3,"cacheReadTokens":1,"numToolCalls":1,"cost":0.2}}' + Write-Output '{"type":"session.task_complete","id":"e7","parentId":"e6","data":{}}' + Write-Output '{"type":"future.event.v99","payload":"fixture"}' +} +'@ + foreach ($harness in @('codex', 'opencode', 'copilot')) { + [System.IO.File]::WriteAllText((Join-Path $fakeBin "$harness.ps1"), $fakeCli, [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $fakeBin "$harness.cmd"), "@echo off`r`npwsh -NoProfile -File ""%~dp0$harness.ps1"" %*`r`n", [System.Text.UTF8Encoding]::new($false)) + } + $fakeGh = @' +[CmdletBinding()] +param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) +if ($RemainingArguments.Count -eq 2 -and $RemainingArguments[0] -eq 'auth' -and $RemainingArguments[1] -eq 'token') { + $config = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') + if (-not [string]::IsNullOrWhiteSpace($config) -and (Test-Path -LiteralPath (Join-Path $config 'auth-marker.txt') -PathType Leaf)) { + Write-Output 'recorded-gh-fallback-token-not-logged' + exit 0 + } + [Console]::Error.WriteLine('not logged in') + exit 1 +} +[Console]::Error.WriteLine('unsupported gh fixture command') +exit 2 +'@ + [System.IO.File]::WriteAllText((Join-Path $fakeBin 'gh.ps1'), $fakeGh, [System.Text.UTF8Encoding]::new($false)) + $env:PATH = "$fakeBin$([System.IO.Path]::PathSeparator)$recordedOldPath" + $env:OPENAI_API_KEY = 'recorded-canary-not-logged' + $env:AGENTIC_GLOBAL_SECRET = 'recorded-unrelated-canary-not-logged' + $env:OPENCODE_DISABLE_PROJECT_CONFIG = '1' + $env:COPILOT_GITHUB_TOKEN = 'recorded-copilot-canary-not-logged' + $env:GH_TOKEN = 'recorded-gh-canary-not-logged' + $env:GITHUB_TOKEN = 'recorded-github-canary-not-logged' + $recordedGhConfig = Join-Path $recordedRoot 'github-cli-auth' + New-Item -ItemType Directory -Path $recordedGhConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $recordedGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [System.Text.UTF8Encoding]::new($false)) + $env:GH_CONFIG_DIR = $recordedGhConfig + $ambientCopilotHome = Join-Path $recordedRoot 'ambient-copilot-home' + New-Item -ItemType Directory -Path $ambientCopilotHome -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $ambientCopilotHome 'copilot-instructions.md'), '# ambient-personal-instruction-not-logged', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $ambientCopilotHome 'config.json'), '{"loggedInUsers":[{"login":"ambient-profile-not-logged"}]}', [System.Text.UTF8Encoding]::new($false)) + $env:COPILOT_HOME = $ambientCopilotHome + $recordedProfiles = [ordered]@{} + foreach ($runnerName in @('codex', 'opencode', 'copilot')) { + $profilePath = Join-Path $recordedRoot "$runnerName-profile.json" + $profileModel = switch ($runnerName) { + 'copilot' { 'claude-haiku-4.5' } + 'codex' { 'gpt-5.6-luna' } + 'opencode' { 'opencode/muse-spark-1.2-contributor-free' } + } + Write-TestJson -Path $profilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = if ($runnerName -eq 'copilot') { 'github-copilot' } else { $runnerName } + model = $profileModel + reasoning_effort = 'medium' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = if ($runnerName -eq 'opencode') { 2 } else { 1 } + }) + $recordedProfiles[$runnerName] = $profilePath + } + $resolvedRecordedCodex = Resolve-ExternalCommand -Name 'codex' + Assert-True (@((Join-Path $fakeBin 'codex.ps1'), (Join-Path $fakeBin 'codex.cmd')) -contains [string]$resolvedRecordedCodex.Source) 'recorded Codex command is selected before the installed CLI' + $recordedVersion = Get-ExternalCommandVersion -CommandInfo $resolvedRecordedCodex -WorkingDirectory (Join-Path $with.Root 'repo') + if (-not $recordedVersion.Available) { throw "recorded Codex --version is not observable (exit=$($recordedVersion.Process.ExitCode), timed_out=$($recordedVersion.Process.TimedOut), stdout='$($recordedVersion.Process.Stdout)', stderr='$($recordedVersion.Process.Stderr)')" } + Assert-Equal 'recorded-codex 9.1' $recordedVersion.Version 'recorded Codex exact version helper' + foreach ($fixtureName in @( + 'copilot-scripted-turn-1-events.jsonl', + 'copilot-scripted-turn-2-events.jsonl', + 'opencode-scripted-turn-1-events.jsonl', + 'opencode-scripted-turn-2-events.jsonl' + )) { + $fixturePath = Join-Path $recordedFixtureRoot $fixtureName + Assert-True (Test-Path -LiteralPath $fixturePath -PathType Leaf) "recorded scripted fixture exists: $fixtureName" + $fixtureEvents = [System.Collections.Generic.List[object]]::new() + foreach ($line in @(Get-Content -LiteralPath $fixturePath)) { + if ([string]::IsNullOrWhiteSpace([string]$line)) { continue } + try { $fixtureEvents.Add(($line | ConvertFrom-Json -Depth 100)) } + catch { throw "recorded scripted fixture '$fixtureName' contains invalid JSON: $($_.Exception.Message)" } + } + Assert-True ($fixtureEvents.Count -ge 3) "recorded scripted fixture has structured events: $fixtureName" + Assert-True (@($fixtureEvents | Where-Object { [string]$_.type -in @('assistant.message', 'text') }).Count -ge 1) "recorded scripted fixture has an assistant message: $fixtureName" + } + $debugConfigFixturePath = Join-Path $recordedFixtureRoot 'opencode-debug-config.json' + $debugConfigFixture = [IO.File]::ReadAllText($debugConfigFixturePath, [Text.UTF8Encoding]::new($false)) | ConvertFrom-Json -Depth 50 + Assert-Equal 'deny' ([string]$debugConfigFixture.permission.skill) 'Recorded OpenCode debug config fixture preserves deny-all skill policy syntax' + foreach ($runnerName in @('codex', 'opencode', 'copilot')) { + $runnerDir = if ($runnerName -eq 'copilot') { 'github-copilot' } else { $runnerName } + $runnerPath = Join-Path $runnerRoot "$runnerDir\runner.ps1" + $description = Invoke-AdapterJson -RunnerPath $runnerPath -Command describe -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + [void](Assert-RunnerDescriptor -Descriptor $description) + Assert-True ($description.PSObject.Properties.Name -contains 'delegation') "$runnerName descriptor declares native delegation" + $expectedDispatchOwner = 'runner' + Assert-Equal $expectedDispatchOwner $description.delegation.dispatch_owner "$runnerName descriptor declares its native dispatch owner" + Assert-True (-not [bool]$description.delegation.nested_model_execution) "$runnerName descriptor forbids nested model execution" + Assert-True (-not [string]::IsNullOrWhiteSpace([string]$description.delegation.mechanism)) "$runnerName descriptor records its native delegation mechanism" + Assert-Equal 'conditional' $description.capabilities.native_worker_delegation "$runnerName descriptor does not present native delegation as terminal proof" + Assert-Equal 'conditional' $description.delegation.model_lock "$runnerName descriptor leaves child model resolution conditional" + if ($runnerName -in @('copilot', 'opencode')) { + Assert-Equal 'conditional' $description.capabilities.scripted_multi_turn_same_session "$runnerName descriptor gates scripted continuation on installed capability proof" + } + $expectedVersion = switch ($runnerName) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-unknown 9.3' } } + Assert-Equal $expectedVersion $description.harness.version "$runnerName exact describe version" + $preflightWith = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + $preflightWithout = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $without.Path -ProfilePath $recordedProfiles[$runnerName] + Assert-Equal 'compatible' $preflightWith.status "$runnerName with_skill pragmatic preflight: $([string]::Join('; ', @($preflightWith.reasons)))" + Assert-Equal 'compatible' $preflightWithout.status "$runnerName without_skill pragmatic preflight" + Assert-Equal $expectedVersion $preflightWith.harness.version "$runnerName exact preflight version" + Assert-Equal 'pragmatic' $preflightWith.isolation.level "$runnerName pragmatic preflight level" + Assert-Equal 'conditional' $preflightWith.delegation.status "$runnerName native delegation preflight requires terminal evidence" + Assert-Equal $expectedDispatchOwner $preflightWith.delegation.dispatch_owner "$runnerName preflight preserves native dispatch ownership" + Assert-True ([bool]$preflightWith.delegation.terminal_evidence_required) "$runnerName preflight requires terminal delegation evidence" + if ($runnerName -in @('copilot', 'opencode')) { + Assert-Equal 'conditional' $preflightWith.resolved_capabilities.scripted_multi_turn_same_session "$runnerName single-turn preflight leaves scripted capability conditional" + } + if ($runnerName -eq 'copilot') { + Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Copilot preflight accepts explicit environment authentication' + Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq '--allow-all-tools broad tool approval' }).Count -eq 1) 'Copilot preflight describes --allow-all-tools as broad tool approval' + Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)' }).Count -eq 1) 'Copilot preflight records preserved path and URL verification' + } + if ($runnerName -eq 'opencode') { + Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'parallel_dispatch' -and $_.status -eq 'passed' }).Count -eq 1) 'OpenCode preflight requires bounded concurrent dispatch' + Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq 'deterministic runner-owned concurrent fan-out' }).Count -eq 1) 'OpenCode preflight records the runner-owned concurrent fan-out' + Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'effective_home' -and $_.status -eq 'passed' }).Count -eq 1) 'OpenCode preflight proves the effective runtime home' + Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'skill_isolation_policy' -and $_.status -eq 'passed' }).Count -eq 1) 'OpenCode preflight proves the arm skill policy' + Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'skill_permission_debug' -and $_.status -eq 'passed' }).Count -eq 1) 'OpenCode preflight proves the installed debug config permission layer' + Assert-Equal 'node.os.homedir' $preflightWith.protocol_observations.effective_home.effective_runtime_home_source 'OpenCode preflight uses the model-free Node homedir proof' + Assert-True ([bool]$preflightWith.protocol_observations.effective_home.windows_profile_parts_coherent) 'OpenCode preflight proves coherent Windows profile parts' + Assert-Equal 'supported_and_verified' $preflightWith.protocol_observations.skill_isolation.permission_layer 'OpenCode preflight records verified native skill permission support' + } + if ($runnerName -eq 'opencode') { + Assert-True (@($preflightWith.warnings | Where-Object { $_ -match 'child-tool environment filter' }).Count -gt 0) "$runnerName reports the child credential-filter limitation" + } + if ($runnerName -eq 'codex') { + Assert-True (-not [bool]$preflightWith.protocol_observations.allow_provider_model_fallback) 'Codex installed schema reports that provider fallback control is unavailable' + Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'native_worker_delegation' -and $_.detail -match 'structurally proves' }).Count -eq 1) 'Codex preflight uses structural app-server schema validation' + Assert-Equal 'aggregate_v2_bundle' $preflightWith.protocol_observations.schema_source_kind 'Codex fixture resolves the aggregate v2 schema bundle' + Assert-True ([string]$preflightWith.protocol_observations.schema_source -match 'codex_app_server_protocol\.v2\.schemas\.json$') 'Codex fixture records the aggregate v2 schema source' + Assert-Equal 'read-only,workspace-write,danger-full-access' ([string]::Join(',', @($preflightWith.protocol_observations.sandbox_modes))) 'Codex fixture validates the installed sandbox enum' + + $individualSchemaMarker = Join-Path $with.Root 'home\codex-schema-individual-v2' + [IO.File]::WriteAllText($individualSchemaMarker, 'fixture', [Text.UTF8Encoding]::new($false)) + try { + $individualPreflight = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + Assert-Equal 'compatible' $individualPreflight.status 'Codex recursively discovers namespaced individual v2 schemas' + Assert-Equal 'recursive_individual_files' $individualPreflight.protocol_observations.schema_source_kind 'Codex records recursive individual schema discovery' + } finally { + Remove-Item -LiteralPath $individualSchemaMarker -Force + } + + $withoutThreadReadMarker = Join-Path $with.Root 'home\codex-schema-without-thread-read' + [IO.File]::WriteAllText($withoutThreadReadMarker, 'fixture', [Text.UTF8Encoding]::new($false)) + try { + $withoutThreadReadPreflight = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + Assert-Equal 'compatible' $withoutThreadReadPreflight.status 'Codex accepts a protocol without the supplemental thread/read method' + Assert-True (-not [bool]$withoutThreadReadPreflight.protocol_observations.thread_read_schema_available) 'Codex records absent supplemental thread/read schemas as unavailable' + $withoutThreadReadDelegationCheck = @($withoutThreadReadPreflight.checks | Where-Object { $_.name -eq 'native_worker_delegation' }) | Select-Object -First 1 + Assert-True ([string]$withoutThreadReadDelegationCheck.detail -match 'thread/read is supplemental and not advertised') 'Codex reports the supplemental thread/read decision deterministically' + } finally { + Remove-Item -LiteralPath $withoutThreadReadMarker -Force + } + + $missingSchemaMarker = Join-Path $with.Root 'home\codex-schema-missing-ThreadStartParams' + [IO.File]::WriteAllText($missingSchemaMarker, 'fixture', [Text.UTF8Encoding]::new($false)) + try { + $missingPreflight = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + Assert-Equal 'incompatible' $missingPreflight.status 'Codex missing schema is a controlled incompatible preflight' + $missingText = [string]($missingPreflight | ConvertTo-Json -Depth 100) + Assert-True ($missingText -match 'Installed Codex app-server schema is missing required v2 schema: ThreadStartParams\.') 'Codex reports the exact missing logical schema' + Assert-True ($missingText -notmatch 'Cannot bind argument to parameter .Schema. because it is null') 'Codex missing schema never emits a null-binding exception' + } finally { + Remove-Item -LiteralPath $missingSchemaMarker -Force + } + + $multipleMissingSchemaMarker = Join-Path $with.Root 'home\codex-schema-missing-required' + [IO.File]::WriteAllText($multipleMissingSchemaMarker, 'fixture', [Text.UTF8Encoding]::new($false)) + try { + $multipleMissingPreflight = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + Assert-Equal 'incompatible' $multipleMissingPreflight.status 'Codex reports multiple missing schemas as a controlled incompatible preflight' + $multipleMissingText = [string]($multipleMissingPreflight | ConvertTo-Json -Depth 100) + Assert-True ($multipleMissingText -match 'Installed Codex app-server schemas are missing required v2 schemas: ThreadStartParams, TurnStartResponse\.') 'Codex reports all missing logical schemas in one deterministic message' + Assert-True ($multipleMissingText -notmatch 'Cannot bind argument to parameter .Schema. because it is null') 'Codex multiple missing schemas never emits a null-binding exception' + } finally { + Remove-Item -LiteralPath $multipleMissingSchemaMarker -Force + } + } + $resultWith = Invoke-AdapterJson -RunnerPath $runnerPath -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + $resultWithout = Invoke-AdapterJson -RunnerPath $runnerPath -Command execute -RunPath $without.Path -ProfilePath $recordedProfiles[$runnerName] + foreach ($result in @($resultWith, $resultWithout)) { + [void](Assert-ExecutionResult -Result $result) + Assert-Equal 'completed' $result.status "$runnerName recorded completion: $([string](Get-JsonProperty -Object $result.exit.failure -Name 'message' -Default ''))" + Assert-Equal $expectedVersion $result.harness.version "$runnerName exact execution version" + Assert-Equal 'accepted_request' $result.resolved.status "$runnerName accepted configuration provenance" + Assert-True ($null -eq $result.resolved.model) "$runnerName does not claim concrete model resolution" + Assert-Equal 'pragmatic' $result.isolation.level "$runnerName pragmatic execution level" + Assert-True (-not $result.isolation.hard_filesystem_confinement) "$runnerName pragmatic execution has no hard confinement" + Assert-Equal 1 $result.attempt_count "$runnerName one semantic attempt" + Assert-Equal 'available' $result.final_response.status "$runnerName captures final response" + $resultRoot = if ($result.run.configuration -eq 'with_skill') { $with.Root } else { $without.Root } + foreach ($artifact in @($result.artifacts)) { + Assert-True ($artifact.path -notmatch '(^|/|\\)\.\.(/|\\|$)') "$runnerName artifact path remains relative" + Assert-True (Test-Path -LiteralPath (Join-Path $resultRoot ($artifact.path -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -PathType Leaf) "$runnerName artifact exists inside its run" + } + # Runner-owned convergence: Copilot and OpenCode behavioral transport + # now emits transport-owned terminal evidence for its own fresh + # session, with no orchestrator-authored reconstruction. Codex's + # runner-owned app-server evidence is validated in its dedicated + # subscription block below; its API-key `codex exec` compatibility + # path is intentionally not the runner-owned behavioral transport. + if ($runnerName -in @('copilot', 'opencode')) { + Assert-Equal 'runner' ([string]$result.evidence.delegation.dispatch_owner) "$runnerName execution evidence is runner-owned" + Assert-Equal 'harness_native_transport' ([string]$result.evidence.capture.source) "$runnerName capture provenance is transport-owned" + Assert-True (-not [bool]$result.evidence.capture.worker_authored) "$runnerName capture is not orchestrator/worker-authored" + Assert-Equal ([string]$result.session.id) ([string]$result.evidence.delegation.worker_session_id) "$runnerName terminal evidence binds to the fresh session id" + $convergenceRunPath = if ($result.run.configuration -eq 'with_skill') { $with.Path } else { $without.Path } + $convergenceRun = Resolve-RunContract -RunPath $convergenceRunPath + $convergenceEvidence = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $result -Run $convergenceRun -RequestedModel ([string]$result.requested.model) -ExpectedWorkerSessionId ([string]$result.session.id) -ExpectedMechanism ([string]$description.delegation.mechanism) + Assert-True $convergenceEvidence.Valid "$runnerName produces valid runner-owned terminal evidence: $([string]::Join(', ', @($convergenceEvidence.Failures)))" + } + } + $logPath = Join-Path $with.Root "repo\$runnerName-fake-cli-log.jsonl" + Assert-True (Test-Path -LiteralPath $logPath -PathType Leaf) "$runnerName recorded process log exists" + $records = @(Get-Content -LiteralPath $logPath | ForEach-Object { $_ | ConvertFrom-Json }) + $executionRecords = @($records | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -or [bool](Get-JsonProperty -Object $_ -Name 'prompt_via_arg' -Default $false) }) + Assert-Equal 1 $executionRecords.Count "$runnerName one execution process per checked arm" + $execution = $executionRecords[0] + Assert-True $execution.stdin_received "$runnerName receives a non-empty stdin prompt" + Assert-Equal 1 $execution.stdin_delivery_count "$runnerName delivers one prompt through stdin" + $promptDiagnostic = if ($runnerName -eq 'codex') { " ($($execution | ConvertTo-Json -Depth 20 -Compress))" } else { '' } + Assert-True $execution.stdin_exact "$runnerName fake CLI received the exact staged prompt bytes$promptDiagnostic" + Assert-True $execution.stdin_utf8_round_trip "$runnerName preserves arbitrary UTF-8 prompt content" + Assert-True (-not $execution.unrelated_present) "$runnerName does not pass unrelated credential canary" + Assert-True (-not $execution.disable_project_config_present) "$runnerName does not pass ambient project-disable override" + Assert-True (-not $execution.worker_auth_file_visible) "$runnerName worker probe cannot read a copied Codex auth file" + Assert-True (-not $execution.worker_global_secret_visible) "$runnerName worker probe cannot read the parent/global canary" + Assert-True (-not $execution.worker_project_disable_visible) "$runnerName worker probe cannot read the parent project-disable variable" + if ($runnerName -eq 'codex') { + Assert-True (-not $execution.worker_provider_visible) 'Codex shell policy hides the provider API-key variable from the worker probe' + } elseif ($runnerName -eq 'copilot') { + Assert-True (-not $execution.worker_copilot_token_visible) 'Copilot secret COPILOT_GITHUB_TOKEN is unavailable to the worker probe' + Assert-True (-not $execution.worker_gh_token_visible) 'Copilot secret GH_TOKEN is unavailable to the worker probe' + Assert-True (-not $execution.worker_github_token_visible) 'Copilot secret GITHUB_TOKEN is unavailable to the worker probe' + } else { + Assert-True (-not $execution.worker_provider_visible) "$runnerName free-model fixture does not require a provider API key" + } + $args = @($execution.args) + foreach ($forbidden in @('--continue', '--session', '--resume')) { Assert-True ($args -notcontains $forbidden) "$runnerName does not pass '$forbidden'" } + if ($runnerName -eq 'codex') { + Assert-True ($args -contains '--ask-for-approval') 'Codex uses explicit approval policy' + Assert-True ($args -contains 'never') 'Codex approval policy is never' + Assert-True ($args -contains '--sandbox' -and $args -contains 'workspace-write') 'Codex retains workspace-write sandbox' + Assert-True ($args -notcontains '--approve-for-me') 'Codex avoids the conflicting approve-for-me flag' + $modelIndex = [Array]::IndexOf([string[]]$args, '--model') + Assert-Equal 'gpt-5.6-luna' $args[$modelIndex + 1] 'Codex opaque model selector propagates to the CLI invocation' + $outputIndex = [Array]::IndexOf([string[]]$args, '--output-last-message') + Assert-Equal (Join-Path $resultWith.evidence.execution_paths.physical_run_root 'evidence\codex-final.txt') $args[$outputIndex + 1] 'Codex output path uses the runner-declared physical projection' + } elseif ($runnerName -eq 'opencode') { + Assert-True ($args -notcontains '--pure') 'OpenCode preserves repository-owned project configuration' + Assert-True ($args -contains '--auto') 'OpenCode is noninteractive' + Assert-True $execution.project_config_visible 'OpenCode paired arm retains repository-owned project configuration' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$execution.working_directory))) 'OpenCode physical execution cwd is outside the source-repository fixture ancestry' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$execution.home))) 'OpenCode physical HOME is outside the source-repository fixture ancestry' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$execution.appdata))) 'OpenCode APPDATA is outside the source-repository fixture ancestry' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$execution.local_appdata))) 'OpenCode LOCALAPPDATA is outside the source-repository fixture ancestry' + Assert-True ([string]::IsNullOrWhiteSpace([string]$execution.node_path)) 'OpenCode does not inherit NODE_PATH from the ambient environment' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$execution.config_directory))) 'OpenCode config directory is outside the source-repository fixture ancestry' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$execution.config_file))) 'OpenCode config file is outside the source-repository fixture ancestry' + Assert-Equal ([string]$execution.home) ([string]$execution.node_homedir) 'OpenCode Node runtime resolves the isolated HOME' + Assert-Equal ([string]$execution.home) ([string]$execution.userprofile) 'OpenCode USERPROFILE matches the isolated HOME' + Assert-Equal ([string]$execution.home) ([string]$execution.homedrive + [string]$execution.homepath) 'OpenCode HOMEDRIVE/HOMEPATH resolve the isolated HOME' + Assert-True (Test-PathInside -BasePath ([string]$execution.home) -CandidatePath ([string]$execution.xdg_config_home)) 'OpenCode XDG_CONFIG_HOME is isolated' + Assert-Equal '1' $execution.disable_external_skills 'OpenCode disables external skill discovery' + Assert-Equal '1' $execution.disable_claude_code_skills 'OpenCode disables Claude/agents skill discovery' + Assert-True (-not [bool]$execution.source_ancestor_candidate_skill_visible) 'OpenCode execution cannot discover the source-repository candidate skill canary through projected ancestry' + Assert-True (-not [bool]$execution.source_ancestor_forms_visible) 'OpenCode execution cannot discover the source-repository FORMS.md canary through projected ancestry' + Assert-True (-not [bool]$execution.source_ancestor_agents_visible) 'OpenCode execution cannot discover source-repository ancestor AGENTS.md through projected ancestry' + Assert-True (-not [bool]$execution.source_ancestor_copilot_visible) 'OpenCode execution cannot discover source-repository ancestor copilot instructions through projected ancestry' + Assert-True (-not [bool]$execution.staged_repo_canary_visible) 'OpenCode staged repository contains no source candidate canary' + Assert-True (-not [bool]$execution.home_canary_visible) 'OpenCode isolated HOME contains no source candidate canary' + Assert-True (-not [bool]$execution.config_canary_visible) 'OpenCode isolated config contains no source candidate canary' + Assert-True ([bool]$execution.fake_ambient_candidate_fixture_visible) 'OpenCode ambient-skill canary fixture exists outside the isolated boundary' + Assert-True (-not [bool]$execution.ambient_agents_skill_visible) 'OpenCode hides fake global .agents skills' + Assert-True (-not [bool]$execution.ambient_claude_skill_visible) 'OpenCode hides fake global .claude skills' + Assert-True (-not [bool]$execution.ambient_opencode_skill_visible) 'OpenCode hides fake global native OpenCode skills' + Assert-True ([bool]$execution.staged_candidate_skill_visible) 'OpenCode with_skill projection contains the intended staged candidate skill' + Assert-Equal 'included' $execution.candidate_skill_exposure 'OpenCode with_skill execution records candidate-skill exposure as included' + Assert-True (-not [bool]$execution.candidate_skill_path_in_arguments) 'OpenCode does not receive a candidate-skill path through arguments' + Assert-True (-not [bool]$execution.candidate_skill_path_in_environment) 'OpenCode does not receive a candidate-skill path through environment variables' + Assert-True (-not [bool]$execution.candidate_canary_in_environment) 'OpenCode environment contains no ambient-skill canary marker' + Assert-True (-not [bool]$execution.candidate_canary_in_stdin) 'OpenCode with_skill stdin does not receive the source-repository canary material' + Assert-Equal 'deny' (Get-JsonProperty -Object $execution.skill_permission -Name '*' -Default '') 'OpenCode with_skill denies all ambient skill names by default' + Assert-Equal 'allow' (Get-JsonProperty -Object $execution.skill_permission -Name 'candidate' -Default '') 'OpenCode with_skill allows only the prepared candidate skill name' + Assert-True ([bool]$resultWith.evidence.execution_paths.physical_cwd_outside_source_repository) 'OpenCode result records the physical cwd ancestry boundary' + Assert-True ([bool]$resultWith.evidence.execution_paths.physical_home_outside_source_repository) 'OpenCode result records the physical HOME ancestry boundary' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$resultWith.evidence.execution_paths.physical_config_directory))) 'OpenCode result records a physical config directory outside the source-repository ancestry' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$resultWith.evidence.execution_paths.physical_config_file))) 'OpenCode result records a physical config file outside the source-repository ancestry' + Assert-True ([bool]$resultWith.evidence.candidate_skill_exposure.hash_match) 'OpenCode result proves the projected candidate skill hash matches the prepared skill hash' + Assert-Equal 1 $resultWith.evidence.skill_isolation.candidate_skill_count 'OpenCode result exposes exactly one native candidate skill' + Assert-Equal 0 $resultWith.evidence.skill_isolation.ambient_skill_count 'OpenCode result exposes no ambient skill roots with_skill' + Assert-Equal 'removed' $resultWith.evidence.execution_paths.projection_cleanup 'OpenCode removes the physical projection after evidence capture' + Assert-True (-not (Test-Path -LiteralPath ([string]$resultWith.evidence.execution_paths.physical_projection_root))) 'OpenCode physical projection is cleaned up by default' + Assert-True ([string]$resultWith.evidence.candidate_skill_exposure.physical_path -match '(?i)[\\/]\.opencode[\\/]skills[\\/]candidate$') 'OpenCode evidence identifies the intended native projected candidate skill location' + Assert-True ([bool]$resultWith.evidence.effective_home.valid) 'OpenCode result proves effective runtime home isolation' + Assert-True ([bool]$resultWith.evidence.effective_home.windows_profile_parts_coherent) 'OpenCode result proves coherent Windows profile-part isolation' + Assert-True ([bool]$resultWith.evidence.ambient_skill_policy.ambient_skill_roots_hidden) 'OpenCode result proves ambient skill roots remain hidden during with_skill' + Assert-True ($resultWith.evidence.timing.turns[0].PSObject.Properties.Name -contains 'event_timing') 'OpenCode records structured event timing when event timestamps are available' + $modelIndex = [Array]::IndexOf([string[]]$args, '--model') + Assert-Equal 'opencode/muse-spark-1.2-contributor-free' $args[$modelIndex + 1] 'OpenCode opaque model selector propagates to the CLI invocation' + $variantIndex = [Array]::IndexOf([string[]]$args, '--variant') + Assert-True ($variantIndex -ge 0) 'OpenCode includes --variant only because the execution profile requested reasoning_effort and preflight proved support' + Assert-Equal 'medium' $args[$variantIndex + 1] 'OpenCode reasoning effort propagates to the --variant argument' + } elseif ($runnerName -eq 'copilot') { + Assert-True (@($args | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count -eq 0) 'Copilot does not place the prompt in argv' + Assert-Equal 0 $execution.prompt_arg_count 'Copilot has no prompt argument' + Assert-True $execution.stdin_received 'Copilot reads the prompt from stdin' + Assert-True ($args -contains '--output-format' -and $args -contains 'json') 'Copilot uses structured JSONL output' + $modelIndex = [Array]::IndexOf([string[]]$args, '--model') + Assert-Equal 'claude-haiku-4.5' $args[$modelIndex + 1] 'Copilot reference model claude-haiku-4.5 propagates to the CLI invocation' + Assert-True ($args -contains '--allow-all-tools') 'Copilot grants broad tool approval for noninteractive execution' + Assert-True ($args -contains '--no-ask-user') 'Copilot does not pause for interactive questions' + Assert-True ($args -notcontains '--no-custom-instructions') 'Copilot preserves repository-owned custom instructions' + Assert-True ($args -contains '--disable-builtin-mcps') 'Copilot disables ambient built-in MCP servers' + foreach ($broad in @('--yolo', '--allow-all', '--allow-all-paths', '--allow-all-urls', '--session-id', '--connect', '-r')) { Assert-True ($args -notcontains $broad) "Copilot avoids the over-broad or session option '$broad'" } + Assert-Equal 1 (@($args | Where-Object { $_ -like '--secret-env-vars=*' }).Count) 'Copilot filters protected variables with --secret-env-vars' + Assert-Equal 'COPILOT_GITHUB_TOKEN,GH_TOKEN,GITHUB_TOKEN' ([string]($args | Where-Object { $_ -like '--secret-env-vars=*' }) -replace '^--secret-env-vars=', '') 'Copilot protects every forwarded token variable' + Assert-True (-not $execution.custom_instructions_disabled -and $execution.builtin_mcps_disabled) 'Copilot preserves repository instructions while disabling built-in MCPs' + Assert-True ($execution.repository_agents_visible -and $execution.repository_copilot_instructions_visible -and $execution.repository_instruction_marker_visible) 'Copilot sees staged repository-owned instructions' + Assert-True $execution.candidate_skill_staged 'Copilot with_skill arm retains the staged candidate skill independently of repository instructions' + Assert-True (-not $execution.ambient_copilot_instructions_visible) 'Copilot does not see the ambient personal instruction file' + Assert-Equal 'explicit_environment' $execution.copilot_authentication_source 'Copilot uses explicit environment authentication in the token fixture' + Assert-Equal 3 @($execution.copilot_auth_names_present).Count 'Copilot process receives all protected token variables without logging values' + Assert-True ([string]::IsNullOrWhiteSpace([string]$execution.gh_config_dir)) 'Copilot explicit-token path does not forward host GH_CONFIG_DIR' + Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_cache_home)) 'Copilot cache is run-local' + Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_home)) 'Copilot COPILOT_HOME is the run''s isolated home' + Assert-Equal 'stdin' $resultWith.evidence.prompt_delivery 'Copilot result records stdin prompt delivery' + Assert-Equal 'COPILOT_GITHUB_TOKEN' $resultWith.evidence.credential.github_token_variable 'Copilot follows explicit token precedence' + Assert-True (-not $resultWith.evidence.credential.github_cli_config_forwarded) 'Copilot result records that GH_CONFIG_DIR was not forwarded with an explicit token' + Assert-Equal 'supported' $resultWith.isolation.capabilities.credential_child_filtering 'Copilot documents protected child-environment filtering' + Assert-Equal 'shell,mcp' ([string]::Join(',', @($resultWith.evidence.credential.secret_env_var_scope))) 'Copilot evidence names the documented filtering scope' + Assert-Equal 'recorded Copilot final response' $resultWith.final_response.text 'Copilot final response is the last assistant message, not an intermediate one' + Assert-Equal 'claude-haiku-4.5' $resultWith.requested.model 'Copilot requested model is preserved as the Codebelt reference model' + Assert-True ($null -eq $resultWith.resolved.model) 'Copilot does not claim a distinct backend model resolution' + Assert-Equal 'claude-haiku-4.5' $resultWith.evidence.observed_model 'Copilot observed model is captured separately from the requested model' + Assert-Equal 'available' $resultWith.telemetry.tokens.status 'Copilot reports available token telemetry' + Assert-Equal 2 ([int]$resultWith.telemetry.tokens.value.input_tokens) 'Copilot input tokens are parsed from assistant.usage' + Assert-Equal 3 ([int]$resultWith.telemetry.tokens.value.output_tokens) 'Copilot output tokens are parsed from assistant.usage' + Assert-Equal 'available' $resultWith.telemetry.tool_calls.status 'Copilot reports available tool-call telemetry' + Assert-True ([int]$resultWith.telemetry.tool_calls.value -ge 1) 'Copilot parses documented tool.execution events' + Assert-Equal 'unavailable' $resultWith.telemetry.cost.status 'Copilot does not estimate a currency cost' + } + $logText = [System.IO.File]::ReadAllText($logPath, [System.Text.UTF8Encoding]::new($false)) + Assert-True ($logText -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') "$runnerName logs do not contain credential values" + Assert-True (($resultWith | ConvertTo-Json -Depth 100) -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') "$runnerName result evidence does not contain credential values" + $withoutLogPath = Join-Path $without.Root "repo\$runnerName-fake-cli-log.jsonl" + Assert-True (Test-Path -LiteralPath $withoutLogPath -PathType Leaf) "$runnerName baseline process log exists" + $withoutRecords = @(Get-Content -LiteralPath $withoutLogPath | ForEach-Object { $_ | ConvertFrom-Json }) + $withoutExecution = @($withoutRecords | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -or [bool](Get-JsonProperty -Object $_ -Name 'prompt_via_arg' -Default $false) }) + Assert-Equal 1 $withoutExecution.Count "$runnerName baseline has one execution process" + if ($runnerName -eq 'opencode') { + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$withoutExecution[0].working_directory))) 'OpenCode without_skill physical cwd is outside the source-repository fixture ancestry' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$withoutExecution[0].appdata))) 'OpenCode without_skill APPDATA is outside the source-repository fixture ancestry' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$withoutExecution[0].local_appdata))) 'OpenCode without_skill LOCALAPPDATA is outside the source-repository fixture ancestry' + Assert-True ([string]::IsNullOrWhiteSpace([string]$withoutExecution[0].node_path)) 'OpenCode without_skill does not inherit NODE_PATH from the ambient environment' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$withoutExecution[0].config_directory))) 'OpenCode without_skill config directory is outside the source-repository fixture ancestry' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$withoutExecution[0].config_file))) 'OpenCode without_skill config file is outside the source-repository fixture ancestry' + Assert-Equal ([string]$withoutExecution[0].home) ([string]$withoutExecution[0].node_homedir) 'OpenCode without_skill Node runtime resolves the isolated HOME' + Assert-Equal ([string]$withoutExecution[0].home) ([string]$withoutExecution[0].userprofile) 'OpenCode without_skill USERPROFILE matches the isolated HOME' + Assert-Equal ([string]$withoutExecution[0].home) ([string]$withoutExecution[0].homedrive + [string]$withoutExecution[0].homepath) 'OpenCode without_skill HOMEDRIVE/HOMEPATH resolve the isolated HOME' + Assert-True (Test-PathInside -BasePath ([string]$withoutExecution[0].home) -CandidatePath ([string]$withoutExecution[0].xdg_config_home)) 'OpenCode without_skill XDG_CONFIG_HOME is isolated' + Assert-Equal '1' $withoutExecution[0].disable_external_skills 'OpenCode without_skill disables external skill discovery' + Assert-Equal '1' $withoutExecution[0].disable_claude_code_skills 'OpenCode without_skill disables Claude/agents skill discovery' + Assert-True ([bool]$withoutExecution[0].fake_ambient_candidate_fixture_visible) 'OpenCode without_skill ambient-skill canary fixture exists outside the isolated boundary' + Assert-True (-not [bool]$withoutExecution[0].ambient_agents_skill_visible) 'OpenCode without_skill hides fake global .agents skills' + Assert-True (-not [bool]$withoutExecution[0].ambient_claude_skill_visible) 'OpenCode without_skill hides fake global .claude skills' + Assert-True (-not [bool]$withoutExecution[0].ambient_opencode_skill_visible) 'OpenCode without_skill hides fake global native OpenCode skills' + Assert-Equal 'deny' ([string]$withoutExecution[0].skill_permission) 'OpenCode without_skill denies every skill name' + Assert-True (-not [bool]$withoutExecution[0].candidate_canary_in_environment) 'OpenCode without_skill environment contains no ambient-skill canary marker' + Assert-True (-not [bool]$withoutExecution[0].source_ancestor_candidate_skill_visible) 'OpenCode without_skill cannot discover the source candidate skill canary through projected ancestors' + Assert-True (-not [bool]$withoutExecution[0].source_ancestor_forms_visible) 'OpenCode without_skill cannot discover the FORMS.md canary through projected ancestors' + Assert-True (-not [bool]$withoutExecution[0].source_ancestor_agents_visible) 'OpenCode without_skill cannot discover source ancestor AGENTS.md through projected ancestors' + Assert-True (-not [bool]$withoutExecution[0].source_ancestor_copilot_visible) 'OpenCode without_skill cannot discover source ancestor copilot instructions through projected ancestors' + Assert-True (-not [bool]$withoutExecution[0].staged_repo_canary_visible) 'OpenCode without_skill staged repo contains no source candidate canary' + Assert-True (-not [bool]$withoutExecution[0].home_canary_visible) 'OpenCode without_skill HOME contains no source candidate canary' + Assert-True (-not [bool]$withoutExecution[0].config_canary_visible) 'OpenCode without_skill config contains no source candidate canary' + Assert-True (-not [bool]$withoutExecution[0].staged_candidate_skill_visible) 'OpenCode without_skill projection contains no candidate skill' + Assert-Equal 'excluded' $withoutExecution[0].candidate_skill_exposure 'OpenCode without_skill execution records candidate-skill exposure as excluded' + Assert-True (-not [bool]$withoutExecution[0].candidate_skill_path_in_arguments) 'OpenCode without_skill has no candidate-skill path in arguments' + Assert-True (-not [bool]$withoutExecution[0].candidate_skill_path_in_environment) 'OpenCode without_skill has no candidate-skill path in environment' + Assert-True (-not [bool]$withoutExecution[0].candidate_canary_in_stdin) 'OpenCode without_skill stdin contains no candidate canary material' + Assert-True (-not ([string]($resultWithout | ConvertTo-Json -Depth 100) -match 'CODEBELT_BASELINE_LEAK_CANARY_|CODEBELT_BASELINE_FORMS_CANARY_')) 'OpenCode without_skill result contains no baseline canary material' + Assert-Equal 'excluded' $resultWithout.evidence.candidate_skill_exposure.status 'OpenCode without_skill result records candidate-skill exposure as excluded' + Assert-True ([string]::IsNullOrWhiteSpace([string]$resultWithout.evidence.candidate_skill_exposure.physical_path)) 'OpenCode without_skill result has no projected candidate skill path' + Assert-Equal 'removed' $resultWithout.evidence.execution_paths.projection_cleanup 'OpenCode without_skill removes the physical projection after evidence capture' + Assert-True (-not (Test-Path -LiteralPath ([string]$resultWithout.evidence.execution_paths.physical_projection_root))) 'OpenCode without_skill physical projection is cleaned up by default' + Assert-True ([bool]$resultWithout.evidence.effective_home.valid) 'OpenCode without_skill result proves effective runtime home isolation' + Assert-Equal 'deny' ([string]$resultWithout.evidence.skill_policy.configured_permission_skill) 'OpenCode without_skill result records deny-all skill policy' + Assert-Equal 0 $resultWithout.evidence.skill_isolation.discovered_skills.Count 'OpenCode without_skill result discovers zero candidate or ambient skills' + Assert-True ([bool]$resultWithout.evidence.ambient_skill_policy.ambient_skill_roots_hidden) 'OpenCode without_skill result proves ambient skill roots remain hidden' + Assert-True ([string]$resultWithout.evidence.execution_paths.physical_working_directory -notmatch [regex]::Escape($recordedRoot)) 'OpenCode without_skill result physical cwd does not point into the source repository' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$resultWithout.evidence.execution_paths.physical_config_directory))) 'OpenCode without_skill result records a physical config directory outside the source-repository ancestry' + Assert-True (-not (Test-PathInside -BasePath $recordedRoot -CandidatePath ([string]$resultWithout.evidence.execution_paths.physical_config_file))) 'OpenCode without_skill result records a physical config file outside the source-repository ancestry' + } + $withoutLogText = [System.IO.File]::ReadAllText($withoutLogPath, [System.Text.UTF8Encoding]::new($false)) + Assert-True ($withoutLogText -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') "$runnerName baseline log does not contain credential values" + Assert-True $withoutExecution[0].stdin_exact "$runnerName baseline receives exact prompt bytes" + if ($runnerName -eq 'copilot') { + Assert-True $withoutExecution[0].repository_agents_visible 'Copilot baseline sees the same staged AGENTS.md instruction' + Assert-True $withoutExecution[0].repository_copilot_instructions_visible 'Copilot baseline sees the same staged repository instruction' + Assert-True (-not $withoutExecution[0].candidate_skill_staged) 'Copilot baseline does not receive the candidate skill directory' + Assert-True (-not $withoutExecution[0].ambient_copilot_instructions_visible) 'Copilot baseline excludes the ambient personal instruction' + Assert-True (($resultWithout | ConvertTo-Json -Depth 100) -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') 'Copilot baseline result evidence does not contain credential values' + } + } + $scriptedInteraction = [ordered]@{ + schema = (Get-RunnerSchemaNames).Interaction + mode = 'scripted' + turns = @( + [ordered]@{ role = 'user'; content = 'recorded scripted turn one' } + [ordered]@{ role = 'user'; source = 'future-turn/turn-2.txt' } + ) + } + foreach ($runnerName in @('copilot', 'opencode')) { + $scriptedIteration = Join-Path $recordedRoot ("scripted-{0}" -f $runnerName) + New-Item -ItemType Directory -Path $scriptedIteration -Force | Out-Null + $scriptedWith = New-TestRun -IterationDirectory $scriptedIteration -Configuration with_skill -EvalName 'scripted-conformance' -Interaction $scriptedInteraction + $scriptedWithout = New-TestRun -IterationDirectory $scriptedIteration -Configuration without_skill -EvalName 'scripted-conformance' -Interaction $scriptedInteraction + foreach ($scriptedRun in @($scriptedWith, $scriptedWithout)) { + [IO.File]::WriteAllText((Join-Path $scriptedRun.Root 'home\scripted-session-fixture'), 'fixture', [Text.UTF8Encoding]::new($false)) + if ($runnerName -eq 'copilot') { [IO.File]::WriteAllText((Join-Path $scriptedRun.Root 'home\copilot-exact-session-help'), 'fixture', [Text.UTF8Encoding]::new($false)) } + if ($runnerName -eq 'opencode') { [IO.File]::WriteAllText((Join-Path $scriptedRun.Root 'home\opencode-timing-fixture'), 'fixture', [Text.UTF8Encoding]::new($false)) } + Add-TestInteractionSources -TestRun $scriptedRun + } + if ($runnerName -eq 'opencode') { + $preexistingOpenCodeIgnore = Join-Path $scriptedWith.Root 'repo\.opencode\.gitignore' + New-Item -ItemType Directory -Path (Split-Path -Parent $preexistingOpenCodeIgnore) -Force | Out-Null + [IO.File]::WriteAllText($preexistingOpenCodeIgnore, 'fixture-existing', [Text.UTF8Encoding]::new($false)) + } + $scriptedRunnerRelativePath = if ($runnerName -eq 'copilot') { 'github-copilot\runner.ps1' } else { 'opencode\runner.ps1' } + $scriptedRunnerPath = Join-Path $runnerRoot $scriptedRunnerRelativePath + $scriptedPreflight = Invoke-AdapterJson -RunnerPath $scriptedRunnerPath -Command preflight -RunPath $scriptedWith.Path -ProfilePath $recordedProfiles[$runnerName] + Assert-Equal 'compatible' $scriptedPreflight.status "$runnerName scripted preflight: $([string]::Join('; ', @($scriptedPreflight.reasons)))" + Assert-Equal 'supported' $scriptedPreflight.resolved_capabilities.scripted_multi_turn_same_session "$runnerName scripted same-session capability is supported only after deterministic proof" + Assert-True ([bool]$scriptedPreflight.protocol_observations.scripted_multi_turn_same_session.available) "$runnerName records deterministic scripted same-session proof" + Assert-True (-not [bool]$scriptedPreflight.protocol_observations.scripted_multi_turn_same_session.implicit_continuation) "$runnerName rejects implicit continuation in the protocol observation" + $scriptedResult = Invoke-AdapterJson -RunnerPath $scriptedRunnerPath -Command execute -RunPath $scriptedWith.Path -ProfilePath $recordedProfiles[$runnerName] + [void](Assert-ExecutionResult -Result $scriptedResult) + Assert-Equal 'completed' $scriptedResult.status "$runnerName exact-session scripted execution completes" + [void](Assert-InteractionResultEvidence -ExecutionResult $scriptedResult -RunData (Resolve-RunContract -RunPath $scriptedWith.Path)) + Assert-True ([bool]$scriptedResult.evidence.interaction.same_session) "$runnerName scripted result proves one same session" + Assert-Equal 2 @($scriptedResult.evidence.interaction.native_turns).Count "$runnerName records both native turns" + Assert-Equal ([string]$scriptedResult.session.id) ([string]$scriptedResult.evidence.interaction.session_id) "$runnerName shared interaction evidence uses the result session id" + Assert-True ([bool]$scriptedResult.evidence.capture.complete_structured_transcript) "$runnerName records a complete structured multi-turn transcript" + Assert-Equal 1 $scriptedResult.evidence.delegation.model_execution_count "$runnerName keeps one runner-owned model execution worker across scripted invocations" + $nativeTurns = @($scriptedResult.evidence.interaction.native_turns) + $firstNativeTurn = $nativeTurns[0] + $secondNativeTurn = $nativeTurns[1] + Assert-Equal ([string]$scriptedResult.session.id) ([string]$scriptedResult.evidence.exact_session_continuation.exact_session_id) "$runnerName exact continuation evidence uses the captured session id" + Assert-True ([bool]$scriptedResult.evidence.exact_session_continuation.turns_started_after_prior_terminal) "$runnerName starts continuation only after a terminal first turn" + Assert-Equal 'fresh' $firstNativeTurn.invocation "$runnerName turn 1 is fresh" + Assert-Equal 'explicit_session_resume' $secondNativeTurn.invocation "$runnerName turn 2 explicitly resumes" + Assert-Equal ([string]$firstNativeTurn.session_id) ([string]$secondNativeTurn.session_id) "$runnerName native turn evidence has one session id" + Assert-True (-not [string]::IsNullOrWhiteSpace([string]$firstNativeTurn.session_id)) "$runnerName captures an exact session id from turn 1 structured events" + Assert-Equal ([string]$firstNativeTurn.session_id) ([string]$secondNativeTurn.target_session_id) "$runnerName turn 2 targets turn 1's exact session id" + Assert-True ([bool]$secondNativeTurn.target_session_match) "$runnerName turn 2 native session identity matches its target" + Assert-Equal ([string]$firstNativeTurn.working_directory) ([string]$secondNativeTurn.working_directory) "$runnerName preserves the working directory across turns" + Assert-Equal ([string]$firstNativeTurn.home) ([string]$secondNativeTurn.home) "$runnerName preserves the isolated home across turns" + Assert-Equal ([string]$firstNativeTurn.requested_model) ([string]$secondNativeTurn.requested_model) "$runnerName preserves the requested model across turns" + Assert-True (@($firstNativeTurn.observed_models | Where-Object { [string]$_ -eq [string]$firstNativeTurn.requested_model }).Count -gt 0) "$runnerName records the requested model in first-turn structured evidence" + Assert-True (@($secondNativeTurn.observed_models | Where-Object { [string]$_ -eq [string]$secondNativeTurn.requested_model }).Count -gt 0) "$runnerName records the requested model in resumed-turn structured evidence" + Assert-Equal 0 $firstNativeTurn.exit_code "$runnerName first turn exits cleanly" + Assert-Equal 0 $secondNativeTurn.exit_code "$runnerName resumed turn exits cleanly" + Assert-True ([bool]$firstNativeTurn.terminal -and [bool]$secondNativeTurn.terminal) "$runnerName records terminal native turn evidence" + Assert-True ([bool]$firstNativeTurn.terminal_assistant_response -and [bool]$firstNativeTurn.terminal_event_observed) "$runnerName proves turn 1 has a terminal assistant response before continuation" + Assert-True ([bool]$secondNativeTurn.terminal_assistant_response -and [bool]$secondNativeTurn.terminal_event_observed) "$runnerName proves the resumed turn has a terminal assistant response" + Assert-True ([DateTime]::Compare([DateTime]$firstNativeTurn.finished_utc, [DateTime]$secondNativeTurn.started_utc) -le 0) "$runnerName starts turn 2 after turn 1 finishes" + Assert-True @($firstNativeTurn.event_timestamps).Count -gt 0 "$runnerName records first-turn event timestamps" + Assert-True @($secondNativeTurn.event_timestamps).Count -gt 0 "$runnerName records second-turn event timestamps" + $firstArgs = @($firstNativeTurn.arguments) + $secondArgs = @($secondNativeTurn.arguments) + Assert-True ($firstArgs -notcontains '--continue' -and $secondArgs -notcontains '--continue') "$runnerName never uses implicit last-session continuation" + if ($runnerName -eq 'opencode') { + Assert-Equal 'opencode-run-explicit-session-continuation' $scriptedResult.evidence.interaction.transport 'OpenCode scripted result records direct CLI exact-session transport' + Assert-True (-not [bool]$scriptedResult.evidence.future_turn_secrecy.interaction_json_projected) 'OpenCode interaction.json is absent from the physical projection' + Assert-True (-not [bool]$scriptedResult.evidence.future_turn_secrecy.future_source_files_projected) 'OpenCode future source files are absent from the physical projection' + Assert-True (-not [bool]$scriptedResult.evidence.future_turn_secrecy.canary_in_physical_projection) 'OpenCode future-turn canary is absent from the turn-1 physical projection' + Assert-True (-not [bool]$scriptedResult.evidence.future_turn_secrecy.canary_in_environment) 'OpenCode future-turn canary is absent from the environment' + Assert-True (-not [bool]$scriptedResult.evidence.future_turn_secrecy.canary_in_arguments) 'OpenCode future-turn canary is absent from turn-1 arguments' + Assert-True (-not [bool]$scriptedResult.evidence.future_turn_secrecy.turn_1_input_contains_future_canary) 'OpenCode turn 1 stdin excludes future input' + Assert-True ([bool]$scriptedResult.evidence.future_turn_secrecy.turn_2_sent_only_after_turn_1_terminal_completed) 'OpenCode turn 2 is sent only after turn 1 terminal completion' + Assert-True ($firstArgs -notcontains '--session') 'OpenCode turn 1 has no explicit session argument' + Assert-True ($secondArgs -contains '--session') 'OpenCode turn 2 uses explicit --session continuation' + $continuationIndex = [Array]::IndexOf([string[]]$secondArgs, '--session') + Assert-Equal ([string]$firstNativeTurn.session_id) ([string]$secondArgs[$continuationIndex + 1]) 'OpenCode turn 2 passes the exact turn-1 session id' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $scriptedWith.Root 'repo\.opencode\node_modules') -PathType Container)) 'OpenCode runtime node_modules does not leak into the logical fixture' + Assert-True (Test-Path -LiteralPath (Join-Path $scriptedWith.Root 'repo\.opencode\.gitignore') -PathType Leaf) 'OpenCode pre-existing .opencode/.gitignore remains in the logical fixture' + Assert-Equal 'runtime' (Get-Content -LiteralPath (Join-Path $scriptedWith.Root 'repo\.opencode\.gitignore') -Raw).Trim() 'OpenCode legitimate pre-existing .opencode mutation synchronizes into the logical fixture' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $scriptedWith.Root 'repo\.opencode\skills\candidate') -PathType Container)) 'OpenCode candidate skill is removed before copy-back' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $scriptedWithout.Root 'repo\.opencode\.gitignore') -PathType Leaf)) 'OpenCode newly generated .opencode/.gitignore is excluded from copy-back when absent initially' + Assert-True (Test-Path -LiteralPath (Join-Path $scriptedWith.Root 'repo\normal-task-output.snk') -PathType Leaf) 'OpenCode normal task output synchronizes into the logical fixture' + + $baselinePreflight = Invoke-AdapterJson -RunnerPath $scriptedRunnerPath -Command preflight -RunPath $scriptedWithout.Path -ProfilePath $recordedProfiles[$runnerName] + Assert-Equal 'compatible' $baselinePreflight.status 'OpenCode baseline scripted preflight is compatible' + Assert-Equal 'supported' $baselinePreflight.resolved_capabilities.scripted_multi_turn_same_session 'OpenCode baseline advertises exact-session scripted capability' + $baselineResult = Invoke-AdapterJson -RunnerPath $scriptedRunnerPath -Command execute -RunPath $scriptedWithout.Path -ProfilePath $recordedProfiles[$runnerName] + [void](Assert-ExecutionResult -Result $baselineResult) + Assert-Equal 'completed' $baselineResult.status 'OpenCode baseline scripted execution completes' + [void](Assert-InteractionResultEvidence -ExecutionResult $baselineResult -RunData (Resolve-RunContract -RunPath $scriptedWithout.Path)) + Assert-True ([bool]$baselineResult.evidence.interaction.same_session) 'OpenCode baseline proves one same session' + Assert-Equal 2 @($baselineResult.evidence.interaction.native_turns).Count 'OpenCode baseline records both native turns' + Assert-True (-not [bool]$baselineResult.evidence.future_turn_secrecy.interaction_json_projected -and + -not [bool]$baselineResult.evidence.future_turn_secrecy.future_source_files_projected -and + -not [bool]$baselineResult.evidence.future_turn_secrecy.canary_in_physical_projection -and + -not [bool]$baselineResult.evidence.future_turn_secrecy.canary_in_environment -and + -not [bool]$baselineResult.evidence.future_turn_secrecy.canary_in_arguments -and + -not [bool]$baselineResult.evidence.future_turn_secrecy.turn_1_input_contains_future_canary) 'OpenCode baseline keeps future input out of repo/home/config/env/args and turn 1' + Assert-True ([bool]$baselineResult.evidence.future_turn_secrecy.turn_2_sent_only_after_turn_1_terminal_completed) 'OpenCode baseline sends turn 2 only after turn 1 terminal completion' + $baselineRecords = @(Get-Content -LiteralPath (Join-Path $scriptedWithout.Root 'repo\opencode-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json }) + $baselineExecutions = @($baselineRecords | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + Assert-Equal 2 $baselineExecutions.Count 'OpenCode baseline starts exactly two direct CLI invocations' + Assert-Equal ([string]$baselineResult.session.id) ([string]$baselineExecutions[1].continuation_session_id) 'OpenCode baseline turn 2 receives the exact captured session id' + Assert-True ([bool]$baselineExecutions[0].stdin_exact -and [bool]$baselineExecutions[1].stdin_exact) 'OpenCode baseline sends both scripted turn inputs through stdin' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $scriptedWithout.Root 'repo\.opencode\node_modules') -PathType Container)) 'OpenCode baseline runtime node_modules does not leak into the logical fixture' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $scriptedWithout.Root 'repo\.opencode\.gitignore') -PathType Leaf)) 'OpenCode baseline generated .opencode/.gitignore is excluded from copy-back' + Assert-True (Test-Path -LiteralPath (Join-Path $scriptedWithout.Root 'repo\normal-task-output.snk') -PathType Leaf) 'OpenCode baseline normal task output synchronizes into the logical fixture' + } else { + Assert-Equal ([string]$firstNativeTurn.copilot_home) ([string]$secondNativeTurn.copilot_home) "$runnerName preserves the isolated COPILOT_HOME across turns" + Assert-True ($firstArgs -notcontains '--resume') "$runnerName fresh turn does not carry a continuation flag" + Assert-True ($secondArgs -contains '--resume') "$runnerName resumed turn carries its explicit continuation flag" + $continuationIndex = [Array]::IndexOf([string[]]$secondArgs, '--resume') + Assert-Equal ([string]$firstNativeTurn.session_id) ([string]$secondArgs[$continuationIndex + 1]) "$runnerName resumed invocation passes the exact captured session id" + } + $scriptedLogPath = Join-Path $scriptedWith.Root ("repo\{0}-fake-cli-log.jsonl" -f $runnerName) + $scriptedRecords = @(Get-Content -LiteralPath $scriptedLogPath | ForEach-Object { $_ | ConvertFrom-Json }) + $scriptedExecutions = @($scriptedRecords | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + Assert-Equal 2 $scriptedExecutions.Count "$runnerName scripted transport starts exactly two recorded invocations" + Assert-Equal ([string]$firstNativeTurn.session_id) ([string]$scriptedExecutions[1].continuation_session_id) "$runnerName recorded transport receives the exact session id on turn 2" + Assert-True ([bool]$scriptedExecutions[0].stdin_exact -and [bool]$scriptedExecutions[1].stdin_exact) "$runnerName sends both scripted turn inputs through stdin" + $scriptedFailureValidation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $scriptedResult -Run (Resolve-RunContract -RunPath $scriptedWith.Path) -RequestedModel ([string]$scriptedResult.requested.model) -ExpectedWorkerSessionId ([string]$scriptedResult.session.id) -ExpectedMechanism ([string]$scriptedResult.evidence.delegation.mechanism) + Assert-True ([bool]$scriptedFailureValidation.Valid) "$runnerName scripted result satisfies the shared native terminal evidence contract" + if ($runnerName -eq 'opencode') { + Assert-Equal 'authoritative_cached_preflight' $scriptedResult.evidence.timing.preflight_source 'OpenCode scripted execution reuses the authoritative same-run preflight observation' + Assert-Equal 5 $scriptedResult.evidence.timing.preflight.probe_count 'OpenCode preflight records version, effective-home, run-help, debug-help, and debug-config probes' + Assert-True ([double]$scriptedResult.evidence.timing.preflight.version_probe_duration_seconds -gt 0) 'OpenCode preflight records version probe duration' + Assert-True ([double]$scriptedResult.evidence.timing.preflight.help_probe_duration_seconds -gt 0) 'OpenCode preflight records help probe duration' + Assert-Equal 2 @($scriptedResult.evidence.timing.turns).Count 'OpenCode timing evidence records both scripted turns' + $firstTiming = @($scriptedResult.evidence.timing.turns)[0] + $secondTiming = @($scriptedResult.evidence.timing.turns)[1] + Assert-Equal 'fresh' $firstTiming.invocation 'OpenCode timing evidence labels turn 1 as a fresh CLI run' + Assert-Equal 'explicit_session_resume' $secondTiming.invocation 'OpenCode timing evidence labels turn 2 as explicit session continuation' + Assert-True ([double]$firstTiming.process_duration_seconds -ge 0 -and [double]$secondTiming.process_duration_seconds -ge 0) 'OpenCode timing evidence records both CLI process durations' + $timingRecords = @(Get-Content -LiteralPath (Join-Path $scriptedWith.Root 'repo\opencode-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 1 @($timingRecords | Where-Object { $_.invocation_kind -eq 'version_probe' }).Count 'OpenCode performs one version probe for the preflight/execute pair' + Assert-Equal 1 @($timingRecords | Where-Object { $_.invocation_kind -eq 'help_probe' }).Count 'OpenCode performs one help probe for the preflight/execute pair' + Assert-Equal 1 @($timingRecords | Where-Object { $_.invocation_kind -eq 'debug_help_probe' }).Count 'OpenCode performs one model-free debug-help probe for the preflight/execute pair' + Assert-Equal 1 @($timingRecords | Where-Object { $_.invocation_kind -eq 'debug_config_probe' }).Count 'OpenCode performs one model-free debug-config probe for the preflight/execute pair' + } + } + $isolatedHomeFailureIteration = Join-Path $recordedRoot 'opencode-isolation-failure' + New-Item -ItemType Directory -Path $isolatedHomeFailureIteration -Force | Out-Null + $isolatedHomeFailureRun = New-TestRun -IterationDirectory $isolatedHomeFailureIteration -Configuration without_skill -EvalName 'opencode-isolation-failure' + [IO.File]::WriteAllText((Join-Path $isolatedHomeFailureRun.Root 'home\opencode-fake-home'), 'fixture', [Text.UTF8Encoding]::new($false)) + $failureOldPath = $env:PATH + $recordedNodeCommand = Resolve-ExternalCommand -Name 'node' + $recordedNodeDirectory = if ($null -eq $recordedNodeCommand) { $null } else { Split-Path -Parent ([string]$recordedNodeCommand.Source) } + $failurePathParts = @($failureOldPath -split [regex]::Escape([string][IO.Path]::PathSeparator) | Where-Object { + -not [string]::IsNullOrWhiteSpace([string]$_) -and + ([string]::IsNullOrWhiteSpace([string]$recordedNodeDirectory) -or -not [string]::Equals( + ([IO.Path]::GetFullPath([string]$_)).TrimEnd([char[]]@('\', '/')), + ([IO.Path]::GetFullPath([string]$recordedNodeDirectory)).TrimEnd([char[]]@('\', '/')), + [StringComparison]::OrdinalIgnoreCase + )) + }) + $env:PATH = [string]::Join([IO.Path]::PathSeparator, @($fakeBin) + @($failurePathParts)) + try { + $failedPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'opencode\runner.ps1') -Command preflight -RunPath $isolatedHomeFailureRun.Path -ProfilePath $recordedProfiles['opencode'] + Assert-Equal 'incompatible' $failedPreflight.status 'OpenCode fails preflight when the model-free runtime-home probe resolves the fake ambient profile' + Assert-True (@($failedPreflight.checks | Where-Object { $_.name -eq 'effective_home' -and $_.status -eq 'failed' }).Count -eq 1) 'OpenCode marks the effective-home check failed for the fake profile' + $failedResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'opencode\runner.ps1') -Command execute -RunPath $isolatedHomeFailureRun.Path -ProfilePath $recordedProfiles['opencode'] + Assert-Equal 'incompatible' $failedResult.status 'OpenCode execution remains incompatible after effective-home failure' + Assert-Equal 'preflight_incompatible' $failedResult.final_response.reason 'OpenCode reports the fail-closed preflight isolation reason' + $failedLogPath = Join-Path $isolatedHomeFailureRun.Root 'repo\opencode-fake-cli-log.jsonl' + if (Test-Path -LiteralPath $failedLogPath -PathType Leaf) { + $failedRecords = @(Get-Content -LiteralPath $failedLogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 0 @($failedRecords | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }).Count 'OpenCode effective-home failure starts zero model executions' + } + } finally { + $env:PATH = $failureOldPath + } + foreach ($runnerName in @('copilot', 'opencode')) { + $unsupportedIteration = Join-Path $recordedRoot ("unsupported-scripted-{0}" -f $runnerName) + New-Item -ItemType Directory -Path $unsupportedIteration -Force | Out-Null + $unsupportedRun = New-TestRun -IterationDirectory $unsupportedIteration -Configuration with_skill -EvalName 'unsupported-scripted' -Interaction $scriptedInteraction + Add-TestInteractionSources -TestRun $unsupportedRun + [IO.File]::WriteAllText((Join-Path $unsupportedRun.Root 'home\scripted-session-fixture'), 'fixture', [Text.UTF8Encoding]::new($false)) + [IO.File]::WriteAllText((Join-Path $unsupportedRun.Root ("home\{0}-no-exact-session-help" -f $runnerName)), 'fixture', [Text.UTF8Encoding]::new($false)) + $unsupportedRunnerRelativePath = if ($runnerName -eq 'copilot') { 'github-copilot\runner.ps1' } else { 'opencode\runner.ps1' } + $unsupportedRunnerPath = Join-Path $runnerRoot $unsupportedRunnerRelativePath + $unsupportedPreflight = Invoke-AdapterJson -RunnerPath $unsupportedRunnerPath -Command preflight -RunPath $unsupportedRun.Path -ProfilePath $recordedProfiles[$runnerName] + Assert-Equal 'incompatible' $unsupportedPreflight.status "$runnerName unsupported exact-session help fails preflight" + Assert-Equal 'unsupported' $unsupportedPreflight.resolved_capabilities.scripted_multi_turn_same_session "$runnerName unsupported continuation is not advertised as supported" + Assert-True (@($unsupportedPreflight.checks | Where-Object { $_.name -eq 'scripted_multi_turn_same_session' -and $_.status -eq 'failed' }).Count -eq 1) "$runnerName records a failed scripted continuation preflight check" + Assert-True (([string]$unsupportedPreflight.protocol_observations.scripted_multi_turn_same_session.reason) -match '(?i)explicit|session|implicit') "$runnerName explains why implicit or ambiguous continuation is rejected" + $unsupportedResult = Invoke-AdapterJson -RunnerPath $unsupportedRunnerPath -Command execute -RunPath $unsupportedRun.Path -ProfilePath $recordedProfiles[$runnerName] + [void](Assert-ExecutionResult -Result $unsupportedResult) + Assert-Equal 'incompatible' $unsupportedResult.status "$runnerName unsupported continuation never executes" + $unsupportedLogPath = Join-Path $unsupportedRun.Root ("repo\{0}-fake-cli-log.jsonl" -f $runnerName) + $unsupportedRecords = @(Get-Content -LiteralPath $unsupportedLogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 0 @($unsupportedRecords | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }).Count "$runnerName unsupported continuation starts zero model invocations" + + if ($runnerName -eq 'opencode') { + $singleTurnRun = New-TestRun -IterationDirectory (Join-Path $unsupportedIteration 'single-turn') -Configuration with_skill -EvalName 'unsupported-scripted-single-turn' + [IO.File]::WriteAllText((Join-Path $singleTurnRun.Root 'home\opencode-no-exact-session-help'), 'fixture', [Text.UTF8Encoding]::new($false)) + $singleTurnPreflight = Invoke-AdapterJson -RunnerPath $unsupportedRunnerPath -Command preflight -RunPath $singleTurnRun.Path -ProfilePath $recordedProfiles[$runnerName] + Assert-Equal 'compatible' $singleTurnPreflight.status 'OpenCode single-turn preflight remains compatible when --session capability is absent' + } + } + foreach ($runnerName in @('copilot', 'opencode')) { + foreach ($failureMarker in @('scripted-no-session-first', 'scripted-session-mismatch', 'scripted-no-terminal-first')) { + $failureIteration = Join-Path $recordedRoot ("scripted-failure-{0}-{1}" -f $runnerName, ($failureMarker -replace '^scripted-', '')) + New-Item -ItemType Directory -Path $failureIteration -Force | Out-Null + $failureRun = New-TestRun -IterationDirectory $failureIteration -Configuration with_skill -EvalName 'scripted-failure' -Interaction $scriptedInteraction + Add-TestInteractionSources -TestRun $failureRun + [IO.File]::WriteAllText((Join-Path $failureRun.Root 'home\scripted-session-fixture'), 'fixture', [Text.UTF8Encoding]::new($false)) + if ($runnerName -eq 'copilot') { [IO.File]::WriteAllText((Join-Path $failureRun.Root 'home\copilot-exact-session-help'), 'fixture', [Text.UTF8Encoding]::new($false)) } + [IO.File]::WriteAllText((Join-Path $failureRun.Root ("home\{0}" -f $failureMarker)), 'fixture', [Text.UTF8Encoding]::new($false)) + $failureRunnerRelativePath = if ($runnerName -eq 'copilot') { 'github-copilot\runner.ps1' } else { 'opencode\runner.ps1' } + $failureRunnerPath = Join-Path $runnerRoot $failureRunnerRelativePath + $failureResult = Invoke-AdapterJson -RunnerPath $failureRunnerPath -Command execute -RunPath $failureRun.Path -ProfilePath $recordedProfiles[$runnerName] + [void](Assert-ExecutionResult -Result $failureResult) + Assert-Equal 'incompatible' $failureResult.status "$runnerName $failureMarker fails closed" + $expectedFailureCode = switch ($failureMarker) { 'scripted-no-session-first' { 'session_id_unobservable' } 'scripted-session-mismatch' { 'session_identity_mismatch' } 'scripted-no-terminal-first' { 'terminal_turn_status' } } + Assert-True (@($failureResult.evidence.native_worker_evidence_failures | Where-Object { $_ -eq $expectedFailureCode }).Count -gt 0) "$runnerName $failureMarker records the native interaction failure" + $failureLogPath = Join-Path $failureRun.Root ("repo\{0}-fake-cli-log.jsonl" -f $runnerName) + $failureRecords = @(Get-Content -LiteralPath $failureLogPath | ForEach-Object { $_ | ConvertFrom-Json }) + $failureExecutions = @($failureRecords | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + $expectedFailureExecutions = if ($failureMarker -eq 'scripted-session-mismatch') { 2 } else { 1 } + Assert-Equal $expectedFailureExecutions $failureExecutions.Count "$runnerName $failureMarker does not continue after an unproven first turn" + } + } + $serialOpenCodeProfile = Join-Path $recordedRoot 'opencode-serial-profile.json' + $serialOpenCodeData = Read-RunnerJson -Path $recordedProfiles['opencode'] + $serialOpenCodeData.concurrency = 1 + Write-TestJson -Path $serialOpenCodeProfile -Value $serialOpenCodeData + $serialOpenCodePreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'opencode\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $serialOpenCodeProfile + Assert-Equal 'incompatible' $serialOpenCodePreflight.status 'OpenCode rejects a serial execution profile' + Assert-True (@($serialOpenCodePreflight.reasons | Where-Object { $_ -match 'concurrency >= 2|Sequential dispatch' }).Count -gt 0) 'OpenCode serial-profile failure explains the concurrency requirement' + $staleCli = $fakeCli.Replace('"--format json`n--dir `n--model `n--auto`n--variant `n--session continue by session id"', '"--format json`n--dir `n--model `n--variant `n--session continue by session id"') + [System.IO.File]::WriteAllText((Join-Path $fakeBin 'opencode.ps1'), $staleCli, [System.Text.UTF8Encoding]::new($false)) + $stalePreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'opencode\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['opencode'] + Assert-Equal 'incompatible' $stalePreflight.status 'stale OpenCode help contract is rejected during preflight' + Assert-True (@($stalePreflight.reasons | Where-Object { $_ -match '--auto' }).Count -gt 0) 'stale OpenCode option failure identifies the missing flag' + [System.IO.File]::WriteAllText((Join-Path $fakeBin 'opencode.ps1'), $fakeCli, [System.Text.UTF8Encoding]::new($false)) + $fileAuthHome = Join-Path $recordedRoot 'codex-file-auth' + New-Item -ItemType Directory -Path $fileAuthHome -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'auth.json'), '{"canary":"not-logged"}', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'config.toml'), 'model = "ambient-not-used"', [System.Text.UTF8Encoding]::new($false)) + New-Item -ItemType Directory -Path (Join-Path $fileAuthHome 'skills'), (Join-Path $fileAuthHome 'agents'), (Join-Path $fileAuthHome 'sessions'), (Join-Path $fileAuthHome 'memories'), (Join-Path $fileAuthHome 'plugins'), (Join-Path $fileAuthHome 'mcp') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'skills\ambient.md'), 'ambient skill must not be copied', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'agents\ambient.md'), 'ambient agent must not be copied', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'mcp.json'), '{"ambient":true}', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'AGENTS.md'), 'ambient instructions must not be copied', [System.Text.UTF8Encoding]::new($false)) + $env:OPENAI_API_KEY = $null + $env:CODEX_HOME = $fileAuthHome + $fileAuthPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'compatible' $fileAuthPreflight.status 'Codex subscription auth is accepted through app-server' + Assert-True (@($fileAuthPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Codex subscription authentication is explicit in preflight' + $fileAuthResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'completed' $fileAuthResult.status ("Codex subscription app-server execution completes (native_failures={0}; failure={1})" -f ([string]::Join(',', @($fileAuthResult.evidence.native_worker_evidence_failures))), ([string](Get-JsonProperty -Object $fileAuthResult.exit.failure -Name 'message' -Default ''))) + Assert-Equal 'recorded subscription response' $fileAuthResult.final_response.text 'Codex app-server captures the final agent message' + Assert-Equal 'recorded-subscription-thread' $fileAuthResult.session.id 'Codex app-server preserves thread identity' + Assert-Equal 'recorded-subscription-turn' $fileAuthResult.evidence.turn_id 'Codex app-server preserves turn identity' + Assert-Equal 'available' $fileAuthResult.telemetry.tokens.status 'Codex app-server maps token usage notifications' + Assert-Equal 2 ([int]$fileAuthResult.telemetry.tokens.value.input_tokens) 'Codex app-server maps input token usage' + Assert-Equal 3 ([int]$fileAuthResult.telemetry.tokens.value.output_tokens) 'Codex app-server maps output token usage' + Assert-Equal 2 ([int]$fileAuthResult.telemetry.tool_calls.value) 'Codex app-server counts command and file-change evidence' + Assert-Equal 1 @($fileAuthResult.evidence.commands).Count 'Codex app-server preserves command evidence' + Assert-Equal 1 @($fileAuthResult.evidence.files).Count 'Codex app-server preserves file-change evidence' + Assert-Equal 'runner' $fileAuthResult.evidence.delegation.dispatch_owner 'Codex native evidence identifies runner-owned dispatch' + Assert-Equal 'gpt-5.6-luna' $fileAuthResult.evidence.delegation.observed_model 'Codex native evidence uses observed thread/start model' + Assert-True (Test-ExactObservedPath -Expected $fileAuthResult.evidence.execution_paths.physical_working_directory -Observed $fileAuthResult.evidence.delegation.observed_working_directory) 'Codex native evidence uses the runner-declared physical cwd' + Assert-Equal (Join-Path $with.Root 'repo') $fileAuthResult.evidence.execution_paths.logical_working_directory 'Codex evidence preserves the logical package cwd' + Assert-True ([bool]$fileAuthResult.evidence.delegation.fresh_worker) 'Codex native evidence proves ephemeral fresh worker' + Assert-True ([bool]$fileAuthResult.evidence.delegation.home_config_isolated) 'Codex native evidence proves auth-only home cleanup' + Assert-True ([bool]$fileAuthResult.evidence.delegation.prompt_fidelity) 'Codex native evidence proves exact prompt hash' + Assert-True ([bool]$fileAuthResult.evidence.delegation.terminal_result_capture) 'Codex native evidence proves turn completion capture' + Assert-Equal 'harness_native_transport' $fileAuthResult.evidence.capture.source 'Codex capture provenance is app-server transport-owned' + Assert-True ([bool]$fileAuthResult.evidence.capture.terminal -and -not [bool]$fileAuthResult.evidence.capture.worker_authored) 'Codex capture is terminal and not authored by the worker/orchestrator' + Assert-True ([bool]$fileAuthResult.evidence.delegation.thread_read_observed) 'Codex native evidence records thread/read observation' + Assert-Equal 'thread/start' $fileAuthResult.evidence.app_server.thread_start_request.method 'Codex evidence retains the exact thread/start request' + Assert-Equal 'turn/start' $fileAuthResult.evidence.app_server.turn_start_request.method 'Codex evidence retains the exact turn/start request' + Assert-Equal 'gpt-5.6-luna' $fileAuthResult.evidence.app_server.thread_start_request.params.model 'Codex exact thread/start request preserves model' + Assert-Equal $fileAuthResult.evidence.execution_paths.physical_working_directory $fileAuthResult.evidence.app_server.turn_start_request.params.cwd 'Codex exact turn/start request preserves the physical cwd' + Assert-Equal 'gpt-5.6-luna' $fileAuthResult.evidence.app_server.thread_start_response.result.model 'Codex evidence retains observed thread/start model' + Assert-Equal 'recorded-subscription-thread' $fileAuthResult.evidence.app_server.thread_start_response.result.thread.id 'Codex parses the installed thread/start.result.thread shape' + Assert-Equal 'recorded-subscription-turn' $fileAuthResult.evidence.app_server.turn_start_response.result.turn.id 'Codex parses the installed turn/start.result.turn shape' + Assert-Equal 'completed' $fileAuthResult.evidence.app_server.terminal_turn.status 'Codex evidence retains terminal turn/completed status' + Assert-Equal 'recorded-subscription-thread' $fileAuthResult.evidence.app_server.thread_read.request.threadId 'Codex evidence retains the thread/read request identity' + Assert-Equal 'recorded-subscription-thread' $fileAuthResult.evidence.app_server.thread_read.response.id 'Codex parses the installed thread/read.result.thread shape' + Assert-True ($fileAuthResult.evidence.app_server.thread_start_request.params.PSObject.Properties.Name -notcontains 'allowProviderModelFallback') 'Codex does not invent unsupported provider fallback control' + Assert-True (@($fileAuthResult.evidence.app_server.thread_start.instruction_sources | Where-Object { (Test-PathInside -BasePath $fileAuthResult.evidence.execution_paths.physical_run_root -CandidatePath $_) }).Count -eq 1) 'allowed staged instruction source is inside the physical arm boundary' + Assert-Equal 'physical_projection_boundary' $fileAuthResult.evidence.delegation.instruction_source_proof 'Codex records the independent physical instruction boundary proof' + Assert-True (-not (Test-Path -LiteralPath $fileAuthResult.evidence.execution_paths.physical_run_root)) 'Codex removes the temporary physical projection after capture' + $fileAuthEvidenceJson = ConvertTo-Json -InputObject $fileAuthResult -Depth 100 + Assert-True ($fileAuthEvidenceJson -notmatch 'recorded-canary|not-logged' -and $fileAuthEvidenceJson -notmatch [regex]::Escape($fileAuthHome)) 'Codex result evidence does not include the copied credential or auth path' + $subscriptionLogPath = Join-Path $with.Root 'repo\codex-fake-cli-log.jsonl' + $subscriptionRecord = Get-Content -LiteralPath $subscriptionLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { $_.PSObject.Properties.Name -contains 'rpc_methods' } | Select-Object -Last 1 + $codexNativeValidation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $fileAuthResult -Run (Resolve-RunContract -RunPath $with.Path) -RequestedModel 'gpt-5.6-luna' -ExpectedRunner 'codex' -ExpectedMechanism $fileAuthResult.evidence.delegation.mechanism + Assert-True ([bool]$codexNativeValidation.Valid) 'Codex app-server result satisfies the common native terminal evidence contract' + [void](Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $fileAuthResult -Run (Resolve-RunContract -RunPath $with.Path) -RequestedModel 'gpt-5.6-luna' -ExpectedRunner 'codex' -ExpectedMechanism $fileAuthResult.evidence.delegation.mechanism) + [void](Assert-NativeTerminalCaptureArtifact -ExecutionResult $fileAuthResult) + $authHomePath = [string]$subscriptionRecord.parent_codex_home + Assert-True (-not (Test-Path -LiteralPath $authHomePath)) 'Codex temporary auth-only home is removed after the arm completes' + + $threadReadUnavailableMarker = Join-Path $with.Root 'home\codex-thread-read-unavailable' + [System.IO.File]::WriteAllText($threadReadUnavailableMarker, 'fixture', [System.Text.UTF8Encoding]::new($false)) + $threadReadUnavailableResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'completed' $threadReadUnavailableResult.status 'Optional thread/read absence does not invalidate a proven terminal turn' + Assert-Equal 'unavailable_optional' $threadReadUnavailableResult.evidence.app_server.thread_read.observation 'Codex records missing thread/read as unavailable supplemental evidence' + Assert-True (@($threadReadUnavailableResult.evidence.native_worker_evidence_failures | Where-Object { $_ -eq 'thread_read_metadata' }).Count -eq 0) 'Optional thread/read absence does not add a false incompatibility' + Remove-Item -LiteralPath $threadReadUnavailableMarker -Force + + $rerouteMarker = Join-Path $with.Root 'home\codex-reroute' + [System.IO.File]::WriteAllText($rerouteMarker, 'fixture', [System.Text.UTF8Encoding]::new($false)) + $reroutedResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'incompatible' $reroutedResult.status 'Codex model/rerouted notification fails closed' + Assert-True (@($reroutedResult.evidence.native_worker_evidence_failures | Where-Object { $_ -eq 'model_rerouted' }).Count -eq 1) 'Codex reroute incompatibility is recorded as transport evidence' + Assert-True ([string]$reroutedResult.exit.failure.message -match 'model_rerouted') 'Codex reroute reason survives in normalized exit failure' + Remove-Item -LiteralPath $rerouteMarker -Force + + $ambientInstructionMarker = Join-Path $with.Root 'home\codex-ambient-instruction' + [System.IO.File]::WriteAllText($ambientInstructionMarker, 'fixture', [System.Text.UTF8Encoding]::new($false)) + $ambientInstructionResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'incompatible' $ambientInstructionResult.status 'Codex unexpected instruction source fails closed' + Assert-True (@($ambientInstructionResult.evidence.native_worker_evidence_failures | Where-Object { $_ -eq 'unexpected_instruction_sources' }).Count -eq 1) 'Codex ambient instruction rejection is recorded as transport evidence' + Assert-True ([string]$ambientInstructionResult.exit.failure.message -match 'unexpected_instruction_sources') 'Codex instruction-source reason survives in normalized exit failure' + Remove-Item -LiteralPath $ambientInstructionMarker -Force + Assert-Equal 'initialize,initialized,thread/start,turn/start,thread/read' ([string]::Join(',', @($subscriptionRecord.rpc_methods))) 'Codex app-server follows the required handshake and post-completion read order' + Assert-Equal 'gpt-5.6-luna' $subscriptionRecord.thread_params.model 'Codex app-server thread receives the requested model' + Assert-True ([bool]$subscriptionRecord.thread_params.ephemeral) 'Codex app-server thread is ephemeral' + Assert-Equal 'read-only' $subscriptionRecord.thread_params.sandbox 'Codex app-server thread uses the installed request enum' + Assert-Equal 'gpt-5.6-luna' $subscriptionRecord.turn_params.model 'Codex app-server turn receives the requested model' + Assert-Equal 'medium' $subscriptionRecord.turn_params.effort 'Codex app-server turn receives the requested reasoning effort' + Assert-Equal $fileAuthResult.evidence.execution_paths.physical_working_directory $subscriptionRecord.turn_params.cwd 'Codex app-server turn receives the physical working directory' + Assert-Equal 'never' $subscriptionRecord.turn_params.approvalPolicy 'Codex app-server turn rejects interactive approvals' + Assert-Equal 'workspaceWrite' $subscriptionRecord.turn_params.sandboxPolicy.type 'Codex app-server turn receives workspace-write sandbox policy' + Assert-True ($subscriptionRecord.parent_codex_home -ne $fileAuthHome) 'Codex app-server does not expose the ambient subscription CODEX_HOME' + Assert-True ([bool]$subscriptionRecord.parent_auth_file_visible) 'Codex app-server parent can read the subscription auth file' + Assert-True ([bool]$subscriptionRecord.auth_only_home) 'Codex app-server temporary CODEX_HOME contains auth.json only' + Assert-True (-not [bool]$subscriptionRecord.parent_config_file_visible) 'Codex app-server temporary CODEX_HOME excludes config.toml' + Assert-True (-not [bool]$subscriptionRecord.parent_skills_directory_visible) 'Codex app-server temporary CODEX_HOME excludes skills' + Assert-True (-not [bool]$subscriptionRecord.parent_agents_directory_visible) 'Codex app-server temporary CODEX_HOME excludes agents' + Assert-True (-not [bool]$subscriptionRecord.parent_sessions_directory_visible) 'Codex app-server temporary CODEX_HOME excludes sessions' + Assert-True (-not [bool]$subscriptionRecord.parent_memories_directory_visible) 'Codex app-server temporary CODEX_HOME excludes memories' + Assert-True (-not [bool]$subscriptionRecord.parent_plugins_directory_visible) 'Codex app-server temporary CODEX_HOME excludes plugins' + Assert-True (-not [bool]$subscriptionRecord.parent_mcp_configuration_visible) 'Codex app-server temporary CODEX_HOME excludes MCP configuration' + Assert-True (-not [bool]$subscriptionRecord.parent_agents_file_visible) 'Codex app-server temporary CODEX_HOME excludes AGENTS.md' + Assert-True (-not [bool]$subscriptionRecord.unrelated_present) 'Codex app-server parent excludes unrelated inherited environment variables' + Assert-True (-not [bool]$subscriptionRecord.worker_auth_file_visible) 'Codex app-server worker fixture does not receive auth.json' + Assert-True (@($subscriptionRecord.args) -contains 'shell_environment_policy.inherit=none') 'Codex app-server disables child shell environment inheritance' + $env:OPENAI_API_KEY = 'recorded-canary-not-logged' + $env:CODEX_HOME = $recordedOldCodexHome + # GitHub Copilot authentication: explicit env, OS-keychain, GitHub CLI, and + # no-auth fixtures are all deterministic and contain no credential values. + $env:COPILOT_GITHUB_TOKEN = $null + $env:GH_TOKEN = $null + $env:GITHUB_TOKEN = $null + $missingGhConfig = Join-Path $recordedRoot 'missing-github-cli-auth' + + # The fixture marker is fake-CLI input only; it models a positive OS + # keychain lookup without naming or reading a real credential-store file. + $copilotKeychainHome = Join-Path $recordedRoot 'copilot-keychain-home' + New-Item -ItemType Directory -Path $copilotKeychainHome -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $with.Root 'home\.copilot') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available'), 'fixture marker only', [Text.UTF8Encoding]::new($false)) + $env:COPILOT_HOME = $copilotKeychainHome + $env:GH_CONFIG_DIR = $missingGhConfig + $copilotKeychainPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotKeychainPreflight.status 'Copilot tokenless OS-keychain authentication remains compatible' + Assert-True (@($copilotKeychainPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'unavailable' }).Count -eq 1) 'Copilot preflight leaves native keychain readiness conditional' + Assert-True (@($copilotKeychainPreflight.warnings | Where-Object { $_ -match 'cannot be proven' }).Count -gt 0) 'Copilot preflight explains the unverified keychain/service boundary' + $copilotKeychainResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'completed' $copilotKeychainResult.status 'Copilot keychain fixture executes without an exported token' + $keychainRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'copilot_authentication_source' -Default '') -eq 'os_keychain' }) + Assert-Equal 1 $keychainRecords.Count 'Copilot fake observes the simulated OS-keychain path' + Remove-Item -LiteralPath (Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available') -Force + + $copilotGhFallbackHome = Join-Path $recordedRoot 'copilot-gh-fallback-home' + New-Item -ItemType Directory -Path $copilotGhFallbackHome -Force | Out-Null + $copilotGhConfig = Join-Path $recordedRoot 'copilot-gh-config' + New-Item -ItemType Directory -Path $copilotGhConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $copilotGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:COPILOT_HOME = $copilotGhFallbackHome + $env:GH_CONFIG_DIR = $copilotGhConfig + $copilotGhPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotGhPreflight.status 'Copilot GitHub CLI fallback remains compatible' + $copilotGhResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'completed' $copilotGhResult.status 'Copilot GitHub CLI fallback fixture executes without an exported token' + Assert-True $copilotGhResult.evidence.credential.github_cli_token_resolved 'Copilot records GitHub CLI token fallback without storing the token value' + Assert-True (-not $copilotGhResult.evidence.credential.github_cli_config_forwarded) 'Copilot GitHub CLI fallback does not forward host GH_CONFIG_DIR' + $ghRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }) + Assert-Equal 1 $ghRecords.Count 'Copilot fake observes only the protected GH_TOKEN produced by trusted GitHub CLI fallback' + + $copilotNoAuthHome = Join-Path $recordedRoot 'copilot-no-auth-home' + New-Item -ItemType Directory -Path $copilotNoAuthHome -Force | Out-Null + $env:COPILOT_HOME = $copilotNoAuthHome + $env:GH_CONFIG_DIR = $missingGhConfig + $copilotNoAuthPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotNoAuthPreflight.status 'Copilot preflight does not require an exported token when native auth is not observable' + Assert-True (@($copilotNoAuthPreflight.warnings | Where-Object { $_ -match 'conditional' }).Count -gt 0) 'Copilot no-auth preflight is explicitly conditional' + $copilotNoAuthResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'failed' $copilotNoAuthResult.status 'Copilot no-auth execution failure is captured without a model request' + Assert-Equal 'copilot_os_keychain_or_github_cli_unverified' $copilotNoAuthResult.evidence.credential.source 'Copilot no-auth evidence does not claim authentication' + Assert-True (($copilotNoAuthResult | ConvertTo-Json -Depth 100) -notmatch 'ambient-profile-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot authentication fixtures never expose credential values' + $env:COPILOT_HOME = $recordedOldCopilotHome + Write-Output 'Real runner deterministic adapter conformance: PASS' +} finally { + $env:PATH = $recordedOldPath + $env:OPENAI_API_KEY = $recordedOldOpenAi + $env:CODEX_HOME = $recordedOldCodexHome + $env:AGENTIC_GLOBAL_SECRET = $recordedOldGlobalSecret + $env:OPENCODE_DISABLE_PROJECT_CONFIG = $recordedOldProjectDisable + $env:COPILOT_GITHUB_TOKEN = $recordedOldCopilotToken + $env:GH_TOKEN = $recordedOldGhToken + $env:GITHUB_TOKEN = $recordedOldGithubToken + $env:COPILOT_HOME = $recordedOldCopilotHome + $env:GH_CONFIG_DIR = $recordedOldGhConfigDir + $env:AGENTIC_RECORDED_FIXTURES = $recordedOldFixtures + if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } +} +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -ne [string]$Actual) { throw "ASSERT: $Message (expected '$Expected', got '$Actual')" } +} + +function Assert-Throws { + param([scriptblock]$Action, [string]$Message) + $thrown = $false + try { & $Action } catch { $thrown = $true } + if (-not $thrown) { throw "ASSERT: $Message" } +} + +function Write-TestJson { + param([string]$Path, [object]$Value) + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + [System.IO.File]::WriteAllText($Path, ((ConvertTo-Json -InputObject $Value -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Invoke-Fake { + param( + [string]$FakePath, + [string]$Command, + [string]$RunPath, + [string]$ProfilePath, + [string]$Scenario = '' + ) + + $arguments = @('-NoProfile', '-File', $FakePath, $Command, '-Run', $RunPath, '-Profile', $ProfilePath) + if (-not [string]::IsNullOrWhiteSpace($Scenario)) { $arguments += @('-Scenario', $Scenario) } + $output = & pwsh @arguments + if ($LASTEXITCODE -ne 0) { throw "Fake runner '$Command' failed: $([string]::Join(' ', @($output)))" } + $json = [string]::Join([Environment]::NewLine, @($output)) + if ([string]::IsNullOrWhiteSpace($json)) { throw "Fake runner '$Command' returned no JSON." } + return $json | ConvertFrom-Json +} + +function Get-TestTreeHash { + param([Parameter(Mandatory = $true)][string]$Root) + + $entries = foreach ($file in @(Get-ChildItem -LiteralPath $Root -Recurse -File -Force | Sort-Object FullName)) { + $relative = [System.IO.Path]::GetRelativePath($Root, $file.FullName).Replace('\', '/') + "$relative`:$((Get-Sha256HexFromFile -Path $file.FullName))" + } + $joined = [string]::Join("`n", @($entries | Sort-Object)) + return Get-Sha256HexFromBytes -Bytes ([System.Text.UTF8Encoding]::new($false).GetBytes($joined)) +} + +function New-TestRun { + param( + [string]$IterationDirectory, + [ValidateSet('with_skill', 'without_skill')][string]$Configuration, + [string]$EvalName = 'conformance', + [object]$Interaction = $null + ) + + $evalDirectory = Join-Path $IterationDirectory 'conformance' + $runRoot = Join-Path $evalDirectory $Configuration + $repo = Join-Path $runRoot 'repo' + $homeDirectory = Join-Path $runRoot 'home' + New-Item -ItemType Directory -Path $repo,$homeDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'README.txt'), 'isolated home', [System.Text.UTF8Encoding]::new($false)) + New-Item -ItemType Directory -Path (Join-Path $repo '.github') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $repo 'AGENTS.md'), '# repo-owned-agent-instruction', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $repo '.github\copilot-instructions.md'), '# repo-owned-copilot-instruction', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $repo 'opencode.json'), '{"fixture_project_config":true}', [System.Text.UTF8Encoding]::new($false)) + $prompt = "# task`r`n`r`nByte fidelity: Δ and emoji 🚀.`r`n" + ("large-prompt-line-0123456789`r`n" * 4096) + [System.IO.File]::WriteAllBytes((Join-Path $runRoot 'prompt.md'), [System.Text.UTF8Encoding]::new($false).GetBytes($prompt)) + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'expected-prompt-sha256.txt'), (Get-Sha256HexFromFile -Path (Join-Path $runRoot 'prompt.md')), [System.Text.UTF8Encoding]::new($false)) + if ($Configuration -eq 'with_skill') { + $skill = Join-Path $runRoot 'skill\candidate' + New-Item -ItemType Directory -Path $skill -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $skill 'SKILL.md'), '# candidate', [System.Text.UTF8Encoding]::new($false)) + $skillDirectory = 'skill/candidate' + $skillHash = Get-TestTreeHash -Root $skill + } else { + $skillDirectory = $null + $skillHash = $null + } + $run = [ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = 1 + evalName = $EvalName + skillName = if ($Configuration -eq 'with_skill') { 'candidate' } else { $null } + iteration = 1 + mode = $Configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $skillDirectory + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + inputFiles = @() + fixtureHash = Get-TestTreeHash -Root $repo + skillHash = $skillHash + contract = [ordered]@{ + sandboxRoot = '.' + workingDirectory = 'repo' + homeDirectory = 'home' + mustNotReadOutsideSandbox = $true + mustNotExposeGlobalSkillsOrConfig = $true + } + } + if ($null -ne $Interaction) { + $interactionPath = Join-Path $runRoot 'interaction.json' + Write-TestJson -Path $interactionPath -Value $Interaction + $run.interactionFile = 'interaction.json' + $run.interactionHash = Get-Sha256HexFromFile -Path $interactionPath + } + $path = Join-Path $runRoot 'run.json' + Write-TestJson -Path $path -Value $run + return [pscustomobject]@{ Root = $runRoot; Path = $path; Contract = $run } +} + +function Add-TestInteractionSources { + param([Parameter(Mandatory = $true)][object]$TestRun) + + $runJson = Read-RunnerJson -Path $TestRun.Path + $futureCanary = 'CODEBELT_FUTURE_TURN_CANARY_' + ([string]$runJson.interactionHash).Substring(0, 16).ToUpperInvariant() + $interaction = Read-RunnerJson -Path (Join-Path $TestRun.Root ([string]$runJson.interactionFile)) + foreach ($turn in @($interaction.turns)) { + $source = [string](Get-JsonProperty -Object $turn -Name 'source' -Default '') + if ([string]::IsNullOrWhiteSpace($source)) { continue } + $sourcePath = Join-Path $TestRun.Root ($source -replace '/', [IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $sourcePath) -Force | Out-Null + [IO.File]::WriteAllText($sourcePath, "recorded scripted turn two $futureCanary", [Text.UTF8Encoding]::new($false)) + } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-runner-conformance-' + [Guid]::NewGuid().ToString('N')) +try { + $iteration = Join-Path $testRoot 'iteration-1' + New-Item -ItemType Directory -Path $iteration -Force | Out-Null + $canaryPath = Join-Path $testRoot 'eval-metadata.json' + [System.IO.File]::WriteAllText($canaryPath, 'grading-key-canary', [System.Text.UTF8Encoding]::new($false)) + $globalRoot = Join-Path $testRoot 'seeded-global-profile' + New-Item -ItemType Directory -Path $globalRoot -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $globalRoot 'same-name-SKILL.md'), 'must remain invisible', [System.Text.UTF8Encoding]::new($false)) + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_GLOBAL_RULES', $globalRoot, 'Process') + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_MEMORY', 'seeded-memory', 'Process') + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_PLUGINS', 'seeded-plugins', 'Process') + + $profilePath = Join-Path $iteration 'execution-profile.json' + Write-TestJson -Path $profilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fake' + model = 'fixture-model' + reasoning_effort = 'high' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) + $unsupportedProfilePath = Join-Path $iteration 'unsupported-profile.json' + Write-TestJson -Path $unsupportedProfilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fake' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'unsupported' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) + $legacyProviderProfilePath = Join-Path $iteration 'legacy-provider-profile.json' + Write-TestJson -Path $legacyProviderProfilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fake' + provider = 'fixture-provider' + model = 'fixture-model' + reasoning_effort = 'high' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) + + $with = New-TestRun -IterationDirectory $iteration -Configuration with_skill + $without = New-TestRun -IterationDirectory $iteration -Configuration without_skill + $fakePath = Join-Path $runnerRoot 'fake\runner.ps1' + + $descriptor = Invoke-Fake -FakePath $fakePath -Command describe -Run $with.Path -Profile $profilePath + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + Assert-Equal 'fake' $descriptor.name 'descriptor identity' + Assert-Equal (Get-RunnerSchemaNames).Protocol $descriptor.protocol_version 'descriptor protocol' + Assert-Equal 'unsupported' $descriptor.capabilities.native_worker_delegation 'deterministic fake does not advertise a native delegation surface' + Assert-Throws { Assert-RunnerDescriptor -Descriptor ([pscustomobject]@{ schema = $descriptor.schema; protocol_version = 'changed'; name = 'fake' }) } 'changed protocol must fail descriptor validation' + Assert-Throws { Resolve-ExecutionProfile -ProfilePath $legacyProviderProfilePath } 'execution profile rejects the removed provider field' + + $unsupported = Invoke-Fake -FakePath $fakePath -Command preflight -Run $with.Path -Profile $unsupportedProfilePath + Assert-Equal 'incompatible' $unsupported.status 'unsupported capability/profile must fail during preflight' + Assert-True (@($unsupported.reasons).Count -gt 0) 'incompatible preflight must explain its reason' + + $withResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $with.Path -Profile $profilePath + $withoutResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath + foreach ($result in @($withResult, $withoutResult)) { + [void](Assert-ExecutionResult -Result $result) + Assert-Equal 'completed' $result.status 'normal completion status' + Assert-True $result.session.fresh 'fresh session flag' + Assert-True (-not $result.session.resumed) 'resume must be false' + Assert-Equal 1 $result.attempt_count 'answer-quality retry is forbidden' + Assert-True ($result.requested.PSObject.Properties.Name -notcontains 'provider') 'portable execution result must not expose provider' + Assert-Equal 'fixture-model' $result.requested.model 'model must pass unchanged' + Assert-Equal 'high' $result.requested.reasoning_effort 'reasoning effort must pass unchanged' + Assert-Equal 'isolated-default' $result.requested.configuration_profile 'configuration profile must pass unchanged' + Assert-Equal 'default' $result.requested.tool_profile 'tool profile must pass unchanged' + Assert-Equal 'unavailable' $result.telemetry.tokens.status 'missing token telemetry must be explicit' + Assert-True ($result.telemetry.tokens.PSObject.Properties.Name -notcontains 'value') 'missing token telemetry must not contain a zero placeholder' + foreach ($artifact in @($result.artifacts)) { + Assert-True ($artifact.path -notmatch '(^|/|\\)\.\.(/|\\|$)') 'artifact path must not escape the run' + $artifactPath = Join-Path $($with.Root) ($artifact.path -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if ($result.run.configuration -eq 'without_skill') { $artifactPath = Join-Path $($without.Root) ($artifact.path -replace '/', [System.IO.Path]::DirectorySeparatorChar) } + Assert-True (Test-Path -LiteralPath $artifactPath -PathType Leaf) 'artifact must exist inside its run' + Assert-Equal $artifact.sha256 ((Get-FileHash -Algorithm SHA256 -LiteralPath $artifactPath).Hash.ToLowerInvariant()) 'artifact hash' + Assert-Equal $artifact.size (Get-Item -LiteralPath $artifactPath).Length 'artifact size' + Assert-True (-not [string]::IsNullOrWhiteSpace($artifact.media_type)) 'artifact media type' + } + } + Assert-True ($withResult.session.id -ne $withoutResult.session.id) 'paired arms must have distinct session ids' + Assert-True ($withResult.run.configuration -ne $withoutResult.run.configuration) 'paired arms must retain distinct configurations' + + $withPromptEvidence = Get-Content (Join-Path $with.Root 'evidence\prompt-delivery.json') -Raw | ConvertFrom-Json + $withoutPromptEvidence = Get-Content (Join-Path $without.Root 'evidence\prompt-delivery.json') -Raw | ConvertFrom-Json + foreach ($pair in @( + [pscustomobject]@{ Run = $with; Evidence = $withPromptEvidence; Result = $withResult } + [pscustomobject]@{ Run = $without; Evidence = $withoutPromptEvidence; Result = $withoutResult } + )) { + $promptBytes = [System.IO.File]::ReadAllBytes((Join-Path $pair.Run.Root 'prompt.md')) + Assert-Equal (Get-Sha256HexFromBytes -Bytes $promptBytes) $pair.Evidence.first_task_input_sha256 'prompt must be the first task input byte-for-byte' + Assert-Equal $promptBytes.Length $pair.Evidence.first_task_input_bytes 'prompt byte length' + Assert-True $pair.Evidence.byte_exact 'prompt fidelity evidence' + Assert-Equal (Join-Path $pair.Run.Root 'repo') $pair.Evidence.working_directory 'working directory' + Assert-Equal (Join-Path $pair.Run.Root 'home') $pair.Evidence.home_directory 'isolated home' + Assert-True (-not $pair.Evidence.global_rules_visible -and -not $pair.Evidence.global_memory_visible -and -not $pair.Evidence.global_plugins_visible -and -not $pair.Evidence.global_same_name_skill_visible) 'seeded ambient rules/memory/plugins/skill must remain invisible' + } + Assert-True $withPromptEvidence.candidate_skill_exposed 'candidate skill is exposed only for with_skill' + Assert-True (-not $withoutPromptEvidence.candidate_skill_exposed) 'candidate skill is excluded for without_skill' + Assert-True (Test-Path -LiteralPath (Join-Path $with.Root 'skill\candidate\SKILL.md')) 'with_skill has staged skill' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $without.Root 'skill'))) 'without_skill has no staged skill' + + $escape = Invoke-Fake -FakePath $fakePath -Command execute -Run $with.Path -Profile $profilePath -Scenario escape + $escapeEvidence = Get-Content (Join-Path $with.Root 'evidence\boundary-probes.json') -Raw | ConvertFrom-Json + Assert-True $escapeEvidence.read_outside_run.attempted 'read escape probe was exercised' + Assert-True $escapeEvidence.read_outside_run.blocked 'read escape probe was blocked' + Assert-True $escapeEvidence.write_outside_run.attempted 'write escape probe was exercised' + Assert-True $escapeEvidence.write_outside_run.blocked 'write escape probe was blocked' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $testRoot 'escape-write.txt'))) 'escape write did not create a file' + + $refusal = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario refusal + Assert-Equal 'completed' $refusal.status 'refusal is a completed captured response' + Assert-True $refusal.final_response.text.Contains('cannot') 'refusal response is retained' + $timeout = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario timeout + Assert-Equal 'timed_out' $timeout.status 'timeout normalization' + Assert-Equal 'unavailable' $timeout.final_response.status 'timeout has no final response' + Assert-True ($null -eq $timeout.exit.status) 'timeout exit status is unavailable' + $failure = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario failure + Assert-Equal 'failed' $failure.status 'harness failure normalization' + Assert-Equal 17 $failure.exit.status 'harness failure exit status' + $incompatible = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario incompatible + Assert-Equal 'incompatible' $incompatible.status 'incompatible normalization' + $unknown = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario unknown-event + Assert-True (@($unknown.warnings | Where-Object { $_ -match 'future\.event\.v99' }).Count -gt 0) 'unknown events produce explicit warnings' + + $resolvedWith = Resolve-RunContract -RunPath $with.Path + $resolvedProfile = Resolve-ExecutionProfile -ProfilePath $profilePath + $mandatoryCapabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + } + $pragmaticCapabilities = [ordered]@{} + foreach ($name in $mandatoryCapabilities.Keys) { $pragmaticCapabilities[$name] = $mandatoryCapabilities[$name] } + $pragmaticCapabilities['filesystem_confinement'] = 'unsupported' + $strictCapabilities = [ordered]@{} + foreach ($name in $mandatoryCapabilities.Keys) { $strictCapabilities[$name] = $mandatoryCapabilities[$name] } + $strictCapabilities['filesystem_confinement'] = 'supported' + $pragmaticResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status completed -FinalResponse 'pragmatic response' -ExitStatus ([Nullable[int]]0) -IsolationCapabilities $pragmaticCapabilities -AttemptCount 1 + [void](Assert-ExecutionResult -Result $pragmaticResult) + Assert-Equal 'completed' $pragmaticResult.status 'pragmatic completed result remains usable' + Assert-Equal 'pragmatic' $pragmaticResult.isolation.level 'missing hard confinement downgrades confidence' + Assert-True (-not $pragmaticResult.isolation.hard_filesystem_confinement) 'pragmatic result does not claim hard confinement' + $strictResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status completed -FinalResponse 'strict response' -ExitStatus ([Nullable[int]]0) -IsolationCapabilities $strictCapabilities -AttemptCount 1 + [void](Assert-ExecutionResult -Result $strictResult) + Assert-Equal 'strict' $strictResult.isolation.level 'proven hard confinement reports strict isolation' + Assert-True $strictResult.isolation.hard_filesystem_confinement 'strict result claims hard confinement' + $failedResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status failed -ExitStatus ([Nullable[int]]17) -Failure (New-ExecutionFailure -Code 'fixture_failure' -Message 'fixture failure') -IsolationCapabilities $pragmaticCapabilities -AttemptCount 1 + [void](Assert-ExecutionResult -Result $failedResult) + Assert-Equal 'failed' $failedResult.status 'failed execution keeps proven pragmatic isolation' + Assert-Equal 'verified' $failedResult.isolation.status 'failed execution retains control verification' + $timedOutResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status timed_out -IsolationCapabilities $strictCapabilities -AttemptCount 1 + [void](Assert-ExecutionResult -Result $timedOutResult) + Assert-Equal 'timed_out' $timedOutResult.status 'timed out execution keeps proven strict isolation' + $missingCapability = [ordered]@{} + foreach ($name in $mandatoryCapabilities.Keys) { $missingCapability[$name] = $mandatoryCapabilities[$name] } + $missingCapability.Remove('response_capture') + $rejectedResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status completed -FinalResponse 'must be rejected' -IsolationCapabilities $missingCapability -AttemptCount 1 + [void](Assert-ExecutionResult -Result $rejectedResult) + Assert-Equal 'incompatible' $rejectedResult.status 'unproven mandatory control rejects completion' + Assert-Equal 'unverified' $rejectedResult.isolation.status 'rejected completion is unverified' + Assert-Equal 'unsupported' $rejectedResult.isolation.level 'rejected completion has unsupported isolation' + $preflightRejected = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status incompatible -FinalResponseReason 'preflight_incompatible' -IsolationCapabilities ([ordered]@{}) -AttemptCount 1 + [void](Assert-ExecutionResult -Result $preflightRejected) + Assert-Equal 'unverified' $preflightRejected.isolation.status 'preflight incompatibility is unverified' + Assert-Equal 'unsupported' $preflightRejected.isolation.level 'preflight incompatibility is unsupported' + $translated = Get-SandboxVisiblePath -HostPath (Join-Path $with.Root 'repo\file.txt') -RunRoot $with.Root -Platform 'linux' + Assert-Equal '/run/repo/file.txt' $translated 'Linux hard sandbox paths use the child namespace' + $macPath = Get-SandboxVisiblePath -HostPath (Join-Path $with.Root 'repo\file.txt') -RunRoot $with.Root -Platform 'macos' + Assert-Equal ([System.IO.Path]::GetFullPath((Join-Path $with.Root 'repo\file.txt'))) $macPath 'macOS sandbox paths remain host-visible' + Assert-Equal 'recorded-cli 1.2.3' (Get-ObservableVersionFromText "`nrecorded-cli 1.2.3`n") 'observable version capture keeps the exact line' + Assert-True ($null -eq (Get-ObservableVersionFromText "`n `n")) 'empty version output has no observable value' + + foreach ($fixture in @('codex-events.jsonl', 'opencode-events.jsonl', 'copilot-events.jsonl')) { + $fixturePath = Join-Path $PSScriptRoot "fixtures\$fixture" + $parsed = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText($fixturePath, [System.Text.UTF8Encoding]::new($false))) + Assert-Equal 0 $parsed.Errors.Count "recorded $fixture has valid JSONL" + Assert-True ($parsed.Events.Count -ge 4) "recorded $fixture has events" + Assert-True (@($parsed.Events | Where-Object { $_.type -eq 'future.event.v99' }).Count -eq 1) "recorded $fixture includes an unknown event" + } + $threadStartRejection = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText((Join-Path $PSScriptRoot 'fixtures\codex-thread-start-rejection.jsonl'), [System.Text.UTF8Encoding]::new($false))) + Assert-Equal 0 $threadStartRejection.Errors.Count 'recorded Codex thread/start rejection fixture is valid JSONL' + $threadStartError = @($threadStartRejection.Events | Where-Object { [int](Get-JsonProperty -Object (Get-JsonProperty -Object $_ -Name 'error' -Default $null) -Name 'code' -Default 0) -eq -32600 })[0] + Assert-Equal -32600 $threadStartError.error.code 'recorded Codex thread/start rejection preserves JSON-RPC error code' + Assert-True ($threadStartError.error.message -match 'read-only.*workspace-write.*danger-full-access') 'recorded Codex thread/start rejection preserves the installed sandbox enum' + $copilotFixture = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText((Join-Path $PSScriptRoot 'fixtures\copilot-events.jsonl'), [System.Text.UTF8Encoding]::new($false))) + Assert-True (@($copilotFixture.Events | Where-Object { $_.type -eq 'assistant.message' }).Count -ge 1) 'recorded copilot fixture includes documented assistant.message output' + Assert-True (@($copilotFixture.Events | Where-Object { $_.type -eq 'assistant.usage' }).Count -eq 1) 'recorded copilot fixture includes documented assistant.usage output' + Assert-True (@($copilotFixture.Events | Where-Object { $_.type -eq 'tool.execution_start' }).Count -eq 1) 'recorded copilot fixture includes documented tool.execution output' + $prepareText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\prepare-skill-evals.ps1'), [System.Text.UTF8Encoding]::new($false)) + $reportText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\generate-eval-report.ps1'), [System.Text.UTF8Encoding]::new($false)) + $bridgeText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'bridge-execution-result.ps1'), [System.Text.UTF8Encoding]::new($false)) + $recordText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'record-native-result.ps1'), [System.Text.UTF8Encoding]::new($false)) + $manifestBridgeText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'bridge-manifest-results.ps1'), [System.Text.UTF8Encoding]::new($false)) + $runnerOwnedText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'invoke-runner-owned-arms.ps1'), [System.Text.UTF8Encoding]::new($false)) + $commonText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'runner-common.ps1'), [System.Text.UTF8Encoding]::new($false)) + $orchestrationText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'orchestration.ps1'), [System.Text.UTF8Encoding]::new($false)) + $opencodeRunnerText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'opencode/runner.ps1'), [System.Text.UTF8Encoding]::new($false)) + Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'portable preparation must not contain harness-specific CLI invocations or provider-field branches' + Assert-True ($orchestrationText -notmatch '(?i)capture-native-results\.ps1|synthesize|worker_authored') 'generic orchestration must not manufacture native terminal envelopes' + Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'reporting must not contain harness-specific or provider-field branches' + Assert-True ($bridgeText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'the raw-to-portable bridge must remain runner-neutral' + Assert-True ($prepareText.Contains('execution-freeze.json') -and $prepareText.Contains('grading.json') -and $prepareText.Contains('apply-eval-grading.ps1') -and $prepareText.Contains('finalize-eval-package.ps1')) 'handoff preparation must expose the shared freeze, grading, and finalization boundaries' + Assert-True ($prepareText.Contains('Read the selected runner descriptor and its `delegation.dispatch_owner`.') -and $prepareText.Contains('invoke-runner-owned-arms.ps1') -and $prepareText.Contains('package-computed Phase 1 allowance') -and $prepareText.Contains('must be started exactly once')) 'handoff preparation must expose one foreground Phase 1 invocation with a computed caller timeout' + Assert-True ($prepareText.Contains('Do not create outer workers') -and $prepareText.Contains('edit raw result/evidence files')) 'handoff preparation must forbid outer runner-owned workers and raw evidence edits' + Assert-True ($prepareText.Contains('The Grader may author exactly one package-root `grading.json`') -and $prepareText.Contains('It must not edit raw execution results')) 'handoff preparation must isolate the Grader to the grading-only artifact' + Assert-True ($prepareText.Contains('Return only its machine-readable JSON summary') -and $prepareText.Contains('Never repair, re-freeze, re-bridge a changed raw result')) 'handoff preparation must make finalizer success and fail-closed recovery explicit' + Assert-True ($prepareText.Contains('evaluation is incomplete') -and $prepareText.Contains('Only persisted runner-produced evidence')) 'handoff preparation must fail closed when runner evidence cannot be persisted' + Assert-True ($prepareText.Contains('fresh package/code fix is required') -and $prepareText.Contains('Never patch package-local runner code') -and $prepareText.Contains('delete execution results') -and $prepareText.Contains('delete or replace `execution-freeze.json`') -and $prepareText.Contains('rerun Phase 1') -and $prepareText.Contains('manually broaden a capability check')) 'generated handoff must forbid package-local repair, state deletion, retry, and manual capability broadening' + $generatedHandoff = Invoke-GeneratedRunnerPrompt + Assert-True ($generatedHandoff.Contains('evaluation is incomplete and a fresh package/code fix is required') -and $generatedHandoff.Contains('Never patch package-local runner code') -and $generatedHandoff.Contains('delete orchestration state') -and $generatedHandoff.Contains('delete execution results') -and $generatedHandoff.Contains('delete or replace `execution-freeze.json`') -and $generatedHandoff.Contains('rerun Phase 1') -and $generatedHandoff.Contains('manually broaden a capability check')) 'generated handoff output forbids package-local repair, state deletion, retry, and manual capability broadening' + Assert-True ($generatedHandoff.Contains('invoke-runner-owned-arms.ps1') -and $generatedHandoff.Contains('package-computed Phase 1 allowance') -and $generatedHandoff.Contains('must be started exactly once') -and $generatedHandoff.Contains('If execution is interrupted and no valid `execution-freeze.json` exists')) 'generated handoff exposes one foreground Phase 1 invocation with fail-closed interruption handling' + Assert-True (-not $generatedHandoff.Contains('control-runner-owned-phase1.ps1') -and -not $generatedHandoff.Contains('WaitSeconds') -and -not $generatedHandoff.Contains('SAME controller command again')) 'generated handoff contains no durable controller polling' + Assert-True ($prepareText -notmatch '(?i)runs\.|record-native-result\.ps1|Assert-NativeWorkerDelegation|Assert-OrchestrationConcurrency|capture\.worker_authored') 'runner-owned handoff must not teach manual orchestration, recorder, or evidence repair trivia' + Assert-True ($bridgeText.Contains('Get-PackageRunnerDescriptor') -and $bridgeText.Contains('Assert-NativeTerminalCaptureArtifact') -and $bridgeText.Contains('ExpectedMechanism')) 'native bridge must require runner-produced terminal evidence' + Assert-True ($recordText.Contains('eval-native-worker-result/1') -and $recordText.Contains('New-ExecutionResult')) 'native terminal recording must use the runner-owned result builder' + Assert-True ($commonText.Contains('exit.status must be a JSON number or null')) 'execution results must reject textual exit statuses' + Assert-True ($commonText.Contains('requested.timeout_seconds') -and $commonText.Contains('execution-result.json run.$field')) 'raw execution results must retain the complete run and requested configuration contract' + Assert-True ($runnerOwnedText.Contains('Invoke-RunnerPreflight') -and $runnerOwnedText.Contains('Get-PreflightGateSummary') -and $runnerOwnedText.Contains('execution_started = $false')) 'runner-owned helper must gate all execute processes behind deterministic preflight' + Assert-True ($runnerOwnedText.Contains('return 120') -and $runnerOwnedText -notmatch 'Get-RunnerPreflightTimeoutSeconds\s*\{[^}]*ProfileTimeoutSeconds') 'runner-owned preflight uses an independent deterministic timeout' + Assert-True ($runnerOwnedText.Contains('This helper is the runner-owned Phase 1 external-handoff surface') -and -not $runnerOwnedText.Contains('Assert-RunnerOwnedFanoutAuthorization') -and -not $runnerOwnedText.Contains('SupervisorId') -and -not (Test-Path -LiteralPath (Join-Path $runnerRoot 'control-runner-owned-phase1.ps1') -PathType Leaf) -and -not (Test-Path -LiteralPath (Join-Path $runnerRoot 'supervise-runner-owned-phase1.ps1') -PathType Leaf) -and -not (Test-Path -LiteralPath (Join-Path $runnerRoot 'phase1-control-common.ps1') -PathType Leaf)) 'foreground fan-out replaces durable controller authorization' + Assert-True ($prepareText -notmatch '') 'handoff preparation must not expose an unconstrained result-file placeholder' + Assert-True ($reportText -notmatch 'function Get-ResultPath') 'reporting must not contain a configuration-derived result path helper' + Assert-True ($manifestBridgeText.Contains('Get-ManifestRunRecords') -and $manifestBridgeText.Contains('$record.ResultPath')) 'package-level bridge must resolve exact manifest records' + Assert-True ($manifestBridgeText -notmatch 'with[-_]skill\.result\.json|without[-_]skill\.result\.json') 'package-level bridge must not encode arm-derived result filenames' + Assert-Equal 1 ([regex]::Matches($opencodeRunnerText, '\$directoryArgument = Get-SandboxVisiblePath').Count) 'OpenCode CLI argument construction assigns the sandbox directory once' + $opencodeAst = Get-OpenCodeRunnerAst + $scriptedFunctionAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Invoke-OpenCodeScriptedExecute' }, $true) | Select-Object -First 1) + $executeFunctionAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Invoke-OpenCodeExecute' }, $true) | Select-Object -First 1) + $legacyFunctionAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Invoke-OpenCodeScriptedExecuteLegacy' }, $true)) + $continuationParserAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Get-OpenCodeContinuationCapability' }, $true)) + $continuationArgumentAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'New-OpenCodeContinuationArguments' }, $true)) + $turnProcessAst = @($opencodeAst.FindAll({ param($node) $node -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $node.Name -eq 'Invoke-OpenCodeTurnProcess' }, $true)) + Assert-Equal 1 $scriptedFunctionAst.Count 'OpenCode has one selected scripted execution function' + Assert-Equal 1 $executeFunctionAst.Count 'OpenCode has one execution dispatcher' + Assert-Equal 0 $legacyFunctionAst.Count 'OpenCode removes the dead legacy scripted continuation function' + Assert-Equal 1 $continuationParserAst.Count 'OpenCode keeps installed-help exact --session parsing' + Assert-Equal 1 $continuationArgumentAst.Count 'OpenCode keeps explicit --session argument construction' + Assert-Equal 1 $turnProcessAst.Count 'OpenCode uses a direct CLI turn-process helper' + $scriptedFunctionText = [string]$scriptedFunctionAst[0].Extent.Text + $executeFunctionText = [string]$executeFunctionAst[0].Extent.Text + Assert-True ($scriptedFunctionText.Contains('Invoke-OpenCodeTurnProcess') -and $scriptedFunctionText.Contains('New-OpenCodeContinuationArguments') -and $scriptedFunctionText.Contains('Get-OpenCodeFutureTurnCanary')) 'selected OpenCode scripted transport uses direct CLI exact-session continuation and future-turn secrecy checks' + Assert-True ($opencodeRunnerText -notmatch '(?i)Start-OpenCodeServer|Invoke-OpenCodeHttpRequest|SessionCreatePath|SessionMessagePath|/global/health|/doc|opencode\s+serve|serve_help_probe|opencode-server-synchronous-http') 'OpenCode runner has no eval server transport or loopback API probes' + Assert-True ($executeFunctionText.Contains('Invoke-OpenCodeScriptedExecute') -and $executeFunctionText -notmatch '(?i)Invoke-OpenCodeScriptedExecuteLegacy\s+-Inputs') 'OpenCode dispatcher selects the direct CLI scripted transport for interaction runs' + + $rawPath = Join-Path $iteration 'conformance\results\with-skill.execution-result.json' + $withoutRawPath = Join-Path $iteration 'conformance\results\without-skill.execution-result.json' + $resultPath = Join-Path $iteration 'conformance\results\with-skill.result.json' + $withoutResultPath = Join-Path $iteration 'conformance\results\without-skill.result.json' + New-Item -ItemType Directory -Path (Split-Path -Parent $rawPath) -Force | Out-Null + Write-TestJson -Path $resultPath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + eval_id = 1 + eval_name = 'conformance' + configuration = 'with_skill' + execution_status = 'unrun' + grading = @([ordered]@{ text = 'preserved assertion'; passed = $null; evidence = '' }) + }) + Write-TestJson -Path $withoutResultPath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + eval_id = 1 + eval_name = 'conformance' + configuration = 'without_skill' + execution_status = 'unrun' + grading = @([ordered]@{ text = 'preserved assertion'; passed = $null; evidence = '' }) + }) + $bridgeResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $with.Path -Profile $profilePath + $withoutBridgeResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath + Write-TestJson -Path $rawPath -Value $bridgeResult + Write-TestJson -Path $withoutRawPath -Value $withoutBridgeResult + Write-TestJson -Path (Join-Path $iteration 'conformance\eval-metadata.json') -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-metadata/2' + eval_id = 1 + eval_name = 'conformance' + assertions = @('preserved assertion') + }) + $oneArmManifest = [ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + configurations = @('with_skill', 'without_skill') + execution_freeze = 'execution-freeze.json' + evals = @([ordered]@{ + eval_id = 1 + eval_name = 'conformance' + directory = 'conformance' + metadata = 'conformance/eval-metadata.json' + runs = [ordered]@{ + with_skill = [ordered]@{ mode = 'with_skill'; run_manifest = 'conformance/with_skill/run.json'; execution_result = 'conformance/results/with-skill.execution-result.json'; result = 'conformance/results/with-skill.result.json' } + without_skill = [ordered]@{ mode = 'without_skill'; run_manifest = 'conformance/without_skill/run.json'; execution_result = 'conformance/results/without-skill.execution-result.json'; result = 'conformance/results/without-skill.result.json' } + } + }) + } + Write-TestJson -Path (Join-Path $iteration 'manifest.json') -Value $oneArmManifest + $oneArmManifestObject = Read-RunnerJson -Path (Join-Path $iteration 'manifest.json') + $oneArmRecords = @(Get-ManifestRunRecords -IterationDirectory $iteration -Manifest $oneArmManifestObject) + $oneArmProfile = Resolve-ExecutionProfile -ProfilePath $profilePath + $oneArmStatePath = Join-Path $iteration 'orchestration-state.json' + $oneArmState = [ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestration-state/1' + completed = [ordered]@{ + 'arm-1-with_skill' = [ordered]@{ + worker_id = 'arm-1-with_skill' + eval_id = 1 + configuration = 'with_skill' + status = [string]$bridgeResult.status + evidence_validation = [ordered]@{ status = 'passed'; reasons = @() } + } + 'arm-1-without_skill' = [ordered]@{ + worker_id = 'arm-1-without_skill' + eval_id = 1 + configuration = 'without_skill' + status = [string]$withoutBridgeResult.status + evidence_validation = [ordered]@{ status = 'passed'; reasons = @() } + } + } + execution_freeze = $null + } + Write-TestJson -Path $oneArmStatePath -Value $oneArmState + + $codexProfilePath = Join-Path $iteration 'codex-native-profile.json' + Write-TestJson -Path $codexProfilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'codex' + model = 'fixture-model' + reasoning_effort = 'high' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) + $codexRunData = Resolve-RunContract -RunPath $with.Path + $nativeEventRelativePath = 'evidence/native-worker-events.jsonl' + $nativeEventPath = Join-Path $with.Root ($nativeEventRelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $nativeEventPath) -Force | Out-Null + [System.IO.File]::WriteAllText($nativeEventPath, '{"type":"terminal"}' + [Environment]::NewLine, [System.Text.UTF8Encoding]::new($false)) + $nativeArtifact = New-ArtifactReference -Run $codexRunData -Path $nativeEventRelativePath -Scope run -MediaType 'application/x-ndjson' + $codexDescriptor = Get-PackageRunnerDescriptor -RunnerName 'codex' + $nativeInputPath = Join-Path $iteration 'conformance\results\native-worker-result.json' + $nativeOutputPath = Join-Path $iteration 'conformance\results\recorded.execution-result.json' + $nativeEnvelope = [ordered]@{ + schema = 'codebeltnet/agentic/eval-native-worker-result/1' + run_id = 'native-fixture-run' + session = [ordered]@{ id = 'native-fixture-session'; fresh = $true; resumed = $false } + status = 'completed' + run = [ordered]@{ eval_id = 1; eval_name = 'conformance'; configuration = 'with_skill' } + final_response = [ordered]@{ status = 'available'; text = 'native fixture response' } + timing = [ordered]@{ started_utc = '2024-01-01T00:00:00Z'; finished_utc = '2024-01-01T00:00:01Z'; duration_seconds = 1 } + exit = [ordered]@{ status = 0; failure = $null } + isolation = [ordered]@{ + capabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + filesystem_confinement = 'unavailable' + } + mechanisms = @('native-fixture-worker') + } + telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = $nativeEventRelativePath; complete = $true }) + tokens = New-UnavailableMetric -Reason 'fixture does not expose token telemetry' + tool_calls = New-AvailableMetric -Value 0 + cost = New-UnavailableMetric -Reason 'fixture does not expose cost telemetry' + } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = [string]$codexDescriptor.delegation.mechanism + worker_session_id = 'native-fixture-session' + observed_model = 'fixture-model' + observed_working_directory = $codexRunData.WorkingDirectoryPath + observed_home = $codexRunData.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = $codexRunData.PromptHash + terminal_result_capture = $true + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + } + } + capture = [ordered]@{ + source = 'harness_native_transport' + terminal = $true + worker_authored = $false + } + artifacts = @($nativeArtifact) + warnings = @() + compatibility_deviations = @() + attempt_count = 1 + resolved = [ordered]@{ status = 'accepted_request'; reason = 'native fixture accepted the requested configuration'; observations = [ordered]@{ model = 'fixture-model'; reasoning_effort = 'high' } } + } + Write-TestJson -Path $nativeInputPath -Value $nativeEnvelope + $recordPath = Join-Path $runnerRoot 'record-native-result.ps1' + $recordOutput = & pwsh -NoProfile -File $recordPath -Runner codex -Run $with.Path -Profile $codexProfilePath -NativeResult $nativeInputPath -Output $nativeOutputPath 2>&1 + if ($LASTEXITCODE -ne 0) { throw "native terminal recording failed: $([string]::Join(' ', @($recordOutput)))" } + $recordedResult = Read-RunnerJson -Path $nativeOutputPath + [void](Assert-ExecutionResult -Result $recordedResult) + Assert-Equal 'native-fixture-run' $recordedResult.run_id 'native terminal recording preserves the opaque worker run id' + Assert-Equal 'conformance' $recordedResult.run.eval_name 'native terminal recording derives exact arm identity from run.json' + Assert-Equal 'fixture-model' $recordedResult.requested.model 'native terminal recording derives model from execution-profile.json' + Assert-Equal 30 $recordedResult.requested.timeout_seconds 'native terminal recording preserves the requested timeout' + Assert-Equal '2024-01-01T00:00:00.000Z' ([DateTime]$recordedResult.started_utc).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ss.fffZ', [Globalization.CultureInfo]::InvariantCulture) 'native terminal recording writes canonical started_utc' + Assert-True ($recordedResult.telemetry.transcript.status -eq 'available') 'native terminal recording preserves transcript evidence' + Assert-Equal 'harness_native_transport' $recordedResult.evidence.capture.source 'native terminal recording preserves capture provenance' + Assert-True (-not [bool]$recordedResult.evidence.capture.worker_authored) 'native terminal recording rejects worker-authored capture provenance' + + $legacyNativeInputPath = Join-Path $iteration 'conformance\results\legacy-summary.json' + Write-TestJson -Path $legacyNativeInputPath -Value $bridgeResult + $legacyOutput = & pwsh -NoProfile -File $recordPath -Runner codex -Run $with.Path -Profile $codexProfilePath -NativeResult $legacyNativeInputPath -Output $nativeOutputPath 2>&1 + Assert-True ($LASTEXITCODE -ne 0) 'legacy summary-shaped worker output is rejected by the native recording boundary' + Assert-True (([string]::Join(' ', @($legacyOutput))) -match 'eval-native-worker-result/1') 'legacy summary rejection identifies the required native envelope' + + $oneArmFreeze = New-ExecutionFreezeDocument -IterationDirectory $iteration -Manifest $oneArmManifestObject -Records $oneArmRecords -Profile $oneArmProfile + $oneArmFreezePath = Write-ExecutionFreezeDocument -IterationDirectory $iteration -Freeze $oneArmFreeze + $oneArmState.execution_freeze = [ordered]@{ schema = (Get-RunnerSchemaNames).ExecutionFreeze; path = 'execution-freeze.json'; sha256 = Get-Sha256HexFromFile -Path $oneArmFreezePath } + Write-TestJson -Path $oneArmStatePath -Value $oneArmState + + Write-TestJson -Path $rawPath -Value $bridgeResult + $bridgePath = Join-Path $runnerRoot 'bridge-execution-result.ps1' + $bridgeOutput = & pwsh -NoProfile -File $bridgePath -Run $with.Path -ExecutionResult $rawPath -Result $resultPath + if ($LASTEXITCODE -ne 0) { throw "execution-result bridge failed: $([string]::Join(' ', @($bridgeOutput)))" } + $portable = Get-Content -LiteralPath $resultPath -Raw | ConvertFrom-Json + Assert-Equal 'codebeltnet/agentic/eval-result/2' $portable.schema 'bridge preserves existing result schema' + Assert-Equal 'completed' $portable.execution_status 'bridge carries execution status' + Assert-Equal 'fixture-model' $portable.model 'bridge carries resolved model' + Assert-True ($portable.PSObject.Properties.Name -notcontains 'provider') 'bridge removes provider from portable result' + Assert-True ($null -eq $portable.total_tokens) 'bridge keeps unavailable total tokens unavailable' + Assert-Equal 0 $portable.tool_calls 'bridge carries available tool-call count' + Assert-True $portable.isolation.transcript_captured 'bridge carries transcript availability' + Assert-Equal 'strict' $portable.isolation.level 'bridge carries isolation confidence level' + Assert-Equal 'verified' $portable.isolation.status 'bridge carries isolation verification status' + Assert-True (@($portable.isolation.mechanisms).Count -gt 0) 'bridge carries isolation mechanisms' + Assert-True (@($portable.output_files).Count -gt 0) 'bridge carries confined evidence paths' + Assert-Equal 'preserved assertion' $portable.grading[0].text 'bridge preserves the canonical grading entry' + Assert-True ($null -eq $portable.grading[0].passed) 'bridge preserves the canonical grading state before grading' + + $manifestPackage = Join-Path $iteration 'manifest-path-regression' + $manifestEval = Join-Path $manifestPackage 'conformance' + New-Item -ItemType Directory -Path $manifestEval -Force | Out-Null + $manifestPackageTools = Join-Path $manifestPackage 'tools/eval-runners' + New-Item -ItemType Directory -Path $manifestPackageTools -Force | Out-Null + foreach ($toolItem in @(Get-ChildItem -LiteralPath $runnerRoot -Force | Where-Object { $_.Name -ne 'tests' })) { + Copy-Item -LiteralPath $toolItem.FullName -Destination $manifestPackageTools -Recurse -Force + } + $manifestWith = New-TestRun -IterationDirectory $manifestPackage -Configuration with_skill -EvalName 'manifest-path-regression' + $manifestWithout = New-TestRun -IterationDirectory $manifestPackage -Configuration without_skill -EvalName 'manifest-path-regression' + $manifestMetadataPath = Join-Path $manifestEval 'eval-metadata.json' + Write-TestJson -Path $manifestMetadataPath -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-metadata/2' + eval_id = 1 + eval_name = 'manifest-path-regression' + assertions = @('preserved assertion', 'completed execution is bridged') + }) + $manifestWithResult = Join-Path $manifestEval 'results\with-skill.result.json' + $manifestWithoutResult = Join-Path $manifestEval 'results\without-skill.result.json' + $manifestWithExecution = Join-Path $manifestEval 'results\with-skill.execution-result.json' + $manifestWithoutExecution = Join-Path $manifestEval 'results\without-skill.execution-result.json' + foreach ($resultPathForStub in @($manifestWithResult, $manifestWithoutResult)) { + $configurationForStub = if ($resultPathForStub -eq $manifestWithResult) { 'with_skill' } else { 'without_skill' } + Write-TestJson -Path $resultPathForStub -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + eval_id = 1 + configuration = $configurationForStub + execution_status = 'unrun' + grading = @( + [ordered]@{ text = 'preserved assertion'; passed = $null; evidence = '' } + [ordered]@{ text = 'completed execution is bridged'; passed = $null; evidence = '' } + ) + }) + } + $manifestToolIntegrity = Get-PackageTreeIntegrity -Root $manifestPackageTools + $manifest = [ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + configurations = @('with_skill', 'without_skill') + runner_tools = 'tools/eval-runners' + runner_tools_integrity = [ordered]@{ schema = 'codebeltnet/agentic/package-tree-integrity/1'; path = 'tools/eval-runners'; sha256 = $manifestToolIntegrity.Sha256; file_count = $manifestToolIntegrity.FileCount } + execution_freeze = 'execution-freeze.json' + evals = @([ordered]@{ + eval_id = 1 + eval_name = 'manifest-path-regression' + directory = 'conformance' + metadata = 'conformance/eval-metadata.json' + runs = [ordered]@{ + with_skill = [ordered]@{ + mode = 'with_skill' + run_manifest = 'conformance/with_skill/run.json' + execution_result = 'conformance/results/with-skill.execution-result.json' + result = 'conformance/results/with-skill.result.json' + } + without_skill = [ordered]@{ + mode = 'without_skill' + run_manifest = 'conformance/without_skill/run.json' + execution_result = 'conformance/results/without-skill.execution-result.json' + result = 'conformance/results/without-skill.result.json' + } + } + }) + } + Write-TestJson -Path (Join-Path $manifestPackage 'manifest.json') -Value $manifest + $parallelProfilePath = Join-Path $manifestPackage 'execution-profile.json' + $parallelProfile = Read-RunnerJson -Path $profilePath + $parallelProfile.concurrency = 2 + Write-TestJson -Path $parallelProfilePath -Value $parallelProfile + $manifestWithExecutionResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $manifestWith.Path -Profile $parallelProfilePath + $manifestWithoutExecutionResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $manifestWithout.Path -Profile $parallelProfilePath + Write-TestJson -Path $manifestWithExecution -Value $manifestWithExecutionResult + Write-TestJson -Path $manifestWithoutExecution -Value $manifestWithoutExecutionResult + + $manifestObject = Get-Content -LiteralPath (Join-Path $manifestPackage 'manifest.json') -Raw | ConvertFrom-Json + $preBridgeValidation = Test-ManifestResults -IterationDirectory $manifestPackage -Manifest $manifestObject + Assert-True (-not $preBridgeValidation.Success) 'terminal execution plus an unrun canonical result fails validation before bridging' + Assert-True (@($preBridgeValidation.Errors | Where-Object { $_ -match 'remains unrun' }).Count -gt 0) 'pre-bridge validation reports the canonical unrun result' + + $incompatibleManifestResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $manifestWithout.Path -Profile $parallelProfilePath -Scenario incompatible + Write-TestJson -Path $manifestWithoutExecution -Value $incompatibleManifestResult + $incompatibleValidation = Test-ManifestResults -IterationDirectory $manifestPackage -Manifest $manifestObject -RequireComplete + Assert-True (-not $incompatibleValidation.Complete) 'incompatible execution evidence fails the completion gate' + Assert-True (@($incompatibleValidation.Errors | Where-Object { $_ -match 'diagnostic only' }).Count -gt 0) 'incompatible completion rejection explains that the arm is diagnostic only' + Write-TestJson -Path $manifestWithoutExecution -Value $manifestWithoutExecutionResult + + $manifestStatePath = Join-Path $manifestPackage 'orchestration-state.json' + $manifestState = [ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestration-state/1' + requested_concurrency = 2 + parallel_dispatch_required = $true + minimum_parallel_workers = 2 + capacity_limit_reported = $false + max_observed_active = 2 + pending_worker_ids = @() + active = [ordered]@{} + completed = [ordered]@{ + 'arm-1-with_skill' = [ordered]@{ + worker_id = 'arm-1-with_skill' + eval_id = 1 + configuration = 'with_skill' + status = 'completed' + evidence_validation = [ordered]@{ status = 'passed'; reasons = @() } + } + 'arm-1-without_skill' = [ordered]@{ + worker_id = 'arm-1-without_skill' + eval_id = 1 + configuration = 'without_skill' + status = 'completed' + evidence_validation = [ordered]@{ status = 'passed'; reasons = @() } + } + } + delegation_rejections = [ordered]@{} + eval_attempts = [ordered]@{ 'arm-1-with_skill' = 1; 'arm-1-without_skill' = 1 } + execution_freeze = $null + } + Write-TestJson -Path $manifestStatePath -Value $manifestState + $manifestRecords = @(Get-ManifestRunRecords -IterationDirectory $manifestPackage -Manifest $manifestObject) + $manifestProfileData = Resolve-ExecutionProfile -ProfilePath $parallelProfilePath + $manifestFreeze = New-ExecutionFreezeDocument -IterationDirectory $manifestPackage -Manifest $manifestObject -Records $manifestRecords -Profile $manifestProfileData + $manifestFreezePath = Write-ExecutionFreezeDocument -IterationDirectory $manifestPackage -Freeze $manifestFreeze + $manifestState.execution_freeze = [ordered]@{ schema = (Get-RunnerSchemaNames).ExecutionFreeze; path = 'execution-freeze.json'; sha256 = Get-Sha256HexFromFile -Path $manifestFreezePath } + Write-TestJson -Path $manifestStatePath -Value $manifestState + + $shadowPath = Join-Path $manifestEval 'results\with_skill.result.json' + Write-TestJson -Path $shadowPath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + eval_id = 1 + configuration = 'with_skill' + execution_status = 'completed' + grading = @() + }) + $manifestBridgePath = Join-Path $manifestPackageTools 'bridge-manifest-results.ps1' + $shadowOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete 2>&1 + $shadowExitCode = $LASTEXITCODE + Assert-True ($shadowExitCode -ne 0) 'manifest bridge rejects an unreferenced underscore shadow result' + Assert-True (([string]::Join(' ', @($shadowOutput))) -match 'unreferenced result-like sibling') 'shadow rejection explains the manifest collision' + $canonicalBeforeBridge = Get-Content -LiteralPath $manifestWithResult -Raw | ConvertFrom-Json + Assert-Equal 'unrun' $canonicalBeforeBridge.execution_status 'shadow result is never selected as the canonical result' + Remove-Item -LiteralPath $shadowPath -Force + + $manifestState.max_observed_active = 1 + Write-TestJson -Path $manifestStatePath -Value $manifestState + $serialBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete -RequireParallelDispatch 2>&1 + Assert-True ($LASTEXITCODE -ne 0) 'manifest bridge rejects post-freeze orchestration mutation' + Assert-True (([string]::Join(' ', @($serialBridgeOutput))) -match 'orchestration-state.json changed') 'state mutation rejection preserves the immutable concurrency ledger' + $manifestState.max_observed_active = 2 + Write-TestJson -Path $manifestStatePath -Value $manifestState + $manifestBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete -RequireParallelDispatch 2>&1 + if ($LASTEXITCODE -ne 0) { throw "manifest path bridge failed: $([string]::Join(' ', @($manifestBridgeOutput)))" } + $canonicalWith = Get-Content -LiteralPath $manifestWithResult -Raw | ConvertFrom-Json + Assert-Equal 'completed' $canonicalWith.execution_status 'manifest bridge populates the canonical hyphen result' + Assert-Equal 'fixture-model' $canonicalWith.model 'manifest bridge carries the model to the canonical result' + Assert-Equal 'deterministic-fake 1' $canonicalWith.harness 'manifest bridge carries the harness to the canonical result' + Assert-True (-not [string]::IsNullOrWhiteSpace([string]$canonicalWith.output)) 'manifest bridge carries output to the canonical result' + Assert-Equal 'results/with-skill.execution-result.json' $canonicalWith.execution_result_file 'manifest bridge records the exact manifest execution path' + Assert-Equal 2 @($canonicalWith.grading).Count 'manifest bridge preserves the canonical grading count' + Assert-Equal 'preserved assertion' $canonicalWith.grading[0].text 'manifest bridge preserves the canonical grading text' + + $canonicalWith.grading[0].passed = $true + $canonicalWith.grading[0].evidence = 'graded after the first bridge' + Write-TestJson -Path $manifestWithResult -Value $canonicalWith + $repeatBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete -RequireParallelDispatch 2>&1 + if ($LASTEXITCODE -ne 0) { throw "repeat manifest path bridge failed: $([string]::Join(' ', @($repeatBridgeOutput)))" } + $canonicalAfterRepeat = Get-Content -LiteralPath $manifestWithResult -Raw | ConvertFrom-Json + Assert-True ([bool]$canonicalAfterRepeat.grading[0].passed) 'repeat manifest bridge preserves completed grading' + Assert-Equal 'graded after the first bridge' $canonicalAfterRepeat.grading[0].evidence 'repeat manifest bridge preserves grading evidence' + + $frozenManifestRawBytes = [System.IO.File]::ReadAllBytes($manifestWithExecution) + $canonicalBeforeIntegrityFailure = [System.IO.File]::ReadAllBytes($manifestWithResult) + $staleReplacement = Get-Content -LiteralPath $manifestWithExecution -Raw | ConvertFrom-Json + $staleReplacement.run_id = 'replacement-terminal-result' + $staleReplacement.final_response.text = 'replacement terminal output' + Write-TestJson -Path $manifestWithExecution -Value $staleReplacement + $replacementBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete -RequireParallelDispatch 2>&1 + Assert-True ($LASTEXITCODE -ne 0) 'manifest bridge rejects a raw execution result changed after the freeze' + Assert-True (([string]::Join(' ', @($replacementBridgeOutput))) -match 'Execution integrity failure|requires fresh Phase 1 execution') 'raw mutation rejection identifies frozen evidence integrity' + Assert-True ([Convert]::ToBase64String([System.IO.File]::ReadAllBytes($manifestWithResult)) -eq [Convert]::ToBase64String($canonicalBeforeIntegrityFailure)) 'raw integrity failure does not rewrite the canonical result' + [System.IO.File]::WriteAllBytes($manifestWithExecution, $frozenManifestRawBytes) + $restoredBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete -RequireParallelDispatch 2>&1 + if ($LASTEXITCODE -ne 0) { throw "restored manifest path bridge failed: $([string]::Join(' ', @($restoredBridgeOutput)))" } + + $invalidExitPath = Join-Path $manifestEval 'results\invalid-exit.execution-result.json' + $invalidExitResult = $bridgeResult | ConvertTo-Json -Depth 100 | ConvertFrom-Json + $invalidExitResult.exit.status = 'completed' + Write-TestJson -Path $invalidExitPath -Value $invalidExitResult + $invalidExitThrew = $false + try { [void](Assert-ExecutionResult -Result $invalidExitResult) } catch { + $invalidExitThrew = $true + Assert-True ($_.Exception.Message -match 'JSON number or null') 'textual exit rejection explains the numeric contract' + } + Assert-True $invalidExitThrew 'execution-result validator rejects a textual exit status' + + Write-Output 'Eval Runner conformance: PASS' +} finally { + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_GLOBAL_RULES', $null, 'Process') + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_MEMORY', $null, 'Process') + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_PLUGINS', $null, 'Process') + if (Test-Path -LiteralPath $testRoot) { Remove-Item -LiteralPath $testRoot -Recurse -Force } +} + +Invoke-RecordedRunnerTests diff --git a/scripts/generate-eval-report.ps1 b/scripts/generate-eval-report.ps1 index 63be59f..6e4e454 100644 --- a/scripts/generate-eval-report.ps1 +++ b/scripts/generate-eval-report.ps1 @@ -26,6 +26,10 @@ .PARAMETER SkillCreatorPath Optional skill-creator installation or package-local tools/skill-creator path. The package-local path is the default so a prepared package remains self-contained after preparation. + +.PARAMETER RequireComplete + Retained for command-line compatibility. Report generation always requires every manifest-declared arm to have a + terminal execution result and a validated canonical result bridged from that exact path. #> [CmdletBinding()] param( @@ -38,13 +42,23 @@ param( [string]$BenchmarkMarkdownPath, - [string]$SkillCreatorPath + [string]$SkillCreatorPath, + + [switch]$RequireComplete ) $ErrorActionPreference = 'Stop' Set-StrictMode -Version Latest $utf8NoBom = [System.Text.UTF8Encoding]::new($false) +# Ensure this process reads UTF-8 from the child Python tooling's stdout even +# when the Windows console default is cp1252, so non-ASCII report evidence is +# captured and forwarded intact. +[Console]::OutputEncoding = $utf8NoBom +$OutputEncoding = $utf8NoBom + +. (Join-Path $PSScriptRoot 'eval-runners/manifest-paths.ps1') +. (Join-Path $PSScriptRoot 'eval-runners/execution-freeze.ps1') function Read-JsonFile { param([string]$Path) @@ -167,26 +181,33 @@ function Invoke-PythonScript { [string[]]$Arguments ) - $output = & $PythonCommand $ScriptPath @Arguments 2>&1 - if ($LASTEXITCODE -ne 0) { - throw "Python script '$ScriptPath' failed with exit code ${LASTEXITCODE}:`n$($output -join [Environment]::NewLine)" - } + # Force CPython UTF-8 Mode for the child so the packaged upstream + # skill-creator Python tooling (aggregate_benchmark.py, generate_review.py) + # reads and writes UTF-8 regardless of the Windows console/locale default + # (cp1252). This fixes non-ASCII report generation on Windows WITHOUT + # modifying any packaged upstream Python source. Setting PYTHONUTF8=1 is + # equivalent to `python -X utf8` and works for both the `python` interpreter + # and the `py` launcher, which does not reliably forward interpreter -X + # options placed before the script path. + $previousUtf8 = [Environment]::GetEnvironmentVariable('PYTHONUTF8') + $previousIoEncoding = [Environment]::GetEnvironmentVariable('PYTHONIOENCODING') + $env:PYTHONUTF8 = '1' + $env:PYTHONIOENCODING = 'utf-8' + try { + $output = & $PythonCommand $ScriptPath @Arguments 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Python script '$ScriptPath' failed with exit code ${LASTEXITCODE}:`n$($output -join [Environment]::NewLine)" + } - foreach ($line in @($output)) { - Write-Host $line + foreach ($line in @($output)) { + Write-Host $line + } + } finally { + if ($null -eq $previousUtf8) { Remove-Item Env:PYTHONUTF8 -ErrorAction SilentlyContinue } else { $env:PYTHONUTF8 = $previousUtf8 } + if ($null -eq $previousIoEncoding) { Remove-Item Env:PYTHONIOENCODING -ErrorAction SilentlyContinue } else { $env:PYTHONIOENCODING = $previousIoEncoding } } } -function Get-ResultPath { - param( - [string]$EvalDirectory, - [string]$Configuration - ) - - $fileName = if ($Configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } - return Join-Path (Join-Path $EvalDirectory 'results') $fileName -} - function Copy-RecordedOutputFiles { param( [object]$Result, @@ -400,7 +421,10 @@ function Get-ReportRun { configuration = $Configuration feedback_key = "eval-$EvalId-$Configuration" model = [string](Get-Property -Object $Result -Name 'model' -Default '') - provider = [string](Get-Property -Object $Result -Name 'provider' -Default '') + requested_model = [string](Get-Property -Object $Result -Name 'requested_model' -Default '') + resolved_model = [string](Get-Property -Object $Result -Name 'resolved_model' -Default '') + configuration_resolution_status = [string](Get-Property -Object $Result -Name 'configuration_resolution_status' -Default '') + configuration_resolution_reason = [string](Get-Property -Object $Result -Name 'configuration_resolution_reason' -Default '') harness = [string](Get-Property -Object $Result -Name 'harness' -Default '') executed_utc = [string](Get-Property -Object $Result -Name 'executed_utc' -Default '') output = $output @@ -412,6 +436,9 @@ function Get-ReportRun { stdout = [string](Get-Property -Object $Result -Name 'stdout' -Default '') stderr = [string](Get-Property -Object $Result -Name 'stderr' -Default '') exit_status = Get-Property -Object $Result -Name 'exit_status' -Default $null + execution_status = Get-Property -Object $Result -Name 'execution_status' -Default $null + execution_run_id = Get-Property -Object $Result -Name 'execution_run_id' -Default $null + execution_result_file = Get-Property -Object $Result -Name 'execution_result_file' -Default $null metrics = $metrics isolation = Get-Property -Object $Result -Name 'isolation' -Default $null grades = @($grades) @@ -422,12 +449,14 @@ function Get-ReportRun { function Get-ReportSkillStats { param( [object]$Manifest, + [object[]]$ManifestRecords, [string]$IterationPath ) $skillRoot = $null - foreach ($entry in @($Manifest.evals)) { - $candidate = Join-Path (Join-Path (Join-Path $IterationPath ([string]$entry.directory)) 'with_skill') ("skill/$($Manifest.skill_name)") + foreach ($record in @($ManifestRecords | Where-Object { [string]$_.Configuration -eq 'with_skill' })) { + $runPackageDirectory = Split-Path -Parent ([string]$record.RunManifestPath) + $candidate = Join-Path $runPackageDirectory ("skill/$($Manifest.skill_name)") if (Test-Path -LiteralPath $candidate -PathType Container) { $skillRoot = $candidate break @@ -449,6 +478,8 @@ function Get-ReportSkillStats { function Write-FirstPartyReport { param( [object]$Manifest, + [object[]]$ManifestRecords, + [object]$Validation, [string]$IterationPath, [string]$OutputPath, [object]$Benchmark @@ -456,26 +487,33 @@ function Write-FirstPartyReport { $evals = [System.Collections.Generic.List[object]]::new() $allModels = [System.Collections.Generic.List[string]]::new() - $allProviders = [System.Collections.Generic.List[string]]::new() - $completedRuns = 0 + $completedRuns = [int]$Validation.BridgedResults foreach ($entry in @($Manifest.evals)) { - $evalDirectory = Join-Path $IterationPath ([string]$entry.directory) - $metadata = Read-JsonFile -Path (Join-Path $evalDirectory 'eval-metadata.json') + $entryRecords = @($ManifestRecords | Where-Object { [int]$_.EvalId -eq [int]$entry.eval_id }) + if ($entryRecords.Count -eq 0) { + throw "$($entry.eval_name) does not have manifest-declared arm paths." + } + $evalDirectory = [string]$entryRecords[0].EvalDirectory + $metadata = Read-JsonFile -Path ([string]$entryRecords[0].MetadataPath) $runMap = [ordered]@{} $assertions = @($metadata.assertions | ForEach-Object { [string]$_ }) foreach ($configuration in @('with_skill', 'without_skill')) { - $resultPath = Get-ResultPath -EvalDirectory $evalDirectory -Configuration $configuration + $records = @($ManifestRecords | Where-Object { + [int]$_.EvalId -eq [int]$entry.eval_id -and [string]$_.Configuration -eq $configuration + }) + if ($records.Count -ne 1) { + throw "$($entry.eval_name)/$configuration does not have exactly one manifest-declared result path." + } + $runRecord = $records[0] + $resultPath = [string]$runRecord.ResultPath $result = if (Test-Path -LiteralPath $resultPath) { Read-JsonFile -Path $resultPath } else { $null } if ($null -ne $result) { - $run = Get-ReportRun -Result $result -Configuration $configuration -EvalName ([string]$entry.eval_name) -EvalId ([int]$metadata.eval_id) -Assertions $assertions -RunPackageDirectory (Join-Path $evalDirectory $configuration) -EvalDirectory $evalDirectory -IterationPath $IterationPath + $runPackageDirectory = Split-Path -Parent ([string]$runRecord.RunManifestPath) + $run = Get-ReportRun -Result $result -Configuration $configuration -EvalName ([string]$entry.eval_name) -EvalId ([int]$metadata.eval_id) -Assertions $assertions -RunPackageDirectory $runPackageDirectory -EvalDirectory $evalDirectory -IterationPath $IterationPath $runMap[$configuration] = $run - if (-not [string]::IsNullOrWhiteSpace([string]$run.output) -or @($run.output_files).Count -gt 0) { $completedRuns++ } if (-not [string]::IsNullOrWhiteSpace([string]$run.model) -and -not $allModels.Contains([string]$run.model)) { $allModels.Add([string]$run.model) } - if (-not [string]::IsNullOrWhiteSpace([string]$run.provider) -and -not $allProviders.Contains([string]$run.provider)) { - $allProviders.Add([string]$run.provider) - } } else { $runMap[$configuration] = $null } @@ -492,7 +530,6 @@ function Write-FirstPartyReport { $metadata = [ordered]@{ model = if ($allModels.Count -gt 0) { $allModels -join ', ' } else { $null } - provider = if ($allProviders.Count -gt 0) { $allProviders -join ', ' } else { $null } completed_runs = $completedRuns expected_runs = @($Manifest.evals).Count * 2 generated_utc = [string](Get-Property -Object $Manifest -Name 'generated_utc' -Default '') @@ -501,7 +538,7 @@ function Write-FirstPartyReport { skill_name = [string]$Manifest.skill_name iteration = [int]$Manifest.iteration metadata = $metadata - skill = Get-ReportSkillStats -Manifest $Manifest -IterationPath $IterationPath + skill = Get-ReportSkillStats -Manifest $Manifest -ManifestRecords $ManifestRecords -IterationPath $IterationPath evals = @($evals) benchmark = $Benchmark } @@ -544,14 +581,19 @@ function Write-UpstreamGrading { $duration = Get-Property -Object $Result -Name 'duration_seconds' -Default $null $tokens = Get-Property -Object $Result -Name 'total_tokens' -Default $null $toolCalls = Get-Property -Object $Result -Name 'tool_calls' -Default $null + $exitStatus = Get-Property -Object $Result -Name 'exit_status' -Default $null + $errorsEncountered = if ($null -eq $exitStatus -or [string]::IsNullOrWhiteSpace([string]$exitStatus)) { $null } elseif ([int]$exitStatus -eq 0) { 0 } else { 1 } - $durationSeconds = if ($null -eq $duration) { 0.0 } else { [double]$duration } - $totalTokens = if ($null -eq $tokens) { 0 } else { [int64]$tokens } - Write-JsonFile -Path (Join-Path $RunDirectory 'timing.json') -Value ([ordered]@{ - total_tokens = $totalTokens - duration_ms = [math]::Round($durationSeconds * 1000, 0) - total_duration_seconds = $durationSeconds - }) + $timing = [ordered]@{} + if ($null -ne $duration -and -not [string]::IsNullOrWhiteSpace([string]$duration)) { + $durationSeconds = [double]$duration + $timing.duration_ms = [math]::Round($durationSeconds * 1000, 0) + $timing.total_duration_seconds = $durationSeconds + } + if ($null -ne $tokens -and -not [string]::IsNullOrWhiteSpace([string]$tokens)) { + $timing.total_tokens = [int64]$tokens + } + Write-JsonFile -Path (Join-Path $RunDirectory 'timing.json') -Value $timing $gradingDocument = [ordered]@{ expectations = @($expectations) @@ -563,7 +605,7 @@ function Write-UpstreamGrading { } execution_metrics = [ordered]@{ total_tool_calls = $toolCalls - errors_encountered = if ([int](Get-Property -Object $Result -Name 'exit_status' -Default 0) -eq 0) { 0 } else { 1 } + errors_encountered = $errorsEncountered } # The upstream aggregator reads timing.json when grading.json does not claim a duration. Keep the # portable run's timing in that sibling file so both elapsed time and token usage survive aggregation. @@ -581,6 +623,7 @@ function Write-UpstreamGrading { function New-UpstreamWorkspace { param( [object]$Manifest, + [object[]]$ManifestRecords, [string]$IterationPath, [string]$WorkspacePath ) @@ -589,8 +632,12 @@ function New-UpstreamWorkspace { $workspaceEntries = [System.Collections.Generic.List[object]]::new() foreach ($entry in @($Manifest.evals)) { - $evalDirectory = Join-Path $IterationPath ([string]$entry.directory) - $metadata = Read-JsonFile -Path (Join-Path $evalDirectory 'eval-metadata.json') + $entryRecords = @($ManifestRecords | Where-Object { [int]$_.EvalId -eq [int]$entry.eval_id }) + if ($entryRecords.Count -eq 0) { + throw "$($entry.eval_name) does not have manifest-declared arm paths." + } + $evalDirectory = [string]$entryRecords[0].EvalDirectory + $metadata = Read-JsonFile -Path ([string]$entryRecords[0].MetadataPath) $evalFolder = Join-Path $WorkspacePath ("eval-{0}-{1}" -f $entry.eval_id, (Get-SafeSegment -Value ([string]$entry.eval_name))) New-Item -ItemType Directory -Path $evalFolder -Force | Out-Null @@ -611,7 +658,14 @@ function New-UpstreamWorkspace { New-Item -ItemType Directory -Path $outputsDirectory -Force | Out-Null Write-JsonFile -Path (Join-Path $configurationDirectory 'eval_metadata.json') -Value $upstreamMetadata - $resultPath = Get-ResultPath -EvalDirectory $evalDirectory -Configuration $configuration + $records = @($ManifestRecords | Where-Object { + [int]$_.EvalId -eq [int]$entry.eval_id -and [string]$_.Configuration -eq $configuration + }) + if ($records.Count -ne 1) { + throw "$($entry.eval_name)/$configuration does not have exactly one manifest-declared result path." + } + $runRecord = $records[0] + $resultPath = [string]$runRecord.ResultPath if (-not (Test-Path -LiteralPath $resultPath)) { continue } @@ -632,7 +686,8 @@ function New-UpstreamWorkspace { Write-JsonFile -Path (Join-Path $outputsDirectory 'isolation.json') -Value $isolation } - Copy-RecordedOutputFiles -Result $result -RunPackageDirectory (Join-Path $evalDirectory $configuration) -EvalDirectory $evalDirectory -IterationPath $IterationPath -OutputDirectory $outputsDirectory + $runPackageDirectory = Split-Path -Parent ([string]$runRecord.RunManifestPath) + Copy-RecordedOutputFiles -Result $result -RunPackageDirectory $runPackageDirectory -EvalDirectory $evalDirectory -IterationPath $IterationPath -OutputDirectory $outputsDirectory Write-UpstreamGrading -Result $result -RunDirectory $runDirectory -Assertions @($metadata.assertions | ForEach-Object { [string]$_ }) } } @@ -642,10 +697,23 @@ function New-UpstreamWorkspace { $iterationPath = (Resolve-Path -LiteralPath $IterationDirectory).Path $manifest = Read-JsonFile -Path (Join-Path $iterationPath 'manifest.json') +$freezeValidation = Assert-ExecutionFreeze -IterationDirectory $iterationPath -RequireOrchestrationState +[void](Assert-FanoutPhase1Success -Aggregate $freezeValidation.Aggregate -MessagePrefix 'Report generation Phase 1') +$manifestRecords = @($freezeValidation.Records) +$validation = Test-ManifestResults -IterationDirectory $iterationPath -Manifest $manifest -Records $manifestRecords -RequireComplete +if (-not $validation.Success) { + throw ([string]::Join([Environment]::NewLine, @($validation.Errors))) +} +foreach ($warning in @($validation.Warnings)) { + Write-Host "[WARN] $warning" +} +if (-not $validation.Complete) { + throw "Evaluation completion gate failed: expected $($validation.ExpectedArmCount) bridged terminal arms, found $($validation.BridgedResults)." +} $skillCreatorPathResolved = Resolve-SkillCreatorPath -RequestedPath $SkillCreatorPath $pythonCommand = Resolve-PythonCommand $workspacePath = Join-Path $iterationPath '.skill-creator-report' -$workspaceEntries = New-UpstreamWorkspace -Manifest $manifest -IterationPath $iterationPath -WorkspacePath $workspacePath +$workspaceEntries = New-UpstreamWorkspace -Manifest $manifest -ManifestRecords $manifestRecords -IterationPath $iterationPath -WorkspacePath $workspacePath $aggregatePath = Join-Path $skillCreatorPathResolved 'scripts/aggregate_benchmark.py' $viewerPath = Join-Path $skillCreatorPathResolved 'eval-viewer/generate_review.py' @@ -655,10 +723,25 @@ Invoke-PythonScript -PythonCommand $pythonCommand -ScriptPath $aggregatePath -Ar $benchmarkWorkspacePath = Join-Path $workspacePath 'benchmark.json' $benchmark = Read-JsonFile -Path $benchmarkWorkspacePath +$benchmarkRuns = @(Get-Property -Object $benchmark -Name 'runs' -Default @()) +if ($RequireComplete) { + if ($benchmarkRuns.Count -eq 0) { + throw 'Evaluation completion gate failed: benchmark output contains zero completed runs.' + } + if ($benchmarkRuns.Count -ne $validation.ExpectedArmCount -or $benchmarkRuns.Count -ne $validation.BridgedResults) { + throw "Evaluation completion gate failed: benchmark completed-run count $($benchmarkRuns.Count) does not match expected bridged count $($validation.BridgedResults) of $($validation.ExpectedArmCount)." + } +} $models = [System.Collections.Generic.List[string]]::new() foreach ($entry in @($manifest.evals)) { foreach ($configuration in @('with_skill', 'without_skill')) { - $resultPath = Get-ResultPath -EvalDirectory (Join-Path $iterationPath ([string]$entry.directory)) -Configuration $configuration + $records = @($manifestRecords | Where-Object { + [int]$_.EvalId -eq [int]$entry.eval_id -and [string]$_.Configuration -eq $configuration + }) + if ($records.Count -ne 1) { + throw "$($entry.eval_name)/$configuration does not have exactly one manifest-declared result path." + } + $resultPath = [string]$records[0].ResultPath if (Test-Path -LiteralPath $resultPath) { $model = [string](Get-Property -Object (Read-JsonFile -Path $resultPath) -Name 'model' -Default '') if (-not [string]::IsNullOrWhiteSpace($model) -and -not $models.Contains($model)) { @@ -698,7 +781,15 @@ $viewerArguments = @( ) Invoke-PythonScript -PythonCommand $pythonCommand -ScriptPath $viewerPath -Arguments $viewerArguments -Write-FirstPartyReport -Manifest $manifest -IterationPath $iterationPath -OutputPath $htmlOutputPath -Benchmark $benchmark +Write-FirstPartyReport -Manifest $manifest -ManifestRecords $manifestRecords -Validation $validation -IterationPath $iterationPath -OutputPath $htmlOutputPath -Benchmark $benchmark + +if ($RequireComplete) { + foreach ($output in @($benchmarkOutputPath, $benchmarkMarkdownOutputPath, $upstreamHtmlOutputPath, $htmlOutputPath)) { + if (-not (Test-Path -LiteralPath $output -PathType Leaf) -or (Get-Item -LiteralPath $output).Length -eq 0) { + throw "Evaluation completion gate failed: report artifact '$output' is missing or empty." + } + } +} Write-Host "Anthropic skill-creator tools: $skillCreatorPathResolved" Write-Host "Anthropic viewer template: $viewerTemplatePath" @@ -706,3 +797,7 @@ Write-Host "Wrote $benchmarkOutputPath" Write-Host "Wrote $benchmarkMarkdownOutputPath" Write-Host "Wrote $upstreamHtmlOutputPath" Write-Host "Wrote $htmlOutputPath" +Write-Host "Manifest-declared terminal arms: $($validation.BridgedResults)/$($validation.ExpectedArmCount)" +if (-not $validation.Complete) { + Write-Host 'This is a partial report; evaluation completion was not asserted.' +} diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index c73787a..13c1911 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -5,19 +5,21 @@ .DESCRIPTION This script computes and prints. It never executes a prompt, never spawns an agent, and never calls a model. It turns skills//evals/evals.json into a paste-ready evaluation package that a human can run in whatever - harness, provider, and model they choose, then validates the results that come back. + harness and model they choose, then validates the results that come back. Prepare mode writes one directory per eval. The grading key and result stubs stay at the eval-case level, outside - the two hermetic run directories a worker actually sees: + the two isolated run directories a worker actually sees: eval-metadata.json id, name, prompt, expected output, assertions, fixtures, hashes, assumptions results/ one prefilled result stub per configuration - with_skill/ a hermetic run: prompt.md, run.json, repo/ (materialized fixtures), home/, skill// + with_skill/ an isolated run: prompt.md, run.json, repo/ (materialized fixtures), home/, skill// without_skill/ the same run without any skill/ directory and no skill instructions - Each run directory is the worker's sandbox root: repo/ is the working tree, home/ is an isolated profile, and skill/ + Each run directory is the worker's staged root: repo/ is the working tree, home/ is an isolated profile, and skill/ (with_skill only) holds the candidate skill revision. run.json is a harness-neutral contract naming only paths inside - the run directory. Preparation validates the isolation invariants and fails early if a package would let a baseline - reach the skill, let a worker reach the source repository, or stage mismatched fixtures. + the run directory. The selected runner enforces the isolation and reports strict confidence when it also proves hard + filesystem confinement or pragmatic confidence when it does not. Preparation validates the isolation invariants and + fails early if a package would let a baseline reach the skill, stage the source repository into a run, or stage + mismatched fixtures. Collect mode reads a prepared package plus whatever result files were filled in, validates them, and invokes the packaged Anthropic skill-creator aggregator and static eval viewer after writing a deterministic comparison. The @@ -42,6 +44,36 @@ .PARAMETER Force Overwrite an existing iteration directory. +.PARAMETER Runner + Required package-local Eval Runner id written to execution-profile.json when -CodebeltReference is not used. + +.PARAMETER Model + Required runner-native model selector written to execution-profile.json when -CodebeltReference is not used. + +.PARAMETER CodebeltReference + Resolve the Codebelt reference configuration by discovering current GitHub Copilot CLI models and selecting + claude-haiku-4.5 only when it is still available. + +.PARAMETER ModelCatalogPath + Optional deterministic catalog JSON used by the model discovery helper. Intended for tests and offline validation. + +.PARAMETER ReasoningEffort + Optional runner-supported reasoning/effort setting written to execution-profile.json. Codex defaults to low + when this is omitted. + +.PARAMETER ConfigurationProfile + Runner configuration profile. Defaults to isolated-default. + +.PARAMETER ToolProfile + Runner tool profile. Defaults to default. + +.PARAMETER TimeoutSeconds + Per-arm runner timeout. Defaults to 900 seconds. + +.PARAMETER Concurrency + Requested concurrency. Defaults to 16 for Codex and GitHub Copilot. OpenCode defaults to 2 when this parameter is + omitted; an explicitly bound value is always honored. It does not change paired-arm semantics. + .PARAMETER Changed Prepares a package for every repo-managed skill this branch changed, including uncommitted work. This is the form the eval completion gate uses after adding or modifying a skill. @@ -53,13 +85,14 @@ Path to a prepared iteration directory. Validates the result files in it, invokes the packaged Anthropic skill-creator aggregator and static viewer, and writes comparison.md, benchmark.json, benchmark.md, the first-party side-by-side report.html, and the exact upstream skill-creator-report.html compatibility artifact. - This is the fallback for results that were not finalized by the external evaluator. + This is an explicitly authorized forensic recovery path for an existing package, not normal handoff recovery. It + cannot repair missing, malformed, or unproven runner-produced execution evidence. .EXAMPLE - pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-strong-name-signing + pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-strong-name-signing -Runner github-copilot -Model claude-haiku-4.5 .EXAMPLE - pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed + pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -CodebeltReference .EXAMPLE pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults $env:TEMP/dotnet-strong-name-signing-workspace/iteration-1 @@ -93,6 +126,44 @@ param( [Parameter(ParameterSetName = 'Changed')] [switch]$Force, + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$Runner, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$Model, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [switch]$CodebeltReference, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$ModelCatalogPath, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$ReasoningEffort, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$ConfigurationProfile = 'isolated-default', + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$ToolProfile = 'default', + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [ValidateRange(1, 86400)] + [int]$TimeoutSeconds = 900, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [ValidateRange(1, 128)] + [int]$Concurrency = 16, + [Parameter(ParameterSetName = 'Collect', Mandatory = $true)] [string]$CollectResults ) @@ -108,16 +179,22 @@ $utf8NoBom = [System.Text.UTF8Encoding]::new($false) [Console]::OutputEncoding = $utf8NoBom $OutputEncoding = $utf8NoBom +. (Join-Path $PSScriptRoot 'eval-runners/manifest-paths.ps1') + $packageSchema = 'codebeltnet/agentic/eval-package/2' $metadataSchema = 'codebeltnet/agentic/eval-metadata/2' $resultSchema = 'codebeltnet/agentic/eval-result/2' $runSchema = 'codebeltnet/agentic/eval-run/1' +$executionProfileSchema = 'codebeltnet/agentic/eval-execution-profile/1' +$executionResultSchema = 'codebeltnet/agentic/eval-execution-result/1' +$runnerProtocolSchema = 'codebeltnet/agentic/eval-runner-protocol/1' $maxFixtureInlineBytes = 32768 -# A materialized run is hermetic: the harness treats the run directory as the worker's sandbox root, mounts repo/ as +# A materialized run is self-contained: the runner treats the run directory as the worker's staged root, uses repo/ as # the working directory and home/ as the isolated user profile, and exposes skill/ only for a with_skill run. Nothing -# else in the package - the grading key, the paired run, other evals, or results - lives inside a run directory, so a -# worker confined to its run directory cannot reach any of it. +# else in the package - the grading key, the paired run, other evals, or results - is staged inside a run directory, so a +# worker that stays within its run directory is never handed any of it. Hard filesystem confinement is an optional +# confidence signal a runner may add on top; it is not required for this staging boundary. $runDirectoryNames = [ordered]@{ Working = 'repo' Home = 'home' @@ -127,6 +204,7 @@ $runDirectoryNames = [ordered]@{ } $reportToolRelativePath = 'tools/generate-eval-report.ps1' $skillCreatorToolRelativePath = 'tools/skill-creator' +$evalRunnerToolRelativePath = 'tools/eval-runners' $skillCreatorEvalFiles = @( 'LICENSE.txt', 'agents/grader.md', @@ -315,6 +393,90 @@ function Assert-WorkspaceLocation { } } +function Get-HarnessName { + param([Parameter(Mandatory = $true)][string]$RunnerName) + + switch ($RunnerName) { + 'github-copilot' { return 'GitHub Copilot CLI' } + 'codex' { return 'Codex CLI' } + 'opencode' { return 'OpenCode' } + 'fake' { return 'Deterministic fake runner' } + default { return $RunnerName } + } +} + +function Get-SupportedRunnerIds { + param([Parameter(Mandatory = $true)][string]$RepoRoot) + + $runnerRoot = Join-Path $RepoRoot 'scripts/eval-runners' + if (-not (Test-Path -LiteralPath $runnerRoot -PathType Container)) { + return @() + } + + return @(Get-ChildItem -LiteralPath $runnerRoot -Directory -Force | + Where-Object { Test-Path -LiteralPath (Join-Path $_.FullName 'runner.ps1') -PathType Leaf } | + Sort-Object Name | + ForEach-Object { $_.Name }) +} + +function Resolve-ExecutionSelection { + param([Parameter(Mandatory = $true)][string]$RepoRoot) + + $referenceRunner = 'github-copilot' + $referenceModel = 'claude-haiku-4.5' + $supportedRunners = @(Get-SupportedRunnerIds -RepoRoot $RepoRoot) + $supportedText = if ($supportedRunners.Count -gt 0) { $supportedRunners -join ', ' } else { '(none found)' } + + if ($CodebeltReference -and (-not [string]::IsNullOrWhiteSpace($Runner) -or -not [string]::IsNullOrWhiteSpace($Model))) { + throw 'Choose either -CodebeltReference or an explicit -Runner/-Model pair, not both.' + } + + if ($CodebeltReference) { + if ($supportedRunners -notcontains $referenceRunner) { + throw "Codebelt Reference requires runner '$referenceRunner', but it is unavailable. Supported runner IDs: $supportedText." + } + $discoveryScript = Join-Path $RepoRoot 'scripts/Get-HarnessModels.ps1' + if (-not (Test-Path -LiteralPath $discoveryScript -PathType Leaf)) { + throw "Cannot resolve Codebelt Reference because '$discoveryScript' is missing." + } + + $arguments = @('-Runner', $referenceRunner, '-RequireModel', $referenceModel) + if (-not [string]::IsNullOrWhiteSpace($ModelCatalogPath)) { + $arguments += @('-CatalogPath', $ModelCatalogPath) + } + $discoveryOutput = & pwsh -NoProfile -File $discoveryScript @arguments 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Codebelt Reference requires $referenceRunner + $referenceModel, but current model discovery could not verify it. $($discoveryOutput -join [Environment]::NewLine)" + } + + return [pscustomobject]@{ + Runner = $referenceRunner + Model = $referenceModel + Harness = Get-HarnessName -RunnerName $referenceRunner + Preset = 'Codebelt Reference' + } + } + + $hasRunner = -not [string]::IsNullOrWhiteSpace($Runner) + $hasModel = -not [string]::IsNullOrWhiteSpace($Model) + if (-not $hasRunner -and -not $hasModel) { + throw "Evaluation preparation requires a resolved Harness + Model before RUN-THIS.prompt.md can be generated. Pass -Runner and -Model, or use -CodebeltReference after verifying the current catalog. Supported runner IDs: $supportedText." + } + if ($hasRunner -ne $hasModel) { + throw 'Runner/model selection is atomic: pass both -Runner and -Model, or neither when no package will be generated.' + } + if ($supportedRunners -notcontains $Runner) { + throw "Unsupported runner '$Runner'. Supported runner IDs: $supportedText." + } + + return [pscustomobject]@{ + Runner = $Runner + Model = $Model + Harness = Get-HarnessName -RunnerName $Runner + Preset = 'Custom' + } +} + function Get-EvalName { param([object]$EvalEntry) @@ -506,7 +668,6 @@ function New-ResultStub { eval_name = $EvalName configuration = $Configuration model = '' - provider = '' harness = '' executed_utc = '' output = '' @@ -518,6 +679,9 @@ function New-ResultStub { stdout = '' stderr = '' exit_status = $null + execution_status = 'unrun' + execution_run_id = '' + execution_result_file = '' duration_seconds = $null total_tokens = $null tool_calls = $null @@ -542,6 +706,41 @@ function New-ResultStub { } } +function New-ExecutionProfile { + param( + [Parameter(Mandatory = $true)][object]$ExecutionSelection, + [Parameter(Mandatory = $true)][int]$EffectiveConcurrency + ) + + return [ordered]@{ + schema = $executionProfileSchema + runner = $ExecutionSelection.Runner + model = $ExecutionSelection.Model + reasoning_effort = if (-not [string]::IsNullOrWhiteSpace($ReasoningEffort)) { $ReasoningEffort } elseif ($ExecutionSelection.Runner -eq 'codex') { 'low' } else { $null } + configuration_profile = $ConfigurationProfile + tool_profile = $ToolProfile + timeout_seconds = $TimeoutSeconds + concurrency = $EffectiveConcurrency + } +} + +function Resolve-EffectiveConcurrency { + param( + [Parameter(Mandatory = $true)][string]$RunnerName, + [Parameter(Mandatory = $true)][int]$RequestedConcurrency, + [Parameter(Mandatory = $true)][bool]$ConcurrencyWasExplicit + ) + + if ($RunnerName -eq 'opencode' -and -not $ConcurrencyWasExplicit) { + return [pscustomobject]@{ Value = 2; Source = 'OpenCode safe default'; Explicit = $false } + } + return [pscustomobject]@{ + Value = $RequestedConcurrency + Source = if ($ConcurrencyWasExplicit) { 'explicit -Concurrency' } else { 'repository default' } + Explicit = $ConcurrencyWasExplicit + } +} + function Get-JsonProperty { param( [object]$Object, @@ -592,6 +791,40 @@ function Get-Assertions { return @() } +function New-InteractionDocument { + param([Parameter(Mandatory = $true)][object]$EvalEntry) + + $declared = Get-JsonProperty -Object $EvalEntry -Name 'interaction' -Default $null + if ($null -eq $declared) { return $null } + if ([string](Get-JsonProperty -Object $declared -Name 'mode' -Default 'scripted') -ne 'scripted') { + throw "Eval $($EvalEntry.id) interaction mode must be 'scripted'." + } + $turns = @(Get-JsonProperty -Object $declared -Name 'turns' -Default @()) + if ($turns.Count -lt 2) { throw "Eval $($EvalEntry.id) scripted interaction must contain at least two turns." } + $normalizedTurns = [System.Collections.Generic.List[object]]::new() + foreach ($turn in $turns) { + $role = [string](Get-JsonProperty -Object $turn -Name 'role' -Default 'user') + if ($role -ne 'user') { throw "Eval $($EvalEntry.id) interaction turns must all be user turns." } + $source = [string](Get-JsonProperty -Object $turn -Name 'source' -Default '') + $content = [string](Get-JsonProperty -Object $turn -Name 'content' -Default '') + if (($source -and $content) -or (-not $source -and -not $content)) { + throw "Eval $($EvalEntry.id) interaction turns must declare exactly one source or content." + } + $normalizedTurn = [ordered]@{ role = 'user' } + if ($source) { + $normalizedTurn.source = $source + } else { + $normalizedTurn.content = $content + } + $normalizedTurns.Add($normalizedTurn) + } + return [ordered]@{ + schema = 'codebeltnet/agentic/eval-interaction/1' + mode = 'scripted' + turns = @($normalizedTurns.ToArray()) + } +} + function Get-Sha256Hex { param([byte[]]$Bytes) @@ -879,6 +1112,32 @@ function Copy-SkillCreatorEvalTools { return $destinationRoot } +function Copy-EvalRunnerTools { + param( + [string]$RepoRoot, + [string]$IterationDirectory + ) + + $sourceRoot = Join-Path (Join-Path $RepoRoot 'scripts') 'eval-runners' + if (-not (Test-Path -LiteralPath $sourceRoot -PathType Container)) { + throw "Missing Eval Runner protocol source '$sourceRoot'." + } + + $destinationRoot = Join-Path $IterationDirectory $evalRunnerToolRelativePath + $files = Get-ChildItem -LiteralPath $sourceRoot -Recurse -File -Force | + Where-Object { $_.FullName -notmatch '[\\/]tests[\\/]' } | + ForEach-Object { Get-RelativePath -BasePath $sourceRoot -FullPath $_.FullName } | + Sort-Object + foreach ($relative in $files) { + $source = Join-Path $sourceRoot ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $destination = Join-Path $destinationRoot ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $destination) -Force | Out-Null + Copy-Item -LiteralPath $source -Destination $destination -Force + } + + return $destinationRoot +} + # The candidate skill's fingerprint, computed from the source over exactly the files Copy-SkillTree stages. The staged # copy in each with_skill run must reproduce this value, which is how preparation proves the worker received the # revision under development rather than a globally installed one. @@ -913,12 +1172,14 @@ function New-RunManifest { [string[]]$RepoFiles, [string]$FixtureHash, [string]$SkillHash, - [bool]$GitWorkspace + [bool]$GitWorkspace, + [string]$InteractionFile = '', + [string]$InteractionHash = '' ) $skillDirectory = if ($Configuration -eq 'with_skill') { "$($runDirectoryNames.Skill)/$SkillName" } else { $null } - return [ordered]@{ + $manifest = [ordered]@{ schema = $runSchema evalId = [int]$EvalEntry.id evalName = $EvalName @@ -944,10 +1205,16 @@ function New-RunManifest { mustNotExposeGlobalSkillsOrConfig = $true } } + if (-not [string]::IsNullOrWhiteSpace($InteractionFile)) { + $manifest.interactionFile = $InteractionFile + $manifest.interactionHash = $InteractionHash + } + return $manifest } -# Fail package generation the moment a run violates an isolation invariant, so a contaminated package never reaches a -# harness. These checks operate on the materialized run directories, not on prose. +# Fail package generation the moment a run violates an experimental isolation invariant, so a contaminated package never +# reaches a harness. The filesystemIsolationRequired field declares the staged workspace boundary; hard OS confinement is +# evaluated separately by the selected runner and reported as strict or pragmatic confidence. function Assert-RunIsolation { param( [string]$EvalName, @@ -1000,7 +1267,7 @@ function Assert-RunIsolation { throw "$EvalName/$configuration run.json must require fresh context." } if (-not [bool]$runManifest.filesystemIsolationRequired -or -not [bool]$runManifest.isolatedHomeRequired) { - throw "$EvalName/$configuration run.json must require filesystem and home isolation." + throw "$EvalName/$configuration run.json must require the staged workspace boundary and isolated home." } # 6. No run manifest references the source repository, and 7. none references a global skill install. @@ -1075,6 +1342,9 @@ function Invoke-PrepareMode { throw "No evals selected for '$Skill'." } + $executionSelection = Resolve-ExecutionSelection -RepoRoot $repoRoot + $effectiveConcurrency = Resolve-EffectiveConcurrency -RunnerName ([string]$executionSelection.Runner) -RequestedConcurrency $Concurrency -ConcurrencyWasExplicit ($scriptBoundParameters.ContainsKey('Concurrency')) + $workspaceRoot = if ([string]::IsNullOrWhiteSpace($OutputRoot)) { Join-Path (Join-Path $repoRoot '.bot') "$Skill-workspace" } else { @@ -1108,6 +1378,14 @@ function Invoke-PrepareMode { [void](Copy-ReportTool -RepoRoot $repoRoot -IterationDirectory $iterationDirectory) $skillCreatorSourcePath = Resolve-SkillCreatorSourcePath -RequestedPath $null [void](Copy-SkillCreatorEvalTools -IterationDirectory $iterationDirectory -SourcePath $skillCreatorSourcePath) + $copiedRunnerTools = Copy-EvalRunnerTools -RepoRoot $repoRoot -IterationDirectory $iterationDirectory + $runnerToolsIntegrity = [ordered]@{ + schema = 'codebeltnet/agentic/package-tree-integrity/1' + path = $evalRunnerToolRelativePath + sha256 = Get-TreeHash -Root $copiedRunnerTools + file_count = @(Get-ChildItem -LiteralPath $copiedRunnerTools -Recurse -File -Force).Count + } + ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'execution-profile.json') -Value (New-ExecutionProfile -ExecutionSelection $executionSelection -EffectiveConcurrency ([int]$effectiveConcurrency.Value)) $skillText = [System.IO.File]::ReadAllText($skillMarkdownPath, $utf8NoBom) $skillBody = if ($skillText -match '(?ms)\A---\r?\n.*?\r?\n---\r?\n(?.*)\z') { $Matches['body'] } else { $skillText } @@ -1157,9 +1435,9 @@ function Invoke-PrepareMode { $repoFiles = @($fixtures | ForEach-Object { $_.RepoRelative } | Sort-Object) - # Materialize both runs. Each run directory is the worker's sandbox root: repo/ is the working tree, home/ is an + # Materialize both runs. Each run directory is the worker's staged root: repo/ is the working tree, home/ is an # isolated profile, and skill/ (with_skill only) holds the candidate. The grading key and results live one level - # up, outside every run directory, so a worker confined to its run directory can never reach them. + # up, outside every run directory, so a worker that stays within its run directory is never handed them. foreach ($configuration in @('with_skill', 'without_skill')) { $runDir = Join-Path $evalDirectory $configuration New-Item -ItemType Directory -Path $runDir -Force | Out-Null @@ -1190,6 +1468,7 @@ function Invoke-PrepareMode { $inputFilesSection = New-InputFilesSection -Fixtures @($fixtures) $assertions = Get-Assertions -EvalEntry $evalEntry + $interactionDocument = New-InteractionDocument -EvalEntry $evalEntry $withSkillPrompt = New-PromptDocument -EvalEntry $evalEntry -InstructionSection $withSkillInstructions -InputFilesSection $inputFilesSection $withoutSkillPrompt = New-PromptDocument -EvalEntry $evalEntry -InstructionSection $withoutSkillPreamble -InputFilesSection $inputFilesSection @@ -1197,12 +1476,23 @@ function Invoke-PrepareMode { Write-Utf8File -Path (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Prompt) -Content $withSkillPrompt Write-Utf8File -Path (Join-Path (Join-Path $evalDirectory 'without_skill') $runDirectoryNames.Prompt) -Content $withoutSkillPrompt - ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'with_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $skillHash -GitWorkspace $workspaceOption.Git) - ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'without_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'without_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $null -GitWorkspace $workspaceOption.Git) + $interactionFile = '' + $interactionHash = '' + if ($null -ne $interactionDocument) { + $interactionFile = 'interaction.json' + foreach ($configuration in @('with_skill', 'without_skill')) { + $interactionPath = Join-Path (Join-Path $evalDirectory $configuration) $interactionFile + ConvertTo-JsonFile -Path $interactionPath -Value $interactionDocument + if ([string]::IsNullOrWhiteSpace($interactionHash)) { $interactionHash = Get-FileSha256 -Path $interactionPath } + if ((Get-FileSha256 -Path $interactionPath) -ne $interactionHash) { throw "Scripted interaction sidecar diverged between configurations for '$evalName'." } + } + } + ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'with_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'with_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $skillHash -GitWorkspace $workspaceOption.Git -InteractionFile $interactionFile -InteractionHash $interactionHash) + ConvertTo-JsonFile -Path (Join-Path (Join-Path $evalDirectory 'without_skill') $runDirectoryNames.Run) -Value (New-RunManifest -SkillName $Skill -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'without_skill' -RepoFiles $repoFiles -FixtureHash $fixtureHash -SkillHash $null -GitWorkspace $workspaceOption.Git -InteractionFile $interactionFile -InteractionHash $interactionHash) $assumptions = [System.Collections.Generic.List[string]]::new() $assumptions.Add('Run with_skill and without_skill on the same model, same version, and same configuration. Different models measure the model, not the skill.') - $assumptions.Add('Each run is hermetic: launch a fresh worker with its run directory as the sandbox root, its repo/ as the working directory, and its home/ as the isolated profile.') + $assumptions.Add('Each run is isolated: launch a fresh worker with its run directory as the staged root, its repo/ as the working directory, and its home/ as the isolated profile. The runner reports strict confidence when it also proves hard filesystem confinement and pragmatic confidence when it does not.') $assumptions.Add("Both runs share an identical materialized repository. Only the with_skill run exposes the candidate skill under skill/$Skill/.") $notInlinedFixtures = @($fixtures | Where-Object { -not $_.Inlined }) if ($notInlinedFixtures.Count -gt 0) { @@ -1212,6 +1502,9 @@ function Invoke-PrepareMode { $assumptions.Add('This eval stages a real .git in repo/ so repository-root detection and version-deriving tools behave as on a developer machine.') } $assumptions.Add('The expected output and assertions in this file are the grading key. They live outside every run directory and must never reach a worker.') + if ($null -ne $interactionDocument) { + $assumptions.Add('This eval uses a deterministic scripted same-session interaction sidecar. Every configuration receives the same ordered user turns; the runner must continue the same session and capture every user/assistant turn.') + } $metadata = [ordered]@{ schema = $metadataSchema @@ -1239,16 +1532,19 @@ function Invoke-PrepareMode { prompt_file = "with_skill/$($runDirectoryNames.Prompt)" run_manifest = "with_skill/$($runDirectoryNames.Run)" result_file = 'results/with-skill.result.json' + execution_result_file = 'results/with-skill.execution-result.json' } without_skill = [ordered]@{ run_directory = 'without_skill' prompt_file = "without_skill/$($runDirectoryNames.Prompt)" run_manifest = "without_skill/$($runDirectoryNames.Run)" result_file = 'results/without-skill.result.json' + execution_result_file = 'results/without-skill.execution-result.json' } } assumptions = @($assumptions) } + if ($null -ne $interactionDocument) { $metadata.interaction = $interactionDocument } ConvertTo-JsonFile -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value $metadata ConvertTo-JsonFile -Path (Join-Path $evalDirectory 'results/with-skill.result.json') -Value (New-ResultStub -SkillName $Skill -IterationNumber $iterationNumber -EvalEntry $evalEntry -EvalName $evalName -Configuration 'with_skill' -Assertions $assertions) @@ -1275,6 +1571,7 @@ function Invoke-PrepareMode { home_directory = "$evalName/with_skill/$($runDirectoryNames.Home)" skill_directory = "$evalName/with_skill/$($runDirectoryNames.Skill)/$Skill" result = "$evalName/results/with-skill.result.json" + execution_result = "$evalName/results/with-skill.execution-result.json" } without_skill = [ordered]@{ mode = 'without_skill' @@ -1285,6 +1582,7 @@ function Invoke-PrepareMode { home_directory = "$evalName/without_skill/$($runDirectoryNames.Home)" skill_directory = $null result = "$evalName/results/without-skill.result.json" + execution_result = "$evalName/results/without-skill.execution-result.json" } } }) @@ -1297,8 +1595,22 @@ function Invoke-PrepareMode { iteration = $iterationNumber generated_utc = $generatedUtc configurations = @('with_skill', 'without_skill') - execution = 'external_handoff' + execution = 'runner_handoff' + execution_selection = [ordered]@{ + harness = $executionSelection.Harness + runner = $executionSelection.Runner + model = $executionSelection.Model + preset = $executionSelection.Preset + } runner_prompt = 'RUN-THIS.prompt.md' + execution_profile = 'execution-profile.json' + runner_protocol = $runnerProtocolSchema + runner_tools = $evalRunnerToolRelativePath + runner_tools_integrity = $runnerToolsIntegrity + execution_result_schema = $executionResultSchema + execution_freeze = 'execution-freeze.json' + grading = 'grading.json' + finalizer = "$evalRunnerToolRelativePath/finalize-eval-package.ps1" report = [ordered]@{ tool = $reportToolRelativePath template = 'tools/eval-report-template.html' @@ -1328,7 +1640,7 @@ function Invoke-PrepareMode { 'fresh context', 'isolated HOME/config', 'isolated CWD', - 'filesystem sandbox', + 'staged filesystem/workspace boundary', 'candidate skill exposure', 'transcript capture' ) @@ -1341,11 +1653,26 @@ function Invoke-PrepareMode { } ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'manifest.json') -Value $manifest - Write-Utf8File -Path (Join-Path $iterationDirectory 'README.md') -Content (New-PackageReadme -SkillName $Skill -IterationNumber $iterationNumber -IterationDirectory $iterationDirectory -ManifestEvals @($manifestEvals)) + Write-Utf8File -Path (Join-Path $iterationDirectory 'README.md') -Content (New-PackageReadme -SkillName $Skill -IterationNumber $iterationNumber -IterationDirectory $iterationDirectory -ManifestEvals @($manifestEvals) -ExecutionSelection $executionSelection -EffectiveConcurrency $effectiveConcurrency) $runnerPath = Join-Path $iterationDirectory 'RUN-THIS.prompt.md' - Write-Utf8File -Path $runnerPath -Content (New-RunnerPrompt -IterationDirectory $iterationDirectory -IterationNumber $iterationNumber -ManifestEvals @($manifestEvals)) + Write-Utf8File -Path $runnerPath -Content (New-RunnerPrompt -IterationDirectory $iterationDirectory -IterationNumber $iterationNumber -ManifestEvals @($manifestEvals) -ExecutionSelection $executionSelection -RequestedConcurrency ([int]$effectiveConcurrency.Value) -PerArmTimeoutSeconds $TimeoutSeconds) - Write-Host "Prepared $($manifestEvals.Count) eval case(s) for '$Skill' (iteration $iterationNumber) as $($manifestEvals.Count * 2) hermetic run package(s)." + Write-Host 'Evaluation package prepared.' + Write-Host '' + Write-Host 'Execution:' + Write-Host " Harness: $($executionSelection.Harness)" + Write-Host " Runner: $($executionSelection.Runner)" + Write-Host " Model: $($executionSelection.Model)" + Write-Host " Timeout: $TimeoutSeconds seconds" + Write-Host " Concurrency: $($effectiveConcurrency.Value) ($($effectiveConcurrency.Source))" + if (-not [string]::IsNullOrWhiteSpace([string]$executionSelection.Preset)) { + Write-Host " Preset: $($executionSelection.Preset)" + } + Write-Host '' + Write-Host "Cases: $($manifestEvals.Count)" + Write-Host "Arms: $($manifestEvals.Count * 2)" + Write-Host '' + Write-Host "Prepared $($manifestEvals.Count) eval case(s) for '$Skill' (iteration $iterationNumber) as $($manifestEvals.Count * 2) isolated run package(s)." Write-Host "Package: $iterationDirectory" Write-Host '' Write-Host 'Every run is a self-contained directory: repo/ is the working tree, home/ is an isolated' @@ -1358,139 +1685,87 @@ function Invoke-PrepareMode { Write-Host 'Point the harness at that path. Do not reproduce its contents in chat: a pasted copy' Write-Host 'loses the absolute paths it depends on, and the harness then cannot find the package.' Write-Host '' - Write-Host 'The runner makes the selected agent the evaluator, grader, and report producer. It must create' - Write-Host 'one isolated fresh worker per run, then grade the collected results and generate both report artifacts.' + Write-Host 'The runner-aware handoff uses execution-profile.json, the package-local Eval Runner protocol,' + Write-Host 'and its deterministic native-worker orchestration plan.' + Write-Host 'The selected external orchestrator must honor each runner descriptor''s dispatch_owner: orchestrator-owned native workers stay delegated, while runner-owned transports are owned by one foreground Phase 1 fan-out command.' + Write-Host 'The orchestrator coordinates; it must not execute an eval arm in its own model context or nest a runner-owned model execution inside another worker.' + Write-Host 'Independent workers run concurrently up to execution-profile.json.concurrency; harness capacity remains authoritative.' Write-Host '' Write-Host 'This script prepared prompts only. It did not run them, and nothing here will.' - Write-Host 'The selected evaluator should finish the package in one run. If it cannot write back to this package,' - Write-Host 'bring back the result objects and use the repository collector as a fallback:' - Write-Host (" pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults `"$iterationDirectory`"") + Write-Host 'The selected evaluator must write valid runner-produced execution results back into this package.' + Write-Host 'If it cannot, the evaluation is incomplete and must fail closed; only persisted runner-produced evidence at' + Write-Host 'the manifest-declared paths may proceed.' } function New-RunnerPrompt { param( [string]$IterationDirectory, [int]$IterationNumber, - [object[]]$ManifestEvals + [object[]]$ManifestEvals, + [Parameter(Mandatory = $true)][object]$ExecutionSelection, + [Parameter(Mandatory = $true)][int]$RequestedConcurrency, + [int]$PerArmTimeoutSeconds = 900, + [int]$RunnerGraceSeconds = 30 ) $builder = [System.Text.StringBuilder]::new() - [void]$builder.AppendLine('# Run, grade, and report this evaluation package') - [void]$builder.AppendLine() - [void]$builder.AppendLine('START NOW. You are the evaluator, grader, and report producer for this package. Do not ask me which role to perform, whether to run the workers, or whether to continue to grading. Execute the complete workflow in this one handoff and return the finished report.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('Do not execute evaluation prompts in the current agent context. Create an isolated fresh-context worker for every run, then continue in this coordinator context with grading and report generation after all workers finish.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('Everything you need is in this directory:') - [void]$builder.AppendLine() - [void]$builder.AppendLine(" $IterationDirectory") - [void]$builder.AppendLine() - [void]$builder.AppendLine("This file is `"$(Join-Path $IterationDirectory 'RUN-THIS.prompt.md')`". Use that package path as authoritative. If the package is inaccessible, report the concrete access failure; do not ask the user to choose another workflow.") - [void]$builder.AppendLine() - [void]$builder.AppendLine("It holds $(@($ManifestEvals).Count) eval case(s), each with a `with_skill` and `without_skill` run. Every run is a self-contained directory that holds a `prompt.md`, a `run.json` contract, a `repo/` working tree, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate. The runs differ only in whether that candidate skill is present. You run the experiment, grade the completed results, and generate the report.") - [void]$builder.AppendLine() - [void]$builder.AppendLine('A person selected this harness and handed you this package. That authorizes these runs, grading, aggregation, and report generation. The repository prohibition on automated model evaluation binds repository scripts and the agent that prepared the package; it does not bind the user-directed external executor completing this specific handoff.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('## Execute now') - [void]$builder.AppendLine() - [void]$builder.AppendLine('This package assumes the harness can give each run a fresh context, an isolated HOME/config root, an isolated working directory, and a filesystem sandbox scoped to the run directory. A harness that cannot provide fresh, independent sessions is incompatible with these evals. Continue without asking for role clarification, and report per run which guarantees you satisfied - fresh context, isolated HOME/config, isolated CWD, filesystem sandbox, candidate skill exposure, and transcript capture.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('## Orchestration contract') - [void]$builder.AppendLine() - [void]$builder.AppendLine('1. For every eval case, create one isolated fresh-context worker for `with_skill` and a second isolated fresh-context worker for `without_skill`. Never reuse a worker or session between runs, between cases, or between iterations.') - [void]$builder.AppendLine('2. Launch each worker from its own run directory, which is the worker''s sandbox root. Set the working directory to that run''s `repo/`, set HOME and the platform-equivalent profile and config roots to its `home/`, and confine filesystem access to the run directory. Read the run''s `run.json` for the exact contract: `workingDirectory`, `homeDirectory`, `skillDirectory`, and the fresh-context, filesystem, and home isolation flags.') - [void]$builder.AppendLine('3. Give each worker only its `prompt.md` and the files already staged in its run directory. Do not expose this runner, `manifest.json`, any `eval-metadata.json`, `comparison.md`, result files, grading criteria, expectations, the paired run, another case''s output, or any note that an experiment is underway. All of those live outside the run directory, so keeping the worker inside it keeps them hidden.') - [void]$builder.AppendLine('4. The candidate skill is already inlined in the with_skill run''s `prompt.md` and staged under its `skill/` directory. Do not load, summarize, or add it yourself. The without_skill run carries no skill instructions and no `skill/` directory; do not expose the candidate skill to that worker by any route, including a globally installed copy.') - [void]$builder.AppendLine('5. Send each `prompt.md` unchanged as the worker''s first message. The input files are already real files in the worker''s `repo/`; the worker reads and edits them there rather than from attachments.') - [void]$builder.AppendLine('6. Use the same model, version, configuration, tools, and limits for every worker. Disable persistent memory or cross-session recall. Independent runs may execute concurrently when the selected harness and token budget allow it.') - [void]$builder.AppendLine('7. Record the worker''s complete response, transcript when available, token usage, elapsed time, and tool-call count. When the harness exposes them, also record the shell commands, files read and written, stdout and stderr, and exit status, and which isolation guarantees you satisfied. Record refusals, questions, and failures as results. Do not retry to improve an answer.') - [void]$builder.AppendLine('8. Work only inside this package. Do not read or modify the source repository around it. Do not begin grading until every available worker has completed or failed and its result is recorded.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('For each case in `manifest.json`, the `runs.with_skill` and `runs.without_skill` entries give each run''s directory, its `prompt`, its `run_manifest` (`run.json`), and the `result` file to write. Run the two prompts in separate workers, then overwrite the matching result file without reading its existing contents. A partial package is valid: record every completed run, continue to grading/reporting, and mark missing arms honestly instead of asking what to do next.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('## Result shape') - [void]$builder.AppendLine() - [void]$builder.AppendLine('```json') - [void]$builder.AppendLine('{') - [void]$builder.AppendLine(' "schema": "codebeltnet/agentic/eval-result/2",') - [void]$builder.AppendLine(" `"iteration`": $IterationNumber,") - [void]$builder.AppendLine(' "eval_id": 1,') - [void]$builder.AppendLine(' "eval_name": "the directory name",') - [void]$builder.AppendLine(' "configuration": "with_skill",') - [void]$builder.AppendLine(' "model": "the exact model id you used",') - [void]$builder.AppendLine(' "provider": "who served it",') - [void]$builder.AppendLine(' "harness": "what you are",') - [void]$builder.AppendLine(' "executed_utc": "2026-01-01T00:00:00Z",') - [void]$builder.AppendLine(' "output": "the complete response the run produced",') - [void]$builder.AppendLine(' "output_files": ["paths of any files the run wrote"],') - [void]$builder.AppendLine(' "transcript": "the complete worker transcript when the harness exposes it",') - [void]$builder.AppendLine(' "shell_commands": ["commands the run executed, when exposed"],') - [void]$builder.AppendLine(' "files_read": ["paths the run read, when exposed"],') - [void]$builder.AppendLine(' "files_written": ["paths the run wrote, when exposed"],') - [void]$builder.AppendLine(' "exit_status": 0,') - [void]$builder.AppendLine(' "duration_seconds": 12.5,') - [void]$builder.AppendLine(' "total_tokens": 1234,') - [void]$builder.AppendLine(' "tool_calls": 6,') - [void]$builder.AppendLine(' "turns": 12,') - [void]$builder.AppendLine(' "base_input_tokens": 27,') - [void]$builder.AppendLine(' "output_tokens": 3800,') - [void]$builder.AppendLine(' "cache_read_tokens": 515605,') - [void]$builder.AppendLine(' "cache_write_1h_tokens": 129582,') - [void]$builder.AppendLine(' "estimated_cost_usd": 2.27,') - [void]$builder.AppendLine(' "model_effort": "high",') - [void]$builder.AppendLine(' "isolation": {') - [void]$builder.AppendLine(' "fresh_context": true,') - [void]$builder.AppendLine(' "isolated_home": true,') - [void]$builder.AppendLine(' "isolated_cwd": true,') - [void]$builder.AppendLine(' "filesystem_sandbox": true,') - [void]$builder.AppendLine(' "candidate_skill_exposed": true,') - [void]$builder.AppendLine(' "transcript_captured": true') - [void]$builder.AppendLine(' },') - [void]$builder.AppendLine(' "grading": [],') - [void]$builder.AppendLine(' "notes": "anything that would change how this result reads"') - [void]$builder.AppendLine('}') - [void]$builder.AppendLine('```') - [void]$builder.AppendLine() - [void]$builder.AppendLine('`transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, `tool_calls`, the optional efficiency telemetry fields, and every `isolation` flag are optional. Include each when the harness exposes it and omit it otherwise. Never estimate a missing value. For `with_skill`, set `isolation.candidate_skill_exposed` to how the skill actually reached the worker.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('`configuration` is `with_skill` or `without_skill` and must match the prompt you ran. Read `eval_id` and `eval_name` from `manifest.json`; do not send them to the worker. Put the full model response in `output`. If it is very long, write it beside the result file and list that path in `output_files` with a summary in `output`.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('`output` is the model''s message in full, including questions, caveats, explanations, or a refusal. Tool output is evidence from the run, not a replacement for the model response. Put the full worker event history in `transcript` when the harness exposes it.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('## Grade and report immediately') - [void]$builder.AppendLine() - [void]$builder.AppendLine('After all available workers finish, read each eval''s `eval-metadata.json`. Only now may you read `expected_output` and `assertions`; they are the grading key and were intentionally hidden from the workers.') - [void]$builder.AppendLine('1. Grade every completed result against every assertion. Use deterministic checks for mechanical assertions and concrete output, transcript, and file evidence for process assertions. Use judgement only where the assertion is genuinely qualitative, and say so in the evidence. Never infer a tool or file action from the model''s self-report when process evidence is absent.') - [void]$builder.AppendLine('2. Write grading back into the matching result file using exactly `grading[].text`, `grading[].passed`, and `grading[].evidence`. Use `passed: null` when an assertion cannot be judged from captured evidence. Do not grade a missing run as passed.') - [void]$builder.AppendLine('The package carries Anthropic skill-creator under `tools/skill-creator`. Use `tools/skill-creator/agents/grader.md` for grading guidance, and use `tools/skill-creator/scripts/aggregate_benchmark.py` plus `tools/skill-creator/eval-viewer/generate_review.py` as the source-of-truth aggregation and review tools.') - $reportCommand = 'pwsh -NoProfile -File "' + (Join-Path $IterationDirectory $reportToolRelativePath) + '" -IterationDirectory "' + $IterationDirectory + '"' - [void]$builder.AppendLine(('3. Run the package report adapter now; do not ask the user to run a second command: ' + $reportCommand + '. It stages the recorded results into the upstream skill-creator workspace contract, invokes the exact upstream aggregator and static viewer, then writes the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` at the package root.')) - [void]$builder.AppendLine('4. If the harness can open local files, open `report.html` after it is written. Otherwise return its absolute path as the primary artifact. Do not wait for browser feedback before finishing the handoff.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('The report is the completion artifact. Do not stop after worker execution, do not return a prose-only recap, and do not ask whether grading or HTML generation is wanted.') + $profilePath = Join-Path $IterationDirectory 'execution-profile.json' + $runnerOwnedFanoutPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/invoke-runner-owned-arms.ps1" + $manifestBridgePath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-manifest-results.ps1" + $finalizerPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/finalize-eval-package.ps1" + $maxScriptedUserTurns = 1 + foreach ($manifestEval in @($ManifestEvals)) { + $metadataPath = Join-Path $IterationDirectory ([string](Get-JsonProperty -Object $manifestEval -Name 'metadata' -Default '')) + if (Test-Path -LiteralPath $metadataPath -PathType Leaf) { + try { + $metadata = Get-Content -LiteralPath $metadataPath -Raw -Encoding UTF8 | ConvertFrom-Json + $turns = @(Get-JsonProperty -Object (Get-JsonProperty -Object $metadata -Name 'interaction' -Default $null) -Name 'turns' -Default @()) + if ($turns.Count -gt $maxScriptedUserTurns) { $maxScriptedUserTurns = $turns.Count } + } catch { + # Prompt generation remains usable for the unit-test helper, + # while real packages already validate metadata before this + # function is called. + } + } + } + $armCount = [Math]::Max(1, @($ManifestEvals).Count * 2) + $effectiveConcurrency = [Math]::Max(1, $RequestedConcurrency) + $executionBatches = [Math]::Max(1, [int][Math]::Ceiling($armCount / [double]$effectiveConcurrency)) + $preflightTimeoutSeconds = [Math]::Max(120, [Math]::Max(1, $PerArmTimeoutSeconds) + [Math]::Max(0, $RunnerGraceSeconds)) + $serialPreflightAllowanceSeconds = $armCount * $preflightTimeoutSeconds + $longestChildAllowanceSeconds = ([Math]::Max(1, $maxScriptedUserTurns) * [Math]::Max(1, $PerArmTimeoutSeconds)) + [Math]::Max(0, $RunnerGraceSeconds) + $executionAllowanceSeconds = $executionBatches * $longestChildAllowanceSeconds + $orchestrationGraceSeconds = [Math]::Max(1, $RunnerGraceSeconds) + $phaseOneAllowanceSeconds = $serialPreflightAllowanceSeconds + $executionAllowanceSeconds + $orchestrationGraceSeconds + [void]$builder.AppendLine('# Execute, grade, and report this evaluation package') [void]$builder.AppendLine() - [void]$builder.AppendLine('## Final handoff') + [void]$builder.AppendLine('START NOW. You are the external Eval Orchestrator for this user-directed handoff. Do not execute an eval prompt in your own context. The repository preparation and validation flow remain model-free.') [void]$builder.AppendLine() - [void]$builder.AppendLine('The finished artifacts are the first-party paired review and the exact upstream skill-creator viewer report, not a request for another command. If you can write to the package machine, leave every result, grading field, `benchmark.json`, `benchmark.md`, `report.html`, and `skill-creator-report.html` in place. Return the absolute first-party report path, the completed/expected run count, any missing arms, the model/provider, and a concise quality summary.') + [void]$builder.AppendLine("Package: $IterationDirectory") + [void]$builder.AppendLine("Profile: $profilePath") [void]$builder.AppendLine() - [void]$builder.AppendLine('If you cannot write to the package machine, return one fenced JSON block containing every completed result object, including its `grading` array, plus the generated report as an artifact when the harness supports file handoff. Do not return separate blocks or a human summary in place of the result objects. State any missing arms and the concrete artifact-transfer limitation.') + [void]$builder.AppendLine('## Phase 1 — blind execution') [void]$builder.AppendLine() - [void]$builder.AppendLine('If you cannot write to that machine - a different product, a browser, a sandbox that shares no disk with it - the results have to travel as text. End with one fenced block, and say plainly that it is meant to be pasted into the repository session as-is:') + [void]$builder.AppendLine(("Phase 1 is one long-running foreground command. Set the caller shell/tool timeout to at least the package-computed Phase 1 allowance of {0} seconds ({1}-second serial preflight allowance for {2} arm(s) at the max(120, profile.timeout_seconds + runner grace) policy + {3}-second execution allowance for {4} concurrent batch(es) + {5}-second orchestration grace), not a default short timeout. For OpenCode, pass this allowance to the shell tool's explicit timeout setting before invoking the command." -f $phaseOneAllowanceSeconds, $serialPreflightAllowanceSeconds, $armCount, $executionAllowanceSeconds, $executionBatches, $orchestrationGraceSeconds)) + [void]$builder.AppendLine('A caller-side shell timeout is not permission to invoke Phase 1 again. `invoke-runner-owned-arms.ps1` must be started exactly once for this iteration. If execution is interrupted and no valid `execution-freeze.json` exists, the package is incomplete and requires a fresh iteration.') [void]$builder.AppendLine() - [void]$builder.AppendLine('```') - [void]$builder.AppendLine('Eval results, grading, and report artifact.') - [void]$builder.AppendLine("Package: $IterationDirectory") - [void]$builder.AppendLine('Model: via , harness ') + [void]$builder.AppendLine('Read the selected runner descriptor and its `delegation.dispatch_owner`. For runner-owned behavioral transport, invoke:') + [void]$builder.AppendLine("pwsh -NoProfile -File `"$runnerOwnedFanoutPath`" -IterationDirectory `"$IterationDirectory`"") + [void]$builder.AppendLine('It performs every preflight before any execute process, preserves exact manifest paths, owns concurrency/backpressure, timeout/watchdog handling, terminal registration, orchestration evidence, and immutable `execution-freeze.json` before Phase 2. Consume its terminal JSON summary. If Phase 1 reports incompatible or fails, stop: the evaluation is incomplete and must fail closed. The evaluation is incomplete and a fresh package/code fix is required. Never patch package-local runner code, delete orchestration state, delete execution results, delete or replace `execution-freeze.json`, rerun Phase 1, or manually broaden a capability check. Do not create outer workers, execute an arm yourself, write orchestration state, or edit raw result/evidence files. If dispatch ownership is orchestrator-owned, use only the descriptor-declared native worker transport, the exact manifest paths, and then run the shared freeze boundary; do not synthesize or repair transport evidence. Only persisted runner-produced evidence at the manifest-declared paths may proceed.') + [void]$builder.AppendLine('Workers receive only their isolated run directory. Keep the paired arm, metadata, expected output, assertions, grading, reports, and orchestration files out of Phase 1. Preserve runner-owned terminal results and all referenced raw transcript/event artifacts exactly as written.') [void]$builder.AppendLine() - [void]$builder.AppendLine('') + [void]$builder.AppendLine('## Phase 2 — grading and finalization') [void]$builder.AppendLine() - [void]$builder.AppendLine('Still unfilled: ') - [void]$builder.AppendLine('```') + [void]$builder.AppendLine('Only after Phase 1 returns a successful terminal JSON summary, invoke the deterministic manifest bridge to validate the freeze and populate the canonical result paths before grading:') + [void]$builder.AppendLine("pwsh -NoProfile -File `"$manifestBridgePath`" -IterationDirectory `"$IterationDirectory`" -RequireComplete -RequireParallelDispatch") + [void]$builder.AppendLine('Only if that bridge succeeds, reveal the grading key in `eval-metadata.json` to the Grader. The Grader may author exactly one package-root `grading.json` with schema `codebeltnet/agentic/eval-grading/1`; each entry contains only `eval_id`, `eval_name`, `configuration`, `assertion_index`, `assertion`, `passed`, and `evidence`. It must not edit raw execution results, canonical non-grading fields, hashes, paths, telemetry, or orchestration state.') + [void]$builder.AppendLine('Write that artifact. Do not invoke the application helper separately; the finalizer invokes `apply-eval-grading.ps1` deterministically. Then invoke exactly once:') + [void]$builder.AppendLine("pwsh -NoProfile -File `"$finalizerPath`" -IterationDirectory `"$IterationDirectory`"") + [void]$builder.AppendLine('The finalizer revalidates the manifest, profile, terminal orchestration/concurrency evidence, immutable freeze, raw artifacts, bridge, canonical results, and grading; it then generates and verifies all required reports. Return only its machine-readable JSON summary and artifact paths. A non-zero exit, missing artifact, integrity error, or report error means the evaluation is incomplete. Never repair, re-freeze, re-bridge a changed raw result, or report prose success.') [void]$builder.AppendLine() - [void]$builder.AppendLine('One block covering everything you ran, not one per case, and the outputs go in it verbatim - a summary written for a human to skim cannot be graded against assertions.') + [void]$builder.AppendLine('The four required package-root artifacts are `report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md`. Same-session scripted evals, when present in a run, are handled by the selected runner only if its descriptor/preflight proves `scripted_multi_turn_same_session`; otherwise preflight fails before execution. The paired configurations receive identical scripted user turns.') [void]$builder.AppendLine() - [void]$builder.AppendLine('The repository collector is only a fallback when result files were transferred without the report artifacts: `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults `. It validates the returned files and invokes the same packaged skill-creator aggregator and viewer; it is not the normal next step after this prompt.') - + [void]$builder.AppendLine("This package contains $(@($ManifestEvals).Count) eval case(s), each with paired `with_skill` and `without_skill` runs. The human reviewer remains the final evaluator.") return $builder.ToString() } @@ -1499,13 +1774,38 @@ function New-PackageReadme { [string]$SkillName, [int]$IterationNumber, [string]$IterationDirectory, - [object[]]$ManifestEvals + [object[]]$ManifestEvals, + [Parameter(Mandatory = $true)][object]$ExecutionSelection, + [Parameter(Mandatory = $true)][object]$EffectiveConcurrency ) + $maxScriptedUserTurns = 1 + foreach ($manifestEval in @($ManifestEvals)) { + $metadataPath = Join-Path $IterationDirectory ([string](Get-JsonProperty -Object $manifestEval -Name 'metadata' -Default '')) + if (Test-Path -LiteralPath $metadataPath -PathType Leaf) { + try { + $metadata = Get-Content -LiteralPath $metadataPath -Raw -Encoding UTF8 | ConvertFrom-Json + $turns = @(Get-JsonProperty -Object (Get-JsonProperty -Object $metadata -Name 'interaction' -Default $null) -Name 'turns' -Default @()) + if ($turns.Count -gt $maxScriptedUserTurns) { $maxScriptedUserTurns = $turns.Count } + } catch { } + } + } + $armCount = [Math]::Max(1, @($ManifestEvals).Count * 2) + $requestedConcurrency = [Math]::Max(1, [int]$EffectiveConcurrency.Value) + $executionBatches = [Math]::Max(1, [int][Math]::Ceiling($armCount / [double]$requestedConcurrency)) + $runnerGraceSeconds = 30 + $preflightTimeoutSeconds = [Math]::Max(120, [Math]::Max(1, $TimeoutSeconds) + $runnerGraceSeconds) + $serialPreflightAllowanceSeconds = $armCount * $preflightTimeoutSeconds + $longestChildAllowanceSeconds = ([Math]::Max(1, $maxScriptedUserTurns) * [Math]::Max(1, $TimeoutSeconds)) + $runnerGraceSeconds + $executionAllowanceSeconds = $executionBatches * $longestChildAllowanceSeconds + $orchestrationGraceSeconds = $runnerGraceSeconds + $phaseOneAllowanceSeconds = $serialPreflightAllowanceSeconds + $executionAllowanceSeconds + $orchestrationGraceSeconds $builder = [System.Text.StringBuilder]::new() [void]$builder.AppendLine("# Eval package: $SkillName (iteration $IterationNumber)") [void]$builder.AppendLine() - [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. You choose the harness, provider, and model; the selected external evaluator runs both configurations, grades them, and generates the report.') + [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, runner-native model, and limits; the external Eval Orchestrator follows the selected descriptor''s `delegation.dispatch_owner`, using either orchestrator-owned native workers or runner-owned native transports, then grades and generates the report.') + [void]$builder.AppendLine() + [void]$builder.AppendLine("Execution selection: runner=$($ExecutionSelection.Runner); model=$($ExecutionSelection.Model); timeout_seconds=$TimeoutSeconds; concurrency=$($EffectiveConcurrency.Value); concurrency_source=$($EffectiveConcurrency.Source).") [void]$builder.AppendLine() [void]$builder.AppendLine('## What is here') [void]$builder.AppendLine() @@ -1513,39 +1813,54 @@ function New-PackageReadme { [void]$builder.AppendLine("- ``$($entry.eval_name)/`` - eval $($entry.eval_id)") } [void]$builder.AppendLine() - [void]$builder.AppendLine('Each eval directory holds the grading key (`eval-metadata.json`), result stubs under `results/`, and two hermetic run directories: `with_skill/` and `without_skill/`. A run directory holds `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate skill. The grading key and results sit outside both run directories, so a worker confined to its run directory never sees them.') + [void]$builder.AppendLine('Each eval directory holds the grading key (`eval-metadata.json`), result stubs under `results/`, and two isolated run directories: `with_skill/` and `without_skill/`. A run directory holds `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate skill. The grading key and results sit outside both run directories, so a worker that stays within its run directory is never handed them.') + [void]$builder.AppendLine('The package root also holds `execution-profile.json`, the package-local Eval Runner protocol and deterministic native-worker queue under `tools/eval-runners/`, and raw `execution-result.json` paths beside the existing result stubs. `run.json` defines what one blind arm must execute; the profile defines with what runner/model/configuration; the selected runner defines how its native worker is created.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('## Orchestration topology') + [void]$builder.AppendLine() + [void]$builder.AppendLine('```text') + [void]$builder.AppendLine('Eval Orchestrator') + [void]$builder.AppendLine(' |') + [void]$builder.AppendLine(' +-- Eval Worker -> one eval arm') + [void]$builder.AppendLine(' +-- Eval Worker -> one eval arm') + [void]$builder.AppendLine(' +-- Eval Worker -> one eval arm') + [void]$builder.AppendLine(' +-- ...') + [void]$builder.AppendLine('```') + [void]$builder.AppendLine('The Eval Orchestrator coordinates and collects; it never executes an eval arm in its own model context. One arm equals one fresh native Eval Worker and one model-backed eval execution. With `dispatch_owner=orchestrator`, the native subagent/task is the worker; with `dispatch_owner=runner`, the runner process/thread is the worker and no outer model subagent is created. Independent workers must run concurrently up to `execution-profile.json.concurrency` when capacity permits; harness capacity is authoritative, so a rejected dispatch stays queued and is not an attempt. A serial result without persisted capacity-limit evidence fails the completion gate.') [void]$builder.AppendLine('The package also carries the exact Anthropic skill-creator assets used after execution under `tools/skill-creator`: `tools/skill-creator/agents/grader.md`, `tools/skill-creator/agents/comparator.md`, `tools/skill-creator/agents/analyzer.md`, `tools/skill-creator/references/schemas.md`, `tools/skill-creator/scripts/aggregate_benchmark.py`, and `tools/skill-creator/eval-viewer/generate_review.py` plus `tools/skill-creator/eval-viewer/viewer.html`.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Isolation model') [void]$builder.AppendLine() [void]$builder.AppendLine('The package guarantees what a generator can: identical materialized repositories for both runs, the candidate skill staged only under `with_skill/skill/`, an empty isolated `home/` per run, and a `run.json` that names only paths inside the run directory. Fixture and skill hashes are recorded so you can prove what each worker received.') [void]$builder.AppendLine() - [void]$builder.AppendLine('The harness must supply the rest at runtime: a fresh context per run, the run directory as the working and config root (working directory `repo/`, HOME `home/`), and a filesystem sandbox that keeps the worker inside its run directory so global skills, global config, the source repository, the paired run, and the grading key stay out of reach. Prompt wording alone does not enforce this; the sandbox does.') + [void]$builder.AppendLine('The harness must supply the rest at runtime: a fresh context per run, the run directory as the working and config root (working directory `repo/`, HOME `home/`), controlled candidate-skill exposure, prompt fidelity, and sufficient response capture. Because global skills, global config, the source repository, the paired run, and the grading key are never staged inside a run directory, a worker that stays within its run directory is not handed them. The selected runner enforces these controls - prompt wording alone does not - and reports strict confidence when it also proves hard filesystem confinement or pragmatic confidence when the mandatory controls hold without it. Hard filesystem confinement is an added confidence signal, not a prerequisite, so Windows and other hosts without a hard sandbox run in pragmatic mode.') [void]$builder.AppendLine() [void]$builder.AppendLine('## How to run') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. Pick one model and configuration. Use the same one for every run in this iteration.') - [void]$builder.AppendLine('2. For each eval, launch a fresh worker for `with_skill/` with its run directory as the sandbox root, `repo/` as the working directory, and `home/` as HOME. Send `prompt.md` as the first message. Read `run.json` for the contract.') - [void]$builder.AppendLine('3. Launch a second fresh worker for `without_skill/` the same way. Never reuse a worker between runs.') - [void]$builder.AppendLine('4. Save each response into the matching file under the eval''s `results/` directory, grade every completed result using the packaged `agents/grader.md` guidance, and run `tools/generate-eval-report.ps1`. The resulting `report.html` is the first-party side-by-side review; `skill-creator-report.html` is the exact upstream viewer.') + [void]$builder.AppendLine(("Phase 1 allowance: {0} seconds ({1} arm(s), concurrency {2}, {3} execution batch(es), timeout_seconds {4}). For OpenCode, use the shell tool's explicit timeout setting with at least this value." -f $phaseOneAllowanceSeconds, $armCount, $requestedConcurrency, $executionBatches, $TimeoutSeconds)) + [void]$builder.AppendLine() + [void]$builder.AppendLine(('1. Read `execution-profile.json` and the selected runner descriptor. If `runner` or `model` is missing or unsupported, fail clearly instead of guessing. For `delegation.dispatch_owner=runner`, invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/invoke-runner-owned-arms.ps1") + ' exactly once with the caller shell/tool timeout set to at least the package-computed Phase 1 allowance. It performs all preflight, native dispatch, concurrency, terminal registration, timeout handling, and raw-evidence freezing. For `delegation.dispatch_owner=orchestrator`, use only the descriptor-declared native worker mechanism, then invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/freeze-execution-evidence.ps1") + ' after every arm is terminal.')) + [void]$builder.AppendLine('2. A caller/tool timeout or interrupted conversation does not authorize rerunning Phase 1. Do not execute an arm in the parent context, create a second worker for a runner-owned arm, expose grading material during execution, or author/repair raw evidence. If Phase 1 reports incompatible or Phase 1/freezing fails, stop: the evaluation is incomplete and a fresh package/code fix is required. Never patch package-local runner code, delete orchestration state, delete execution results, delete or replace `execution-freeze.json`, rerun Phase 1, or manually broaden a capability check.') + [void]$builder.AppendLine(('3. After the freeze succeeds, invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-manifest-results.ps1") + ' with `-RequireComplete -RequireParallelDispatch` and add `-RequireNativeDelegation` when the selected descriptor has `dispatch_owner=runner`. Only after that deterministic bridge succeeds, give the grading key to the Grader. The Grader writes only the package-root `grading.json` grading-only artifact. It must not modify execution results, canonical non-grading fields, hashes, paths, telemetry, or orchestration state.')) + [void]$builder.AppendLine(('4. Invoke ' + (Join-Path $IterationDirectory "$evalRunnerToolRelativePath/finalize-eval-package.ps1") + ' exactly once. It invokes the deterministic apply-eval-grading boundary, validates the frozen evidence, idempotent bridge, complete grading, and report outputs. Return only its machine-readable summary.')) [void]$builder.AppendLine() - [void]$builder.AppendLine('`RUN-THIS.prompt.md` turns a harness that can create isolated workers or sessions into the evaluator, grader, and report producer. It reads the package, creates one new worker per run from its run directory, keeps runner instructions and grading data out of every worker, records results, grades after collection, and invokes Anthropic skill-creator''s aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') + [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It never executes an eval arm in its own model context. Same-session scripted interactions are allowed only when the selected runner proves that capability; paired runs receive identical deterministic turns. The package is complete only when the finalizer exits successfully.') [void]$builder.AppendLine() - [void]$builder.AppendLine('A harness that cannot provide fresh, independent sessions with isolated working and config roots is incompatible with these evals. `-CollectResults` accepts a partial iteration and reports unfilled runs as missing.') + [void]$builder.AppendLine('A harness that cannot provide fresh, independent sessions with isolated working and config roots is incompatible with these evals. If the selected external process cannot write valid runner-produced execution results back into the package, the evaluation is incomplete and must fail closed. Only persisted runner-produced evidence at the manifest-declared paths may proceed to Phase 2.') [void]$builder.AppendLine() [void]$builder.AppendLine('A with_skill run on one model compared against a baseline on another measures both the model and the skill. That is not a skill-effectiveness result, so do not report it as one. If you do mix models, say so explicitly and treat the comparison as directional only.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Report artifacts') [void]$builder.AppendLine() - [void]$builder.AppendLine('Fill in each `results/*.result.json`:') + [void]$builder.AppendLine('The package bridge writes each `results/*.result.json` from the runner-produced execution result at its exact manifest-declared path. The external Grader authors only the package-root grading artifact; the deterministic application helper projects it onto canonical results without changing any other field:') [void]$builder.AppendLine() - [void]$builder.AppendLine('- `model`, `provider`, `harness` - what actually ran it, as specifically as you know') + [void]$builder.AppendLine('- `model`, `harness` - what actually ran it, as specifically as you know') [void]$builder.AppendLine('- `executed_utc` - when') [void]$builder.AppendLine('- `output` - the produced output, or a summary plus paths in `output_files`') [void]$builder.AppendLine('- `transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, `tool_calls` - include the values the harness exposes; omit unavailable values rather than estimating them') [void]$builder.AppendLine('- Optional efficiency telemetry: `turns`, `base_input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens` or `cache_write_1h_tokens`, `estimated_cost_usd`, and `model_effort`. These are shown when recorded and never inferred from totals.') [void]$builder.AppendLine('- `isolation` - the guarantees the harness satisfied for this run; process-dependent assertions can only be graded from a run that captured the needed evidence') - [void]$builder.AppendLine('- `grading[].passed` - `true`, `false`, or `null` per assertion once the external evaluator or a deterministic script has checked it, with `evidence`') + [void]$builder.AppendLine('- `grading.json` - schema `codebeltnet/agentic/eval-grading/1`, with exact assertion identities and only `passed` plus `evidence` decisions') [void]$builder.AppendLine('- `notes` - anything that would change how the result reads') [void]$builder.AppendLine() [void]$builder.AppendLine('The normal handoff finishes with these package-root artifacts:') @@ -1557,9 +1872,6 @@ function New-PackageReadme { [void]$builder.AppendLine('benchmark.md Anthropic skill-creator human-readable benchmark summary') [void]$builder.AppendLine('```') [void]$builder.AppendLine() - [void]$builder.AppendLine('If the harness cannot write to the package machine, its final handoff should contain one paste-ready JSON array of completed result objects including grading, plus the report as a file artifact when supported. A prose-only recap is not sufficient.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('For transferred results without report artifacts, the repository-side fallback is `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults `, which validates the files and invokes the same packaged skill-creator aggregator and viewer.') [void]$builder.AppendLine() [void]$builder.AppendLine('This workspace is temporary. Do not commit it to the repository unless someone explicitly asks for a checked-in example.') @@ -1681,39 +1993,87 @@ function Invoke-CollectMode { $errors = [System.Collections.Generic.List[string]]::new() $warnings = [System.Collections.Generic.List[string]]::new() $rows = [System.Collections.Generic.List[object]]::new() + $manifestRecords = @(Get-ManifestRunRecords -IterationDirectory $iterationDirectory -Manifest $manifest) + $runnerAware = $manifest.PSObject.Properties.Name -contains 'execution_profile' + if ($runnerAware) { + $packageBridgePath = Join-Path $iterationDirectory "$($manifest.runner_tools)/bridge-manifest-results.ps1" + if (-not (Test-Path -LiteralPath $packageBridgePath -PathType Leaf)) { + $packageBridgePath = Join-Path (Join-Path (Get-RepoRoot) 'scripts') 'eval-runners/bridge-manifest-results.ps1' + } + if (-not (Test-Path -LiteralPath $packageBridgePath -PathType Leaf)) { + $errors.Add("Manifest-driven bridge is missing at '$packageBridgePath'.") + } else { + $requireNativeDelegation = $true + $profilePath = Join-Path $iterationDirectory ([string]$manifest.execution_profile) + if (Test-Path -LiteralPath $profilePath -PathType Leaf) { + $profileForCollection = [System.IO.File]::ReadAllText($profilePath, $utf8NoBom) | ConvertFrom-Json + if ([string]$profileForCollection.runner -eq 'fake') { + # The deterministic fake is a compatibility fixture, not a + # harness-native worker. Never treat its execute output as + # native delegation evidence. + $requireNativeDelegation = $false + } + } + $bridgeArguments = @('-NoProfile', '-File', $packageBridgePath, '-IterationDirectory', $iterationDirectory) + if ($requireNativeDelegation) { $bridgeArguments += '-RequireNativeDelegation' } + $bridgeOutput = & pwsh @bridgeArguments 2>&1 + if ($LASTEXITCODE -ne 0) { + $errors.Add("Manifest-driven execution-result bridge failed: $([string]::Join(' ', @($bridgeOutput)))") + } else { + foreach ($line in @($bridgeOutput)) { + if (-not [string]::IsNullOrWhiteSpace([string]$line)) { + Write-Host $line + } + } + } + } + } foreach ($entry in @($manifest.evals)) { - $evalDirectory = Join-Path $iterationDirectory $entry.directory - $metadata = [System.IO.File]::ReadAllText((Join-Path $evalDirectory 'eval-metadata.json'), $utf8NoBom) | ConvertFrom-Json + $entryRecords = @($manifestRecords | Where-Object { [int]$_.EvalId -eq [int]$entry.eval_id }) + if ($entryRecords.Count -eq 0) { + $errors.Add("$($entry.eval_name) has no manifest-declared arm paths.") + continue + } + $evalDirectory = [string]$entryRecords[0].EvalDirectory + $metadata = [System.IO.File]::ReadAllText([string]$entryRecords[0].MetadataPath, $utf8NoBom) | ConvertFrom-Json $observed = @{} - foreach ($configuration in @('with_skill', 'without_skill')) { - $fileName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } - $resultPath = Join-Path (Join-Path $evalDirectory 'results') $fileName + foreach ($configuration in @(Get-ManifestConfigurations -Manifest $manifest)) { + $runRecords = @($entryRecords | Where-Object { [string]$_.Configuration -eq $configuration }) + if ($runRecords.Count -ne 1) { + $errors.Add("$($entry.eval_name)/$configuration must have exactly one manifest-declared arm path set.") + continue + } + $runRecord = $runRecords[0] + $resultPath = [string]$runRecord.ResultPath + $resultRelative = [string]$runRecord.ResultRelative if (-not (Test-Path -LiteralPath $resultPath)) { - $warnings.Add("$($entry.eval_name)/$configuration - no result file at results/$fileName.") + $warnings.Add("$($entry.eval_name)/$configuration - no result file at the manifest path '$resultRelative'.") continue } try { $result = [System.IO.File]::ReadAllText($resultPath, $utf8NoBom) | ConvertFrom-Json } catch { - $errors.Add("$($entry.eval_name)/$configuration - results/$fileName is not valid JSON: $($_.Exception.Message)") + $errors.Add("$($entry.eval_name)/$configuration - manifest result '$resultRelative' is not valid JSON: $($_.Exception.Message)") continue } if ([string]$result.configuration -ne $configuration) { - $errors.Add("$($entry.eval_name)/$configuration - results/$fileName declares configuration '$($result.configuration)'.") + $errors.Add("$($entry.eval_name)/$configuration - manifest result '$resultRelative' declares configuration '$($result.configuration)'.") continue } if ([int]$result.eval_id -ne [int]$metadata.eval_id) { - $errors.Add("$($entry.eval_name)/$configuration - results/$fileName declares eval_id $($result.eval_id) but the package says $($metadata.eval_id).") + $errors.Add("$($entry.eval_name)/$configuration - manifest result '$resultRelative' declares eval_id $($result.eval_id) but the package says $($metadata.eval_id).") continue } $outputText = [string](Get-JsonProperty -Object $result -Name 'output' -Default '') $outputFiles = @(Get-JsonProperty -Object $result -Name 'output_files' -Default @()) - $hasOutput = -not [string]::IsNullOrWhiteSpace($outputText) -or $outputFiles.Count -gt 0 + $executionStatus = [string](Get-JsonProperty -Object $result -Name 'execution_status' -Default '') + $hasExecution = -not [string]::IsNullOrWhiteSpace($executionStatus) -and $executionStatus -ne 'unrun' + $hasOutput = -not [string]::IsNullOrWhiteSpace($outputText) -or $outputFiles.Count -gt 0 -or $hasExecution if (-not $hasOutput) { $warnings.Add("$($entry.eval_name)/$configuration - not run yet (empty output and no output_files).") continue @@ -1727,7 +2087,12 @@ function Invoke-CollectMode { # A transferred or partial result may arrive without grading. Fall back to the assertion count from the # package so the row still shows how much is left to check. $grading = @(Get-JsonProperty -Object $result -Name 'grading' -Default @()) - $graded = @($grading | Where-Object { $null -ne (Get-JsonProperty -Object $_ -Name 'passed') }) + if ($executionStatus -eq 'incompatible') { + $warnings.Add("$($entry.eval_name)/$configuration - execution is incompatible; its output is diagnostic only and cannot contribute grading evidence.") + $graded = @() + } else { + $graded = @($grading | Where-Object { $null -ne (Get-JsonProperty -Object $_ -Name 'passed') }) + } $passed = @($graded | Where-Object { [bool]$_.passed }).Count $total = if ($grading.Count -gt 0) { $grading.Count } else { @($metadata.assertions).Count } if ($graded.Count -eq 0) { @@ -1748,13 +2113,13 @@ function Invoke-CollectMode { $observed[$configuration] = [pscustomobject]@{ Model = $model - Provider = [string](Get-JsonProperty -Object $result -Name 'provider' -Default '') Graded = $graded.Count Passed = $passed Total = $total TranscriptRecorded = -not [string]::IsNullOrWhiteSpace($transcriptText) ProcessEvidence = $hasProcessEvidence IsolationReport = $isolationReport + ExecutionStatus = $executionStatus DurationSeconds = Get-JsonProperty -Object $result -Name 'duration_seconds' TotalTokens = Get-JsonProperty -Object $result -Name 'total_tokens' ToolCalls = Get-JsonProperty -Object $result -Name 'tool_calls' @@ -1780,10 +2145,25 @@ function Invoke-CollectMode { }) } + $completion = Test-ManifestResults ` + -IterationDirectory $iterationDirectory ` + -Manifest $manifest ` + -Records $manifestRecords ` + -RequireComplete + if (-not $completion.Complete) { + foreach ($completionError in @($completion.Errors)) { + $errors.Add("Completion gate: $completionError") + } + } + $builder = [System.Text.StringBuilder]::new() - [void]$builder.AppendLine("# Eval comparison: $($manifest.skill_name) (iteration $($manifest.iteration))") + $comparisonKind = if ($errors.Count -gt 0) { 'Diagnostic comparison (incomplete)' } else { 'Eval comparison' } + [void]$builder.AppendLine("# $comparisonKind`: $($manifest.skill_name) (iteration $($manifest.iteration))") [void]$builder.AppendLine() [void]$builder.AppendLine('This repository-side comparison validates recorded grading and never invokes a model. Grading may have been performed by the user-directed external evaluator before this report was generated.') + if ($errors.Count -gt 0) { + [void]$builder.AppendLine('This package did not pass the completion gate. The rows below are diagnostic evidence only and must not be treated as a valid paired benchmark.') + } [void]$builder.AppendLine() [void]$builder.AppendLine('| Eval | Assertions | with_skill model | with_skill graded | without_skill model | without_skill graded |') [void]$builder.AppendLine('| --- | --- | --- | --- | --- | --- |') @@ -1815,7 +2195,7 @@ function Invoke-CollectMode { [void]$builder.AppendLine() [void]$builder.AppendLine('## Isolation reported') [void]$builder.AppendLine() - [void]$builder.AppendLine('Flags each run''s harness confirmed: fresh context, isolated home, isolated cwd, filesystem sandbox, candidate skill exposure, transcript capture (Y/N, ? unknown). Process-dependent assertions are only gradeable from a run with process evidence.') + [void]$builder.AppendLine('Flags each run''s harness confirmed: fresh context, isolated home, isolated cwd, filesystem confinement (strict when a hard sandbox is proven, otherwise pragmatic), candidate skill exposure, transcript capture (Y/N, ? unknown). Process-dependent assertions are only gradeable from a run with process evidence.') [void]$builder.AppendLine() [void]$builder.AppendLine('| Eval | Configuration | Isolation | Process evidence |') [void]$builder.AppendLine('| --- | --- | --- | --- |') @@ -1844,14 +2224,18 @@ function Invoke-CollectMode { Write-Host '' Write-Host "Wrote $comparisonPath" - $reportScript = Join-Path (Join-Path (Get-RepoRoot) 'scripts') 'generate-eval-report.ps1' - $reportOutput = & pwsh -NoProfile -File $reportScript -IterationDirectory $iterationDirectory 2>&1 - if ($LASTEXITCODE -ne 0) { - $errors.Add("Report generation failed: $($reportOutput -join [Environment]::NewLine)") - } else { - foreach ($line in @($reportOutput)) { - Write-Host $line + if ($errors.Count -eq 0) { + $reportScript = Join-Path (Join-Path (Get-RepoRoot) 'scripts') 'generate-eval-report.ps1' + $reportOutput = & pwsh -NoProfile -File $reportScript -IterationDirectory $iterationDirectory 2>&1 + if ($LASTEXITCODE -ne 0) { + $errors.Add("Report generation failed: $($reportOutput -join [Environment]::NewLine)") + } else { + foreach ($line in @($reportOutput)) { + Write-Host $line + } } + } else { + Write-Host 'Skipping report generation because the completion gate failed; comparison.md is diagnostic only.' } if ($errors.Count -gt 0) { diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index f66755c..8edf85b 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1191,6 +1191,7 @@ Add-ValidationResult -Results $results -Name 'Repository automation cannot launc $automationPaths = @($automationPaths | Where-Object { $normalized = $_ -replace '\\', '/' ($normalized.StartsWith('scripts/') -or $normalized.StartsWith('.github/')) -and + -not $normalized.StartsWith('scripts/eval-runners/') -and $normalized -ne 'scripts/validate-skill-templates.ps1' -and $automationExtensions -contains [System.IO.Path]::GetExtension($normalized).ToLowerInvariant() } | Sort-Object -Unique) @@ -1231,10 +1232,131 @@ Add-ValidationResult -Results $results -Name 'Repository automation cannot launc Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'this repository does not provide an opt-in path around that rule.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Model-backed comparisons are not a repository completion gate.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'This rule is Priority 1.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner' Assert-Contains -Name 'README.md' -Content $readme -Needle 'There is no repository opt-in switch.' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'Eval Runner' Assert-Contains -Name 'README.md' -Content $readme -Needle 'validate-skill-templates.ps1 -MetadataOnly' } +Add-ValidationResult -Results $results -Name 'Eval Runner protocol conformance remains deterministic' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { + return + } + $conformancePath = Join-Path $repoRoot 'scripts/eval-runners/tests/test-runner-conformance.ps1' + if (-not (Test-Path -LiteralPath $conformancePath -PathType Leaf)) { + throw 'The Eval Runner conformance suite is missing.' + } + $conformanceOutput = & pwsh -NoProfile -File $conformancePath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Eval Runner conformance failed: $($conformanceOutput -join [Environment]::NewLine)" + } + if (@($conformanceOutput -join [Environment]::NewLine) -notmatch 'Eval Runner conformance:\s+PASS') { + throw 'Eval Runner conformance did not report PASS.' + } +} + +Add-ValidationResult -Results $results -Name 'Runner-owned orchestration remains deterministic' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { + return + } + $orchestrationPath = Join-Path $repoRoot 'scripts/eval-runners/tests/test-orchestration.ps1' + if (-not (Test-Path -LiteralPath $orchestrationPath -PathType Leaf)) { + throw 'The native-worker orchestration suite is missing.' + } + $orchestrationOutput = & pwsh -NoProfile -File $orchestrationPath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Native-worker orchestration failed: $($orchestrationOutput -join [Environment]::NewLine)" + } + if (@($orchestrationOutput -join [Environment]::NewLine) -notmatch 'Native worker orchestration:\s+PASS') { + throw 'Native-worker orchestration did not report PASS.' + } +} + +Add-ValidationResult -Results $results -Name 'Foreground Phase 1 lifecycle remains deterministic' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { + return + } + $lifecyclePath = Join-Path $repoRoot 'scripts/eval-runners/tests/test-phase1-controller-lifecycle.ps1' + if (-not (Test-Path -LiteralPath $lifecyclePath -PathType Leaf)) { + throw 'The foreground Phase 1 lifecycle suite is missing.' + } + $lifecycleOutput = & pwsh -NoProfile -File $lifecyclePath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Foreground Phase 1 lifecycle regressions failed: $($lifecycleOutput -join [Environment]::NewLine)" + } + if (@($lifecycleOutput -join [Environment]::NewLine) -notmatch 'Runner-owned foreground Phase 1 lifecycle:\s+PASS') { + throw 'Foreground Phase 1 lifecycle regressions did not report PASS.' + } +} + +Add-ValidationResult -Results $results -Name 'Phase 1 aggregate fail-closed regressions remain deterministic' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { + return + } + $aggregatePath = Join-Path $repoRoot 'scripts/eval-runners/tests/test-phase1-aggregate-regressions.ps1' + if (-not (Test-Path -LiteralPath $aggregatePath -PathType Leaf)) { + throw 'The Phase 1 aggregate regression suite is missing.' + } + $aggregateOutput = & pwsh -NoProfile -File $aggregatePath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Phase 1 aggregate regressions failed: $($aggregateOutput -join [Environment]::NewLine)" + } + if (@($aggregateOutput -join [Environment]::NewLine) -notmatch 'Phase 1 aggregate regressions:\s+PASS') { + throw 'Phase 1 aggregate regressions did not report PASS.' + } +} + +Add-ValidationResult -Results $results -Name 'Frozen evidence, grading isolation, and finalization remain deterministic' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { + return + } + $integrityPath = Join-Path $repoRoot 'scripts/eval-runners/tests/test-integrity-finalization.ps1' + if (-not (Test-Path -LiteralPath $integrityPath -PathType Leaf)) { + throw 'The frozen-evidence and finalization regression suite is missing.' + } + $integrityOutput = & pwsh -NoProfile -File $integrityPath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Frozen-evidence/finalization regression failed: $($integrityOutput -join [Environment]::NewLine)" + } + if (@($integrityOutput -join [Environment]::NewLine) -notmatch 'Eval package integrity and finalization:\s+PASS') { + throw 'Frozen-evidence/finalization regression did not report PASS.' + } +} + +Add-ValidationResult -Results $results -Name 'Windows UTF-8 report generation succeeds without patching upstream skill-creator' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { + return + } + $reportUtf8Path = Join-Path $repoRoot 'scripts/eval-runners/tests/test-report-utf8.ps1' + if (-not (Test-Path -LiteralPath $reportUtf8Path -PathType Leaf)) { + throw 'The Windows UTF-8 report regression is missing.' + } + $reportUtf8Output = & pwsh -NoProfile -File $reportUtf8Path 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Windows UTF-8 report regression failed: $($reportUtf8Output -join [Environment]::NewLine)" + } + if (@($reportUtf8Output -join [Environment]::NewLine) -notmatch 'Report UTF-8 regression:\s+(PASS|SKIP)') { + throw 'Windows UTF-8 report regression did not report PASS or SKIP.' + } +} + +Add-ValidationResult -Results $results -Name 'Model-free harness probes resolve a writable temp separate from eval isolation' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { + return + } + $probeEnvironmentPath = Join-Path $repoRoot 'scripts/eval-runners/tests/test-probe-environment.ps1' + if (-not (Test-Path -LiteralPath $probeEnvironmentPath -PathType Leaf)) { + throw 'The probe-environment regression is missing.' + } + $probeEnvironmentOutput = & pwsh -NoProfile -File $probeEnvironmentPath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Probe-environment regression failed: $($probeEnvironmentOutput -join [Environment]::NewLine)" + } + if (@($probeEnvironmentOutput -join [Environment]::NewLine) -notmatch 'Probe environment regression:\s+PASS') { + throw 'Probe-environment regression did not report PASS.' + } +} + Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable prompts instead of executing them' -Action { $agents = Get-FileText -RepoRoot $repoRoot -RelativePath 'AGENTS.md' -GitRef $Ref $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref @@ -1249,7 +1371,10 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'the same model, the same version, and the same configuration' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'a baseline handed the answer key is not a baseline' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'repository automation remains deterministic and never invokes a model.' - Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill ' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Resolve the execution configuration before running the package preparation script.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'scripts/Get-HarnessModels.ps1' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'OpenCode discovery is free-only' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Handing the package over' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Executing a package you were handed' @@ -1264,7 +1389,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'The user asked for eval results, not a second workflow decision.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Asking for an eval' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '`eval `, `evaluate `' - Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Run the script immediately when asked. Do not reply with a plan, a menu of options' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Resolve the execution configuration before running the package preparation script.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Eval preparation is a completion gate' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Adding or modifying any repo-managed skill triggers this workflow.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed' @@ -1273,7 +1398,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'README.md' -Content $readme -Needle 'a completion gate an agent cannot skip' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed' Assert-Contains -Name 'README.md' -Content $readme -Needle 'prepares the paired candidate and baseline inputs as a portable package and stops' - Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill ' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'Before running the script, choose a Harness + Model.' Assert-NotContains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'run-skill-benchmark.ps1' Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'Eval packages inside this repository must live under .bot/.' Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'git does not ignore it' @@ -1286,10 +1412,127 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $scriptPath = Join-Path $repoRoot 'scripts/prepare-skill-evals.ps1' + $modelDiscoveryPath = Join-Path $repoRoot 'scripts/Get-HarnessModels.ps1' $packageRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-eval-package-' + [Guid]::NewGuid().ToString('N')) $taskMarker = "`n# Task`n" try { - $prepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $packageRoot 2>&1 + $catalogPath = Join-Path $packageRoot 'fake-model-catalog.json' + New-Item -ItemType Directory -Path $packageRoot -Force | Out-Null + [System.IO.File]::WriteAllText($catalogPath, (@' +{ + "models": [ + { "id": "claude-haiku-4.5", "display_name": "Claude Haiku 4.5", "availability": "paid", "operation": "language" }, + { "id": "gpt-5.6-luna", "display_name": "GPT-5.6 Luna", "availability": "paid", "operation": "language" }, + { "id": "deepseek/deepseek-v4-flash", "display_name": "DeepSeek V4 Flash", "availability": "free", "operation": "language" }, + { "id": "paid-model", "display_name": "Paid Model", "availability": "paid", "operation": "language" }, + { "id": "unknown-model", "display_name": "Unknown Model", "availability": "unknown", "operation": "language" }, + { "id": "opencode/muse-spark-1.2-contributor-free", "display_name": "Muse Spark 1.2", "cost": { "input": 0, "output": 0, "cache": { "read": 0, "write": 0 } }, "operation": "language" } + ] +} +'@), $utf8NoBom) + + $copilotDiscovery = (& pwsh -NoProfile -File $modelDiscoveryPath -Runner 'github-copilot' -CatalogPath $catalogPath 2>&1) + if ($LASTEXITCODE -ne 0) { throw "Get-HarnessModels.ps1 failed for Copilot fixture: $($copilotDiscovery -join [Environment]::NewLine)" } + $copilotModels = ($copilotDiscovery -join [Environment]::NewLine) | ConvertFrom-Json + if (@($copilotModels.models).Count -ne 6) { throw 'Copilot discovery must return all available fixture models.' } + + $codexDiscovery = (& pwsh -NoProfile -File $modelDiscoveryPath -Runner 'codex' -CatalogPath $catalogPath 2>&1) + if ($LASTEXITCODE -ne 0) { throw "Get-HarnessModels.ps1 failed for Codex fixture: $($codexDiscovery -join [Environment]::NewLine)" } + $codexModels = ($codexDiscovery -join [Environment]::NewLine) | ConvertFrom-Json + if (@($codexModels.models).Count -ne 6) { throw 'Codex discovery must return all available fixture models.' } + + foreach ($runnerName in @('opencode')) { + $discoveryOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner $runnerName -CatalogPath $catalogPath 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Get-HarnessModels.ps1 failed for ${runnerName}: $($discoveryOutput -join [Environment]::NewLine)" } + $discovery = ($discoveryOutput -join [Environment]::NewLine) | ConvertFrom-Json + $ids = @($discovery.models | ForEach-Object { [string]$_.id }) + if ($ids -notcontains 'deepseek/deepseek-v4-flash' -or $ids -notcontains 'opencode/muse-spark-1.2-contributor-free') { + throw "$runnerName discovery must retain free fixture model selectors." + } + if ($ids -contains 'paid-model' -or $ids -contains 'unknown-model') { + throw "$runnerName discovery must not include paid or unknown-availability models." + } + } + + $paidCatalogPath = Join-Path $packageRoot 'paid-model-catalog.json' + [System.IO.File]::WriteAllText($paidCatalogPath, (@' +{ + "models": [ + { "id": "paid-model", "display_name": "Paid Model", "availability": "paid", "operation": "language" }, + { "id": "unknown-model", "display_name": "Unknown Model", "availability": "unknown", "operation": "language" } + ] +} +'@), $utf8NoBom) + $noFreeOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner 'opencode' -CatalogPath $paidCatalogPath 2>&1 + if ($LASTEXITCODE -eq 0 -or ($noFreeOutput -join ' ') -notmatch 'No free OpenCode models') { + throw 'OpenCode discovery must fail clearly when free discovery returns zero models.' + } + + $missingCatalogOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner 'codex' -CatalogPath (Join-Path $packageRoot 'missing-catalog.json') 2>&1 + if ($LASTEXITCODE -eq 0 -or ($missingCatalogOutput -join ' ') -notmatch 'does not exist') { + throw 'Discovery failures must remain local and must not invent fallback models.' + } + + $referenceOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner 'github-copilot' -CatalogPath $catalogPath -RequireModel 'claude-haiku-4.5' 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Codebelt Reference fixture should resolve: $($referenceOutput -join [Environment]::NewLine)" } + $missingReferenceOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner 'github-copilot' -CatalogPath $paidCatalogPath -RequireModel 'claude-haiku-4.5' 2>&1 + if ($LASTEXITCODE -eq 0 -or ($missingReferenceOutput -join ' ') -notmatch 'Required model') { + throw 'Codebelt Reference discovery must fail instead of silently substituting a model.' + } + + $referencePackageRoot = Join-Path $packageRoot 'reference-package' + $referencePrepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $referencePackageRoot -CodebeltReference -ModelCatalogPath $catalogPath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 -CodebeltReference failed against the fake current catalog: $($referencePrepareOutput -join [Environment]::NewLine)" + } + $referenceProfile = [System.IO.File]::ReadAllText((Join-Path $referencePackageRoot 'iteration-1\execution-profile.json'), $utf8NoBom) | ConvertFrom-Json + if ([string]$referenceProfile.runner -ne 'github-copilot' -or [string]$referenceProfile.model -ne 'claude-haiku-4.5' -or [int]$referenceProfile.concurrency -ne 16) { + throw 'Codebelt Reference preparation must write github-copilot + claude-haiku-4.5 atomically and preserve default concurrency 16.' + } + + $codexPackageRoot = Join-Path $packageRoot 'codex-package' + $codexPrepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $codexPackageRoot -Runner 'codex' -Model 'gpt-5.6-luna' 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 failed for the Codex default fixture: $($codexPrepareOutput -join [Environment]::NewLine)" + } + $codexProfile = [System.IO.File]::ReadAllText((Join-Path $codexPackageRoot 'iteration-1\execution-profile.json'), $utf8NoBom) | ConvertFrom-Json + if ([string]$codexProfile.runner -ne 'codex' -or [string]$codexProfile.model -ne 'gpt-5.6-luna' -or [string]$codexProfile.reasoning_effort -ne 'low') { + throw 'Codex preparation must preserve gpt-5.6-luna and default omitted reasoning effort to low.' + } + if ([int]$codexProfile.concurrency -ne 16) { throw 'Codex omitted -Concurrency must preserve the repository default of 16.' } + + $opencodePackageRoot = Join-Path $packageRoot 'opencode-package' + $opencodePrepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $opencodePackageRoot -Runner 'opencode' -Model 'opencode/muse-spark-1.2-contributor-free' 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 failed for the OpenCode fixture: $($opencodePrepareOutput -join [Environment]::NewLine)" + } + $opencodeProfile = [System.IO.File]::ReadAllText((Join-Path $opencodePackageRoot 'iteration-1\execution-profile.json'), $utf8NoBom) | ConvertFrom-Json + if ([string]$opencodeProfile.runner -ne 'opencode' -or [string]$opencodeProfile.model -ne 'opencode/muse-spark-1.2-contributor-free' -or [int]$opencodeProfile.concurrency -ne 2) { + throw 'OpenCode preparation must preserve the selected runner-native model and use concurrency 2 when -Concurrency is omitted.' + } + if (($opencodePrepareOutput -join [Environment]::NewLine) -notmatch 'Concurrency:\s+2 \(OpenCode safe default\)') { throw 'OpenCode preparation output must identify the safe default concurrency source.' } + + $explicitOpenCodePackageRoot = Join-Path $packageRoot 'opencode-explicit-concurrency-package' + $explicitOpenCodeOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $explicitOpenCodePackageRoot -Runner 'opencode' -Model 'opencode/muse-spark-1.2-contributor-free' -Concurrency 16 2>&1 + if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed for explicit OpenCode concurrency: $($explicitOpenCodeOutput -join [Environment]::NewLine)" } + $explicitOpenCodeProfile = [System.IO.File]::ReadAllText((Join-Path $explicitOpenCodePackageRoot 'iteration-1\execution-profile.json'), $utf8NoBom) | ConvertFrom-Json + if ([int]$explicitOpenCodeProfile.concurrency -ne 16) { throw 'Explicit OpenCode -Concurrency 16 must be honored without clamping.' } + if (($explicitOpenCodeOutput -join [Environment]::NewLine) -notmatch 'Concurrency:\s+16 \(explicit -Concurrency\)') { throw 'Explicit OpenCode preparation output must identify -Concurrency as explicit.' } + + $missingSelectionRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-eval-missing-selection-' + [Guid]::NewGuid().ToString('N')) + $missingSelectionOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $missingSelectionRoot 2>&1 + if ($LASTEXITCODE -eq 0) { + throw 'prepare-skill-evals.ps1 must refuse to generate RUN-THIS.prompt.md without a resolved runner/model selection.' + } + if (($missingSelectionOutput -join ' ') -notmatch 'requires a resolved Harness \+ Model') { + throw 'prepare-skill-evals.ps1 must explain that Harness + Model selection is required before handoff generation.' + } + if (Test-Path -LiteralPath $missingSelectionRoot) { + Remove-Item -LiteralPath $missingSelectionRoot -Recurse -Force + throw 'prepare-skill-evals.ps1 must not create an unresolved eval package.' + } + + $prepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $packageRoot -Runner 'github-copilot' -Model 'claude-haiku-4.5' 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed: $($prepareOutput -join [Environment]::NewLine)" } @@ -1302,6 +1545,12 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable if ([string]$manifest.schema -ne 'codebeltnet/agentic/eval-package/2') { throw "The package manifest must declare schema eval-package/2; got '$($manifest.schema)'." } + if ([string]$manifest.runner_tools_integrity.schema -ne 'codebeltnet/agentic/package-tree-integrity/1' -or + [string]$manifest.runner_tools_integrity.path -ne 'tools/eval-runners' -or + [string]$manifest.runner_tools_integrity.sha256 -notmatch '^[0-9a-f]{64}$' -or + [int]$manifest.runner_tools_integrity.file_count -lt 1) { + throw 'The package manifest must anchor the complete package-local Eval Runner tool tree.' + } if ([string]$manifest.report.tool -ne 'tools/generate-eval-report.ps1' -or [string]$manifest.report.template -ne 'tools/eval-report-template.html' -or [string]$manifest.report.skill_creator -ne 'tools/skill-creator' -or @@ -1323,7 +1572,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } foreach ($isolationField in @('fresh_context_required', 'isolated_home_required', 'isolated_cwd_required')) { if (-not [bool]$manifest.isolation.$isolationField) { - throw "manifest.isolation.$isolationField must be true so a harness knows the run is hermetic." + throw "manifest.isolation.$isolationField must be true so a harness knows the run requires an isolated context." } } @@ -1333,22 +1582,28 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $runner = [System.IO.File]::ReadAllText($runnerPath, $utf8NoBom) foreach ($needle in @( - 'START NOW. You are the evaluator, grader, and report producer', - 'Do not execute evaluation prompts in the current agent context.', - 'create one isolated fresh-context worker for `with_skill` and a second isolated fresh-context worker for `without_skill`', - 'Never reuse a worker or session between runs', - 'Do not expose this runner', - 'The candidate skill is already inlined in the with_skill run', - 'Launch each worker from its own run directory', - 'Use the same model, version, configuration, tools, and limits for every worker.', - 'Record the worker''s complete response, transcript when available, token usage, elapsed time, and tool-call count.', - 'Do not begin grading until every available worker has completed or failed', - '## Grade and report immediately', - 'grading[].text', - 'tools/skill-creator/agents/grader.md', - 'scripts/aggregate_benchmark.py', - 'eval-viewer/generate_review.py', - 'report.html' + 'START NOW. You are the external Eval Orchestrator', + 'Do not execute an eval prompt in your own context.', + 'execution-freeze.json', + 'invoke-runner-owned-arms.ps1', + 'package-computed Phase 1 allowance', + 'must be started exactly once', + 'If execution is interrupted and no valid `execution-freeze.json` exists', + 'Do not create outer workers', + 'edit raw result/evidence files', + 'Only after Phase 1 returns a successful terminal JSON summary', + 'grading.json', + 'codebeltnet/agentic/eval-grading/1', + 'assertion_index', + 'passed', + 'evidence', + 'apply-eval-grading.ps1', + 'finalize-eval-package.ps1', + 'machine-readable JSON summary', + 'report.html', + 'skill-creator-report.html', + 'benchmark.json', + 'benchmark.md' )) { if (-not $runner.Contains($needle)) { throw "RUN-THIS.prompt.md must state '$needle'." @@ -1357,12 +1612,93 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable foreach ($forbidden in @( 'this context has read the runner instructions and can no longer produce a clean run', '## If you can only hold one context', - 'If you truly cannot, this package is not for you' + 'If you truly cannot, this package is not for you', + 'Choose evaluation configuration', + 'Codebelt Reference', + 'discover current models', + '', + 'record-native-result.ps1', + 'manufacture a native envelope', + 'recompute or re-bless', + 'hand-author preflight', + 'paste-ready JSON', + 'control-runner-owned-phase1.ps1', + '-WaitSeconds', + 'SAME controller command again', + 'Never invoke `invoke-runner-owned-arms.ps1` directly' )) { if ($runner.Contains($forbidden)) { - throw "RUN-THIS.prompt.md must not contain the refusal path '$forbidden'." + throw "RUN-THIS.prompt.md must not contain forbidden handoff text '$forbidden'." } } + + $generatedHandoffs = @( + [pscustomobject]@{ Name = 'GitHub Copilot generated handoff'; Iteration = $iterationDirectory }, + [pscustomobject]@{ Name = 'Codex generated handoff'; Iteration = (Join-Path $codexPackageRoot 'iteration-1') }, + [pscustomobject]@{ Name = 'OpenCode generated handoff'; Iteration = (Join-Path $opencodePackageRoot 'iteration-1') } + ) + foreach ($generatedHandoff in $generatedHandoffs) { + $generatedPromptPath = Join-Path $generatedHandoff.Iteration 'RUN-THIS.prompt.md' + if (-not (Test-Path -LiteralPath $generatedPromptPath -PathType Leaf)) { + throw "$($generatedHandoff.Name) must be prepared through the real handoff generator." + } + $generatedPrompt = [System.IO.File]::ReadAllText($generatedPromptPath, $utf8NoBom) + foreach ($needle in @( + 'invoke-runner-owned-arms.ps1', + 'delegation.dispatch_owner', + 'package-computed Phase 1 allowance', + 'must be started exactly once', + 'bridge-manifest-results.ps1', + 'Only if that bridge succeeds', + 'Do not create outer workers', + 'execution-freeze.json', + 'grading.json', + 'apply-eval-grading.ps1', + 'finalize-eval-package.ps1', + 'evaluation is incomplete and must fail closed', + 'Only persisted runner-produced evidence at the manifest-declared paths may proceed' + )) { + Assert-Contains -Name $generatedHandoff.Name -Content $generatedPrompt -Needle $needle + } + foreach ($forbidden in @( + 'bring back the result objects', + 'paste-ready JSON array', + 'If the harness cannot write to the package machine', + 'state it in chat', + '-CollectResults', + 'control-runner-owned-phase1.ps1', + '-WaitSeconds', + 'SAME controller command again' + )) { + Assert-NotContains -Name $generatedHandoff.Name -Content $generatedPrompt -Needle $forbidden + } + + $generatedReadmePath = Join-Path $generatedHandoff.Iteration 'README.md' + $generatedReadme = [System.IO.File]::ReadAllText($generatedReadmePath, $utf8NoBom) + Assert-Contains -Name "$($generatedHandoff.Name) package README" -Content $generatedReadme -Needle 'evaluation is incomplete and must fail closed' + Assert-Contains -Name "$($generatedHandoff.Name) package README" -Content $generatedReadme -Needle 'timeout_seconds=' + Assert-Contains -Name "$($generatedHandoff.Name) package README" -Content $generatedReadme -Needle 'concurrency_source=' + Assert-NotContains -Name "$($generatedHandoff.Name) package README" -Content $generatedReadme -Needle 'paste-ready JSON array' + Assert-NotContains -Name "$($generatedHandoff.Name) package README" -Content $generatedReadme -Needle '-CollectResults' + } + + $opencodePrompt = [System.IO.File]::ReadAllText((Join-Path $opencodePackageRoot 'iteration-1\RUN-THIS.prompt.md'), $utf8NoBom) + foreach ($forbidden in @( + 'OpenCode NATIVE TASK DISPATCH', + 'native `Task` tool', + 'full-capability built-in `General` worker', + 'sibling `Task` tool calls', + 'same assistant turn', + 'Want me to re-dispatch' + )) { + Assert-NotContains -Name 'OpenCode generated handoff' -Content $opencodePrompt -Needle $forbidden + } + + $copilotPrompt = [System.IO.File]::ReadAllText((Join-Path $iterationDirectory 'RUN-THIS.prompt.md'), $utf8NoBom) + foreach ($forbidden in @('general-purpose', 'fleet', 'native `Task` tool', 'sibling `Task` tool calls')) { + Assert-NotContains -Name 'GitHub Copilot generated handoff' -Content $copilotPrompt -Needle $forbidden + } + foreach ($entry in @($manifest.evals)) { $metadataForLeak = [System.IO.File]::ReadAllText((Join-Path (Join-Path $iterationDirectory $entry.directory) 'eval-metadata.json'), $utf8NoBom) | ConvertFrom-Json if ($runner.Contains([string]$metadataForLeak.expected_output)) { @@ -1381,11 +1717,17 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable throw "$($entry.eval_name)/$configuration manifest entry must declare '$pathProperty'." } } + if ($run.PSObject.Properties.Name -notcontains 'execution_result') { + throw "$($entry.eval_name)/$configuration manifest entry must declare 'execution_result'." + } foreach ($mustExist in @($run.prompt, $run.run_manifest, $run.working_directory, $run.home_directory)) { if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory $mustExist))) { throw "$($entry.eval_name)/$configuration manifest path '$mustExist' does not exist." } } + if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory $run.result) -PathType Leaf)) { + throw "$($entry.eval_name)/$configuration manifest result path '$($run.result)' does not exist." + } } $withRunDir = Join-Path $evalDirectory 'with_skill' @@ -1411,14 +1753,14 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } } - # 10. run.json requires fresh context and isolation; 6/7. it references nothing outside the run package. + # 10. run.json requires fresh context, the staged workspace boundary, and isolated home; 6/7. it references nothing outside the run package. $withRunJson = [System.IO.File]::ReadAllText((Join-Path $withRunDir 'run.json'), $utf8NoBom) $withoutRunJson = [System.IO.File]::ReadAllText((Join-Path $withoutRunDir 'run.json'), $utf8NoBom) $withRun = $withRunJson | ConvertFrom-Json $withoutRun = $withoutRunJson | ConvertFrom-Json foreach ($run in @($withRun, $withoutRun)) { if (-not [bool]$run.freshContextRequired -or -not [bool]$run.filesystemIsolationRequired -or -not [bool]$run.isolatedHomeRequired) { - throw "$($entry.eval_name) run.json must require fresh context, filesystem, and home isolation." + throw "$($entry.eval_name) run.json must require fresh context, the staged workspace boundary, and isolated home." } if ([string]$run.workingDirectory -ne 'repo' -or [string]$run.homeDirectory -ne 'home') { throw "$($entry.eval_name) run.json must set workingDirectory=repo and homeDirectory=home." @@ -1487,84 +1829,191 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } foreach ($configuration in @('with_skill', 'without_skill')) { - $resultFile = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } - $stubPath = Join-Path (Join-Path $evalDirectory 'results') $resultFile + $run = $entry.runs.$configuration + $resultLabel = [string]$run.result + $stubPath = Join-Path $iterationDirectory $run.result $stub = [System.IO.File]::ReadAllText($stubPath, $utf8NoBom) | ConvertFrom-Json if ([string]$stub.configuration -ne $configuration) { - throw "$($entry.eval_name) result stub $resultFile must declare configuration '$configuration'." + throw "$($entry.eval_name) result stub $resultLabel must declare configuration '$configuration'." } if (@($stub.grading).Count -ne @($metadata.assertions).Count) { - throw "$($entry.eval_name) result stub $resultFile must carry one grading entry per assertion." + throw "$($entry.eval_name) result stub $resultLabel must carry one grading entry per assertion." } foreach ($propertyName in @('transcript', 'shell_commands', 'files_read', 'files_written', 'exit_status', 'duration_seconds', 'total_tokens', 'tool_calls', 'turns', 'base_input_tokens', 'output_tokens', 'cache_read_tokens', 'cache_write_tokens', 'cache_write_1h_tokens', 'estimated_cost_usd', 'model_effort', 'isolation')) { if ($stub.PSObject.Properties.Name -notcontains $propertyName) { - throw "$($entry.eval_name) result stub $resultFile must expose optional field '$propertyName'." + throw "$($entry.eval_name) result stub $resultLabel must expose optional field '$propertyName'." } } foreach ($isolationField in @('fresh_context', 'isolated_home', 'isolated_cwd', 'filesystem_sandbox', 'candidate_skill_exposed', 'transcript_captured')) { if ($stub.isolation.PSObject.Properties.Name -notcontains $isolationField) { - throw "$($entry.eval_name) result stub $resultFile must expose isolation flag '$isolationField'." + throw "$($entry.eval_name) result stub $resultLabel must expose isolation flag '$isolationField'." } } } } + if ([string]$manifest.execution -ne 'runner_handoff' -or + [string]$manifest.execution_profile -ne 'execution-profile.json' -or + [string]$manifest.runner_protocol -ne 'codebeltnet/agentic/eval-runner-protocol/1' -or + [string]$manifest.runner_tools -ne 'tools/eval-runners' -or + [string]$manifest.execution_result_schema -ne 'codebeltnet/agentic/eval-execution-result/1') { + throw 'Runner-aware packages must declare the execution profile, runner protocol, runner tools, and execution-result schema.' + } + $profilePath = Join-Path $iterationDirectory ([string]$manifest.execution_profile) + $profile = [System.IO.File]::ReadAllText($profilePath, $utf8NoBom) | ConvertFrom-Json + foreach ($profileField in @('schema', 'runner', 'model', 'reasoning_effort', 'configuration_profile', 'tool_profile', 'timeout_seconds', 'concurrency')) { + if ($profile.PSObject.Properties.Name -notcontains $profileField) { + throw "execution-profile.json must declare '$profileField'." + } + } + if ($profile.PSObject.Properties.Name -contains 'provider') { + throw 'execution-profile.json must not declare provider.' + } + if ([string]$profile.schema -ne 'codebeltnet/agentic/eval-execution-profile/1' -or + [string]::IsNullOrWhiteSpace([string]$profile.runner) -or + [string]::IsNullOrWhiteSpace([string]$profile.model) -or + [int]$profile.timeout_seconds -lt 1 -or [int]$profile.concurrency -lt 1) { + throw 'execution-profile.json has an invalid schema or execution limit.' + } + $runnerTools = [System.Collections.Generic.List[string]]::new() + foreach ($runnerTool in @( + 'runner-common.ps1', + 'package-integrity.ps1', + 'invoke-runner-owned-arms.ps1', + 'resolve-runner.ps1', + 'orchestration.ps1', + 'execution-freeze.ps1', + 'freeze-execution-evidence.ps1', + 'apply-eval-grading.ps1', + 'finalize-eval-package.ps1', + 'bridge-execution-result.ps1', + 'bridge-manifest-results.ps1', + 'manifest-paths.ps1', + 'contracts/execution-profile.schema.json', + 'contracts/execution-result.schema.json', + 'contracts/execution-freeze.schema.json', + 'contracts/grading.schema.json', + 'contracts/interaction.schema.json' + )) { $runnerTools.Add($runnerTool) } + $runnerSourceRoot = Join-Path $repoRoot 'scripts/eval-runners' + foreach ($runnerDirectory in Get-ChildItem -LiteralPath $runnerSourceRoot -Directory -Force | Sort-Object Name) { + if (Test-Path -LiteralPath (Join-Path $runnerDirectory.FullName 'runner.ps1') -PathType Leaf) { + $runnerTools.Add("$($runnerDirectory.Name)/runner.ps1") + } + } + foreach ($runnerTool in $runnerTools) { + if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory "tools/eval-runners/$runnerTool") -PathType Leaf)) { + throw "Prepared package is missing runner tool '$runnerTool'." + } + } + . (Join-Path $iterationDirectory 'tools/eval-runners/runner-common.ps1') + . (Join-Path $iterationDirectory 'tools/eval-runners/package-integrity.ps1') + $preparedToolIntegrity = Get-PackageTreeIntegrity -Root (Join-Path $iterationDirectory 'tools/eval-runners') + if ($preparedToolIntegrity.Sha256 -ne [string]$manifest.runner_tools_integrity.sha256 -or $preparedToolIntegrity.FileCount -ne [int]$manifest.runner_tools_integrity.file_count) { + throw 'Prepared package runner-tool integrity does not match manifest.json.' + } + $collectOutput = & pwsh -NoProfile -File $scriptPath -CollectResults $iterationDirectory 2>&1 - if ($LASTEXITCODE -ne 0) { - throw "prepare-skill-evals.ps1 -CollectResults failed on an unrun package: $($collectOutput -join [Environment]::NewLine)" + if ($LASTEXITCODE -eq 0) { + throw 'prepare-skill-evals.ps1 -CollectResults must fail closed on an unrun package.' + } + if (($collectOutput -join ' ') -notmatch 'Completion gate') { + throw "prepare-skill-evals.ps1 -CollectResults must report the completion gate for an unrun package: $($collectOutput -join [Environment]::NewLine)" } if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'comparison.md'))) { throw 'prepare-skill-evals.ps1 -CollectResults must write comparison.md.' } - if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'report.html'))) { - throw 'prepare-skill-evals.ps1 -CollectResults must write report.html.' - } - if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'benchmark.json'))) { - throw 'prepare-skill-evals.ps1 -CollectResults must write benchmark.json.' - } - if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'benchmark.md'))) { - throw 'prepare-skill-evals.ps1 -CollectResults must write benchmark.md.' + foreach ($invalidCompletionArtifact in @('report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { + if (Test-Path -LiteralPath (Join-Path $iterationDirectory $invalidCompletionArtifact)) { + throw "prepare-skill-evals.ps1 -CollectResults must not write '$invalidCompletionArtifact' for an incomplete diagnostic package." + } } $firstEntry = @($manifest.evals)[0] - $firstEvalDirectory = Join-Path $iterationDirectory $firstEntry.directory - foreach ($resultFile in @('with-skill.result.json', 'without-skill.result.json')) { - $resultPath = Join-Path (Join-Path $firstEvalDirectory 'results') $resultFile - $result = [System.IO.File]::ReadAllText($resultPath, $utf8NoBom) | ConvertFrom-Json - $result.model = 'validator-model' - $result.provider = 'validator-provider' - $result.harness = 'validator-harness' - $result.executed_utc = '2026-01-01T00:00:00Z' - $result.output = 'validator output' - $result.transcript = 'validator transcript' - $result.duration_seconds = 1.25 - $result.total_tokens = 123 - $result.tool_calls = 2 - $result.turns = 4 - $result.base_input_tokens = 27 - $result.output_tokens = 123 - $result.cache_read_tokens = 456 - $result.cache_write_1h_tokens = 78 - $result.estimated_cost_usd = 0.12 - $result.model_effort = 'high' - foreach ($grade in @($result.grading)) { - $grade.passed = $true - $grade.evidence = 'validator evidence' - } - $result.isolation.fresh_context = $true - $result.isolation.isolated_home = $true - $result.isolation.isolated_cwd = $true - $result.isolation.filesystem_sandbox = $true - $result.isolation.candidate_skill_exposed = $true - $result.isolation.transcript_captured = $true - [System.IO.File]::WriteAllText($resultPath, (($result | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) + # Use the package-local runner-owned fixture for this deterministic + # package check. It exercises the same Phase 1 fan-out and freeze + # boundary as a real runner, including the scripted interaction case; + # no validator step authors raw or canonical evidence after freezing. + $fixtureRunnerDirectory = Join-Path $iterationDirectory 'tools/eval-runners/fixture' + New-Item -ItemType Directory -Path $fixtureRunnerDirectory -Force | Out-Null + Copy-Item -LiteralPath (Join-Path $repoRoot 'scripts/eval-runners/tests/fixtures/runner-owned-fixture.ps1') -Destination (Join-Path $fixtureRunnerDirectory 'runner.ps1') -Force + $profilePath = Join-Path $iterationDirectory ([string]$manifest.execution_profile) + $deterministicProfile = [System.IO.File]::ReadAllText($profilePath, $utf8NoBom) | ConvertFrom-Json + $deterministicProfile.runner = 'fixture' + $deterministicProfile.model = 'fixture-model' + [System.IO.File]::WriteAllText($profilePath, (($deterministicProfile | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) + . (Join-Path $iterationDirectory 'tools/eval-runners/runner-common.ps1') + . (Join-Path $iterationDirectory 'tools/eval-runners/package-integrity.ps1') + $deterministicToolIntegrity = Get-PackageTreeIntegrity -Root (Join-Path $iterationDirectory 'tools/eval-runners') + $manifest.runner_tools_integrity.sha256 = $deterministicToolIntegrity.Sha256 + $manifest.runner_tools_integrity.file_count = $deterministicToolIntegrity.FileCount + [System.IO.File]::WriteAllText((Join-Path $iterationDirectory 'manifest.json'), (($manifest | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) + $fixtureMetricEnvironment = [ordered]@{ + AGENTIC_RUNNER_FIXTURE_FINAL_RESPONSE = 'validator output' + AGENTIC_RUNNER_FIXTURE_DURATION_SECONDS = '1.25' + AGENTIC_RUNNER_FIXTURE_METRICS = '1' + } + $fixtureMetricEnvironmentBefore = [ordered]@{} + foreach ($environmentName in $fixtureMetricEnvironment.Keys) { + $fixtureMetricEnvironmentBefore[$environmentName] = [Environment]::GetEnvironmentVariable($environmentName) + [Environment]::SetEnvironmentVariable($environmentName, [string]$fixtureMetricEnvironment[$environmentName]) + } + try { + $fanoutPath = Join-Path $iterationDirectory 'tools/eval-runners/invoke-runner-owned-arms.ps1' + $fanoutOutput = & pwsh -NoProfile -File $fanoutPath -IterationDirectory $iterationDirectory 2>&1 + $fanoutExitCode = $LASTEXITCODE + $fanoutDocument = ([string]::Join([Environment]::NewLine, @($fanoutOutput)) | ConvertFrom-Json -Depth 100) + if ($fanoutExitCode -ne 0 -or [string]$fanoutDocument.status -ne 'completed') { + throw "The deterministic runner-owned fixture could not complete the package: $($fanoutOutput -join [Environment]::NewLine)" + } + } finally { + foreach ($environmentName in $fixtureMetricEnvironment.Keys) { + [Environment]::SetEnvironmentVariable($environmentName, $fixtureMetricEnvironmentBefore[$environmentName]) + } + } + $executionCollectOutput = & pwsh -NoProfile -File $scriptPath -CollectResults $iterationDirectory 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 -CollectResults failed while bridging the complete deterministic fixture: $($executionCollectOutput -join [Environment]::NewLine)" + } + # The only post-execution artifact authored by this validator is a + # grading-only document with exact metadata identities. Canonical + # result grading is projected by the deterministic application helper. + $gradingEntries = [System.Collections.Generic.List[object]]::new() + foreach ($entryToGrade in @($manifest.evals)) { + $metadataPath = Join-Path $iterationDirectory ([string]$entryToGrade.metadata) + $metadataForGrade = [System.IO.File]::ReadAllText($metadataPath, $utf8NoBom) | ConvertFrom-Json + foreach ($configuration in @('with_skill', 'without_skill')) { + for ($assertionIndex = 0; $assertionIndex -lt @($metadataForGrade.assertions).Count; $assertionIndex++) { + $gradingEntries.Add([ordered]@{ + eval_id = [int]$entryToGrade.eval_id + eval_name = [string]$entryToGrade.eval_name + configuration = $configuration + assertion_index = $assertionIndex + assertion = [string]$metadataForGrade.assertions[$assertionIndex] + passed = $true + evidence = 'validator evidence' + }) + } + } + } + $gradingPath = Join-Path $iterationDirectory ([string]$manifest.grading) + [System.IO.File]::WriteAllText($gradingPath, (([ordered]@{ schema = 'codebeltnet/agentic/eval-grading/1'; grading = @($gradingEntries.ToArray()) } | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) + $applyGradingPath = Join-Path $iterationDirectory 'tools/eval-runners/apply-eval-grading.ps1' + $applyOutput = & pwsh -NoProfile -File $applyGradingPath -IterationDirectory $iterationDirectory -GradingPath ([string]$manifest.grading) 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "The deterministic grading-only application failed: $($applyOutput -join [Environment]::NewLine)" + } + $finalizerPath = Join-Path $iterationDirectory 'tools/eval-runners/finalize-eval-package.ps1' + $finalizerOutput = & pwsh -NoProfile -File $finalizerPath -IterationDirectory $iterationDirectory -GradingPath ([string]$manifest.grading) 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "The deterministic eval finalizer failed: $($finalizerOutput -join [Environment]::NewLine)" } $metricsOutput = & pwsh -NoProfile -File $scriptPath -CollectResults $iterationDirectory 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 -CollectResults failed on recorded metrics: $($metricsOutput -join [Environment]::NewLine)" } $comparison = [System.IO.File]::ReadAllText((Join-Path $iterationDirectory 'comparison.md'), $utf8NoBom) - foreach ($needle in @('## Run metrics', '| 1.25 | 123 | 2 | recorded |', '## Isolation reported', 'fresh=Y home=Y cwd=Y fs=Y skill=Y tx=Y')) { + foreach ($needle in @('## Run metrics', '| 1.25 | 123 | 2 | recorded |', '## Isolation reported', 'fresh=Y home=Y cwd=Y fs=N skill=Y tx=Y', 'fresh=Y home=Y cwd=Y fs=N skill=N tx=Y')) { if (-not $comparison.Contains($needle)) { throw "comparison.md must report available run metric '$needle'." } @@ -1579,7 +2028,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $benchmark = [System.IO.File]::ReadAllText((Join-Path $iterationDirectory 'benchmark.json'), $utf8NoBom) | ConvertFrom-Json $expectedEvalCount = @($manifest.evals).Count - if ([int]$benchmark.metadata.runs_per_configuration -ne 1 -or @($benchmark.metadata.evals_run).Count -ne $expectedEvalCount -or @($benchmark.runs).Count -ne 2) { + if ([int]$benchmark.metadata.runs_per_configuration -ne 1 -or @($benchmark.metadata.evals_run).Count -ne $expectedEvalCount -or @($benchmark.runs).Count -ne ($expectedEvalCount * 2)) { throw "benchmark.json must use the upstream skill-creator schema for the recorded paired run set (runs_per_configuration=$($benchmark.metadata.runs_per_configuration), evals=$(@($benchmark.metadata.evals_run).Count), runs=$(@($benchmark.runs).Count), expected_evals=$expectedEvalCount)." } if ([int]$benchmark.run_summary.with_skill.tokens.mean -ne 123 -or [int]$benchmark.run_summary.without_skill.tokens.mean -ne 123) { @@ -1595,7 +2044,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $insideRepo = Join-Path $repoRoot 'agentic-eval-isolation-check' - $isolationOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $insideRepo 2>&1 + $isolationOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $insideRepo -Runner 'github-copilot' -Model 'claude-haiku-4.5' 2>&1 if ($LASTEXITCODE -eq 0) { throw 'prepare-skill-evals.ps1 must refuse an output root inside this repository but outside .bot/.' } @@ -1610,7 +2059,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable # .bot/ is the sanctioned in-repository home, and it only works while git ignores it. $botRoot = Join-Path (Join-Path $repoRoot '.bot') 'agentic-eval-bot-check' try { - $botOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $botRoot 2>&1 + $botOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $botRoot -Runner 'github-copilot' -Model 'claude-haiku-4.5' 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 must accept an output root under .bot/: $($botOutput -join [Environment]::NewLine)" } @@ -1887,6 +2336,7 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Invocation Routing Lock' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Interpret `Please do a git bot commit yolo` as `git bot commit` identity plus auto-approval for the full current worktree.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '`yolo` is not the commit message, and it does not request a changelog.' + Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'complete the commit workflow in the current turn after the required checks pass' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Full-Skill Read and Subject Lock' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Before running any Git command or composing a subject, read this `SKILL.md` completely from the first line through EOF.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'If a tool truncates the file, continue from the first unread line until EOF before proceeding.' @@ -1907,6 +2357,7 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Recovery Safety Rule' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Prefer non-destructive recovery first: targeted unstaging, precise re-staging, or `git stash`' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '`yolo` / `auto` skips user confirmation only.' + Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'the user''s `yolo` or `auto` is already the approval for this commit request' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'If the user did **not** say `yolo` or `auto`, and session-level auto mode is not already enabled, do **not** run any commit command yet.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Commit Language Lock' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Resolve that path from this skill''s own bundled `references/` directory or installed skill folder first.' @@ -1927,6 +2378,7 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces Assert-NotContains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Allowed Prefixes' Assert-NotContains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Emoji Selection' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Even in auto-approval mode, surface the commit buckets explicitly before committing.' + Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'The summary is status output, not a review request.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Do not pass literal `\n` escape sequences and assume the shell will rewrite them.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Prefer grammatical sentence and paragraph breaks over column-based hard wrapping.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Then always run `git log -1 --format="%an <%ae>"` and verify that the author matches the requested identity mode before reporting success.' @@ -1998,6 +2450,8 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Triggers the single-category context quality gate because more than one file is being placed in one category' Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Recognizes exactly one changed file as the explicit exception and skips the single-category context quality gate' Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Please do a git bot commit yolo.' + Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Treats yolo as explicit approval to complete the commit workflow in the same turn after required checks pass' + Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Does not ask whether to proceed, wait for another approval, or return a pending commit plan after presenting the status summary' Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Does not replace bot identity with a human-authored commit plus a Co-authored-by trailer' Assert-Contains -Name 'README.md' -Content $readme -Needle '**Single-category context gate**' Assert-Contains -Name 'README.md' -Content $readme -Needle 'Multi-file plans that initially collapse to one category also require a visible full-context quality gate' @@ -2111,6 +2565,11 @@ Add-ValidationResult -Results $results -Name 'Git keep a changelog skill updates Assert-Contains -Name 'git-keep-a-changelog/scripts/resolve-release-entity.ps1' -Content $entityResolver -Needle "'Added'" Assert-Contains -Name 'git-keep-a-changelog/scripts/resolve-release-entity.ps1' -Content $entityResolver -Needle "'Unchanged'" Assert-Contains -Name 'git-keep-a-changelog/scripts/test-resolve-release-entity.ps1' -Content $entityResolverTests -Needle "Assert-Classification -EntityPath 'skills/dotnet-test' -Expected 'Added'" + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle '### Layered Capability Classification' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'Do not use a top-level directory or the first framework commit as the only release entity.' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'A child adapter absent at the resolved base and present at `HEAD` is `Added`' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'Do not repeat an adapter in a parent `Added` bullet and again in `Changed`' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'A planned, blocked, or unsupported CLI/TUI/harness is not support' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $evals -Needle 'Updates CHANGELOG.md directly instead of only drafting notes in chat' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $evals -Needle 'Reads full commit subjects and bodies before writing the release entry' @@ -2155,6 +2614,8 @@ Add-ValidationResult -Results $results -Name 'Git summary skills reduce ranges t Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not add a Security or other section entry when the final diff contradicts the commit message claim' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not create a Changed section or Changed bullet for dotnet-test refinements made before its first release' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not preserve the earlier draft bullet as a frozen baseline that forces later refinements into `Changed`' + Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not treat the top-level scripts/eval-runners directory as the only release entity when independently selectable adapters have distinct final states' + Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not claim the planned or blocked Freebuff TUI is a supported runner or a completed Added capability' Assert-Contains -Name 'git-nuget-release-notes/SKILL.md' -Content $nugetSkill -Needle 'History is evidence; the resulting state is truth.' Assert-Contains -Name 'git-nuget-release-notes/SKILL.md' -Content $nugetSkill -Needle 'Classify each user-facing package capability from whether it existed at the resolved base' diff --git a/skills/agent-smith/SKILL.md b/skills/agent-smith/SKILL.md index 8a22b7a..ed86fb2 100644 --- a/skills/agent-smith/SKILL.md +++ b/skills/agent-smith/SKILL.md @@ -1,7 +1,7 @@ --- name: agent-smith -description: > - Use this skill to apply a rigorous, evidence-driven software-craftsmanship standard across an engineering task. Invoke explicitly as `/agent-smith task`, or automatically for architecture, implementation, refactoring, review, API compatibility and Semantic Versioning, testing, benchmarking, performance, skill authoring, documentation, security and DevSecOps, CI/CD, delivery, governance, and engineering assessment. For .NET, also use for IDE or CA diagnostic remediation, EditorConfig cleanup, code-style compliance, informational diagnostics, and `dotnet format` conformance. Perform the requested work, respect repository scope and conventions, validate before completion, and report evidence and risk honestly. Technology-neutral core; specialist guidance loads on demand. Do NOT use for ordinary prose, casual conversation, translation, image generation, or unrelated factual questions. +description: > + Use when the user wants evidence-driven architecture, implementation, refactoring, review, API compatibility/SemVer, testing, performance, skill authoring, documentation, security/DevSecOps, CI/CD, delivery, governance, or .NET IDE/CA and EditorConfig remediation. --- # Agent Smith @@ -14,21 +14,21 @@ The name is a deliberate, understated nod to a relentless *agent* combined with Apply **one coherent engineering standard** across design, implementation, validation, documentation, delivery, and governance. When this skill is active you do not merely advise — you **perform the requested task** to that standard, then validate it and report honestly. -The standard is technology-neutral. Specialist guidance (including .NET, Git, GitHub, CI/CD, REST, and software-supply-chain security) is loaded only when the task calls for it, and is never imposed on work where it does not apply. - -## Critical skill-authoring lock - -When creating, modifying, reviewing, or evaluating a skill, read `references/skill-authoring.md` before editing. Inspect the actual skill, applicable repository instructions, real execution traces, repeated work, and failure evidence before recommending changes; if unavailable, make that inspection the first required step. Always analyze the task graph for safe parallelism and concurrency. State how independent retrieval, execution, validation, and grading can use bounded fan-out; encode it in the skill when useful. Keep dependencies, shared mutations, rate-limited calls, and fragile ordered workflows sequential. - -For non-trivial reusable scripts and deterministic validators bundled with a skill, choose C# and .NET by default in this .NET-first skill collection. Cross-repository portability alone is not a reason to retain or introduce Python, Bash, or PowerShell. Use another language only for a concrete repository/host constraint, vendor SDK, or materially simpler native operation; state the evidence. Resolve the latest supported .NET LTS dynamically from Microsoft's official support policy when compatible repository SDK/target-framework pins or explicit user constraints do not decide. Do not replace a simpler native command with a C# program merely to satisfy the preference. - -## Critical .NET conformance lock - -When the task selects .NET EditorConfig conformance mode, read both `references/dotnet.md` and `references/dotnet-editorconfig-conformance.md` from the activated skill directory before the first formatter command. Do not search for those resources relative to the target repository or improvise the workflow if a required reference cannot be read. - -For informational diagnostics, informational-or-higher conformance, and any conformance task that does not explicitly set a different minimum, every discovery, investigation, retry, and final `dotnet format` command must include both `--severity info` and `--verify-no-changes`. The formatter defaults to `warn` when `--severity` is omitted, which can hide the findings that define success. Keep the resolved severity explicit and identical throughout the remediation loop. `--no-restore` changes restore behaviour only; it never replaces either required flag or proves conformance. - -If a prior mutating formatter pass has produced `Unmerged change from project` annotations, use the bundled `scripts/repair-roslyn-multiproject-artifacts.ps1` from this skill directory. The tool detects the neutral Roslyn multi-project artifact signature and reports the structural pattern independently of diagnostic ID. It currently repairs only the proven `whole-document-namespace-conversion` pattern. Run its default check mode first and use `-Apply` only when every artifact is reported as `recoverable`; an unrecognized or differing candidate prevents all writes. Pattern-specific recovery is not permission to use the formatter in mutating mode. +The standard is technology-neutral. Specialist guidance (including .NET, Git, GitHub, CI/CD, REST, and software-supply-chain security) is loaded only when the task calls for it, and is never imposed on work where it does not apply. + +## Critical skill-authoring lock + +When creating, modifying, reviewing, or evaluating a skill, read `references/skill-authoring.md` before editing. Inspect the actual skill, applicable repository instructions, real execution traces, repeated work, and failure evidence before recommending changes; if unavailable, make that inspection the first required step. Always analyze the task graph for safe parallelism and concurrency. State how independent retrieval, execution, validation, and grading can use bounded fan-out; encode it in the skill when useful. Keep dependencies, shared mutations, rate-limited calls, and fragile ordered workflows sequential. + +For non-trivial reusable scripts and deterministic validators bundled with a skill, choose C# and .NET by default in this .NET-first skill collection. Cross-repository portability alone is not a reason to retain or introduce Python, Bash, or PowerShell. Use another language only for a concrete repository/host constraint, vendor SDK, or materially simpler native operation; state the evidence. Resolve the latest supported .NET LTS dynamically from Microsoft's official support policy when compatible repository SDK/target-framework pins or explicit user constraints do not decide. Do not replace a simpler native command with a C# program merely to satisfy the preference. + +## Critical .NET conformance lock + +When the task selects .NET EditorConfig conformance mode, read both `references/dotnet.md` and `references/dotnet-editorconfig-conformance.md` from the activated skill directory before the first formatter command. Do not search for those resources relative to the target repository or improvise the workflow if a required reference cannot be read. + +For informational diagnostics, informational-or-higher conformance, and any conformance task that does not explicitly set a different minimum, every discovery, investigation, retry, and final `dotnet format` command must include both `--severity info` and `--verify-no-changes`. The formatter defaults to `warn` when `--severity` is omitted, which can hide the findings that define success. Keep the resolved severity explicit and identical throughout the remediation loop. `--no-restore` changes restore behaviour only; it never replaces either required flag or proves conformance. + +If a prior mutating formatter pass has produced `Unmerged change from project` annotations, use the bundled `scripts/repair-roslyn-multiproject-artifacts.ps1` from this skill directory. The tool detects the neutral Roslyn multi-project artifact signature and reports the structural pattern independently of diagnostic ID. It currently repairs only the proven `whole-document-namespace-conversion` pattern. Run its default check mode first and use `-Apply` only when every artifact is reported as `recoverable`; an unrecognized or differing candidate prevents all writes. Pattern-specific recovery is not permission to use the formatter in mutating mode. ## Activation and invocation @@ -74,16 +74,16 @@ A task may select **multiple** modes. Load core principles for every invocation, | Mode | Use when the task involves | Load | |------|----------------------------|------| | Architecture | system design, boundaries, distributed systems, integration, DDD, CQRS, event-driven design, deployment topology, migration | `references/architecture.md` | -| API design & compatibility | public/HTTP APIs, libraries, contracts, serialization, versioning, Semantic Versioning | `references/api-design-and-compatibility.md` | -| Implementation | coding and refactoring | `references/implementation.md` | -| .NET | .NET or C# is relevant | `references/dotnet.md` | -| .NET EditorConfig conformance | the user explicitly requests EditorConfig, code-style, formatter-supported analyzer, informational IDE, or named diagnostic remediation or verification | `references/dotnet.md` + `references/dotnet-editorconfig-conformance.md` | -| Testing | test design/review, regression, functional/integration/contract testing | `references/testing.md` | +| API design & compatibility | public/HTTP APIs, libraries, contracts, serialization, versioning, Semantic Versioning | `references/api-design-and-compatibility.md` | +| Implementation | coding and refactoring | `references/implementation.md` | +| .NET | .NET or C# is relevant | `references/dotnet.md` | +| .NET EditorConfig conformance | the user explicitly requests EditorConfig, code-style, formatter-supported analyzer, informational IDE, or named diagnostic remediation or verification | `references/dotnet.md` + `references/dotnet-editorconfig-conformance.md` | +| Testing | test design/review, regression, functional/integration/contract testing | `references/testing.md` | | Performance | benchmarking, profiling, optimization, latency, throughput, allocation, scalability | `references/performance.md` | | Security & DevSecOps | identity, authorization, secrets, dependencies, pipelines, supply chain, permissions, deployment security | `references/security-and-devsecops.md` | | Delivery & repository engineering | CI/CD, Git, branching, repo structure, releases, automation, containers, deployment | `references/delivery-and-repositories.md` | | Documentation | public API docs, README, architecture docs, guides, release notes, examples, DocFX | `references/documentation.md` | -| Skill authoring | creating, modifying, reviewing, describing, or evaluating agent skills and their bundled resources | `references/skill-authoring.md` | +| Skill authoring | creating, modifying, reviewing, describing, or evaluating agent skills and their bundled resources | `references/skill-authoring.md` | | Governance | policies, standards, compliance, metrics, enterprise repo governance, guardrails | `references/governance.md` | **Load .NET guidance only when .NET or C# is actually relevant.** For non-.NET work, apply the core principles and let local conventions govern language-specific detail. @@ -92,10 +92,10 @@ A task may select **multiple** modes. Load core principles for every invocation, - **Small implementation** (`/agent-smith add validation for an optional config property`): core principles + implementation (+ platform reference if relevant) + testing. Proportional process, no architecture document. - **Benchmark assessment**: core principles + decision framework + performance + implementation + platform reference (e.g. `dotnet.md`) + response contract; agent-handoff template only if delegation is requested. -- **Public API review**: core principles + decision framework + api-design-and-compatibility + implementation + platform reference + documentation + response contract. -- **CI/CD pipeline**: core principles + decision framework + security-and-devsecops + delivery-and-repositories + governance (when policy is involved) + response contract. -- **Skill authoring**: core principles + decision framework (for material choices) + skill authoring + implementation/testing/documentation as applicable. Make concurrency and script-runtime choices explicit. -- **Scoped .NET diagnostic remediation** (`/agent-smith fix the named IDE and CA diagnostics in src/Codebelt.Core`): core principles + implementation + .NET + .NET EditorConfig conformance + testing. Preserve the user-supplied diagnostic IDs and path through discovery, edits, final verification, build, tests, and reporting. +- **Public API review**: core principles + decision framework + api-design-and-compatibility + implementation + platform reference + documentation + response contract. +- **CI/CD pipeline**: core principles + decision framework + security-and-devsecops + delivery-and-repositories + governance (when policy is involved) + response contract. +- **Skill authoring**: core principles + decision framework (for material choices) + skill authoring + implementation/testing/documentation as applicable. Make concurrency and script-runtime choices explicit. +- **Scoped .NET diagnostic remediation** (`/agent-smith fix the named IDE and CA diagnostics in src/Codebelt.Core`): core principles + implementation + .NET + .NET EditorConfig conformance + testing. Preserve the user-supplied diagnostic IDs and path through discovery, edits, final verification, build, tests, and reporting. ## Repository precedence @@ -130,7 +130,7 @@ For each material finding: Issue → Why it matters → Evidence or reasoning ## Response behaviour -**Be concise. Sacrifice grammar for the sake of concision.** Prefer clear fragments when they shorten feedback. Remain respectful and technically defensible. Challenge weak assumptions; preserve good existing decisions; prioritize material issues; avoid empty praise; distinguish recommendation from requirement; avoid exaggerated certainty; explain non-obvious trade-offs. Never omit required evidence, validation limits, blockers, compatibility impact, or material risk. +**Be concise. Sacrifice grammar for the sake of concision.** Prefer clear fragments when they shorten feedback. Remain respectful and technically defensible. Challenge weak assumptions; preserve good existing decisions; prioritize material issues; avoid empty praise; distinguish recommendation from requirement; avoid exaggerated certainty; explain non-obvious trade-offs. Never omit required evidence, validation limits, blockers, compatibility impact, or material risk. For substantial assessments, use the structure in `references/response-contract.md` (Assessment → Findings → Recommendation → Trade-offs → Validation → Actionable handoff). Do not force that structure onto every response. When producing a formal assessment or a delegation prompt, use `references/engineering-assessment-template.md` or `references/agent-handoff-template.md`. @@ -162,7 +162,7 @@ This skill must not: - turn every small task into an architecture exercise; - produce advice without completing the requested work when implementation is possible; - fabricate evidence or claim unperformed validation; -- sacrifice correctness, required evidence, or material context for terseness; +- sacrifice correctness, required evidence, or material context for terseness; - broaden the task without justification; - introduce dependencies or abstractions without demonstrating value. @@ -174,15 +174,15 @@ Load on demand, per the routing table: - `references/decision-framework.md` — structured reasoning for material decisions. - `references/architecture.md` — system design and boundaries. - `references/api-design-and-compatibility.md` — public and HTTP API contracts and versioning. -- `references/implementation.md` — coding and refactoring discipline. -- `references/dotnet.md` — .NET/C#-specific guidance (load only when relevant). -- `references/dotnet-editorconfig-conformance.md` — scope-aware, read-only `dotnet format` discovery and verification for explicit .NET EditorConfig, code-style, and supported analyzer conformance work. -- `references/testing.md` — test design and review. +- `references/implementation.md` — coding and refactoring discipline. +- `references/dotnet.md` — .NET/C#-specific guidance (load only when relevant). +- `references/dotnet-editorconfig-conformance.md` — scope-aware, read-only `dotnet format` discovery and verification for explicit .NET EditorConfig, code-style, and supported analyzer conformance work. +- `references/testing.md` — test design and review. - `references/performance.md` — benchmarking, profiling, optimization. - `references/security-and-devsecops.md` — identity, secrets, dependencies, pipelines, supply chain. - `references/delivery-and-repositories.md` — CI/CD, Git, releases, repository engineering. - `references/documentation.md` — documentation as part of the product. -- `references/skill-authoring.md` — skill design, parallelism, .NET-first scripts, descriptions, and eval loops. +- `references/skill-authoring.md` — skill design, parallelism, .NET-first scripts, descriptions, and eval loops. - `references/governance.md` — policies, standards, and metrics (Intent → Drivers → Metrics → Actions). - `references/response-contract.md` — review severity, finding shape, and assessment structure. - `references/engineering-assessment-template.md` — fill-in template for a formal assessment. diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md index 8d35f41..be8bc2c 100644 --- a/skills/dotnet-benchmark/SKILL.md +++ b/skills/dotnet-benchmark/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-benchmark description: > - Discover, prioritize, and author trustworthy BenchmarkDotNet performance experiments for a .NET type while following codebelt engineering conventions and using the Codebelt.Extensions.BenchmarkDotNet Console runner. Use whenever a user wants to benchmark, micro-benchmark, performance-test, profile, optimize, compare implementations, investigate allocations or contention, or find likely bottlenecks in a .NET type or method. The skill inspects source and usage evidence, ranks high-value operations instead of every public member, selects representative workloads, rejects misleading microbenchmarks, creates or reuses the tuning/ and tooling/ harness, preflights existing-report skips, semantic-preflights workload correctness, validates discovery, and keeps full runs human-initiated. When the user says yolo, it auto-accepts routine defaults and proceeds through safe validation without confirmation churn. + Use when the user wants to design, author, review, or diagnose BenchmarkDotNet experiments for .NET code, compare implementations, measure allocations or contention, or benchmark a specific type or method. Also use to judge whether profiling or load testing is the better instrument. --- # Evidence-Driven .NET Benchmarking diff --git a/skills/dotnet-change-impact/SKILL.md b/skills/dotnet-change-impact/SKILL.md index 47e5b64..be53457 100644 --- a/skills/dotnet-change-impact/SKILL.md +++ b/skills/dotnet-change-impact/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-change-impact -description: > - Classifies .NET library or NuGet package changes and recommends the correct release bump: Major, Minor, or Patch. Applies both Semantic Versioning (MAJOR.MINOR.PATCH) and .NET assembly/file versioning (Major.Minor.Build.Revision), grounded in Microsoft’s official .NET library compatibility rules. Use when evaluating the current branch, breaking changes, API diffs, public API changes, dependency updates, TFM/platform support, interface or enum changes, overloads, analyzers, source generators, or binary/source/behavioral/design-time/backwards compatibility. When no explicit change details or compare range are provided, inspects the current Git branch and compares it against the upstream default branch automatically. Always returns structured compatibility reasoning with the recommendation. +description: > + Use when the user wants a Major, Minor, or Patch recommendation for .NET library or NuGet package changes, including API diffs, behavior changes, dependencies, target frameworks, analyzers, or source generators, based on compatibility impact. --- # .NET Change Impact diff --git a/skills/dotnet-docfx-digest/SKILL.md b/skills/dotnet-docfx-digest/SKILL.md index 9fa6aad..3ec78d7 100644 --- a/skills/dotnet-docfx-digest/SKILL.md +++ b/skills/dotnet-docfx-digest/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-docfx-digest description: > - Create and maintain developer-friendly DocFX documentation digests for .NET public APIs: repo-wide no-input audits, namespace pages, purpose-first API summaries, extension-member documentation, overwrite files, examples, availability notes, AGENTS.md maintenance, and verification. Use when the user asks to document a .NET API, update DocFX docs, create namespace pages, improve API summaries, add extension-member tables, update XML comments, add examples, maintain DocFX overwrite files, or verify documentation builds. Treat "use dotnet-docfx-digest", "complete missing documentation", and public .NET API changes as automatic triggers. + Use when the user wants to create, repair, audit, or complete DocFX docs for .NET public APIs, or has changed public API that needs namespace pages, XML comments, overwrite files, extension-member tables, examples, or build verification. Exclude private/internal APIs. --- # .NET DocFX Digest Steward diff --git a/skills/dotnet-new-app-slnx/SKILL.md b/skills/dotnet-new-app-slnx/SKILL.md index 3bffe36..625e48f 100644 --- a/skills/dotnet-new-app-slnx/SKILL.md +++ b/skills/dotnet-new-app-slnx/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-new-app-slnx -description: > - Scaffold a new .NET standalone application solution following codebelt engineering conventions. Use this skill when the user wants to create a new .NET application — Console, Web, or Worker service. Also use when the user mentions "new app", "new console app", "new web api", "new mvc app", "new razor app", "new web app", "new worker service", "scaffold app", "dotnet new web", "dotnet new webapi", "dotnet new mvc", "dotnet new webapp", "dotnet new worker", "dotnet new console", or wants a .NET application project with CI/CD pipeline, functional tests, and code quality tooling. ALWAYS use this skill when asked to scaffold or create a new .NET application solution. +description: > + Use when the user wants to scaffold a complete codebelt-style .NET application solution (`.slnx`) for Console, Web API, MVC, Razor, empty Web, or Worker hosts, with repository tooling and functional tests. Do not use for a quick throwaway project or application logic. --- # .NET Application Solution Setup (Codebelt Conventions) diff --git a/skills/dotnet-new-lib-slnx/SKILL.md b/skills/dotnet-new-lib-slnx/SKILL.md index 0f464d1..5a54b6e 100644 --- a/skills/dotnet-new-lib-slnx/SKILL.md +++ b/skills/dotnet-new-lib-slnx/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-new-lib-slnx description: > - Scaffold a new .NET NuGet library solution following codebelt engineering conventions. Use this skill when the user wants to create a new NuGet library, class library, or reusable .NET package. Also use when the user mentions "new library", "new NuGet package", "scaffold library", "class library solution", "dotnet new classlib", or wants a .NET library project with multi-target frameworks, strong-name signing, NuGet packaging, DocFX documentation, CI/CD pipeline, and code quality tooling. ALWAYS use this skill when asked to scaffold or create a new .NET library solution. + Use when the user wants to scaffold a complete codebelt-style .NET library or NuGet package solution (`.slnx`), including packaging, tests, DocFX, benchmarks, and strong-name signing. Do not use merely to add a library project to an established solution. --- # .NET Library Solution Setup (Codebelt Conventions) diff --git a/skills/dotnet-remote-testing/SKILL.md b/skills/dotnet-remote-testing/SKILL.md index 1d9a698..c9cd53b 100644 --- a/skills/dotnet-remote-testing/SKILL.md +++ b/skills/dotnet-remote-testing/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-remote-testing -description: > - Run .NET tests inside a resolved remote Docker environment — Visual Studio's Remote Testing without hand-writing container plumbing. Invoking this skill IS the request: run the tests immediately. Never reply with a menu of options or a questionnaire. Use when asked to remote test, run tests in Docker or a container, target a specific .NET SDK, list or select test environments, or honor an existing testenvironments.json. Honors configured Docker environments, or derives them from Microsoft's live .NET release index using mcr.microsoft.com/dotnet/sdk images, plus codebeltnet/ubuntu-testrunner for multi-targeted repos, via the runner scripts/remote-test.cs. Docker only; WSL and SSH are unsupported. Do NOT use to author or refactor test code, choose a testing framework, generate Dockerfiles, or run tests on the host. +description: > + Use when the user wants to run, list, or plan .NET tests in Docker remote-test environments, including `testenvironments.json` or a requested SDK/container. Do not use to write or refactor tests, create Dockerfiles, use WSL/SSH, or run tests on the host. compatibility: > Requires the .NET 10 SDK or later (`dotnet run --file`), a running Docker daemon, and PowerShell 7+. Zero-config discovery needs network access; a cache enables offline reuse. --- diff --git a/skills/dotnet-segregated-assets/SKILL.md b/skills/dotnet-segregated-assets/SKILL.md index cb2513f..feef224 100644 --- a/skills/dotnet-segregated-assets/SKILL.md +++ b/skills/dotnet-segregated-assets/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-segregated-assets description: > - Migrate or configure an ASP.NET Core web application so developers keep authoring static files in the conventional wwwroot while deployed static content is served by Codebelt Static Content Provider (codebeltnet/web-cdn-origin:2.0.0), a separate asset host rather than the web app. Use when asked to segregate static assets, move wwwroot off the web app, stop shipping wwwroot with the app, or reconcile Cuemon App/CDN TagHelpers with a segregated topology. Reuse existing Cuemon or project abstractions, distinguish App assets from shared CDN assets, preserve Static Web Assets, and verify publish/local invariants deterministically. Do NOT use to build a general-purpose CDN or migrate non-ASP.NET static sites. + Use when the user wants an ASP.NET Core app to keep authoring static files in `wwwroot` while serving deployed app assets from `codebeltnet/web-cdn-origin`, including Cuemon `app-*`/`cdn-*` migration. Do not use for general CDN design or non-ASP.NET sites. compatibility: > Requires the .NET SDK 10+ and PowerShell 7+. NuGet.org access is required when plan resolves an existing Cuemon package reference. Docker is optional (only for the local origin). CI guidance targets GitHub Actions, which is the assumed delivery surface. --- diff --git a/skills/dotnet-strong-name-signing/SKILL.md b/skills/dotnet-strong-name-signing/SKILL.md index 634d786..543d600 100644 --- a/skills/dotnet-strong-name-signing/SKILL.md +++ b/skills/dotnet-strong-name-signing/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-strong-name-signing -description: > - Generate a strong name key (.snk) file for signing .NET assemblies using pure .NET cryptography — no Visual Studio Developer PowerShell or sn.exe required. Works in any terminal. Use this skill when the user wants to create a strong name key, generate an .snk file, sign .NET assemblies, or mentions "strong-name", "snk", "AssemblyOriginatorKeyFile", "SignAssembly", or asks how to sign a .NET library. Also use when scaffolding .NET libraries or NuGet packages that need assembly signing. ALWAYS use this skill when asked to generate or create a strong name key file. +description: > + Use when the user wants to generate a `.snk` strong-name key, configure `SignAssembly` or `AssemblyOriginatorKeyFile`, or sign a .NET assembly, library, or NuGet package. Do not use for Authenticode, code-signing certificates, or secret-management tasks. --- # Strong Name Signing for .NET diff --git a/skills/dotnet-strong-name-signing/evals/evals.json b/skills/dotnet-strong-name-signing/evals/evals.json index 258261d..6853358 100644 --- a/skills/dotnet-strong-name-signing/evals/evals.json +++ b/skills/dotnet-strong-name-signing/evals/evals.json @@ -4,22 +4,56 @@ { "id": 1, "prompt": "Generate a strong name key for this repository using the defaults.", - "expected_output": "A summary-first confirmation flow that generates a .snk file with the default 1024-bit key size in the repo root.", + "expected_output": "Turn 1 computes the defaults, presents the 1024-bit repository-root summary, and waits for confirmation without generating the key. After the deterministic confirmation turn, turn 2 generates and verifies the .snk file and reports its details.", "expectations": [ "Computes defaults silently before asking for confirmation", + "Presents the default path and 1024-bit size and explicitly asks for confirmation before the protected operation", + "Does not generate the key before the confirmation turn", "Defaults to a 1024-bit RSA key", - "Uses the RSACryptoServiceProvider ExportCspBlob flow" - ] + "Uses the RSACryptoServiceProvider ExportCspBlob flow", + "Continues the same runner session for the confirmation turn", + "Generates and verifies the key only after confirmation and reports its path, size, and key details" + ], + "interaction": { + "mode": "scripted", + "turns": [ + { + "role": "user", + "source": "prompt.md" + }, + { + "role": "user", + "content": "Yes, proceed." + } + ] + } }, { "id": 2, "prompt": "Create a 4096-bit strong name key called signing-key in C:\\temp\\keys.", - "expected_output": "The workflow overrides the defaults for key size, file name, and output directory, then reports the generated file details.", + "expected_output": "Turn 1 summarizes the requested 4096-bit key, signing-key filename, and output directory, then waits for confirmation without generating it. After the deterministic confirmation turn, turn 2 performs the generation and reports the final file details.", "expectations": [ "Allows overriding individual values after the summary", + "Requests confirmation before generating the overridden key", + "Does not perform the protected generation before the confirmation turn", "Uses the requested 4096-bit key size", - "Reports output path, size, and key size after generation" - ] + "Reports output path, size, and key size after generation", + "Continues the same runner session for the confirmation turn", + "Performs the overridden generation only after confirmation" + ], + "interaction": { + "mode": "scripted", + "turns": [ + { + "role": "user", + "source": "prompt.md" + }, + { + "role": "user", + "content": "Yes, proceed." + } + ] + } }, { "id": 3, diff --git a/skills/dotnet-test/SKILL.md b/skills/dotnet-test/SKILL.md index c095854..869d159 100644 --- a/skills/dotnet-test/SKILL.md +++ b/skills/dotnet-test/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-test description: > - Move .NET xUnit test projects onto Codebelt's entrypoint-owned test hosts, replacing Microsoft's WebApplicationFactory and hand-rolled host plumbing with WebApplicationTestFactory, WebApplicationTest, ApplicationTestFactory, and ApplicationTest — for ASP.NET Core, console, and worker applications alike. Invoking this skill IS the request: inspect the repository and refactor immediately, never opening with a menu, a capability list, or a questionnaire. Use for WebApplicationFactory migration, xUnit v2-to-v3 modernization, Microsoft Testing Platform adoption, managed fixtures, reusable functional-test harnesses, in-process console or worker tests, and unit-test bootstrap. Preserve behavior, test names, and package ownership, then validate restore/build/test. Do NOT use for NUnit/MSTest-only work, production refactoring without a test-project goal, or process-launching end-to-end harnesses. + Use when the user wants to bootstrap or refactor .NET xUnit tests onto Codebelt entrypoint-owned hosts, migrate `WebApplicationFactory`, modernize xUnit v2 to v3 or Microsoft Testing Platform, or add web, console, or worker fixtures. Exclude NUnit/MSTest and process-based end-to-end harnesses. compatibility: > Requires .NET SDK, PowerShell 7+, and network access to NuGet for dynamic package resolution. --- diff --git a/skills/git-keep-a-changelog/SKILL.md b/skills/git-keep-a-changelog/SKILL.md index c151fd0..2ec032b 100644 --- a/skills/git-keep-a-changelog/SKILL.md +++ b/skills/git-keep-a-changelog/SKILL.md @@ -1,7 +1,7 @@ --- name: git-keep-a-changelog description: > - Create or update CHANGELOG.md from git history using Keep a Changelog 1.1.0 style. Use when the user explicitly asks to create or update a changelog, draft release notes, prepare or finalize a release changelog, or requests a SemVer-aware release summary. Treat `ready to release` and `rtr` as triggers only in a versioned release context. Treat `yolo` and `auto` only as autonomy modifiers after explicit changelog or release-note intent; they are never standalone triggers. Never select this skill for `git bot commit yolo`, `git commit auto`, or another commit-execution request unless the user also explicitly asks to update the changelog or release notes. Reads full commit bodies and diffs, isolates branch history, includes pending changes automatically only in scoped yolo or auto mode, and writes curated surviving base-to-HEAD outcomes for review. + Use when the user wants to create or update `CHANGELOG.md`, follow Keep a Changelog, or finalize a versioned changelog. Treat `ready to release` or `rtr` as triggers only with version context. Do not trigger for GitHub releases, NuGet package notes, commit execution, or bare `yolo`/`auto`. compatibility: > Requires Git and PowerShell 7+ for deterministic branch-scope resolution. --- @@ -80,6 +80,17 @@ Reduce first. Interpret second. Summarize last. Establish the classification baseline at the user-facing release-entity boundary, not independently for every changed file. For a repo-managed skill, the entity is the skill capability together with its dedicated files and inseparable registration, catalog, documentation, validation, and eval wiring. If that entity is absent at the base and present at `HEAD`, its introduction is `Added`; intermediate commits that refine, fix, document, or validate it cannot create `Changed` or `Fixed` outcomes for that same new entity. A change to a separately pre-existing shared capability remains its own outcome and is classified from its own base state. +### Layered Capability Classification + +Do not use a top-level directory or the first framework commit as the only release entity. Classify at the smallest independently selectable user-facing boundary. For layered eval tooling, the protocol/framework and each selectable runner, CLI, TUI, or harness adapter can have different release states. + +- A child adapter absent at the resolved base and present at `HEAD` is `Added` even when its parent directory already existed at the base or was introduced earlier in the branch. Run the entity resolver for the parent and each independent child path when the diff supports that decomposition. +- Keep implementation, fixtures, conformance tests, and wiring with the capability they introduce. Do not repeat an adapter in a parent `Added` bullet and again in `Changed` as “added to the lineup.” +- Refinements to a pre-existing adapter or framework are `Changed` or `Fixed` from their final delta. A defect repaired before first release of a base-absent capability remains part of that capability's `Added` outcome. +- A planned, blocked, or unsupported CLI/TUI/harness is not support and must not be listed as an `Added` adapter. + +Use the final state, not commit verbs: newly usable execution support belongs under `Added`; changes to existing runner, orchestration, report, telemetry, or package behavior belong under `Changed`; and distinct supported repairs belong under `Fixed`. For example, a new framework followed by Cline or GitHub Copilot adapters gets `Added` outcomes for the framework and adapters, while changes to an existing Codex adapter are classified separately. + 1. Inspect cumulative manifest and version deltas across `diff_range`. 2. Inspect the cumulative base-to-`HEAD` diff. 3. Inspect any approved pending worktree changes that are part of the draft. diff --git a/skills/git-keep-a-changelog/evals/evals.json b/skills/git-keep-a-changelog/evals/evals.json index 4d080ab..c9504aa 100644 --- a/skills/git-keep-a-changelog/evals/evals.json +++ b/skills/git-keep-a-changelog/evals/evals.json @@ -251,6 +251,20 @@ "Does not reinterpret yolo as a release or changelog trigger", "Defers the request to the git-visual-commits workflow" ] + }, + { + "id": 22, + "prompt": "Create a deterministic temp git repo outside the current repository under `$env:TEMP`, then use git-keep-a-changelog there. Start from a tagged base release containing `CHANGELOG.md`, an existing `scripts/eval-runners/README.md`, an existing `scripts/eval-runners/codex/runner.ps1`, an existing report generator, and an existing conformance test. On branch `v1.1.0/eval-runner-support`, add the shared execution contracts and package bridge, add `scripts/eval-runners/cline/runner.ps1` and `scripts/eval-runners/github-copilot/runner.ps1` with their fixtures and documentation, refine the existing report telemetry and Codex command behavior, add a `freebuff-readiness.md` file that says its TUI transport is planned and blocked, and repair a distinct existing Codex evidence-directory defect. Include commits from multiple contributors. Update `CHANGELOG.md` and stop after the edit.", + "expected_output": "The changelog separates the new execution framework and independently selectable Cline and GitHub Copilot adapters as Added outcomes, classifies changes to the pre-existing report/Codex behavior separately, records the supported Codex repair when justified, does not repeat the new adapters under Changed, and does not claim the planned Freebuff TUI is supported.", + "expectations": [ + "Does not treat the top-level scripts/eval-runners directory as the only release entity when independently selectable adapters have distinct final states", + "Runs or applies path-backed base-versus-HEAD classification to the shared framework and the Cline and GitHub Copilot adapter boundaries", + "Places first-time Cline and GitHub Copilot support under Added even when the parent eval-runner framework already existed at the base or earlier in the branch", + "Classifies refinements to the pre-existing report generator or Codex adapter from their surviving base-to-HEAD behavior rather than calling them new support", + "Does not repeat a newly supported adapter in both Added and Changed or describe it merely as added to an existing lineup under Changed", + "Does not claim the planned or blocked Freebuff TUI is a supported runner or a completed Added capability", + "Keeps the changelog grounded in the final state and includes contributions from every selected author" + ] } ] } diff --git a/skills/git-nuget-readme/SKILL.md b/skills/git-nuget-readme/SKILL.md index 79c187c..94a6b4f 100644 --- a/skills/git-nuget-readme/SKILL.md +++ b/skills/git-nuget-readme/SKILL.md @@ -1,7 +1,7 @@ --- name: git-nuget-readme description: > - Create or update a NuGet package README.md from git history and real .NET project metadata for repositories that ship a package from `src/`. Use this skill whenever the user asks to write a package README, refresh NuGet-facing docs, improve the repo README for a library, summarize the current branch into README copy, or make a package more compelling to adopt on NuGet. Treat requests like "update the README for this package", "write a NuGet README from git", "refresh the library README", "make this NuGet package easier to pick", or "generate a devex-friendly README for this assembly" as automatic triggers. The skill discovers the advertised packable project, grounds the README in real package and source metadata, preserves honest claims, and writes forthcoming, adoption-friendly copy instead of generic marketing fluff. + Use when the user wants to create or refresh a NuGet-facing `README.md` for a .NET package, grounded in the packable `src/` project and current repository changes. Do not use for general repository docs, DocFX API pages, changelogs, or release notes. --- # Git NuGet README diff --git a/skills/git-nuget-release-notes/SKILL.md b/skills/git-nuget-release-notes/SKILL.md index b91002e..2b49771 100644 --- a/skills/git-nuget-release-notes/SKILL.md +++ b/skills/git-nuget-release-notes/SKILL.md @@ -1,7 +1,7 @@ --- name: git-nuget-release-notes description: > - Create or update per-package NuGet release notes from git history for .NET repositories that store cumulative `.nuget/{ProjectName}/PackageReleaseNotes.txt` files. Use this skill whenever the user asks for NuGet release notes, `PackageReleaseNotes.txt`, per-assembly or per-package release notes, or wants git commits turned into package release notes instead of a repo-wide changelog. Treat requests like "update PackageReleaseNotes.txt", "write NuGet release notes from git", "summarize this release per assembly", or "create missing package release notes under .nuget" as automatic triggers. The skill discovers packable `src/` projects, resolves concrete release version and availability per package, creates missing files when needed, preserves cumulative newest-first history, reduces each package to its surviving base-to-HEAD delta before using history as context, and avoids raw commit-log dumps or unsupported claims. + Use when the user wants per-package NuGet release notes in cumulative `.nuget/{ProjectName}/PackageReleaseNotes.txt` files, including creating missing files for packable `src/` projects. Do not use for repository `CHANGELOG.md`, GitHub releases, or package README work. --- # Git NuGet Release Notes diff --git a/skills/git-remote-release/SKILL.md b/skills/git-remote-release/SKILL.md index 1610c65..7409e03 100644 --- a/skills/git-remote-release/SKILL.md +++ b/skills/git-remote-release/SKILL.md @@ -1,7 +1,7 @@ --- name: git-remote-release description: > - Generate GitHub release notes by summarizing all commits and pull requests between two Git tags, branches, or the current branch and the upstream default branch. Use when the user asks to write release notes, generate release notes, draft a GitHub release, create release notes from tags, summarize changes between versions, summarize the current branch, or provides a GitHub compare URL. Trigger phrases: "release notes", "generate release notes", "what changed between", "summarize changes from v1 to v2", "GitHub release", "summarize this branch", compare URLs like "github.com/owner/repo/compare/v1...v2". When no explicit input is given, detects the current branch and compares against the upstream default branch automatically. + Use when the user wants GitHub release notes or a human-readable change summary from commits and pull requests between tags, branches, or a GitHub compare URL. Do not use for `CHANGELOG.md`, NuGet `PackageReleaseNotes.txt`, commit messages, or squash summaries. --- # Git Remote Release diff --git a/skills/git-repo-digest/SKILL.md b/skills/git-repo-digest/SKILL.md index f1e0dc0..0d41847 100644 --- a/skills/git-repo-digest/SKILL.md +++ b/skills/git-repo-digest/SKILL.md @@ -1,7 +1,7 @@ --- name: git-repo-digest description: > - Generate source-grounded repository digest markdown from deterministic local evidence bundles. Use when the user asks to create, refresh, or complete repo/package digests, family or project overview pages, .bot/digests output, digest workspace workflows, or result/Index.md plus result/{PackageName}.md files for any repository URL. The skill runs its bundled .NET file-based evidence generator over a git clone, separates authoritative XML evidence from Markdown prompts and reading aids, writes package digests first, then writes the overview from completed package digests, and enforces complete-read grounding and no-invention rules even when file output is capped. + Use when the user wants source-grounded repository or package digest Markdown from a repository URL or existing `.bot/digests` workspace, especially `result/Index.md` and per-package pages. Do not use for README, DocFX, changelog, release-note, or one-off prose summaries. --- # Git Repo Digest diff --git a/skills/git-visual-commits/SKILL.md b/skills/git-visual-commits/SKILL.md index d5664f1..0700436 100644 --- a/skills/git-visual-commits/SKILL.md +++ b/skills/git-visual-commits/SKILL.md @@ -1,7 +1,7 @@ --- name: git-visual-commits description: > - Execute the structured git commit workflow whenever the user says `git bot commit`, `git commit`, or `git our commit`; asks the agent to commit or stage changes; or asks to write or review a commit message. Treat `Please do a git bot commit yolo` and equivalent wording as an authoritative invocation of this skill: select bot identity, enable auto-approval, and never treat `yolo` as the message or route the request to changelog or release-note skills. Treat commit wording as an automatic trigger for this skill, not as a casual hint. `yolo` and `auto` are modifiers only inside an explicit commit request and never standalone triggers. Apply full-worktree semantic grouping unless narrowed, validated emoji-first lowercase subjects, conventional prefixes only on explicit request, and post-commit identity and body verification. + Use when the user asks to stage or commit changes, write or review a commit message, or says `git bot commit`, `git commit`, or `git our commit`. Treat commit wording as an automatic trigger for this skill, not as a casual hint. `yolo` and `auto` only modify an explicit commit request. --- # Git Visual Commits @@ -16,6 +16,7 @@ This skill drives the entire git commit workflow — reviewing changes, grouping - An explicit `git bot commit`, `git commit`, or `git our commit` phrase is an authoritative request to use this skill. Do not substitute a changelog, release-note, squash-summary, or generic commit workflow. - Interpret `Please do a git bot commit yolo` as `git bot commit` identity plus auto-approval for the full current worktree. `yolo` is not the commit message, and it does not request a changelog. +- That exact combination is also an instruction to complete the commit workflow in the current turn after the required checks pass. Treat the visible plan as status information, not as a request for another approval; do not end with a pending plan or ask whether to proceed. - Equivalent word order and punctuation, such as `git bot commit, yolo` or `yolo — do a git bot commit`, preserve the same routing when both the explicit commit command and modifier are present. - A competing skill may run only when the user also explicitly requests its distinct output, such as updating `CHANGELOG.md`, drafting release notes, or producing a squash summary. @@ -72,6 +73,7 @@ The validator enforces an emoji present in the bundled reference table, exactly - mixed-scope validation - deterministic subject validation - post-commit author verification +- In auto-approval mode, the user's `yolo` or `auto` is already the approval for this commit request. After the required checks pass, execute the commit command(s) in the same turn. Do not ask "Proceed?", "Should I commit?", or any equivalent confirmation question, and do not return a plan as if approval were still pending. Stop only for a concrete blocker such as a missing alias or failed validation, and report that blocker directly. If the user did **not** say `yolo` or `auto`, and session-level auto mode is not already enabled, do **not** run any commit command yet. You must stop after Step 4, present the plan, and wait for approval. @@ -403,7 +405,9 @@ If auto-approval is **not** active, Step 4 is a hard stop. Do not stage, do not Auto-committing: 🔧 build config → 🚚 rename auth to identity → ✅ identity tests → 💬 update changelog ``` -Even in auto-approval mode, surface the commit buckets explicitly before committing. Auto-approval removes the wait, not the planning step. +Even in auto-approval mode, surface the commit buckets explicitly before committing. Auto-approval removes the wait, not the planning step. + +The summary is status output, not a review request. Step 5 is mandatory in the same turn once its preconditions pass: never ask "Proceed with committing these groups?" (or an equivalent question), wait for a reply, or finish with a pending commit plan. If the user did not narrow scope, the plan you surface must account for the full worktree rather than an arbitrarily chosen subset. diff --git a/skills/git-visual-commits/evals/evals.json b/skills/git-visual-commits/evals/evals.json index 73d8b1e..6ab5043 100644 --- a/skills/git-visual-commits/evals/evals.json +++ b/skills/git-visual-commits/evals/evals.json @@ -273,11 +273,13 @@ { "id": 25, "prompt": "Please do a git bot commit yolo.", - "expected_output": "An unambiguous git-visual-commits invocation that uses bot identity and auto-approval for the full worktree without treating yolo as a message or routing to changelog work.", + "expected_output": "An unambiguous git-visual-commits invocation that uses bot identity and auto-approval for the full worktree, performs the required checks, and completes the commit workflow in the same turn without asking for another approval or treating yolo as a message or routing to changelog work.", "expectations": [ "Routes the request to git-visual-commits rather than git-keep-a-changelog, release notes, or a generic commit workflow", "Interprets git bot commit as an identity lock and executes the git bot commit command rather than regular git commit", "Interprets yolo as auto-approval for the commit workflow rather than as the commit subject or message", + "Treats yolo as explicit approval to complete the commit workflow in the same turn after required checks pass", + "Does not ask whether to proceed, wait for another approval, or return a pending commit plan after presenting the status summary", "Treats the full current worktree as scope because the user did not narrow it", "Does not replace bot identity with a human-authored commit plus a Co-authored-by trailer" ] diff --git a/skills/git-visual-squash-summary/SKILL.md b/skills/git-visual-squash-summary/SKILL.md index 71c5ddf..3e6fe6d 100644 --- a/skills/git-visual-squash-summary/SKILL.md +++ b/skills/git-visual-squash-summary/SKILL.md @@ -1,7 +1,7 @@ --- name: git-visual-squash-summary description: > - Turn many commits into a curated grouped squash summary for squash-and-merge contexts. Use when the user asks to squash a branch, summarize PR commits, or clean up history. Defaults to full feature branch against base (not tracking remote), includes all authors unless narrowed, and acts immediately—the skill is read-only with no permission-seeking. Returns grouped lines only, resolving the cumulative diff to drop reverted churn, preserving identifiers and overlap, and avoiding changelog wording. A bare invocation is a complete request: run git commands immediately and return summary lines, never an instruction recap or permission question. + Use when the user wants a read-only, ready-to-paste grouped squash-and-merge or PR branch summary from a commit range or the current feature branch. Do not use to mutate history, write changelog or release notes, execute commits, or summarize a single commit message. --- # Git Visual Squash Summary diff --git a/skills/markdown-illustrator/SKILL.md b/skills/markdown-illustrator/SKILL.md index 22329df..f0af52c 100644 --- a/skills/markdown-illustrator/SKILL.md +++ b/skills/markdown-illustrator/SKILL.md @@ -1,7 +1,7 @@ --- name: markdown-illustrator description: > - Turn a markdown document into a visualization-first chat response consisting of one Visual Brief and one high-quality diffuser prompt generated with best-effort reasoning. Use when the user references a .md file and wants a hero image, cover image, visual digest, keynote opener, illustration, or diffuser prompt, especially for requests like "turn roadmap.md into a keynote opener image" or "create a visual digest for onboarding-notes.md". Default to zero follow-up questions, no file creation, and no style/theme/model menus; infer a compact visual strategy from the request and document, and only honor extra specificity when the user explicitly asks for a named model, aesthetic, or visual treatment such as whiteboard or blackboard. + Use when the user references a Markdown file and wants one hero, cover, keynote-opener, visual-digest, or illustration prompt derived from it, including whiteboard, blackboard, or named image-model styling. Do not trigger for image generation itself or general Markdown summarization. --- # Markdown Illustrator diff --git a/skills/skill-creator-agnostic/SKILL.md b/skills/skill-creator-agnostic/SKILL.md index e8ac43a..4f68913 100644 --- a/skills/skill-creator-agnostic/SKILL.md +++ b/skills/skill-creator-agnostic/SKILL.md @@ -1,7 +1,8 @@ --- name: skill-creator-agnostic description: > - DEPRECATED — no longer maintained and scheduled for removal in 1.0.0. Retained only for backward compatibility. Do not use for new skill creation, modification, or benchmarking. Use Anthropic's `skill-creator` directly and apply this repository's skill-authoring rules from `AGENTS.md` instead. + DEPRECATED — no longer maintained and scheduled for removal in 1.0.0. Use only when the user explicitly invokes `skill-creator-agnostic` or asks about its status. Redirect skill creation, modification, evaluation, and benchmarking to Anthropic's `skill-creator` plus repository `AGENTS.md`. +disable-model-invocation: true --- # Skill Creator Agnostic diff --git a/skills/trunk-first-repo/SKILL.md b/skills/trunk-first-repo/SKILL.md index a1168d6..d1aabb0 100644 --- a/skills/trunk-first-repo/SKILL.md +++ b/skills/trunk-first-repo/SKILL.md @@ -1,7 +1,7 @@ --- name: trunk-first-repo -description: > - Initialize a folder as a git repository following scaled trunk-based development. Sets up an empty main branch (seed commit only), creates a versioned feature branch, pushes main before feature branches, and enforces a PR-first workflow where content only reaches main through pull requests. Use this skill when the user wants to initialize a git repo, set up a new repository, start a project with proper git workflow, safely push the first trunk-first branches later with "push remote", or mentions "trunk-based", "PR workflow", "branch protection", "git init", or wants to follow GitHub PR best practices. ALWAYS use this skill when asked to initialize or set up a git repository. +description: > + Use when the user wants to initialize a new Git repository with a scaled trunk-based, PR-first workflow, including an empty seeded `main`, a versioned feature branch, branch protection, or a guarded first remote push. Do not use for commits or history cleanup in an existing repository. --- # Trunk-First Repo