[Jaylyn Barbee (Jaylyn-Barbee)]

What and why

Defines the file every benchmark trial writes, so the dashboard and README can read results from one place. Part of #1179 (second in the stack, uses the arm names from #1180).

This draft is for the team to review the format. It adds the schema, a doc, a validator and one example, and makes both harnesses write the files (see "Where the files are written"). It also adds publish-results.ps1, which stages a release and opens a PR into benchmark-data. It does not create that branch.

Refs #1181

The schema

One JSON file per trial (one agent attempt: one scenario, one model, one arm). Release results go on the benchmark-data branch, one folder per release:

v0.7.1\winforms-msix__cli-only__sonnet-5.5__1.json
       <scenario>__<arm>__<model>__<trial>.json

Dev trials use the same format with "release": "dev" and stay local.

Field	Type	Meaning
schemaVersion	1	Format version
release	vX.Y.Z or dev	winapp release tested
commit	40-char SHA	winappCli commit tested
benchmark	apps | tasks	Which harness produced it
scenario	string	Scenario (apps) or task id (tasks)
set	tuning | held-out	Scenario set
arm	none | cli-only | cli-plus-plugins	What the agent was given
model	string	Model under test
trial	int ≥ 1	Repeat number
startedAt	UTC time	Start time
passed	bool	true when every required check passed
improvements	{ met, total }	improvement checks met out of total
checks[]	list	Every check: id, kind (required | improvement), passed, optional source (harness | judge), optional detail
judge	optional	apps only: judge model and designScore (0-10). Detail only, never decides pass/fail
usage	object	Tokens, credits, time (below)
skills[]	optional	Plugin skills loaded: name, count, optional atMs
winappCommands[]	optional	winapp commands run: command, count
error	optional	Set only when the harness failed, not the agent

Pass/fail. No single score. If any required check fails, the trial fails. If an improvement check fails, the trial still passes and improvements.met goes down by one.

• tasks: required = the checks already marked required. The rest are improvements. A partly done task is not a pass.
• apps: required = the gates (builds, launches, right framework, Native AOT). Quality checks and judge-checked requirements are improvements. A scenario can mark a requirement as required.

Usage. No dollar costs, no price table.

"usage": {
  "tokens": { "input": 412000, "output": 18500, "cacheRead": 380000 },
  "subAgentTokens": null,
  "nanoAiu": 5230000000,
  "premiumRequests": 1,
  "wallClockMs": 612000
}

nanoAiu is the AI credits Copilot CLI reports, as an exact integer (1 AIU = 1,000,000,000). null if not reported.

Where the files are written

• tasks: <run dir>\trials\<scenario>__<arm>__<model>__<trial>.json. release is the pinned release with -Winapp release, else dev. set comes from task.json (default tuning).
• apps: <trial dir>\trial-result.json. release comes from BENCH_RELEASE (default dev). set and required_requirements come from the scenario config. Setups that are not one of the three arms (published) write no file.
• Both keep their existing output (runs.jsonl, results.json). apps token counts are now also kept as exact integers.

Publishing (one manual step)

Runs on any machine stay local. For a release, the release owner runs:

pwsh benchmarks\publish-results.ps1 -Path .\<run folder> -OpenPr

It validates the files, writes <release>\index.json (a summary per scenario/arm/model, so readers load one file instead of hundreds), and opens a PR into benchmark-data (not main). A second person reviews; merging is the publish. Without -OpenPr it only previews. -OpenPr refuses a dev run, mixed commits, a dirty or different checkout, or a release already on the branch. A maintainer still needs to create benchmark-data and add a ruleset that only allows PRs.

Example

benchmarks\examples\v0.7.1\winforms-msix__cli-only__sonnet-5.5__1.json: passed, 1 of 2 improvements met.

Validate

pwsh benchmarks\validate-results.ps1 -Path .\v0.7.1
# OK: 1 result file(s) valid.

Besides the schema, it checks that passed and improvements match checks, check ids are unique, the file name matches its fields, and the folder matches release.

Full reference: benchmarks\RESULTS-FORMAT.md.

Real runs (one trial each, dev)

Run	Result	File
tasks winforms-msix, cli-only	passed, 14.9 credits	validates; publish-results.ps1 preview staged it and wrote index.json
apps calc, cli-only	passed, 12/16 improvements	trial-result.json valid
apps calc, cli-plus-plugins@main	score 92	counts from the real log: skill winui-dev-workflow x1; winapp new x1, run x4, ui ... x5

The last run exposed a bug (the arm name cli-plus-plugins@main was not recognized, so no file was written); fixed in this PR.

Before / after

Nothing a user runs changes.

Not tested / risks

• publish-results.ps1 -OpenPr is tested only up to its guards; it has not pushed or opened a PR (the branch does not exist).
• Only one trial per harness was run for real (below), to check the files, not the numbers.
• subAgentTokens is always null for tasks.
• Questions for the team:
  1. Keep subAgentTokens and premiumRequests? The decision lists only input, output, cache-read and credits.
  2. Is a reviewed PR into benchmark-data the right publish gate?
  3. set is a new optional field on each scenario/task, default tuning. Is that the right place?

Checklist

• Tests added or updated (Pester: validator and publish script, 8 passing)
• Tested locally on Windows
• Docs, samples, and plugins/winapp/skills/ updated if user-facing behavior changed (new benchmarks\RESULTS-FORMAT.md; no plugin changes needed)

[github-actions]

Build Metrics Report

Validation passed. All required build and validation jobs succeeded.

Binary Sizes

Artifact	Baseline	Current	Delta
CLI (ARM64)	65.43 MB	65.43 MB	✅ 0.0 KB (0.00%)
CLI (x64)	65.33 MB	65.33 MB	✅ 0.0 KB (0.00%)
MSIX (ARM64)	27.08 MB	27.08 MB	📈 +0.3 KB (+0.00%)
MSIX (x64)	28.69 MB	28.69 MB	📈 +0.3 KB (+0.00%)
NPM Package	56.40 MB	56.40 MB	📈 +0.2 KB (+0.00%)
NuGet Package	56.55 MB	56.55 MB	📈 +0.1 KB (+0.00%)

.NET Test Results (TRX reports)

Other suites are reflected in the overall validation status above.

✅ 10920 passed, 44 skipped out of 10964 tests in 2067.0s (+281.3s vs. baseline)

Test Coverage

✅ 87.6% line coverage, 81.6% branch coverage · ✅ no change vs. baseline

CLI Startup Time

73ms median (x64, winapp --version) · ✅ no change vs. baseline

Try This Build

Installs the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing.

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 1200

Switching between builds often?

Put the tool on your PATH once:

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPath

Then this build is just:

winapp-pr 1200

Run winapp-pr with no arguments to pick from a list of open PRs.

---

Updated 2026-11-03 21:42:40 UTC · commit a92e0c4 · workflow run

[Zach Teutsch (zateutsch)]

🤖 AI-generated review (winappcli pr-review skill) — verify before acting.

Decision

Changes required — previewing results can delete the input files, and exported results can report successes, source commits, and activity counts incorrectly.

Reviewed commit 821426963da81cc7356555d241fd1081a00e9108 against main.

Must fix

Preview can delete the results it is reading

• What is wrong: The publisher deletes its destination before checking whether it contains the source files. It also uses release as a path before validating it.
• Show me: In an isolated fixture, -Path .\results -OutDir .\results deleted the original release JSON, then failed copying it. A malformed "release":"..\\victim" also deleted a sibling marker before validation rejected the file.
• Why it matters: A preview can destroy files without -OpenPr.
• Smallest fix: Validate source data before filesystem changes, constrain destinations to the staging directory, and reject destinations containing source files.
• Location: benchmarks\publish-results.ps1:50–59

Failed judging becomes a passing app trial

• What is wrong: When the judge—the model checking the app’s requirements—fails, the writer substitutes an empty requirement list and ignores the recorded failure.
• Show me: Passing build/launch checks, failReason: "Validation timed out", and an untested required requirement produced "outcome":"valid","passed":true. The real validator accepted it, and the publisher counted validTrials: 1, passed: 1.
• Why it matters: A tooling failure inflates the published success rate.
• Smallest fix: Skip trial emission when attempted judging fails, matching the documented current policy for harness failures.
• Location: benchmarks\apps\src\runner\benchmark.ts:4730–4751

Release results can identify the wrong plugin commit

• What is wrong: The exported commit comes from the harness checkout, although the plugins can come from another Git ref. The default plugin ref is main.
• Show me: From release checkout A, --agent cli-plus-plugins@B resolves plugins at B, but the result records A through currentCommit(). The publisher’s checkout check then passes against A.
• Why it matters: Published release numbers can describe different, potentially unreleased plugins.
• Smallest fix: Reject release-labeled exports when the resolved plugin commit differs from the claimed commit.
• Location: benchmarks\apps\src\runner\benchmark.ts:4733–4734

Task exports lose skill and command counts

• What is wrong: The converter reads object properties, but its production caller passes an ordered dictionary. Those lookups miss the dictionary entries.
• Show me: A production-shaped record with three winapp-setup loads and four package commands exported skills: [{"name":"winapp-setup"}] and no winappCommands.
• Why it matters: Published summaries undercount skill use and omit command activity.
• Smallest fix: Pass [pscustomobject]$record, as the preceding serialization already does, and cover the production dictionary shape in a test.
• Location: benchmarks\tasks\lib\TaskBench.psm1:219–220

Non-blocking

Failed PR creation can return success

• What is wrong: The publisher does not check gh pr create’s exit code; successful cleanup overwrites it.
• Show me: Executing the final block with a simulated gh failure and successful cleanup left $LASTEXITCODE at 0.
• Why it matters: Callers checking the exit status can miss a failed publishing handoff.
• Smallest fix: Check the exit code immediately after gh pr create and throw on failure.
• Location: benchmarks\publish-results.ps1:137–140

What was exercised

• Invoke-Pester -Path .\benchmarks\tests, .\benchmarks\tasks\tests, .\benchmarks\agents\tests — 235 passed.
• npm --prefix .\benchmarks\apps test — 68 passed.
• npm --prefix .\benchmarks\apps run typecheck — passed.
• Isolated reproductions confirmed deletion, count loss, failed-judge acceptance through publication staging, and exit-code masking.
• No paid benchmark trials or GitHub publication were run. The plugin-commit mismatch was traced through source resolution and export code, not exercised with a full trial.

[Jaylyn Barbee (Jaylyn-Barbee)]

Fixed in 05bc7e5: publishing a preview no longer deletes its own results (unsafe release/scenario/arm/model values and an -OutDir that overlaps the source are rejected first). Apps trials with failed judging, or plugins from a different commit than a release, no longer write a trial file. Task exports keep skill and command counts. A failed gh pr create now fails the script.

[Copilot]

🟡 Changes recommended

Unresolved critical issues allow unchecked requirements to pass and stale or incorrectly attributed benchmark results to be published.

7 open issues

• Preserve original trial identity during revalidation · New
• Remove stale trial results when evaluation cannot be exported · New
• Fail unchecked required requirements when validation is skipped · New
• Verify plugin sources before exporting release results · New
• Keep summary collections consistently serialized as arrays · New
• Skip unsupported configurations in the shared-format writer · New
• Accept non-valid trials when no checks fail · New

What changed in this PR

Defines a shared per-trial results format for the apps and tasks benchmarks, giving future dashboard and README summaries a common data source.

Changes:

• Adds the JSON schema, format reference, example, and validation script.
• Updates both harnesses to export checks, outcomes, exact usage, and activity counts.
• Adds release staging, summary indexes, and an opt-in PR publishing workflow targeting benchmark-data.

File	Description
benchmarks/​validate-results.ps1	Validates schema, result consistency, and file layout.
benchmarks/​tests/​validate-results.Tests.ps1	Tests accepted results and validation failures.
benchmarks/​tests/​publish-results.Tests.ps1	Tests staging and publishing safeguards.
benchmarks/​tasks/​tests/​TaskBench.Tests.ps1	Tests task result conversion and usage counts.
benchmarks/​tasks/​run.ps1	Writes task trial files and captures activity counts.
benchmarks/​tasks/​lib/​TaskBench.psm1	Converts task records into the shared format.
benchmarks/​results.schema.json	Defines the trial result structure.
benchmarks/​RESULTS-FORMAT.md	Documents fields, outcomes, storage, and publishing.
benchmarks/​publish-results.ps1	Stages results, builds indexes, and optionally opens a PR.
benchmarks/​examples/​v0.7.1/​winforms-msix__cli-only__sonnet-5.5__1.json	Provides an example task result.
benchmarks/​apps/​src/​types.ts	Adds scenario sets and required requirement IDs.
benchmarks/​apps/​src/​runner/​trial-result.ts	Converts app trials into the shared format.
benchmarks/​apps/​src/​runner/​trial-result.test.ts	Tests app result mapping and naming.
benchmarks/​apps/​src/​runner/​benchmark.ts	Captures exact usage and writes app trial files.
benchmarks/​apps/​src/​runner/​activity.ts	Counts skill loads and winapp commands from events.
benchmarks/​apps/​src/​runner/​activity.test.ts	Tests activity parsing and counting.
benchmarks/​agents/​lib/​Benchmark.psm1	Preserves exact credits and optionally retains repeated commands.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Critical gaps in release provenance and export handling can allow mismatched, stale, or failed trials to be published as valid results.

5 open issues

• Reject release exports when the resolved CLI version mismatches · New
• Delete stale trial exports before starting a rerun · New
• Treat unexpected revalidation failures as judge failures · New
• Verify the checkout matches the pinned release commit · New
• Preserve attempt identity independently of the trial export · New

1 previously missed issue

In code that hasn't changed since last review

Prevent subcommand matching from crossing newlines

benchmarks/​apps/​src/​runner/​activity.ts:28

Do not consume the next command as a subcommand. For a multiline shell call containing winapp restore\nwinapp run, the optional second-token match crosses the newline and consumes the second winapp, so the exported activity records only restore. Restrict token spacing to spaces and tabs so each command remains available to the next match.

7 resolved since last review

• Verify plugin sources before exporting release results
• Fail unchecked required requirements when validation is skipped
• Remove stale trial results when evaluation cannot be exported
• Preserve original trial identity during revalidation
• Accept non-valid trials when no checks fail
• Skip unsupported configurations in the shared-format writer
• Keep summary collections consistently serialized as arrays

_{Review effort: Balanced}