WingFoil Benchmark

A benchmark that measures where a harness for AI coding agents makes a difference, and where it does not. A harness is the process and memory tooling around an agent; the one measured first is WingFoil. The same agent and model work through the same multi-step tasks with and without the harness, and the results are compared on quality, cost and determinism, by rules published before any run.

v0.1: the first preliminary result

• Results: https://wingfoil.github.io/wingfoil-benchmark/c82a5e74885b/2/
• How it works: the method page, every rule, its limits included.
• Release notes: docs/releases/v0.1.md: what was compared, the results in short, the spending, and the known limits.

v0.1 compares three arms (baseline, baseline-docs and wingfoil, on WingFoil v0.2.2) on four scenarios (S1, S2, S3 and S8), with Claude Code and claude-sonnet-5, plus one run of S1 on Opus 5. Its results are preliminary: only S1 was repeated (three times), and no difference is claimed as significant. Competing harnesses (Spec Kit, OpenSpec) are planned for v0.2.

The benchmark is built by WingFoil's maintainer. That is threat T1 of the experiment design. The questions, metrics and scenarios were fixed before any run, losses are published like wins, and every run's record, its cost and its agent transcript are public, so that the results can be checked and contested.

What is where

Path	What it holds
docs/01_vision/	the product vision, the personas, the features, the release sequencer and the experiment design
docs/02_specification/	requirements, acceptance scenarios (Gherkin), scenario specifications, traceability
docs/calibration/	the calibration of v0.1 (dry runs, measured costs, budget) and the spending ledger
docs/memory/	the project's records, managed with WingFoil: releases, tasks, bugs, decisions, campaigns
docs/releases/	release notes
scenarios/	the scenarios, their seeds, prompts and public hidden tests, per version
arms/	each arm's operating manual, setup and configuration
campaigns/	campaign files; the reference campaign of v0.1 is v0-1-reference.yaml
results/	every execution's records and scores, run by run (transcripts are release assets)
findings/	finding notes, generated from results, for WingFoil's maintainer to file
eligibility/	the eligibility register: every assessed tool and version, WingFoil included, with its verdict
site-content/	the method and eligibility pages' prose; the rest of the site is generated
src/	the bench command: runner, scoring, aggregation, site

The hold-out tests and S2's answer key live in a separate, private repository, so that public material cannot be used to pass them (threats T5 and T13).

Tooling

This repository is itself managed with WingFoil. The WingFoil version that manages this repository is pinned in vendor/ (built from a fixed WingFoil commit) and is independent of the WingFoil version under test, which each benchmark campaign selects on its own.

npm install
npx wingfoil --help

Private oracles and hold-out scenarios live in a separate, private repository.

Development

One TypeScript package (Node.js ≥ 22.12), organized as modules under src/ that depend only downwards (docs/02_specification/requirements.md REQ-ARC-01/02; the lint enforces the rule).

npm test          # unit and acceptance tests, coverage at or above 80%
npm run lint      # ESLint and Prettier
npm run typecheck
npm run build     # compiles src/ into dist/

Acceptance tests live in test/acceptance/, one per Gherkin scenario of docs/02_specification/acceptance/, named @<feature> <scenario title>. A traceability test fails when a scenario of a started task has no test.

Scenarios live in scenarios/<id>/<version>/scenario.yaml. The trivial scenario used by the runner's own tests is test/fixtures/scenarios/T0/1.0/.

Several sessions work the repository at once. Their rules (among them: one linked worktree each, no rewrite of main, approval commands that name their checkout, real-agent runs from the main checkout) are the multi-session directive: .wingfoil/directives/custom/multi-session.md, listed for every role by npx wingfoil directives list --role <role>.

Commands

npm run build compiles the bench command; then:

npx bench campaign validate campaigns/<name>.yaml

BENCH_FAKE_SCRIPT=<script.json> npx bench campaign run campaigns/<name>.yaml

BENCH_AGENT_TOKEN_FILE=<token-file> npx bench campaign run campaigns/<name>.yaml --allow-spending

validate checks the campaign file (docs/02_specification/requirements.md REQ-FMT-01): every harness pinned to a released version or a commit that eligibility/register.yaml admits (F7.4), the baseline arm present, every scenario it names present under scenarios/<id>/<version>/, and, when the file pins arm_digests, every arm's files unchanged since (REQ-FMT-13). It prints the campaign's identity, the digest that names its results. It then lists what the campaign's runs need from the machine, one requires line each, set, missing or invalid with the reason (bug-014):

Variable	Names	Needed by	When
BENCH_AGENT_TOKEN_FILE	the file holding the agent's token	run	a real agent
BENCH_FAKE_SCRIPT	the fake agent's script	run	the fake agent
BENCH_WINGFOIL_REPO	the local WingFoil clone the harness is built from	run	a wingfoil harness
BENCH_SPECKIT_REPO	the local Spec Kit clone (github/spec-kit) the harness is built from	run	a speckit harness
BENCH_HOLDOUT_PATH	the hold-out (or --holdout)	score	a scenario that declares one

run refuses, before its estimate and naming every one, when a variable it needs is not set, or names the wrong kind of path. run executes every scenario × arm × repetition with the default model, then each model slice's scenarios × arms × repetitions with the slice's model: one Docker image per campaign, one container per run, whose only mount is that run's fresh workspace (a copy of the scenario seed, made a git repository). Workspaces land in runs/<campaign-id>/<n>/, the execution in results/<campaign-id>/<n>/, which holds run.json per run and steps/<NN>/{usage.json, transcript.jsonl, diff.patch} per step. Transcripts are git-ignored.

Each step is a fresh agent session: nothing but the repository carries state from one step to the next, and after each step the workspace is committed as step <NN>.

When a session ends waiting for input — a question, or a request for approval — the neutral approver answers it in the same session, with the fixed reply of the approver policy the campaign pins (approver_policy: v1): "Approved. Proceed." to an approval request, "No further input is available. Make the most reasonable choice, record it, and proceed." to a question. Each reply is an intervention, recorded in run.json with its step, kind and text. After three in one step, a session still waiting gets no reply and the step ends as intervention cap reached; the run goes on. The policy is the same in every arm.

Which agent runs is the campaign's agent.name. The scripted fake agent costs nothing and reads its script from BENCH_FAKE_SCRIPT; a script may name a recorded session for the fake to replay, and one recorded session per reply it expects from the approver, so the whole pipeline can be exercised without an agent. Claude Code reads its long-lived token from the file named by BENCH_AGENT_TOKEN_FILE — created with claude setup-token — which reaches the container as an environment variable and is scrubbed out of every stored transcript. Because it spends real money, a campaign naming it refuses to start unless --allow-spending is passed.

npx bench scenario dry-run <id>@<version> --arm <arm> [--model <id>] [--allow-spending]

A dry run is one real run of a scenario version in one arm, made to measure what it costs. It takes what a campaign file would pin — agent, harnesses, approver policy, caps, currency rate — from scenarios/dry-run.yaml, and it is stored in results/dry-runs/<n>/, marked dry_run: true in its run.json and never counted as a campaign's result. Before it starts, it prints what the latest completed dry run of the same version, arm and model cost; when it ends, it prints what this one cost per step and in total.

npx bench campaign estimate campaigns/<name>.yaml

estimate prices a campaign before it runs, from its scenarios' dry runs. For every scenario version, arm and model the campaign file describes — its model slices included — it reads every completed dry run and prints the lowest, the mean and the highest cost times the repetitions. A key with a single dry run is priced at that cost plus a stated margin of 35 %. Then the totals, the highest in euro at the campaign's rate, as an API-equivalent cost. It starts nothing. A key with no dry run fails the estimate, naming the dry run to make. A key is never priced from another model's dry run. campaign run prints the same totals before it starts, and what the campaign spent when it ends.

Before anything is built, campaign run applies the budget guard. Its figure is what every execution of the campaign already spent, earlier attempts included, plus the highest estimate of what is left: the whole campaign for a new execution, only the runs a resume will run for --resume. Both commands print it as budget guard: spent + estimate = figure. A campaign whose cost cannot be estimated does not start, nor does one whose figure is above its ceiling_eur — no option overrides the ceiling. Above warn_eur it shows the figure and the threshold and starts only when the maintainer answers y on a terminal; without a terminal it does not start.

While it runs, the caps hold. Each session is given what is left of the run's run_cost_eur as the agent's own budget, and stops there: the step and the run end cap reached, the step's snapshot kept. Every command of a step runs under timeout with what is left of step_time_s: a step killed there ends time cap reached, and since a killed session reports no cost, it is counted at the most it can have cost. A session whose tokens exceed step_tokens is not resumed (token cap reached). No run starts once the campaign's executions so far have spent ceiling_eur (campaign ended: budget exhausted), nor after a session met the subscription's usage limit (quota exhausted).

Stopping a campaign. The first Ctrl-C (SIGINT) or SIGTERM stops it cleanly: the current step's agent is killed with its container at once and counted at its bound, as a step killed at its time cap; the run ends interrupted; no further run starts (campaign ended: interrupted). A second signal ends the process at once, leaving execution.json running and possibly the run's container (see below). A campaign file may also set fail_fast: {same_error: <k>}: once k runs failed at the same step with the same error, no further run starts (campaign ended: fail_fast). Without it a failed run never stops the campaign. Each execution records its state in results/<campaign-id>/<n>/execution.json: running, then completed, or stopped with the reason (interrupt, quota, ceiling, fail_fast, or error when the execution itself could not go on) and the dates it started and ended. scenario dry-run stops the same way, without an execution.json.

Resuming a campaign. npx bench campaign run campaigns/<name>.yaml --resume <campaign-id>/<n> goes on with execution n rather than starting a new one. It runs only the runs that never started and those that ended failed, interrupted or quota exhausted; a run that completed, or reached its cost cap, is never run again, since that would select results. Each earlier attempt is kept under its run's attempts/<k>/, and run.json records attempts. The execution's ceiling counts what it already spent. A resume is refused when the campaign file changed (another campaign identity), when the execution has no execution.json (v0.1's), when it is already aggregated, and when one of its containers is running.

A run that is killed some other way — a killed process, a closed terminal, a reboot — can leave its container behind. The runner never removes a container it did not create: at the start of a campaign it lists the ones earlier runs of that campaign left, warns about each with the docker rm --force that clears it, and fails the run whose container name is still taken, with the same command.

npx bench score <campaign-id>/<n> [--final] [--holdout <path>]
npx bench score dry-runs/<n>

An execution whose execution.json does not say completed is scored but not aggregated, so that it can still be resumed. --final declares it final: it is aggregated as it stands, aggregate.json lists the runs it never started under not_run, and execution.json records final: true, after which it can no longer be resumed. An execution without execution.json (v0.1's) aggregates as before.

score scores every run of an execution with its scenario's hidden tests, outside every run container. Each step's snapshot is rebuilt from what the run stored — the seed, the setup's patch and each step's patch — and checked against the tree the run recorded. Each suite then runs under node:test with tsx, in a scoring container built from docker/score-image/, with no network, the suite mounted read-only and the snapshot copied in. A suite's total is what it counts on the seed; a test a snapshot does not pass, or never reports, fails. The result is score.json beside run.json: M-Q1 for every step a suite is bound to and for the final snapshot, byte-identical when scored again (adr-004). With --holdout (or BENCH_HOLDOUT_PATH), the hold-out's additions run the same way, in containers of their own beside the public suite, and are stored apart under holdout, in counts only: no name, message or output of a hold-out test is printed or stored. Without one, holdout says why it was not scored. score.json also records the run's cost per step and in total — tokens by kind, API-equivalent cost in USD and euro, wall time, turns and interventions — from what the run stored; a step killed at its time cap counts at the most it can have cost, marked cost_reported: false.

An arm with a harness declares the capabilities its harness provides (arms/<arm>/arm.yaml); a scenario declares the ones it needs. A run whose harness lacks one is an expected failure: it runs and is scored like any other, and is marked in run.json and score.json with the missing capability named — never skipped. campaign validate lists them before anything runs. The baseline arms are the reference and are never marked. A harness arm also declares telemetry_off, the NAME=value settings that turn its tool's telemetry off in its container ([] when it has none), and a docs control declares docs_of, the harness arm it is generated from. One line per run: [email protected] baseline fake-model r1: step 01 0/1, step 02 1/1, final 1/1; hold-out final 1/2.

npx bench run show <campaign-id>/<n>/runs/<scenario>@<ver>/<arm>/<model>/r<k> [--full]
npx bench run compare <run> <run>

run show prints one stored run: its pins, every step's usage, models, commits, diff and test results. run compare sets two runs side by side, step by step.

npx bench campaign compare <earlier-campaign-id>/<n> <newer-campaign-id>/<m>

campaign compare compares two aggregated executions, the earlier first, as you order them. It works group by group, where a group is a scenario version and an arm present in both. For each metric of the scenario's category, it records each execution's delta against its own baseline and the change of that delta: across campaigns only deltas are compared. Each delta carries both sides' n and range, and is preliminary unless both sides have at least 3 runs. It writes results/comparisons/<newer-id>-<m>/<earlier-id>-<n>.json and prints the pins that differ (model, agent, effort, harnesses). Rows on two different default models are marked cross-model. A scenario whose content changed is not compared, and the comparison names the two hashes; so is one whose hash is unknown, whose category has no map of metrics or cannot be read, and a slice with no counterpart. A side with no delta (no baseline, not measured, not comparable) says why and keeps the arm's own value when it has one, and its row has no change. A dry run, an execution not yet aggregated, or one execution twice is refused. site build publishes each comparison whose newer side is the execution it builds as compare-<earlier-id>-<n>.html, linked from that execution's landing page.

npx bench finding <campaign-id>/<n> --scenario <id>@<version> --metric <metric> --arms <arm>,… --as bug|decision-log

finding writes a finding note under findings/ from an aggregated execution: the campaign, the WingFoil commit, the runs and the metric's values, ending with a block in the shape of a WingFoil bug or decision record.

npx bench site build <campaign-id>/<n>
npx bench site publish [--remote <name>]
npx bench transcripts pack <campaign-id>/<n>

site build writes the static site of an execution into site/<campaign-id>/<n>/, a permanent address that later builds of other executions leave as it is. Besides the landing and category pages, it writes one page per result profile of site-content/profiles.yaml (a copy of the personas' profiles): the landing page restricted to the profile's categories, with its own headline. site/index.html lists every execution built, with the day each ended. site publish pushes site/ to the gh-pages branch, only when the repository is public and only what a fresh build gives; it refuses to drop an execution gh-pages already publishes, so that a permanent address never breaks. transcripts pack packs an execution's transcripts into one release asset and the harness artifacts its runs installed (from .cache/harnesses/) into a second, harnesses.tar.gz, and prints the gh release create command, which it does not run.

npx bench images prune [--dry-run]

images prune removes the images the benchmark built — dry-<12 hex>:latest (dry runs), <12 hex>:latest (campaigns), bench-score:<12 hex> (scoring) — that no container uses and that are not the current scoring image, and prints each one, those kept and why, and the count. It never touches another image, and refuses while a named benchmark container (bench-…) is running. Run it only when no campaign, dry run or scoring is in progress anywhere on the host: between two runs no container holds a run's image, and removing it fails the next run. The size printed per image is the image's, shared by the tags of one build. --dry-run lists without removing (bug-015).

Exit codes: 0 success, 1 failure (one line per problem, or a run that failed), 2 usage error.

npm run test:bin     # the built command line
npm run test:docker  # W1's to W3's runs, a dry run, bug-003's, W6's scoring, the image port; in real Docker
                     # (the speckit arm's test runs when BENCH_SPECKIT_REPO names a clone of github/spec-kit)

License

MIT — see LICENSE.

Full suites

scripts/full-suites.sh runs the full suites — lint, npm test with coverage, the build, test:bin and test:docker — on the main checkout and every task/* worktree, or only on the branches it is given, one at a time, niced and at idle I/O priority with 2 vitest workers (the containers test:docker starts run under Docker's daemon, at its priority). Each branch's log is .cache/full-suites/<date>/<branch>-<commit>-<time>.txt (git-ignored), never overwritten, and summary.txt gets a line per branch: the commit, DIRTY when the worktree had changes not committed, each stage's exit code, and CHANGED when the worktree's commit or files changed during the run. A docker test skips itself when its harness clone is missing (BENCH_WINGFOIL_REPO, BENCH_SPECKIT_REPO): read the log's counts, not only its exit code.

scripts/full-suites.sh                 # main and every task/* worktree
scripts/full-suites.sh task/task-075-… # one branch

They take tens of minutes and load the machine, so they run on demand when it is free, never from a cron on a machine that sleeps (dl-016, task-075). Their green log is what the approver approves a task on (kanban-delivery version 5).