Recursive Agent Harnessing for an Open World

Fork notice. This is an unofficial fork of google-research/harness2, not affiliated with Google or the paper's authors. It adds pi (Pi coding agent) and dsh (DeepSeek Harness) executor substrates for JobBench and the custom benchmark, a custom benchmark for your own tasks without reference answers, and harness2-export / HARNESS2_CUSTOM_SEED for carrying a harness between runs. The paper's results below were produced upstream with the OpenCode and Codex executors. The additions here have been run end to end on a handful of tasks: a Pi run through the full refinement loop on one JobBench task, a DeepSeek Harness rollout on the same task, and custom-benchmark rollouts with both substrates - one of them on a second provider, through Pi. That is a smoke test, not a benchmark, and LAB and WorkBuddy-Bench do not support the new substrates.

A harness is the system around a model that enables it to act as an agent: prompts, tools, memory, skills, orchestration, and more. Harnessing is the process of optimizing that system.

Harness² is a plug-and-play framework for recursive agent harnessing in an open world, where agents encounter tasks unseen during harness design. It connects two levels of recursion while keeping model weights fixed:

• Task-level recursion refines the execution harness for the current task through a contrastive Propose–Probe–Compose step: propose edits, probe their effects on the same task, and compose a refined harness from behavioral contrasts between executions, without ground-truth feedback.
• Harnessing-level recursion updates a persistent improvement harness with adopted harnesses and evidence-supported lessons from each task to guide future refinement.

Upstream paper results, from the OpenCode and Codex executors. They do not describe this fork's additions: across three professional benchmarks and four model–harness configurations, a single Harness² step improves GLM-5 and Gemini 3.5 Flash over their base harnesses, with gains of up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB's partial pass rate. Increasing the task-level recursion budget brings further gains.

Upstream's release supports parallel and sequential recursion, OpenCode and Codex executors, and JobBench, LAB, and WorkBuddy-Bench; this fork adds two more executors and a fourth benchmark, described below.

Video overview

harness2-video.mp4

Contents

• Video overview
• Harness components
• Install
• Run a benchmark
• Executor substrates: Pi and DeepSeek Harness
• Your own tasks
• Experiments
• Web access
• Results and resume
• Citation
• Contributing
• License
• Disclaimer

Harness components

Harness² exposes eight editable components around the model: system prompt, guardrails, memory, tool guidance, subagents, skills, scripts, and plugins. They control context construction, the tool interface, delegation, and behavior inside the tool loop.

Install

Use Linux, Python 3.12, and uv. From the repository root:

uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e '.[render]'
harness2-demo --output runs/demo --k 2

The local demo runs both recursion modes with deterministic editors and real Python executions. It requires no model credentials.

The custom benchmark with the pi executor needs no Google Cloud project, no Docker and no OpenCode checkout: Node.js, npm, git and uv (already required to install this package), one executor key, and a directory of your own tasks. A dsh run also installs Pi, which runs the improver. Judging a task that has no check.py needs a second key for the judge. That path is exercised on macOS as well as Linux. LAB and WorkBuddy-Bench keep the requirements below.

For model-backed benchmarks, install Node.js 20+, the Google Cloud CLI, and the benchmark dependencies. LAB needs Pandoc and Poppler; WorkBuddy-Bench needs Docker. Then configure your models:

cp .env.example .env
# Edit .env with your Vertex AI project and model IDs.
gcloud auth application-default login

See the setup guide for model settings, authentication alternatives, and the Codex executor.

Run a benchmark

The following example installs LAB and runs one task, then evaluates and reports its result. Model-backed runs incur API costs; start with a small task selection.

If you came for what this fork adds: --substrate pi or dsh runs on JobBench and on the custom benchmark, and the next two sections cover the substrate choice and custom, which is the shortest path to a real run.

harness2-setup lab
harness2-check --bench lab
harness2 --bench lab --k 1 --mode parallel \
  --only-domains antitrust-competition --max-tasks 1
harness2-judge --bench lab --k 1 --mode parallel
harness2-report --bench lab --k 1 --mode parallel

Sequential recursion runs the same commands with --mode sequential. --k is the task-level recursion budget; see Experiments. Judging and reporting must use the same benchmark, executor, subset, --k, --mode, and seed as the refinement run. Add --dry-run to harness2 to preview task selection without model calls.

Benchmark	Tasks	Install	Guide
JobBench	Professional work	harness2-setup jb	JobBench
LAB	Legal work	harness2-setup lab	LAB
WorkBuddy-Bench	Office, web, and code	harness2-setup wb	WorkBuddy
Custom	Your own tasks	harness2-setup custom --substrate pi	Your own tasks

Custom is the fork's addition and takes no dataset. Choose one benchmark per command. WorkBuddy also takes --subset office, --subset web, or --subset code on every command after setup. The benchmark guides cover each one's data, domains, and grading.

Executor substrates: Pi and DeepSeek Harness

--substrate chooses what runs the task. Four are wired in; pi has been run through the full refinement loop on one JobBench task, and dsh through one complete JobBench rollout and the custom benchmark:

Substrate	CLI	Notes
oc	OpenCode	Upstream default; runs from a pinned checkout through bun
cx	Codex CLI	Pinned binary under dependencies/codex
pi	Pi	npm install -g @earendil-works/pi-coding-agent, or let setup install 1.3.0 into dependencies/pi
dsh	DeepSeek Harness	Setup installs 0.2.1-rc.2 into dependencies/dsh, plus Pi for the improver; slower per task, see below

harness2-setup jb --substrate pi
harness2-check --bench jb --substrate pi
harness2 --bench jb --substrate pi --k 1 --only-domains technical_writers --max-tasks 1
harness2-judge --bench jb --substrate pi --k 1

pi and dsh run as coding-agent CLIs in a scratch workspace, not from a pinned checkout. Each rollout copies the harness to the workspace root and points the agent at an empty home (PI_CODING_AGENT_DIR or DSH_HOME/DSH_AGENTS_HOME), so discovery starts from the harness rather than from your own setup -- unless you set HARNESS2_PI_AGENT_DIR, which deliberately hands Pi an existing directory holding saved logins. Pi is given its skills and extensions explicitly; dsh reads .agents/skills/ itself and has its plugins loaded by path. The workspace lives under HARNESS2_SCRATCH (default: the system temp directory) and is refused if it would sit inside a git checkout, because both agents find their project root by walking up to .git.

Model access is configuration, not code. HARNESS2_PI_MODEL and HARNESS2_DSH_MODEL default to HARNESS2_SOLVER_MODEL, and both CLIs read their credentials from the provider's environment variable (OPENCODE_API_KEY, DEEPSEEK_API_KEY, ...) -- though DeepSeek's own API is the one route that has not been called here, its bare-id mapping being verified from dsh's composed configuration instead. A provider that authenticates by saved login instead works through HARNESS2_PI_AGENT_DIR, which points Pi at an agent directory holding that login. Nothing here is tied to a particular vendor; an OpenCode Go key and a saved OpenAI Codex login have both been exercised, and any other gateway is the first thing to smoke-test.

DeepSeek Harness works an assignment in more, smaller steps than Pi, so it needs more wall clock: on one JobBench task it took 24 minutes and 191 tool calls, against 6 minutes and 42 for Pi; both wrote the same four files. The rollout deadline is 2400 s for Pi and DeepSeek Harness on JobBench (Codex gets 3600 s) and 2400 s for every substrate on the custom benchmark; LAB allows 3600 s and the CLI exposes no flag to change any of them, so an unusually large task reaches that limit on dsh first.

Both substrates also validate a proposed harness while it is still a proposal, by loading it offline with credentials stripped: Pi exits before its first model call if an extension will not load, and silently drops a skill whose name it cannot parse; dsh refuses to import a plugin that is not a module exporting name and apply. Either failure is reported back to the improver instead of being paid for as a rollout.

Your own tasks

--bench custom refines a harness on work you care about, with no reference answers and no dataset to download. Point it at a directory:

my-tasks/
  writing/
    exec-summary/
      TASK.md          the assignment, verbatim
      files/           optional reference material
      check.py         optional check, run against the deliverables

export HARNESS2_CUSTOM_TASKS=~/my-tasks
harness2-check --bench custom --substrate pi
harness2 --bench custom --substrate pi --k 1 --only-domains writing --max-tasks 1
harness2-judge --bench custom --substrate pi --k 1

check.py is optional and, when present, decides the verdict by exiting 0. With no check.py and a judge configured, each deliverable is rated against its own assignment: the task text plus what the agent produced, never a golden answer. Rated output is read as text, so a binary deliverable such as a spreadsheet is described to the judge by name and size instead of its contents -- give such a task a check.py. With neither a check nor a judge, rollouts still run and are reported, and the measurement is absent; a run that produced no deliverable at all is the one case recorded as zero. OpenCode and Codex are not supported here, because they need a benchmark-owned checkout, render step and credential file.

Use several tasks per area. A single task gives the improver one example to reason from, and it will write the specifics of that example into the harness; harness2-check warns when an area holds only one.

A rollout is not a sandbox: pi and dsh run with your permissions, and the assignment only tells the agent to stay in its workspace. A vague task invites it to look around the machine, and if it reads anything that looks like a verdict the harness refuses to proceed rather than let it into the improver's context. Keep HARNESS2_SCRATCH and HARNESS2_RUNS_DIR away from the code you are refining, and if you want the boundary enforced rather than requested, that is what the in-loop lever is for: an extension or plugin you write can refuse a read outside the workspace, and the prompt surfaces document how.

Carrying a harness between runs

harness2-export writes a harness out as a portable seed, and HARNESS2_CUSTOM_SEED imports it as the tree the recursion starts from:

# a harness adopted while refining on JobBench
harness2-export --bench jb --k 1 --task technical_writers/task1 \
  --substrate pi --to ~/my-harness

# or any tree you wrote by hand
harness2-export --from ./some/harness --substrate pi --to ~/my-harness

# start from it on your own tasks
HARNESS2_CUSTOM_TASKS=~/my-tasks HARNESS2_CUSTOM_SEED=~/my-harness \
  harness2 --bench custom --substrate pi --k 1

The export is the editable tree itself plus harness2.json, which records the source, the fingerprint, and the files it contains. Unset HARNESS2_CUSTOM_SEED to fall back to the built-in one-file seed.

Experiments

Scale task-level recursion in two ways: parallel recursion explores multiple candidate harnesses and composes refinements from their execution evidence; sequential recursion carries the refined harness and accumulated evidence through successive Propose–Probe–Compose steps.

Parallel recursion explores wider; sequential recursion refines deeper.

Option	Behavior
--k 1	One Propose–Probe–Compose step
--k K --mode parallel	Probe K general and K domain-specific candidates, then produce K compositions
--k K --mode sequential	Refine the harness through K successive steps
--substrate cx	Use Codex instead of the default OpenCode executor

After each task, harnessing-level recursion retains the adopted harness and evidence-supported lessons to guide future refinement. See the experiment guide for task selection, concurrency, reproducibility, and baseline comparisons.

Web access

Executor network access is unrestricted, and Codex runs with its sandbox disabled. Public benchmark rubrics may be accessible online. Before evaluation, configure network restrictions to prevent access to benchmark answers and grading criteria.

Results and resume

Results are saved under runs/, organized by benchmark and subset where applicable. They include task summaries, deliverables, adopted harnesses, reflection notes, and evaluation results.

Repeat the same command to resume. Use a new HARNESS2_RUNS_DIR when changing models or task selection. See results and resume for details; each command also supports --help.

Citation

@article{xu2026harness2,
  title   = {Harness$^2$: Recursive Agent Harnessing for an Open World},
  author  = {Xu, Ruiyao and Chen, Yanfei and CuiZhu, Zhongying and Dalvi Mishra, Bhavana and Ming, Yifei and Yu, Han and Han, Rujun and Lee, Chen-Yu and Pfister, Tomas},
  journal = {arXiv preprint arXiv:2610.14224},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.14224}
}

Contributing

See CONTRIBUTING.md.

License

Harness² is licensed under Apache 2.0. Third-party code under third_party/, benchmarks, and datasets retain their upstream licenses; see third-party notices.

Disclaimer

This is not an officially supported Google product. This project is not eligible for the Google Open Source Software Vulnerability Rewards Program.