Repository navigation
Harness² CLI: a general-purpose version of Harness² - #11
Merged
Merged
Conversation
Describe the editable surface of the new Claude Code executor (CLAUDE.md, .claude/skills, .claude/scripts, sub_agents installed as .claude/agents, .claude/hooks.json merged into .claude/settings.json) and of the generic command executor, whose surface is a list of path patterns. Codex harness scripts move to .agents/scripts so they cannot collide with a project's own scripts/ directory. The shared INV template now takes the script and skill levers per surface instead of hardcoding scripts/ and trigger lines. Co-authored-by: Claude Opus 5.5 <[email protected]>
settings.py resolves configuration from defaults, the global ~/.config/harness2/config.toml, the project's .harness2/config.toml, a named profile, HARNESS2_<SECTION>__<KEY> environment variables and CLI overrides, and records where every value came from. It also finds the project root, loads .harness2/.env, and edits TOML files in place without dropping comments. providers.py turns a provider profile (anthropic, openai, vertex, bedrock) into the environment or config each agent CLI expects, so no code path assumes Vertex AI any more. Secrets are only ever referenced through api_key_env. registry.py finds task suites: the built-in "tasks" suite plus any installed through the harness2.suites entry-point group. Co-authored-by: Claude Opus 5.5 <[email protected]>
…#3) JobBench, LAB and WorkBuddy-Bench now live in their own distribution under plugins/harness2-benchmarks/ (package harness2_benchmarks) and register through the harness2.suites entry-point group as the jb, lab, wb-office, wb-web and wb-code suites. The core package keeps only the method, the executors and the CLI. Moved with history: the benchmark adapters and renderers, the installer, preflight and setup, the vendored runners and patches (third_party/), the dependency metadata, the benchmark scripts and the benchmark docs. Downloaded benchmark dependencies now go to the user cache (~/.cache/harness2/benchmarks, or HARNESS2_BENCH_DEPS) instead of the repository, and each run's root comes from the suite context. The plugin keeps its env-driven configuration (HARNESS2_SOLVER_MODEL and friends) so the paper setup keeps working; runner assets ship as package data, so a wheel install works outside a checkout. Co-authored-by: Claude Opus 5.5 <[email protected]>
Improver, selector and reflector sessions now run on the claude, codex or opencode CLI found on PATH (or a configured binary), each with the environment its provider profile produces and an isolated config dir, so neither Vertex AI nor the user's global agent setup is assumed. A "human" channel lets the CLI answer selections in the terminal, and the reflector can run on its own channel. core/events.py appends one JSON line per rollout, session, selection and task to the run's events.jsonl and enforces --budget-usd and --max-calls across resumes: work stops before the next call once a limit is reached, and Ctrl-C stops the same way. core.configure() sets phase width, transport attempts and session concurrency at run time. Persisted run-directory paths are stored relative to the run root and the session cache keys on paths relative to the audit dir, so a run directory can be moved and still resumes without repeating work. The benchmarks plugin passes its pinned OpenCode command and Vertex provider through the new ChannelSpec fields directly. Co-authored-by: Claude Opus 5.5 <[email protected]>
…#5) Adds two executors next to Codex and OpenCode: - claude: runs the Claude Code CLI with an isolated CLAUDE_CONFIG_DIR. Its harness is CLAUDE.md, .claude/skills, .claude/scripts, subagents installed as .claude/agents/<name>.md and hooks merged into .claude/settings.json; stream-json output renders into the shared trajectory grammar. - command: runs any agent through a user-supplied command template; its editable surface is a list of glob patterns. Each executor can now import a harness from a project's native layout, strip it from a workspace, install a candidate into one, and export an adopted harness back, touching only the files its surface manages (and refusing symlinked paths). Codex harness scripts move to .agents/scripts so they never collide with a project's scripts/ directory. Executors resolve their CLI from PATH by default (or a configured binary) instead of pinned downloads, and the bridge serves any provider profile as a Responses endpoint instead of Vertex only. Co-authored-by: Claude Opus 5.5 <[email protected]>
Replaces the seven harness2-* scripts with one `harness2` command (`python -m harness2` works too) whose global options -C, --profile, -c KEY=VALUE, -q, -v and --json apply to every subcommand: run, doctor, setup, judge, report, runs, diff, task, config, demo, bridge, completion, plus init, status, chat and apply (stubs for now). The old script names stay as aliases. A run lives in <project>/.harness2/runs/<name>/ and records run.json: the identity settings, the task stream and the schedule. Resuming with different identity settings is refused with the differing keys, and judge, report, status, diff and apply find everything from the run name (default: the most recent run) instead of repeating flags. `run --dry-run` prints the task stream and the number of rollouts and model sessions the run will make; --budget-usd and --max-calls stop a run cleanly (exit 4) and Ctrl-C stops it for a later resume (exit 130). judged.json records the judge model it was graded with. harness2/config.py is gone: its generic parts were replaced by settings.py, its benchmark parts moved to the plugin. Co-authored-by: Claude Opus 5.5 <[email protected]>
harness2.suite reads tasks from .harness2/tasks/<id>/ (task.md plus an
optional task.toml, files/, check/ and rubric.md) and runs them on the
claude, codex or command executor:
- project tasks run in a throwaway git worktree at `ref`, with the
project's own harness and .harness2/ removed and the candidate harness
installed; the result is the patch the agent made, harness files
excluded;
- files tasks run in a scratch directory seeded from files/, and the
result is what the agent saved in output/.
The base harness is imported from the project once per run and kept, so
a resumed run compares like with like. Rollouts are idempotent by run
id and record exit code, time and reported cost.
Grading is optional and never used for refinement: a `check` command is
run against the result in a fresh workspace (exit status, or a JSON
{n_passed, n_criteria} line), or an LLM judge applies rubric.md. Cached
grades are reused only while the check, rubric and judge model are
unchanged. Also adds task listing, scaffolding and JSONL/CSV import,
and the improver prompt profiles for repository and files work.
Co-authored-by: Claude Opus 5.5 <[email protected]>
Drives `harness2` in-process against a temporary git project with a CLAUDE.md, a skill and a .claude/settings.json, using a fake `claude` that plays both executor and improver. Covers a project task graded by a check command and a files task, runs at k=1 and k=2, judge, report, diff and adoption back into the project (the unrelated settings key and the skill survive), and a resumed run that makes no new model calls. Co-authored-by: Claude Opus 5.5 <[email protected]>
- `harness2 init` asks which agent CLI runs the tasks (detecting claude and codex on PATH), which models to use and where the harness lives, writes a commented .harness2/config.toml, a .gitignore and the tasks directory, and says whether the needed API key is visible. --yes takes the defaults. - `harness2 status [--watch]` shows each task's state, the harness it picked, rollouts and spend, from run.json and events.jsonl only, so it works during a run, after it stopped and from another terminal. - `harness2 apply` applies only what the run changed relative to the harness it started from, so edits made to the project since are kept; a file changed on both sides is skipped unless --force. It lists the changes first and asks, or reviews file by file with -i, refuses to write over uncommitted harness files, and can create a branch. - `harness2 chat` starts the user's own Claude Code or Codex in the project with a guide to harness2 and the project's tasks and runs. - `harness2 run --selector human` hands the final pick to the person at the terminal, showing the anonymised candidates. Every subcommand now has help text, and run, doctor, config, task and judge print readable results (doctor also flags a model whose provider its CLI cannot use). Untracks the bytecode and egg-info that slipped into the repository and adds a .gitignore. Co-authored-by: Claude Opus 5.5 <[email protected]>
Rewrites the README around tuning your own agent's harness: what it does, install with uv/pipx, a quick start from `harness2 init` to `harness2 apply`, the command list, configuration, other agents through the command executor, and where the paper's benchmarks went. Adds docs/how-it-works.md (the method, what a run costs and stores), docs/tasks.md (task layout, workspaces, deliverables, grading, import), docs/configuration.md (settings layers, every key, providers per CLI, profiles, secrets) and docs/cli.md (every command and flag, exit codes, the run directory). CONTRIBUTING.md now describes this fork's setup, layout and conventions; the benchmarks plugin gets a README, and its setup guide no longer points at a missing .env.example or the old Vertex-only bridge. Also orders `harness2 --help` the way the commands are used, makes `task new` say what to do next, and fills in the package metadata. Co-authored-by: Claude Opus 5.5 <[email protected]>
Harness² adopts one harness per task. `harness2 consolidate` gives the improver the run's starting harness, every task's adopted harness, its request and the lesson the run recorded, and asks for one harness for the project's next tasks; the result is validated against the executor's surface (one retry with the problems listed) and stored in <run>/consolidated/<mode>/ with the improver's notes. `harness2 eval` runs the starting and the consolidated harness on the run's gradable tasks (optionally repeated), grades both and prints the per-task and mean pass rates with the difference. `apply` and `diff` now default to the consolidated harness when there is one; `--task` still picks a single task's adopted harness. Co-authored-by: Claude Opus 5.5 <[email protected]>
…#13) - The distribution is now harness2-cli (the command and the import package stay `harness2`); the benchmarks plugin depends on it, and the project URLs point at 3ae3ae/harness2-cli with the upstream linked. - GitHub Actions runs the core and plugin tests on Linux and macOS with Python 3.12 and 3.13, plus ruff check and ruff format --check. - `harness2 init` asks whether the models are reached through the agent CLI's own login (a Claude or ChatGPT subscription, the default) or an API key, and says whether that CLI is logged in; --auth answers it. - `harness2 runs rm` deletes a run after asking (--yes to skip), and `runs ls` / `runs show` print tables instead of raw values. Co-authored-by: Claude Opus 5.5 <[email protected]>
…LIs (#14) Two built-in providers, claude-login and chatgpt-login, use the login the installed `claude` / `codex` already has, so a Claude or ChatGPT subscription works without an API key: - claude-login keeps the user's own config dir (where the login lives) and adds --setting-sources project,local, --strict-mcp-config and --no-session-persistence, so personal CLAUDE.md, skills, plugins, MCP servers and session history stay out of the runs; - chatgpt-login keeps the user's own CODEX_HOME and adds --ignore-user-config and --ignore-rules (doctor notes when a global AGENTS.md is still part of every Codex run). API-key providers keep their fully isolated per-run config dirs. Doctor checks `claude auth status` / `codex login status` and rejects a login provider on a CLI that cannot use it. Hitting a provider's usage limit now stops the run cleanly instead of recording failed work: the interrupted rollout is discarded and the same command resumes later. Project worktrees left by killed runs are pruned before new ones are created. Co-authored-by: Claude Opus 5.5 <[email protected]>
A run now ends by combining the harnesses it adopted per task, with the lessons it recorded, into one final harness per recursion mode; that is what `apply`, `diff` and `eval` use, so there is a single result to review and apply. The separate `consolidate` command is gone. If the final step fails, the run's exit code is unchanged and rerunning the same command retries just that step (every earlier step is reused). The dry-run estimate counts the extra session, and a run stopped by a provider's usage limit says to rerun once the limit resets. Co-authored-by: Claude Opus 5.5 <[email protected]>
`harness2 guide` prints how an AI agent should drive harness2 on a person's behalf: ask before spending, bound runs with --max-calls, read --json, answer prompts with flags, never edit runs or harness files directly, the init/task/doctor/run --detach/status/stop/diff/eval/apply workflow, where things are stored and what the exit codes mean, followed by the current project's settings, tasks and runs. `guide --install claude` installs the same text as a user-level Claude Code skill, which harness2's own runs never load. `harness2 chat` now builds on it. `init --json` and `apply --json` print a result object on stdout and keep their human-readable text on stderr; `apply --json` refuses to ask for confirmation and needs --yes or --dry-run. Co-authored-by: Claude Opus 5.5 <[email protected]>
- `harness2 run --detach` starts the same run as a background process,
logging to <run>/run.log, and returns its name, pid and log path at
once (an immediate failure is reported with the log's tail).
- A running run records itself in <run>/run.pid; a second start of the
same run is refused, and `harness2 stop [NAME]` asks it to stop: the
calls in flight finish, the run exits, and rerunning resumes it. Only
a detached run's own process group is signalled, never the shell or
agent that launched a foreground run.
- With --json, a failing command prints {"error", "exit_code"} on
stdout instead of an error line; `diff --json` returns the changes
and per-file diffs.
- `apply -i` and `chat` refuse to start without a terminal instead of
waiting for input, and the agent guide says how to change init's
choices with `config set`.
Co-authored-by: Claude Opus 5.5 <[email protected]>
* docs: describe Harness² CLI as the general-purpose version of Harness² The README now presents the fork as Harness² CLI: what it adds to the original release (a comparison table), the run -> diff -> eval -> apply flow around one final harness, subscriptions as the default way to reach models, and how AI agents drive it (--json, --detach, stop, guide). The configuration guide documents claude-login and chatgpt-login, what is and is not isolated, and usage limits; the command reference covers stop, eval, guide, runs rm, --detach, JSON errors and the new run files; How it works explains the final harness step that the CLI adds to the paper's method. Adds a changelog. Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(status): say whether a run is still going `harness2 status` (and its --json) now reports whether the run's process is alive and its pid, and says when a run stopped without finishing, for example after being killed, so the user knows to resume it. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
A live run with Codex on a ChatGPT subscription (one task, k=1, nine model calls, all without an API key) went through run, status, diff, judge, eval and apply. Fixes from reading its output: - Codex and subscriptions report no money, so run and status say "spend not reported" instead of "$0.00 spent", which read as free. - eval and apply call the run's result the final harness, as the docs do, and eval's table is aligned. - init --yes no longer prints stray blank lines. Co-authored-by: Claude Opus 5.5 <[email protected]>
The previous commit renamed eval's column to FINAL and aligned the table, but three assertions in tests/test_consolidate.py still expected the old text, so the suite failed on general-cli. They now match the current output, with the column spacing left flexible. Co-authored-by: Claude Opus 5.5 <[email protected]>
GitHub's Ubuntu runners have no zsh, so the completion test failed there on /bin/zsh. It now finds bash and zsh on PATH and only checks the generated script's text for a shell that is missing. Co-authored-by: Claude Opus 5.5 <[email protected]>
3ae3ae
changed the title
…#22) Codex has no flag to skip a personal ~/.codex/hooks.json or ~/.codex/AGENTS.md: with --ignore-user-config it still runs and reads them (verified with marker-writing hooks), so every chatgpt-login rollout and improver session fired the user's own hooks. Claude Code's flags, by contrast, do keep user hooks and plugins out (verified the same way). chatgpt-login now runs Codex on a home kept for harness2, ~/.local/state/harness2/codex-home, which the user signs in to ChatGPT once with the new `harness2 login codex`; that is a separate login, so the user's own ~/.codex is never read or changed and its refresh token is never used. `providers.chatgpt-login.home` points runs at another home instead, and doctor then lists the personal files every run will include. `harness2 login` shows whether both subscription logins are ready, and init and the agent guide point at it. Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR turns the fork into Harness² CLI, a general-purpose version of google-research/harness2. Its main use is to refine the harness of the coding agent you already use (Claude Code's
CLAUDE.md/.claude/, or Codex'sAGENTS.md/.agents/) on tasks from your own project.The method and its prompts are the paper's; everything around them is new. The work came in through these PRs into
general-cli:commandsurfaces; Codex scripts move to.agents/scriptsharness2-benchmarkspluginevents.jsonl,--budget-usd/--max-calls, resume, relocatable runscommandexecutors; native harness import/export; system CLIs by defaultharness2command with subcommands, run names, dry-run call estimatestaskssuite: git-worktree and files tasks, optionalcheck/rubric gradinginitwizard,status --watch, three-wayapply,chat,--selector humanrunfrom every task's adopted harness and lesson;evalcompares it with the starting harnessharness2-cli; CI (Linux/macOS × Python 3.12/3.13 + ruff);runs rmclaude-login/chatgpt-login: subscriptions through the installed CLIs; usage limits stop runs cleanlyguide,--jsonerrors,run --detach,stop, no blocking promptsharness2 login codex), because Codex runs a personal~/.codex/hooks.jsoneven with--ignore-user-config; Claude Code's user hooks were verified not to runDeliberately left out: sandboxing and HTML reports.
Test plan
codex execcall saw an API key.run --detach,status,diff,judge(100%),evalandapply --dry-run.~/.claude/CLAUDE.md, user skills and the Codexconfig.tomlare not sent in runs.harness2 demo.claude-loginwas not done; only Codex was tested live.🤖 Generated with Claude Code