diff --git a/.agents/skills/dual-review/SKILL.md b/.agents/skills/dual-review/SKILL.md index ce3ff62a..4bfe7138 100644 --- a/.agents/skills/dual-review/SKILL.md +++ b/.agents/skills/dual-review/SKILL.md @@ -1,6 +1,6 @@ --- name: dual-review -description: Review a PR with two independent reviewers (this session and a fresh Codex gpt-6.1-sol at high effort), merge and verify their findings, fix, and repeat until the PR is ready for merge; minor points are recorded, not fixed. Ends with the merge verdict, the changes made and every unresolved finding with its reason. Invoke with /dual-review [repo]. +description: Review a PR with two independent reviewers (this session and a fresh Codex gpt-6.2-sol at high effort), merge and verify their findings, fix, and repeat until the PR is ready for merge; minor points are recorded, not fixed. Ends with the merge verdict, the changes made and every unresolved finding with its reason. Invoke with /dual-review [repo]. user-invocable: true --- @@ -14,7 +14,7 @@ left alone, since fixing them costs rounds and tends to break real paths. It sto that blocks the merge. Run it in a Claude Opus 5.5 session at high effort (`/model`, `/effort`); say so at the start if the session is on -something else. The independent reviewer is always Codex with `gpt-6.1-sol` at `model_reasoning_effort="high"`. +something else. The independent reviewer is always Codex with `gpt-6.2-sol` at `model_reasoning_effort="high"`. ## Inputs @@ -45,7 +45,7 @@ status) so it does not re-report them. Then run, with `run_in_background: true` `W` the worktree): ```bash -codex exec --skip-git-repo-check -C "$W" -s read-only -m gpt-6.1-sol -c model_reasoning_effort="high" \ +codex exec --skip-git-repo-check -C "$W" -s read-only -m gpt-6.2-sol -c model_reasoning_effort="high" \ -o "$S/dual-review--r-codex.md" - < "$S/dual-review--r-prompt.md" \ > "$S/dual-review--r-codex.log" 2>&1 ``` diff --git a/.agents/skills/prepare-pr/SKILL.md b/.agents/skills/prepare-pr/SKILL.md index 5f092e4f..2dbe4fbf 100644 --- a/.agents/skills/prepare-pr/SKILL.md +++ b/.agents/skills/prepare-pr/SKILL.md @@ -23,6 +23,8 @@ affected interfaces, and the validation evidence available. [test checklist](references/test-checklist.md). - For PRs that add or change a method or a dataset and start a process or change what a sandbox shows, also complete the [sandbox checklist](references/sandbox-checklist.md). +- For PRs that add a benchmark or change how a benchmark's tasks are written or read, + also complete the [benchmark checklist](references/benchmark-checklist.md). Keep checklist questions in their owning files. Answer shared items once, using actual code, checks and records rather than planned work. diff --git a/.agents/skills/prepare-pr/references/benchmark-checklist.md b/.agents/skills/prepare-pr/references/benchmark-checklist.md new file mode 100644 index 00000000..9f947c4f --- /dev/null +++ b/.agents/skills/prepare-pr/references/benchmark-checklist.md @@ -0,0 +1,28 @@ +# Benchmark checklist + +Answer these items alongside the [general checklist](general-pr-checklist.md) when a pull request adds a benchmark +or changes how a benchmark's tasks are written or read. The contract is in +[Task directories](../../../../docs/framework/task.md#task-directories) and +[Benchmarks](../../../../docs/framework/task.md#benchmarks); how a task runs in Harbor is in +[How Harbor runs a task](../../../../docs/developer/harbor-design.md). + +- [ ] **Task directories.** Are the tasks task directories in Harbor's format, rather than a new row format? + `hermit.task.directory.write_task` is the standard helper for a `prepare.py`; `python -m hermit.harbor.dataset + prepare ` prepares the Harbor dataset its `dataset.ref` names (give the reason for each + `dataset.patch` configured). A task set whose tasks each + need an image of their own runs in Harbor's environments with `hermit.task.verify`, never a runner of its own. +- [ ] **Public and private halves.** Do the rows the agent and the learner read hold only the public half (the + instruction, its settings, its metadata), with answers, tests and the verifier in the private half or in + judge-owned storage? Name each field a `data.view` adds and why the learner may see it, and what private + information the judge's feedback may show under the benchmark's protocol. +- [ ] **What a task asks.** Does each task's `task.toml` declare the network (`network_mode`, `allowed_hosts`), + memory and timeouts its agent and its verifier need? Hermit's bubblewrap task sandbox refuses a task whose network + or memory the run cannot give; on the host, with `environment.image`, or in Harbor's environments (where the run's + overrides apply: `environment.harbor.environment.override_memory_mb`, ...), state how the configs meet them. On + every backend, state the effective timeouts: a run's own (`agent.timeout_min`, + `environment.harbor.agent_timeout_sec`) may be shorter than a task's. +- [ ] **Parity, for a moved benchmark.** For a benchmark moved from rows to task directories, do its tasks read back + through each of its configs as exactly the old rows (prompts, learning material, splits)? State how this was + compared, with the counts. +- [ ] **A real run.** Link one smoke run through `hermit eval` (and `hermit train` when a method is part of the + change), with its scores and no infrastructure errors. diff --git a/.agents/skills/prepare-pr/references/sandbox-checklist.md b/.agents/skills/prepare-pr/references/sandbox-checklist.md index 26ab62fd..13fbb25b 100644 --- a/.agents/skills/prepare-pr/references/sandbox-checklist.md +++ b/.agents/skills/prepare-pr/references/sandbox-checklist.md @@ -5,11 +5,17 @@ changes a method or a dataset starts a process or changes what a sandbox shows. [How the sandbox works](../../../../docs/guides/sandbox.md). - [ ] **Reuse the framework's sandbox.** Does every process the change starts (agent, judge, learner and its roles, - tools, the adapter's code) run through `hermit.sandbox` or `hermit/isolate.py`, with its needs declared in the - existing settings (`environment.*`, `judge.*`, the method's view, `sandbox.evolver.*`)? Link one - `sandbox/-.json` from a run of the change. For each process that cannot use this abstraction, state why - and how it is confined instead. + tools, the adapter's code) run in a confined environment the framework gives (`hermit.environment`: `Sandboxed`, + `Container`, or a task's Harbor environment) or through `hermit.sandbox` / `hermit/isolate.py`, with its needs + declared in the existing settings (`environment.*`, `judge.*`, a task's `task.toml`, the method's view, + `sandbox.evolver.*`)? `Local` is not confinement: it runs on this machine as it is. Link the evidence from a run of + the change that the confinement was enforced: a launch record (`sandbox/-.json` whose isolation probe + passed, `probe.passed: true`), or, where the framework keeps no launch record (a container, a Harbor trial with its + `trial_dir` in the rollout's verdict, the strict adapter sandbox), a check run inside it that shows what it can read, + write and reach. For each process that runs otherwise (`Local` + included), state why and how it is confined instead, or, for trusted code on the host (a judge that only reads a + verdict), exactly what it does. - [ ] **Sandbox changes are core changes.** Does the pull request change the sandbox itself (`hermit/sandbox.py`, - `hermit/sandbox_relay.py`, `hermit/isolate.py`, the Trainer's evolver sandbox, or their defaults)? If so, link the - issue where the change was agreed with the core team, and answer it under **Core changes** in the general - checklist. + `hermit/sandbox_relay.py`, `hermit/isolate.py`, `hermit/environment/`, the Trainer's evolver sandbox, or their + defaults)? If so, link the issue where the change was agreed with the core team, and answer it under **Core + changes** in the general checklist. diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c142530..0acd4a49 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -423,6 +423,14 @@ Changes are grouped by software version. Entries under Unreleased describe work ### Fixed +- In a task's Harbor environment, a CLI's output kept in `/logs/agent` as it ran reached a shared verifier unscrubbed + when Harbor's agent timeout ended the phase before Hermit's own logs were copied over it. The phase's end now scrubs + each such stream there of the credentials passed to the agent's commands (their variables and descriptors) before + the verifier runs, in a local Docker environment, whose `/logs/agent` is the trial's folder on this machine (best + effort: a stream this machine cannot read stays as it came). Elsewhere a shared verifier still gets it as it came; + the trial's files are scrubbed after the trial. + A multi-step task (`[[steps]]`) was refused as an environment that failed to start and run again after + `infra_wait_s`; it is now refused as unsupported (`unsupported_task`) and never run again. - `Sandbox.wrap` ran a virtual environment's interpreter (`/bin/python`, a link to its base interpreter) by its real path, so it started without the environment and its packages failed to import. An interpreter whose environment is granted (the sandbox's `readable` or `writable`, the call's, or `interpreters`) now runs by its own diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 075b8b5b..e78538c4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -59,6 +59,11 @@ also answers the [sandbox checklist](.agents/skills/prepare-pr/references/sandbo framework's sandbox, states why for anything that cannot, and treats a change to the sandbox itself as a core change agreed with the core team first. +A pull request that adds a benchmark or changes how its tasks are written or read also answers the +[benchmark checklist](.agents/skills/prepare-pr/references/benchmark-checklist.md): task directories in Harbor's +format, answers and tests kept out of the tasks the agent and the learner read (the judge's feedback shows what the +benchmark's protocol allows), what each task asks of its environment declared, and parity for a moved benchmark. + GitHub Actions (`.github/workflows/tests.yml`) runs `python -m pytest --changed` against the base branch on every pull request, and the complete suite on main, daily and on demand, on Ubuntu with bubblewrap. Tests whose dependencies it does not install (agent CLIs, AgentDyn, prepared benchmark data) skip there, as they do locally. diff --git a/docs/_includes/harbor-agents.html b/docs/_includes/harbor-agents.html new file mode 100644 index 00000000..bb189aa7 --- /dev/null +++ b/docs/_includes/harbor-agents.html @@ -0,0 +1,45 @@ +
+
+ + + + + +CLI agent: Codex, Claude Code + +Hermit +renders its home, prompt +reads its stream, logs + +set up, then run + +its files at the end + +task's container + +CLI session + +ls · git · pytest · edits +run by the CLI itself +none through the bridge + + +model +calls +Loop agent: a ReactAgent, the answer-only agent + +Hermit +the agent's loop is here +Hermit calls the model + + + +1 exec per command + +task's container +runs the one command it is sent, returns the result +once for each command the agent runs + +
+
A CLI is an ordinary process in the container, like a terminal opened there; the bridge only sets it up, starts it and brings back what it left. A loop agent's commands each pass through the bridge.
+
diff --git a/docs/_includes/harbor-processes.html b/docs/_includes/harbor-processes.html new file mode 100644 index 00000000..bcaf12e4 --- /dev/null +++ b/docs/_includes/harbor-processes.html @@ -0,0 +1,57 @@ +
+
+ + + + + + + +Hermit process + +acting agent +starts a CLI, or runs a loop + + +HarborEnvironment +the Environment API + +judge +hermit.task.verify + + +driver · Harbor's python + +Trial.run() + +HermitBridge +relays; no logic of its own +pull or build · setup +agent phase · verifier +stop · result.json · scrub + + +task's container + +agent's commands +a loop agent's commands, +or one CLI session +running its own + +tests/test.sh +only for the verifier + + +JSON lines +stdin, stdout + +exec · upload +download + +verifier + +its verdict + +
+
Purple is Hermit's, teal is Harbor's. The bridge is the agent Harbor's trial runs; it carries Hermit's agent's commands into the container and decides nothing itself. The verifier is shown in shared mode, in the agent's container.
+
diff --git a/docs/_includes/harbor-trial.html b/docs/_includes/harbor-trial.html new file mode 100644 index 00000000..ccd65457 --- /dev/null +++ b/docs/_includes/harbor-trial.html @@ -0,0 +1,53 @@ +
+
+ + + + + + +Hermit process + +driver · Trial.run() + +task's container + + + + + +① start, after the disk gate + + +② pull or build, healthcheck + +③ bridge setup(): install what the agent needs +④ agent phase: its network, its user, its clock + + +⑤ start answered: the instruction, the workspace + + +⑥ Hermit's agent works; each of its commands goes through the bridge + +exec / upload / download + +exec as the agent user; transfers by Harbor's copy + + +exec: exit code, output + + +⑦ done (or Harbor's agent timeout) +bridge: kill what still runs, remove sign-ins, scrub, copy logs + + +⑧ tests in, verifier, reward; stop; scrub + + +⑨ rewards, exception (result.json) +Hermit: the outcome, retries, usage; the judge scores + +
+
A trial with a shared verifier, in the order of Harbor's Trial.run(). Hermit's agent works only in ⑥, where each command it runs is one exec relayed by the bridge.
+
diff --git a/docs/_static/css/harbor-diagrams.css b/docs/_static/css/harbor-diagrams.css new file mode 100644 index 00000000..138de274 --- /dev/null +++ b/docs/_static/css/harbor-diagrams.css @@ -0,0 +1,48 @@ +/* Harbor diagrams (docs/_includes/harbor-*.html): purple for Hermit, teal for Harbor, neutral for the container. */ +.ha-hb { + --hb-text: #1d2430; + --hb-muted: #5d6676; + --hb-line: #c9d0da; + --hb-surface: #fff; + --hb-hermit: #8a4f9e; + --hb-hermit-soft: #f3e9f6; + --hb-harbor: #1f6f8b; + --hb-harbor-soft: #e3f0f5; + --hb-box: #f4f6f8; + margin: 1.5rem 0; + color: var(--hb-text); +} +html.dark .ha-hb { + --hb-text: #e4e8ee; + --hb-muted: #9aa4b2; + --hb-line: #3a4452; + --hb-surface: #1a2028; + --hb-hermit: #c79bd8; + --hb-hermit-soft: #2e2236; + --hb-harbor: #5fb3cf; + --hb-harbor-soft: #16313b; + --hb-box: #212833; +} +.ha-hb-scroll { overflow-x: auto; } +.ha-hb svg { display: block; width: 100%; min-width: 640px; height: auto; } +.ha-hb figcaption { margin-top: 0.5rem; font-size: 0.875em; color: var(--hb-muted); } +.ha-hb text { fill: var(--hb-text); font-size: 12.5px; } +.ha-hb .t-h { font-size: 14px; font-weight: 700; } +.ha-hb .t-b { font-weight: 600; } +.ha-hb .t-m { fill: var(--hb-muted); font-size: 11.5px; } +.ha-hb .mono { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: 11.5px; } +.ha-hb .c-hermit { fill: var(--hb-hermit); } +.ha-hb .c-harbor { fill: var(--hb-harbor); } +.ha-hb .b-hermit { fill: var(--hb-hermit-soft); stroke: var(--hb-hermit); stroke-width: 1.6; } +.ha-hb .b-harbor { fill: var(--hb-harbor-soft); stroke: var(--hb-harbor); stroke-width: 1.6; } +.ha-hb .b-box { fill: var(--hb-box); stroke: var(--hb-line); stroke-width: 1.4; } +.ha-hb .b-inner { fill: var(--hb-surface); stroke: var(--hb-line); stroke-width: 1.2; } +.ha-hb .b-loop { fill: none; stroke: var(--hb-hermit); stroke-width: 1.3; stroke-dasharray: 5 4; } +.ha-hb .lane { stroke: var(--hb-line); stroke-width: 1.2; stroke-dasharray: 4 4; } +.ha-hb .a { fill: none; stroke-width: 1.6; } +.ha-hb .a-hermit { stroke: var(--hb-hermit); } +.ha-hb .a-harbor { stroke: var(--hb-harbor); } +.ha-hb .a-plain { stroke: var(--hb-muted); } +.ha-hb .ah-hermit { fill: var(--hb-hermit); } +.ha-hb .ah-harbor { fill: var(--hb-harbor); } +.ha-hb .ah-plain { fill: var(--hb-muted); } diff --git a/docs/conf.py b/docs/conf.py index d22d2c0f..c3b9f767 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -38,7 +38,8 @@ html_title = f"{project} {version}" html_baseurl = "https://infini-ai-lab.github.io/hermit/docs/" html_static_path = ["_static", "../assets"] -html_css_files = ["css/custom.css", "css/overview-diagrams.css", "css/agent-layers.css", "css/sandbox-diagrams.css"] +html_css_files = ["css/custom.css", "css/overview-diagrams.css", "css/agent-layers.css", "css/sandbox-diagrams.css", + "css/harbor-diagrams.css"] html_js_files = ["js/component-explorer.js", "js/overview-diagrams.js", "js/faq.js", "js/agent-layers.js"] html_favicon = "../assets/logo.svg" html_theme_options = { diff --git a/docs/developer/harbor-design.md b/docs/developer/harbor-design.md new file mode 100644 index 00000000..671210fb --- /dev/null +++ b/docs/developer/harbor-design.md @@ -0,0 +1,134 @@ +# How Harbor runs a task + +This page records how a task in its own Harbor environment runs: which process does what, and where an agent's +command goes. It shows the common path (a local Docker environment, a shared verifier, a built-in agent); the +contract, with every setting, outcome and refusal, is in [Benchmarks](../framework/task.md#benchmarks). + +## In one line + +Harbor runs the whole trial with its own `Trial.run()` (the environment, the setup, the agent phase, the verifier, the +teardown); the agent in that trial is a bridge that relays the commands of Hermit's agent. Every agent stays +Hermit's: it is written against the [Environment API](../framework/agent-layers.md#environments) (`exec`, `upload`, +`download`) and knows nothing of Harbor. Only a task set in Harbor's format, run in Harbor's environments, goes +through the bridge; an agent that works through the environment its call is given needs nothing for Harbor (a plain +`run(task, adapter)` function, which takes no environment, cannot run there). + +## The processes + +```{raw} html +:file: ../_includes/harbor-processes.html +``` + +Harbor runs in a venv of its own (Python 3.13 and `~/.cache/hermit/harbor-venv` in Hermit's setup; the python beside +the configured `harbor` command). Each environment is one driver process under that python +(`hermit/environment/harbor_driver.py`), which imports Harbor and what it depends on, never Hermit, and +speaks JSON lines with Hermit on its stdin and stdout (`hermit/environment/harbor.py`). The driver closes the trial +when its stdin closes; a driver that had to be killed has its trial's compose projects taken down by name (unless +`environment.harbor.environment.kwargs.keep_containers` is set). + +## Who does what + +| Step | Who | Where | +|---|---|---| +| Wait for disk room (the host's /tmp, Docker's data root; with `image_sizes`, room kept for the images still to pull) | Hermit | `harbor_docker.py` | +| Check the task's capabilities, then start the environment: the prebuilt image pulled or the Dockerfile built, the healthcheck | Harbor | `Trial.run()` | +| The agent's setup: install what a built-in agent declares (`Agent.installs`), within `setup_timeout_sec` (1200 s) | Harbor, on the bridge's `setup()` | Harbor's installers | +| The agent phase: its network policy, its user, its timeout | Harbor | `Trial.run()` | +| The work in the agent phase | Hermit's agent, through the bridge | the agent's own code | +| End the agent phase: kill what still runs, remove sign-ins, copy the logs | the bridge | `harbor_driver.py` | +| The verifier: the tests uploaded only now (shared mode), run, the reward read | Harbor | `Trial.run()` | +| Stop the environment, write `result.json`, scrub the credentials the trial knows from its files | Harbor | `Trial.run()` | +| Scrub the credentials passed through the bridge from the trial's files, then reply | the driver | `harbor_driver.py` | +| Read the outcome: the agent's result or infra | Hermit | `harbor_errors.py` | +| Remove the task's prebuilt image as `docker_cleanup` says (best effort) | Hermit | `harbor_docker.py` | +| Run again, mark unscored, or score from the verifier's verdict | Hermit, its judge | `hermit/trainer/rollout.py`, `hermit.task.verify` | + +The trial's steps and their order are Harbor's. Hermit adds the warm start around the trial (the disk gate and the +image cleanup, which Hermit's earlier Harbor runner had) and reads the outcome as that runner read it. With +`[verifier] environment_mode = "separate"`, Harbor stops the agent's environment and verifies in a fresh one, which +gets Harbor's collected artifacts and its own mounts (not the agent's workspace or `/logs/agent`), and whose image may +bring the tests itself. + +## A trial and a command's path + +```{raw} html +:file: ../_includes/harbor-trial.html +``` + +In ⑥ the bridge does not interpret a command; it runs it and returns the result. In a local Docker environment a +variable the agent passes reaches the container by name, never on a command line of this machine (another environment +type passes it as Harbor's own code does). Each command runs in a process group of its own, so the bridge can end the +ones still running when the phase ends. `upload` and `download` take the same path, one transfer at a time, as +Harbor's file copies (an upload is given to the agent user). Commands are accepted only during +the agent phase. + +## Two kinds of agent + +The commands a CLI's model asks for do not pass through the bridge: the CLI runs in the container, like a terminal +opened there. + +```{raw} html +:file: ../_includes/harbor-agents.html +``` + +| | CLI agent (Codex, Claude Code) | Loop agent (a `ReactAgent`, the answer-only agent) | +|---|---|---| +| Where its loop runs | in the container (the CLI) | in Hermit's process | +| Who calls the model | the CLI, from the container | Hermit | +| What passes through the bridge | a few `exec`s that find the CLI and lay out its home, one that runs the session, the transfers around it | one `exec` for each command, and transfers (the answer-only agent: its answer) | +| How it gets into the container | installed by Harbor's setup (`Agent.installs`; python3 for an adapter's hooks, best effort), unless the image already has the version asked for or the environment mounts a binary of this machine the agent user can run at that version (`agent..bin`) | nothing to install | +| Network | the model's host, which it calls from the container | the model is called from this machine | + +For either kind, the run adds the agent's model hosts to the hosts a task without a public network lets the agent +phase reach (`allowed_hosts`); a task whose tools need more declares it in its `task.toml`. + +Whether an agent is a CLI agent follows its Base Agent, not the benchmark: an agent built on a CLI Base Agent runs as +a CLI session whatever its Layer 3 adds; one built on `ReactAgent` is a loop agent. For a CLI session, Hermit renders +its home here and uploads it (to `/installed-agent/hermit-home` unless `remote_home` says otherwise), starts the CLI +with the prompt on its stdin, reads its stream (the reply, the steps, the usage) from the `exec`'s output as it reads +any session of the CLI, and downloads what the session left: Codex's last-message file (Claude Code's reply is in +its stream) and, with a ChatGPT sign-in, the session files and the refreshed sign-in. + +## How the agent phase ends + +The phase ends when Hermit's agent is done, or when Harbor's agent timeout ends it. The agent's own budget ends 60 s +before Harbor's (10% of a short one): time kept for its session's end and logs to reach the trial. Then the bridge, +as the earlier runner's agents did: + +1. kills the process group of each command it started that still runs; what a finished command left running (a + server the task asked for) stays; +2. removes the credential files the agent registered (a CLI's sign-in); +3. scrubs a CLI's stream, written to `/logs/agent` as it ran, in place, best effort (in a local Docker environment, + whose `/logs/agent` is the trial's folder on this machine; elsewhere a shared verifier gets it as it came), then + copies the log directory Hermit's agent supplied with `done`, if it came in time, scrubbed, to `/logs/agent`, which a + shared verifier may read. + +If a kill, a removal or the copy fails, the trial is infra and its verifier's result is not the agent's; a stream +this machine cannot read is left as it came. Otherwise Harbor runs the +verifier on what the agent left whenever the session's end is the agent's result (a timeout, a CLI that exited with +an error). Otherwise the verifier is skipped: an adapter over Codex's context cap is the agent's 0; a CLI that never +started, or a session the provider could not serve, is infra. + +## Outcomes + +| Outcome | Examples | What happens | +|---|---|---| +| The agent's result | the verifier's reward; a verifier that found no reward file (0, when no infra signal explains it) or ran out of time on the agent's solution (0, unless `verifier_timeout: infra`); a CLI that exited with an error; an adapter that failed before any session | scored, 0 included | +| Infra | the environment did not start; a verdict with neither a reward (under `reward_key`) nor an exception; a verifier whose tools never arrived or whose disk filled; a session that never started; a provider that could not serve | run again in a fresh environment, up to `infra_retries` (2), each after `infra_wait_s` (300 s); a session a CLI's provider could not serve under `busy_retries` (3, after 120 s × n) instead; a cause that runs no better again (the disk gate, a refused configuration, a task Harbor refuses as `unsupported_task`) is not run again | +| Still infra after the retries | | `unscored`: the judge refuses to score it; `hermit eval` counts it in `errors` and as 0 in `mean_score`; `python -m hermit.harbor.report` gives pass@1 over the agent's results only (with `pass_reward` set) | + +A lost sign-in or a spent quota ends the run; a session a subscription plan refused runs again once the plan allows, +outside these budgets. + +Infra is never the agent's 0. The classification is the earlier runner's, ported +(`hermit/environment/harbor_errors.py`). + +## Why this shape + +A first design ran Harbor's trial steps from Hermit (the network switch, the verifier's order, the user, the +teardown). Each of them had to match Harbor's behavior, and the differences kept turning up in review. With Harbor's +own `Trial.run()` owning the trial, those steps are Harbor's code and cannot drift from it; Hermit keeps the agent +and the outcome code it already had. One bridge serves every agent, so a new agent never touches Harbor. + +What a Harbor environment does not support yet (a multi-step task, the agent's mounts, ...) is refused rather than +dropped: see [Benchmarks](../framework/task.md#benchmarks). diff --git a/docs/developer/index.md b/docs/developer/index.md index e050a0de..1ab2dd3f 100644 --- a/docs/developer/index.md +++ b/docs/developer/index.md @@ -4,7 +4,7 @@ Start with the extension boundary you need. A task integration normally requires | Change | Entry point | Verify | |---|---|---| -| New task, acting agent, or judge | [Bring your task](../tutorials/your-own-task.md), [task contract](../framework/task.md) | No-adapter probe, adapter consumption, split isolation, logs, scoring | +| New task, acting agent, or judge | [Bring your task](../tutorials/your-own-task.md), [task contract](../framework/task.md), [task directories](../framework/task.md#task-directories), [benchmarks](../framework/task.md#benchmarks) | No-adapter probe, adapter consumption, split isolation, answers and tests kept to the judge (a task's private half or judge-owned storage), logs, scoring | | New instruction/skill/tool structure | [Customize an adapter](../tutorials/customize-an-adapter.md), [adapter metadata](../framework/adapter-manifest.md) | Learnable/fixed rules, checker, exported schemas | | New learning method | [Evolver tutorial](../tutorials/custom-evolver.md), [full contract](../framework/custom-evolver.md) | Failure handling, state persistence, usage, final selection | | Reproduction of a published method | [Reproduce a published method](reproducing-a-method.md) | The adapter and the Evolver named, the differences from the authors' code, the protocol, the first evaluation on the shared settings | @@ -77,4 +77,5 @@ component contracts are described in [Adapter](../framework/adapter.md), [Evolve Describe the user-visible behavior, the affected contract, how you verified it, and any environment requirements. Include a small reproducer for failures. Update commands, defaults, and docs together; do not describe a proposed feature as available. The generic [service lifecycle proposal](service-design.md) is kept separately from supported usage. -The [sandbox design](sandbox-design.md) records how the task, judge and evolver sandboxes are built. +The [sandbox design](sandbox-design.md) records how the task, judge and evolver sandboxes are built, and +[How Harbor runs a task](harbor-design.md) how a task in its own Harbor environment runs. diff --git a/docs/framework/task.md b/docs/framework/task.md index e788bf46..19cb23fa 100644 --- a/docs/framework/task.md +++ b/docs/framework/task.md @@ -297,7 +297,8 @@ which is refused. A benchmark enters Hermit as a task set in [Harbor](https://github.com/laude-institute/harbor)'s format (`task.toml`, `instruction.md`, `environment/` with a Dockerfile or a prebuilt `docker_image`, `tests/test.sh` writing -`/logs/verifier/reward.txt`): what Terminal-Bench, SWE-bench and the datasets on the Harbor hub share. A task set that +`/logs/verifier/reward.txt`): what Terminal-Bench, SWE-bench and the datasets on the Harbor hub share (the design, with +diagrams: [How Harbor runs a task](../developer/harbor-design.md)). A task set that needs the environments its tasks name (an image of its own each) runs in Harbor's environments: Harbor gives the task's data, its environment and its verifier, and every agent is Hermit's. The run config says so in `environment.harbor`: @@ -322,8 +323,9 @@ judge: **A rollout.** The run (`hermit/trainer/rollout.py`) starts the task's own Harbor environment (`hermit/environment/harbor.py`), hands the acting agent that environment with the task's public half -(`run(task, adapter, environment=env)`), runs Harbor's verifier on what the agent left there, whatever its session's -end (a timeout, a CLI that exited with an error: Harbor verifies all of them), and stops the environment. The +(`run(task, adapter, environment=env)`), runs Harbor's verifier on what the agent left there whenever the session's +end is the agent's result (a timeout, a CLI that exited with an error: Harbor verifies all of them; a session that is +infra, or an adapter that failed before any session, is read without it), and stops the environment. The verifier's verdict is the output's `extra["verified"]`, which the judge reads: the reward is the score (the reward under `reward_key`; none there is infra, as the runner read it), `passed` says whether it reached `pass_reward`, the feedback is the verifier's `feedback.txt` (Hermit's, beyond Harbor) or its output's tail. A verifier that left no reward, or ran out of its time @@ -350,30 +352,31 @@ over the runs that are the agent's result ([Evaluation](../guides/evaluation.md) (`environment.harbor.infra_retries=0`). AlgoTune's judge (`examples/algotune/utils/judge.py`) reads the same verdict and scores its speedup. -**The environment.** Harbor runs in its own venv (Python 3.13, `harbor` 0.25.0, `~/.cache/hermit/harbor-venv`), so -each environment is a driver process under Harbor's python (`hermit/environment/harbor_driver.py`) that speaks JSON -lines with Hermit. Harbor runs the whole trial with its own `Trial.run()`: the prebuilt image pulled or the Dockerfile -built (with the build timeout and its multiplier), the healthcheck, Harbor's capability check (a task with MCP servers, -a skills folder or a Windows container is refused: `unsupported_task`, never run again), the agent's setup (within its -setup timeout, under the baseline network policy), the agent phase (its network policy, its user, its timeout), the -task's tests uploaded only for the verifier, the verifier's environment (`[verifier] environment_mode`: shared, or a -separate one), its timeout and multiplier, the reward files, the scrub of credentials, the trial's `result.json` -(under `jobs_dir/__/`) and the environment's stop. The agent in that trial is a bridge with no logic of its -own (`HermitBridge`, a Harbor agent in the driver): its setup installs what the acting agent declares it needs -(`Agent.installs`: the CLI unless `bin` names one, python3 for an adapter's hooks) with Harbor's own installers, -Debian bullseye's archived sources repaired first; its run relays the agent's `exec`, `upload` and `download` to the +**The environment.** Harbor runs in its own venv (Python 3.13, `harbor` 0.25.0, `~/.cache/hermit/harbor-venv`), so each +environment is a driver process under Harbor's python (`hermit/environment/harbor_driver.py`) that speaks JSON lines +with Hermit. Harbor runs the whole trial with its own `Trial.run()`: Harbor's capability check (a task with MCP servers, +a skills folder or a Windows container is refused: `unsupported_task`, never run again), the prebuilt image pulled or +the Dockerfile built (with the build timeout and its multiplier), the healthcheck, the agent's setup (within its setup +timeout, under the baseline network policy), the agent phase (its network policy, its user, its timeout), the task's +tests uploaded only for the verifier, the verifier's environment (`[verifier] environment_mode`: shared, or a separate +one), its timeout and multiplier, the reward files, the environment's stop, the trial's `result.json` (under +`jobs_dir/__/`) and the scrub of credentials. The agent in that trial is a bridge with no logic of its own +(`HermitBridge`, a Harbor agent in the driver): its setup installs what the acting agent declares it needs +(`Agent.installs`: the CLI unless `bin` names one, python3 for an adapter's hooks) with Harbor's own installers, Debian +bullseye's archived sources repaired first; its run relays the agent's `exec`, `upload` and `download` to the environment until Hermit's agent is done, or Harbor's agent timeout ends the phase, then, as the runner's agents did: kills the process group of each command it started that still runs (each runs under `setsid`; a kill that fails is -infra), removes the credential files the agent registered (a sign-in; one not removed is infra), and copies the -agent's logs (a CLI's stream is also written to `/logs/agent` as it runs); what an ended command left running, a -server the task asked for, stays. Harbor then verifies. The agent's own budget ends 60 s before Harbor's agent timeout (10% of a short one), so that its session's -end and logs reach the trial first. Around it Hermit keeps its warm start -(`hermit/environment/harbor_docker.py`): no environment starts while the host's /tmp or Docker's data root has less -than its gate free (room kept for the task's image when it is not here yet, `image_sizes`, and for the images other -environments of the run are still to pull), the task's prebuilt image is removed after it as `docker_cleanup` says, -and a driver that had to be killed (or the process that ended without stopping it) has its trial's compose projects -taken down by name. Every environment Harbor supports is a `type` (docker by default; e2b, modal, daytona, ... with -Harbor's extra installed in its venv). +infra), removes the credential files the agent registered (a sign-in; one not removed is infra), and copies the agent's +logs (a CLI's stream is also written to `/logs/agent` as it runs, and scrubbed there at the phase's end, best effort, in +a local Docker environment; elsewhere a shared verifier gets it as it came); what an ended command left running, a +server the task asked for, stays. Harbor then verifies. The agent's own budget ends 60 s before Harbor's agent timeout +(10% of a short one), so that its session's end and logs reach the trial first. Around it Hermit keeps its warm start +(`hermit/environment/harbor_docker.py`): no environment starts while the host's /tmp or Docker's data root has less than +its gate free (room kept for the task's image when it is not here yet, `image_sizes`, and for the images other +environments of the run are still to pull), the task's prebuilt image is removed after it as `docker_cleanup` says, and +a driver that had to be killed (or the process that ended without stopping it) has its trial's compose projects taken +down by name. Every environment Harbor supports is a `type` (docker by default; e2b, modal, daytona, ... with Harbor's +extra installed in its venv). An environment's operations are the Environment API's ([Agent layers](agent-layers.md#environments)): `exec` runs a command in the task's main container as the task's agent user, from its working directory (the workspace), each @@ -419,7 +422,8 @@ Not supported in a Harbor environment yet, and refused rather than dropped: the server runs on this machine), the agent's `mounts`, `prepare` and `collect`, readable mounts, hidden paths or a network limit of Hermit's (the task's task.toml says what its environment reaches), and a multi-step task (`[[steps]]`, which needs one environment across the Trainer's steps). The credentials an agent command was given are -scrubbed from the logs handed to the verifier and from the trial's files. +scrubbed from the logs Hermit hands the verifier (and, in a local Docker environment, from a CLI's stream there) and +from the trial's files after the trial. | `environment.harbor` | Meaning | |---|---| diff --git a/docs/index.md b/docs/index.md index 4656a7b6..00ee61cc 100644 --- a/docs/index.md +++ b/docs/index.md @@ -108,6 +108,7 @@ developer/releasing developer/reproducing-a-method developer/service-design developer/sandbox-design +developer/harbor-design framework/agent-layers tutorials/custom-evolver framework/custom-evolver diff --git a/docs/tutorials/your-own-task.md b/docs/tutorials/your-own-task.md index c64da86e..f60e749c 100644 --- a/docs/tutorials/your-own-task.md +++ b/docs/tutorials/your-own-task.md @@ -44,6 +44,10 @@ A split file lists one id per line. Keep training, validation, and test disjoint Train task rows become learner-visible material. Keep private labels in judge-owned storage, not in fields you merely intend the pipeline to ignore. Restrict learner filesystem grants consistently with that separation. +A benchmark, or any task set whose tasks need an environment or a verifier of their own, is written as task +directories in Harbor's format: see +[Task directories](../framework/task.md#task-directories) and [Benchmarks](../framework/task.md#benchmarks). + ## Implement the acting pipeline The pipeline receives a task and an adapter path, or `None` for the no-adapter baseline. It must actually supply diff --git a/hermit/environment/harbor_driver.py b/hermit/environment/harbor_driver.py index a12b4874..1f554182 100644 --- a/hermit/environment/harbor_driver.py +++ b/hermit/environment/harbor_driver.py @@ -3,12 +3,12 @@ /bin/python -I harbor_driver.py # hermit.environment.harbor starts it, one per task -Harbor runs the whole trial with its own `Trial.run()`: the environment's start (the prebuilt image pulled or the -Dockerfile built, its build timeout and multiplier, the healthcheck), Harbor's capability check (a task's skills or -MCP servers, a Windows container: refused for this bridge), -the agent's setup (under Harbor's setup timeout, the baseline network policy, the agent user), the agent phase (the -task's agent network policy, its user, its timeout), the verifier (`[verifier] environment_mode` shared or separate, -the tests uploaded only now, its timeout), the scrub of the trial's files, result.json and the environment's stop. +Harbor runs the whole trial with its own `Trial.run()`: Harbor's capability check (a task's skills or MCP servers, a +Windows container: refused for this bridge), the environment's start (the prebuilt image pulled or the Dockerfile +built, its build timeout and multiplier, the healthcheck), the agent's setup (under Harbor's setup timeout, the +baseline network policy, the agent user), the agent phase (the task's agent network policy, its user, its timeout), +the verifier (`[verifier] environment_mode` shared or separate, the tests uploaded only now, its timeout), the +environment's stop, result.json and the scrub of the trial's files. The agent in it is HermitBridge, a Harbor agent with no logic of its own (an external agent, as Harbor calls one): its setup installs what Hermit asked for with Harbor's own installers, and its run relays Hermit's agent commands to the environment until Hermit says it is done, or Harbor's agent timeout or a cancellation ends it; either way it ends @@ -44,6 +44,7 @@ import asyncio import contextlib +import errno import json import os from pathlib import Path @@ -380,6 +381,7 @@ def __init__(self, out): self.transfers = asyncio.Lock() # a transfer at a time, never beside the phase's end self.secrets: set = set() # the credentials the agent's commands were given (scrubbed) self.secret_files: set = set() # files there holding a credential: removed at the phase's end + self.streams: set = set() # the agent logs a command's output is kept in as it runs self.exceptions: list = [] # every exception the trial recorded or skipped, in order self.repaired = False self.lock = asyncio.Lock() # one request that changes the trial's phase at a time @@ -410,8 +412,8 @@ def record(exc, _recorded=recorded): # Harbor keeps the first; the d self.exceptions.append(exc) _recorded(exc) trial._record_exception = record - if trial.task.has_steps: - raise ValueError(f"{trial.task.name} has [[steps]]: a multi-step task is not served here") + if trial.task.has_steps: # refused as Harbor refuses a task: never run again + raise Failed("UnsupportedTask", f"{trial.task.name} has [[steps]]: a multi-step task is not served here") self.trial, self.installs = trial, list(req.get("installs") or []) self.run_task = asyncio.create_task(trial.run()) await asyncio.wait({self.run_task, self.phase_started}, return_when=asyncio.FIRST_COMPLETED) @@ -576,7 +578,8 @@ async def _end_phase(self) -> None: """The end of the agent phase, at the runner's: the agent's requests in flight cancelled (their Docker clients killed with them), no transfer running (they hold the same lock), the process group of each command still running killed as root and checked gone (kill_groups), the credential files the agent registered removed - (each confirmed gone), and Hermit's logs copied, scrubbed, into the trial's agent logs. A command that ended + (each confirmed gone), the commands' streams scrubbed (_scrub_streams), and Hermit's logs copied, scrubbed, + into the trial's agent logs. A command that ended left what it started (a server the task asked for), as the runner's sessions did. A kill or a removal that fails is HarborDriverError: Harbor records it and runs no verifier (infra).""" for task in list(self.agent_ops): @@ -595,6 +598,7 @@ async def _end_phase(self) -> None: failed = failed or str(exc) if failed: raise HarborDriverError(failed) + await self._scrub_streams() await self._publish_logs() async def _remove_credentials(self) -> None: @@ -609,6 +613,42 @@ async def _remove_credentials(self) -> None: raise HarborDriverError("a credential file could not be removed from the environment: " f"{((gone.stdout or '') + (gone.stderr or ''))[-300:]}") + async def _scrub_streams(self) -> None: + """Each stream a command's output was kept in as it ran (a CLI's, in /logs/agent), scrubbed of the agent's + credentials before a shared verifier reads it, whether or not Hermit's own logs follow (Harbor's timeout ends + the phase without them). In a mounted environment (a local Docker one) that folder is the trial's on this + machine: each stream is read without following a link, scrubbed, and put back through safe_copy when that + changed it. It is best effort: a stream this machine cannot read or replace is logged and left as it came (as + before this scrub), never a failure of the phase's end. Elsewhere Harbor carries the stream to its verifier + as it came, and only the trial's files are scrubbed, after the trial.""" + if not (self.secrets and self.streams and getattr(getattr(self.env, "capabilities", None), "mounted", False)): + return + await asyncio.to_thread(self._scrub_stream_files, Path(self.trial.paths.agent_dir)) + + def _scrub_stream_files(self, agent_dir: Path) -> None: + try: + with tempfile.TemporaryDirectory() as kept: + for name in sorted(self.streams): + staged = Path(kept) / name + try: # a link is never followed (O_NOFOLLOW: ELOOP, skipped) + fd = os.open(agent_dir / name, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC) + with os.fdopen(fd, "rb") as given: # scrubbed before any copy of it is written here + if not stat.S_ISREG(os.fstat(given.fileno()).st_mode): + continue + data = given.read() + clean = scrubbed(data, self.secrets) + if clean != data: + staged.write_bytes(clean) + except OSError as exc: + with contextlib.suppress(OSError): # a part written is never put back + staged.unlink(missing_ok=True) + if exc.errno not in (errno.ENOENT, errno.ELOOP): + log(f"scrub: the stream {name} left as it came: {exc}") + if any(Path(kept).iterdir()): + safe_copy(Path(kept), agent_dir) + except OSError as exc: # no room to stage, or the copy failed: left as they came + log(f"scrub: the streams left as they came: {exc}") + async def _publish_logs(self) -> None: if self.agent_logs and Path(self.agent_logs).is_dir(): await asyncio.to_thread(safe_copy, Path(self.agent_logs), self.trial.paths.agent_dir, self.secrets) @@ -707,6 +747,7 @@ async def _run_command(self, req: dict, command: str, timeout, user, mark: str) await env.exec(f"chmod 644 {shlex.quote(staged)}", user="root") command = f"( {command} ) < {shlex.quote(staged)}" if req.get("stream"): + self.streams.add(Path(str(req["stream"])).name) logged = shlex.quote(f"{EnvironmentPaths.agent_dir}/{Path(str(req['stream'])).name}") command = f"set -o pipefail; {{ {command} ; }} | {{ tee {logged} 2>/dev/null; true; }}" q, pids = shlex.quote, shlex.quote(f"{PIDS}{mark}") diff --git a/tests/core/test_harbor_environment.py b/tests/core/test_harbor_environment.py index 9745412a..d66de508 100644 --- a/tests/core/test_harbor_environment.py +++ b/tests/core/test_harbor_environment.py @@ -2544,6 +2544,93 @@ async def end(how=failing): assert line == "RESULT " + str(["infra", "infra"]), line +@pytest.mark.skipif(not HARBOR_PYTHON.exists(), reason="needs Harbor's venv (~/.cache/hermit/harbor-venv)") +def test_a_clis_live_stream_is_scrubbed_before_the_verifier_even_when_harbors_timeout_ends_the_phase(tmp_path): + """A CLI's output is kept in /logs/agent as it runs, which a shared verifier reads. Harbor's timeout ends the phase + with no logs of Hermit's to copy over it, so the phase's end scrubs the stream there itself (a local Docker + environment shares that folder with this machine); a link planted in its place is never followed. Elsewhere it is + left for Harbor's download, which the trial's scrub covers.""" + agent_dir, outside = tmp_path / "agent", tmp_path / "outside.txt" + script = tmp_path / "probe.py" + script.write_text(textwrap.dedent(f""" + import asyncio, importlib.util, sys + from pathlib import Path + from types import SimpleNamespace as NS + spec = importlib.util.spec_from_file_location("drv", {str(DRIVER_FILE)!r}) + drv = importlib.util.module_from_spec(spec); sys.modules["drv"] = drv; spec.loader.exec_module(drv) + agent_dir, outside = Path({str(agent_dir)!r}), Path({str(outside)!r}) + + async def ran(command, **kwargs): + return NS(return_code=0, stdout="yes", stderr="") + + async def main(): + out = [] + for mounted in (True, False): + agent_dir.mkdir(exist_ok=True) + outside.write_text("{AGENT_KEY}") + (agent_dir / "planted.jsonl").unlink(missing_ok=True) + (agent_dir / "planted.jsonl").symlink_to(outside) + d = drv.Driver(None) + d.env = NS(exec=ran, capabilities=NS(mounted=mounted), _merge_env=lambda env: env) + d.trial = NS(paths=NS(agent_dir=agent_dir)) + key = {{"OPENAI_API_KEY": {AGENT_KEY!r}}} + for name in ("agent-codex.jsonl", "planted.jsonl"): # the CLI's session, its key passed by name + await d.exec({{"command": "codex exec -", "stream": name, "env": key}}) + (agent_dir / "agent-codex.jsonl").write_text("401: bad key {AGENT_KEY}\\n") # what tee kept + await d._end_phase() # no agent_logs: Harbor's timeout came first + out.append([(agent_dir / "agent-codex.jsonl").read_text(), outside.read_text(), + (agent_dir / "planted.jsonl").is_symlink()]) + # a stream this machine cannot read (another uid, a strict umask) whose own log follows: never infra + logs = agent_dir.parent / "logs" + logs.mkdir(exist_ok=True) + (logs / "agent-codex.jsonl").write_text("401: bad key {AGENT_KEY}\\n") + (agent_dir / "agent-codex.jsonl").chmod(0) + d.env, d.agent_logs = NS(exec=ran, capabilities=NS(mounted=True)), str(logs) + await d._end_phase() + out.append((agent_dir / "agent-codex.jsonl").read_text()) + print("RESULT", out) + asyncio.run(main()) + """)) + done = subprocess.run([str(HARBOR_PYTHON), "-I", str(script)], capture_output=True, text=True, timeout=60) + line = next((x for x in done.stdout.splitlines() if x.startswith("RESULT")), done.stdout + done.stderr) + assert line == "RESULT " + str([["401: bad key [REDACTED]\n", AGENT_KEY, True], + [f"401: bad key {AGENT_KEY}\n", AGENT_KEY, True], + "401: bad key [REDACTED]\n"]), line + + +@pytest.mark.skipif(not HARBOR_PYTHON.exists(), reason="needs Harbor's venv (~/.cache/hermit/harbor-venv)") +def test_a_multi_step_harbor_task_is_refused_as_unsupported_and_never_run_again(tmp_path): + """A task with [[steps]] is refused at the driver's start as a task Harbor runs for none of Hermit's agents + (UnsupportedTask), which Hermit reads as unsupported_task and does not run again, rather than an environment that + failed to start (sandbox_failed), which spent the infra retries on a task that can never run.""" + (tmp_path / "task").mkdir() + script = tmp_path / "probe.py" + script.write_text(textwrap.dedent(f""" + import asyncio, importlib.util, sys + from types import SimpleNamespace as NS + spec = importlib.util.spec_from_file_location("drv", {str(DRIVER_FILE)!r}) + drv = importlib.util.module_from_spec(spec); sys.modules["drv"] = drv; spec.loader.exec_module(drv) + + async def create(config): + return NS(task=NS(has_steps=True, name="multi"), _record_exception=lambda exc: None) + + async def main(): + drv.Trial.create = create + try: + await drv.Driver(None).start({{"task_dir": {str(tmp_path / "task")!r}, + "trials_dir": {str(tmp_path / "trials")!r}, "trial_name": "t"}}) + print("RESULT started") + except drv.Failed as exc: + print("RESULT", exc.kind) + asyncio.run(main()) + """)) + done = subprocess.run([str(HARBOR_PYTHON), "-I", str(script)], capture_output=True, text=True, timeout=60) + line = next((x for x in done.stdout.splitlines() if x.startswith("RESULT")), done.stdout + done.stderr) + assert line == "RESULT UnsupportedTask", line + from hermit.environment import harbor_errors + assert "unsupported_task" in harbor_errors.NO_RETRY + + def test_a_harbor_agents_adapter_code_runs_in_the_strict_adapter_sandbox(tmp_path, monkeypatch): """No worker for a Harbor task set's agent: its call in this process still confines the adapter's own code as the worker did (the runner's adapter sandbox): a context tool's Session is sandboxed, its exports' render goes through