Skip to content

Task/environment refactor: task directories, Harbor as environments, every agent Hermit's (#112-#116) - #117

Merged
haizhongzheng merged 5 commits into
mainfrom
task-environment-refactor
Nov 20, 2026
Merged

haizhongzheng merged 5 commits into
mainfrom
task-environment-refactor

Conversation

@haizhongzheng

@haizhongzheng haizhongzheng commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Change

The task/environment refactor (design doc "Hermit agent 与 benchmark 重构设计", 第二阶段), developed on the
integration branch task-environment-refactor as five reviewed PRs. Merge with a merge commit (not squash), so
each step stays one commit on main:

PR Commit What
#112 7e217556 hermit/task and hermit/environment beside the agents, hermit/common (structure only)
#113 52843e3b Task directories in Harbor's format as task sets, their own verifier script, the bubblewrap backend, data.view; OfficeQA as task directories
#114 125a09f7 Harbor gives only data, environments and the verifier; every agent is Hermit's
#115 b56562b1 Every remaining benchmark's tasks as task directories, read through a view
#116 22e97d70 Fixes from the final whole-branch review

Design in one paragraph. A task set is a directory of task directories in Harbor's format (or a tasks.jsonl); a
task has a public half (what the agent is handed) and a private half (only the judge reads). Benchmarks of two kinds:
those that really use Harbor (Docker, E2B, Modal; Harbor's verifier) and those that only use Harbor's format and run on
Hermit's own bubblewrap or local environment. For Harbor tasks, Harbor's own Trial.run() runs each trial and one thin
bridge agent relays Hermit's exec / upload / download during the agent phase, so agents stay independent of
Harbor (they only use Hermit's Environment API); the old harbor run runner and harborside (~17.8k lines) are gone.
Infrastructure failures run again in a fresh environment and are never scored as the agent's 0. Every benchmark's
prompts, records and learning inputs are byte-identical to before (data.view).

Breaking changes (CHANGELOG, Unreleased): every moved task set needs its prepare.py run again (OfficeQA, LIBERO,
Robosuite, RoboDojo-Sim, ServeLearnBench, CL-Bench, AgentDyn/AgentDojo, the claude_subscription toy); agent.name: harbor configs moved to agent.base + environment.harbor; hermit.agent.environment forwards to
hermit.environment for one release; setup_timeout_sec for Harbor tasks defaults to 1200 s.

Not supported yet (refused with a clear error): multi-step Harbor tasks (both paths); adapter tools inside a Harbor
environment; Pi, OpenCode, dsh, mini-swe-agent there; a CLI-backed model behind the answer agent there; session resume
in a remote environment.

Known issues (the old runner behaved the same; listed in #114 and #116): the bridge's pid file in agent-writable
/tmp (checked against the command's token); background processes of a finished command survive; transfer completion
not confirmed; a ChatGPT refresh lost on Harbor's hard timeout; a verifier failing before its tests run is scored 0; the
report and RRSI read output.status; a billed quota refusal that stops the run is not counted; the SWE-bench README's
token mean includes retries.

Validation

Linux (RHEL 9, x86_64), bubblewrap, Docker, Harbor 0.25.0 in its own venv, ASPIRE (pinned) with its services on an L40.

Program Path exercised Result
Terminal-Bench 2.0 smoke, GLM 5.3 Flash (Fireworks), mounted Codex Harbor path 1.0, no errors (twice)
Terminal-Bench AHE smoke, login: chatgpt, Codex installed by Harbor subscription login, install, learner train 1.0, 1.0; eval of the final adapter 1.0, no errors
OfficeQA smoke eval task directories, bubblewrap 2/12, no errors (twice; the same range as main)
LIBERO, one task ASPIRE engine on GPU, moved task set completed, no errors
CL-Bench database, one question moved stream benchmark 1.0, the learner wrote an adapter
AlgoTune SkillOpt, login: chatgpt (on 22e97d70) Harbor path, learner, validation gate train 0.410, 0.410 (validation 0.411 then 0.513); eval of the final adapter 0.410, no errors
LAB-Bench SkillOpt, login: chatgpt (on 22e97d70) Harbor path, learner, validation gate train 1.0, 1.0; eval of the final adapter 1.0, no errors

No API key, ChatGPT access/refresh/id token or Fireworks key in any run record, trial directory or log; no leftover
containers, and Harbor removed the task images it built.

Review

General checklist (added after the merge; the items above were the template's only)

  • Core changes. Yes: hermit/task/ (tasks, task sets, task directories, the script verifier),
    hermit/environment/ (the Environment API, the bubblewrap and Docker backends, Harbor's environment, its driver and
    harbor_errors), hermit/common.py, hermit/trainer/rollout.py (a task's environment started, handed to the
    agent, verified), hermit/isolate.py (the strict adapter sandbox for an adapter's own code), hermit/sandbox.py
    (small additions); the old Harbor runner and harborside removed. Why: the design doc's second phase (Change).
  • Agent layers. Agents use only the Environment API; the bridge in Harbor's driver holds no agent logic; tasks
    and environments import no agent (test_layers); benchmark agents stay subclasses in their example folders (Refactor 4/4: every remaining benchmark's tasks as task directories, read through a view #115).
  • Implementation correctness and failures. Traced through 22 review rounds (Review). Errors: a rollout's
    outcome is read as the old runner read it (harbor_errors.py, ported); infra runs again in a fresh environment and
    is never the agent's 0. Timeouts: the agent's budget ends 60 s before Harbor's; a failed end of the agent phase is
    infra, with no verifier. Cancellation: stop cancels the trial through Harbor's teardown; a killed driver's
    compose projects are taken down by name. What still behaves as the old runner did is under Known issues.
  • Real execution paths and tests. hermit train and hermit eval on the real programs in Validation, through
    Harbor in Docker (Terminal-Bench, AlgoTune, LAB-Bench), bubblewrap (OfficeQA), ASPIRE on a GPU (LIBERO) and a stream
    benchmark (CL-Bench); tests/core/test_harbor_environment.py runs a real Harbor trial in Docker where Docker and
    Harbor's venv are present.
  • Documentation. 24 documentation files in these PRs (docs/, the README, the CHANGELOG's Unreleased); the contributor checklists and a
    Harbor design page with diagrams follow in Harbor design page and contributor checklists; scrub a CLI's live stream, refuse multi-step tasks as unsupported #119.
  • This description. Read end to end against the merged tree and the run records.

Sandbox checklist

  • Reuse the framework's sandbox. A task's agent and its script verifier run in the bubblewrap backend
    (hermit/environment/bwrap.py, built on hermit.sandbox; launch records runs/<run>/sandbox/*.json) or, for a Harbor task, in the task's container (its trial directory and result.json under
    environment.harbor.jobs_dir); an adapter's own code (its context tool, its exports' render) runs in the strict
    adapter sandbox (hermit/isolate.py). Not confined: Harbor's driver, which runs under Harbor's python on this
    machine as harbor run did before, since it drives Docker; no agent, adapter or task code runs in it.
  • Sandbox changes are core changes. Yes (hermit/environment/, hermit/isolate.py, hermit/sandbox.py).
    Agreed with the repository's maintainer in the design doc; no separate issue.

Test checklist

  • Group and selection. New tests in tests/core/ (test_task.py, test_task_directory.py,
    test_harbor_environment.py) and the examples' own utils/tests/; tests/selection.py lists hermit/task/*,
    hermit/environment/* and hermit/common.py as shared; no new helper or fixture directory.
  • Names. Sentences (test_the_environments_moved_to_the_top_level_and_the_old_path_forwards).
  • Docstrings. Each new file opens with one naming the code under test and what it covers.
  • What it asserts. The tests assert what a caller sees (a status, a record, a raised error). Not done: a
    systematic check that each review fix's regression test fails on the code before the fix.
  • Real entry points and stand-ins. Rollouts go through hermit/trainer/rollout.py and Config.load; the
    driver is tested against real Harbor in Docker; the model and the CLI are the stand-ins.
  • Isolation. tmp_path; the Docker test skips, with its reason, without Docker or Harbor's venv.
  • Time and order. One order flake under parallel load (reharness) predates these PRs, on main (Refactor: fixes from the final review (verifier script's own environment, host faults, usage) #116).
  • Cost. Not measured per new test; the full suite took 21 min (Validation).
  • Hygiene. The changed files linted; no processes or containers left after the suite or the real runs.

🤖 Generated with Claude Code

haizhongzheng and others added 5 commits November 19, 2026 13:03
…tructure only) (#112)

* Tasks and environments beside the agents: hermit/task, hermit/environment, hermit/common

The first of four stacked changes toward Harbor-format tasks run by Hermit's own
agents (design: "Hermit agent 与 benchmark 重构设计", 第二阶段). Structure only:
no behavior changes, the requests every agent sends are byte for byte main's.

- hermit/environment/: the environments move out of hermit/agent/ to the top
  level, since a task declares what it needs of one and a judge may run in one;
  hermit.agent.environment forwards for one release.
- hermit/task/: task.py becomes a package. Task holds a public half (what the
  acting agent is handed, what a run records as its input) and a private half
  only the judge reads; Benchmark is a task set and its splits; verify.py wraps a
  judge as a verifier of a Task. Every name importable from hermit.task before
  still is.
- hermit/common.py: RunOutput and load_callable, shared by agents and tasks so
  that neither imports the other.
- The run (hermit/trainer/rollout.py) hands the agent task.given() and the
  verifier the whole task; the trainer, eval and hermit check read a Benchmark.
  The validation file the learner gets holds each task's public half.
- tests/core/test_layers.py: tasks and agents never import each other; the
  environments import neither. tests/core/test_task.py covers the new types.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* hermit check: report the probe task by its id (a Task, not a row)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 1: a bare judge reads the whole row again; records keep the row's id

- judge_verifier calls the judge with Task.row() (the public half and `_private`), as
  run_one did before tasks were typed; the agent still gets only the public half.
- Records keep the row's own id (7, not "7"); run_tasks keys its results by task.
- run_one/run_tasks keep the `judge` parameter name; the Verifier protocol states
  `takes_task`; agent-layers.md shows hermit/environment/ beside hermit/agent/.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* rollout: the record's id in one name (line length)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* eval: a record's numeric id finds its task (after the record kept the row's id)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
…, the bwrap backend; OfficeQA as task directories (#113)

* Tasks and environments beside the agents: hermit/task, hermit/environment, hermit/common

The first of four stacked changes toward Harbor-format tasks run by Hermit's own
agents (design: "Hermit agent 与 benchmark 重构设计", 第二阶段). Structure only:
no behavior changes, the requests every agent sends are byte for byte main's.

- hermit/environment/: the environments move out of hermit/agent/ to the top
  level, since a task declares what it needs of one and a judge may run in one;
  hermit.agent.environment forwards for one release.
- hermit/task/: task.py becomes a package. Task holds a public half (what the
  acting agent is handed, what a run records as its input) and a private half
  only the judge reads; Benchmark is a task set and its splits; verify.py wraps a
  judge as a verifier of a Task. Every name importable from hermit.task before
  still is.
- hermit/common.py: RunOutput and load_callable, shared by agents and tasks so
  that neither imports the other.
- The run (hermit/trainer/rollout.py) hands the agent task.given() and the
  verifier the whole task; the trainer, eval and hermit check read a Benchmark.
  The validation file the learner gets holds each task's public half.
- tests/core/test_layers.py: tasks and agents never import each other; the
  environments import neither. tests/core/test_task.py covers the new types.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* hermit check: report the probe task by its id (a Task, not a row)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 1: a bare judge reads the whole row again; records keep the row's id

- judge_verifier calls the judge with Task.row() (the public half and `_private`), as
  run_one did before tasks were typed; the agent still gets only the public half.
- Records keep the row's own id (7, not "7"); run_tasks keys its results by task.
- run_one/run_tasks keep the `judge` parameter name; the Verifier protocol states
  `takes_task`; agent-layers.md shows hermit/environment/ beside hermit/agent/.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* rollout: the record's id in one name (line length)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* eval: a record's numeric id finds its task (after the record kept the row's id)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Task directories in Harbor's format, their own verifier, the bwrap backend; OfficeQA as task directories

The second of four stacked changes (design: 第二阶段). A task set may now be a
directory of task directories in Harbor's format, run on Hermit's own sandbox
(B-class: Harbor's format, not Harbor's runtime).

- hermit/task/directory.py: Harbor's task format read without Harbor (moved from
  hermit/harbor/dataset.py, which imports it): the instruction (line endings
  kept), settings and metadata public; the task's directory and tests private.
  Benchmark loads `data.tasks: <directory>` (`data.rollouts: k`).
- hermit/task/verify.py: a task's own verifier, `judge: {path:
  hermit.task.verify, name: script}`, run in the judge's sandbox as Harbor runs
  it: tests at /tests, a fresh /logs/verifier, reward.txt or reward.json, and
  feedback.txt for learning methods. A judge is called with Task.row() (the row
  plus `_private`); an agent still gets only the public half.
- hermit/environment/spec.py and bwrap.py: EnvironmentSpec (the run config's
  `environment` section) and the bubblewrap backend the task sandbox is built
  by; a task asking for more network or memory than the run gives is refused.
- examples/officeqa: the task set moves to utils/benchmark/ (prepare.py writes
  task directories; judge.py, task-info/, splits, data.json, seed adapter); the
  agents read each task as the row they always read (officeqa_fields.py), so the
  prompts are byte for byte the same.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 1: OfficeQA's methods on the new layout; task directories as Harbor reads them

OfficeQA (the methods the move missed): ReHarness's launcher tests the tasks
directory; RRSI's rows (byte-identical to before on the real data) and AHE's come
from the task directories through officeqa_fields.question_rows; RRSI and AHE
score with utils/benchmark/judge.py; Harness²'s source hook reads a directory
row; OPRO shows the instruction; utils/.gitignore keeps the old paths ignored;
AHE's real-data tests find the prepared data again.

Task directories: a step's [verifier] settings come before the task's;
[verifier.env] is set for the script (${NAME} resolved as Harbor does);
reward.json takes a boolean as Harbor does; a task's [agent] timeout_sec bounds
its calls and the judge waits at least the verifier's timeout_sec; an allowlist
task's hosts and (with the script verifier) its verifier's network are checked.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 2: a view of OfficeQA's rows for agents and learners; Harbor's verifier defaults and allowlists

- data.view: <file.py>:<function> turns a task directory's row into the row the
  task set's agents and learners read. OfficeQA's configs view each task as
  SkillOpt's row, so records and what a learning method reads (GEPA, ACE, OPRO,
  SkillOpt) are the same as with its tasks.jsonl (246/246 on the real data);
  OPRO's question_field is `question` again.
- What a task asks of its environment is kept in its private half (`needs`) and
  checked before each call (hermit.trainer.rollout asks a sandboxed agent's
  `refuses`), viewed or not.
- As Harbor: a verifier's timeout_sec defaults to 600; [verifier.environment]
  implies a separate verifier; each phase's allowlist (a step's first) is its own;
  wildcard hosts are refused under an allowlist run (the proxy matches hosts as
  written).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 3: a step's verifier as Harbor resolves it; a viewed task keeps its timeout; finite rewards

- A step's verifier mode and environment are resolved as Harbor's
  verifier_mode.py does (explicit mode, else an environment implies separate,
  the step's level before the task's, else shared), and the verifier's network
  and allowlist come from that effective environment.
- A step's tests/ are laid over the task's in the judge's sandbox (its test.sh
  first), as Harbor uploads them.
- A task's agent timeout is kept in its needs, which the run hands the sandboxed
  agent, so a data.view that leaves it out of the row still bounds the call.
- reward.txt must hold a finite number, as in Harbor.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 4: multi-step tasks refused on this path; the verifier's workspace and memory

- A multi-step task directory is refused here (its steps share one environment in
  order, which a per-call sandbox does not give); PR 4 brings that.
- The script verifier's workspace defaults to the directory the agent worked in,
  when its output names none.
- A separate verifier's own memory_mb is checked against the run's limit.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
…ent is Hermit's (#114)

* Tasks and environments beside the agents: hermit/task, hermit/environment, hermit/common

The first of four stacked changes toward Harbor-format tasks run by Hermit's own
agents (design: "Hermit agent 与 benchmark 重构设计", 第二阶段). Structure only:
no behavior changes, the requests every agent sends are byte for byte main's.

- hermit/environment/: the environments move out of hermit/agent/ to the top
  level, since a task declares what it needs of one and a judge may run in one;
  hermit.agent.environment forwards for one release.
- hermit/task/: task.py becomes a package. Task holds a public half (what the
  acting agent is handed, what a run records as its input) and a private half
  only the judge reads; Benchmark is a task set and its splits; verify.py wraps a
  judge as a verifier of a Task. Every name importable from hermit.task before
  still is.
- hermit/common.py: RunOutput and load_callable, shared by agents and tasks so
  that neither imports the other.
- The run (hermit/trainer/rollout.py) hands the agent task.given() and the
  verifier the whole task; the trainer, eval and hermit check read a Benchmark.
  The validation file the learner gets holds each task's public half.
- tests/core/test_layers.py: tasks and agents never import each other; the
  environments import neither. tests/core/test_task.py covers the new types.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* hermit check: report the probe task by its id (a Task, not a row)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 1: a bare judge reads the whole row again; records keep the row's id

- judge_verifier calls the judge with Task.row() (the public half and `_private`), as
  run_one did before tasks were typed; the agent still gets only the public half.
- Records keep the row's own id (7, not "7"); run_tasks keys its results by task.
- run_one/run_tasks keep the `judge` parameter name; the Verifier protocol states
  `takes_task`; agent-layers.md shows hermit/environment/ beside hermit/agent/.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* rollout: the record's id in one name (line length)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* eval: a record's numeric id finds its task (after the record kept the row's id)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Task directories in Harbor's format, their own verifier, the bwrap backend; OfficeQA as task directories

The second of four stacked changes (design: 第二阶段). A task set may now be a
directory of task directories in Harbor's format, run on Hermit's own sandbox
(B-class: Harbor's format, not Harbor's runtime).

- hermit/task/directory.py: Harbor's task format read without Harbor (moved from
  hermit/harbor/dataset.py, which imports it): the instruction (line endings
  kept), settings and metadata public; the task's directory and tests private.
  Benchmark loads `data.tasks: <directory>` (`data.rollouts: k`).
- hermit/task/verify.py: a task's own verifier, `judge: {path:
  hermit.task.verify, name: script}`, run in the judge's sandbox as Harbor runs
  it: tests at /tests, a fresh /logs/verifier, reward.txt or reward.json, and
  feedback.txt for learning methods. A judge is called with Task.row() (the row
  plus `_private`); an agent still gets only the public half.
- hermit/environment/spec.py and bwrap.py: EnvironmentSpec (the run config's
  `environment` section) and the bubblewrap backend the task sandbox is built
  by; a task asking for more network or memory than the run gives is refused.
- examples/officeqa: the task set moves to utils/benchmark/ (prepare.py writes
  task directories; judge.py, task-info/, splits, data.json, seed adapter); the
  agents read each task as the row they always read (officeqa_fields.py), so the
  prompts are byte for byte the same.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 1: OfficeQA's methods on the new layout; task directories as Harbor reads them

OfficeQA (the methods the move missed): ReHarness's launcher tests the tasks
directory; RRSI's rows (byte-identical to before on the real data) and AHE's come
from the task directories through officeqa_fields.question_rows; RRSI and AHE
score with utils/benchmark/judge.py; Harness²'s source hook reads a directory
row; OPRO shows the instruction; utils/.gitignore keeps the old paths ignored;
AHE's real-data tests find the prepared data again.

Task directories: a step's [verifier] settings come before the task's;
[verifier.env] is set for the script (${NAME} resolved as Harbor does);
reward.json takes a boolean as Harbor does; a task's [agent] timeout_sec bounds
its calls and the judge waits at least the verifier's timeout_sec; an allowlist
task's hosts and (with the script verifier) its verifier's network are checked.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 2: a view of OfficeQA's rows for agents and learners; Harbor's verifier defaults and allowlists

- data.view: <file.py>:<function> turns a task directory's row into the row the
  task set's agents and learners read. OfficeQA's configs view each task as
  SkillOpt's row, so records and what a learning method reads (GEPA, ACE, OPRO,
  SkillOpt) are the same as with its tasks.jsonl (246/246 on the real data);
  OPRO's question_field is `question` again.
- What a task asks of its environment is kept in its private half (`needs`) and
  checked before each call (hermit.trainer.rollout asks a sandboxed agent's
  `refuses`), viewed or not.
- As Harbor: a verifier's timeout_sec defaults to 600; [verifier.environment]
  implies a separate verifier; each phase's allowlist (a step's first) is its own;
  wildcard hosts are refused under an allowlist run (the proxy matches hosts as
  written).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 3: a step's verifier as Harbor resolves it; a viewed task keeps its timeout; finite rewards

- A step's verifier mode and environment are resolved as Harbor's
  verifier_mode.py does (explicit mode, else an environment implies separate,
  the step's level before the task's, else shared), and the verifier's network
  and allowlist come from that effective environment.
- A step's tests/ are laid over the task's in the judge's sandbox (its test.sh
  first), as Harbor uploads them.
- A task's agent timeout is kept in its needs, which the run hands the sandboxed
  agent, so a data.view that leaves it out of the row still bounds the call.
- reward.txt must hold a finite number, as in Harbor.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Harbor environment backend: a task's Harbor environment as a Hermit Environment

hermit/environment/harbor.py (HarborEnvironments, HarborEnvironment) and its driver,
hermit/environment/harbor_driver.py, run under Harbor's own venv: Harbor's trial of the
task with its nop agent, driven a piece at a time (start, exec, upload, download,
install, verify, stop) over JSON lines. hermit/harbor/docker.py moves unchanged to
hermit/environment/harbor_docker.py (the disk gate, image cleanup, compose_down).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Hermit drives each Harbor task: the agent works in the task's Harbor environment, Harbor verifies

- Acting agents take the task's environment: run(task, adapter, environment=...).
  The CLI Base Agents (Codex, Claude Code) run in a remote environment through its
  exec (the home rendered here and uploaded, the CLI installed by Harbor or mounted);
  the CLI acting agent deploys the adapter for a session there (hooks rewritten
  relative to the workspace); the answer-only agent (agent.name: answer) replaces
  HermitAnswer.
- hermit/environment/remote.py: the environment served to the agent's worker in the
  task sandbox (exec, upload, download, install; never verify).
- The run (hermit/trainer/rollout.py) starts each task's environment
  (environment.harbor), hands it to the agent, verifies with Harbor's verifier and
  stops it; judge hermit.task.verify:environment reads the verdict.
- Deleted: hermit/harbor/runner.py, harborside/ and the runner's parts and tests.
- Configs: every agent.name: harbor config moves to agent.base + environment.harbor.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Tests for the Harbor environment, the served environment, remote CLI sessions, the answer agent and the run

tests/core/test_harbor_environment.py: the driver's protocol with a stand-in driver,
the warm start's calls, the verifier's verdicts, Codex and Claude Code in a remote
environment, the CLI acting agent's staging, the answer agent, the environment server
(no verify), the run's start/verify/stop, the environment judge; and a real Docker +
Harbor test (skipped without them). OfficeQA's Harbor config and judge, and the
example tests, follow the new path.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Docs and changelog: Harbor environments, every agent Hermit's

docs/framework/task.md's Benchmarks section describes the new path (environment.harbor,
the driver, the warm start, the agents in a remote environment, what is refused); the
agent layers, configuration, sandbox design, evaluation, model backends,
troubleshooting and compatibility pages, examples/README.md, the benchmark READMEs and
the OfficeQA recipe follow it. CHANGELOG: Unreleased, Breaking.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* A ReactAgent in a task's environment: its tools run there, the loop without the environment's budget

Codex's install in a remote environment: with a version set, Harbor's installer is
asked (it keeps a CLI already at that version).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* rollout: wrap a long line (after rebasing onto PR 2)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Harbor environments keep the runner's infra semantics: run again, and never the agent's 0

- hermit/environment/harbor_errors.py: the runner's errors.py ported (Harbor's error
  policy, INFRA_CAUSES, GRADED, HOST_FAILURE, the verifier that ran no test, classify),
  with its verifier_timeout_verdict and verifier_verdict, NO_RETRY, the defaults
  (infra_retries 2, infra_wait_s 300 s), and the causes of an environment that did
  not start or a session that never did.
- HarborEnvironments.verdict reads the verifier's answer as the runner read a trial;
  EnvironmentUnavailable carries its cause (disk, sandbox_failed, harbor_failed,
  harbor_timeout).
- hermit/trainer/rollout.py: a rollout that ends infra runs again in a fresh
  environment up to environment.harbor.infra_retries times after infra_wait_s, never
  for disk or model_rejected; every run's usage is the rollout's (extra.attempts, the
  runner's _attempts); one that stays infra is marked unscored and the environment
  judge refuses it (Unscored: a judge_error record, as the runner's judge did).
- report.kind counts output.status infra again; tests, task.md, evaluation guide,
  examples README and CHANGELOG follow.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 4: multi-step tasks refused on this path; the verifier's workspace and memory

- A multi-step task directory is refused here (its steps share one environment in
  order, which a per-call sandbox does not give); PR 4 brings that.
- The script verifier's workspace defaults to the directory the agent worked in,
  when its output names none.
- A separate verifier's own memory_mb is checked against the run's limit.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 1 (#114): restore what the Harbor runner did for a session in a task's environment

- The CLI on PATH after Harbor's install (nvm's, ~/.local/bin), the prompt on stdin
  and Harbor's own instruction as the prompt (byte for byte, as the runner gave it);
  the task's own agent budget, also for the sandboxed worker's deadline; a session
  is not resumed (refused).
- Every sign-in file leaves the environment before the verifier (confirmed, from /);
  the ChatGPT plan's usage reports come back; web search off on a restricted agent
  network; the login's hosts allowed; a login's plan waited for before the
  environment starts; the disk gate and the infra pause stop with the run.
- Codex's lost sign-in, provider failure, plan limit and refused configuration read
  from its notices (codex_stream, ported from signin.py); a refused configuration is
  model_rejected (never run again); provider and environment failures across the
  worker boundary are infra; the CLI's provider allowance is the rollout's one.
- Usage of every attempt counted; status/failure_cause beside the verdict (AHE);
  hooks' audit and state collected; python3 ensured for hooks; the image's context
  file merged within Codex's cap, an unreadable one infra.
- Driver: agent commands still running are ended before the verifier (not a server
  an ended one left); logs copied without following links; installs under the
  baseline network policy, within Harbor's setup timeout, Debian bullseye sources
  repaired; a timeout without `timeout` still ends the command; requests written by
  a writer thread. Harbor task directories are run secrets. Answer agent: Harbor's
  default /app/answer.txt.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* A CLI install that failed spent nothing: its run again's cost is 0, not unknown

Seen in the Harbor-install smoke: Harbor's Codex install ran past Harbor's setup timeout
on a 1-CPU task (infra, run again, then served); the failed run's cost made the eval's
cost unknown, where the runner counted a setup failure as 0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 2 (#114): the boundaries the Harbor runner kept, and its reading of a session

- Security: no variable an agent's command names touches the driver or the host's
  compose client (each travels under a HERMIT_ENV_<n> carrier and is exported in the
  container); a by_fd secret (Claude Code's token) is moved into a pipe by the
  outermost shell before `timeout` or the CLI start, so no persistent process holds it.
- An agent command's process tree is killed (descendants of its marked processes,
  whatever their environment); installs exclude agent commands while the setup
  network policy is open; Harbor's scrub of the trial's files runs on verify and stop,
  and the verifier's exception is scrubbed before Hermit reads it.
- The driver's teardown kills a driver whose writer is stuck; the working directory
  is read from a marked line; environment-operation and host failures are typed
  infra (host_failed never run again).
- The runner's session_verdict ported: a configuration refused before any token is
  model_rejected, a refused bill or spent quota ends the run (usage_limit), a prompt
  too long or the model's refusal stays the agent's result; Codex's failure text read.
- A failed attempt keeps its usage (the agent returns it as an infra run); the plan's
  refused sessions' usage counts; a rotated ChatGPT sign-in comes back however the
  exec ended; the image's context file is read once across provider retries.
- On a login the plan's wait and slot are held around the whole environment (the CLI
  takes none inside; a refused session holds the plan and the task runs again, not
  counted); the answer and react agents keep the task's budget on a host pipeline.
- verifier_reward_raw beside the status (AHE); eval rows keep the rollout's outcome;
  environment.harbor.* overridable on the command line.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test: an unused import

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 3 (#114): the runner's results kept on the long paths

- A CLI session's provider failure runs the task again in a fresh environment
  (environment.harbor.busy_retries/busy_wait_s, the runner's allowance over a task's
  runs, not infra_retries), never again in the environment the session changed.
- An adapter that failed before any session (deployment, per-task context, context over
  Codex's cap) is the agent's result scored 0 (adapter_failed, context_over_limit), not
  left out of the denominator; a minimal row's context is computed from Harbor's
  instruction.
- The image's linked context file is read through the environment and the link
  replaced (real Docker test); host failures are host_failed (not run again); a
  teardown failure after the verdict keeps it; a lost sign-in stops new work before
  the task's teardown; the verifier watchdog follows Harbor's own verifier timeout.
- The install gate admits and counts a command at once and reopens on every path;
  an admitted session that drew credits holds the plan; a refused session without
  a plan of the run's ends the run; local sign-in copies go though a take-back fails.
- ReAct keeps what its loop spent on a timeout and asks no follow-up after it; the
  answer agent's whole run is bounded, its log descriptor closed, model.max_retries
  honoured, attachments sent once, its endpoint's refusals read as the runner read
  them (bill: usage_limit; configuration: model_rejected; input: its result).
- An acting agent's model a CLI reaches is refused for Harbor environments; run_agent
  and RRSI's smoke see a Harbor rollout's outcome; SWE-bench's token mean filters by
  outcome.status.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 4 (#114): race-proof log copy, confirmed kills, the runner's Codex cap

- The agent's logs reach the trial's folder through O_NOFOLLOW directory descriptors and O_EXCL temp files renamed
  into place: an entry swapped for a link while the copy runs is replaced, never written through.
- A failed restore of the agent phase's network policy breaks the environment: no further agent command, install or
  ensure; _ensure_python3 re-raises it.
- kill_marked confirms its kill (exit 1 when a process is left); _drain raises before the verifier, keeps the mark
  and tries again; cleanup runs from /.
- The sandbox proxy refuses network, hidden-path and readable limits instead of ignoring them.
- A hook audit that cannot be read keeps the session's result and usage (hooks_audit_missing).
- Codex in a task's environment keeps the runner's project_doc_max_bytes of 262144 unless the config sets one.
- Answer: adapter context failures scored 0 (adapter_failed), preparation failures infra, the answer moved into place
  only within the budget; ReAct's answer-tag follow-up is bounded; a call given up in flight counts as a call with
  unknown cost.
- The disk gate rechecks the stop flag (between gates, after them): nothing starts, the rollout ends as the stopped
  run's.
- logins_of treats an acting agent in environment.harbor as contained; the benchmark-file form of environment.harbor
  takes nested overrides; config-fields and examples/README updated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 5 (#114): the Hermit-side fixes that outlive the bridge redesign

- The credentials an agent command is given are scrubbed from the logs handed to the verifier and from the trial's
  files (the driver records them from each exec's env and by_fd).
- A ChatGPT sign-in comes back as the runner took it (signin.take_back, ported into codex_stream): a private
  download, a regular file of at most 64 KiB with both tokens, stamped at most SKEW/OTHER_CLOCK ahead; a failed local
  write of the refresh keeps the finished session.
- The sandboxed agent provisions and takes back the configured login_root.
- A Codex rejection keeps its head (the status) for the verdict, as the runner did.
- A remote session that reached its CLI but reported no usage counts as an unknown cost.
- A refused session whose plan hold cannot be written pauses HOLD_UNWRITTEN_S before the next environment.
- reward_key is authoritative (none there is infra, the runner's classify); verifier_redact restored; an adapter
  failure has no pass on a benchmark without pass_reward.
- The answer agent extracts as the runner did (group 1 only when it took part) and sends max_tokens only when given.
- Docs: the agents that work in a Harbor environment; report docstring; the Codex cap comment.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Harbor runs the whole trial: one bridge agent relays Hermit's commands (#114)

Harbor's own Trial.run() now owns each task's trial: the environment's start, its capability check, the agent's
setup (Harbor's setup timeout, the baseline network policy, the agent user), the agent phase (its network policy,
user and timeout), the verifier in its mode, the scrub, result.json and the stop. The driver no longer drives the
trial a piece at a time.

- HermitBridge (a Harbor BaseAgent in the driver, imported by AgentConfig.import_path) has no agent logic: its setup
  installs what the acting agent declares (Agent.installs: the CLI unless `bin` names one, probed first unless a
  version is pinned; python3 for an adapter's hooks) with Harbor's own installers; its run starts the phase, relays
  Hermit's exec/upload/download until `done`, Harbor's agent timeout or a cancellation, and ends every command it
  started that still runs (their Docker clients too, the kill confirmed) before it returns. A kill not confirmed
  fails the run (HarborDriverError: no verifier, infra). Hermit's logs reach /logs/agent scrubbed.
- Deleted: the Hermit-side network-policy switching and install gate, the install/ensure/verify ops, the separate
  drain step, the piecewise start/verify/stop. HarborEnvironment.install reports what setup installed; verify sends
  `done` and reads the trial's verdict. Harbor refusing a task's skills, MCP servers or Windows is unsupported_task
  (never run again). The agent's own budget ends 60 s before Harbor's agent timeout so its logs arrive first; a
  phase Harbor ended (AgentPhaseOver) is read as the trial's verdict.
- Agents stay Harbor-free: they use Hermit's Environment API; only the rollout of a Harbor task set starts a session.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Bridge: transfers as the agent user, its uid:gid, limits refused, no missed death (#114)

- r5 #2: an upload or download the agent's process asks for through the sandbox's proxy (as_agent) runs with the task
  agent user's own permissions when it is not root (staged as root, copied into place as that user; a tar made as
  that user, unpacked as data); root only for Hermit's own provisioning. PermissionError, not infra.
- r5 #9: the bridge reads the uid:gid the agent runs as (an image's own USER too), and Hermit's provisioning is
  chowned to it, confirmed (the runner's _owner).
- r5 #8: the CLI agent keeps an environment's other limits when it cuts the time, and refuses network, hidden or
  readable limits in a remote environment; the sandbox's proxy refuses them before it is served.
- r5 #13: the driver's reader publishes its death, and replies, under the lock a request registers under.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* One implementation each: usage ledger, credential lifecycle, deadline, retry states (#114)

- hermit/agent/model/usage.py: Ledger (an agent run's usage: every attempt, every call given up in flight, every
  session cut off before its usage; cost unknown when any entry's is; what an empty ledger reports is the caller's
  word) and Attempts (a rollout's runs and their sum, per-attempt records: the runner's _attempts). The workspace
  agent, ReAct, the answer agent and base React's total use it; rollout._attempts and the hand sums are gone.
- hermit/agent/base/credentials.py: held(provision, take_back, remove), the sign-in's lifecycle with its finally:
  the CLI's remote session and the sandboxed agent's worker both go through it.
- hermit/agent/base/deadline.py: one absolute Deadline (remaining, cut, a chat client bound to it): ReAct's and the
  answer agent's model calls, the CLI's limits before its session and the workspace's deadline_s.
- hermit/trainer/rollout.py: Retries, the rollout's one retry and admission state machine with its states named
  (DONE, INFRA, PROVIDER_BUSY, PLAN_WAIT, STOP); one run in a fresh environment is _environment_run.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 6 (#114): the agent phase's end, outside Harbor's timeout, does what it must

Structural fix for the end of the agent phase (findings 1-7, 11): the bridge's end-of-phase work runs as a task of
its own that its run awaits through any cancellation (Harbor's agent timeout can no longer cut it short), bounded by
END_PHASE_S; anything it cannot confirm raises HarborDriverError, so Harbor runs no verifier. It: lets transfers in
flight finish, cancels the commands, ends their clients and kills their processes (running and earlier unconfirmed
kills alike), keeps a running command's stream (exec `stream=`, tee'd there as it runs) in the agent logs under its
name, removes the credential files the CLI registered before its upload (Environment `secret`, confirmed), and copies
Hermit's logs scrubbed. A cancelled wait on any Docker client kills it (compose up after a build timeout too); stop
answers outside the lifecycle lock, ending the start's clients before the teardown.

- The verdict: a verifier that failed after the agent's timeout is read by its infra policy (the driver keeps every
  exception Harbor recorded or skipped); a result.json Harbor could not write falls back to the trial's in-memory
  result, and no verdict at all is the driver's error.
- A disabled verifier is refused; Windows refused at the trial's creation is UnsupportedTask.
- Agents: install discovery never raises on an adapter (the deployment scores it); a deployment Harbor's phase end
  cut short is not adapter_failed; python3 for hooks is best effort at setup; the claude-code alias installs Claude
  Code; the answer agent's log is a folder with the runner's answer.json; eval checks the sign-in before replacing
  results. Docs: the judge's place with Harbor, installs is built-in only, the driver's intro, AHE's trace note.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Ported from the runner: a mounted CLI's checks, Claude's key, attachments, infra usage (#114)

- A mounted binary (`bin`) serves only when it runs there as the pinned version (any, when none is) and the agent's
  user may execute it (the runner's _use_mounted): linked onto PATH, with the release's rg and this machine's CA bundle
  for an image that has none (harborside.codex's _after_mount); else the bridge's setup installs the CLI and the
  session runs the installed program (bin_served, through the proxy too).
- Claude Code in a task's environment gets no raw key variable of its binding: it reads its key from its own
  (session_env), as the runner's launch left it.
- The answer agent warns when it leaves out an attachment over 20 MiB.
- A run that is not the agent's result after a session ran, with no usage reported, has an unknown cost and token
  count (the runner's usage None), never 0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 7 (#114): no verifier after an unfinished end, the agent's requests as the agent

- P1 #1: the phase's end that does not finish (its bound, or any other way) is HarborDriverError, read from the
  shield's own exception too: Harbor runs no verifier after it (_close_phase, tested past its bound).
- P1 #2: nothing the worker asks for runs as root: credential files are removed as the agent user, an upload's
  missing folders are made by that user (cp -f replaces what it may replace), and the CLI's home there is the
  bridge's own provisioning, declared by the agent (Agent.installs `home`), made and given to the user at the phase's
  start.
- #3: each command writes its pid at its start, the root of what kill_marked ends where root cannot read another
  user's environment (no CAP_SYS_PTRACE). #4: the commands are stopped as the phase closes, before a pending
  transfer is let finish. #5: the driver's environment drops the ambient credentials the runner's harbor_env dropped.
  #6: a stop Harbor acknowledged with its compose projects still there takes them down by name (a local, deleted
  Docker environment). #7: cp -f. #8: one absolute deadline from the phase's start (deadline_at, through the proxy):
  every command's time and ReAct's and the answer agent's model calls come off it. #9: a timed-out rollout's log is
  the trial's agent logs (the stream the bridge kept, stderr too). #10: the start's and a separate verifier's
  watchdogs from Harbor's budgets. #11: attachments measured and read in the environment as the agent user (through
  a link; one over 20 MiB never read). #12: eval checks the acting agent's sign-in only. #13: a pipeline error on a
  run again keeps what the runs before it spent. #14: the agent logs reach a shared verifier only (documented).
- Tests no longer depend on prepared data (the eval test writes its own task set); the affected tests pass in a
  clean checkout.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 8 (#114): the phase's end as confirmed steps, commands by their own word, the runner's checks

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Round 8 triage: out-of-scope items (#8, #12, #17, #18) left as the runner had them

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 9 (#114): no worker request as root, archives extracted member by member, the phase's end confirmed

- P1 #1: a worker's upload is never promoted to root by its name: the deployment names the agent declares are made
  the agent user's by the bridge itself as the phase starts (before any agent command runs), and every worker
  upload stays the agent user's own.
- P1 #2: archives another process made (the worker's upload on this machine, the container's tar of a download)
  are written member by member: regular files and folders only, no absolute or `..` names, files created anew
  (O_EXCL | O_NOFOLLOW) through folders the extractor made, whatever Python's tarfile filter does.
- #3 a transfer still running or failed at the phase's end fails it; #4 a stream that began and cannot be read
  fails it; #5 the deployment record (.hermit-deploy.json, Claude's .mcp.json) and an image's context link are
  provisioned; #6 credentials are removed though an earlier step failed; #7 Harbor ending the phase during the
  answer's delivery keeps its spend; #8 a command whose end is unknown is ended at the phase's end; #9 a zombie is
  not a live client; #10 the host kills the driver's client groups when it has to kill the driver; #11
  keep_containers holds on the fallback too; #12 the loopback check knows `cli` is Codex; #13 an attachment's bytes
  are read between marks of their own.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Harbor tasks back to the simple design: no worker, no proxy, the runner's end of the phase (#114)

- A task in Harbor's environment runs its agent in this process (host_pipeline): every command and CLI runs in the
  task's container, the agent's loop here with the HarborEnvironment. Deleted: hermit/environment/remote.py
  (EnvironmentServer, ProxyEnvironment, the host-side extraction of a worker's archives), the sandboxed agent's
  environment server, the worker's environment, the agent-user transfer rules (as_agent) and the deployment and home
  provisioning they needed; Hermit's uploads are its provisioning again, root's copy given to the agent user.
- The end of the agent phase at the runner's level: each command runs under setsid, its group leader's pid in a file
  (validated by the command's word on its command line); at done or Harbor's timeout the bridge cancels the requests
  in flight, kills each group still running as root with one check it is gone, removes the registered credential
  files, copies Hermit's logs, and returns; a kill or a removal that fails is infra (no verifier). Transfers hold one
  lock with the phase's end. A CLI's stream is written to /logs/agent as it runs (the runner's stream lived there).
  Deleted: the per-step bounded/confirmed machinery, the stream keeping, the transfer drain, the environment marks.
  Kept: Harbor's Trial.run, the compose clients' process groups killed on cancellation, keep_containers, the warm
  start, the disk gate, the cleanup.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Harbor: the agent's setup within 20 minutes by default; any failure of the phase's end is infra (#114)

- environment.harbor.setup_timeout_sec defaults to 1200 s (Harbor's agent setup, a CLI's install: its own 360 s
  default ran out on all three attempts of a real run under load); a setting still wins, and the start's watchdog
  follows it.
- A failure of the agent phase's end of any kind is HarborDriverError (no verifier, infra), never the agent's 0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Round 10: confine the adapter's code, no implicit sign-in, keep cut-off usage

- A Harbor task set's agent call confines the adapter's own code as the worker
  did: its context tool's Session and its exports' render run in the strict
  adapter sandbox (hermit.adapter.session.CONFINED, per call).
- A CLI in Harbor's environments needs a model binding or a login (the
  runner's refusal), and a remote session never copies this machine's sign-in.
- The usage a CLI reported before Harbor's timeout cut it off is read from the
  stream it left in the trial's agent logs and counted.
- stop() calls end_clients() with its current signature; the host's forced
  teardown signals only the driver's own group; kill_groups keeps a pid file
  until its group is confirmed gone; a failed kill exec still removes the
  credentials; a download of a link is refused; the log a record names is
  scrubbed of the run's keys.
- Docs: no worker or proxy for Harbor; Hermit's judge on the host.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* kill_groups in POSIX sh: no `--` for dash's kill; dead processes gone

CI's Ubuntu runner runs `sh` as dash, whose kill reads `--` as a pid ("Illegal
number: -"), so the group was never killed and the check failed. The kill is
now `kill -KILL -<pgid>` (POSIX: dash, busybox ash, bash), and a process in
state X counts as gone with a zombie. Harbor's main container runs the line
under bash, so trials were not affected; a sidecar or another shell would be.

The tests run kill_groups under sh and bash with a PID 1 that never reaps
(Python, as a container's `sleep infinity`), count the group's live processes
from /proc (no ps), and once more in Debian's python:3.12-slim (dash).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
…read through a view (#115)

* Tasks and environments beside the agents: hermit/task, hermit/environment, hermit/common

The first of four stacked changes toward Harbor-format tasks run by Hermit's own
agents (design: "Hermit agent 与 benchmark 重构设计", 第二阶段). Structure only:
no behavior changes, the requests every agent sends are byte for byte main's.

- hermit/environment/: the environments move out of hermit/agent/ to the top
  level, since a task declares what it needs of one and a judge may run in one;
  hermit.agent.environment forwards for one release.
- hermit/task/: task.py becomes a package. Task holds a public half (what the
  acting agent is handed, what a run records as its input) and a private half
  only the judge reads; Benchmark is a task set and its splits; verify.py wraps a
  judge as a verifier of a Task. Every name importable from hermit.task before
  still is.
- hermit/common.py: RunOutput and load_callable, shared by agents and tasks so
  that neither imports the other.
- The run (hermit/trainer/rollout.py) hands the agent task.given() and the
  verifier the whole task; the trainer, eval and hermit check read a Benchmark.
  The validation file the learner gets holds each task's public half.
- tests/core/test_layers.py: tasks and agents never import each other; the
  environments import neither. tests/core/test_task.py covers the new types.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* hermit check: report the probe task by its id (a Task, not a row)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 1: a bare judge reads the whole row again; records keep the row's id

- judge_verifier calls the judge with Task.row() (the public half and `_private`), as
  run_one did before tasks were typed; the agent still gets only the public half.
- Records keep the row's own id (7, not "7"); run_tasks keys its results by task.
- run_one/run_tasks keep the `judge` parameter name; the Verifier protocol states
  `takes_task`; agent-layers.md shows hermit/environment/ beside hermit/agent/.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* rollout: the record's id in one name (line length)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* eval: a record's numeric id finds its task (after the record kept the row's id)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Task directories in Harbor's format, their own verifier, the bwrap backend; OfficeQA as task directories

The second of four stacked changes (design: 第二阶段). A task set may now be a
directory of task directories in Harbor's format, run on Hermit's own sandbox
(B-class: Harbor's format, not Harbor's runtime).

- hermit/task/directory.py: Harbor's task format read without Harbor (moved from
  hermit/harbor/dataset.py, which imports it): the instruction (line endings
  kept), settings and metadata public; the task's directory and tests private.
  Benchmark loads `data.tasks: <directory>` (`data.rollouts: k`).
- hermit/task/verify.py: a task's own verifier, `judge: {path:
  hermit.task.verify, name: script}`, run in the judge's sandbox as Harbor runs
  it: tests at /tests, a fresh /logs/verifier, reward.txt or reward.json, and
  feedback.txt for learning methods. A judge is called with Task.row() (the row
  plus `_private`); an agent still gets only the public half.
- hermit/environment/spec.py and bwrap.py: EnvironmentSpec (the run config's
  `environment` section) and the bubblewrap backend the task sandbox is built
  by; a task asking for more network or memory than the run gives is refused.
- examples/officeqa: the task set moves to utils/benchmark/ (prepare.py writes
  task directories; judge.py, task-info/, splits, data.json, seed adapter); the
  agents read each task as the row they always read (officeqa_fields.py), so the
  prompts are byte for byte the same.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 1: OfficeQA's methods on the new layout; task directories as Harbor reads them

OfficeQA (the methods the move missed): ReHarness's launcher tests the tasks
directory; RRSI's rows (byte-identical to before on the real data) and AHE's come
from the task directories through officeqa_fields.question_rows; RRSI and AHE
score with utils/benchmark/judge.py; Harness²'s source hook reads a directory
row; OPRO shows the instruction; utils/.gitignore keeps the old paths ignored;
AHE's real-data tests find the prepared data again.

Task directories: a step's [verifier] settings come before the task's;
[verifier.env] is set for the script (${NAME} resolved as Harbor does);
reward.json takes a boolean as Harbor does; a task's [agent] timeout_sec bounds
its calls and the judge waits at least the verifier's timeout_sec; an allowlist
task's hosts and (with the script verifier) its verifier's network are checked.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 2: a view of OfficeQA's rows for agents and learners; Harbor's verifier defaults and allowlists

- data.view: <file.py>:<function> turns a task directory's row into the row the
  task set's agents and learners read. OfficeQA's configs view each task as
  SkillOpt's row, so records and what a learning method reads (GEPA, ACE, OPRO,
  SkillOpt) are the same as with its tasks.jsonl (246/246 on the real data);
  OPRO's question_field is `question` again.
- What a task asks of its environment is kept in its private half (`needs`) and
  checked before each call (hermit.trainer.rollout asks a sandboxed agent's
  `refuses`), viewed or not.
- As Harbor: a verifier's timeout_sec defaults to 600; [verifier.environment]
  implies a separate verifier; each phase's allowlist (a step's first) is its own;
  wildcard hosts are refused under an allowlist run (the proxy matches hosts as
  written).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 3: a step's verifier as Harbor resolves it; a viewed task keeps its timeout; finite rewards

- A step's verifier mode and environment are resolved as Harbor's
  verifier_mode.py does (explicit mode, else an environment implies separate,
  the step's level before the task's, else shared), and the verifier's network
  and allowlist come from that effective environment.
- A step's tests/ are laid over the task's in the judge's sandbox (its test.sh
  first), as Harbor uploads them.
- A task's agent timeout is kept in its needs, which the run hands the sandboxed
  agent, so a data.view that leaves it out of the row still bounds the call.
- reward.txt must hold a finite number, as in Harbor.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Harbor environment backend: a task's Harbor environment as a Hermit Environment

hermit/environment/harbor.py (HarborEnvironments, HarborEnvironment) and its driver,
hermit/environment/harbor_driver.py, run under Harbor's own venv: Harbor's trial of the
task with its nop agent, driven a piece at a time (start, exec, upload, download,
install, verify, stop) over JSON lines. hermit/harbor/docker.py moves unchanged to
hermit/environment/harbor_docker.py (the disk gate, image cleanup, compose_down).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Hermit drives each Harbor task: the agent works in the task's Harbor environment, Harbor verifies

- Acting agents take the task's environment: run(task, adapter, environment=...).
  The CLI Base Agents (Codex, Claude Code) run in a remote environment through its
  exec (the home rendered here and uploaded, the CLI installed by Harbor or mounted);
  the CLI acting agent deploys the adapter for a session there (hooks rewritten
  relative to the workspace); the answer-only agent (agent.name: answer) replaces
  HermitAnswer.
- hermit/environment/remote.py: the environment served to the agent's worker in the
  task sandbox (exec, upload, download, install; never verify).
- The run (hermit/trainer/rollout.py) starts each task's environment
  (environment.harbor), hands it to the agent, verifies with Harbor's verifier and
  stops it; judge hermit.task.verify:environment reads the verdict.
- Deleted: hermit/harbor/runner.py, harborside/ and the runner's parts and tests.
- Configs: every agent.name: harbor config moves to agent.base + environment.harbor.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Tests for the Harbor environment, the served environment, remote CLI sessions, the answer agent and the run

tests/core/test_harbor_environment.py: the driver's protocol with a stand-in driver,
the warm start's calls, the verifier's verdicts, Codex and Claude Code in a remote
environment, the CLI acting agent's staging, the answer agent, the environment server
(no verify), the run's start/verify/stop, the environment judge; and a real Docker +
Harbor test (skipped without them). OfficeQA's Harbor config and judge, and the
example tests, follow the new path.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Docs and changelog: Harbor environments, every agent Hermit's

docs/framework/task.md's Benchmarks section describes the new path (environment.harbor,
the driver, the warm start, the agents in a remote environment, what is refused); the
agent layers, configuration, sandbox design, evaluation, model backends,
troubleshooting and compatibility pages, examples/README.md, the benchmark READMEs and
the OfficeQA recipe follow it. CHANGELOG: Unreleased, Breaking.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* A ReactAgent in a task's environment: its tools run there, the loop without the environment's budget

Codex's install in a remote environment: with a version set, Harbor's installer is
asked (it keeps a CLI already at that version).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* rollout: wrap a long line (after rebasing onto PR 2)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Harbor environments keep the runner's infra semantics: run again, and never the agent's 0

- hermit/environment/harbor_errors.py: the runner's errors.py ported (Harbor's error
  policy, INFRA_CAUSES, GRADED, HOST_FAILURE, the verifier that ran no test, classify),
  with its verifier_timeout_verdict and verifier_verdict, NO_RETRY, the defaults
  (infra_retries 2, infra_wait_s 300 s), and the causes of an environment that did
  not start or a session that never did.
- HarborEnvironments.verdict reads the verifier's answer as the runner read a trial;
  EnvironmentUnavailable carries its cause (disk, sandbox_failed, harbor_failed,
  harbor_timeout).
- hermit/trainer/rollout.py: a rollout that ends infra runs again in a fresh
  environment up to environment.harbor.infra_retries times after infra_wait_s, never
  for disk or model_rejected; every run's usage is the rollout's (extra.attempts, the
  runner's _attempts); one that stays infra is marked unscored and the environment
  judge refuses it (Unscored: a judge_error record, as the runner's judge did).
- report.kind counts output.status infra again; tests, task.md, evaluation guide,
  examples README and CHANGELOG follow.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 4: multi-step tasks refused on this path; the verifier's workspace and memory

- A multi-step task directory is refused here (its steps share one environment in
  order, which a per-call sandbox does not give); PR 4 brings that.
- The script verifier's workspace defaults to the directory the agent worked in,
  when its output names none.
- A separate verifier's own memory_mb is checked against the run's limit.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 1 (#114): restore what the Harbor runner did for a session in a task's environment

- The CLI on PATH after Harbor's install (nvm's, ~/.local/bin), the prompt on stdin
  and Harbor's own instruction as the prompt (byte for byte, as the runner gave it);
  the task's own agent budget, also for the sandboxed worker's deadline; a session
  is not resumed (refused).
- Every sign-in file leaves the environment before the verifier (confirmed, from /);
  the ChatGPT plan's usage reports come back; web search off on a restricted agent
  network; the login's hosts allowed; a login's plan waited for before the
  environment starts; the disk gate and the infra pause stop with the run.
- Codex's lost sign-in, provider failure, plan limit and refused configuration read
  from its notices (codex_stream, ported from signin.py); a refused configuration is
  model_rejected (never run again); provider and environment failures across the
  worker boundary are infra; the CLI's provider allowance is the rollout's one.
- Usage of every attempt counted; status/failure_cause beside the verdict (AHE);
  hooks' audit and state collected; python3 ensured for hooks; the image's context
  file merged within Codex's cap, an unreadable one infra.
- Driver: agent commands still running are ended before the verifier (not a server
  an ended one left); logs copied without following links; installs under the
  baseline network policy, within Harbor's setup timeout, Debian bullseye sources
  repaired; a timeout without `timeout` still ends the command; requests written by
  a writer thread. Harbor task directories are run secrets. Answer agent: Harbor's
  default /app/answer.txt.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* A CLI install that failed spent nothing: its run again's cost is 0, not unknown

Seen in the Harbor-install smoke: Harbor's Codex install ran past Harbor's setup timeout
on a 1-CPU task (infra, run again, then served); the failed run's cost made the eval's
cost unknown, where the runner counted a setup failure as 0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Task directories for the Claude subscription example; write_task writes one

The first benchmark of refactor 4/4: examples/claude_subscription's nine tasks are
task directories in Harbor's format under utils/benchmark/ (with the judge, the
splits and task-info), each read through data.view as the row {id, prompt} it was
in tasks.jsonl, so the prompts and records are byte for byte the same.
hermit.task.directory.write_task writes a task directory (instruction.md, task.toml
with [metadata] and other tables) whose values tomllib reads back as given.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* ASPIRE's engine, Robosuite and LIBERO: tasks as task directories

Refactor 4/4: examples/aspire_robosuite's and examples/libero's tasks (each task,
its verifier and its rerun) are task directories in Harbor's format under their
utils/benchmark/ (prepare.py, judge.py, task-info/, splits), asking for no network
and judged on the host, where the simulator is. aspire_fields.py, every config's
data.view and the readers on the host (sim_service.py and stage_docs.py --tasks),
reads each back as the row it was in tasks.jsonl.

Parity: 21/21 Robosuite rows on its four configs and 630/630 LIBERO rows on its
fourteen, both presets, every split file byte for byte; the simulator service's
rows the same. A short LIBERO run before and after (prompt_only/smoke.yaml, eval of
rerun-libero_object-t00, no adapter): the actor's prompt.txt (2471 bytes) and the
service's registration identical.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* RoboDojo-Sim's tasks as task directories

Refactor 4/4: examples/robodojo_sim's forty-two tasks are task directories in
Harbor's format under utils/benchmark/ (prepare.py, judge.py, task-info/, splits),
each asking for no network and judged on the host. robodojo_fields.py, every
config's data.view and the simulator service's reader (dojo_service.py --tasks),
reads each back as the row it was in tasks.jsonl: 42/42 rows identical on all seven
configs, both presets, and the eight split files byte for byte. RoboDojo and Isaac
Sim are not on this machine: no smoke.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* ServeLearnBench's episodes as task directories

Refactor 4/4: examples/servelearnbench's scenario export writes one task directory in
Harbor's format per episode under utils/benchmark/ (with prepare.py, judge.py,
windows.py, windows.json, task-info/ and the splits), the stamped instruction as
instruction.md. slb_fields.py, both configs' data.view and windows.py's reader,
reads each back as the row it was: 1161/1161 rows identical (retail_l3, from the
old prepare's export) on both configs, the eleven split files byte for byte. Each
episode stays one task (the adapter is what carries across the stream), so no
multi-step task is needed. The package is not installed here: no smoke.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* CL-Bench's database and poker streams: tasks as task directories

Refactor 4/4: each question (hand) of examples/clbench_database's and
examples/clbench_poker's ordered streams is a task directory in Harbor's format
under utils/benchmark/ (with prepare.py, judge.py, task-info/ and the per-order
splits); clbench_fields.py and poker_fields.py, every config's data.view, read each
back as the row it was. Each row stays one task: the stream's state is the host's,
carried by the agent's own loop in its run directory, so no multi-step task is
needed. The runners read the venue texts from utils/benchmark/task-info/mwm.

Parity: 200/200 (database) and 520/520 (poker) rows identical on all three configs
each, from the old prepare run against ~/code/clbench, splits byte for byte. Short
prompt_only runs before and after (order 0's first question, first hand; GLM 5.3
Flash): the recorded input row and the host's first query identical (1184 and 1350
bytes), the evidence README identical.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 2 (#114): the boundaries the Harbor runner kept, and its reading of a session

- Security: no variable an agent's command names touches the driver or the host's
  compose client (each travels under a HERMIT_ENV_<n> carrier and is exported in the
  container); a by_fd secret (Claude Code's token) is moved into a pipe by the
  outermost shell before `timeout` or the CLI start, so no persistent process holds it.
- An agent command's process tree is killed (descendants of its marked processes,
  whatever their environment); installs exclude agent commands while the setup
  network policy is open; Harbor's scrub of the trial's files runs on verify and stop,
  and the verifier's exception is scrubbed before Hermit reads it.
- The driver's teardown kills a driver whose writer is stuck; the working directory
  is read from a marked line; environment-operation and host failures are typed
  infra (host_failed never run again).
- The runner's session_verdict ported: a configuration refused before any token is
  model_rejected, a refused bill or spent quota ends the run (usage_limit), a prompt
  too long or the model's refusal stays the agent's result; Codex's failure text read.
- A failed attempt keeps its usage (the agent returns it as an infra run); the plan's
  refused sessions' usage counts; a rotated ChatGPT sign-in comes back however the
  exec ended; the image's context file is read once across provider retries.
- On a login the plan's wait and slot are held around the whole environment (the CLI
  takes none inside; a refused session holds the plan and the task runs again, not
  counted); the answer and react agents keep the task's budget on a host pipeline.
- verifier_reward_raw beside the status (AHE); eval rows keep the rollout's outcome;
  environment.harbor.* overridable on the command line.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test: an unused import

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* AgentDyn and AgentDojo: each protocol's frozen cases as task directories

Refactor 4/4: AgentDyn's and AgentDojo's prepare scripts, with their judge,
task-info/ and DATA.md, move to utils/benchmark/; each protocol directory now holds
its cases as task directories in Harbor's format (tasks/<id>/, the case's row as
task.toml's [metadata], its id in its place among sorted keys), cases.txt (the rows'
order, which a later stage and --full-test read), the splits and the manifest, whose
tasks_sha256 is still the SHA-256 of the rows as JSON lines (DATA.md's hashes hold).
cases.py's case_row is every config's data.view; the later stages, report.py,
compare.py and the experiment runner read a protocol's tasks/ (or a tasks.jsonl
prepared before); the experiment snapshot copies a task set's directory and hashes
it whole. Each case stays one task (the agent loop is AgentDyn's own).

Parity (AgentDyn and recorded collections are not on this machine): the old and the
new pipelines run on the same synthetic collections, the native audit set to all
eligible: expanded-v2, its full/ inventory, thirty-v1, all-groups-v1, full-test-v1,
prepare.py's protocol, AgentDojo's comparison-v1 and prepare.py's protocol give the
same rows (54 to 275 each), splits byte for byte and manifests (tasks_sha256 too);
all eleven configs read the old rows through data.view. No smoke.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Docs: the examples' task sets in utils/benchmark/; unique test names

examples/README.md's layout names utils/benchmark/ (prepare.py writing task
directories, judge.py, task-info/), and its rule on the examples whose agent owns its
loop says where they now stand: task directories in Harbor's format, each row of a
stream one task, on Hermit's own sandbox or the host. docs/framework/task.md names
write_task and that route. The examples' new tests get names of their own
(pytest refuses two test modules of one basename), and write_task's float case reads
more plainly.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 1 (#115): an AgentDyn case keeps its nulls; RoboDojo's README on its format

- Each case is kept in its task.toml as one JSON line ([metadata] case): a clean
  case of an old tasks.jsonl (injection_task_id null, injections {}) reads back
  as it was, key order and hash unchanged; the nullable fixture is back.
- RoboDojo's README: its task set is in Harbor's format; its tasks run on the
  host simulator service, not in Harbor's environments.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 3 (#114): the runner's results kept on the long paths

- A CLI session's provider failure runs the task again in a fresh environment
  (environment.harbor.busy_retries/busy_wait_s, the runner's allowance over a task's
  runs, not infra_retries), never again in the environment the session changed.
- An adapter that failed before any session (deployment, per-task context, context over
  Codex's cap) is the agent's result scored 0 (adapter_failed, context_over_limit), not
  left out of the denominator; a minimal row's context is computed from Harbor's
  instruction.
- The image's linked context file is read through the environment and the link
  replaced (real Docker test); host failures are host_failed (not run again); a
  teardown failure after the verdict keeps it; a lost sign-in stops new work before
  the task's teardown; the verifier watchdog follows Harbor's own verifier timeout.
- The install gate admits and counts a command at once and reopens on every path;
  an admitted session that drew credits holds the plan; a refused session without
  a plan of the run's ends the run; local sign-in copies go though a take-back fails.
- ReAct keeps what its loop spent on a timeout and asks no follow-up after it; the
  answer agent's whole run is bounded, its log descriptor closed, model.max_retries
  honoured, attachments sent once, its endpoint's refusals read as the runner read
  them (bill: usage_limit; configuration: model_rejected; input: its result).
- An acting agent's model a CLI reaches is refused for Harbor environments; run_agent
  and RRSI's smoke see a Harbor rollout's outcome; SWE-bench's token mean filters by
  outcome.status.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 4 (#114): race-proof log copy, confirmed kills, the runner's Codex cap

- The agent's logs reach the trial's folder through O_NOFOLLOW directory descriptors and O_EXCL temp files renamed
  into place: an entry swapped for a link while the copy runs is replaced, never written through.
- A failed restore of the agent phase's network policy breaks the environment: no further agent command, install or
  ensure; _ensure_python3 re-raises it.
- kill_marked confirms its kill (exit 1 when a process is left); _drain raises before the verifier, keeps the mark
  and tries again; cleanup runs from /.
- The sandbox proxy refuses network, hidden-path and readable limits instead of ignoring them.
- A hook audit that cannot be read keeps the session's result and usage (hooks_audit_missing).
- Codex in a task's environment keeps the runner's project_doc_max_bytes of 262144 unless the config sets one.
- Answer: adapter context failures scored 0 (adapter_failed), preparation failures infra, the answer moved into place
  only within the budget; ReAct's answer-tag follow-up is bounded; a call given up in flight counts as a call with
  unknown cost.
- The disk gate rechecks the stop flag (between gates, after them): nothing starts, the rollout ends as the stopped
  run's.
- logins_of treats an acting agent in environment.harbor as contained; the benchmark-file form of environment.harbor
  takes nested overrides; config-fields and examples/README updated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 5 (#114): the Hermit-side fixes that outlive the bridge redesign

- The credentials an agent command is given are scrubbed from the logs handed to the verifier and from the trial's
  files (the driver records them from each exec's env and by_fd).
- A ChatGPT sign-in comes back as the runner took it (signin.take_back, ported into codex_stream): a private
  download, a regular file of at most 64 KiB with both tokens, stamped at most SKEW/OTHER_CLOCK ahead; a failed local
  write of the refresh keeps the finished session.
- The sandboxed agent provisions and takes back the configured login_root.
- A Codex rejection keeps its head (the status) for the verdict, as the runner did.
- A remote session that reached its CLI but reported no usage counts as an unknown cost.
- A refused session whose plan hold cannot be written pauses HOLD_UNWRITTEN_S before the next environment.
- reward_key is authoritative (none there is infra, the runner's classify); verifier_redact restored; an adapter
  failure has no pass on a benchmark without pass_reward.
- The answer agent extracts as the runner did (group 1 only when it took part) and sends max_tokens only when given.
- Docs: the agents that work in a Harbor environment; report docstring; the Codex cap comment.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Harbor runs the whole trial: one bridge agent relays Hermit's commands (#114)

Harbor's own Trial.run() now owns each task's trial: the environment's start, its capability check, the agent's
setup (Harbor's setup timeout, the baseline network policy, the agent user), the agent phase (its network policy,
user and timeout), the verifier in its mode, the scrub, result.json and the stop. The driver no longer drives the
trial a piece at a time.

- HermitBridge (a Harbor BaseAgent in the driver, imported by AgentConfig.import_path) has no agent logic: its setup
  installs what the acting agent declares (Agent.installs: the CLI unless `bin` names one, probed first unless a
  version is pinned; python3 for an adapter's hooks) with Harbor's own installers; its run starts the phase, relays
  Hermit's exec/upload/download until `done`, Harbor's agent timeout or a cancellation, and ends every command it
  started that still runs (their Docker clients too, the kill confirmed) before it returns. A kill not confirmed
  fails the run (HarborDriverError: no verifier, infra). Hermit's logs reach /logs/agent scrubbed.
- Deleted: the Hermit-side network-policy switching and install gate, the install/ensure/verify ops, the separate
  drain step, the piecewise start/verify/stop. HarborEnvironment.install reports what setup installed; verify sends
  `done` and reads the trial's verdict. Harbor refusing a task's skills, MCP servers or Windows is unsupported_task
  (never run again). The agent's own budget ends 60 s before Harbor's agent timeout so its logs arrive first; a
  phase Harbor ended (AgentPhaseOver) is read as the trial's verdict.
- Agents stay Harbor-free: they use Hermit's Environment API; only the rollout of a Harbor task set starts a session.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Bridge: transfers as the agent user, its uid:gid, limits refused, no missed death (#114)

- r5 #2: an upload or download the agent's process asks for through the sandbox's proxy (as_agent) runs with the task
  agent user's own permissions when it is not root (staged as root, copied into place as that user; a tar made as
  that user, unpacked as data); root only for Hermit's own provisioning. PermissionError, not infra.
- r5 #9: the bridge reads the uid:gid the agent runs as (an image's own USER too), and Hermit's provisioning is
  chowned to it, confirmed (the runner's _owner).
- r5 #8: the CLI agent keeps an environment's other limits when it cuts the time, and refuses network, hidden or
  readable limits in a remote environment; the sandbox's proxy refuses them before it is served.
- r5 #13: the driver's reader publishes its death, and replies, under the lock a request registers under.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* One implementation each: usage ledger, credential lifecycle, deadline, retry states (#114)

- hermit/agent/model/usage.py: Ledger (an agent run's usage: every attempt, every call given up in flight, every
  session cut off before its usage; cost unknown when any entry's is; what an empty ledger reports is the caller's
  word) and Attempts (a rollout's runs and their sum, per-attempt records: the runner's _attempts). The workspace
  agent, ReAct, the answer agent and base React's total use it; rollout._attempts and the hand sums are gone.
- hermit/agent/base/credentials.py: held(provision, take_back, remove), the sign-in's lifecycle with its finally:
  the CLI's remote session and the sandboxed agent's worker both go through it.
- hermit/agent/base/deadline.py: one absolute Deadline (remaining, cut, a chat client bound to it): ReAct's and the
  answer agent's model calls, the CLI's limits before its session and the workspace's deadline_s.
- hermit/trainer/rollout.py: Retries, the rollout's one retry and admission state machine with its states named
  (DONE, INFRA, PROVIDER_BUSY, PLAN_WAIT, STOP); one run in a fresh environment is _environment_run.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 6 (#114): the agent phase's end, outside Harbor's timeout, does what it must

Structural fix for the end of the agent phase (findings 1-7, 11): the bridge's end-of-phase work runs as a task of
its own that its run awaits through any cancellation (Harbor's agent timeout can no longer cut it short), bounded by
END_PHASE_S; anything it cannot confirm raises HarborDriverError, so Harbor runs no verifier. It: lets transfers in
flight finish, cancels the commands, ends their clients and kills their processes (running and earlier unconfirmed
kills alike), keeps a running command's stream (exec `stream=`, tee'd there as it runs) in the agent logs under its
name, removes the credential files the CLI registered before its upload (Environment `secret`, confirmed), and copies
Hermit's logs scrubbed. A cancelled wait on any Docker client kills it (compose up after a build timeout too); stop
answers outside the lifecycle lock, ending the start's clients before the teardown.

- The verdict: a verifier that failed after the agent's timeout is read by its infra policy (the driver keeps every
  exception Harbor recorded or skipped); a result.json Harbor could not write falls back to the trial's in-memory
  result, and no verdict at all is the driver's error.
- A disabled verifier is refused; Windows refused at the trial's creation is UnsupportedTask.
- Agents: install discovery never raises on an adapter (the deployment scores it); a deployment Harbor's phase end
  cut short is not adapter_failed; python3 for hooks is best effort at setup; the claude-code alias installs Claude
  Code; the answer agent's log is a folder with the runner's answer.json; eval checks the sign-in before replacing
  results. Docs: the judge's place with Harbor, installs is built-in only, the driver's intro, AHE's trace note.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Ported from the runner: a mounted CLI's checks, Claude's key, attachments, infra usage (#114)

- A mounted binary (`bin`) serves only when it runs there as the pinned version (any, when none is) and the agent's
  user may execute it (the runner's _use_mounted): linked onto PATH, with the release's rg and this machine's CA bundle
  for an image that has none (harborside.codex's _after_mount); else the bridge's setup installs the CLI and the
  session runs the installed program (bin_served, through the proxy too).
- Claude Code in a task's environment gets no raw key variable of its binding: it reads its key from its own
  (session_env), as the runner's launch left it.
- The answer agent warns when it leaves out an attachment over 20 MiB.
- A run that is not the agent's result after a session ran, with no usage reported, has an unknown cost and token
  count (the runner's usage None), never 0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 7 (#114): no verifier after an unfinished end, the agent's requests as the agent

- P1 #1: the phase's end that does not finish (its bound, or any other way) is HarborDriverError, read from the
  shield's own exception too: Harbor runs no verifier after it (_close_phase, tested past its bound).
- P1 #2: nothing the worker asks for runs as root: credential files are removed as the agent user, an upload's
  missing folders are made by that user (cp -f replaces what it may replace), and the CLI's home there is the
  bridge's own provisioning, declared by the agent (Agent.installs `home`), made and given to the user at the phase's
  start.
- #3: each command writes its pid at its start, the root of what kill_marked ends where root cannot read another
  user's environment (no CAP_SYS_PTRACE). #4: the commands are stopped as the phase closes, before a pending
  transfer is let finish. #5: the driver's environment drops the ambient credentials the runner's harbor_env dropped.
  #6: a stop Harbor acknowledged with its compose projects still there takes them down by name (a local, deleted
  Docker environment). #7: cp -f. #8: one absolute deadline from the phase's start (deadline_at, through the proxy):
  every command's time and ReAct's and the answer agent's model calls come off it. #9: a timed-out rollout's log is
  the trial's agent logs (the stream the bridge kept, stderr too). #10: the start's and a separate verifier's
  watchdogs from Harbor's budgets. #11: attachments measured and read in the environment as the agent user (through
  a link; one over 20 MiB never read). #12: eval checks the acting agent's sign-in only. #13: a pipeline error on a
  run again keeps what the runs before it spent. #14: the agent logs reach a shared verifier only (documented).
- Tests no longer depend on prepared data (the eval test writes its own task set); the affected tests pass in a
  clean checkout.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 8 (#114): the phase's end as confirmed steps, commands by their own word, the runner's checks

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Round 8 triage: out-of-scope items (#8, #12, #17, #18) left as the runner had them

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review round 9 (#114): no worker request as root, archives extracted member by member, the phase's end confirmed

- P1 #1: a worker's upload is never promoted to root by its name: the deployment names the agent declares are made
  the agent user's by the bridge itself as the phase starts (before any agent command runs), and every worker
  upload stays the agent user's own.
- P1 #2: archives another process made (the worker's upload on this machine, the container's tar of a download)
  are written member by member: regular files and folders only, no absolute or `..` names, files created anew
  (O_EXCL | O_NOFOLLOW) through folders the extractor made, whatever Python's tarfile filter does.
- #3 a transfer still running or failed at the phase's end fails it; #4 a stream that began and cannot be read
  fails it; #5 the deployment record (.hermit-deploy.json, Claude's .mcp.json) and an image's context link are
  provisioned; #6 credentials are removed though an earlier step failed; #7 Harbor ending the phase during the
  answer's delivery keeps its spend; #8 a command whose end is unknown is ended at the phase's end; #9 a zombie is
  not a live client; #10 the host kills the driver's client groups when it has to kill the driver; #11
  keep_containers holds on the fallback too; #12 the loopback check knows `cli` is Codex; #13 an attachment's bytes
  are read between marks of their own.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Harbor tasks back to the simple design: no worker, no proxy, the runner's end of the phase (#114)

- A task in Harbor's environment runs its agent in this process (host_pipeline): every command and CLI runs in the
  task's container, the agent's loop here with the HarborEnvironment. Deleted: hermit/environment/remote.py
  (EnvironmentServer, ProxyEnvironment, the host-side extraction of a worker's archives), the sandboxed agent's
  environment server, the worker's environment, the agent-user transfer rules (as_agent) and the deployment and home
  provisioning they needed; Hermit's uploads are its provisioning again, root's copy given to the agent user.
- The end of the agent phase at the runner's level: each command runs under setsid, its group leader's pid in a file
  (validated by the command's word on its command line); at done or Harbor's timeout the bridge cancels the requests
  in flight, kills each group still running as root with one check it is gone, removes the registered credential
  files, copies Hermit's logs, and returns; a kill or a removal that fails is infra (no verifier). Transfers hold one
  lock with the phase's end. A CLI's stream is written to /logs/agent as it runs (the runner's stream lived there).
  Deleted: the per-step bounded/confirmed machinery, the stream keeping, the transfer drain, the environment marks.
  Kept: Harbor's Trial.run, the compose clients' process groups killed on cancellation, keep_containers, the warm
  start, the disk gate, the cleanup.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Harbor: the agent's setup within 20 minutes by default; any failure of the phase's end is infra (#114)

- environment.harbor.setup_timeout_sec defaults to 1200 s (Harbor's agent setup, a CLI's install: its own 360 s
  default ran out on all three attempts of a real run under load); a setting still wins, and the start's watchdog
  follows it.
- A failure of the agent phase's end of any kind is HarborDriverError (no verifier, infra), never the agent's 0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Round 10: confine the adapter's code, no implicit sign-in, keep cut-off usage

- A Harbor task set's agent call confines the adapter's own code as the worker
  did: its context tool's Session and its exports' render run in the strict
  adapter sandbox (hermit.adapter.session.CONFINED, per call).
- A CLI in Harbor's environments needs a model binding or a login (the
  runner's refusal), and a remote session never copies this machine's sign-in.
- The usage a CLI reported before Harbor's timeout cut it off is read from the
  stream it left in the trial's agent logs and counted.
- stop() calls end_clients() with its current signature; the host's forced
  teardown signals only the driver's own group; kill_groups keeps a pid file
  until its group is confirmed gone; a failed kill exec still removes the
  credentials; a download of a link is refused; the log a record names is
  scrubbed of the run's keys.
- Docs: no worker or proxy for Harbor; Hermit's judge on the host.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* kill_groups in POSIX sh: no `--` for dash's kill; dead processes gone

CI's Ubuntu runner runs `sh` as dash, whose kill reads `--` as a pid ("Illegal
number: -"), so the group was never killed and the check failed. The kill is
now `kill -KILL -<pgid>` (POSIX: dash, busybox ash, bash), and a process in
state X counts as gone with a zombie. Harbor's main container runs the line
under bash, so trials were not affected; a sidecar or another shell would be.

The tests run kill_groups under sh and bash with a PID 1 that never reaps
(Python, as a container's `sleep infinity`), count the group's live processes
from /proc (no ps), and once more in Debian's python:3.12-slim (dash).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
…ent, host faults, usage) (#116)

* Final review fixes: verifier script's own environment, host faults, usage

- A task's own verifier script (hermit.task.verify:script, shared or
  separate) runs in a sandbox built from the task's environment only: never
  the acting model's key variable, provider hosts or forwarded ports.
- The strict adapter sandbox is preflighted at startup for a Harbor task set
  (the runner's sandbox_fault), and a deployment or context that fails because
  it stopped starting is staging_failed (infra, run again), not a scored 0.
- The answer agent counts the usage an HTTP error response reports, on the
  scored result and on the infra end.
- A ledger of only unreported sessions or calls has unknown tokens, not 0.
- data.view must keep each task's id.
- An unreadable optional feedback.txt leaves the verifier's reward.
- AgentDyn/AgentDojo snapshots freeze data.view like the agent and judge.
- Docs: Hermit's Harbor judge on the host; the verifier script's sandbox.
- Minor: the RRSI trials judge loads its score module once; a dead fixture.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* PR 116 review: edges of the final fixes

- sandbox_fault: bubblewrap missing under a required sandbox is a fault (the
  runner refused it at startup), at startup and at run time.
- The answer agent's session construction (its context tool's server) is
  inside the guard: a sandbox that stopped starting there is infra too.
- A staging failure on this machine (the strict sandbox not starting, an
  OSError such as a full disk) is an infra output, staging_failed: run again,
  nothing spent, so a later passing run keeps a known cost.
- data.view reads a copy of the row: a view that rewrites its input cannot
  change the task or merge two tasks.
- The verifier script's sandbox does not apply sandbox.hide (the agent's), as
  a trusted judge's never did; the model key, hosts and ports stay out.
- What a failed call billed (its error's usage) is counted on the failed
  record outside Harbor too, across the task sandbox's worker.
- Snapshots refuse a module-form data.view (read live where it is installed).
- Docs: the verifier script's boundary stated as the grants it lacks.
- Minor: the RRSI judge loads under a lock; the 503 test asserts its attempts
  and their combined cost.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* PR 116 round 2: staging on this machine, a terminal failure's spend

- A CLI agent's staging folders that cannot be made (a full disk) in a task's
  own environment are staging_failed (run again, nothing spent), as the
  runner guarded its job's folders with the staging; an answer agent whose
  context server's files cannot be made (OSError) is staging_failed too.
- A pipeline's own failure after an infra run again sums every attempt's
  spend through Attempts, the failed one's included when it said.
- An error body's all-zero usage (a session's placeholder) is no report, as
  hermit.tools.failed_usage reads it.
- Docs: a shared judge's verifier script runs in a sandbox of its own.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
@haizhongzheng
haizhongzheng merged commit b65a5c6 into main Nov 20, 2026
1 check passed
@haizhongzheng
haizhongzheng deleted the task-environment-refactor branch November 20, 2026 15:01
haizhongzheng added a commit that referenced this pull request Nov 20, 2026
…eam, refuse multi-step tasks as unsupported (#119)

* Docs: Harbor design page, sandbox and benchmark checklists after the refactor

After #117 the contributor checklists and the task tutorial still described
the framework before it. Add docs/developer/harbor-design.md with three
diagrams (the processes, a trial in order, CLI and loop agents) and who does
each step, how the agent phase ends and the outcomes; let the sandbox
checklist accept hermit.environment and count hermit/environment/ as core;
add a benchmark checklist (task directories, public and private halves, what
a task asks, parity, a real run) and say when it applies; point the task
tutorial and the developer index to task directories; the dual-review
reviewer is gpt-6.2-sol.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Review fixes: design page and checklists checked against the code; scrub a CLI's live stream; refuse multi-step tasks as unsupported

The independent review (/dual-review, 9 rounds) checked every claim of the
Harbor design page, its diagrams and the checklists against the code and
Harbor 0.25.0, and corrected what was false: Local is not confinement and
sandbox evidence must show the isolation probe passed; only the bubblewrap
task sandbox refuses a task the run cannot give, and every backend states its
effective timeouts; Harbor's trial order, what is infra, which retries apply,
what passes through the bridge, CLI install and reuse, the separate verifier.

Two driver fixes the maintainer asked to include:
- A CLI's output kept in /logs/agent as it ran reached a shared verifier
  unscrubbed when Harbor's timeout ended the phase before Hermit's logs were
  copied. The phase's end now scrubs each stream there, best effort, in a
  local Docker environment (never a failure of the phase's end).
- A multi-step task was refused as sandbox_failed and run again; it is now
  refused as unsupported_task and never run again.
Each has a test that fails on main.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant