Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,20 @@ Changes are grouped by software version. Entries under Unreleased describe work

### Breaking

- The robot benchmarks and ASPIRE no longer share code: a benchmark holds nothing of a method, and ASPIRE nothing of
a benchmark. The engine Robosuite and LIBERO run on (the simulator service, `bin/sim`, the neutral task functions,
the judge, the task writer, the row view) is `examples/cap_harness/`, which RoboDojo's service also builds on;
`examples/aspire_robosuite/` is `examples/robosuite/`, the benchmark's tasks and an `aspire/` folder. ASPIRE's
session (its fix-loop and verifier prompts, the evolutionary search, the facts the coordinator reads) is
`hermit/agent/evolver/methods/aspire/actor.py`, built on the engine's task functions (`agent.params.engine`), and its
direct-chat actor `direct.py` (`agent.name: hermit.agent.evolver.methods.aspire.direct:AspireActor`). A benchmark's
`prepare.py` writes its tasks and their reruns (`rerun_of`) only; ASPIRE's verifier rows are derived by `rows.py`,
which an ASPIRE folder's `tasks.sh` runs before the run, into that folder's `tasks/` and `splits/`. The service's
attempt budget is a row's `max_programs`, and the judge scores a row's `eval_seeds` (a verifier row's are its own
five); a verifier's verdict rule is ASPIRE's `collect`, which withholds the program unless `verified` (`not_scored`).
The prompts the sessions see are unchanged, byte for byte. A task function may be named `package.module:function`.
LIBERO's `prompt_only/` configs drop `evosearch: false`.

- The old import paths of moved modules are gone, with no forward: 0.1's, which 0.2 forwarded (`hermit.agents`,
`hermit.evolver`, `hermit.llm`, `hermit.backends`, `hermit.trajectory`), and 0.2's `hermit.agent.base.direct` and
`hermit.agent.base.container`, and `hermit.agent.environment` (never released). Import instead:
Expand Down
7 changes: 4 additions & 3 deletions docs/framework/task.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,9 @@ agent:

`codex`, `claude-code`, `pi`, `mini-swe-agent`, `opencode`, `dsh` and `hermes` act with a coding-agent CLI in a
workspace of Hermit's (or, for Codex and Claude Code, in a task's own Harbor environment: [Benchmarks](#benchmarks)):
a task gives what is its own, in configuration and, where its workspace is
built in code, in functions it names (`file.py:function`); the agent does the rest. OfficeQA's Codex runs are one such
a task gives what is its own, in configuration and, where its workspace is built in code, in functions it names
(`file.py:function`, or `package.module:function` for a method that brings its own session, as ASPIRE's
`hermit.agent.evolver.methods.aspire.actor`); the agent does the rest. OfficeQA's Codex runs are one such
configuration (`examples/officeqa/utils/task.py`):

```yaml
Expand Down Expand Up @@ -456,7 +457,7 @@ of your own is written as task directories by your `utils/benchmark/prepare.py`
goes into `tests/`, uploaded after the agent ran, so the answer is never where the agent can read it; see
`examples/officeqa/utils/harbor/build_tasks.py`). A task set of task directories that Hermit runs in its own sandbox or
on the host, with a judge on the host (`judge.mode: host`, no `tests/`), is what the examples whose agent owns its loop
use (`examples/aspire_robosuite/`, `clbench_*`, `servelearnbench/`, `agentdyn/`, ...; [Task directories](#task-directories)).
use (`examples/robosuite/`, `libero/`, `clbench_*`, `servelearnbench/`, `agentdyn/`, ...; [Task directories](#task-directories)).
A task set is checked without a model by running a command of a known answer in each task's environment and verifying
it (`tests/core/test_harbor_environment.py` does this against Docker).

Expand Down
61 changes: 37 additions & 24 deletions docs/methods/aspire.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,21 @@ learns only the library: the actor's harness and prompts are fixed.

```yaml
template: aspire # the skill library: skills/README.md and skills/<topic>.md
data: {train: all.txt, val: verify.txt, test: rerun.txt, step_size: 1, shuffle: false}
data: # the benchmark's tasks with ASPIRE's verifiers (tasks.sh: rows.py)
{tasks: tasks, train: splits/all.txt, val: splits/verify.txt, test: splits/rerun.txt, step_size: 1, shuffle: false}
agent:
base: codex # the actor: ASPIRE's fix-loop subagent prompt, verbatim
params: {prompt: ../utils/agent/task.py:prompt, prepare: ../utils/agent/task.py:prepare,
answer: ../utils/agent/task.py:answer, collect: ../utils/agent/task.py:collect}
params: # ASPIRE's task functions, over the engine's
prompt: hermit.agent.evolver.methods.aspire.actor:prompt
prepare: hermit.agent.evolver.methods.aspire.actor:prepare
answer: hermit.agent.evolver.methods.aspire.actor:answer
collect: hermit.agent.evolver.methods.aspire.actor:collect
engine: ../../cap_harness/utils/agent/task.py
codex: {codex_sandbox: danger-full-access, shell_env: all, web_search: false} # the task sandbox is the boundary
environment: # the task sandbox: its code, ASPIRE's prompts and docs, the service
read: [../utils/agent, $ASPIRE_DOCS, ...]
environment: # the task sandbox: the engine's code, ASPIRE's prompts and docs
read: [../../cap_harness/utils/agent, $ASPIRE_DOCS, ...]
env: [ASPIRE_SIM_URL]
judge: {path: ../utils/benchmark/judge.py} # on the host: Stage 2
judge: {path: ../../cap_harness/utils/benchmark/judge.py} # the engine's, on the host: Stage 2
sandbox: {enabled: true, enforce: required}
evolver:
name: aspire # the coordinator: a Codex session with ASPIRE's guide
Expand All @@ -40,16 +45,22 @@ privileged access (simulator internals, success predicates, reflection), and ref
No existing template fits: `prompt-only` is one file; `continual-harness` carries a policy, sub-agents and memory
ASPIRE does not have; `reharness` retrieves per instance from an embedded bank.

The example's task functions (`examples/aspire_robosuite/utils/agent/task.py`) give the actor its workspace in the
task sandbox: `bin/sim`; ASPIRE's API and artifact docs and the benchmark's page (Robosuite's, or LIBERO's); the scene-snapshot program; and a page that maps
the template's paths and commands onto the workspace. The sandbox shows nothing else of ASPIRE's checkout: not its
shipped library, worked solutions or examples. It has no GPU, so the simulator runs on the host as a service
(`utils/agent/sim_service.py`), which `bin/sim` reaches on a forwarded loopback port; the session registers there
before the agent starts (its workspace and task id: one session per rollout), and the service runs only the seeds of
the task row it reads on the host, the unseen check once and a verifier's two programs. The judge (`utils/benchmark/judge.py`,
on the host) runs the selected program once on the held-out seeds: ASPIRE's Stage 2, the coordinator's in ASPIRE. The
verifier is the same actor on the task's verifier row (`verify-<task>`, `data.val`): ASPIRE's verifier prompt, five
seeds, two attempts, scored on its own seeds with its own verdict.
The tasks run on the [cap-harness engine](https://github.com/Infini-AI-Lab/hermit/tree/main/examples/cap_harness),
which knows nothing of ASPIRE; ASPIRE brings its own session on it. ASPIRE's task functions
(`hermit/agent/evolver/methods/aspire/actor.py`, named by module in the configs) build on the engine's
(`agent.params.engine`): the engine gives the actor its workspace in the task sandbox (`bin/sim`; ASPIRE's API and
artifact docs and the benchmark's page, Robosuite's or LIBERO's; the scene-snapshot program), and ASPIRE adds its
assignment, the evolutionary search's page and a page that maps the template's paths and commands onto the
workspace. The sandbox shows nothing else of ASPIRE's checkout: not its shipped library, worked solutions or
examples. It has no GPU, so the simulator runs on the host as the engine's service (`utils/agent/sim_service.py`),
which `bin/sim` reaches on a forwarded loopback port; the session registers there before the agent starts (its
workspace and task id: one session per rollout), and the service runs only the seeds of the task row it reads on the
host, the unseen check once and, for a row with an attempt budget, that many programs. The engine's judge
(`utils/benchmark/judge.py`, on the host) runs the selected program once on the held-out seeds: ASPIRE's Stage 2, the
coordinator's in ASPIRE. The verifier is the same actor on the task's verifier row (`verify-<task>`, `data.val`),
which ASPIRE derives from the benchmark's tasks before a run (`rows.py`, run by the method folder's `tasks.sh`): ASPIRE's
verifier prompt, five seeds, an attempt budget of two programs, scored on its own seeds and only with its own verdict
`verified` (ASPIRE's `collect` withholds the program otherwise).

## The step

Expand Down Expand Up @@ -98,11 +109,12 @@ is the evaluation on the test split (`rerun-<task>` rows, the same seeds), compa

The Codex actor's settings (`agent.params`) are `aspire_root`, `aspire_docs` (the docs as the launcher stages them), `max_steps` and
`evosearch` (default true; false leaves out the evolutionary search: its part of the prompt, its page in the workspace, and its
program as the scored one); its seeds are the task rows'
(`utils/benchmark/prepare.py`, which the launcher runs: `ASPIRE_EVAL_SEEDS` for the held-out ones), and the simulator's
program as the scored one) and `engine` (the engine's task functions, relative to the config); its seeds are the
task rows' (the benchmark's `utils/benchmark/prepare.py`, which the launcher runs: `ASPIRE_EVAL_SEEDS` for the
held-out ones), and the simulator's
environment and workers are the simulator service's (`ASPIRE_SIM_ENV`). The direct comparison actor
(`run_direct.yaml`) reads `debug_seeds`, `workers`, `aspire_root` and `sim_env` from `agent.params`, a task row's own
seeds winning; the held-out seeds are the task rows' for both actors (the judge reads them there). The verifier's rows
(`run_direct.yaml`) reads `engine` (the engine's `sim.py`), `debug_seeds`, `workers`, `aspire_root` and `sim_env` from
`agent.params`, a task row's own seeds winning; the held-out seeds are the task rows' for both actors (the judge reads them there). The verifier's rows
carry their own seeds.

## The authors' implementation against this one
Expand All @@ -124,12 +136,13 @@ carry their own seeds.
| Library history | `<file>-changelog.md` beside each file | `state/changelog.md` | the adapter holds only the library |
| Phase 3 | rerun of tasks below 80% whose library changed | the test split reruns every task | the evaluation measures it per task |
| Models | Claude Opus 4.6 (and others) | GLM 5.3 Flash in every role | the first-evaluation row of the framework |
| Perception | SAM3 | SAM3, or any service with its wire format (`utils/seg_service/`: Grounding-DINO + SAM) | SAM3's weights are gated |
| Perception | SAM3 | SAM3, or any service with its wire format (the engine's `utils/seg_service/`: Grounding-DINO + SAM) | SAM3's weights are gated |
| Evolutionary search (§2.3) | a separate evosearch subagent for tasks the fix loop leaves below 90%: K = 8 candidates per round from the top 3, stop at 80% or 5 rounds, Stage 2 on its best | the same template, verbatim, as Part 2 of the actor's session; the judge runs Stage 2 on `evosearch_best_code.py` | a decision: one session keeps the task's rollout one call |
| Not ported | — | BEHAVIOR-1K, the real robot | scope |

`examples/aspire_robosuite/aspire/run_direct.yaml` replaces the Codex actor with a structured loop of `direct` calls
(`utils/agent/agent.py`), which sees a digest of each run instead of the files: a comparison, not the method.
`examples/robosuite/aspire/run_direct.yaml` replaces the Codex actor with a structured loop of `direct` calls
(`hermit/agent/evolver/methods/aspire/direct.py`, on the engine's `sim.py`), which sees a digest of each run instead of
the files: a comparison, not the method.

## What it is not

Expand All @@ -140,7 +153,7 @@ Phase 3 does).

## Example

[`examples/aspire_robosuite/`](https://github.com/Infini-AI-Lab/hermit/tree/main/examples/aspire_robosuite): the
[`examples/robosuite/aspire/`](https://github.com/Infini-AI-Lab/hermit/tree/main/examples/robosuite/aspire): the
seven Robosuite tasks of ASPIRE's harness.
[`examples/libero/aspire/`](https://github.com/Infini-AI-Lab/hermit/tree/main/examples/libero/aspire): LIBERO's five
suites and LIBERO-Pro's eighty tasks on the same engine.
10 changes: 6 additions & 4 deletions examples/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# Examples

Each folder is one task. Inside it the three things a task owner provides are kept apart, and the configs of every
method that has been run on the task sit beside them:
Each folder is one task, except [`cap_harness/`](cap_harness/), the engine that Robosuite and LIBERO run on (RoboDojo's
simulator service builds on it too), and [`claude_subscription/`](claude_subscription/), the configs of the Claude
subscription guide. Inside a task's folder the three things a task owner provides are kept apart, and the configs of
every method that has been run on the task sit beside them:

```text
examples/<task>/
Expand Down Expand Up @@ -36,8 +38,8 @@ a `.hermit/CARD.md`.
| [`clbench_database/`](clbench_database/) | The CL-Bench database exploration stream (accuracy) | `CLBenchDatabaseAgent`: metaworld's host seat in CL-Bench's loop | [`prompt_only`](clbench_database/prompt_only/), [`mwm`](clbench_database/mwm/), [`reharness`](clbench_database/reharness/) |
| [`clbench_poker/`](clbench_poker/) | The CL-Bench exploitable-poker stream (bb/hand) | `CLBenchPokerAgent` | [`prompt_only`](clbench_poker/prompt_only/), [`mwm`](clbench_poker/mwm/), [`reharness`](clbench_poker/reharness/) |
| [`servelearnbench/`](servelearnbench/) | A ServeLearnBench scenario: serving stream, per-window tests | `ServeLearnBenchAgent`: the benchmark's own loop | [`prompt_only`](servelearnbench/prompt_only/), [`mwm`](servelearnbench/mwm/) |
| [`aspire_robosuite/`](aspire_robosuite/) | The Robosuite tasks of ASPIRE's Code-as-Policy harness: development and held-out seeds, task success | Codex with ASPIRE's fix-loop subagent and verifier prompts (`utils/agent/task.py`), GLM 5.3 Flash, ASPIRE's robot execution engine in its own venv through `bin/sim` | [`aspire`](aspire_robosuite/aspire/) |
| [`libero/`](libero/) | LIBERO's five suites and LIBERO-Pro's `_swap` and `_task` perturbations, a program per task in ASPIRE's Code-as-Policy harness: development and held-out seeds, task success | Codex writing the program, on the `aspire_robosuite/` engine (ASPIRE's LIBERO venv, seeded initial states), with a method's prompt or a method-neutral one | [`aspire`](libero/aspire/), [`prompt_only`](libero/prompt_only/) |
| [`robosuite/`](robosuite/) | The Robosuite tasks of ASPIRE's Code-as-Policy harness: development and held-out seeds, task success | Codex writing the program, GLM 5.3 Flash, on the [`cap_harness/`](cap_harness/) engine (ASPIRE's robot execution engine in its own venv through `bin/sim`), with a method's own session (ASPIRE's fix-loop subagent and verifier) | [`aspire`](robosuite/aspire/) |
| [`libero/`](libero/) | LIBERO's five suites and LIBERO-Pro's `_swap` and `_task` perturbations, a program per task in ASPIRE's Code-as-Policy harness: development and held-out seeds, task success | Codex writing the program, on the `cap_harness/` engine (ASPIRE's LIBERO venv, seeded initial states), with a method's own session or a method-neutral prompt | [`aspire`](libero/aspire/), [`prompt_only`](libero/prompt_only/) |
| [`robodojo_sim/`](robodojo_sim/) | RoboDojo-Sim's forty-two bimanual manipulation tasks: a program that controls the robot, developed on one layout set and scored on two others, task success (a first implementation: its smoke run on the real simulator, no training run yet) | Codex writing `policy.py` (`utils/agent/task.py`), GLM 5.3 Flash, RoboDojo evaluating it as an XPolicyLab policy whose program runs in a sandbox of its own | [`prompt_only`](robodojo_sim/prompt_only/) |
| [`agentdyn/`](agentdyn/) | AgentDyn's frozen benign and injected cases (Shopping, GitHub, DailyLife), clean utility and attack success rate | `bridge.py:run`: AgentDyn's native agent loop, with the defense bundle's hooks around every tool call | [`evosafe`](agentdyn/evosafe/) (EvoSafeHarness) |
| [`agentdojo/`](agentdojo/) | Official banking, Slack, travel and workspace suites; frozen 57/54/406 local protocol, native utility/ASR and errors | `task.py:run`: native AgentDojo loop through the shared AgentDyn supervised runtime | [`evosafe`](agentdojo/evosafe/), [`she`](agentdojo/she/), completed Lightning/Codex reference comparisons |
Expand Down
1 change: 0 additions & 1 deletion examples/aspire_robosuite/.gitignore

This file was deleted.

20 changes: 0 additions & 20 deletions examples/aspire_robosuite/utils/agent/aspire_fields.py

This file was deleted.

Loading
Loading