Repository navigation
Robotics: the cap-harness engine, the benchmarks and ASPIRE no longer share code - #125
Merged
Merged
Conversation
… share code A benchmark now holds nothing of a method, and ASPIRE nothing of a benchmark. - examples/cap_harness/: the engine Robosuite and LIBERO run on (simulator service, bin/sim, neutral task functions, judge, task writer, row view, the launcher with a method folder's tasks.sh hook); RoboDojo's service builds on it. - examples/robosuite/ (was aspire_robosuite): the Robosuite tasks and an aspire/ folder. A benchmark's prepare.py writes its tasks and their reruns (rerun_of). - hermit/agent/evolver/methods/aspire/: actor.py (ASPIRE's fix-loop, search and verifier sessions over the engine's task functions, agent.params.engine; the verdict rule withholds an unverified verifier's program), rows.py (ASPIRE's verifier rows, derived before a run), direct.py (the direct-chat comparison). - The service's attempt budget is a row's max_programs; the judge scores a row's eval_seeds. A task function may be named package.module:function. - The Codex sessions' prompts and workspaces are unchanged byte for byte; ASPIRE's header names the CLI's context file (CLAUDE.md for Claude Code); Robosuite gains a Claude Code smoke config. - Tests: the engine's in examples/cap_harness/utils/tests (selected through ENGINES), ASPIRE's in tests/methods/aspire. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ase service, the direct actor's scope; Claude's Bash timeout - The engine and Robosuite READMEs and actor.py said every session is Codex; they now name Claude Code too (smoke_claude.yaml). - RoboDojo's README and two comments still called its base "the ASPIRE example's service"; it is the cap-harness engine's. - direct.py's docstring said any cap-harness task; its prompts and runs are Robosuite's. - smoke_claude.yaml raises Claude Code's Bash timeout (2 min by default, 10 at most) through the session's environment, since a bin/sim run-batch can take longer. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…tted smoke_claude.yaml (task sandbox, login: claude) needs CLAUDE_CODE_OAUTH_TOKEN, which the machine the PR was validated on does not have, so it never ran in the form it was committed in. ASPIRE's header still names the CLI's context file (CLAUDE.md for Claude Code); a Claude Code config can follow once it can run. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The {context_file} line existed only for a Claude Code session, and this PR no
longer ships a Claude Code config. ASPIRE's prompt pages are now byte-identical
to main's; the Claude Code wording can come with a Claude Code config.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The index said every folder under examples/ is one task; cap_harness/ is the engine Robosuite and LIBERO run on, and claude_subscription/ holds a guide's configs. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
The robot benchmarks and the ASPIRE method shared code. ASPIRE's fix loop, evolutionary search and verifier lived in the Robosuite benchmark's task functions, and the benchmark's
prepare.pywrote ASPIRE's verifier rows. So another method on LIBERO had to strip ASPIRE's parts (evosearch: false, deletingtask.md), and LIBERO ran on another example's code. Now a benchmark holds nothing of a method, and ASPIRE nothing of a benchmark.examples/cap_harness/(new)bin/sim, neutral task functions (prepare,collect), the judge, the task writer (utils/benchmark/tasks.py), the row view (fields.py), andlaunch.sh, which sources a method folder'stasks.sh. RoboDojo's service subclasses its service. Its README has the engine's setup.examples/robosuite/(wasaspire_robosuite)prepare.py), task-info, and anaspire/folder of configs.examples/libero/aspire/and aprompt_only/folder. The prompt-only configs use the engine's workspace and record as they are.hermit/agent/evolver/methods/aspire/actor.pyholds ASPIRE's acting sessions (fix-loop and verifier prompts, the search, the facts the coordinator reads), built on the engine's task functions (agent.params.engine).rows.pyderives ASPIRE's verifier rows before a run, through the engine's task writer.direct.pyis the direct-chat comparison actor.Interfaces:
prepare.pywrites its tasks and their reruns (rerun-<id>,rerun_of). Reruns stay with the benchmark, because a test split may not share ids with the training split, and prompt-only uses them too.verify-<id>) come fromrows.py, whichaspire/tasks.shruns into that folder'stasks/andsplits/.max_programs(it used to follow the row's role). The judge scores a row'seval_seeds, so a verifier row sets them to its own five, withscore_label: verifierfor the feedback.collectand direct actor (not_scored): an unverified verifier's program is withheld.hermit/agent/acting/workspace.pyaccepts a task function namedpackage.module:function, matched by a strict dotted regex, so ASPIRE's configs namehermit.agent.evolver.methods.aspire.actor:prompt.tests/selection.pygainsENGINES: a change toexamples/cap_harness/runs the groups of the methods on its benchmarks.tests/core/test_environment.pyreads the source of apackage.module:Classagent without importing it.Two behavioural notes:
main's (prompts/*.mdcompared file by file). A{context_file}line for Claude Code was reverted along with the dropped Claude config. Run A ran with that line, so its Claude sessions were toldCLAUDE.md; at this head a Claude Code session would be toldAGENTS.md.smoke_claude.yamlwas dropped: it needsCLAUDE_CODE_OAUTH_TOKENto run as committed (task sandbox,login: claude), and it could not be run here. Run A below used Claude Code with a run-only config.Validation
Parity
main. Checked 13 on the test checkout, and 8 on ASPIRE's real templates at the pinned commitfa00f0ef(216 KB): actor, no search, max_steps, LIBERO, verifier ×2, neutral ×2.prepare()workspaces are identical in 5 cases (ASPIRE robo / no search / LIBERO / verifier, and prompt-only); only the per-run path and port inbin/sim.jsondiffer.bin/sim_cli.py, which is copied into every workspace, is unchanged.Tests
python -m pytest --changed --base origin/mainonecc8cd94(the review fixes and a merge ofmain). The two later commits only dropsmoke_claude.yamland revert the{context_file}line, and the affected tests pass (86, then 45). selected every group: 1797 passed, 148 skipped, 7 failed. All 7 fail the same way on a cleanmainworktree in this environment:test_the_moved_modules_old_paths_are_gone×4;harness2package is missing:test_harness2_on_pi×2;test_rrsi_sees_a_harbor_rollouts_failure...×1.tests/methods/aspire,examples/cap_harness/utils/tests, RoboDojo's tests,test_layersandtest_environment: 93 passed.Entry points
robosuite/aspire,libero/prompt_only,libero/aspire) run end to end with stub providers and a wrapper in place of the finalhermit train:prepare,tasks.sh, the staged docs, the simulator service up and healthy, and the train command.hermit check --no-probepasses on one config per folder against the real ASPIRE checkout.GPU, no model
bin/simon a development seed, a held-out seed refused, the unseen check once, the engine'scollect, and the judge on held-out seeds 1-3 (a fixed minimal program scored 0/3, as expected).max_programs.Real smoke runs (Robosuite ASPIRE smoke: two tasks, ASPIRE's providers on free L40 GPUs)
Both run-only configs below were deleted after the runs; neither is committed.
verify-*rows thatrows.pyderived), the judge scored the verified programs, and it promoted entries intograsp.md,transport.mdandlocalize.md.actor.promptbuilt the prompt,preparebuilt the workspace, and the actor ranbin/sim. The launch records reportprobe.passed: trueforrollout-1andlearner-1/-2.Found, not fixed (predates this PR; worth issues)
environment.runner: hostputs its workspace under.logs/workspaces, outside the directory the simulator service serves, so the service refuses it. Run A passedagent.params.workspaces=$ASPIRE_OUT/actor-logs/rollout.evolver.yamlsetshome: [], so a coordinator without a login or binding gets no sign-in. Run A setevolver.params.home: [.credentials.json].~/.local/bin/codexis a symlink into its standalone package), needscodex-code-mode-hostbeside its real binary. Hermit's task sandbox shows only the binary, so a sandboxed Codex session has no tools. Run B setagent.params.binto the real path and added its folder toenvironment.read.Review
docs/methods/aspire.md,docs/framework/task.md,examples/README.md, the engine, Robosuite and LIBERO READMEs,tests/README.md.ASPIRE_*launcher variables unchanged.tests/core/test_layers.pypasses (the agent layer imports no task layer). The direct actor's own loop predates this PR (a comparison; cf. Move the benchmarks' own agent loops to ReactAgent subclasses (CL-Bench, ServeLearnBench, OfficeQA, AgentDyn) #122).sim.confine) are as before. Run B's launch records reportprobe.passed: true(rollout and learner); run A ran unsandboxed, by the user's choice.write_task); network and timeouts are unchanged; parity as above.smoke_claude.yamlcould not be run in its committed form (it needsCLAUDE_CODE_OAUTH_TOKEN), so it was dropped from this PR (5100ea94).direct.py's docstring overstated its scope (it is Robosuite's);bin/simrun-batches: no timeouts in run A. A Claude Code config for full runs should raise it (BASH_DEFAULT_TIMEOUT_MS/BASH_MAX_TIMEOUT_MSin the session's env).a.b:cwould be read as a module function; reruns'instruction.mdwording; the record of an unverified verifier no longer carriesfix_code(it stays infiles/);rows.py's ownseeds_of; the launcher does not check for a Claude token up front.Not changed, for the maintainer's review (agents read this text at run time):
bin/sim_cli.pystill says "a verifier's two programs";skill_reportfor every record;🤖 Generated with Claude Code