Repository navigation
feat(robolab): integrate RoboLab as an RPent simulation environment - #329
lizuojun04 wants to merge 3 commits into
Conversation
Signed-off-by: lizuojun <[email protected]>
Signed-off-by: lizuojun <[email protected]>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Documentation build overview
19 files changed ·
|
Documentation build overview
19 files changed ·
|
|
|
||
| .. code-block:: bash | ||
| bash script/install_robolab.sh |
There was a problem hiding this comment.
where is bash script/install_robolab.sh? And is better to align with other env, just add uv install command
|
|
||
| .. code-block:: bash | ||
| export PI05_CHECKPOINT_PATH=/absolute/path/to/pi05_droid_jointpos_torch |
There was a problem hiding this comment.
if the ckpt is open-source, better to tell user where to get this ckpt, reference https://rpent.readthedocs.io/zh-cn/latest/rst_source/simulators/libero.html#vla
|
|
||
| .. code-block:: bash | ||
| bash script/run_robolab.sh "$PI05_CHECKPOINT_PATH" BananaInBowlTask 0 ./logs/first-robolab |
There was a problem hiding this comment.
same, this script is not existed
| ~~~~~~~~ | ||
|
|
||
| 仿真进程从已安装的 ``robolab`` 包及其相邻的 ``assets/`` 读取资源，不读取 | ||
| RPent 的 ``ref-repo/`` 或 ``data/``。另行设置 Pi0.5 检查点路径，路径可以在仓库外： |
There was a problem hiding this comment.
why mentioned ref-repo/ and data/ here? Remove it would be better
| 任务记忆 | ||
| -------- | ||
|
|
||
| RoboLab 默认使用本地记忆目录。传入 |
There was a problem hiding this comment.
are there memory for robolab now?
| return daemon, HttpRpcClient(f"http://{host}:{port}") | ||
|
|
||
|
|
||
| def _spawn_vla(args: argparse.Namespace, output_dir: Path): |
There was a problem hiding this comment.
please reuse the implementation of select_am / AMConfig.spawn_worker
| and not args.vla_endpoint | ||
| and not (args.pi05_checkpoint or get_pi05_checkpoint_path()) | ||
| ): | ||
| raise ValueError("RoboLab Dashboard requires a Pi0.5 checkpoint") |
There was a problem hiding this comment.
it's better to support use policy_backend = none in dashboard.
Description
Register RoboLab as an RPent simulation environment so users can run DROID joint-position control tasks through
--robot robolab, using the existing planners, observation and action tools, and RoboLab's native success evaluation.Main Changes
robots/robolab/, reusing RPent'sRobotSpec, Env/VLA RPC, process lifecycle, and run artifact interfaces, with CLI and Dashboard entry points.view_env_state,pi05_act, andexecute_action. The planner can execute 1–15 actions from a prediction or send a single native eight-value action. Further actions are rejected after the environment terminates.states.jsonandresult.json, preservingtrue,false, andnullseparately from the planner'sfinishstatus.Installation and Resources
.[robolab]installs policy dependencies. Isaac Sim 5.0 / Isaac Lab 2.2.0 and RoboLab use a separate simulator environment to avoid conflicts with planner dependencies.script/install_robolab.shinstalls a fixed RoboLab revision (b6cfcf6) outside the RPent repository and downloads its LFS assets. Virtual environments are created at the repository root.--pi05-checkpoint,PI05_CHECKPOINT_PATH, or the evaluation script's--checkpointargument. The installer does not download or convert checkpoints.data/,ref-repo/, or another project's checkout. Outputs default tologs/. Model weights, scene assets, virtual environments, and raw experiment artifacts are outside the submission scope.The integration uses the upstream RLinf OpenPI policy. Codex database storage follows the user's configuration; RPent does not automatically override
sqlite_home.Testing
The following checks have been completed:
ruff check --previewandruff format --check; files changed by the subsequent SQLite cleanup were checked againsphinx-build -W --keep-goingbuilds passeddata/orref-repo/, CLI, simulator, and script help worked, as did an 18-episode dry run with an explicit external checkpoint path; omitting the checkpoint produced an argument erroruv pip checkpassed for the local policy and simulator environmentsHelp and dry-run checks do not establish a complete installation on a fresh machine or real simulator execution. Under the local development constraints, no unit tests were added or run, and the repository-wide
pre-commitcommand was not run. The relevant files were checked directly with the repository's pinned Ruff version. Full CI and regression coverage for other benchmarks remain unverified.Manual Verification
Runtime Integration
A real integration check was completed on an NVIDIA A800 node:
Codex → RPent tools → Pi0.5 inference → RoboLab action execution → dual-camera observations and saved states → owned-process cleanup.The bounded smoke run executed one action, changed the joint state, saved images and states, and exited both the environment and model processes. Native success remained
null; this establishes integration behavior and is not counted as task success.The subsequent single-GPU rendering configuration was also checked through real environment initialization and reset. GPU 3 was the active rendering device, and the previous additional rendering allocation of approximately 5 GiB on GPU 0 disappeared. Other GPUs still had CUDA contexts of approximately 200 MiB.
Six-Task Evaluation
Using the upstream RLinf OpenPI implementation, six tasks were evaluated serially with environment seeds 0, 1, and 2: 15 native successes out of 18 episodes, for an 83.33% success rate.
b6cfcf6, Isaac Sim 5.0 / Isaac Lab 2.2.0gpt-6-astra,lowreasoning, local loginpi05_droid_jointpos_torch, a DROID joint-position PyTorch checkpoint0.4.0, commit2348ece3df113874671478be1360ae3115b346e30.2.2, commita560f4dd8205b8423ecd4c8a0fabb5f54140b8a0pi05_droid_polaris, 10 denoising steps, a 15-step prediction horizon, two imagesNumbers in parentheses are the simulation actions actually executed:
BananaInBowlTaskBowlStackingLeftOnRightTaskBowlStackingRightOnLeftTaskBananaThenRubiksCubeTaskReorientRedMugTaskStackYellowOnRedTaskResult verification and observations:
finish: null; the statistics do not rely on the planner's completion claims.pi05_actcalls and 11execute_actioncalls. Right-on-left bowl stacking seeds 0/1 and yellow-on-red stacking seed 1 reached native success immediately after direct gripper-opening actions. No counterfactual experiment was performed to quantify the benefit of those interventions.Interpretation and Limits
These results evaluate RPent + Codex + Pi0.5 on the selected tasks. With only three episodes per task and a planner that adjusts action chunks and intervenes directly, they do not establish full RoboLab performance, standalone VLA performance, or superiority over the native JAX implementation. A complete normalized score is unavailable, so only native success rates and execution records are reported.
The 18 episodes preceded the GPU routing fix, deployment-path cleanup, and removal of the SQLite override. They used separate SQLite directories, and multi-GPU rendering had not yet been disabled. They therefore provide real evaluation evidence from integration development, rather than a complete rerun of the final submission. Validation of the subsequent changes is described above; the final version has not been reevaluated across all 18 episodes.
Detailed experiment records in
mdnote/include per-episode timing, source hashes, and aggregation rules. Raw images, logs, weights, and local data directories are not included in the PR.Running the Evaluation
After obtaining a compatible external PyTorch checkpoint and completing the local Codex login, use the following commands to install the environments and preview evaluation. Select an available GPU and a fresh output directory:
Remove
--dry-runto start evaluation. The scripts do not automatically isolate Codex databases; configure Codex separately if multiple processes or machines share storage. Policy randomness and planner decisions are not fixed, so this command does not guarantee an exact replay of the recorded trajectories.Checklist