[andylin-hao]

Description

Adds robots/r2/, an RPent extension that lets an LLM planner operate a Robbyant R2 (two 7-DoF arms, parallel grippers, a pan-tilt head RGB-D camera, and two wrist RGB-D cameras) without a VLA.

The R2 support itself lives in RLinf and is in review in RLinf/RLinf#1745 (feat/r2-robot), which this PR also needs for R2Connection.hold(). RPent's rlinf extra pins release/v0.4, which has no R2, so the docs install RLinf from that branch until it is released.

How it fits together

• robots.r2.env_server runs on the robot. It builds RLinf's R2Env locally (no Ray), which owns the controller connection, fault recovery, the rest pose, IK, and the fixed-rate setpoint stream. R2AgentEnv (rlinf_env.py) is a thin adapter that checks every planner command against a site config before the robot moves: a per-arm workspace box in base_link, maximum translation and rotation per call, and a joint-speed cap. Refusals return ok: false with the reason; incomplete moves also return the controller's health.
• RPent and the planner run on a workstation and attach to the server over HTTP (--env-endpoint). RPent never starts the server, because it owns hardware and runs under the operator's control on the robot. No new extra is needed on the workstation.
• Tools: move_to, move_delta, rotate_delta, set_gripper, look, go_home, and move_base (registered only when the site allows chassis motion; off by default). The readers are view_env_state, view_camera_meta, back_project, and segment (registered only when SAM3 is configured).
• back_project turns pixels into base_link points from the step's depth, configured intrinsics and OpenCV distortion, and a camera pose. The pose is computed from the robot's own URDF (which carries its hand-eye calibration) and the measured waist, neck, and arm joints, so it stays correct as the head and arms move.
• segment uses RPent's shared SAM3 service to find an object by a short name or a pixel on it. It returns the mask's [row, col] box and centre and an overlay image for the planner to check. When the camera has depth and calibration, it also returns the object's median base_link point and 5th-95th percentile extent. RPent attaches to --sam3-endpoint, or starts a local server when SAM3_CHECKPOINT_PATH is set, and waits for SAM3 before connecting the env client, because connecting resets the arms.
• Operator verdict. R2 has no automatic success check. request_operator_verdict asks the operator on RPent's terminal (success, failure, continue, or abort, with optional notes). finish is refused until the operator answers, and its status is the operator's verdict, not the planner's claim. A motion after a verdict discards it, an abort refuses further motion, and every answer is saved in operator_events.json. Without a terminal, the run finishes as an unverified failure.
• Stopping mid-motion. Cancelling a planner call sends env.request_stop, which bypasses the server's write lock and calls RLinf's R2Connection.hold(): the arms, head, and chassis brake within their acceleration limits, and the interrupted tool reports stopped: true.
• Mock mode. env_server --mock drives RLinf's simulated R2 controller (it ships with RLinf's tests, so it needs an RLinf source checkout). tests/e2e_tests/r2 uses it for an env component test and the scripted CLI policy chain.
• python -m robots.r2.diagnostics checks a running server without moving the robot.
• Tasks: 0 primitive smoke test, 1 tabletop pick and place. --task-language replaces the instruction.

Reuse and divergence. Env lifecycle, kinematics, motion streaming, and fault handling are RLinf's. Two things RLinf does not provide live in RPent:

• Camera poses: RLinf's R2 kinematics covers the waist and arms only, so RPent computes them from the URDF chain.
• Planner-facing bounds: the per-call and workspace limits that make LLM commands safe to execute.

Shared changes

• run_scripted_policy_chain (tests/e2e_tests/common.py) accepts calls between the action and finish, and starts the CLI with stdin from /dev/null so a scripted session never waits for terminal input. Existing callers are unchanged.
• R2's Dashboard spec sets external_env, because its env server always runs on the robot. Without it the Dashboard rejected R2, and the CLI required a TTY even when no operator is attached.

Known limitations

• The R2 does not publish camera intrinsics (checked ROS topics, parameters, and the sensor configs on the robot). Until a site fills them in, back_project names what is missing, and the planner can only make small moves checked against images. The head lens is wide-angle, so its distortion coefficients are needed too.
• The env server does not authenticate clients; the docs say to bind it only to a network the operator controls.
• The workspace boxes in config/example.yaml are starting values around the measured rest pose and must be reviewed per site.
• The docs page has no robot photo yet.

An AI assistant (Claude Code) was used to write this change.

Testing

• pytest tests/unit_tests/robots/r2 -v: 38 passed. Covers:
  • the adapter against a fake RLinf connection: workspace and per-call refusals, relative moves composing with the measured TCP, IK failure, gripper and neck ranges, base gating, millimetre depth, and URDF camera poses;
  • site-config validation and URDF chains;
  • back_project, including inverting the distortion model;
  • segment against a fake SAM3 client: 3D location, overlay, missing calibration, misses, and exactly one of prompt or point;
  • the toolkit registering optional tools only when configured;
  • the operator verdict: finish refused before an answer, status taken from the verdict, a motion discarding it, abort, an invalid reply, and no terminal;
  • stopping: the adapter reporting a stopped motion, a stop reaching a motion that holds the server's write lock over HTTP, and cancelling a toolkit call sending the stop;
  • HTTP loopbacks through the real facade, client, runtime hook, and diagnostics, and attaching to a SAM3 server.
• pytest tests/unit_tests: 1496 passed, 14 skipped, 10 errors. The errors are test_pi05_vla_server_contracts.py and the RoboDojo env/flash contract tests, which import torch; torch is not installed by the [test] extra in my environment. They are unrelated to this change.
• pytest tests/e2e_tests/r2 -v: 2 passed, against RLinf's simulated controller (RLinf at b72e9c3d, CPU only). The component test observes, moves an arm (final error 0.9 mm in simulation), turns the head, sets a gripper, stops a move_to halfway through a stop request from a second connection, and returns home. The policy chain runs the public CLI with the offline planner: move_delta, request_operator_verdict (no terminal, so unavailable), then finish, recorded as failure. I ran these in a scratch environment that combines RPent with RLinf's dependencies; with RLinf absent, the fixture skips them.
• pre-commit run --all-files: passed.
• sphinx-build -W --keep-going for source-en and source-zh: no warnings.

Manual verification

On an R2 (21-DoF body, grippers; Ubuntu 24.04 aarch64, /opt/py310), with no motion commanded:

• Installed RPent with the documented recipe: a --system-site-packages venv, with its site-packages first on PYTHONPATH after entry_env.sh.
• Started robots.r2.env_server with config/example.yaml and connected R2EnvClient(reset_on_connect=False).
• Results:
  • Metadata and measured state were returned; health was OK in system state MANUAL.
  • All three cameras returned RGB plus depth registered to color: head 1280×800 with 77% valid depth, wrists 640×480.
  • URDF camera poses were plausible: the head about 1.75 m up looking along +x; the wrist cameras looking along the gripper with the fingers in view.
  • One observation took 0.61 s over localhost HTTP, and the server shut down cleanly.

Not yet verified: any motion through RPent (task 0) and stopping a motion on the robot; back_project and segment 3D accuracy on hardware, which need intrinsics; and segment with a real SAM3 checkpoint on R2 images. I'll run task 0 with an operator at the emergency stop before marking this ready.

Checklist

• My code follows the code style of this project.
• I have updated related documentation when needed.
• I have added tests to cover my changes when needed.
• All new and existing tests passed. (All new tests pass; the 10 existing errors above need torch.)

🤖 Generated with Claude Code

[read-the-docs-community]

Documentation build overview

📚 RPent | 🛠️ Build #35538208 | 📁 Comparing 946a5d1 against latest (0e406bb)

  🔍 Preview build  

7 files changed · + 1 added · ± 6 modified

+ Added

• rst_source/real_world_robots/r2.html

± Modified

• index.html
• rst_source/get_started/overview.html
• rst_source/guides/cli.html
• rst_source/real_world_robots/index.html
• rst_source/real_world_robots/so101.html
• rst_source/real_world_robots/yam.html

[read-the-docs-community]

Documentation build overview

📚 RPent-cn | 🛠️ Build #35538209 | 📁 Comparing 946a5d1 against latest (0e406bb)

  🔍 Preview build  

7 files changed · + 1 added · ± 6 modified

+ Added

• rst_source/real_world_robots/r2.html

± Modified

• index.html
• rst_source/get_started/overview.html
• rst_source/guides/cli.html
• rst_source/real_world_robots/index.html
• rst_source/real_world_robots/so101.html
• rst_source/real_world_robots/yam.html