PredActor: Predictive Action Diffusion for Steerable Onboard Humanoid Control

Get started

The quickest way to use this repository is to ask coding agent for one of these concrete workflows:

1. Quick evaluation:

   “Install PredActor and run the quick evaluation for me.”
   

2. Collection and training:

   “Install the data collection module and run the 152-motion collection, dyeing, and training example.” 
   

3. Deployment:

   “Deploy the released checkpoint in MuJoCo, or on a wired/onboard Unitree G1, using the deployment guide.”
   

The agent and contributor guide is a compact index for coding agents. It complements the detailed human guides below.

---

Overview

PredActor augments action diffusion with an internal future-state trajectory for look-ahead guidance while retaining direct action execution. It combines classifier guidance on predicted states with classifier-free text conditioning, using proprioceptive history at runtime and sending the selected action directly to the controller.

The repository is organized around five reader paths: quick evaluation, data collection, training, model evaluation, and deployment. The release checklist below shows what is currently available.

At a glance

• Predictive direct control: jointly predicts future states and actions, keeps states internal, and executes the selected action without a separate motion-reference tracker.
• Proprioceptive inputs: conditions on onboard proprioceptive history without requiring externally estimated full-body states as policy inputs.
• CG and CFG steering: combines test-time objectives on predicted states with learned behavior conditioning.
• Rolling onboard inference: reuses the denoising horizon across control ticks and optimizes runtime for 50 Hz operation on Jetson Orin NX.
• Simulation and hardware demos: demonstrates commands, transitions, steering, and disturbance response.

Release checklist

• Evaluation: browser-based MuJoCo evaluation, G1 assets, and published checkpoints.
• Data collection and labeling: IsaacLab motion collection and MotionCLIP dataset labeling tools.
• BC training: behavior-cloning training code, configurations, and reproducibility smoke test.
• Deployment: diffusion-deploy source, G1 assets, export tooling, and runbooks.
• DAgger: interactive data aggregation and policy refinement pipeline.

---

Usage Guide

Quick Evaluation

Install uv, then run these commands on Linux:

git clone https://github.com/MasterYip/PredActor.git
cd PredActor
uv sync --locked
uv run --locked python scripts/hf_download.py --filter checkpoints
uv run --locked predactor-eval

The download command retrieves the released checkpoints from MasterYip/PredActor_Artifacts.

The last command opens http://127.0.0.1:8765/. It uses CUDA when available and otherwise falls back to CPU. The locked environment targets Python 3.10 and includes the evaluator, MuJoCo, and the browser interface; Isaac Sim and the training repositories are not required.

For a headless startup, use uv run --locked predactor-eval --headless --web-no-browser. Run uv run --locked predactor-eval --help for optional checkpoint, config, device, and Web UI overrides.

Data Collection and Auto Labeling

Download the default set of 152 G1 motion trajectories, RLTracker checkpoint, and G1 MotionCLIP checkpoint:

uv run --locked python scripts/hf_download.py --filter checkpoints motions/g1_152fk

# IsaacLab collection environment
conda create --override-channels -c conda-forge -n predactor-collection python=3.10 -y
conda activate predactor-collection
bash PredActor/collection/install.sh

# Collect 7,600 episodes from the 152 motions
bash PredActor/collection/collect_dataset.sh \
  --mode deterministic --samples_per_motion 50 --num_envs 1000 --headless

# Return to the repository uv environment, dye the collected trajectories, then train
conda deactivate
uv run python PredActor/collection/scripts/data_dyeing/data_dyeing.py

Behavior Clone Training

# Train the text-conditioned policy
uv run python PredActor/train.py \
  --cfg g1prdp_cond_diffuse.yaml --exp_name predactor_bc \
  dataset_dir="$PWD/Artifacts/dataset/g1_152fk" \
  dataset.zarr_path=motion_dyed_text_t0.01.zarr \
  training.num_epochs=50 training.seed=42 dataloader.num_workers=16 \
  logging.mode=offline

To train the condition-free PRDP baseline on the same collected/dyed zarr:

uv run python PredActor/train.py \
  --cfg g1prdp_diffuse.yaml --exp_name predactor_bc_unconditional \
  dataset_dir="$PWD/Artifacts/dataset/g1_152fk" \
  dataset.zarr_path=motion_dyed_text_t0.01.zarr \
  training.seed=42 dataloader.num_workers=16 \
  logging.mode=offline

g1prdp_diffuse.yaml does not use the text latent at inference; the same dyed zarr is used so both runs share the collection and dataset layout.

Artifact placement and environment details are in the collection, data dyeing, training, and motion encoder guides.

Mujoco Evaluation

Set run_dir to the directory printed by training:

run_dir="outputs/<run-directory>"
uv run --locked predactor-eval \
  --checkpoint "$run_dir/checkpoints/latest.ckpt" \
  --config "$run_dir/resolved_config.yaml" \
  --output_dir "$run_dir/eval" \
  --web-ui

Open http://127.0.0.1:8765/ to control the model. Add --web-perturb for mouse perturbations or --web-port 8766 to use another port.

Deployment (Experimental)

Warning

The deployment path is experimental. Please perform pre-deployment evaluation and testing in MuJoCo before attempting to run on the physical G1.

The PredActor/diffusion_deploy/ directory contains the C++ runtime, exporter, G1 assets, platform lock, and launch scripts. Read the single deployment guide for export, build, MuJoCo sim2sim, wired G1, onboard G1, and joystick operation.

The paper defaults use G1, FP32, DDIM-2, and no FK for the PRDP policy. Exported policy bundles and learned weights are supplied separately.

cd PredActor/diffusion_deploy
PYTHONNOUSERSITE=1 PYTHONPATH=.. python tools/export_policy.py \
  --checkpoint /path/to/checkpoint.ckpt --config checkpoint \
  --output models/my_policy --mode backbone --target native \
  --obs-profile G1 --no-use-fk --deploy-config deploy_config.yaml --gui
./build.sh --backend torch --download-runtime auto --with-sdk2 --jobs 4

Joystick Interface

The unified interface reads one command source in-process. A local Xbox controller at /dev/input/js0 uses the left stick for vx/vy, the right stick for wz, and the D-pad for stand, walk, jog, and squat down. LB toggles whole-body guidance; L2 selects the configured guidance gain; L3 toggles vx/vy; and R3 toggles wz. The same reader can consume the Unitree wireless bytes from LowState on a wired or onboard run.

cd PredActor/diffusion_deploy
./scripts/run.sh --config models/my_policy/deploy_config.yaml \
  --fsm-config models/my_policy/fsm_config.yaml --device cuda:0 \
  --mode control --unified-command --command-source joydev \
  --command-device /dev/input/js0 --command-mapping xbox_new

For an SSH-only session, the optional read-only status listener uses a TUI:

python tools/unified_command_dashboard.py --host 127.0.0.1 --port 9880 --mode tui

Train the motion encoder (Experimental)

Train G1 MotionCLIP with the original G1 xyz recipe: G1-retargeted AMASS motions, BABEL labels, and a trainable CLIP text encoder. Follow the motion encoder guide to prepare the training database, then run from the repository root:

uv run --locked bash PredActor/motion_encoder/scripts/train_g1_xyz.sh

This runs 100 epochs with 16 data-loader workers and writes PredActor/motion_encoder/exps/g1-model-xyz-clip/checkpoint_0100.pth.tar. It does not initialize from the released MotionCLIP checkpoint.

---

Demos

Each result is independently playable below. Visit the project website for the complete gallery.

Hardware

Text control Walk, squat down, and return to walking. predactor-text-walk-squat-walk.mp4	Behavior transitions Walk, accelerate to a jog, and transition into a squat. predactor-text-walk-jog-squat.mp4
Physical interaction Walk and stand commands under external interference. predactor-behavioral-response.mp4	Outdoor pathway Outdoor locomotion on the physical G1. predactor-outdoor-pathway.mp4

Simulation

Joystick steering Directional steering with text-selected locomotion modes. predactor-joystick-steering.mp4	Text and joystick Text-selected behavior with simultaneous directional control. predactor-sim-text-joystick.mp4
Text control Behavior selection and transitions from text commands. predactor-sim-text-control.mp4	Semantic interpolation Continuous control between semantic motion endpoints. predactor-sim-semantic-interpolation.mp4
Target tracking Classifier-guided destination following. predactor-sim-target-tracking.mp4	Disturbance response Recovery behavior under external perturbations. predactor-sim-disturbance-response.mp4

Acknowledgements

We thank the authors of the following open-source projects:

• diffusion_implementation by WhoKnowsssss, which provides the diffusion training framework.
• TextOp, which provides the RL tracker training framework and pretrained tracker checkpoint.
• MotionCLIP, which provides the motion encoder foundation.

Citation

@misc{ye2026predactorpredictiveactiondiffusion,
  title={PredActor: Predictive Action Diffusion for Steerable Onboard Humanoid Control},
  author={Lei Ye and Haibo Gao and Yitang Li and Peng Xu and Zetong Jing and Junhan Sun and Fanrong Dong and Ziqi Han and Xue Wang and Jianhua Sun and Cewu Lu and Hao Zhao and Liang Ding},
  year={2026},
  eprint={2609.24840},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2609.24840},
}

License

The contents of this repository are released under the MIT License, unless noted otherwise.

Contact & Community

• Discord: Join the PredActor community for discussion and project updates.
• X: @MasterYip3 for author updates and project news.

PredActor WeChat group Valid through October 27, 2026.

Contact Author Direct contact with the author.

Scan a QR code with WeChat, or click it to view the full-size image. If the group QR code has expired or the group is full, contact the author on WeChat.