Embodied.cpp 🤖

Embodied.cpp is an inference runtime for embodied AI models: Vision-Language-Action (VLA) models and World-Action Models (WAMs) for robotic perception and control. It runs these models efficiently on heterogeneous hardware (CPU / CUDA GPU / NPU) using GGUF weights, and ships with ready-to-use servers and evaluation clients.

---

NEWS

• [2026.09] Added support for Xiaomi-Robotics-0, TurboVLA, and X-VLA runtimes with full LIBERO evaluations (eval/).
• [2026.08] 🔥🔥 Released Embodied.cpp v1.0.
• [2026.07] Added support for Cosmos3-Nano and GR00T N1.7, the RoboLab benchmark, and Isaac Sim.
• [2026.06] Released the initial version of Embodied.cpp with support for pi0.5, HY-VLA, and LingBot-VA, plus the LIBERO and RoboTwin benchmarks.

---

🎬 Demos

GR00T N1.7

groot-n1.7-demo-results.mp4

• Successful execution ✅
• Cumulative inference time 2.5 s → 1.7 s (⬇32%).
• End-to-end execution time 14.6 s → 13.3 s.

HY-VLA

hy-vla-demo-results.mp4

• Successful execution ✅
• Cumulative inference time 9.4 s → 6.6 s (⬇30%).
• End-to-end execution time 14.8 s → 12.0 s.

---

Table of Contents

• Embodied.cpp 🤖
  • NEWS
  • 🎬 Demos
    • GR00T N1.7
    • HY-VLA
  • Table of Contents
  • 1. 🧭 Current Support and Roadmap
    • 1.1 Supported Models
      • Vision-Language-Action Models
      • World Models
    • 1.2 Performance Acceleration
    • 1.3 Runtime Roadmap
  • 2. 🚀 Quick Start
    • 2.1 Clone the Repo
    • 2.2 Get GGUF Weights
    • 2.3 Install System Dependencies
    • 2.4 Build by Model and Backend
    • 2.5 Start a Server
  • 3. 🧪 Evaluation
    • 3.1 LIBERO
    • 3.2 RoboTwin
    • 3.3 RoboLab (Cosmos3-Nano)
  • 4. 🔧 Convert and Quantize Models
  • 5. 🗂️ Project Structure
  • 6. 🚧 Known Limitations & Future Work
  • 7. 📄 Citation
  • 8. ⚖️ License
  • 9. 🙏 Acknowledgements

---

1. 🧭 Current Support and Roadmap

1.1 Supported Models

Vision-Language-Action Models

pi0.5	SmolVLA	HY-VLA	GR00T N1.7
Xiaomi-Robotics-0	TurboVLA	X-VLA

World Models

LingBot-VA

Cosmos3-Nano

We continuously track advances in embodied AI and adapt Embodied.cpp to the latest open models. Pull requests that add support for new models are always welcome.

1.2 Performance Acceleration

Reported VLA comparisons are normalized to each model's Python baseline (1.00), with Python → C++ BF16-policy ratios. Lower inference latency and VRAM are better. The unified benchmark standard and evidence ledger apply to every VLA runtime.

Model	Inference Latency ↓	VRAM ↓
pi0.5	1.00 → 0.90 (10% lower)	1.00 → 0.60 (40% lower)
GR00T N1.7	1.00 → 0.72 (28% lower)	1.00 → 0.93 (7% lower)
HY-VLA	1.00 → 0.48 (52% lower)	1.00 → 0.68 (32% lower)
Xiaomi-Robotics-0	1.00 → 0.37 (63% lower)	1.00 → 0.90 (10% lower)
TurboVLA	1.00 → 0.59 (41% lower)	1.00 → 0.95 (5% lower)
X-VLA	1.00 → 0.88 (12% lower)	1.00 → 0.57 (43% lower)

For World Models, C++ substantially reduces VRAM while keeping the success rate close to the Python baseline.

Model	VRAM ↓
Cosmos3	21.84 GB → 19.49 GB (10.8% lower)
LingBot-VA	24.75 GB → 16.44 GB (33.6% lower)

Precision note: XR0 and TurboVLA baselines are the official BF16 implementations; X-VLA's is the official F32 deployment (BF16-vs-BF16 X-VLA parity is 0.99 latency / 1.04 VRAM). pi0.5, GR00T and HY-VLA keep the original project's reported comparisons and were not rerun here; current-source measurements for XR0/TurboVLA/X-VLA come from matched-boundary runs and the per-model validation reports under eval/.

Highlights: Compared with Python, C++ BF16 reduces VLA inference latency by up to 63% and VRAM by up to 57%. For World Models, it reduces VRAM by up to 33.6%, with success-rate changes limited to 2 percentage points.

1.3 Runtime Roadmap

• This project is still under active construction 🚧
• A more modular and maintainable runtime architecture for Embodied.cpp
• Additional inference optimizations, such as real-time chunking and VLA caching

---

2. 🚀 Quick Start

2.1 Clone the Repo

git clone <repo-url> && cd embodied.cpp
./patches/init_third_party.sh

By default, the setup script prepares a combined llama.cpp source tree for all supported runtimes. For smaller model-specific setups or custom patch profiles, see patches/PATCH.md.

2.2 Get GGUF Weights

Pre-converted GGUF releases for the original models are available on Hugging Face:

• https://huggingface.co/SEU-PAISys/Embodied.cpp

The repository currently hosts GGUF artifacts for the original runtime models:

• pi0.5: main policy GGUF plus multimodal projector GGUF
• GR00T N1.7: truncated Qwen3-VL text GGUF, vision projector GGUF, and action-head GGUF
• HY-VLA-0.5: combined VLA GGUF for RoboTwin and related runtime paths
• LingBot-VA: transformer GGUF and companion artifacts used by the LingBot path

The remaining models are converted locally from their upstream checkpoints with the scripts in scripts/:

• Cosmos3-Nano: RoboLab WAM GGUF with the Wan VAE encoder
• SmolVLA: LeRobot policy GGUF plus SigLIP mmproj GGUF
• Xiaomi-Robotics-0: Qwen3-VL-4B backbone + DiT flow-matching action head, converted with scripts/convert_xr0_to_gguf.py; quantize with scripts/quantize_xr0_gguf.py (q8_0/q6_k/q5_k/q4_k)
• TurboVLA: DINOv3 ViT + BERT + bidirectional cross-attn fusion + ACT decoder, converted with scripts/convert_turbovla_to_gguf.py; create q8_0/q6_k/q4_0 storage variants with scripts/quantize_vla_gguf.py
• X-VLA: Florence-2 DaViT + BART encoder + domain-conditioned flow head, converted with scripts/convert_xvla_to_gguf.py; create q8_0/q6_k/q4_0 storage variants with scripts/quantize_vla_gguf.py

Recommended local layout:

checkpoints/
  pi05/
    pi05.gguf
    pi05-mmproj.gguf
  groot-n1/
    qwen3vl-backbone-bf16.gguf
    qwen3vl-mmproj-bf16.gguf
    groot-n1.7-libero-object-action-head-bf16.gguf
  Hy-Embodied-0.5-VLA-RoboTwin/
    Hy-Embodied-0.5-VLA-RoboTwin_bf16.gguf
    Hy-Embodied-0.5-VLA-RoboTwin_q4_K.gguf
  lingbot_va/
    lingbot_transformer.gguf
    ...
  cosmos3/
    cosmos3_robolab_full_w8_with_vae_encoder.gguf
  smolvla/
    smolvla.gguf
    mmproj-smolvla.gguf
  xr0/
    xr0.gguf              # convert locally via scripts/convert_xr0_to_gguf.py
    xr0-mmproj.gguf
  turbovla/
    turbovla.gguf         # convert locally via scripts/convert_turbovla_to_gguf.py
  xvla/
    xvla-libero.gguf      # convert locally via scripts/convert_xvla_to_gguf.py

You can also convert upstream checkpoints yourself with the scripts in scripts/, but for most users the Hugging Face GGUF releases are the fastest way to get started.

2.3 Install System Dependencies

Install the required system packages for your platform before building.

Minimum build requirements:

• CMake >= 3.22
• A C++17 compiler, such as GCC 11+ or Clang 14+
• CUDA 12.x, optional and required only for GPU builds

Linux: Make sure cmake, protobuf=3.20.3, zeromq, cppzmq, pkg-config and uv are available before building. A typical Ubuntu/Debian native-Linux installation is:

sudo apt-get update
sudo apt-get install -y \
  build-essential cmake pkg-config protobuf-compiler libprotobuf-dev \
  libzmq3-dev cppzmq-dev libegl1-mesa-dev libglu1-mesa-dev \
  libgl1-mesa-dev ffmpeg iproute2

CUDA builds additionally require a compatible NVIDIA driver and CUDA toolkit. Desktop systems may use nvidia-smi for device monitoring; Jetson systems provide tegrastats for resource monitoring. Install uv separately if your distribution does not package it.

2.4 Build by Model and Backend

Model switches default to OFF. Enable only the runtimes you need.

CUDA GPU template:

CUDA_HOME="${CUDA_HOME:-$(dirname "$(dirname "$(command -v nvcc)")")}"
CUDA_ARCH=${CUDA_ARCH:-native}

cmake -S . -B <BUILD_DIR> \
  -DCMAKE_BUILD_TYPE=Release \
  -D<MODEL_BUILD_FLAG>=ON \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER="${CUDA_HOME}/bin/nvcc" \
  -DCMAKE_CUDA_ARCHITECTURES="${CUDA_ARCH}"
cmake --build <BUILD_DIR> --target <SERVER_TARGET> -j$(nproc)

Use a separate <BUILD_DIR> for each model or CMake configuration, such as build-groot-cuda or build-lingbot-cuda; a build directory stores one CMake configuration and its generated artifacts.

Replace the placeholders with the model you want to build:

Model	<MODEL_BUILD_FLAG>	<SERVER_TARGET>
pi0.5	MODEL_BUILD_VLA_PI05	vla-server
SmolVLA	MODEL_BUILD_VLA_SMOLVLA	vla-server
HY-VLA	MODEL_BUILD_VLA_HY_VLA	vla-server
GR00T N1.7	MODEL_BUILD_VLA_GROOT_N1	vla-server
LingBot-VA	MODEL_BUILD_WAM_LINGBOT_VA	wam-lingbot-server
Cosmos3-Nano	MODEL_BUILD_WAM_COSMOS3	wam-server
Xiaomi-Robotics-0	MODEL_BUILD_VLA_XR0	vla-server
TurboVLA	MODEL_BUILD_VLA_TURBOVLA	vla-server
X-VLA	MODEL_BUILD_VLA_XVLA	vla-server

CUDA_ARCH defaults to native, so CMake detects the GPU installed on the build machine. Override it with an explicit architecture when cross-compiling or when using CMake older than 3.24. Common explicit values include 75 (Turing), 80 or 86 (Ampere), 87 (Ampere, Jetson AGX Orin), 89 (Ada), 90 (Hopper), and 120 (Blackwell). The selected CUDA toolkit must support that architecture; for example, Blackwell sm_120 requires CUDA 12.8 or newer.

Jetson AGX Orin with JetPack 6: JetPack 6 commonly provides CMake 3.22, so select Orin's compute capability explicitly instead of using the native value introduced in CMake 3.24:

CUDA_HOME=/usr/local/cuda
cmake -S . -B build-pi05-jetson \
  -DCMAKE_BUILD_TYPE=Release \
  -DMODEL_BUILD_VLA_PI05=ON \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER="${CUDA_HOME}/bin/nvcc" \
  -DCMAKE_CUDA_ARCHITECTURES=87
cmake --build build-pi05-jetson --target vla-server -j4

cuDNN is detected automatically and enables accelerated convolution paths for the LingBot-VA and Cosmos3 WAM CUDA builds. When cuDNN is unavailable, the existing CUDA fallback paths remain enabled. Discovery searches normal system and multiarch locations as well as the CUDA toolkit. Set CUDNN_ROOT for a non-system installation.

2.5 Start a Server

./<BUILD_DIR>/<SERVER_TARGET> <MODEL_ARGUMENTS>

Use the <BUILD_DIR> and <SERVER_TARGET> selected in section 2.4. Replace <MODEL_ARGUMENTS> with the arguments for the selected model:

Model	<SERVER_TARGET>	<MODEL_ARGUMENTS>
pi0.5	vla-server	<MMPROJ_GGUF> <MODEL_GGUF>
SmolVLA	vla-server	<MMPROJ_GGUF> <MODEL_GGUF>
HY-VLA	vla-server	<MODEL_GGUF>
GR00T N1.7	vla-server	--backbone <BACKBONE_GGUF> <MMPROJ_GGUF> <ACTION_HEAD_GGUF>
LingBot-VA	wam-lingbot-server	<TRANSFORMER_GGUF> <TEXT_ENCODER_GGUF> <VAE_ENCODER_GGUF>
Cosmos3-Nano	wam-server	<MODEL_GGUF>
Xiaomi-Robotics-0	vla-server	<MMPROJ_GGUF> <MODEL_GGUF>
TurboVLA	vla-server	<MODEL_GGUF>
X-VLA	vla-server	<MODEL_GGUF>

VLA servers bind to tcp://*:5555 by default. LingBot-VA and Cosmos3-Nano bind to tcp://*:5557 by default. Pass --bind <ADDR> to override the listening address or port.

3. 🧪 Evaluation

Start the required server as described in section 2.5, then select the configuration and runner for the model and benchmark you want to evaluate.

Model	Benchmark	Configuration	Results	Server
pi0.5	LIBERO	pi0.5	-	Manual
SmolVLA	LIBERO	SmolVLA	-	Manual
GR00T N1.7	LIBERO	GR00T	-	Manual
LingBot-VA	LIBERO	LingBot	-	Manual
HY-VLA	RoboTwin	HY-VLA	-	Managed
Cosmos3-Nano	RoboLab	Cosmos3	-	Managed
Xiaomi-Robotics-0	LIBERO	Xiaomi-Robotics-0	report	Manual
TurboVLA	LIBERO	TurboVLA	report	Manual
X-VLA	LIBERO	X-VLA	report	Manual

LIBERO uses eval/client/run_sim_client_direct.py; start its matching server separately. RoboTwin and RoboLab runners start their servers from the selected configuration and stop them when the evaluation finishes.

LIBERO with a manual server (all VLA models follow the same two-step flow):

# 1. Start the server for the model you built (see section 2.5).
./build/vla-server checkpoints/xr0/xr0-mmproj.gguf \
                   checkpoints/xr0/xr0.gguf --bind tcp://*:5555

# 2. Run the matching configuration in a second shell.
MUJOCO_GL=egl PYOPENGL_PLATFORM=egl \
eval/sim/libero/libero_uv/.venv/bin/python \
  eval/client/run_sim_client_direct.py \
  --conf eval/conf/libero_xr0_eval.yaml

Swap the checkpoint and configuration for the model under test; TurboVLA (libero_turbovla_eval.yaml) and X-VLA (libero_xvla_eval.yaml) take a single GGUF, while Xiaomi-Robotics-0 takes the mmproj GGUF first.

SmolVLA on LIBERO:

Convert a LeRobot LIBERO checkpoint into the policy GGUF and the SigLIP identity-proxy mmproj. The pixel-shuffle connector is stored in the policy GGUF and executed by models/smolvla.cpp.

python scripts/convert_smolvla_to_gguf.py \
  --ckpt checkpoints/smolvla_libero \
  --out checkpoints/smolvla/smolvla.gguf
python scripts/convert_smolvla_mmproj_to_gguf.py \
  --ckpt checkpoints/smolvla_libero \
  --out checkpoints/smolvla/mmproj-smolvla.gguf

# Start `vla-server` with MODEL_BUILD_VLA_SMOLVLA=ON, then run the smoke test.
MUJOCO_GL=egl PYOPENGL_PLATFORM=egl \
eval/sim/libero/libero_uv/.venv/bin/python \
  eval/client/run_sim_client_direct.py \
  --conf eval/conf/libero_smolvla_eval.yaml

The serialized SmolVLA processor requires a trailing newline in each task prompt; the direct client applies it automatically. The checked-in configuration uses one replayed action per model request and the full LIBERO episode horizon.

See eval/SMOLVLA_VALIDATION.md for the acceptance matrix, smoke protocol, build matrix, and parity methodology. The implementation and validation report is available at eval/SMOLVLA_TECHNICAL_REPORT_ZH.md.

HY-VLA on RoboTwin:

Each checked-in YAML is a baseline evaluation configuration. Adjust its task selection, episode count, model paths, output location, and other benchmark-specific settings for your run; the exact field names are documented in the corresponding configuration and simulator README.

3.1 LIBERO

LIBERO tests robotic manipulation skills on the spatial, object, goal, short, and long suites. Install the simulator once:

bash eval/sim/libero/setup_libero.sh

After starting the matching server in another terminal, run a checked-in configuration:

# pi0.5
eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
  --conf eval/conf/libero_pi05_eval.yaml

# GR00T N1.7
eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
  --conf eval/conf/libero_groot_n1_eval.yaml

# LingBot-VA
eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
  --conf eval/conf/libero_lingbot_va_eval.yaml

# Xiaomi-Robotics-0
eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
  --conf eval/conf/libero_xr0_eval.yaml

# TurboVLA
eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
  --conf eval/conf/libero_turbovla_eval.yaml

# X-VLA
eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
  --conf eval/conf/libero_xvla_eval.yaml

See eval/sim/libero/README.md for LIBERO suite selection, headless EGL execution, and configuration details.

3.2 RoboTwin

RoboTwin is a dual-arm manipulation benchmark. Install it once:

bash eval/sim/robotwin/setup_robotwin.sh

Run HY-VLA with the standard configuration:

eval/sim/robotwin/robotwin_uv/.venv/bin/python \
  eval/client/run_robotwin_eval.py \
  --conf eval/conf/robotwin_hy_vla_eval.yaml

See eval/sim/robotwin/README.md for detailed setup modes and troubleshooting.

3.3 RoboLab (Cosmos3-Nano)

RoboLab evaluates the native C++ Cosmos3 WAM path. Install RoboLab once without launching its optional Isaac Sim smoke test:

RUN_SMOKE_TEST=0 bash eval/sim/robolab/setup_robolab.sh

Run Cosmos3-Nano with the standard configuration:

python3 eval/client/run_robolab_eval.py \
  --conf eval/conf/robolab_cosmos3_eval.yaml

The runner starts the C++ wam-server automatically. See eval/sim/robolab/README.md for configuration details, the transport-only smoke test, and the PyTorch-reference path.

4. 🔧 Convert and Quantize Models

Pre-converted GGUF releases for the original runtime models are available on Hugging Face; the newer models are converted locally from their upstream checkpoints (see section 2.2). Use the conversion tools when preparing a compatible upstream checkpoint or a custom quantization.

Model	Workflow
pi0.5	Policy and vision projector
GR00T N1.7	Action head and Qwen3-VL backbone
HY-VLA	Combined GGUF and quantization
LingBot-VA	Model artifacts and Wan quantization
Cosmos3-Nano	RoboLab full_w8 GGUF
Xiaomi-Robotics-0	GGUF conversion and k-quantization
TurboVLA	Self-contained GGUF and storage quantization
X-VLA	Policy GGUF and storage quantization

See scripts/README.md for prerequisites, commands, expected outputs, and post-conversion checks.

5. 🗂️ Project Structure

What lives where, in plain language:

Directory	What it contains
models/	C++ implementations of supported models
runtime/	Model registry, architecture detection, shared utilities
adapter/	Typed I/O boundary between observations and model inputs
serving/	ZeroMQ/Protobuf inference servers and API definitions
kernels/	Custom CUDA kernels for GPU builds
scripts/	GGUF conversion and quantization tools
patches/	Third-party setup patches
eval/	Evaluation clients, configurations, and simulator integrations

6. 🚧 Known Limitations & Future Work

Project-wide:

• No unit test framework; coverage relies on smoke tests and LIBERO/RoboTwin evaluation scripts.
• No batched inference (batch_size=1 only).

7. 📄 Citation

If you find Embodied.cpp useful in your research, please consider citing:

@article{xu2026embodiedcpp,
  title={Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots},
  author={Xu, Ling and Han, Chuyu and Li, Borui and Wu, Hao and Jiang, Shiqi and Cao, Ting and Li, Chuanyou and Zhong, Sheng and Wang, Shuai},
  journal={arXiv preprint arXiv:2607.02501},
  year={2026},
  doi={10.48550/arXiv.2607.02501},
  url={https://arxiv.org/abs/2607.02501}
}

8. ⚖️ License

This project is released under the Apache License 2.0. Third-party dependencies, model checkpoints, datasets, and upstream reference implementations are distributed under their own licenses.

9. 🙏 Acknowledgements

Supported models:

• pi0.5 / OpenPI
• NVIDIA Isaac GR00T
• HY-VLA
• LingBot-VA
• NVIDIA Cosmos / Cosmos3-Nano
• SmolVLA
• Xiaomi-Robotics-0
• TurboVLA
• X-VLA

Foundational projects this build depends on:

• llama.cpp (LLM inference engine)
• vla.cpp (unified VLA runtime)
• LIBERO (manipulation benchmark)
• RoboTwin (dual-arm robot benchmark)
• RoboLab (Isaac Sim DROID evaluation benchmark)