PIAE: Physics-Informed Continuous Motion Autoencoder

PIAE is a compact continuous autoencoder for 3D human motion. It maps every eight HumanML3D frames to one 256-dimensional token while explicitly preserving the physical signals whose integration controls global trajectory.

On the official HumanML3D test split, PIAE reaches 8.940 mm global MPJPE and 8.887 mm root-relative MPJPE over 4,210 eligible clips, with no prediction-side root, heading, rigid, or Procrustes alignment.

Method

flowchart LR
    A["Motion<br/>T × 22 × 3"] --> B["Structured representation<br/>T × 70"]
    B --> C["Masked Transformer encoder<br/>stride 2 × 2 × 2"]
    C --> D["One token / 8 frames<br/>256-D continuous latent"]
    D --> E["224-D learned pose code"]
    D --> F["32-D physical carrier"]
    E --> G["Masked Transformer decoder"]
    F --> G
    G --> H["Local pose + contact residuals"]
    F --> I["Exact planar/yaw recovery<br/>constrained root height"]
    H --> J["Reconstructed motion<br/>T × 22 × 3"]
    I --> J

Loading

The 70-D frame representation contains 66 root-local joint coordinates, two planar root velocities, one yaw velocity, and root height. Three masked encoder stages compress time by 2 each. Every output token contains:

• 224 learned dimensions for pose and motion detail.
• 32 deterministic carrier dimensions for the eight planar root-velocity pairs, eight yaw velocities, root-height start/end/mean, four soft support-contact means, and root-height variation.

The carrier remains inside the fixed 256-D token budget. The decoder uses it directly for trajectory-critical quantities and learns residual pose/contact detail. Training combines local and global reconstruction with velocity, acceleration, bone-length, contact, penetration, and terminal SE(2) objectives.

More detail is available in the project report and method notes.

Results

HumanML3D official test split, micro/element-weighted aggregate. Lower is better.

Model	Temporal rate	Global MPJPE (mm)	Root-relative MPJPE (mm)	Velocity (mm/s)	Acceleration (m/s²)	Contact speed (mm/s)	Penetration (%)	Terminal drift (mm)
PIAE	8×	8.940	8.887	62.763	1.369	24.667	0.019	0.000087
MotionGPT VQ	4×	172.085	55.987	212.885	2.934	105.854	0.345	251.731
MLD-1	1 global slot	104.408	43.924	223.479	2.271	66.276	0.038	126.876

The benchmark starts from all 4,384 official test IDs. It evaluates all 4,210 clips that satisfy the shared minimum of 40 frames and records the 174 shorter clips separately; no short clip is padded and no eligible clip is skipped. Each model receives the same centered crop, capped at 192 frames and divisible by eight, at 20 FPS. See the complete result record, machine-readable aggregates, and evaluation protocol.

Installation

PIAE requires Python 3.10 or newer and PyTorch.

git clone https://github.com/EviLooo/PIAE.git
cd PIAE
python -m venv .venv
# Activate .venv, then:
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

For a CUDA system, install the PyTorch build appropriate for that CUDA runtime before installing PIAE.

HumanML3D setup

HumanML3D data is not distributed with this repository. Because HumanML3D is reconstructed from AMASS/SMPL-family assets subject to their original terms, users must obtain the required assets and run the official HumanML3D preparation pipeline themselves. Point PIAE to a prepared directory containing:

HumanML3D/
├── train.txt
├── val.txt
├── test.txt
├── new_joints/
└── new_joint_vecs/     # required for MotionGPT/MLD comparison

Create a frozen, integrity-checked data manifest:

python scripts/prepare_manifest.py \
  --humanml-root /path/to/HumanML3D \
  --output manifests/humanml3d.json

See dataset setup and representation details.

Train from scratch

Run the complete preflight first. It verifies the official splits, data fingerprint, representation round trip, padding invariance, physical carrier, token dimensions, and model configuration without updating weights.

python train.py \
  --humanml-root /path/to/HumanML3D \
  --humanml-manifest manifests/humanml3d.json \
  --save-dir runs/piae_x8 \
  --preflight-only

Then verify a real forward/backward pass:

python train.py \
  --humanml-root /path/to/HumanML3D \
  --humanml-manifest manifests/humanml3d.json \
  --save-dir runs/piae_smoke \
  --smoke-test

Start the full 300-epoch run:

python train.py \
  --humanml-root /path/to/HumanML3D \
  --humanml-manifest manifests/humanml3d.json \
  --save-dir runs/piae_x8 \
  --num-workers 8

The default experiment is fully specified in configs/piae_x8.json. The trainer selects the best checkpoint using validation MPJPE, maintains an EMA model, writes an immutable training contract, and never constructs or loads the test split during training or model selection.

Reconstruct after training

Use the best model produced by the training run to reconstruct any eligible HumanML3D clip:

python reconstruct.py \
  --humanml-root /path/to/HumanML3D \
  --humanml-manifest manifests/humanml3d.json \
  --checkpoint runs/piae_x8/piae_best_mpjpe.pt \
  --clip-id 000000 \
  --output-dir reconstruction/000000 \
  --device auto

This writes the ground truth, reconstructed 22-joint motion, 256-D latent tokens, contact probabilities, and per-clip reconstruction metrics. See the complete training and reconstruction guide or its printable PDF.

Evaluate

The released evaluator runs PIAE, the official MotionGPT VQ-VAE, and the official MLD VAE through one crop, coordinate, recovery, and metric implementation:

python -m evaluation.evaluate_humanml3d_clean \
  --humanml-root /path/to/HumanML3D \
  --humanml-manifest manifests/humanml3d.json \
  --piae-checkpoint /path/to/piae_best_mpjpe.pt \
  --piae-stats /path/to/native_norm_stats.npz \
  --motiongpt-root /path/to/MotionGPT \
  --motiongpt-checkpoint /path/to/motiongpt_s3_h3d.tar \
  --motiongpt-mean /path/to/MotionGPT/mean.npy \
  --motiongpt-std /path/to/MotionGPT/std.npy \
  --mld-root /path/to/motion-latent-diffusion \
  --mld-checkpoint /path/to/1222_mld_humanml3d_FID041.ckpt \
  --mld-mean /path/to/MLD/Mean.npy \
  --mld-std /path/to/MLD/Std.npy \
  --output-dir evaluation_output \
  --device cuda \
  --fail-fast

Verify the published aggregates directly from the per-clip CSV:

python scripts/verify_results.py

Regenerate the quantitative plots:

python -m evaluation.plot_results

Repository contents

• piae/: model, representation, data pipeline, losses, and training utilities.
• evaluation/: shared full-test evaluator and official-model adapters.
• results/: aggregate JSON, all per-clip metrics, eligibility records, and plots.
• tests/: carrier, compression, masking, recovery, and checkpoint-loading tests.
• paper/: latest project report.

Citation

If PIAE is useful in your work, please cite the metadata in CITATION.cff.

Copyright and acknowledgements

Copyright © 2026 Osama Alshammari and Omar Hamad. Dataset and external-model assets remain governed by their upstream terms. See COPYRIGHT.md and THIRD_PARTY_NOTICES.md for details.