Zhengxuan Wei1,2*, Xu Guo3,2*, Xinghui Li2*†, Xunzhi Xiang1, Min Wei2, Yiran Zhu2, Qiulin Wang2, Xintao Wang2, Pengfei Wan2, Xiangwang Hou3, Qi Fan1†
1Nanjing University 2Kling Team, Kuaishou Technology 3Tsinghua University
*Equal contribution †Corresponding author
TL;DR GIM-World compresses an arbitrarily long observation history into a fixed-size implicit memory whose content is supervised to encode 3D scene geometry. Long-horizon autoregressive rollouts stay geometrically coherent and visually consistent with the scene, while explicit-memory (frame retrieval / packing) and geometry-agnostic implicit-memory baselines drift.
- [2026-09] Code and checkpoints (first-person / third-person MIND) released.
- [2026-06] Paper on arXiv.
- [2026] GIM-World is accepted to SIGGRAPH Asia 2026.
- 📖 Introduction
- 🛠️ Installation
- ⬇️ Download Models
- 🎮 Quick Start
- 🔑 Inference
- 📊 Results
- 🗂️ Code layout
- 📚 Citation
- 📧 Contact
- 🙏 Acknowledgements
Video world models generate the future from past observations and actions, but a long rollout depends on what the model remembers once observations leave its context window. Explicit memories keep frames or an online 3D reconstruction and inherit retrieval heuristics, redundant appearance storage or reconstruction artifacts; implicit memories compress history into a compact state, but nothing constrains that state to encode cross-view geometry.
GIM-World is a geometry-aware implicit memory for video world models built on a Wan2.1-1.3B DiT backbone:
- Implicit Memory Encoder — two blocks of compact self-attention + FFN turn a variable-length, camera-indexed history into a fixed set of memory tokens; the tokens are concatenated with the target latents to condition the backbone. The encoder runs in < 0.3 % of the backbone's time.
- Camera-Queryable Geometry Supervision (training only) — a ray-map query of a sampled history camera reads the memory and is matched against frozen VGGT features, forcing the memory to store view-consistent geometry rather than an appearance cache. The geometry head and teacher are discarded at inference; the head and loss are provided for reference in
gim/models/geometry_head.pybut are not used bygenerate.py. - Information-Guided Pruning — before encoding, the history is reduced to a budget of K latents by greedily maximizing mutual information under a pose–time Gaussian-process kernel, so encoding cost stays bounded as the rollout grows.
conda create -n gim-world python=3.10 -y
conda activate gim-world
pip install -r requirements.txt
pip install flash-attn --no-build-isolation # required by the Wan2.1 backboneTested with PyTorch >= 2.4 and CUDA 12.x on a single NVIDIA GPU (bf16 VAE / T5, fp32 DiT).
# Wan2.1 VAE / umT5 + GIM-World checkpoints from ModelScope,
# plus one MIND example clip for the quick start
python download_models.py --with_exampleThis fetches the Wan2.1-T2V-1.3B VAE / umT5 text encoder and the GIM-World checkpoints, and prints the paths to use below. Checkpoints are hosted on ModelScope: nagara214/GIM-World.
| Checkpoint | Trained on |
|---|---|
GIM-World/first_person |
MIND first-person split |
GIM-World/third_person |
MIND third-person split |
Each checkpoint folder holds transformer/ (DiT), memory_encoder.safetensors, camera_proj.safetensors, action_embedding.safetensors and a config.json with the inference hyper-parameters (480×832, 20-latent chunks, pruning budget 200).
bash run.shrun.sh rolls out the example clip in assets/examples/demo_scene for 20 seconds after its mark_time, following the recorded camera trajectory, and writes outputs/demo/demo_scene/video.mp4 (observed prefix + generated frames). Edit the variables at the top of run.sh to point at your own paths.
A single entry point handles everything end to end — VAE encoding of the prefix, text encoding, history pruning, autoregressive rollout and decoding — no offline preprocessing step.
python generate.py \
--input path/to/clip_dir \
--output outputs/my_run \
--gim_ckpt checkpoints/GIM-World/first_person \
--wan_dir checkpoints/Wan2.1-T2V-1.3BA clip is a directory in the MIND layout:
clip_dir/
├── video.mp4 # at least the observed prefix: frames [0, mark_time)
└── action.json
Frames [0, mark_time) of video.mp4 are encoded as the initial history; entries [mark_time, total_time) of data provide the camera path and actions to follow. Positions are divided by 100 (cm → m) and expressed relative to the first history frame inside the model.
Author an action.json for your own footage with a pose string:
# treat all frames of my_clip/video.mp4 as the prefix, then move forward 60 frames,
# turn right 20 frames, move forward 40 frames
python make_trajectory.py --clip_dir my_clip \
--pose "w-60, right-20, w-40" \
--caption "A quiet stone courtyard at dusk."
python generate.py --input my_clip --output outputs/my_clip --gim_ckpt ... --wan_dir ...| Token | Meaning | Token | Meaning |
|---|---|---|---|
w / s |
move forward / backward | up / down |
look up / down |
a / d |
move left / right | left / right |
look left / right |
stay |
hold pose |
Each item is <action>-<frames>; speeds default to 5 cm/frame and 1°/frame (--move_speed, --turn_speed). If your prefix camera is not static, pass per-frame poses with --prefix_pose_json. For arbitrary paths, write action.json directly or use gim.utils.trajectory.frames_from_keyframes.
| Flag | Default | Description |
|---|---|---|
--max_seconds |
20 | Stop this many seconds after mark_time (≤ 0: run to total_time) |
--num_steps |
50 | Denoising steps per chunk (DPM-Solver++) |
--guide_scale |
1.0 | Text CFG scale (1.0 = off) |
--pruning |
from config | information_guided (paper) or uniform |
--pruning_budget |
200 | Max history latents kept before memory encoding |
--perspective |
from config | 1st or 3rd: which pose in action.json is the camera |
--caption |
from json | Override the text prompt |
--include_prefix |
off | Prepend the observed frames to the saved video |
--dry_run |
off | Encode, align and prune only (no diffusion) — quick sanity check |
Every 20-latent chunk corresponds to 77 frames (≈ 3.2 s at 24 fps).
Point --input_root at a MIND split; every sub-directory with video.mp4 + action.json is a clip, and clips are sharded across the GPUs launched by torchrun:
torchrun --nproc_per_node=8 generate.py \
--input_root MIND-Data/1st_data/test/mem_test \
--output outputs/gim_world/1st_data/mem_test \
--gim_ckpt checkpoints/GIM-World/first_person \
--wan_dir checkpoints/Wan2.1-T2V-1.3B
torchrun --nproc_per_node=8 generate.py \
--input_root MIND-Data/3rd_data/test/mem_test \
--output outputs/gim_world/3rd_data/mem_test \
--gim_ckpt checkpoints/GIM-World/third_person \
--wan_dir checkpoints/Wan2.1-T2V-1.3B --perspective 3rdThe output tree outputs/gim_world/{1st_data,3rd_data}/mem_test/{clip}/video.mp4 is exactly what the official MIND evaluator expects:
python src/process.py --gt_root MIND-Data --test_root outputs/gim_world --num_gpus 8Quantitative comparison on the MIND memory test (from the paper, Table 1). Memory: per-frame reconstruction vs. ground truth; Action: relative pose error of the rendered trajectory; Geometry: normalized dense reprojection score.
| Method | First-person | Third-person | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MSE↓ | PSNR↑ | SSIM↑ | LPIPS↓ | Trans↓ | Rot↓ | Reproj↑ | MSE↓ | PSNR↑ | SSIM↑ | LPIPS↓ | Trans↓ | Rot↓ | Reproj↑ | |
| Matrix-Game 2.0 | 0.1188 | – | – | – | 0.0265 | 0.6914 | – | 0.1404 | – | – | – | 0.0622 | 0.9031 | – |
| MIND-World | 0.1035 | – | – | – | 0.0384 | 0.5534 | – | 0.1042 | – | – | – | 0.0321 | 0.3328 | – |
| FramePack | 0.0764 | 12.04 | 0.3763 | 0.7062 | 0.0269 | 0.6536 | 73.97 | 0.0735 | 12.23 | 0.3957 | 0.6770 | 0.0145 | 0.2852 | 70.82 |
| Context-as-Memory | 0.0706 | 12.50 | 0.3917 | 0.6953 | 0.0235 | 0.5697 | 66.14 | 0.0671 | 12.81 | 0.4051 | 0.6480 | 0.0241 | 0.4385 | 69.17 |
| SSM memory | 0.0796 | 11.96 | 0.3953 | 0.7439 | 0.0379 | 0.6702 | 53.95 | 0.0928 | 11.76 | 0.4037 | 0.6830 | 0.0201 | 0.2971 | 61.26 |
| GIM-World | 0.0614 | 13.40 | 0.4135 | 0.6304 | 0.0247 | 0.4126 | 81.70 | 0.0605 | 13.58 | 0.4303 | 0.5974 | 0.0106 | 0.1588 | 87.10 |
generate.py # single inference entry point
make_trajectory.py # author action.json from a pose string
download_models.py # fetch Wan2.1 VAE/T5 + GIM-World checkpoints (+ example clip)
gim/
pipeline.py # GIMWorldPipeline: encode -> prune -> rollout -> decode
models/
memory_encoder.py # Implicit Memory Encoder (Sec. 3.2)
dit.py # memory-conditioned Wan2.1 DiT, camera injection, chunk sampler
action_embedding.py # action embeddings
geometry_head.py # camera-queryable geometry head + L_geo (Sec. 3.3, training only)
utils/
pruning.py # information-guided pruning (Sec. 3.4)
camera.py # poses, relative frames, latent <-> frame alignment
video.py # streaming Wan VAE encode / decode, mp4 I/O
trajectory.py # pose-string / keyframe -> action.json
wan/ # trimmed Wan2.1 (DiT, VAE, umT5, flow-matching solver)
@article{wei2026gim,
title = {Geometry-Aware Implicit Memory for Video World Models},
author = {Wei, Zhengxuan and Guo, Xu and Li, Xinghui and Xiang, Xunzhi and Wei, Min and Zhu, Yiran and Wang, Qiulin and Wang, Xintao and Wan, Pengfei and Hou, Xiangwang and Fan, Qi},
journal = {arXiv preprint arXiv:2606.02436},
year = {2026}
}Questions and issues: open a GitHub issue or email [email protected].
GIM-World builds on Wan2.1 (backbone, VAE, text encoder), is trained and evaluated on MIND, and distills geometry from VGGT during training. We thank the authors for releasing their work.
Code derived from Wan2.1 is released under the Apache 2.0 License; the GIM-World additions follow the same license.
{ "mark_time": 1465, // first frame to generate "total_time": 4352, // frames covered by "data" "caption": "...", // optional text prompt "data": [ // one entry per frame, prefix AND future {"time": 0, "ws": 0, "ad": 0, "ud": 0, "lr": 0, // actions: 0 none, 1 +, 2 − "actor_pos": {"x": 2398.2, "y": -4672.6, "z": 168.6}, // cm (Unreal) "actor_rpy": {"x": 0, "y": 0, "z": -72.5}, // roll/pitch/yaw, degrees "camera_pos": {...}, "camera_rpy": {...}} // third-person only ] }