A complete, from-scratch build of a world model on a single RTX 4090.
A 46M-parameter camera-conditioned RGB-D model trained from scratch to predict novel views from posed observations — then extended to 128-step maze rollouts with discrete actions and geometric memory. Same codebase, synthetic static scenes at 96², one card end to end.
First-frame conditioning is RGB-D + camera pose (Plücker rays). Maze demos are camera/action-conditioned rollouts (DFS paths planned on the GT map) — not autonomous planning.
Same weights, more loops: R1 / R2 / R4 side by side (orbit)
- Loops buy autoregressive quality. Same checkpoint, 24 held-out orbit scenes: R1→R4 adds +1.68 dB AR PSNR (raw JSON; Stage E, 5 Euler steps). See
results/orbit_stage_e_step083864.json. - Loop depth has a useful range. An R×steps grid peaks near R4 + 2 Euler steps; R6/R8 (beyond the train depth of 4) degrade AR. Loops beat spending the same budget on more denoising steps alone.
- Revisit self-consistency ≠ scene correctness. A model can recreate its own earlier (wrong) layout at high PSNR. Oracle-history experiments show substantial sensitivity to rollout and memory fidelity — they diagnose the system; they do not prove a write-only bottleneck.
Install, smoke-check, train on one 4090, then point demos and evals at your EMA with --ckpt / --init. Headline numbers below are backed by published JSON under results/ — reprint them on CPU with print_result_tables.py.
git clone https://github.com/NeuraLiying/LoopWorld.git
cd LoopWorld
# install — see Requirements below
# Geometry / interface smoke
python scripts/run_stage_a_checks.py
# Reprint published tables from results/*.json (CPU, no checkpoint)
python scripts/print_result_tables.py
# Orbit starter train on one 4090 (~1 hour wall for the published YAML)
python scripts/run_stage_d_main.py \
--config configs/r1_main_768_96.yaml \
--device cuda:0 --batch-size 16 --tag my_run
# After training:
python scripts/make_orbit_demo.py \
--ckpt path/to/your_ema.pt \
--out exp_records/my_orbit
python scripts/stage_e_eval.py \
--ckpt path/to/your_ema.pt \
--out exp_records/my_stage_e \
--modes S,T,AR --Rs 1,2,4Maze train → eval → explore demo entrypoints are under Train.
Checkpoint: EMA at step 83864 (multi-continuation Stage D).
| Mode | Conditioning | R4 PSNR | Loop gain R4−R1 | AbsRel |
|---|---|---|---|---|
| S | GT first RGB-D frame | 28.36 | +1.93 dB | 0.026 |
| T | First frame + GT previous | 31.14 | +2.01 dB | 0.012 |
| AR | First frame + own previous | 27.40 | +1.68 dB | 0.032 |
| Demo | File |
|---|---|
| Loop split (R1/R2/R4) | docs/media/simple_orbit/D1_loop_R1_R2_R4_split.gif |
| AR PSNR vs view | docs/media/simple_orbit/D3_psnr_curve.png |
| Four-panel orbit (RGB-D) | docs/media/simple_orbit/orbit_demo_step083864_h264.mp4 |
Raw JSON: results/orbit_stage_e_step083864.json · R×steps grid: results/orbit_loop_rxsteps_grid.json.
Same layout EMA, protocol R4 / 5 steps / T=128 / own_conf. Paths are DFS explorations planned on the GT map; the model receives pose and discrete actions.
| Memory | AR full | Revisit vs GT |
|---|---|---|
| Own (confidence-weighted) | 20.22 | 21.10 |
| Oracle (GT-integrated history) | 33.24 | 35.23 |
Clean teaser — model RGB-D rollout + path (seed 700004, DFS, own_conf). Map is GT reference only.
Diagnostic panel — GT / pred / memory / depth (same trajectory). Prefer the clean teaser on a first pass.
| Demo | File |
|---|---|
| DFS clean (recommended teaser) | docs/media/maze/layout_explore_clean.gif |
| DFS diagnostic (six-panel) | docs/media/maze/layout_explore_diag.gif |
| Earlier SF own-memory AR | docs/media/maze/sf_ar_own_memory.gif |
JSON: results/maze_layout_test_seeds700000_700031.json.
Layout continues from action → memory → self-forcing (continued training), not a standalone from-scratch maze run. Stage figures: docs/technical_report.md.
Keep these distinct when reading the code or papers:
| Dial | Symbol | Meaning |
|---|---|---|
| Transformer loops | R | Shared-weight DiT core iterations (train R=4; eval R1/R2/R4) |
| Flow steps | S | Rectified-flow Euler steps per generated view |
| Autoregressive time | t | Frame index in a rollout |
| Geometric memory | read/write | Lift RGB-D → world points → reproject into the target camera |
Loop mechanics: 1 pre + 2 shared core + 1 post (~46M). The next loop consumes the hidden state after the core (not after the post). Every loop exit is supervised in training.
Parameters stay fixed; R is a quality/latency dial. Three measured points (beyond “more loops look better”):
- Loop gain grows with training on the 24-scene Stage E protocol (5 Euler steps, step 83864): R4−R1 ≈ +1.93 / +2.01 / +1.68 dB on S / T / AR.
- On this flow world model, loops beat extra denoising steps. The R×steps grid peaks at R4 with 2 Euler steps (S 29.26, AR 27.91); R1 with 10 steps is only ~25.5 on S.
- R is not unbounded. Training depth is 4; R6/R8 especially hurt AR (R8 AR ~24.6).
- Shared-weight loop DiT (R1→R4). Extra compute from iterations, not parameters; every exit supervised (
train_loops: 4). - Rectified flow + few Euler steps. Straight-line velocity; Stage E uses 5 steps so long AR rollouts stay cheap.
- RGB-D prediction. Depth is an explicit geometric target and enables point memory.
- Plücker-ray pose conditioning. Dense, resolution-aligned cameras — not a single global pose vector.
- QK-Norm + fp32 attention logits. Keeps long resumed runs finite; log max attention logit as a health signal.
- S / T / AR split. Separates single-frame quality from error accumulation; AR is the headline.
- Action conditioning via AdaLN (mazes). Discrete moves injected every block; init so control starts as a residual on the loaded checkpoint.
- Geometric point memory. z-buffered reprojection with light splatting; own vs oracle history for diagnosis.
- Self-forcing + layout curriculum. Train on EMA rollouts; oversample hard/turn frames; confidence-weight memory reads.
- Single-24GB-GPU profiling. 96² / width 768 / batch 16 saturates one 4090; profile before scaling batch.
- High revisit self-consistency ≠ correct memory. The model can stably recreate its own earlier wrong layout.
- Oracle-history experiments reveal substantial sensitivity to rollout and memory fidelity. They are a diagnostic upper bound, not a claim that writing is the only bottleneck.
- Training-recipe gains saturate. Self-forcing and layout help; further closed-loop gains need better geometry write/read, not endless recipe tweaks.
- Watch attention logits before long runs. After QK-Norm, max logits stay ~18; earlier runs hit ~1e7 while loss still looked fine.
Inspired by Atlas and Genie (camera/action-conditioned world models at scale). LoopWorld is an independent from-scratch implementation: procedural RGB-D data, looped DiT + rectified flow, Plücker conditioning, maze actions, geometric memory, and the experiments above were built and run for this project.
| Python | 3.10 (3.10–3.11 tested) |
| PyTorch | ≥ 2.4 with CUDA (tested 2.4.0 + cu121) |
| GPU | One 24 GB GPU (e.g. RTX 4090) |
| Python deps | see requirements.txt |
| System | ffmpeg with libx264 — demo video export only |
conda create -n loopworld python=3.10 -y && conda activate loopworld
pip install torch --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
# ffmpeg: apt / conda-forge / brew# Geometry / interface smoke
python scripts/run_stage_a_checks.py
# Orbit main train (96², width 768)
python scripts/run_stage_d_main.py \
--config configs/r1_main_768_96.yaml \
--device cuda:0 --batch-size 16 --tag my_run
# Maze: action → memory → self-forcing → layout
python scripts/run_maze_v2_train.py --device cuda:0
python scripts/run_maze_v2_mem_train.py --device cuda:0 --init path/to/action_ckpt.pt
python scripts/run_maze_v2_sf_train.py --device cuda:0 --init path/to/mem_ckpt.pt
python scripts/run_maze_v3_layout_train.py --device cuda:0 --init path/to/sf_ckpt.pt
python scripts/maze_mem_diag.py --ckpt path/to/mem_ckpt.pt --out exp_records/my_mem_diagloopworld/ # Python package (orbit + maze + memory)
configs/ # Train recipes
scripts/ # Train / eval / demo entrypoints
results/ # Published eval JSON (verifiable tables)
weights/ # MODEL_CARD + SHA256SUMS (run fingerprints)
docs/
media/ # Figures, GIFs, H.264 demos
technical_report.md
docs/technical_report.md— stage chronology, maze progression figures, eval protocols
@software{loopworld2026,
title = {LoopWorld: A Looped RGB-D World Model, from Orbits to Interactive Mazes},
author = {Li, Ying},
year = {2026},
url = {https://github.com/NeuraLiying/LoopWorld}
}


