trainer: DDP devices are per node; eval-mode val subsampling is single-rank

.gitignore

@@ -45,12 +45,25 @@ annotations_test/**
 log_conversion/**
 debug_model_inputs/**
 temp_dir/**
+/datasets/
+apikey.txt
+slurm-*.out
+slurmoutputs/
+*.log
+.inductor_cache/
+
 scratch/
 sample_data/
 external_ckpts/
 external/MANO/
+ann_backup/
+aria_ann_backup/
+sort_annotations/
 
 # setuptools build output
 build/
 dist/
 docs/
+
+# per-worktree venv for the Pi benchmarks (the shared emimic/ venv is off-pin)
+.venv-pi/

AGENTS.md

@@ -25,4 +25,39 @@ use plan mode for anything except extremely simple tasks
 
 ## Slurm rules
 If you're on a slurm cluster, request a GPU before running or testing training.
-On sky1/sky2: salloc -p rl2-lab -A rl2-lab --gres=gpu:a40:1 -c 12 --mem=30G
+On sky1/sky2: salloc -p rl2-lab -A rl2-lab --gres=gpu:a40:1 -c 12 --mem=30G
+
+## Cancelled jobs and leaked GPU memory
+A cancelled DDP job can leave whole GPUs allocated, with `nvidia-smi` blaming PIDs that are not in
+`/proc`. Nothing is wrong with the driver: the memory is held by **live** `pt_data_worker` processes
+reparented to init.
+
+Dataloader workers are forked after CUDA init, so each inherits the rank's `/dev/nvidia*` fds — and
+ranks open *every* visible GPU, so one survivor pins the whole node. The worker cannot exit on its
+own: it inherited both ends of its result pipe, so a dead rank never gives it an `EPIPE`, it blocks
+in `pipe_write`, and PyTorch's parent check (which only runs between `index_queue` timeouts) is
+never reached. `ProctrackType=proctrack/linuxproc` then walks the PPID tree and never sees it.
+
+Whole ranks leak the same way. A rank wedged in `torch.cuda.synchronize()` inside inductor's
+`cudagraph_trees` does not answer SIGTERM, Slurm hits `UnkillableStepTimeout` (60s) and gives up
+(`srun: error: Timed out waiting for job step to complete`), and the job leaves the queue with its
+ranks still spinning on every GPU. Enabling `torch.compile` makes this more likely, not less.
+
+`egomimic/utils/gpu_orphans.py` owns the fix:
+
+- `orphan_guarded(params)` / `die_with_parent` set `PR_SET_PDEATHSIG` on every worker, so the kernel
+  kills it with its rank regardless of what it is blocked on. **Any new `DataLoader` on a training
+  path needs one of them** — they are already on the train/val loaders and the norm-stats pass.
+- `arm_rank_pdeathsig()` does the same for the rank process itself, so a launcher that gives up
+  takes its ranks with it. Slurm-gated, and skipped when SIGHUP is ignored, so a `nohup` run in
+  a `salloc` shell is not shot when that shell exits.
+- `reap_orphans()` runs from `trainHydra.main` before CUDA init and clears what an earlier job left
+  on the node, so a leak cannot outlive the job that made it. It only ever signals processes that
+  are yours, reparented to init and holding an nvidia fd; a live job's workers have a live parent,
+  so they are never touched. Off outside Slurm and on non-zero `SLURM_LOCALID`; disable with
+  `EGOMIMIC_REAP_GPU_ORPHANS=0`.
+- `./reap_gpu_orphans.sh [--kill] <node>` is the manual version, for nodes you are not about to
+  submit to. It attaches to your own job on a busy node rather than queueing behind it.
+
+The cluster-level fix is `ProctrackType=proctrack/cgroup`, which sweeps by cgroup instead of by PPID
+tree and would make all of this unnecessary. That needs admin; `/etc/slurm` is read-only to us.

CONTRIBUTING_DATA.md

@@ -644,7 +644,7 @@ If you are contributing egocentric human data, you use `human_bimanual` (or the
 There is a **single concrete `Human` embodiment class** ([`egomimic/rldb/embodiment/human.py`](egomimic/rldb/embodiment/human.py)) shared by all human data. You do **not** write a per-vendor subclass, and there is **no per-vendor embodiment identifier** — every human contributor uses `human_*` and records their source in the `lab` field. Camera intrinsics travel with the data (`zarr.attrs`, §6.3 / §6.4); per-vendor structural choices are passed as explicit arguments from the data config:
 
 - `Human.get_keymap(keymap_mode="cartesian"|"keypoints", has_head_pose=<bool>, include_aria_keypoints=<bool>)`
-- `Human.get_transform_list(mode="cartesian"|"keypoints_headframe_ypr"|..., stride=<int>)`
+- `Human.get_transform_list(mode="cartesian_6d"|"keypoints_headframe_ypr"|..., stride=<int>)`
 
 Onboarding human data is just two steps:
 
@@ -947,7 +947,7 @@ import torch
 HAS_HEAD_POSE = True
 key_map = Human.get_keymap(keymap_mode="cartesian", has_head_pose=HAS_HEAD_POSE)
 transform_list = (
-    Human.get_transform_list(mode="cartesian", stride=3) if HAS_HEAD_POSE else None
+    Human.get_transform_list(mode="cartesian_6d", stride=3) if HAS_HEAD_POSE else None
 )
 
 resolver = LocalEpisodeResolver(
@@ -987,7 +987,7 @@ from egomimic.rldb.filters import DatasetFilter
 resolver = LocalEpisodeResolver(
     folder_path    = "/local/processed",
     key_map        = Human.get_keymap(keymap_mode="cartesian"),
-    transform_list = Human.get_transform_list(mode="cartesian", stride=3),
+    transform_list = Human.get_transform_list(mode="cartesian_6d", stride=3),
 )
 filters = DatasetFilter(filter_lambdas=[
     "lambda row: row['episode_hash'] == '2026-03-15-14-22-10-000000'"

README.md

@@ -101,6 +101,7 @@ Every cluster-specific default in the configs can be overridden without editing
 | `EGOVERSE_CACHE_DIR` | `paths.cache_dir`; norm stats cached by content under `<cache_dir>/norm_stats/` | `$EGOVERSE_LOG_DIR/cache` |
 | `EGOVERSE_ROOT` | `paths.root_dir` | the directory you launch from |
 | `EGOVERSE_PI05_WEIGHTS` | pi0.5 base weights directory (`model.robomimic_model.config.pytorch_weight_path`) | lab PACE path |
+| `EGOVERSE_PALIGEMMA_WEIGHTS` | PaliGemma checkpoint for the from-scratch pi arm (`...config.paligemma_weight_path`), a local dir or an HF repo id | `google/paligemma-3b-pt-224` |
 | `WANDB_ENTITY`, `WANDB_PROJECT` | `logger.wandb.entity` / `.project` | `rl2-group` / `zarr_test` |
 
 ### Other Settings