[aidang3019]

ponytail(u1): drop bench/ and duplicate viz scripts, reuse gpu_orphans in the reaper

• delete bench/ (15 files): unreferenced harnesses; the shell drivers source a
  wt-env.sh that is not in the repo
• delete abc_process/viz_eva_episode.py, mecka_process/viz_mecka_episode.py and
  the empty abc_process package; the inspector (dataset_view.py) covers them
• AGENTS.md: the leaked-GPU section shrinks to a pointer at gpu_orphans.py
  (the admin-only proctrack/cgroup note moves into its docstring)
• reap_gpu_orphans.sh runs python -m egomimic.utils.gpu_orphans [--kill]
  instead of re-implementing find_orphans in bash; same CLI
• orphan_guarded: always set worker_init_fn=die_with_parent (no caller passes
  its own; DataLoader ignores it at num_workers=0)
• mecka_to_zarr: drop the dead rot_left local and unused hand_index parameter
• rollout.py: one get_transform_list call (mode or LEGACY), one ARM_EMBODIMENT map

Co-Authored-By: Claude Opus 5.5 [email protected]

ponytail(u2): shrink data/model configs, derive pad_proprio_gripper, drop dead configs

• mecka_20pct_6d: 4471 inline train pins -> resources/blocklists/mecka_20pct_6d_train.txt
  (4078 = pins minus the 393 mecka_bad_egomotion drops); DatasetFilter takes a
  hash-list name/file for episode_hashes via load_hash_list
• delete the six *_hpt_keypoints data twins; the keypoint delta lives once in the
  data: block of train_zarr_mecka_kp_wrist_hpt / train_zarr_mecka_flagship_kp_hpt
  (and their train_viz_hpt_keypoints_wrist default line is gone)
• model/hpt_bc_keypoints_wrist_300M inherits hpt_bc_mecka_6d_300M + delta
• delete data/mecka_fold_freeform_6d, data/cotrain_pi_lang_wrist,
  evaluator/eval_pi_wristframe, trainer/ddp_bf16_compress + utils/ddp_hooks.py
• freeform/flagship opsplit inherit mecka_all_6d; opscale4 inherits
  freeform opsplit; opscale1/2 set only train_operators (lambdas string-identical)
• flagship metric_frames_per_episode: auto (top3/topop overrides dropped)
• drop explicit train_viz_datasets/params (trainHydra's default reuses train)
• Human.get_transform_list derives pad_proprio_gripper; key dropped from configs
• human revert-6d builders: no action_key/obs_key kwargs, no duplicate
  SelectCurrentStep in the wristframe ones
• MultiDataModuleWrapper: **_config_only replaces four config-only kwargs
• tests: drop config-echo / golden-dims tests, repoint ladder + kp tests

Co-Authored-By: Claude Opus 5.5 [email protected]

ponytail(u3): one EvalVideo with a prefix, pipeline-derived replay stride

• delete TrainVizEvalVideo (eval_train_viz.py): EvalVideo takes prefix=
  (/ metric keys, videos_/); trainHydra builds the valid /
  train_viz / unseen_op_valid heads as instantiate(cfg.evaluator, prefix=...)
  in one loop (inlines _build_train_viz_evaluator and
  _build_unseen_op_valid_evaluator)
• delete evaluator/train_viz_{pi,pi_camframe_6d,pi_wristframe_6d,
  hpt_keypoints_wrist,hpt_wristframe_6d}.yaml; flagship_6d_hpt uses the
  default train_viz head; fold ladder drops train_viz_evaluator.base
• delete replay_viz.infer_stride/_interp_step and the _replay_stride /
  _replay_horizon / seen>=40 bookkeeping: trainHydra._action_strides derives
  frames-per-step from each dataset's keymap horizon + resampler
  (mecka 29/99, aria 27/99, eva 44/99); an explicit action_stride still wins
• drop EvalVideo knobs viz_mode, replay_chunks, replay_trail (class
  constants) and _video_fps (fps=30)
• PIEvalVideo's compute_metrics_and_viz becomes EvalVideo's default;
  HPTEvalVideo adds its main/aux/shared heads in one loop over the stateless
  _paired_mse (no torchmetrics) and keeps the cam-frame MSE (not
  revert-invariant: 18-D 6D wrist frame vs 12-D ypr head frame)
• drop metrics rot6d_paired_mse_avg, xyz_final_mse_avg, kp_l2_final,
  wrist_xyz_final_mse_avg and _split_mse
• tests: drop the single-process gate fit test and the duplicate
  closed-gate length test; replace infer_stride tests with a stride
  derivation test and a replay budget test; prefix test

Co-Authored-By: Claude Opus 5.5 [email protected]

ponytail(u4): cut models/algo/pl_model/trainHydra over-engineering

• hpt_nets: drop the pretrained weight-hash verification (hash_state_dict,
  verify_pretrained_weights, snapshot readers, DINOv3/Qwen reference
  overrides, _hpt_pretrained, HPT verify_pretrained). apply_skipping_pretrained
  (kept, with its tests) is the only init pass that reaches the stems.
• Deterministic eval: delete DeterministicEvalMixin + the generator plumbing
  through HPTModel.forward / heads / DenoisingPolicy. ModelWrapper now saves
  the RNG state and seeds python/numpy/torch with global_rank at
  on_validation_start and restores it at on_validation_end.
  _build_prompts moves back into HPT.
• Compile: algos return compile_targets(); ModelWrapper._maybe_compile calls
  nn.Module.compile on each. Algo's no-op and utils/compile_utils.py deleted.
• PaliGemma: drop the gated-repo mirror fallback and select_init_source
  (inlined into PI.init, both-paths check first).
• Qwen text cache: counters, stats/clear helpers, per-entry device check and
  the cache_text_features/text_cache_max_entries knobs gone; LRU capacity is
  TEXT_CACHE_MAX_ENTRIES.
• use_pad_mask knob + DenoisingPolicy.loss_fn's mask branch -> F.mse_loss;
  proprio_history_mask pop removed.
• RDT: drop image cond-drop (null_img, _drop_cameras).
• PI: inline _packed_width memo.
• image_augs: no per-image fallback; unbatchable train augs raise.
• trainHydra: _model_trainer_defaults removed (clip moves to
  trainer/default.yaml, u2); norm-stats loop is compute_norm_stats(), which
  scripts/precompute_norm_stats.py now calls; one set_norm_stats_from loop.
• Tests: source-reading tests and yaml/Lightning echo tests deleted, text-stem
  helpers shared in tests/fixtures/text_stems.py, test_per_sample_augs folded
  into test_image_augs.

Co-Authored-By: Claude Opus 5.5 [email protected]

ponytail(u5): shrink rldb transforms, pose layouts and norm-cache plumbing

• CartesianYPRToRot6D / CartesianRot6DToYPR / KeypointsYPRToRot6D /
  KeypointsRot6DToYPR: one per-arm _PerArmRotation (np.split + rot slice);
  unused output_key kwarg dropped. Bit-identical outputs and errors.
• S3AnnotationCutoffEpisodeResolver back to the 3-line class (no
  require_annotations / resolve override / _episode_has_annotation_spans);
  n_spans debug log dropped from _build_frame_to_ann_end.
• pose_utils: cartesian + keypoint layout tables generated by one
  _bimanual_layout helper (identical tables); key_layout shared by
  rot6d_channels and _bounds_check_channels (complement of rot);
  _rot6d_to_ypr reuses action_utils._reconstruct_R_from_cols.
• PoseCoordinateFrameTransform: per_step_target flag removed (per-step
  framing whenever target_history and shapes match, the old default).
• proprio_history_mask producer deleted (_proprio_history_spec/_mask + emit).
• MultiDataset._identity_channels inlined as rot6d_channels(...).
• InterpolatePadMask: np.interp threshold; mask_key kwarg dropped.
• PadGripperZeros / UnpadGripperZeros: np.insert / np.delete.
• norm cache: one _atomic_write helper; recipe_key via norm_cache_key;
  _git_sha lru wrapper dropped.
• is_legacy_vendor_embodiment and the dead try/except in
  _embodiment_matches removed.
• tests: shared tests/fixtures/poses.py (random SE(3) poses/chunks,
  transform runner, bounds-check shell); structural list tests replaced by
  test_wrist6d_roundtrip running cartesian_6d too; duplicate/echo tests cut.

Co-Authored-By: Claude Opus 5.5 [email protected]

ponytail: cross-unit follow-ups after merging u1-u5

• trainer/default.yaml: gradient_clip_val reads model.gradient_clip_val
  (pi0.5_base's 1.0) now that trainHydra._model_trainer_defaults is gone;
  an explicit trainer.gradient_clip_val still wins. Compose test added.
• tests: drop the deleted keypoint data config row from the vendor-pairing
  test, the removed PI._packed_widths / HPT.use_pad_mask attributes, and
  look for SelectCurrentStep anywhere in the human revert lists.
• comments/docs: stale references to TrainVizEvalVideo, eval_train_viz.py,
  EvalVideo._video_fps, per_step_target, proprio_history_mask, the deleted
  keypoint data config and the PaliGemma mirror fallback.

Co-Authored-By: Claude Opus 5.5 [email protected]

ponytail: verification fixes round 1

Tests restored or added for logic that still exists:

• opscale ladder: mode total on every split, valid_prefix, annotation_key,
  proprio_history K=1, fix_left_wrist_convention, and key_map/transform_list
  instantiated for the 6d rung and the kp recipe.
• flagship: top3 joins the opsplit val-head check; the kp recipe's
  evaluator revert, keypoints viz and "fold clothes" prompt; its data block
  is instantiated.
• PerSampleAugs grad flow and train_image_augs=None pass-through.
• train() val-head wiring (valid_prefix head, train_viz head from
  cfg.evaluator, pipeline action stride, prefixed metric keys).
• pi recipe (trainer=ddp_pi) keeps the model's 1.0 gradient clip.

Fixes:

• val loaders draw their worker/shuffle seeds from their own run-seeded
  generator, so opening them (which Lightning does before
  on_validation_start) no longer shifts the training torch RNG stream.
• precompute_norm_stats: the script's own flags now win over extra hydra
  overrides (a trailing norm_stats.save_cache_dir=null dropped the file).
• kp_wrist recipe comment no longer claims the flagship data works with it.
• stale .gitignore / integration-test comments; dead proprio_history_mask
  assert.

Declared u5 changes (no code change):

• _rot6d_to_ypr: within ~3e-7 rad of pitch +-90 deg the ypr may differ from
  the old numpy Gram-Schmidt (up to a representation flip at scipy's 1e-7
  gimbal threshold); the rotation itself matches.
• InterpolatePadMask: a length-1 mask is now thresholded at >0 like every
  longer mask (only reachable with non-binary masks, which nothing emits).

Co-Authored-By: Claude Opus 5.5 [email protected]

ponytail: verification fixes round 2

Fixes:

• trainHydra: the train_viz head is always instantiate(cfg.evaluator,
  prefix="train_viz"). A train_viz_evaluator key left in a config saved
  before u3 (TrainVizEvalVideo wrapper, deleted) is ignored instead of
  failing to instantiate; its base was always evaluator.
• PILatentEvalVideo: a prefixed head (train_viz, opsplit, valid_prefix)
  computes metrics only and registers no embed_prefix/k_proj hooks, as
  when it was wrapped. Two hooked heads chained embed_prefix wrappers and
  restored them out of order, so the first training step after validation
  called None (evaluator=eval_latent with train_viz on).
• precompute_norm_stats: the "cache now holds these stats" line prints
  only when norm_stats.cache_dir is set; logging.basicConfig restores the
  per-dataset progress lines compute_norm_stats logs.
• _action_pad_mask docstring no longer points at the deleted loss_fn.

Tests added for logic that still exists:

• kp 300M model's state_ee_pose: null (hpt_stems loop).
• two latent heads in ModelWrapper order leave embed_prefix original.
• HPT.compile_targets: trunk + Module heads + encoders, skips a head
  without an nn.Module .model.
• eval-mode _apply_image_augs passes an uncoded camera through.
• train() path: HPTModel.finalize_modules runs once; model
  enable_grad_norm=false reaches ModelWrapper.

Declared changes (no code change):

• u1: the viz-only calibrations (INTRINSICS eva/yam, EXTRINSICS
  abc_fold_viz/yam) lived only in viz_eva_episode.py; recover with
  git show 4179722:egomimic/scripts/abc_process/viz_eva_episode.py (L69-111).
• u2: data=cotrain_pi_lang_wrist and evaluator=eval_pi_wristframe no
  longer compose; old launch lines use data=cotrain_pi_lang
  evaluator=eval_pi, which compose the same.
• u5 InterpolatePadMask: a length-1 mask is thresholded at >0 before
  upsampling (0.8 -> 1.0); binary masks, the only ones emitted, are
  bit-identical.
• u5 _rot6d_to_ypr: <= ~4e-15 away from gimbal lock; within ~3e-7 rad
  of pitch +-90 deg yaw and roll can shift together by up to ~3e-9
  (1 float32 ulp). The rotation is the same.

Co-Authored-By: Claude Opus 5.5 [email protected]

ponytail: verification fixes round 3

Fixes:

• pose_utils: torch and action_utils._reconstruct_R_from_cols are imported
  inside _rot6d_to_ypr, so the module (and the dataset browser's
  inspector_lib/dataset_view, which imports it in a torch-free viz venv)
  loads without torch again.
• test_proprio_history: dropped the stale torchmetrics comment (HPTEvalVideo
  uses the stateless action_metrics._paired_mse now).

Tests added for logic that still exists (runtime checks, no source reading):

• train() settles a requeue's ckpt_path (_prepare_checkpoint_resume) before
  _build_model_config_tree; spy folded into the existing CPU-build train()
  test, so no extra run.
• importing egomimic.trainHydra (as a -m submitit launcher does) leaves
  TORCHINDUCTOR_CACHE_DIR unset.
• ModelWrapper does not override configure_gradient_clipping.

Declared changes (no code change):

• u4 HPT: HPT.init no longer hashes pretrained stems against their HF
  snapshot (verify_pretrained, default on). A corrupted init now trains
  instead of raising, and the "[hpt] pretrained weights of ... verified:
  sha256=..." log lines are gone. Startup is ~3 s and ~1.1 GB per rank
  cheaper. finalize_modules leaving pretrained weights untouched stays
  guarded by test_finalize_modules_leaves_pretrained_weights_untouched and
  test_finalize_modules_keeps_the_real_qwen_encoder_bit_identical.
• u4 DenoisingPolicy: under bf16 autocast (the shipped precision), the old
  all-ones masked mean divided by a bf16-rounded count; F.mse_loss divides
  by the exact BSD. Batches whose BSD is not exactly representable in
  bf16 (in practice only partial last batches: the last val batch of a head
  whose metric split is not a batch multiple, the last train batch of an
  epoch without limit_train_batches) change loss and gradient scale by up
  to ~2^-9 relative (0.27% at N=6000). Full batches are bit-identical. The
  exact mean is the intended value.
• u1: the viz-only calibrations (INTRINSICS eva/yam, EXTRINSICS
  abc_fold_viz/yam) lived only in viz_eva_episode.py and are gone from the
  tree; recover with
  git show 4179722:egomimic/scripts/abc_process/viz_eva_episode.py (L69-111)
  (4179722 = ponytail-base).
• u2 mecka_20pct_6d keeps K=1, but its train_viz head loses mecka_all_6d's
  bs-32 train_viz loader params. Under recipes with valid batch 64 it now
  scores all 4078 episodes (64 batches) instead of the first 2560 (80x32),
  so train_viz/ metrics change. The rdt_mem arms (valid bs 32) are
  unchanged. The config comment ("80 val batches x 64") always assumed this.

Co-Authored-By: Claude Opus 5.5 [email protected]

ponytail: reaper probe runs on the node's bare python3

reap_gpu_orphans.sh called python, which only exists with a venv active; the
bash probe it replaced had no dependencies. Call python3, and import str2bool
(numpy) inside reap_orphans so python3 -m egomimic.utils.gpu_orphans needs
only the stdlib.

Co-Authored-By: Claude Opus 5.5 [email protected]

[aidang3019]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• ponytail(u1): drop bench/ and duplicate viz scripts, reuse gpu_orphans in the reaper #703 👈 (View in Graphite)
• eva: joint-space mode for the ABC YAM action space; robot image history #702
• eval: replay maps chunk steps to real frames for resampled chunks #691
• pi: keep per-sample intrinsics so overlays use the episode camera #690 : 2 other dependent PRs (#707 , #716 )
• rdt: long-range visual memory; image memory window and history stride in seconds #689
• norm stats: cache sampled rows per episode so splits share them #687 : 1 other dependent PR (#688 )
• data: exclude episodes by hash; scan Mecka egomotion for failed head tracking #686
• eval: log validation videos to wandb #684 : 1 other dependent PR (#685 )
• slurm: stop cancelled jobs from leaking whole GPUs #682 : 1 other dependent PR (#683 )
• algo: RDT (Robotics Diffusion Transformer) on the HPT chassis #680 : 1 other dependent PR (#681 )
• pi: initialize from PaliGemma with a scratch action expert #677 : 1 other dependent PR (#678 )
• perf: vectorized image augs and a torch.compile knob for HPT and Pi; Pi clips gradients like openpi #676
• loader: one zarr read per key, not one per keymap entry #674
• eval: build the pinned-episode video batches only where they are used #673
• checkpoints: last.ckpt as a symlink, not a second full copy #672
• trainHydra: metric_frames_per_episode: auto, and warn on empty prompts #671
• transforms: per-step wrist framing for proprio history #670
• viz: one keypoint renderer, the horizon trace #669
• configs: history and backbone-LR knobs on the keypoint wrist HPT model #668
• trainer: ddp_bf16_compress, DDP with a bf16 gradient all-reduce #667
• eval: overlay videos play at the source rate under DDP #666
• hpt: build the policy on CPU and let Lightning place it #664 : 1 other dependent PR (#665 )
• hpt: never re-initialise pretrained stems; hash-check them; mask text padding #645 : 1 other dependent PR (#646 )
• obs: proprio history (K proprio steps, one token each, learned time embedding) #650
• loss: mask the repeat-last action padding out of the HPT loss #649
• resnet: keep a frozen backbone in eval mode; give backbones their own LR group #652
• augs: per-sample image augmentation for HPT train-mode augs #651
• eval: deterministic validation (seeded prompts + flow sampling); log-only grad-norm MAD #644
• data: mecka_fold_flagship_top3_hpt_6d, the 3-operator flagship train split #643
• val: subsampled metric loaders + pinned contiguous video loaders #642
• infra: mecka 6D / keypoint recipes, 300M models, defaults, and the rollout revert #640
• eval: layout-aware frame-invariant metrics, and a train-split viz head #659
• data: stop letting one bad episode or a rotation channel take a run down #658
• norm: guard degenerate channels, horizon pooling, and no rot6d normalization #657
• keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656
• mecka: fix the double-mirrored LEFT wrist frame, and correct existing zarrs #655
• 6d: PI goes 6D-only; Eva and Human gain the continuous-rotation modes #654
• 6d: continuous rotation representation, bimanual layouts, shared norm helpers #653
• train: give each Slurm slurm job gets its own cache dir to avoid conflict #638 : 1 other dependent PR (#639 )
• write git version into the run dir for reproducibility #637
• train: don't load PI base weights on resume and verify that checkpoint path exists #636
• configs: declare proprio and actions dims once per HPT model config #634 : 1 other dependent PR (#635 )
• cache norm stats for data configs to minimize recomputation #633
• import cloud packages lazily to support local training #631 : 1 other dependent PR (#632 )
• cache dataset resolver for faster training initialization #630
• validate episode hashes in data config before training #628
• modify pi camera map to match data #629
• add tests to ensure training works between every model and data vendor #626
• run CI tests on every PR #625
• add env variables for user-specific paths #624
• build: set pyproject and uv as the dependency managers #623
• fix(configs): fix paths for data hydra yaml configs #622
• refactor(algo): delete ACT #627 : 1 other dependent PR (#621 )
• tests: create dedicated test folder with unit and integration tests #620
• add pyright type checker #619
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.