eval: GT/prediction replay video, on by default by RyanPCo · Pull Request #685 · GaTech-RL2/EgoVerse

RyanPCo · 2026-09-23T13:27:18Z

Replay becomes the default validation video: viz_mode=replay plays each action chunk twice over the frames it spans — first the ground-truth hand at each step (green), then the predicted one (red) — instead of drawing the whole chunk as a trace on every frame.

Each step is drawn on its own frame. The chunk (in its anchor frame's camera) is mapped into that frame's camera by a rigid fit of the GT keypoints, so head motion is compensated without a head pose.
126-D keypoint actions render both hands' full 21-keypoint skeletons. Other layouts fall back to the embodiment's viz over a short trail (exact for Eva's fixed camera).
Chunks don't overlap; frames of an incomplete chunk carry over to the next flush.
_visualize_preds moves into EvalVideo (the HPT and PI copies were identical).
Default on. Replay needs consecutive frames, so it runs only on pinned video loaders (data.video_episodes); a head without one keeps the overlay video, as does EvalLatent. viz_mode=overlay restores the old video.
The replay video loader stops after replay_chunks (default 5) chunks rather than viz_max_batches, which at 8 frames per batch could not cover one 100-step chunk. viz_max_batches still bounds a loader that yields no clips.
action_stride is inferred from the GT keypoints (step k of a chunk rigidly matches frame k·stride's own step 0), since data configs use 1 or 3 and cotrain mixes them. Correct on 27/27 real-episode checks; set it explicitly to override.

Checked on rdt_3b epoch 999, mecka val episode: skeletons track the hands through head motion.

Known unrelated failure: tests/unit/test_eval_loaders.py::test_only_rank_zero_writes_the_video already fails on the parent (#684) — its fake trainer has no loggers.

🤖 Generated with Claude Code

https://claude.ai/code/session_01516MaELbiPNmJzkjPGebhu

RyanPCo · 2026-09-23T13:28:07Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

The pi0.5 converters stop doing rotation math. Actions now reach the algo already normalized and already in their model-native layout, so to32_norm_6d / from32_norm_6d are a pure rearrange into (and out of) openpi's 32-slot block layout: RobotBimanual's native 20D [L xyz 6d g | R xyz 6d g] IS that layout and only needs padding, HumanBimanual's 18D inserts a zero gripper slot per arm. BaseActionConverter raises on both, so an unregistered embodiment fails loudly. The four single-arm converters (RobotLeft/Right, HumanLeft/Right) are deleted along with PI._extract_xyz, which existed for the cam-frame MSE. PI's loss reduction and sample post-processing move into _reduce_loss and _postprocess_sampled_actions. The ypr->6D conversion is now a data transform. Eva gains cartesian_6d and cartesian_wristframe_6d, Human the same pair, each 6D-encoding the action AND the proprio ee_pose: normalized YPR saturates yaw/roll at +-pi (wraparound), so per-dim normalization is only meaningful on the continuous representation. get_revert_transform_list / the _build_*_revert_6d{,_wristframe}_ lists map a prediction batch back to cam-frame ypr for the viz overlay. Eva defaults to cartesian_wristframe_6d (eva, cotrain_pi_base, industry_eva_pi; cotrain_pi_lang_wrist now composes the same as cotrain_pi_lang). Human data configs move to cartesian_6d with pad_proprio_gripper, which pads the proprio ee_pose 18 -> 20 (zero grip slots at 9/19) so the pi0.5 State: prompt bins line up positionally with the robot 20D layout; UnpadGripperZeros inverts it in the revert lists. HPT dims follow: eva 14 -> 20, human 12 -> proprio 20 / action 18. With the 6D modes in place, a discontinuous rotation stops being a legal training target. get_transform_list raises on the cartesian ypr modes and on Eva's cartesian_wristframe_quat unless the caller passes an explicit allow_legacy_rotation=True, which only the data/rollout boundary does (the viz tutorial, rollout.py's legacy-checkpoint path, and the fixtures that use a cartesian pipeline as scaffolding). test_data_configs_compose asserts that no shipped data config selects such a mode or sets the opt-out, so the guard cannot be bypassed from a config. The five *_pickplace_qwen* data configs were the only ones still on Euler rotations; they and the two hpt_bc_pickplace_qwen model configs paired with them are deleted. The keypoint modes keep their ypr and quat variants for now -- the keypoints_*_6d modes do not exist yet. Also here, because they are the same widening: SplitKeys rejects a layout whose widths do not sum to the input (an evaluator revert list built for 12-dim ypr fed an 18/20-dim 6D batch used to slice silently), and vendor-tagged human embodiment names (MECKA_/SCALE_/ARIA_/LIGHTWHEEL_*) alias onto HUMAN_* through canonical_embodiment_name, so pins and embodiment checks accept the legacy metadata. Robot (EVA_*) names are never aliased. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp

… zarrs compute_hand_pose_xyzquat built the palm normal as cross(thumb_dir, pinky_dir). Thumb and pinky are anatomically swapped between hands, so that cross product already mirrors chirality for free; applying rot_left on top of it flipped x/y a second time and landed the LEFT hand on the RIGHT hand's spatial convention. Both hands now use the same rot_right relabel. Reconverting every affected zarr would be the obvious fix, so this avoids it: rot_left == rot_right @ diag(-1, -1, 1), which makes right-multiplying the stored left pose by Rz(180 deg) exactly equivalent to reconverting with rot_right. RotateLocalFrame does that right-multiply (R_new = R_old @ R_fix), relabelling the pose's own axes without moving its origin, on (7,) poses and (T, 7) chunks alike, and leaves zero-norm quaternions alone because those mark padded frames. Human.get_transform_list(fix_left_wrist_convention=True) prepends it to the raw LEFT pose keys before any frame math: ee_pose in the cartesian modes, wrist_pose in the keypoints modes, since the converter wrote the same rotation into both. The flag is a property of the DATA, not of the vendor: measured against the production set (1833 of 1838 episodes probed, 2026-09-18), every mecka episode AND all 20 scale episodes still carry the double-mirrored left wrist, with the right-hand control clean, so data/mecka.yaml and data/scale.yaml both opt in. Data reconverted after this commit must not, or it double-flips. Because that correctness lives in the data, left_wrist_convention() re-derives the verdict from an episode's own keypoints -- recomputing both wrists under each relabel and comparing with the stored quaternions, with the RIGHT hand as the control -- and S3EpisodeResolver samples a few resolved episodes at startup. Only the double-flip raises: the flag on against post-fix data would correct what the converter already corrected, which is a fresh error from a stale setting. The flag off against pre-fix data is what every run did before this commit, and the generic human configs (aria, human, the cotrains) have never set it, so that warns and names the knob rather than turning their startup into a crash. Five legacy 2025 episodes match neither convention (a third converter wrote them); their control fails, so they warn too. data/mecka_scale_cotrain_pi.yaml and data/industry_eva_pi.yaml are deleted: both merge mecka and scale into ONE human_bimanual dataset under a single transform_list, and the correction is per-lab, so neither file has a correct value for the flag. model/pi0.5_cotrain_mecka_scale.yaml goes with them. test_wrist6d_roundtrip runs the real transform lists on synthetic world-frame poses, parametrized over the flag, and compares every stage against plain numpy/scipy SE(3) math that shares no code with the pipeline; the convention checker is tested the same way, on synthetic keypoints. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp

…an default Human data trains on hand keypoints instead of a palm-origin cartesian pose. The action is 144-D wrist-first, per hand [wrist xyz (3) | wrist rot6d (6) | 21 MANO keypoints (63)] x {left, right}, built by the keypoints_{wrist,head}frame_6d modes: the existing ypr keypoint builders produce the 138-D layout and KeypointsYPRToRot6D re-encodes just the wrist rotation, leaving the keypoint blocks untouched. Both the action and the observations.state.keypoints proprio are converted, so neither carries a wrapping euler angle. MANO keypoint 0 IS the wrist, so per hand the wrist position rides at both ch 0-2 and ch 9-11; that is kept deliberately (21 keypoints is the MANO interface every consumer reshapes to) and written down in the bimanual_keypoint_layout docstring. include_ee_pose additionally builds the cartesian head-frame ee_pose next to the keypoint action (keymap + transform list), so a keypoint pi0.5 run discretizes the same State: prompt block as the cartesian runs and differs from them only in the action space. The cartesian builder runs with delete_target_world=False to keep obs_head_pose alive for the keypoint builder, and its actions_cartesian by-product is dropped. 144 does not fit openpi's action vector, whose PyTorch port hard-codes 32-wide action projections regardless of Pi0Config.action_dim. PI now reads action_dim and, when it is not 32, swaps in action_dim-wide action_in_proj / action_out_proj AFTER the strict base-weight load: only those two layers start fresh, everything else comes from the checkpoint, and sample_actions already draws noise at config.action_dim. pad_to_width zero-pads narrower embodiments (eva's 20-D) into the same vector and raises rather than truncating a wider one. _reduce_loss then restricts the mean to the slots the converter actually packs, so eva does not average its loss over 112 zero-padded dims and contribute a fraction of the gradient signal it had at width 32. HumanBimanualKeypoints is identity-in, slice-out. HPT human configs move to a 144-D state_keypoints stem. hpt_bc_flow_human holds the body -- there is no vendor notion at the embodiment level, so the canonical config is the embodiment one -- and hpt_bc_flow_{mecka,aria,scale} are two-line aliases of it. HPTModel.add_stem drops null stem entries so a child config can delete a stem its base declares, which hydra merges cannot express, and hpt_cotrain_flow_seperate_head uses it to drop state_ee_pose. With the keypoints_*_6d modes in place, the guard from the previous commit widens to the keypoints ypr and quat modes, so every remaining discontinuous rotation target needs an explicit allow_legacy_rotation=True (the aria->MANO and mecka keypoint viz tutorials, which read pre-6D data). The cartesian configs that were still pinned to the ypr widths move with it: hpt_bc_flow_human_cartesian and the cotrain base to human 20/18, the shared-head cotrains to a 20-wide shared action. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp

Wrist-relative actions made a latent division safe-looking: the t=0 cell of a wrist-frame chunk is exactly the identity pose, so its quantile range is 0 and the historical `(q99 - q1) + 1e-6` denominator scaled any off-convention value by 1e6. A channel whose range is below NORM_MIN_RANGE (1e-4) is now treated as constant -- it normalizes to 0 and unnormalizes to its centre rather than dividing by epsilon. Diffusion Policy / robomimic ignore ranges under 1e-4 and GR00T maps min == max to 0; this is the same rule, applied in the shared _apply_norm_one / _apply_unnorm_one so the dataset and the tests agree. norm_stats.pool_horizon (default false, so nothing changes unless asked) computes one set of stats per channel over every timestep of the chunk, the way openpi does, instead of one per (timestep, channel) cell. The pooled stats are tiled back to the (T, D) shape, so normalize, the bounds check and checkpoints see the same layout either way. It is part of the norm-stat cache key, so flipping it cannot silently reuse stats computed the other way. precompute_norm_stats.py computes and caches the stats for a recipe outside a training run, keyed by the same file + content hash. Which of the two is better is unsettled: the literature is split (LBM per-timestep with a stated rationale, openpi / RDT / GR00T / Diffusion Policy / UMI / lerobot / Octo pooled) and nothing here has compared them on a real recipe, so the default stays the inherited per-timestep one and the config comment says what would make it worth flipping. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp

A 6D rotation is two columns of a rotation matrix: already bounded in [-1, 1], and per-channel centre-and-scale pulls the pair off the manifold without buying any resolution. The two projects that anchor an action chunk at near-identity the way this repo does -- TRI's LBM and UMI -- both exclude rotation from normalization for exactly that reason (LBM: "We avoid this normalization procedure for the 6D rotation in the poses to avoid corrupting the rotation matrix"; UMI sends its rotation channels through an identity normalizer). Per timestep AND normalized was the one combination with no precedent, and it is what manufactures the t=0 zero-range rotation cells NORM_MIN_RANGE exists to catch -- this removes them at the source. rot6d_channels() reads the indices off the existing layout tables, so the rule follows the layout rather than a second hardcoded list: the cartesian 18/20 widths and the 144-D keypoint layout, action and proprio alike. The ypr widths keep the old behaviour; an Euler angle is not a matrix column and those modes are legacy-only now. _apply_norm_one / _apply_unnorm_one take the channels and pass them through untouched, so the dataset path, the evaluator revert and the rollout revert all follow from one place, and the degenerate-channel guard is left to do what it is actually good at: the structural kp0 / kp9 position zeros. Norm-stat files stay valid -- what changes is how they are applied -- but a run before this commit and a run after it are not comparable, and the loss is now weighted differently between translation (unit scale) and rotation (raw). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp

Three independent ways MultiDataset could fail a whole run on data that was fine, or on data it should have rejected up front. Bounds check. Quantile bounds on a rotation channel are meaningless: euler ypr wraps at +-pi and rot6d columns are matrix entries, so neither respects a quantile interval. _bounds_check_channels uses the layout tables to check only translation, gripper and keypoint channels; unrecognized widths still get the full-vector check, and the NaN/Inf check always covers everything. The bounds themselves get slack -- 1e-6 absolute plus half of each cell's own quantile range -- because the check is any-cell over ~13 200 cells per keypoint chunk. Measured on the 2815-frame flagship episode through the real keypoints_wristframe_6d pipeline (zscore, stats from sample_frac 0.10 over its own frames, 403 probe frames, any-key rejection): 39.5 % of frames rejected at slack 0, 17.9 % at 0.05, 11.4 % at 0.1, 5.2 % at 0.25, 3.2 % at 0.5, 1.0 % at 1.0. The 132-channel action chunk drives all of it. Half a range is the knee, not a cushion -- though the absolute numbers move with which frames the stats were drawn from, so the comment says how to reproduce them. Corrupt values sit orders of magnitude outside the range and are still caught. Cells the normalizer treats as constant are exempt, since a tiny offset there cannot blow up the loss -- and which stat decides that is norm_mode's business (std under zscore, the quantile range under quantile), so the exemption asks _degenerate instead of hardcoding q99 - q1; every shipped config is zscore, and a channel that is 0 in 99.6 % of samples and 1.0 in the rest has q99 - q1 = 0 but std = 0.063, i.e. it is still divided by and must still be checked. quantile_0_01 / quantile_99_99 now snap to observed samples ("lower" / "higher") instead of interpolating: under 10k samples the linear percentile lands strictly inside the sample range, so the extreme frames of the very episodes the stats came from were rejected (3-16 % of a single episode in the overfit runs). A stats/sample shape mismatch is logged once per key instead of silently disabling the check. Sampler. Retries stayed inside the failing episode until they exhausted it, so one wholly-bad episode killed the run and NCCL-timed-out every other rank. Retries now widen to the global index space after 25 attempts, and 100 consecutive failures raise. The cap is a systemic-failure detector, not a tolerance dial: attempts resets per __getitem__, so with p = the bad fraction of the whole index space it fires with probability p**(N - 25), a step function in p. At N = 100 the firing band sits at p >~ 0.8, so everything it can reach is systemic; at N = 1000 it needed ~99 % of the store to be bad and got there through ~975 sequential zarr/S3 reads per call per worker, which reads as a hang rather than a failure. Retry logging was one WARNING per attempt -- tens of thousands of lines per run -- now the first 20 per worker and then one in 500. Annotation cutoff. Many Scale-"completed" episodes have no usable annotation span because the injection step lagged, which makes the cutoff a silent no-op on exactly the runs whose point is to clamp chunks at annotation boundaries. S3AnnotationCutoffEpisodeResolver(require_annotations=True) drops them, logs the count it dropped and the count it kept, and raises if nothing is left. The per-episode span line drops to DEBUG: one line per episode per rank is thousands of lines on a real split, and the resolver's aggregate says the same thing. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp

Metrics were paired MSE over whatever the action vector happened to be, which mixes metres and rotation-matrix entries into one number that means nothing in either unit. action_metrics dispatches on the layout instead: cartesian widths report xyz MSE (m^2), geodesic rotation error in degrees (after Gram-Schmidt, so it survives a non-orthonormal prediction), raw rot6d MSE as a training-tracking diagnostic, DTW and Frechet; keypoint widths report keypoint L2 in metres, wrist xyz / rotation error and keypoint DTW. Each also splits early / mid / late (0-10 / 10-50 / 50-100 % of the chunk), because a chunk model can be right at t=0 and drift, and one average hides that. DTW forgives temporal misalignment that paired MSE punishes: a correct trajectory executed slightly early scores near zero. These are all frame-invariant -- the cam-frame revert is a rigid transform applied to prediction and ground truth alike -- so the revert no longer runs for metrics at all, only when rendering the overlay. That also removes the cam-frame MSE, which was the same number as the native one. Rendering is CPU-bound at roughly a second a frame and dominated viz-epoch wall time (unbounded rendering added ~2h per viz epoch on the combined pretrain), so viz_every_n_epochs gates it on Lightning's (current_epoch + 1) convention, the one validation actually runs on, with the final epoch always rendering so short runs get a video; viz_max_batches caps frames per rank. Metrics still run on every batch of every pass. Video fps is divided by world_size, since distributed validation deals indices round-robin and rank 0's consecutive frames are world_size source frames apart -- they were being written as a world_size-times timelapse. train_viz adds a second val loader over the train split (metrics train_viz/, videos videos_train_viz/), on by default for training runs with an evaluator, so train and val curves come from the same evaluator on the same axes. A third unseen_op_valid loader and data.valid_prefix let the held-out-operator splits log seen_op_valid/ and unseen_op_valid/ separately, instead of the seen operators landing in the train_viz slot and the unseen ones in Valid/. ModelWrapper dispatches on val_loader_names by dataloader_idx. val_at_start runs a pre-fit baseline validation, skipped on resumes because trainer.validate there would score the un-resumed base model. Reverse KL is removed: M forward passes per val batch for a number nothing was read off. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

The fold recipes this stack was built for: mecka_all_6d, mecka_fold_freeform_6d and its opsplit variant (wrist frame; cam frame is a transform_list.mode override rather than a near-duplicate config, and test_data_configs_compose rejects any data config that differs from its parent only in that mode), mecka_fold_flagship_opsplit_hpt_6d, mecka_fold_flagship_topop_hpt_6d and mecka_fold_freeform_opsplit_hpt_keypoints, plus the two 300M HPT models and pi0.5_bc_mecka_6d. Defaults, each for a failure seen in a run: - checkpoints every 50 epochs, saved on train-epoch end. With check_val_every_n_epoch > 1 the validation-end hook only fires on val epochs, so every_n_epochs silently skipped the others. - ddp_pi inference_mode: false. The compiled pi sampler does in-place setup on its static buffers during cudagraph capture, which is illegal on inference tensors ("Inplace update to inference tensor outside InferenceMode"); no_grad is equivalent for eval and creates none. - ddp_pi validates every 20 epochs instead of 200, and eval_hpt renders viz every 200. - pi0.5 gets a constant LR after warmup. The cosine schedule decayed to 0 over num_training_steps=60000, so runs past ~60k steps trained at ~0 lr and shorter runs never saw the configured lr for most of training. - The Qwen stem resolves a repo id to its cached snapshot directory when offline: transformers >= 4.57 probes the Hub while loading a tokenizer by repo id even under HF_HUB_OFFLINE, which raises on compute nodes with no internet. Loading from the directory skips the probe. Offline is read the way huggingface_hub reads it -- 1 / ON / YES / TRUE in any case, from the live environment -- rather than by matching the literal "1", which left HF_HUB_OFFLINE=true hitting the probe the helper exists to avoid. viz_mecka_episode.py and viz_eva_episode.py render an episode straight from its zarr, which is how the left-wrist convention bug was found. Both take the dataset dir from egomimic.utils.paths.default_dataset_dir(), which reads paths/default.yaml, instead of each re-stating the cluster path. HumanBimanualCartesianEuler / RobotBimanualCartesianEuler are renamed to HumanBimanualCartesian6D / RobotBimanualCartesian6D: both pack xyz + rot6d and have not touched an Euler angle since the 6D conversion, and this is the commit that makes the human one the converter for a new default recipe. Also prunes unit tests whose assertions another test already makes on the same code path, and parametrized cases that repeat a stricter test's compose call. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

…evert its outputs NOT TESTED ON HARDWARE. The rollout hard-coded cartesian_wristframe_ypr inputs and used the policy's predictions as-is. Now that eva checkpoints can be trained in several modes (cartesian, cartesian_6d, cartesian_wristframe_6d), that silently feeds a model the wrong inputs and sends the wrong frame to the arms. It now reads data.train_datasets.<name>.resolver.transform_list.mode off the checkpoint's own config, builds the matching input pipeline, and maps predictions back to camera-frame xyz+ypr+gripper through Eva.get_revert_transform_list. The revert needs the un-normalized proprio, which process_batch_for_training normalizes in place, so the raw ee_pose is cloned before that call. A checkpoint whose config records no mode keeps exactly the old behaviour -- legacy inputs, no revert -- and says so loudly, so this cannot silently change what an existing checkpoint does. test_eva_revert_for_rollout_recovers_camframe_ypr covers the revert math by cross-checking the wrist-frame revert against the cam-frame cartesian_6d one, which never enters the wrist frame -- so the frame composition is genuinely validated, though not against independently written SE(3) math. The hardware path itself still needs a real rollout. Both get_transform_list calls here pass allow_legacy_rotation=True: this is inference, and a checkpoint that predates the 6D conversion records exactly the modes the training-time guard rejects. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp

The unshuffled val loaders were truncated by limit_val_batches (80 x 64 x W frames) to their leading episode-hash-sorted slice, so at W = 1 the seen-op head scored 25 of 70 episodes and 6 of 11 operators, and the unseen head 33 of 191 episodes with an 81/19 operator mix on a 52/48 split (lane-0 diagnostic, 2026-09-15, measured single-rank). The loaders are unshuffled only so the overlay video is coherent, and the video renders just the first viz_max_batches (10) batches anyway. Split each val head in two: * metric loader = EvenStrideDataset(split, frames_per_episode=K) with K from data.metric_frames_per_episode[head], so every episode -- and so every operator -- is scored, at the same 80 batches and with no rendering; * video loader = <head>_video, the head's split restricted to the episode hashes pinned in data.video_episodes[head] (one per operator), contiguous and unsubsampled, logging no metrics. The pinned episodes are taken out of the already-resolved split, so resolve-once still holds. EvalVideo.on_validation_step gains a mode argument (metrics / video / both); ModelWrapper.validation_step routes <head>_video to the same evaluator in video mode and skips it entirely on a non-viz epoch, before the forward pass. Heads with no pins are called exactly as before, so a data config that declares neither key behaves as today. K = floor(5120 / episodes-in-split), from the measured splits: opsplit valid 73 (70 eps), train_viz 3 (1346 eps), unseen_op_valid 26 (191 eps); topop valid 256 (20 eps), train_viz 13 (385 eps). Those are SINGLE-RANK numbers and trainHydra scales them by trainer.devices x num_nodes, because DistributedSampler strides rather than chunks: the W ranks between them walk the whole split, so the pre-split window was already 80 x 64 x W frames wide. Without the scaling the shipped 4-GPU ddp_pi run would score 5,110 frames of the seen-val head where it previously scored all 13,744 -- a regression on the one head small enough to score whole. EvenStrideDataset keeps an episode entire when K exceeds its length, so the scaling cannot overrun a short split. Expect a shift in the per-batch Frechet _min/_max curves: they now span many episodes instead of ~2 s of one, so they are not comparable with older runs. Eval.on_validation_step, the abstract base, takes the mode kwarg too: the metrics/video path passes it unconditionally, so an evaluator that predates the split would have raised TypeError at the first val step of any config that declares video_episodes. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp

…split Three-operator twin of mecka_fold_flagship_opsplit_hpt_6d (and the wider sibling of the single-operator topop config): train on the 3 largest flagship fold-clothes operators (1,079 eps / 206,276 frames, 65.3 % of flagship), none of them held out. Seen-op val is the seeded 5 % episode split within those operators (53 eps / 9,826 frames); the unseen-operator head is inherited unchanged, so the runs stay comparable. Lane A keys measured from the real split on a compute node 2026-09-15: metric_frames_per_episode valid=floor(5120/53)=96, train_viz=floor(5120/1026)=4 (single-rank values; trainHydra scales them by the world size) (unseen 26 inherited); video_episodes pins one seen-val episode per train operator (191+223+226 = 640 f = exactly the 10-batch video budget) and inherits the train_viz and unseen pins. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp

…only grad-norm MAD Validating one checkpoint twice now gives identical prompts, actions and metrics. * HPT gets a per-pass RNG seed, base + rank * stride + pass counter, where the counter advances once per forward_eval call and is rewound by ModelWrapper.on_validation_start. forward_eval builds a torch.Generator from it and threads it down HPTModel.forward -> DenoisingPolicy.forward -> sample_action, and forks/seeds the global RNG around the BC val-loss call (DenoisingPolicy.predict draws its own noise and timestep, out of reach of any generator kwarg). Every generator kwarg defaults to None, so training and all other callers are untouched. The val-loss seed is offset by EVAL_LOSS_STRIDE rather than by 1: a torch.Generator seeded with S draws the same stream as a global manual_seed(S), so +1 would have handed batch i's val loss exactly the noise batch i+1 samples with. eval_base_seed is a real __init__ kwarg, so a second eval pass can be made to draw different noise on purpose. * _build_prompts picks annotations with that same seeded RNG whenever the algo is in eval mode, regardless of annotation_sampling_mode, so eval keeps prompt variety instead of collapsing to sample[0]. Train mode is unchanged. * The MAD grad-norm spike detector in on_after_backward is now log-only: it still logs the raw norm, the threshold, the flag and the [GRAD_NORM_SPIKE] line, but never rescales gradients, and every step (flagged or not) goes into the rolling median/MAD window. Fixed clipping moves to Lightning: trainer/default.yaml gains gradient_clip_val (null for now) and gradient_clip_algorithm: norm. The on_before_optimizer_step hook goes with it: Precision._after_closure calls that hook BEFORE _clip_gradients, so "policy_grad_norms_clipped" was the pre-clip norm under another name -- bit-identical to policy_grad_norms_raw, and no more informative once gradient_clip_val is set. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp

torchvision draws one random parameter set per call, so applying the configured train_image_augs to the stacked (B, 3, H, W) camera batch gave every image in the batch the same jitter. Wrap train_image_augs in a new PerSampleAugs module so each image draws its own parameters, and move the inline aug block of _robomimic_to_hpt_data into _apply_image_augs with the same gating. Deterministic eval augs stay batched. _apply_image_augs also checks train_image_augs before calling it: __init__ accepts None and train mode would have raised TypeError. The loop is not free: measured on an A40 at batch 64 with the shipped ColorJitter + Normalize, 13.8 -> 92.2 ms at 224x224 and 62.2 -> 105.5 ms at the camera's 360x640, per camera per training step. Correctness first here; the cost is removed further up the stack, where the jitter itself is vectorised over the batch. Co-Authored-By: Claude Fable 5.1 <[email protected]> Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp

An arm for training pi0.5 from the released VLM instead of the pi0.5 base checkpoint, to separate what the VLM pretraining buys from what pi0.5's own robot pretraining buys. The prefix (SigLIP tower, Gemma-2B, the projector) comes from google/paligemma-3b-pt-224; everything pi0.5 adds on top -- the Gemma-300M action expert, action_in/out_proj, the adaRMS time MLPs -- stays at its init. pytorch_weight_path and paligemma_weight_path are mutually exclusive rather than ordered by precedence: silently preferring one would make a typo in the other look like it worked. Setting neither now warns instead of loading nothing quietly. The loader streams shard by shard and then fails if any parameter went untouched or any checkpoint tensor went unused, because a partial load of a 3.6B model is invisible in the loss for a long time. Two places where the release does not line up with what openpi builds, both handled in the key remap: - transformers 5 flattened SiglipVisionModel, so vision_tower.vision_model.* is vision_tower.* there; below 5 (the pin, 4.53.2) the hop is still in the module tree. The loader maps onto whichever the live model has. Mapping to the 5.x spelling unconditionally failed 21 SigLIP tensors on the pin, which the key-string tests could not see; a unit test now loads a tiny real PaliGemmaForConditionalGeneration under the installed transformers, and passes on both 4.53.2 and 5.17.0. - The checkpoint pads the vocabulary to 257216, 64 rows past the 257152 openpi allocates (<image> plus 63 unused). embed_tokens and lm_head are truncated; openpi reaches the image slots through the vision tower, never through an embed_tokens lookup, so the dropped rows are unreachable. google/paligemma-3b-pt-224 is license-gated. With paligemma_allow_mirror: true, resolve_paligemma_dir falls back on a 401/403 (not on a rate limit or an outage) to ungated mirrors whose shards are sha256-checked against the canonical repo's LFS metadata first -- a mirror that has drifted is refused, not used. It is off by default: integrity is checked, but a research arm's weights of record should come from the canonical repo unless a run opts in, and this cluster's HF token is authorized for it. Only *.safetensors are pulled. openpi's own loader source is not an option: its GCS bucket 403s anonymously. pi0.5_paligemma_init is an overlay, not a runnable config, so a keypoint or 6D arm is four lines. It changes the init source only; the constant LR stays, since pi0.5_base documents dropping the cosine schedule on purpose. openpi_compat carries three shims for a transformers 5 venv, which openpi's pin (4.53.2, also this repo's) does not expect; all should be deleted once the fork's transformers_replace patch is ported forward. The third restores openpi's fp32 keep-list, which spells the pre-flattening vision paths, so on 5.x the SigLIP patch and position embeddings ran in bf16. On 5.x only this arm loads: the pi0.5 base checkpoint stores the nested SigLIP names and its strict load_model stops at startup. pi05.md says which setup is supported and what differs. Verified with a full trainHydra run to exit 0 on an H100 (4 epochs, all four validation heads, checkpoint and videos written) and the real 12 GB checkpoint into the real model in tests/integration, both on transformers 5.17, plus the unit tests on 4.53.2 and 5.17.0. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01Xh3wMw9pXYPHUSGjU7qmix

…isode fps in every sample Human.get_keymap(image_history_gap_s=...) adds a <front key>_hist camera entry; ZarrDataset reads it round(gap x fps) frames back, clamped at the episode start. Every sample also carries the episode's fps (zarr attr). The past frame names the front camera's zarr key, so the loader's one-call-per- key read covers both windows. JPEG keys are one frame per chunk, and a plain union span would read every frame in between; the read planner keeps them two rows. Per sample on a synthetic 30 fps episode of 360x640 JPEGs (median of 7x40 samples, login node, noisy): no history 11-12 ms; gap 0.1 s 21-24 -> 19-23 ms (the second decode, which history needs); 0.5 s 30-34 -> 20-21 ms; 2 s 53-61 -> 19 ms, flat in the gap. A test counts the JPEG rows read at 0.1, 0.5 and 2 s. Nothing consumes <front>_hist here; the keymap docstring says so. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_019DbPWJCKtVg5c2PNHs7egw

…spect ratio) Same call convention as ResNet. Loaded through timm: the locked transformers (4.53.2) has no DINOv3 class, and the timm hub repos are ungated. RoPE-only, so image_size may be [H, W]; exposes grid_size. pretrained_reference_state_dict reads the checkpoint back from the local HF cache (never the network), so the pretrained-weight hash check covers this tower as it does the Qwen stems; an uncached checkpoint or a random tower is skipped. Against the real timm/vit_base_patch16_dinov3.lvd1689m file the key sets match (162 tensors) and every value is equal. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_019DbPWJCKtVg5c2PNHs7egw

RDT(HPT) swaps the policy network: no Perceiver pooling; stems hand full token sequences to the head; the DiT blocks live in the trunk slot, shared across embodiments, and cross-attend to language / image tokens in alternating blocks; each FMPolicy head holds one embodiment's action adaptor, position table and output layer. DINOv3 ViT-B/L (fine-tuned at 0.1x) + frozen Qwen3-Embedding, 2-frame image history, ctrl-freq token, per-stream condition dropout. Configs: model/rdt_*; train_zarr_mecka_flagship_6d_rdt. The (past, current) pair goes through the train augs as (B * 2, 3, H, W) with frames=2: one jitter draw per sample covers both frames, and every op runs per frame. Laid side by side instead, ColorJitter's contrast took one mean over both frames (the past frame shifted by up to 0.20 with the current frame's content) and a Resize or CenterCrop would have mixed pixels across the seam. The pair also gets HPT's per-sample absent-camera guard, since both go through _apply_image_augs. compile.enabled=true leaves the trunk off HPT's compile list (no_trunk), but the DiT is still compiled: every head's denoiser calls it, so dynamo traces its blocks into the head's graph (asserted in test_rdt with a capturing backend). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_019DbPWJCKtVg5c2PNHs7egw

A cancelled DDP job left GPUs allocated with nvidia-smi blaming PIDs absent from /proc. The driver is fine: the memory is held by live processes reparented to init. gpu19 had 126 pt_data_worker survivors across three cancellations, each holding /dev/nvidia0-7, pinning 7 GPUs and ~1.7 TB of RAM. Workers fork after CUDA init, so they inherit the rank's device fds, and ranks open every visible GPU -- one survivor pins the node. A worker also inherits both ends of its result pipe, so a dead rank never sends EPIPE; it blocks in pipe_write and never reaches PyTorch's ppid check. Whole ranks leak the same way: one wedged in torch.cuda.synchronize() inside inductor cudagraph_trees ignores SIGTERM until Slurm gives up (UnkillableStepTimeout) and drops the job with its ranks still spinning. ProctrackType=proctrack/linuxproc walks the PPID tree, so it never sweeps anything reparented to init. egomimic/utils/gpu_orphans.py: * orphan_guarded/die_with_parent put PR_SET_PDEATHSIG on every worker, so the kernel kills it with its rank whatever it is blocked on. Wired into the train and val loaders and the norm-stats pass. * arm_rank_pdeathsig does the same for the rank process. A process whose parent is already gone when it arms exits 1, after one line on stderr saying why (os._exit skips logging's flush). SLURM_JOB_ID is also set inside a salloc shell, so a run started there with nohup is left unarmed (SIGHUP ignored): armed, it died with the shell it was meant to outlive. * reap_orphans runs from trainHydra.main before CUDA init and clears what an earlier job left, so a leak cannot outlive the job that made it. It only signals processes that are ours, reparented to init and holding an nvidia fd. Slurm-gated; EGOMIMIC_REAP_GPU_ORPHANS=0 (or false/no) disables it, any truthy spelling keeps it on, and anything else raises. reap_gpu_orphans.sh is the manual version for a node we are not submitting to. It reads PPid from /proc/<pid>/status, as the module does: in /proc/<pid>/stat, field 2 is comm, and a comm with spaces shifted field 4 far enough to read a live process's PPid as 1. The PDEATHSIG test waits for the unguarded worker to block on its pipe, which Linux >= 5.5 reports as pipe_write and 5.4 (sky2) as pipe_wait. It accepts either and skips on anything else, killing the worker first: failing there used to leave it running under init. The kill-switch test never scans the real /proc. The cluster-level fix is ProctrackType=proctrack/cgroup, which needs admin. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01DF1j2AzwzFztRsqvbTC35b

…e-rank trainer/ddp.yaml and ddp_pi.yaml set devices to gpus_per_node * nodes next to num_nodes, but Lightning reads devices per node and multiplies by num_nodes itself, so a 2-node run asked each node for twice the GPUs it has. The submitit launchers made the same mistake in gres, which Slurm also reads per node. Both now use gpus_per_node; single-node runs are unchanged. trainHydra scales each val head's frames-per-episode K by devices * num_nodes, but eval mode forces devices=1 only after the loaders are built. An eval run of a multi-GPU recipe therefore kept W times the frames per episode that its single rank's limit_val_batches window holds, and the window ran out before the last episodes. _trainer_world_size now returns 1 in eval mode. Both predate this stack (the ddp devices expression is from 2025-01). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01UkAZwERXcFKjueZQKfRF6J

Each mp4 EvalVideo writes is also logged to any WandbLogger under <video dir>/<embodiment> (videos/, videos_train_viz/, videos_unseen_op_valid/), captioned with the epoch, so runs can be reviewed without copying files off the cluster. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01415567AQYskZLeSpSMsgwL

viz_mode=replay plays each action chunk twice over the frames it spans: first the ground-truth hand at each step, then the predicted one. Each step is drawn on the frame it belongs to, with the chunk mapped into that frame's camera by a rigid fit of the GT keypoints (the same hand at the same instant in both frames), so head motion needs no head pose. The 126-D keypoint layout draws both hands' 21-keypoint skeletons; other layouts fall back to the embodiment's own viz over a short trail. Chunks don't overlap and span horizon * action_stride frames; frames of an incomplete chunk carry over to the next flush. viz_max_batches must cover at least one chunk (a 100-step horizon needs ~13 batches of 8). _visualize_preds moves from HPTEvalVideo / PIEvalVideo (identical copies) into EvalVideo. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01516MaELbiPNmJzkjPGebhu

Replay becomes EvalVideo's default. It needs consecutive frames, so it runs only on a pinned video loader; a head without one keeps the overlay video, and EvalLatent stays on overlay. The replay video loader is capped at replay_chunks (5) chunks instead of viz_max_batches, which at 8 frames a batch could not cover one 100-step chunk; viz_max_batches still bounds a loader that yields no clips. action_stride defaults to None and is inferred from the GT keypoints (step k of a chunk rigidly matches frame k*stride's own step 0): data configs use 1 or 3 and cotrain mixes them under one embodiment. TrainVizEvalVideo passes the replay settings through and tells its base which mode each step is. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01516MaELbiPNmJzkjPGebhu

RyanPCo · 2026-10-12T20:27:50Z

Folded into #684 to keep the stack under Graphite's 50-branch cap; the changes are unchanged there.

This was referenced Sep 23, 2026

slurm: stop cancelled jobs from leaking whole GPUs #682

Open

trainer: DDP devices are per node; eval-mode val subsampling is single-rank #683

Closed

This was referenced Sep 23, 2026

eval: log validation videos to wandb #684

Draft

data: exclude episodes by hash; scan Mecka egomotion for failed head tracking #686

Draft

RyanPCo changed the title ~~eval: GT/prediction replay video mode~~ eval: GT/prediction replay video, on by default Sep 23, 2026

RyanPCo force-pushed the ryanco/gt-pred-replay branch 2 times, most recently from 804fbf5 to 69a991a Compare September 23, 2026 17:59

RyanPCo force-pushed the ryanco/eval-video-wandb branch from fc272d2 to 4a21de7 Compare September 24, 2026 03:19

RyanPCo force-pushed the ryanco/gt-pred-replay branch from 69a991a to 7959307 Compare September 24, 2026 03:19

aidang3019 mentioned this pull request Sep 29, 2026

eva: joint-space mode for the ABC YAM action space; robot image history #702

Draft

RyanPCo and others added 13 commits September 29, 2026 21:04

RyanPCo and others added 9 commits September 29, 2026 21:04

aidang3019 force-pushed the ryanco/gt-pred-replay branch from 7959307 to e573cfd Compare September 30, 2026 01:07

aidang3019 mentioned this pull request Sep 30, 2026

ponytail(u1): drop bench/ and duplicate viz scripts, reuse gpu_orphans in the reaper #703

Draft

RyanPCo force-pushed the ryanco/eval-video-wandb branch from 388ce59 to b649f4e Compare October 12, 2026 20:26

RyanPCo closed this Oct 12, 2026

Navigation Menu

Repository navigation

eval: GT/prediction replay video, on by default - #685

eval: GT/prediction replay video, on by default#685
RyanPCo wants to merge 41 commits into
ryanco/eval-video-wandbfrom
ryanco/gt-pred-replay

RyanPCo commented Sep 23, 2026 •

edited

Loading

Uh oh!

RyanPCo commented Sep 23, 2026 •

edited

Loading

Uh oh!

RyanPCo commented Oct 12, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

RyanPCo commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RyanPCo commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RyanPCo commented Oct 12, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

RyanPCo commented Sep 23, 2026 •

edited

Loading

RyanPCo commented Sep 23, 2026 •

edited

Loading