[RyanPCo]

val: read the selected metric frames exactly

EvenStrideDataset went through MultiDataset.getitem, whose bounds check
swaps an out-of-quantile frame for an unseeded random frame of the same
episode. The val cohort drifted between passes and lost exactly the hard
frames.

With preserve_indices the wrapper reads base.datasets[episode][frame] (through
nested MultiDatasets, from the pose-glitch-masked index map), checks the leaf's
new frame_idx matches (a JPEG retry inside ZarrDataset moves it), fails on
non-finite tracked keys, then normalizes with the base's stats.
replace_invalid swaps a failing frame for another allowed start frame of the
same episode, seeded per (episode, frame) and logged.

The metric heads use both by default (data.exact_val_reads,
data.replace_invalid_val_frames). Pinned video loaders are unchanged.

Ported from PR #710 (EvenStrideDataset preserve_indices / replace_invalid,
ZarrDataset frame_idx); its random_seed sampling is not ported.

Co-Authored-By: Claude Opus 5.5 [email protected]
Claude-Session: https://claude.ai/code/session_01X86xQuci4asKUkiCwqMiKF

eval: drop padded steps from the layout metrics, add per-third MSE

Past an episode's end the action chunk repeats its last frame, which the model
predicts trivially, so tail samples deflated every in-training metric. The
layout and keypoint metrics now take the dataset's action_pad_mask: means are
per trajectory over real steps, then over the batch, and *_final reads each
trajectory's last real step. DTW / Frechet still see the whole chunk.

New *paired_mse{early,mid,late}_third for cartesian xyz / rot6d and the
keypoint wrist_xyz (the kp metric of record). A trajectory with no real step
in a third does not vote there; a third no trajectory reaches is omitted.

HPT and PI keep action_pad_mask in the processed batch (when it matches the
action horizon) so the evaluators can pass it; the loss masking is unchanged
and still gated by use_pad_mask.

Ported from PR #710's trajectory_mse / trajectory_mse_thirds; its
SampledTrajectoryMSE evaluator is not ported.

Co-Authored-By: Claude Opus 5.5 [email protected]
Claude-Session: https://claude.ai/code/session_01X86xQuci4asKUkiCwqMiKF

pl_model: restart the eval RNG at the first batch of every val loader

The pass counter was only rewound in on_validation_start, so a val set's
prompts and sampling noise depended on how many batches the loaders before
it had run: a change in another head's size, or a viz epoch running the
video loaders, shifted it. validation_step now rewinds it at batch_idx == 0
of each loader. The EMA swap stays in on_validation_start / end.

Also gives the on_validation_start test the ema attributes it needs since
the EMA change.

Ported from PR #710's per-cohort reset in validation_step.

Co-Authored-By: Claude Opus 5.5 [email protected]
Claude-Session: https://claude.ai/code/session_01X86xQuci4asKUkiCwqMiKF

Ported from ideas in #710 (no IDs taken). Combined stack: ruff clean; 562 targeted unit tests pass on CPU; no GPU smoke run yet.

🤖 Generated with Claude Code

https://claude.ai/code/session_01X86xQuci4asKUkiCwqMiKF

[RyanPCo]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• history: learned null token for dropped and episode-start-padded history #715
• mecka private: fixed-budget operator-scaling ladder generator #714
• train: hardened WSD defined in-config, W&B experiment tag, mode=validate on EMA weights #713
• val: exact-index metric frames, pad-masked metrics with thirds, per-loader eval RNG #712 👈 (View in Graphite)
• mecka private: mono converter, opscale configs and generators, IDs kept out of the repo #709
• data/rdt: keypoint RDT recipe, pose-glitch masking, hardened staging #708
• pl_model: RDT-style weight EMA, validated in place of raw weights #701
• data: stage episodes to node-local disk; never read them from /workspace #707
• pi: keep per-sample intrinsics so overlays use the episode camera #690 : 2 other dependent PRs (#691 , #716 )
• rdt: long-range visual memory; image memory window and history stride in seconds #689
• norm stats: cache sampled rows per episode so splits share them #687 : 1 other dependent PR (#688 )
• data: exclude episodes by hash; scan Mecka egomotion for failed head tracking #686
• eval: log validation videos to wandb #684 : 1 other dependent PR (#685 )
• slurm: stop cancelled jobs from leaking whole GPUs #682 : 1 other dependent PR (#683 )
• algo: RDT (Robotics Diffusion Transformer) on the HPT chassis #680 : 1 other dependent PR (#681 )
• pi: initialize from PaliGemma with a scratch action expert #677 : 1 other dependent PR (#678 )
• perf: vectorized image augs and a torch.compile knob for HPT and Pi; Pi clips gradients like openpi #676
• loader: one zarr read per key, not one per keymap entry #674
• eval: build the pinned-episode video batches only where they are used #673
• checkpoints: last.ckpt as a symlink, not a second full copy #672
• trainHydra: metric_frames_per_episode: auto, and warn on empty prompts #671
• transforms: per-step wrist framing for proprio history #670
• viz: one keypoint renderer, the horizon trace #669
• configs: history and backbone-LR knobs on the keypoint wrist HPT model #668
• trainer: ddp_bf16_compress, DDP with a bf16 gradient all-reduce #667
• eval: overlay videos play at the source rate under DDP #666
• hpt: build the policy on CPU and let Lightning place it #664 : 1 other dependent PR (#665 )
• hpt: never re-initialise pretrained stems; hash-check them; mask text padding #645 : 1 other dependent PR (#646 )
• obs: proprio history (K proprio steps, one token each, learned time embedding) #650
• loss: mask the repeat-last action padding out of the HPT loss #649
• resnet: keep a frozen backbone in eval mode; give backbones their own LR group #652
• augs: per-sample image augmentation for HPT train-mode augs #651
• eval: deterministic validation (seeded prompts + flow sampling); log-only grad-norm MAD #644
• data: mecka_fold_flagship_top3_hpt_6d, the 3-operator flagship train split #643
• val: subsampled metric loaders + pinned contiguous video loaders #642
• infra: mecka 6D / keypoint recipes, 300M models, defaults, and the rollout revert #640
• eval: layout-aware frame-invariant metrics, and a train-split viz head #659
• data: stop letting one bad episode or a rotation channel take a run down #658
• norm: guard degenerate channels, horizon pooling, and no rot6d normalization #657
• keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656
• mecka: fix the double-mirrored LEFT wrist frame, and correct existing zarrs #655
• 6d: PI goes 6D-only; Eva and Human gain the continuous-rotation modes #654
• 6d: continuous rotation representation, bimanual layouts, shared norm helpers #653
• train: give each Slurm slurm job gets its own cache dir to avoid conflict #638 : 1 other dependent PR (#639 )
• write git version into the run dir for reproducibility #637
• train: don't load PI base weights on resume and verify that checkpoint path exists #636
• configs: declare proprio and actions dims once per HPT model config #634 : 1 other dependent PR (#635 )
• cache norm stats for data configs to minimize recomputation #633
• import cloud packages lazily to support local training #631 : 1 other dependent PR (#632 )
• cache dataset resolver for faster training initialization #630
• validate episode hashes in data config before training #628
• modify pi camera map to match data #629
• add tests to ensure training works between every model and data vendor #626
• run CI tests on every PR #625
• add env variables for user-specific paths #624
• build: set pyproject and uv as the dependency managers #623
• fix(configs): fix paths for data hydra yaml configs #622
• refactor(algo): delete ACT #627 : 1 other dependent PR (#621 )
• tests: create dedicated test folder with unit and integration tests #620
• add pyright type checker #619
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.