[RyanPCo]

Adds an opt-in metric_frames_per_episode: auto for val heads and a one-time warning when a sample falls back to an empty prompt.

Behaviour changes

None by default. auto is opt-in, explicit ints keep their meaning, and no shipped config uses it yet. The prompt change is a log line only.

metric_frames_per_episode: auto

• trainHydra computes K = floor(limit_val_batches * batch_size / episodes) per head and dataset, from the resolved split and that head's loader batch_size, then scales by world size as before. K follows the live filters instead of a hand count that goes stale when a split is defined by SQL filters rather than an episode pin.
• limit_val_batches=0 (validation off) subsamples nothing.
• A non-int limit_val_batches, or a missing batch size or split, raises.

Other changes

• DeterministicEvalMixin._build_prompts warns once when a sample has no annotation and default_prompt is empty, so the text stem silently sees ''.
• mecka_fold_freeform_opsplit_6d stays frozen. It has no annotation guard (112 of its 462 episodes have segments = NULL), but checkpoints were trained on it. Its header now says so.

The operator-scaling ladder data configs (k = 1/2/4) and their test are kept out of this PR.

Tests

tests/unit/test_eval_loaders.py::test_auto_k_is_derived_from_the_resolved_split covers the formula, the world-size scaling, the limit_val_batches=0 pass-through and the error cases.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw

[RyanPCo]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• pi: initialize from PaliGemma with a scratch action expert #677 : 2 dependent PRs (#678 , #680 )
• perf: vectorized image augs and a torch.compile knob for HPT and Pi; Pi clips gradients like openpi #676
• loader: one zarr read per key, not one per keymap entry #674
• eval: build the pinned-episode video batches only where they are used #673
• checkpoints: last.ckpt as a symlink, not a second full copy #672
• trainHydra: metric_frames_per_episode: auto, and warn on empty prompts #671 👈 (View in Graphite)
• transforms: per-step wrist framing for proprio history #670
• viz: one keypoint renderer, the horizon trace #669
• configs: history and backbone-LR knobs on the keypoint wrist HPT model #668
• trainer: ddp_bf16_compress, DDP with a bf16 gradient all-reduce #667
• eval: overlay videos play at the source rate under DDP #666
• hpt: build the policy on CPU and let Lightning place it #664 : 1 other dependent PR (#665 )
• hpt: never re-initialise pretrained stems; hash-check them; mask text padding #645 : 1 other dependent PR (#646 )
• obs: proprio history (K proprio steps, one token each, learned time embedding) #650
• loss: mask the repeat-last action padding out of the HPT loss #649
• resnet: keep a frozen backbone in eval mode; give backbones their own LR group #652
• augs: per-sample image augmentation for HPT train-mode augs #651
• eval: deterministic validation (seeded prompts + flow sampling); log-only grad-norm MAD #644
• data: mecka_fold_flagship_top3_hpt_6d, the 3-operator flagship train split #643
• val: subsampled metric loaders + pinned contiguous video loaders #642
• infra: mecka 6D / keypoint recipes, 300M models, defaults, and the rollout revert #640
• eval: layout-aware frame-invariant metrics, and a train-split viz head #659
• data: stop letting one bad episode or a rotation channel take a run down #658
• norm: guard degenerate channels, horizon pooling, and no rot6d normalization #657
• keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656
• mecka: fix the double-mirrored LEFT wrist frame, and correct existing zarrs #655
• 6d: PI goes 6D-only; Eva and Human gain the continuous-rotation modes #654
• 6d: continuous rotation representation, bimanual layouts, shared norm helpers #653
• train: give each Slurm slurm job gets its own cache dir to avoid conflict #638 : 1 other dependent PR (#639 )
• write git version into the run dir for reproducibility #637
• train: don't load PI base weights on resume and verify that checkpoint path exists #636
• configs: declare proprio and actions dims once per HPT model config #634 : 1 other dependent PR (#635 )
• cache norm stats for data configs to minimize recomputation #633
• import cloud packages lazily to support local training #631 : 1 other dependent PR (#632 )
• cache dataset resolver for faster training initialization #630
• validate episode hashes in data config before training #628
• modify pi camera map to match data #629
• add tests to ensure training works between every model and data vendor #626
• run CI tests on every PR #625
• add env variables for user-specific paths #624
• build: set pyproject and uv as the dependency managers #623
• fix(configs): fix paths for data hydra yaml configs #622
• refactor(algo): delete ACT #627 : 1 other dependent PR (#621 )
• tests: create dedicated test folder with unit and integration tests #620
• add pyright type checker #619
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[RyanPCo]

Automated adversarial review (read-only; findings were checked by a second skeptical pass before posting): 1 high.

[RyanPCo]

high (correctness): When val_dataloader returns a list (the default, since train_viz is on), the val loaders get no DistributedSampler (#666 and #673 rely on this). Scaling K by _trainer_world_size therefore makes every rank read the same first limit_val_batches*batch_size frames of an episode-major set that is W times that size. On 8 GPUs, Valid/* and unseen_op_valid/* end up scoring only ~1/8 of the episodes, which defeats the per-episode subsampling, and the metrics change with GPU count. The multiplier predates this PR, but auto builds it into the recommended path. Drop the world-size factor whenever the metric loaders are unsharded (or shard them explicitly), and make the _trainer_world_size/_metric_frames_per_episode docstrings agree with EvalVideo._video_fps.

[RyanPCo]

Review findings: status after the fixup pass. The fixes are squashed into this PR's commits, and the whole stack was force-pushed.

Fixed

• metric_frames_per_episode (incl. new auto) is multiplied by world size, but the multi-loader val heads are not sharded: K is no longer multiplied by the world size, for auto and explicit values alike. Removed the now-unused, older _trainer_world_size (and slurm: stop cancelled jobs from leaking whole GPUs #682's eval-mode variant of it and its tests). Updated its test, the auto test and the flagship config comment (both outside the PR). Correct together with the val_dataloader change for eval: overlay videos play at the source rate under DDP #666.