[RyanPCo]

Branch 4 of 8 in the split of #640, which is the tip of the stack and carries the map of all eight. Branches 1-3 (#653-#655) are merged, so this one now sits directly on main.
Unit suite on this branch: 467 passed, 0 failed.

Human data trains on hand keypoints instead of a palm-origin cartesian pose.
The action is 144-D wrist-first, per hand
[wrist xyz (3) | wrist rot6d (6) | 21 MANO keypoints (63)] x {left, right},
built by the keypoints_{wrist,head}frame_6d modes: the existing ypr keypoint
builders produce the 138-D layout and KeypointsYPRToRot6D re-encodes just the
wrist rotation, leaving the keypoint blocks untouched. Both the action and the
observations.state.keypoints proprio are converted, so neither carries a
wrapping euler angle.

include_ee_pose additionally builds the cartesian head-frame ee_pose next to
the keypoint action (keymap + transform list), so a keypoint pi0.5 run
discretizes the same State: prompt block as the cartesian runs and differs
from them only in the action space. The cartesian builder runs with
delete_target_world=False to keep obs_head_pose alive for the keypoint
builder, and its actions_cartesian by-product is dropped.

144 does not fit openpi's action vector, whose PyTorch port hard-codes 32-wide
action projections regardless of Pi0Config.action_dim. PI now reads action_dim
and, when it is not 32, swaps in action_dim-wide action_in_proj /
action_out_proj AFTER the strict base-weight load: only those two layers start
fresh, everything else comes from the checkpoint, and sample_actions already
draws noise at config.action_dim. pad_to_width zero-pads narrower embodiments
(eva's 20-D) into the same vector and raises rather than truncating a wider
one. _reduce_loss then restricts the mean to the slots the converter actually
packs, so eva does not average its loss over 112 zero-padded dims and
contribute a fraction of the gradient signal it had at width 32.
HumanBimanualKeypoints is identity-in, slice-out.

HPT human configs move to a 144-D state_keypoints stem (hpt_bc_flow_mecka,
which human/aria/scale now alias); the cartesian variant survives as
hpt_bc_flow_human_cartesian. HPTModel.add_stem drops null stem entries so a
child config can delete a stem its base declares, which hydra merges cannot
express, and hpt_cotrain_flow_seperate_head uses it to drop state_ee_pose.
Shared-head cotrain configs need one action space and stay cartesian.

Co-Authored-By: Claude Opus 5 (1M context) [email protected]

🤖 Generated with Claude Code

https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw

[RyanPCo]

• hpt: never re-initialise pretrained stems; hash-check them; mask text padding #645 : 2 dependent PRs (#646 , #664 )
• obs: proprio history (K proprio steps, one token each, learned time embedding) #650
• loss: mask the repeat-last action padding out of the HPT loss #649
• resnet: keep a frozen backbone in eval mode; give backbones their own LR group #652
• augs: per-sample image augmentation for HPT train-mode augs #651
• eval: deterministic validation (seeded prompts + flow sampling); log-only grad-norm MAD #644
• data: mecka_fold_flagship_top3_hpt_6d, the 3-operator flagship train split #643
• val: subsampled metric loaders + pinned contiguous video loaders #642
• infra: mecka 6D / keypoint recipes, 300M models, defaults, and the rollout revert #640
• eval: layout-aware frame-invariant metrics, and a train-split viz head #659
• data: stop letting one bad episode or a rotation channel take a run down #658
• norm: guard degenerate channels, horizon pooling, and no rot6d normalization #657
• keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656 👈 (View in Graphite)
• mecka: fix the double-mirrored LEFT wrist frame, and correct existing zarrs #655
• 6d: PI goes 6D-only; Eva and Human gain the continuous-rotation modes #654
• 6d: continuous rotation representation, bimanual layouts, shared norm helpers #653
• train: give each Slurm slurm job gets its own cache dir to avoid conflict #638 : 1 other dependent PR (#639 )
• write git version into the run dir for reproducibility #637
• train: don't load PI base weights on resume and verify that checkpoint path exists #636
• configs: declare proprio and actions dims once per HPT model config #634 : 1 other dependent PR (#635 )
• cache norm stats for data configs to minimize recomputation #633
• import cloud packages lazily to support local training #631 : 1 other dependent PR (#632 )
• cache dataset resolver for faster training initialization #630
• validate episode hashes in data config before training #628
• modify pi camera map to match data #629
• add tests to ensure training works between every model and data vendor #626
• run CI tests on every PR #625
• add env variables for user-specific paths #624
• build: set pyproject and uv as the dependency managers #623
• fix(configs): fix paths for data hydra yaml configs #622
• refactor(algo): delete ACT #627 : 1 other dependent PR (#621 )
• tests: create dedicated test folder with unit and integration tests #620
• add pyright type checker #619
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[RyanPCo]

Automated adversarial review (read-only; findings were checked by a second skeptical pass before posting): 1 high, 1 medium, 2 low, 6 nit.

[RyanPCo]

high (correctness): The human ac_key is now actions_keypoints, but the default evaluator still visualizes actions_cartesian. train_zarr_cartesian mounts eval_hpt -> viz/cartesian.yaml, so Human.viz_gt_preds looks up predictions["human_bimanual_actions_cartesian"] and raises KeyError on the first validation sanity step (e.g. model=hpt_bc_flow_mecka data=mecka). eval_pi.yaml has the same problem: it still uses _build_human_cartesian_revert_eef_frame_transform_list and pi_cartesian viz for human_bimanual. Have the human recipes select a keypoints evaluator (viz keypoints mode on actions_keypoints, plus _build_human_keypoints_revert_6d_wristframe_transform_list), and add a test that runs one validation step instead of ~evaluator.

[RyanPCo]

medium (dead-code): _build_human_keypoints_revert_6d_transform_list and _build_human_keypoints_revert_6d_wristframe_transform_list are only used by tests. eval_pi.yaml and eval_hpt_wrist.yaml still map human_bimanual to _build_human_cartesian_revert_eef_frame_transform_list, which splits actions_cartesian. With the new keypoint defaults (e.g. train_zarr_cartesian_pi + model=pi0.5_bc_mecka + data=mecka), that key is missing from the batch, so the cam-frame validation path breaks. Please either add a keypoint evaluator config (or a Human.get_revert_transform_list(mode) dispatch like Eva's) that points at the wristframe keypoint revert, or drop these builders until something uses them.

[RyanPCo]

low (duplication): This file is a full copy of the old hpt_bc_flow_human.yaml body (trunk, encoders, augs, optimizer, scheduler), and it has already drifted: reverse_kl_samples: 8 is missing here. Consider defaults: [hpt_bc_flow_human, _self_] and override only dims.human_bimanual: {proprio: 20, action: 18}, ac_keys.human_bimanual: actions_cartesian, stem_specs.human_bimanual.state_keypoints: null and a state_ee_pose stem. That uses the null-stem drop this PR adds to HPTModel. The camera_transforms history comment can go too, since this PR deletes it from the parent.

[RyanPCo]

low (simplification): The packed width is a static property of each converter: the cartesian converters always _pad32, and HumanBimanualKeypoints already declares native_dim = 144. Instead of running a forward to32_norm_6d and caching it in self._packed_widths, add a packed_dim class attribute to the converters and read self.action_registry.get(emb, ac_key).packed_dim in _reduce_loss. Then _packed_widths and _packed_width can go.

[RyanPCo]

nit (style): This file gains trailing whitespace (diffusion: true , human_bimanual: , whitespace-only blank lines) and a bare # TODO on pretrained_checkpoint. Please strip the whitespace and either drop the TODO or say what is pending.

[RyanPCo]

nit (simplification): The two inverse transforms derive offsets differently: KeypointsYPRToRot6D reads them from bimanual_keypoint_layout (and recomputes xyz on every loop iteration), while this one hard-codes o + 3 / o + 9. Pick one style for both and hoist the offsets out of the loop. The guard can be just if arr.shape[-1] != 144:, because a 144 width always has a layout.

[RyanPCo]

nit (magic-number): action32 now holds an action_dim-wide (e.g. 144) vector, so the name is misleading. Rename it (e.g. action_vec). Also consider an OPENPI_PRETRAINED_ACTION_DIM = 32 constant for the self.action_dim != 32 check, its log message and _pad32.

[RyanPCo]

nit (comments): '(the 138-D ypr layout of the keypoints_*_ypr modes is no longer the default)' describes the change, not the config. Keep only the layout description, and drop the extra blank line added after dims.

[RyanPCo]

nit (cleanliness): This file brings back trailing whitespace (diffusion: true , human_bimanual: , blank lines with spaces) and the "" # TODO that main had cleaned up. It also drops the comment explaining why camera_transforms: {} is still passed, which hpt_bc_flow_human_cartesian.yaml keeps. Restore main's formatting and that comment, and change only dims, ac_keys and the stem name.

[RyanPCo]

nit (simplification): layout is None or arr.shape[-1] != 144 reduces to arr.shape[-1] != 144. The two inverse transforms also index the same layout differently: KeypointsYPRToRot6D derives offsets from bimanual_keypoint_layout (and recomputes xyz inside the loop), while this one hard-codes o+3/o+9. Pick one style for both so they can't drift apart.

[RyanPCo]

Review findings: status after the fixup pass. The fixes are squashed into this PR's commits, and the whole stack was force-pushed.

Fixed

• 134-line copy of hpt_bc_flow_human instead of inheriting it: Now defaults [hpt_bc_flow_human, self] and overrides only dims, ac_key, state_keypoints: null and the state_ee_pose stem.
• Added trailing whitespace and a bare # TODO: Stripped the whitespace in a keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656 fixup. The TODO is removed in a eval: layout-aware frame-invariant metrics, and a train-split viz head #659 fixup because that line is adjacent to eval: layout-aware frame-invariant metrics, and a train-split viz head #659's reverse_kl_samples removal.
• KeypointsRot6DToYPR hard-codes offsets its sibling reads from the layout: Both transforms read xyz/rot widths from the layout, hoisted out of the loop. The guard is now just shape != 144.
• _packed_width runs a real forward conversion and keeps a cache just to read a static width: Converters declare a packed_dim class attribute (32, or 144 for keypoints). _reduce_loss reads it. Removed _packed_widths/_packed_width and the test setup they needed; rldb, algo, eval: leave an untracked hand out of losses, stats and bounds #725's hand-mask loss reads packed_dim too.
• Comment references the change; extra blank line added: Comment reduced to the layout description. eval: layout-aware frame-invariant metrics, and a train-split viz head #659 already removes the extra blank line.
• hpt_bc_flow_human.yaml regresses earlier cleanup (trailing whitespace, TODO, deleted rationale comment): Restored the camera_transforms rationale comment and stripped the whitespace. TODO removed (in the eval: layout-aware frame-invariant metrics, and a train-split viz head #659 fixup).
• KeypointsRot6DToYPR width check is redundant, and the two keypoint transforms index the layout differently: Same fix as the KeypointsRot6DToYPR offsets finding.

Partial

• openpi pretrained width 32 is a bare literal in several places: Added OPENPI_PRETRAINED_ACTION_DIM, used in pi.py and _pad32; renamed action32 to packed. Did not rename the to32_norm_6d method, which would ripple across the codebase.
• Human HPT/PI recipes now emit actions_keypoints, but the default evaluators still visualize actions_cartesian: Fixed at the tip by eval: layout-aware frame-invariant metrics, and a train-split viz head #659's evaluator switch. Added a test in eval: layout-aware frame-invariant metrics, and a train-split viz head #659 checking the default HPT/PI evaluators' viz action_key and revert match each human recipe. keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656-data: stop letting one bad episode or a rotation channel take a run down #658 on their own stay broken: moving eval: layout-aware frame-invariant metrics, and a train-split viz head #659's evaluator hunks into keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656 would conflict on autosquash.

Not fixed

• Keypoint 6D revert builders are never wired into any evaluator: Already wired at the stack tip: eval: layout-aware frame-invariant metrics, and a train-split viz head #659 switches eval_hpt/eval_pi to the keypoint revert. Moving those hunks into keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656 cannot be done as a fixup without autosquash conflicts. Added a config test in eval: layout-aware frame-invariant metrics, and a train-split viz head #659 (see the keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656 high finding).