Repository navigation
Conversation
Contributor
Author
This was referenced Sep 15, 2026
RyanPCo
force-pushed
the
ryanco/flagship-top3-config
branch
from
September 17, 2026 14:52
994dae9 to
84e177a
Compare
RyanPCo
force-pushed
the
ryanco/eval-loaders
branch
from
September 17, 2026 14:52
c314000 to
690b00f
Compare
This was referenced Sep 17, 2026
RyanPCo
force-pushed
the
ryanco/flagship-top3-config
branch
from
October 12, 2026 23:19
c76e2ad to
c77a643
Compare
Open
…an default
Human data trains on hand keypoints instead of a palm-origin cartesian pose.
The action is 144-D wrist-first, per hand
[wrist xyz (3) | wrist rot6d (6) | 21 MANO keypoints (63)] x {left, right},
built by the keypoints_{wrist,head}frame_6d modes: the existing ypr keypoint
builders produce the 138-D layout and KeypointsYPRToRot6D re-encodes just the
wrist rotation, leaving the keypoint blocks untouched. Both the action and the
observations.state.keypoints proprio are converted, so neither carries a
wrapping euler angle. MANO keypoint 0 IS the wrist, so per hand the wrist
position rides at both ch 0-2 and ch 9-11; that is kept deliberately (21
keypoints is the MANO interface every consumer reshapes to) and written down
in the bimanual_keypoint_layout docstring.
include_ee_pose additionally builds the cartesian head-frame ee_pose next to
the keypoint action (keymap + transform list), so a keypoint pi0.5 run
discretizes the same State: prompt block as the cartesian runs and differs
from them only in the action space. The cartesian builder runs with
delete_target_world=False to keep obs_head_pose alive for the keypoint
builder, and its actions_cartesian by-product is dropped.
144 does not fit openpi's action vector, whose PyTorch port hard-codes 32-wide
action projections regardless of Pi0Config.action_dim. PI now reads action_dim
and, when it is not 32, swaps in action_dim-wide action_in_proj /
action_out_proj AFTER the strict base-weight load: only those two layers start
fresh, everything else comes from the checkpoint, and sample_actions already
draws noise at config.action_dim. pad_to_width zero-pads narrower embodiments
(eva's 20-D) into the same vector and raises rather than truncating a wider
one. _reduce_loss then restricts the mean to the slots the converter actually
packs, so eva does not average its loss over 112 zero-padded dims and
contribute a fraction of the gradient signal it had at width 32.
HumanBimanualKeypoints is identity-in, slice-out.
HPT human configs move to a 144-D state_keypoints stem. hpt_bc_flow_human
holds the body -- there is no vendor notion at the embodiment level, so the
canonical config is the embodiment one -- and hpt_bc_flow_{mecka,aria,scale}
are two-line aliases of it. HPTModel.add_stem drops null stem entries so a
child config can delete a stem its base declares, which hydra merges cannot
express, and hpt_cotrain_flow_seperate_head uses it to drop state_ee_pose.
With the keypoints_*_6d modes in place, the guard from the previous commit
widens to the keypoints ypr and quat modes, so every remaining discontinuous
rotation target needs an explicit allow_legacy_rotation=True (the aria->MANO
and mecka keypoint viz tutorials, which read pre-6D data). The cartesian
configs that were still pinned to the ypr widths move with it:
hpt_bc_flow_human_cartesian and the cotrain base to human 20/18, the
shared-head cotrains to a 20-wide shared action.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
Wrist-relative actions made a latent division safe-looking: the t=0 cell of a wrist-frame chunk is exactly the identity pose, so its quantile range is 0 and the historical `(q99 - q1) + 1e-6` denominator scaled any off-convention value by 1e6. A channel whose range is below NORM_MIN_RANGE (1e-4) is now treated as constant -- it normalizes to 0 and unnormalizes to its centre rather than dividing by epsilon. Diffusion Policy / robomimic ignore ranges under 1e-4 and GR00T maps min == max to 0; this is the same rule, applied in the shared _apply_norm_one / _apply_unnorm_one so the dataset and the tests agree. norm_stats.pool_horizon (default false, so nothing changes unless asked) computes one set of stats per channel over every timestep of the chunk, the way openpi does, instead of one per (timestep, channel) cell. The pooled stats are tiled back to the (T, D) shape, so normalize, the bounds check and checkpoints see the same layout either way. It is part of the norm-stat cache key, so flipping it cannot silently reuse stats computed the other way. precompute_norm_stats.py computes and caches the stats for a recipe outside a training run, keyed by the same file + content hash. Which of the two is better is unsettled: the literature is split (LBM per-timestep with a stated rationale, openpi / RDT / GR00T / Diffusion Policy / UMI / lerobot / Octo pooled) and nothing here has compared them on a real recipe, so the default stays the inherited per-timestep one and the config comment says what would make it worth flipping. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
A 6D rotation is two columns of a rotation matrix: already bounded in [-1, 1], and per-channel centre-and-scale pulls the pair off the manifold without buying any resolution. The two projects that anchor an action chunk at near-identity the way this repo does -- TRI's LBM and UMI -- both exclude rotation from normalization for exactly that reason (LBM: "We avoid this normalization procedure for the 6D rotation in the poses to avoid corrupting the rotation matrix"; UMI sends its rotation channels through an identity normalizer). Per timestep AND normalized was the one combination with no precedent, and it is what manufactures the t=0 zero-range rotation cells NORM_MIN_RANGE exists to catch -- this removes them at the source. rot6d_channels() reads the indices off the existing layout tables, so the rule follows the layout rather than a second hardcoded list: the cartesian 18/20 widths and the 144-D keypoint layout, action and proprio alike. The ypr widths keep the old behaviour; an Euler angle is not a matrix column and those modes are legacy-only now. _apply_norm_one / _apply_unnorm_one take the channels and pass them through untouched, so the dataset path, the evaluator revert and the rollout revert all follow from one place, and the degenerate-channel guard is left to do what it is actually good at: the structural kp0 / kp9 position zeros. Norm-stat files stay valid -- what changes is how they are applied -- but a run before this commit and a run after it are not comparable, and the loss is now weighted differently between translation (unit scale) and rotation (raw). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
MultiDataset.to_state now records rot6d_passthrough. A checkpoint state that has stats but no flag predates the passthrough and normalized the rot6d channels, so from_state turns it off for those; fresh stats (empty state) keep it on. Without this, eval/rescore/requeue of such a checkpoint fed raw rot6d proprio and left predicted rotations in normalized space, silently. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GqAcWA2Lk9zubUhKX5DW1v
Three independent ways MultiDataset could fail a whole run on data that was
fine, or on data it should have rejected up front.
Bounds check. Quantile bounds on a rotation channel are meaningless: euler ypr
wraps at +-pi and rot6d columns are matrix entries, so neither respects a
quantile interval. _bounds_check_channels uses the layout tables to check only
translation, gripper and keypoint channels; unrecognized widths still get the
full-vector check, and the NaN/Inf check always covers everything. The bounds
themselves get slack -- 1e-6 absolute plus half of each cell's own quantile
range -- because the check is any-cell over ~13 200 cells per keypoint chunk.
Measured on the 2815-frame flagship episode through the real
keypoints_wristframe_6d pipeline (zscore, stats from sample_frac 0.10 over its
own frames, 403 probe frames, any-key rejection): 39.5 % of frames rejected at
slack 0, 17.9 % at 0.05, 11.4 % at 0.1, 5.2 % at 0.25, 3.2 % at 0.5, 1.0 % at
1.0. The 132-channel action chunk drives all of it. Half a range is the knee,
not a cushion -- though the absolute numbers move with which frames the stats
were drawn from, so the comment says how to reproduce them.
Corrupt values sit orders of magnitude outside the range and are still caught.
Cells the normalizer treats as constant are exempt, since a tiny offset there
cannot blow up the loss -- and which stat decides that is norm_mode's business
(std under zscore, the quantile range under quantile), so the exemption asks
_degenerate instead of hardcoding q99 - q1; every shipped config is zscore, and
a channel that is 0 in 99.6 % of samples and 1.0 in the rest has q99 - q1 = 0
but std = 0.063, i.e. it is still divided by and must still be checked.
quantile_0_01 / quantile_99_99 now snap to observed samples ("lower" /
"higher") instead of interpolating: under 10k samples the linear percentile
lands strictly inside the sample range, so the extreme frames of the very
episodes the stats came from were rejected (3-16 % of a single episode in the
overfit runs). A stats/sample shape mismatch is logged once per key instead of
silently disabling the check.
Sampler. Retries stayed inside the failing episode until they exhausted it,
so one wholly-bad episode killed the run and NCCL-timed-out every other rank.
Retries now widen to the global index space after 25 attempts, and 100
consecutive failures raise. The cap is a systemic-failure detector, not a
tolerance dial: attempts resets per __getitem__, so with p = the bad fraction
of the whole index space it fires with probability p**(N - 25), a step function
in p. At N = 100 the firing band sits at p >~ 0.8, so everything it can reach
is systemic; at N = 1000 it needed ~99 % of the store to be bad and got there
through ~975 sequential zarr/S3 reads per call per worker, which reads as a
hang rather than a failure. Retry logging was one WARNING per attempt -- tens
of thousands of lines per run -- now the first 20 per worker and then one in
500.
Annotation cutoff. Many Scale-"completed" episodes have no usable annotation
span because the injection step lagged, which makes the cutoff a silent no-op
on exactly the runs whose point is to clamp chunks at annotation boundaries.
S3AnnotationCutoffEpisodeResolver(require_annotations=True) drops them, logs
the count it dropped and the count it kept, and raises if nothing is left. The
per-episode span line drops to DEBUG: one line per episode per rank is
thousands of lines on a real split, and the resolver's aggregate says the same
thing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
Metrics were paired MSE over whatever the action vector happened to be, which mixes metres and rotation-matrix entries into one number that means nothing in either unit. action_metrics dispatches on the layout instead: cartesian widths report xyz MSE (m^2), geodesic rotation error in degrees (after Gram-Schmidt, so it survives a non-orthonormal prediction), raw rot6d MSE as a training-tracking diagnostic, DTW and Frechet; keypoint widths report keypoint L2 in metres, wrist xyz / rotation error and keypoint DTW. Each also splits early / mid / late (0-10 / 10-50 / 50-100 % of the chunk), because a chunk model can be right at t=0 and drift, and one average hides that. DTW forgives temporal misalignment that paired MSE punishes: a correct trajectory executed slightly early scores near zero. These are all frame-invariant -- the cam-frame revert is a rigid transform applied to prediction and ground truth alike -- so the revert no longer runs for metrics at all, only when rendering the overlay. That also removes the cam-frame MSE, which was the same number as the native one. Rendering is CPU-bound at roughly a second a frame and dominated viz-epoch wall time (unbounded rendering added ~2h per viz epoch on the combined pretrain), so viz_every_n_epochs gates it on Lightning's (current_epoch + 1) convention, the one validation actually runs on, with the final epoch always rendering so short runs get a video; viz_max_batches caps frames per rank. Metrics still run on every batch of every pass. Video fps is divided by world_size, since distributed validation deals indices round-robin and rank 0's consecutive frames are world_size source frames apart -- they were being written as a world_size-times timelapse. train_viz adds a second val loader over the train split (metrics train_viz/, videos videos_train_viz/), on by default for training runs with an evaluator, so train and val curves come from the same evaluator on the same axes. A third unseen_op_valid loader and data.valid_prefix let the held-out-operator splits log seen_op_valid/ and unseen_op_valid/ separately, instead of the seen operators landing in the train_viz slot and the unseen ones in Valid/. ModelWrapper dispatches on val_loader_names by dataloader_idx. val_at_start runs a pre-fit baseline validation, skipped on resumes because trainer.validate there would score the un-resumed base model. Reverse KL is removed: M forward passes per val batch for a number nothing was read off. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The fold recipes this stack was built for: mecka_all_6d, mecka_fold_freeform_6d
and its opsplit variant (wrist frame; cam frame is a transform_list.mode
override rather than a near-duplicate config, and test_data_configs_compose
rejects any data config that differs from its parent only in that mode),
mecka_fold_flagship_opsplit_hpt_6d, mecka_fold_flagship_topop_hpt_6d and
mecka_fold_freeform_opsplit_hpt_keypoints, plus the two 300M HPT models and
pi0.5_bc_mecka_6d.
Defaults, each for a failure seen in a run:
- checkpoints every 50 epochs, saved on train-epoch end. With
check_val_every_n_epoch > 1 the validation-end hook only fires on val epochs,
so every_n_epochs silently skipped the others.
- ddp_pi inference_mode: false. The compiled pi sampler does in-place setup on
its static buffers during cudagraph capture, which is illegal on inference
tensors ("Inplace update to inference tensor outside InferenceMode"); no_grad
is equivalent for eval and creates none.
- ddp_pi validates every 20 epochs instead of 200, and eval_hpt renders viz
every 200.
- pi0.5 gets a constant LR after warmup. The cosine schedule decayed to 0 over
num_training_steps=60000, so runs past ~60k steps trained at ~0 lr and
shorter runs never saw the configured lr for most of training.
- The Qwen stem resolves a repo id to its cached snapshot directory when
offline: transformers >= 4.57 probes the Hub while loading a tokenizer by
repo id even under HF_HUB_OFFLINE, which raises on compute nodes with no
internet. Loading from the directory skips the probe. Offline is read the
way huggingface_hub reads it -- 1 / ON / YES / TRUE in any case, from the
live environment -- rather than by matching the literal "1", which left
HF_HUB_OFFLINE=true hitting the probe the helper exists to avoid.
viz_mecka_episode.py and viz_eva_episode.py render an episode straight from
its zarr, which is how the left-wrist convention bug was found. Both take the
dataset dir from egomimic.utils.paths.default_dataset_dir(), which reads
paths/default.yaml, instead of each re-stating the cluster path.
HumanBimanualCartesianEuler / RobotBimanualCartesianEuler are renamed to
HumanBimanualCartesian6D / RobotBimanualCartesian6D: both pack xyz + rot6d and
have not touched an Euler angle since the 6D conversion, and this is the commit
that makes the human one the converter for a new default recipe.
Also prunes unit tests whose assertions another test already makes on the same
code path, and parametrized cases that repeat a stricter test's compose call.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…evert its outputs NOT TESTED ON HARDWARE. The rollout hard-coded cartesian_wristframe_ypr inputs and used the policy's predictions as-is. Now that eva checkpoints can be trained in several modes (cartesian, cartesian_6d, cartesian_wristframe_6d), that silently feeds a model the wrong inputs and sends the wrong frame to the arms. It now reads data.train_datasets.<name>.resolver.transform_list.mode off the checkpoint's own config, builds the matching input pipeline, and maps predictions back to camera-frame xyz+ypr+gripper through Eva.get_revert_transform_list. The revert needs the un-normalized proprio, which process_batch_for_training normalizes in place, so the raw ee_pose is cloned before that call. A checkpoint whose config records no mode keeps exactly the old behaviour -- legacy inputs, no revert -- and says so loudly, so this cannot silently change what an existing checkpoint does. test_eva_revert_for_rollout_recovers_camframe_ypr covers the revert math by cross-checking the wrist-frame revert against the cam-frame cartesian_6d one, which never enters the wrist frame -- so the frame composition is genuinely validated, though not against independently written SE(3) math. The hardware path itself still needs a real rollout. Both get_transform_list calls here pass allow_legacy_rotation=True: this is inference, and a checkpoint that predates the 6D conversion records exactly the modes the training-time guard rejects. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…eckpoint
The rollout looked for data.train_datasets.<eva>.resolver.transform_list.mode
in the checkpoint's config tree, but _build_model_config_tree saves only
model, so every checkpoint fell back to the legacy 14-D ypr pipeline and a
6D policy would have produced garbage commands. trainHydra now records
{dataset: transform_list.mode} as data_transform_modes in that tree, and the
rollout reads it from there. A checkpoint without it (every one trained
before this commit) is taken as legacy only when its saved actions_cartesian
width is 14; otherwise the rollout refuses to guess and asks for the new
--transform-mode flag.
EMBODIMENT_MAP still held the pre-collapse ids (8/7/6 for both/left/right);
it now reads them from EMBODIMENT.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GqAcWA2Lk9zubUhKX5DW1v
The unshuffled val loaders were truncated by limit_val_batches (80 x 64 x W frames) to their leading episode-hash-sorted slice, so at W = 1 the seen-op head scored 25 of 70 episodes and 6 of 11 operators, and the unseen head 33 of 191 episodes with an 81/19 operator mix on a 52/48 split (lane-0 diagnostic, 2026-09-15, measured single-rank). The loaders are unshuffled only so the overlay video is coherent, and the video renders just the first viz_max_batches (10) batches anyway. Split each val head in two: * metric loader = EvenStrideDataset(split, frames_per_episode=K) with K from data.metric_frames_per_episode[head], so every episode -- and so every operator -- is scored, at the same 80 batches and with no rendering; * video loader = <head>_video, the head's split restricted to the episode hashes pinned in data.video_episodes[head] (one per operator), contiguous and unsubsampled, logging no metrics. The pinned episodes are taken out of the already-resolved split, so resolve-once still holds. EvalVideo.on_validation_step gains a mode argument (metrics / video / both); ModelWrapper.validation_step routes <head>_video to the same evaluator in video mode and skips it entirely on a non-viz epoch, before the forward pass. Heads with no pins are called exactly as before, so a data config that declares neither key behaves as today. K = floor(5120 / episodes-in-split), from the measured splits: opsplit valid 73 (70 eps), train_viz 3 (1346 eps), unseen_op_valid 26 (191 eps); topop valid 256 (20 eps), train_viz 13 (385 eps). Those are SINGLE-RANK numbers and trainHydra scales them by trainer.devices x num_nodes, because DistributedSampler strides rather than chunks: the W ranks between them walk the whole split, so the pre-split window was already 80 x 64 x W frames wide. Without the scaling the shipped 4-GPU ddp_pi run would score 5,110 frames of the seen-val head where it previously scored all 13,744 -- a regression on the one head small enough to score whole. EvenStrideDataset keeps an episode entire when K exceeds its length, so the scaling cannot overrun a short split. Expect a shift in the per-batch Frechet _min/_max curves: they now span many episodes instead of ~2 s of one, so they are not comparable with older runs. Eval.on_validation_step, the abstract base, takes the mode kwarg too: the metrics/video path passes it unconditionally, so an evaluator that predates the split would have raised TypeError at the first val step of any config that declares video_episodes. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
RyanPCo
force-pushed
the
ryanco/eval-loaders
branch
from
October 19, 2026 19:38
0093b4e to
e63242c
Compare
RyanPCo
force-pushed
the
ryanco/flagship-top3-config
branch
from
October 19, 2026 19:39
c77a643 to
f77291a
Compare
…split Three-operator twin of mecka_fold_flagship_opsplit_hpt_6d (and the wider sibling of the single-operator topop config): train on the 3 largest flagship fold-clothes operators (1,079 eps / 206,276 frames, 65.3 % of flagship), none of them held out. Seen-op val is the seeded 5 % episode split within those operators (53 eps / 9,826 frames); the unseen-operator head is inherited unchanged, so the runs stay comparable. Lane A keys measured from the real split on a compute node 2026-09-15: metric_frames_per_episode valid=floor(5120/53)=96, train_viz=floor(5120/1026)=4 (single-rank values; trainHydra scales them by the world size) (unseen 26 inherited); video_episodes pins one seen-val episode per train operator (191+223+226 = 640 f = exactly the 10-batch video budget) and inherits the train_viz and unseen pins. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
RyanPCo
force-pushed
the
ryanco/flagship-top3-config
branch
from
October 19, 2026 20:22
f77291a to
ce4c383
Compare
This was referenced Oct 19, 2026
RyanPCo
force-pushed
the
ryanco/eval-loaders
branch
from
October 30, 2026 18:27
e63242c to
939f249
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three-operator twin of mecka_fold_flagship_opsplit_hpt_6d (and the wider
sibling of the single-operator topop config): train on the 3 largest
flagship fold-clothes operators (1,079 eps / 206,276 frames, 65.3 % of
flagship), none of them held out. Seen-op val is the seeded 5 % episode
split within those operators (53 eps / 9,826 frames); the unseen-operator
head is inherited unchanged, so the runs stay comparable.
Lane A keys measured from the real split on a compute node 2026-09-15:
metric_frames_per_episode valid=floor(5120/53)=96, train_viz=floor(5120/1026)=4
(unseen 26 inherited); video_episodes pins one seen-val episode per train
operator (191+223+226 = 640 f = exactly the 10-batch video budget) and
inherits the train_viz and unseen pins.
Co-Authored-By: Claude Opus 5 (1M context) [email protected]