[aidang3019]

eva: joint-space mode for the ABC YAM action space; robot image history

ABC's station and its simulator (amazon-far/abc abc_sim) act in 14-D
absolute joint positions + gripper, [left j1..j6, left grip, right j1..j6,
right grip]. ABC zarrs already carry {left,right}.{obs,cmd}_{joints,gripper},
so a joint-space policy can train on real and sim ABC episodes with one
schema and be stepped in the sim without IK.

• Eva transform/keymap mode "joints": observations.state.joint_positions
  (14) and actions_joints (100, 14), the 45 raw 30 Hz steps resampled to the
  model horizon like the cartesian modes; no revert.
• Embodiment.get_keymap(image_history_gap_s=...): the "_hist" front camera
  Human already had, so a robot domain can share an RDT trunk with
  trunk.image_history: 2.
• layout_metrics(..., ac_key): actions_joints* get joint metrics (abs error in
  degrees, gripper MSE). A 14-D chunk otherwise reads as cartesian ypr.

Co-Authored-By: Claude Opus 5.5 (1M context) [email protected]

train: init_weights_ckpt, a weights-only fine-tune init

ckpt_path is a full Lightning resume (optimizer, schedule, epoch) and fails
on keys a new embodiment adds, so a human-pretrained RDT could not be
fine-tuned on robot data from this stack. The loader that did this on
aidan/abc-stationery-pi never reached main.

init_weights_ckpt= copies every name+shape-matched tensor and leaves
the rest (a new embodiment's stems / head, a different action width) at
their fresh init. It raises when under init_weights_min_fraction (0.5) of
the model's elements matched, so an unrelated checkpoint cannot silently
"fine-tune" from random init, and it is skipped whenever ckpt_path is set,
so a Slurm requeue resumes the run's own weights.

Co-Authored-By: Claude Opus 5.5 (1M context) [email protected]

abc_sim: convert ABC sim_224 teleop episodes to EgoVerse zarrs

amazon-far/abc publishes 24 simulator tasks of teleop data (sim_224, 412 h)
as states_actions.bin + a stacked 3-camera mp4 per episode. The converter
writes each episode as an eva_bimanual zarr with the real ABC zarrs' keys
({left,right}.{obs,cmd}_{joints,gripper}, images.{front_1,left_wrist,
right_wrist}, annotations), so Eva's "joints" keymap reads real and sim ABC
alike.

• episodes are selected by task_name (prepare.py extracts every task into
  one {train,val}_sim folder) and keep ABC's own split as metadata split,
  plus lab=abc_sim and the source episode id;
• prompts follow ABC's rule (prompt_timeline > instruction > task name) with
  the "sim " prefix ABC trains and evaluates sim episodes with;
• intrinsics are the sim's fovy-58 pinhole at the stored resolution;
• idempotent (done marker + tmp dir + rename), parallel, one bad episode is
  reported in conversion_report.json instead of stopping the task.

Co-Authored-By: Claude Opus 5.5 (1M context) [email protected]

abc: RDT-1B joint-space recipes, the fold ladder, a Lambda pack launcher

The model and data side of the ABC sim pipeline (egomimic/scripts/abc_sim/
README.md):

• model/rdt_1b_abc_joints_dinov3_vitb: the 1B DiT with an eva_bimanual head
  and state stem in the ABC station's 14-D joint space beside the mecka
  human head (shared trunk, tower and text stem).
• data/abc_sim_joints (converted sim_224 episodes, ABC's own train/val
  split), abc_real_joints (lab=abc on the mirror; the 2026-09-01 relabel
  makes lab+task+path the stable filter; front frames resized to one size
  because ABC mixes 640x480 and 1280x720), abc_sim_joints_cotrain_mecka
  (single-task human+robot cotrain, one batch of each per step).
• train_zarr_abc_sim_rdt_ft / _real_rdt_ft / _sim_rdt_cotrain recipes and
  evaluator/eval_hpt_abc_joints: joint metrics for the robot, the wrist-frame
  6D overlay for the human; an embodiment without a viz entry gets metrics
  only.
• train_zarr_fold_ladder_rdt1b: the RDT recipe at 1B on one rung of the
  mecka freeform-fold operator ladder (k = 1/2/4), 1000 x 100 steps, a
  checkpoint every 100 epochs -- the bases the ABC fine-tunes start from.
• Eva keymap "joints_front" (front camera only) and get_transform_list
  image_hw (ResizeImages) for mixed-resolution sources.
• EGOVERSE_EPISODE_TABLE: the S3 resolver reads a pickled episode-table
  snapshot where there is no SQL (Lambda).
• scripts/abc_sim/lambda_pack.sbatch: several small-DDP runs on one
  exclusive loaner node, datasets staged to node-local /ephemeral first.

Co-Authored-By: Claude Fable 5.1 [email protected]

abc_sim: policy server, sim client, Lambda rollout launcher

Closed-loop evaluation of a joint-space checkpoint in ABC's simulator.
The sim's Python (py3.12, MuJoCo 3.8, torch 2.11) cannot share the training
venv, so the policy is served over HTTP + JSON (stdlib both ends):

• policy_server.py builds the model input exactly as ZarrDataset builds a
  training sample for the checkpoint's data config (same keymap, transform
  list, the _hist frame image_history_gap_s earlier, fps) and normalizes it
  with the checkpoint's norm stats before process_batch_for_training -- the
  dataset normalizes in getitem, which the robot rollout skipped.
• sim_client.py runs in ABC's venv on abc_minimal's SimTaskEnv with ABC's
  evaluators and eval defaults (seeded worlds, 15 executed steps per
  inference, 236 chunks, early stop on success), no RTC; summary.json has
  ABC's fields. --policy hold is the no-motion floor.
• rollout_lambda.sbatch runs both in one container step on one H100.

tests/unit/test_policy_server.py: the server's sample equals the dataset's
for the same frames, normalization happens before the model, and the HTTP
round trip.

Co-Authored-By: Claude Fable 5.1 [email protected]

abc_sim: val-metric vs sim-success report

val_vs_sim.py: from a manifest of checkpoints (W&B run, epoch, ladder
hours, rollout summary.json), Spearman / Pearson between the validation
metric and success_rate / mean_max_progress with bootstrap CIs, and the
log-linear fit of the metric (and success) against data hours across the
ladder's final checkpoints. report.json + report.md.

Co-Authored-By: Claude Fable 5.1 [email protected]

policy_server: the history lag in frames, as the dataset reads it

ZarrDataset reads the _hist frame round(gap_s * fps) frames back; the
server compared timestamps in seconds and, at 1/30 s resolution, could pick
one frame too far back. Same frame arithmetic now; the equality test covers
t = 0..11.

Co-Authored-By: Claude Fable 5.1 [email protected]

abc_sim: audit fixes (chunk timing, history frame, server load) + ponytail pass

Fixes found by an audit of this branch against ABC's code:

• policy_server could not start: it called .to()/.eval() on the algo, a plain
  object; the Lightning wrapper owns and moves the nets.
• The served chunk played 2.2x too slow: the model's 100 steps span 45 raw
  30 Hz frames (JOINT_RAW_HORIZON resampled to 100), and the client executed
  one per 30 Hz step. The server now resamples back onto the 45-frame grid,
  so the 15 executed steps are 0.5 s of the plan, as in ABC.
• The _hist frame went 0.5-1.5 s stale: the server only ever saw re-plan
  frames. The client now renders the frame lag_frames (3) steps before each
  re-plan and sends it; the server is stateless.
• Checkpoints stored only the model config, so a rollout could not rebuild
  the training inputs (robot/rollout.py's transform-mode lookup always fell
  back to the legacy ypr pipeline). The config tree now also carries each
  dataset's resolved keymap + transform list.
• sim_client now drives ABC's own rollout_worlds + build_summary (the
  unplaceable-seed retry, chunk metrics, resolved physics, the summary
  fields) through an HTTP policy with ABC's infer() surface, instead of a
  hand copy of that loop.
• Prompts: the converter keeps ABC's rule verbatim (no "sim " forced onto
  instruction/timeline prompts); the client conditions on the training prompt
  (the task name) by default, --prompt env for the randomized tasks. The two
  differed on 4 of the 7 candidate tasks. Episodes ABC's loader drops as
  unlabelled are skipped; a states/metadata length mismatch raises.

Ponytail pass (less code, same behaviour): tasks.py deleted (every sim_224
task_name is already an abc_sim alias); val_vs_sim on scipy.stats; the
cotrain data config inherits mecka_all_6d instead of copying it; the
init_weights report dict and its min-fraction config knob removed (the guard
stays); the unreachable stereo-camera alias and --limit removed; config
headers cut to their role line (one pointed at a file that does not exist);
config-echo tests replaced by an end-to-end test: train a tiny RDT, save the
checkpoint, serve it with policy_server, infer.

Co-Authored-By: Claude Opus 5.5 [email protected]

lambda_pack: stage data from a node step; run onto a running job's spare GPUs

The /ephemeral staging ran in the batch shell, so the documented "bash this
file with SLURM_JOB_ID=" use copied onto the gateway, which has
no /ephemeral. It is now a step. Run steps --overlap so they can share a
running allocation's (mostly idle) CPUs; harmless in an exclusive one.

Co-Authored-By: Claude Opus 5.5 [email protected]

lambda_pack: stage to node-local /tmp (/ephemeral is root-only on loaner nodes)

Co-Authored-By: Claude Opus 5.5 [email protected]

eval: the val subsample K is per-rank; the val loaders are not sharded

_metric_frames_per_episode scaled K by devices * num_nodes on the premise
that a DistributedSampler strides the val split across ranks. The val heads
come back inside CombinedLoaders, which Lightning does not wrap (every rank
runs every val batch; measured 2026-09-16, EvalVideo._video_fps), so on W
GPUs each rank scored the first 1/W of a W-times-denser subsample, in
episode-hash order: on the freeform unseen-operator head at 2 GPUs, half of
the 35 episodes. K is now the single-rank value on every rank.

Co-Authored-By: Claude Opus 5.5 [email protected]

fold ladder: the true replay stride (29/99 frames per chunk step)

The replay's stride inference only fits keypoint layouts and falls back to
1 on the 12-D cartesian revert: a 3.4x-slow GT/pred replay that drifts off
the hand. The mecka cartesian chunk is 30 raw frames resampled to 100.

Co-Authored-By: Claude Opus 5.5 [email protected]

[aidang3019]

ABC fine-tune + abc_sim rollout pipeline (stacked on #691), for comparing pretraining runs by closed-loop sim success where offline metrics look alike. Operator doc: egomimic/scripts/abc_sim/README.md.

• Joint space. Eva mode joints = the ABC station's 14-D absolute joints + gripper, which is also abc_sim's action space, so no IK. Real ABC zarrs already carry obs_joints/cmd_joints.
• Fine-tuning. init_weights_ckpt (weights-only init; fails if under half the model matched). RDT-1B recipes: train_zarr_abc_sim_rdt_ft, _real_rdt_ft, _sim_rdt_cotrain (single-task human + robot).
• Data. convert_sim_to_zarr.py converts ABC's sim_224 teleop episodes to zarrs with the real-ABC keys. EGOVERSE_EPISODE_TABLE lets the resolver run from a table snapshot where there is no SQL.
• Rollout. policy_server.py (training venv) builds inputs exactly as the dataset does, including normalization, and returns 45 commands on the 30 Hz grid. sim_client.py (ABC venv) drives ABC's own rollout_worlds + build_summary.
• Analysis. val_vs_sim.py gives the val-metric vs sim-success correlation, with bootstrap CIs and the log-linear fit.
• Bases. train_zarr_fold_ladder_rdt1b is RDT-1B on the freeform-fold operator ladder k=1/2/4. It is running now on the Lambda loaner via lambda_pack.sbatch.

Fixes to code below this PR (each its own commit):

• Val subsample K. It was scaled by world size, but the val loaders are not sharded, so on W GPUs each rank scored only the first 1/W of the subsample.
• Checkpoint config. Checkpoints stored only the model config, so robot/rollout.py's transform-mode lookup could never succeed and always fell back to the legacy ypr pipeline. Checkpoints now also carry each dataset's keymap + transform list.
• Fold-ladder replay stride. The fold-ladder recipe now sets the true replay stride (29/99).

Tests. New unit tests cover the joint pipeline, the converter, the server (sample equals dataset sample; HTTP round trip), and an end-to-end CPU check: train a tiny RDT, save it, serve it, infer. Full tests/unit: 1694 passed, 4 skipped.

Not yet run. The sim rollouts need ABC's venv on Lambda, which is pending.

🤖 Generated with Claude Code

[aidang3019]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• ponytail(u1): drop bench/ and duplicate viz scripts, reuse gpu_orphans in the reaper #703
• eva: joint-space mode for the ABC YAM action space; robot image history #702 👈 (View in Graphite)
• eval: replay maps chunk steps to real frames for resampled chunks #691
• pi: keep per-sample intrinsics so overlays use the episode camera #690 : 2 other dependent PRs (#707 , #716 )
• rdt: long-range visual memory; image memory window and history stride in seconds #689
• norm stats: cache sampled rows per episode so splits share them #687 : 1 other dependent PR (#688 )
• data: exclude episodes by hash; scan Mecka egomotion for failed head tracking #686
• eval: log validation videos to wandb #684 : 1 other dependent PR (#685 )
• slurm: stop cancelled jobs from leaking whole GPUs #682 : 1 other dependent PR (#683 )
• algo: RDT (Robotics Diffusion Transformer) on the HPT chassis #680 : 1 other dependent PR (#681 )
• pi: initialize from PaliGemma with a scratch action expert #677 : 1 other dependent PR (#678 )
• perf: vectorized image augs and a torch.compile knob for HPT and Pi; Pi clips gradients like openpi #676
• loader: one zarr read per key, not one per keymap entry #674
• eval: build the pinned-episode video batches only where they are used #673
• checkpoints: last.ckpt as a symlink, not a second full copy #672
• trainHydra: metric_frames_per_episode: auto, and warn on empty prompts #671
• transforms: per-step wrist framing for proprio history #670
• viz: one keypoint renderer, the horizon trace #669
• configs: history and backbone-LR knobs on the keypoint wrist HPT model #668
• trainer: ddp_bf16_compress, DDP with a bf16 gradient all-reduce #667
• eval: overlay videos play at the source rate under DDP #666
• hpt: build the policy on CPU and let Lightning place it #664 : 1 other dependent PR (#665 )
• hpt: never re-initialise pretrained stems; hash-check them; mask text padding #645 : 1 other dependent PR (#646 )
• obs: proprio history (K proprio steps, one token each, learned time embedding) #650
• loss: mask the repeat-last action padding out of the HPT loss #649
• resnet: keep a frozen backbone in eval mode; give backbones their own LR group #652
• augs: per-sample image augmentation for HPT train-mode augs #651
• eval: deterministic validation (seeded prompts + flow sampling); log-only grad-norm MAD #644
• data: mecka_fold_flagship_top3_hpt_6d, the 3-operator flagship train split #643
• val: subsampled metric loaders + pinned contiguous video loaders #642
• infra: mecka 6D / keypoint recipes, 300M models, defaults, and the rollout revert #640
• eval: layout-aware frame-invariant metrics, and a train-split viz head #659
• data: stop letting one bad episode or a rotation channel take a run down #658
• norm: guard degenerate channels, horizon pooling, and no rot6d normalization #657
• keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656
• mecka: fix the double-mirrored LEFT wrist frame, and correct existing zarrs #655
• 6d: PI goes 6D-only; Eva and Human gain the continuous-rotation modes #654
• 6d: continuous rotation representation, bimanual layouts, shared norm helpers #653
• train: give each Slurm slurm job gets its own cache dir to avoid conflict #638 : 1 other dependent PR (#639 )
• write git version into the run dir for reproducibility #637
• train: don't load PI base weights on resume and verify that checkpoint path exists #636
• configs: declare proprio and actions dims once per HPT model config #634 : 1 other dependent PR (#635 )
• cache norm stats for data configs to minimize recomputation #633
• import cloud packages lazily to support local training #631 : 1 other dependent PR (#632 )
• cache dataset resolver for faster training initialization #630
• validate episode hashes in data config before training #628
• modify pi camera map to match data #629
• add tests to ensure training works between every model and data vendor #626
• run CI tests on every PR #625
• add env variables for user-specific paths #624
• build: set pyproject and uv as the dependency managers #623
• fix(configs): fix paths for data hydra yaml configs #622
• refactor(algo): delete ACT #627 : 1 other dependent PR (#621 )
• tests: create dedicated test folder with unit and integration tests #620
• add pyright type checker #619
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.