[RyanPCo]

Every run writes split.json (train/valid dataset keys per train dataset) and,
with Lance datasets, lance_versions.json (table versions and their tags) next
to git_sha.txt; a requeue or resume that differs keeps the first file and
writes a timestamped one. Resolving an untagged Lance version warns that the
run is reproducible only until cleanup_old_versions drops it. Resuming from a
checkpoint whose run validated on episodes this run trains on warns.

https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw

[RyanPCo]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• aria_to_lance: half-open task and alias end #727
• trainHydra: record the split and Lance versions a run read #726 👈 (View in Graphite)
• rldb, algo, eval: leave an untracked hand out of losses, stats and bounds #725
• rldb/lance: require_hands_in_frame and per-hand sample masks #724
• rldb/lance: action_clamp and history_floor at segment bounds #723
• rldb/lance: one leaf per recording, sampled over its task segments #722
• rldb/mcap: vendor MCAP spec 0.1 and validator #718
• rldb: move MultiDataset and format-neutral pieces out of zarr/ #717
• lance migration #716
• pi: keep per-sample intrinsics so overlays use the episode camera #690 : 2 other dependent PRs (#691 , #707 )
• rdt: long-range visual memory; image memory window and history stride in seconds #689
• norm stats: cache sampled rows per episode so splits share them #687 : 1 other dependent PR (#688 )
• data: exclude episodes by hash; scan Mecka egomotion for failed head tracking #686
• eval: log validation videos to wandb #684 : 1 other dependent PR (#685 )
• slurm: stop cancelled jobs from leaking whole GPUs #682 : 1 other dependent PR (#683 )
• algo: RDT (Robotics Diffusion Transformer) on the HPT chassis #680 : 1 other dependent PR (#681 )
• pi: initialize from PaliGemma with a scratch action expert #677 : 1 other dependent PR (#678 )
• perf: vectorized image augs and a torch.compile knob for HPT and Pi; Pi clips gradients like openpi #676
• loader: one zarr read per key, not one per keymap entry #674
• eval: build the pinned-episode video batches only where they are used #673
• checkpoints: last.ckpt as a symlink, not a second full copy #672
• trainHydra: metric_frames_per_episode: auto, and warn on empty prompts #671
• transforms: per-step wrist framing for proprio history #670
• viz: one keypoint renderer, the horizon trace #669
• configs: history and backbone-LR knobs on the keypoint wrist HPT model #668
• trainer: ddp_bf16_compress, DDP with a bf16 gradient all-reduce #667
• eval: overlay videos play at the source rate under DDP #666
• hpt: build the policy on CPU and let Lightning place it #664 : 1 other dependent PR (#665 )
• hpt: never re-initialise pretrained stems; hash-check them; mask text padding #645 : 1 other dependent PR (#646 )
• obs: proprio history (K proprio steps, one token each, learned time embedding) #650
• loss: mask the repeat-last action padding out of the HPT loss #649
• resnet: keep a frozen backbone in eval mode; give backbones their own LR group #652
• augs: per-sample image augmentation for HPT train-mode augs #651
• eval: deterministic validation (seeded prompts + flow sampling); log-only grad-norm MAD #644
• data: mecka_fold_flagship_top3_hpt_6d, the 3-operator flagship train split #643
• val: subsampled metric loaders + pinned contiguous video loaders #642
• infra: mecka 6D / keypoint recipes, 300M models, defaults, and the rollout revert #640
• eval: layout-aware frame-invariant metrics, and a train-split viz head #659
• data: stop letting one bad episode or a rotation channel take a run down #658
• norm: guard degenerate channels, horizon pooling, and no rot6d normalization #657
• keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656
• mecka: fix the double-mirrored LEFT wrist frame, and correct existing zarrs #655
• 6d: PI goes 6D-only; Eva and Human gain the continuous-rotation modes #654
• 6d: continuous rotation representation, bimanual layouts, shared norm helpers #653
• train: give each Slurm slurm job gets its own cache dir to avoid conflict #638 : 1 other dependent PR (#639 )
• write git version into the run dir for reproducibility #637
• train: don't load PI base weights on resume and verify that checkpoint path exists #636
• configs: declare proprio and actions dims once per HPT model config #634 : 1 other dependent PR (#635 )
• cache norm stats for data configs to minimize recomputation #633
• import cloud packages lazily to support local training #631 : 1 other dependent PR (#632 )
• cache dataset resolver for faster training initialization #630
• validate episode hashes in data config before training #628
• modify pi camera map to match data #629
• add tests to ensure training works between every model and data vendor #626
• run CI tests on every PR #625
• add env variables for user-specific paths #624
• build: set pyproject and uv as the dependency managers #623
• fix(configs): fix paths for data hydra yaml configs #622
• refactor(algo): delete ACT #627 : 1 other dependent PR (#621 )
• tests: create dedicated test folder with unit and integration tests #620
• add pyright type checker #619
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[RyanPCo]

Automated adversarial review (read-only; findings were checked by a second skeptical pass before posting): 1 low.

[RyanPCo]

low (efficiency): manifest() runs here only to find untagged tables, and MultiDataset._from_resolver calls it again right after. Each call lists tags on every table read, which means S3 metadata reads. Since self.versions is fixed by the end of resolve, compute it once (e.g. cache it on self) and reuse it in both places.

[RyanPCo]

Review findings: status after the fixup pass. The fixes are squashed into this PR's commits, and the whole stack was force-pushed.

Fixed

• manifest() runs in resolve and again in _from_resolver: manifest() is cached until self.versions changes and returns a deep copy.