[RyanPCo]

Writes last.ckpt as a relative symlink to the epoch checkpoint just saved (save_last: link), not a second full copy. With save_last: true, ModelCheckpoint calls _save_checkpoint twice per checkpoint event and writes the same tensors twice, ~21 GB each time on Pi 0.5.

Behaviour changes

• last.ckpt is now a symlink in callbacks/checkpoints.yaml. Anything that copies checkpoint dirs must preserve links (rsync -l), or last.ckpt is lost.
• A requeue with a dangling or missing last.ckpt now raises. A link can dangle where a file could not: the target deleted by hand, last.ckpt dropped by an rsync without -l, or a preemption between Lightning's unlink and relink. The requeue path used to warn and silently restart the run from scratch. It now raises FileNotFoundError when last.ckpt dangles, or is missing while the dir holds other .ckpt files, and names the ln -s fix. Only an empty checkpoint dir still falls back to the launch-time ckpt_path.
• The callback relinks on every train epoch once any checkpoint exists, not only on checkpoint epochs. That adds one os.remove + os.symlink + strategy.barrier per epoch.

Why it is safe

• save_top_k is -1, so the target is never collected.
• _should_remove_checkpoint returns False when previous == current, which is always the case for last.ckpt.
• Resume is unaffected: _requeue_resume_path gates on os.path.isfile, which follows symlinks.

Cost removed

On the now-retired ABC-DiT-L the second copy measured 9 GB and ~1.3 min, inside a checkpoint event that blocked the run for 2.7–3.4 min of a 42.3-min 1000-step cycle. Pi 0.5 checkpoints are ~21 GB.

Tests

New in tests/unit/test_checkpoint_resume.py:

• test_save_last_link_writes_a_symlink_not_a_second_copy drives a real two-epoch fit. link leaves a relative symlink to the newest epoch checkpoint; true leaves a regular file the same size as it, which is the copy this removes.
• test_checkpoint_callback_links_last_instead_of_copying checks the composed config.
• test_requeue_resumes_through_a_symlinked_last_ckpt, test_requeue_with_a_dangling_last_ckpt_raises, test_requeue_with_checkpoints_but_no_last_ckpt_raises.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw

[RyanPCo]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• pi: initialize from PaliGemma with a scratch action expert #677 : 2 dependent PRs (#678 , #680 )
• perf: vectorized image augs and a torch.compile knob for HPT and Pi; Pi clips gradients like openpi #676
• loader: one zarr read per key, not one per keymap entry #674
• eval: build the pinned-episode video batches only where they are used #673
• checkpoints: last.ckpt as a symlink, not a second full copy #672 👈 (View in Graphite)
• trainHydra: metric_frames_per_episode: auto, and warn on empty prompts #671
• transforms: per-step wrist framing for proprio history #670
• viz: one keypoint renderer, the horizon trace #669
• configs: history and backbone-LR knobs on the keypoint wrist HPT model #668
• trainer: ddp_bf16_compress, DDP with a bf16 gradient all-reduce #667
• eval: overlay videos play at the source rate under DDP #666
• hpt: build the policy on CPU and let Lightning place it #664 : 1 other dependent PR (#665 )
• hpt: never re-initialise pretrained stems; hash-check them; mask text padding #645 : 1 other dependent PR (#646 )
• obs: proprio history (K proprio steps, one token each, learned time embedding) #650
• loss: mask the repeat-last action padding out of the HPT loss #649
• resnet: keep a frozen backbone in eval mode; give backbones their own LR group #652
• augs: per-sample image augmentation for HPT train-mode augs #651
• eval: deterministic validation (seeded prompts + flow sampling); log-only grad-norm MAD #644
• data: mecka_fold_flagship_top3_hpt_6d, the 3-operator flagship train split #643
• val: subsampled metric loaders + pinned contiguous video loaders #642
• infra: mecka 6D / keypoint recipes, 300M models, defaults, and the rollout revert #640
• eval: layout-aware frame-invariant metrics, and a train-split viz head #659
• data: stop letting one bad episode or a rotation channel take a run down #658
• norm: guard degenerate channels, horizon pooling, and no rot6d normalization #657
• keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656
• mecka: fix the double-mirrored LEFT wrist frame, and correct existing zarrs #655
• 6d: PI goes 6D-only; Eva and Human gain the continuous-rotation modes #654
• 6d: continuous rotation representation, bimanual layouts, shared norm helpers #653
• train: give each Slurm slurm job gets its own cache dir to avoid conflict #638 : 1 other dependent PR (#639 )
• write git version into the run dir for reproducibility #637
• train: don't load PI base weights on resume and verify that checkpoint path exists #636
• configs: declare proprio and actions dims once per HPT model config #634 : 1 other dependent PR (#635 )
• cache norm stats for data configs to minimize recomputation #633
• import cloud packages lazily to support local training #631 : 1 other dependent PR (#632 )
• cache dataset resolver for faster training initialization #630
• validate episode hashes in data config before training #628
• modify pi camera map to match data #629
• add tests to ensure training works between every model and data vendor #626
• run CI tests on every PR #625
• add env variables for user-specific paths #624
• build: set pyproject and uv as the dependency managers #623
• fix(configs): fix paths for data hydra yaml configs #622
• refactor(algo): delete ACT #627 : 1 other dependent PR (#621 )
• tests: create dedicated test folder with unit and integration tests #620
• add pyright type checker #619
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[RyanPCo]

Automated adversarial review (read-only; findings were checked by a second skeptical pass before posting): 1 nit.

[RyanPCo]

nit (simplification): existing = glob.glob(os.path.join(ckpt_dir, "*.ckpt")) replaces the listdir/isdir/endswith comprehension and already returns [] for a missing directory.

[RyanPCo]

Review findings: status after the fixup pass. The fixes are squashed into this PR's commits, and the whole stack was force-pushed.

Fixed

• Hand-rolled .ckpt scan could use glob: Now glob.glob(os.path.join(glob.escape(ckpt_dir), '*.ckpt')). The escape stops brackets in run directory names from being read as a pattern.