[RyanPCo]

• pl_utils/preemption.py: PreemptionCheckpoint callback. On the signal submitit
  sends (USR2 / $SUBMITIT_PREEMPT_SIGNAL) it saves /checkpoints/last.ckpt
  (what trainHydra resumes from on SLURM_RESTART_COUNT) and then chains to the
  previously installed handler (submitit's requeue). Restores the old handler
  on teardown. Details that matter:
  • before saving it points every ModelCheckpoint(save_last=True) in the same
    directory (compared via realpath; Lightning realpaths dirpath) at
    last.ckpt, otherwise their version counter writes every later periodic
    save to last-v1.ckpt, ... (also after resume, since the path is part of
    ModelCheckpoint's saved state) while trainHydra resumes the stale file;
  • writes last.ckpt.preempt and renames, so a GraceTime SIGKILL mid-write
    cannot leave a truncated last.ckpt;
  • a failed save is logged and the requeue still runs;
  • forked DataLoader workers inherit the handler (Slurm signals every
    process in the step); they return immediately instead of saving.
    There is deliberately no self-requeue for non-submitit launches: nothing in
    the repo launches training that way, and a raw sbatch requeue would
    recompose hydra.run.dir with a new timestamp and never find last.ckpt.
• trainHydra: SLURMEnvironment(auto_requeue=False) + PreemptionCheckpoint.
  Lightning's own handler listened on SIGUSR1, which nothing ever sent, and
  Lightning skips registering on a signal submitit already owns. On requeue,
  resume only if last.ckpt exists (preemption during dataset setup happens
  before the callback is installed); warn and start fresh otherwise.
• hydra/launcher/submitit*.yaml: signal_delay_s: 600 (launcher default 120 s is
  too short to write a multi-GB pi0.5 checkpoint to NFS before the time-limit
  SIGKILL). Preemption itself is bounded by the partition GraceTime (300 s on
  overcap), which no launcher setting changes.
• pl_model: torch.distributed.barrier() in on_fit_start / on_validation_end is
  now a no-op without a process group (CPU smoke test, devices=1 non-DDP).
• tests/unit: test_preemption (8: chain, default path, env signal, save
  failure, no temp file, ModelCheckpoint coexistence across a resume, the same
  through a symlinked run dir, forked worker does nothing),
  test_pl_model_barriers (2).

Co-Authored-By: Claude Fable 5.1 [email protected]
Claude-Session: https://claude.ai/code/session_01UHDiN8yfERcsiFdz4mXuZi

🤖 Generated with Claude Code

https://claude.ai/code/session_01UHDiN8yfERcsiFdz4mXuZi

[RyanPCo]

• fix(train): save checkpoint when Slurm kills the job, resume on restart #621 👈 (View in Graphite)
• refactor(algo): delete ACT #627 : 1 other dependent PR (#622 )
• tests: create dedicated test folder with unit and integration tests #620
• add pyright type checker #619
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[RyanPCo]

Not doing preemption checkpointing (decided 2026-09-13); see
docs/critique/00_README.md Phase 1.