write git version into the run dir for reproducibility by RyanPCo · Pull Request #637 · GaTech-RL2/EgoVerse

RyanPCo · 2026-09-13T20:25:23Z

Rank-0, once per process: on fit start, or on validation start for
validate-only (eval) runs; enabled in callbacks/defaults.yaml so every shipped
top-level config gets it. Writes git_sha.txt, git.diff (git diff HEAD --submodule=diff, raw bytes so non-UTF-8 edits neither crash training nor stop
the patch applying; includes file diffs inside external/openpi) and
git_untracked.txt (names only). A failed diff or listing leaves
.error with the reason instead of an empty file, so it never reads as a
clean tree. A process that finds git_sha.txt already in the run dir (Slurm
requeue, resume) writes git_sha..txt etc. beside it, keeping the
first launch's record. No-op with one warning when git is missing or egomimic
is not a checkout; write errors are logged, never raised. wandb records the sha
online; this covers offline runs and keeps provenance next to the checkpoints.

The git helper lives in egomimic/utils/git_info.py (every call bounded by a
30 s timeout, up from zarr_writer's old 5 s); zarr_writer's and norm_cache's
provenance stamps now use it, so norm_cache no longer imports zarr_writer.

Co-Authored-By: Claude Fable 5.1 [email protected]
Claude-Session: https://claude.ai/code/session_01MWtR1x5tjREwoxQZAnUGrh
Co-Authored-By: Claude Opus 5 (1M context) [email protected]
Claude-Session: https://claude.ai/code/session_01BbQNNaMAbTNMypegmJkv4s

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWtR1x5tjREwoxQZAnUGrh
https://claude.ai/code/session_01BbQNNaMAbTNMypegmJkv4s

RyanPCo · 2026-09-13T20:25:44Z

train: give each Slurm slurm job gets its own cache dir to avoid conflict #638 : 2 dependent PRs (#639 , #653 )
write git version into the run dir for reproducibility #637 👈 (View in Graphite)
train: don't load PI base weights on resume and verify that checkpoint path exists #636
configs: declare proprio and actions dims once per HPT model config #634 : 1 other dependent PR (#635 )
cache norm stats for data configs to minimize recomputation #633
import cloud packages lazily to support local training #631 : 1 other dependent PR (#632 )
cache dataset resolver for faster training initialization #630
validate episode hashes in data config before training #628
modify pi camera map to match data #629
add tests to ensure training works between every model and data vendor #626
run CI tests on every PR #625
add env variables for user-specific paths #624
build: set pyproject and uv as the dependency managers #623
fix(configs): fix paths for data hydra yaml configs #622
refactor(algo): delete ACT #627 : 1 other dependent PR (#621 )
tests: create dedicated test folder with unit and integration tests #620
add pyright type checker #619
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

RyanPCo · 2026-09-29T21:26:38Z

@greptileai

greptile-apps · 2026-09-29T21:31:47Z

Confidence Score: 4/5

[Medium risk] Adds git provenance logging to training runs.

The PR should not merge until untracked files used outside egomimic/utils are included in the provenance record.

Findings

Untracked files are missed ▶
CSV reuse skips provenance ▶
Resume records can collide ▶

Summary

The PR adds a Git-provenance callback to the default training callbacks, records tracked changes and untracked filenames beside run checkpoints, and shares the Git SHA helper with the norm cache.

The untracked-file listing misses files outside the helper directory.
One eval route bypasses the callback's hooks, and closely spaced resumes can collide on filenames.

_{Reviews (1) · Last reviewed commit: "callbacks: GitProvenance writes git_sha...."}

greptile-apps · 2026-09-29T21:31:52Z

+def git_untracked() -> bytes:
+    """Newline-separated paths of untracked, non-ignored files (names only;
+    untracked files inside submodules are not listed). Raises GitError."""
+    return _git("ls-files", "--others", "--exclude-standard")


Untracked files are missed

If a run uses a new, untracked file outside egomimic/utils, such as a training config, this command lists only files beneath the helper's working directory. The file is omitted from git_untracked.txt, leaving the run's provenance incomplete. Run the listing from the work-tree root.

greptile-apps · 2026-09-29T21:31:52Z

+    def on_validation_start(self, trainer, pl_module) -> None:
+        # Validate-only (eval) runs never reach on_fit_start.
+        self._write_once(trainer)


CSV reuse skips provenance

Latent evaluation can reuse existing CSVs, render outputs, and return without calling trainer.validate(). Because this callback records eval provenance only when validation starts, that eval path produces outputs without a Git record.

greptile-apps · 2026-09-29T21:31:52Z

+        if (root / "git_sha.txt").exists():
+            tag = "." + datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+        (root / f"git_sha{tag}.txt").write_text(sha + "\n", encoding="utf-8")


Resume records can collide

If two launches write to the same run directory within one second, they use the same timestamped filenames. The later launch can overwrite the earlier launch's SHA and diff, so the earlier provenance is lost. Use a collision-resistant tag or create the files exclusively.

RyanPCo · 2026-10-19T19:23:30Z

Merge activity

Oct 19, 7:23 PM UTC: A user started a stack merge that includes this pull request via Graphite.
Oct 19, 7:27 PM UTC: Graphite rebased this pull request as part of a merge.
Oct 19, 7:27 PM UTC: @RyanPCo merged this pull request with Graphite.

Rank-0, once per process: on fit start, or on validation start for validate-only (eval) runs; enabled in callbacks/defaults.yaml so every shipped top-level config gets it. Writes git_sha.txt, git.diff (`git diff HEAD --submodule=diff`, raw bytes so non-UTF-8 edits neither crash training nor stop the patch applying; includes file diffs inside external/openpi) and git_untracked.txt (names only). A failed diff or listing leaves <file>.error with the reason instead of an empty file, so it never reads as a clean tree. A process that finds git_sha.txt already in the run dir (Slurm requeue, resume) writes git_sha.<UTC time>.txt etc. beside it, keeping the first launch's record. No-op with one warning when git is missing or egomimic is not a checkout; write errors are logged, never raised. wandb records the sha online; this covers offline runs and keeps provenance next to the checkpoints. The git helper lives in egomimic/utils/git_info.py (every call bounded by a 30 s timeout, up from zarr_writer's old 5 s); zarr_writer's and norm_cache's provenance stamps now use it, so norm_cache no longer imports zarr_writer. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MWtR1x5tjREwoxQZAnUGrh Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01BbQNNaMAbTNMypegmJkv4s

RyanPCo force-pushed the ryanco/phase4-git-provenance branch from ad0260c to 6eda7a6 Compare September 13, 2026 21:59

RyanPCo force-pushed the ryanco/phase4-resume-without-base-weights branch from ed4bc94 to 3d8ffd1 Compare September 13, 2026 21:59

RyanPCo changed the title ~~callbacks: GitProvenance writes git_sha.txt + git.diff into the run dir~~ write git version into the run dir for reproducibility Sep 14, 2026

RyanPCo force-pushed the ryanco/phase4-resume-without-base-weights branch from 3d8ffd1 to d25f8ca Compare September 15, 2026 13:31

RyanPCo force-pushed the ryanco/phase4-git-provenance branch from 6eda7a6 to 7a242b3 Compare September 15, 2026 13:31

RyanPCo mentioned this pull request Sep 15, 2026

infra: mecka 6D / keypoint recipes, 300M models, defaults, and the rollout revert #640

Draft

RyanPCo force-pushed the ryanco/phase4-git-provenance branch from 7a242b3 to fd3fecb Compare September 15, 2026 15:04

RyanPCo force-pushed the ryanco/phase4-resume-without-base-weights branch 2 times, most recently from 4a8be78 to 677a62d Compare September 15, 2026 15:14

RyanPCo force-pushed the ryanco/phase4-git-provenance branch from 4b49d7c to 0ce439f Compare September 15, 2026 15:14

RyanPCo marked this pull request as ready for review September 15, 2026 15:16

This was referenced Sep 15, 2026

val: subsampled metric loaders + pinned contiguous video loaders #642

Draft

data: mecka_fold_flagship_top3_hpt_6d, the 3-operator flagship train split #643

Closed

greptile-apps Bot reviewed Sep 29, 2026

View reviewed changes

aidang3019 force-pushed the ryanco/phase4-resume-without-base-weights branch from b9e1287 to ffebd78 Compare September 30, 2026 01:06

aidang3019 force-pushed the ryanco/phase4-git-provenance branch from b195bb5 to 22327ba Compare September 30, 2026 01:07

aidang3019 mentioned this pull request Sep 30, 2026

ponytail(u1): drop bench/ and duplicate viz scripts, reuse gpu_orphans in the reaper #703

Draft

RyanPCo force-pushed the ryanco/phase4-resume-without-base-weights branch from ffebd78 to bcb8775 Compare October 12, 2026 21:50

RyanPCo force-pushed the ryanco/phase4-git-provenance branch from 22327ba to 125facf Compare October 12, 2026 21:50

RyanPCo changed the base branch from ryanco/phase4-resume-without-base-weights to graphite-base/637 October 19, 2026 19:24

RyanPCo changed the base branch from graphite-base/637 to main October 19, 2026 19:25

RyanPCo force-pushed the ryanco/phase4-git-provenance branch from 125facf to 61425db Compare October 19, 2026 19:26

RyanPCo merged commit 201c2cf into main Oct 19, 2026
4 checks passed

Navigation Menu

Repository navigation

write git version into the run dir for reproducibility - #637

write git version into the run dir for reproducibility#637
RyanPCo merged 1 commit into
mainfrom
ryanco/phase4-git-provenance

RyanPCo commented Sep 13, 2026 •

edited

Loading

Uh oh!

RyanPCo commented Sep 13, 2026 •

edited

Loading

Uh oh!

RyanPCo commented Sep 29, 2026

Uh oh!

greptile-apps Bot commented Sep 29, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Sep 29, 2026

Uh oh!

greptile-apps Bot Sep 29, 2026

Uh oh!

greptile-apps Bot Sep 29, 2026

Uh oh!

RyanPCo commented Oct 19, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

RyanPCo commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RyanPCo commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RyanPCo commented Sep 29, 2026

Uh oh!

greptile-apps Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Uh oh!

greptile-apps Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

RyanPCo commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

RyanPCo commented Sep 13, 2026 •

edited

Loading

RyanPCo commented Sep 13, 2026 •

edited

Loading

greptile-apps Bot commented Sep 29, 2026 •

edited

Loading

RyanPCo commented Oct 19, 2026 •

edited

Loading