eval: log validation videos to wandb by RyanPCo · Pull Request #684 · GaTech-RL2/EgoVerse

RyanPCo · 2026-09-22T18:24:45Z

Each mp4 EvalVideo writes is also logged to any WandbLogger under <video dir>/<embodiment> (videos/, videos_train_viz/, videos_unseen_op_valid/), captioned with the epoch, so runs can be reviewed without copying files off the cluster.

It also adds a GT/prediction replay video mode (egomimic/eval/replay_viz.py) and makes it the default viz, with a chunk-based cap and an inferred stride.

Tests: tests/unit/test_eval_video_upload.py, tests/unit/test_replay_viz.py.

Folded in (to keep the stack under Graphite's 50-branch cap):

eval: GT/prediction replay video, on by default #685 — eval: GT/prediction replay video, on by default

🤖 Generated with Claude Code

https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw

RyanPCo · 2026-09-22T18:25:32Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

RyanPCo

Automated adversarial review (read-only; findings were checked by a second skeptical pass before posting): 3 low, 2 nit.

RyanPCo · 2026-10-30T20:55:48Z

+    return image
+
+
+def _draw_skeleton(image, keypoints, intrinsics, edges, color):


low (consistency): _draw_skeleton is a second keypoint renderer, and Human.viz says there is deliberately no static-skeleton path. Move it into egomimic/utils/viz_utils.py next to _viz_keypoints_horizon_trace (sharing _N_KP and the layout checks), and update the Human.viz comment to say the replay video is the one sanctioned per-step skeleton renderer.

RyanPCo · 2026-10-30T20:55:48Z

-                    self.val_image_buffer[key].clear()
-                    self.val_counter[key] += 1
+                if isinstance(images, ReplayClip):
+                    self.val_image_buffer[key].append(images)


low (correctness): Replay mode caps a head at replay_chunks * H * stride frames over one concatenated clip of all its pinned episodes. For unseen_op_valid (two pins) that is 500 frames, so only the first operator's episode is ever rendered, while the yaml says one episode per operator. If a pinned episode is shorter than the cap, a chunk anchored near its end draws on the next episode's frames and rigid_fit pairs hands from two episodes. Either apply the cap per episode and don't anchor chunks across an episode boundary (carry the episode hash per frame in ReplayClip), or document that replay renders only the first pin.

RyanPCo · 2026-10-30T20:55:49Z

+_REPLAY_OWN_KWARGS = {
+    "image_key",
+    "action_key",
+    "annotation_key",


low (correctness): Replay mode removes annotation_key from the viz kwargs, and render_replay never draws the prompt. Pinned-episode videos for configs that set it (cartesian.yaml, cotrain_lang.yaml, pi__lang.yaml) therefore silently lose the prompt overlay now that replay is the default. Carry the per-frame annotation in ReplayClip and draw it next to the GT/PRED label, or at least log once that replay ignores annotation_key.

RyanPCo · 2026-10-30T20:55:49Z

+from egomimic.utils.type_utils import _to_numpy
+from egomimic.utils.viz_utils import _prepare_viz_image
+
+_KP_WIDTH = 126


nit (duplication): _KP_WIDTH = 126 duplicates the layout that viz_utils derives from _N_KP. Define it as 2 * 3 * _N_KP, or export a named constant from viz_utils, so the two cannot drift.

RyanPCo · 2026-10-30T20:55:49Z

+                    self._replay_seen[key] = seen
+                    self._replay_horizon[key] = images.gt.shape[1]
+                    # infer_stride compares steps 1-8 against up to 32 frames on.
+                    if key not in self._replay_stride and seen >= 40:


nit (magic-number): seen >= 40 only works because it exceeds max(candidates) * steps (32) for infer_stride's defaults. If those defaults change, larger strides are silently skipped. Derive it from named constants in replay_viz, e.g. MIN_FRAMES_FOR_STRIDE = max(STRIDE_CANDIDATES) * STRIDE_STEPS + 1.

Each mp4 EvalVideo writes is also logged to any WandbLogger under <video dir>/<embodiment> (videos/, videos_train_viz/, videos_unseen_op_valid/), captioned with the epoch, so runs can be reviewed without copying files off the cluster. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01415567AQYskZLeSpSMsgwL

viz_mode=replay plays each action chunk twice over the frames it spans: first the ground-truth hand at each step, then the predicted one. Each step is drawn on the frame it belongs to, with the chunk mapped into that frame's camera by a rigid fit of the GT keypoints (the same hand at the same instant in both frames), so head motion needs no head pose. The 126-D keypoint layout draws both hands' 21-keypoint skeletons; other layouts fall back to the embodiment's own viz over a short trail. Chunks don't overlap and span horizon * action_stride frames; frames of an incomplete chunk carry over to the next flush. viz_max_batches must cover at least one chunk (a 100-step horizon needs ~13 batches of 8). _visualize_preds moves from HPTEvalVideo / PIEvalVideo (identical copies) into EvalVideo. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01516MaELbiPNmJzkjPGebhu

Replay becomes EvalVideo's default. It needs consecutive frames, so it runs only on a pinned video loader; a head without one keeps the overlay video, and EvalLatent stays on overlay. The replay video loader is capped at replay_chunks (5) chunks instead of viz_max_batches, which at 8 frames a batch could not cover one 100-step chunk; viz_max_batches still bounds a loader that yields no clips. action_stride defaults to None and is inferred from the GT keypoints (step k of a chunk rigidly matches frame k*stride's own step 0): data configs use 1 or 3 and cotrain mixes them under one embodiment. TrainVizEvalVideo passes the replay settings through and tells its base which mode each step is. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01516MaELbiPNmJzkjPGebhu

RyanPCo · 2026-10-30T23:10:13Z

Review findings: status after the fixup pass. The fixes are squashed into this PR's commits, and the whole stack was force-pushed.

Fixed

Static-skeleton renderer that Human.viz forbids: moved to viz_utils._viz_keypoints_skeleton; Human.viz comment updated (human.py comment outside the PR, needed for the fix; kept short next to viz: one keypoint renderer, the horizon trace #669's one-line version); fixup targets fe8933a to avoid autosquash conflicts
_KP_WIDTH = 126 hardcoded: viz_utils.HEAD_KEYPOINT_WIDTH = 23_N_KP, used in replay_viz and test_replay_viz
Stride-inference threshold 40 and flush 1000 are magic numbers: STRIDE_CANDIDATES, STRIDE_STEPS and MIN_FRAMES_FOR_STRIDE in replay_viz; _MAX_BUFFERED_FRAMES in eval_video
Replay mode silently drops the prompt overlay: ReplayClip carries annotation_key prompts and render_replay draws them via viz(mode='annotations'); test added

Partial

Replay merges pinned episodes; chunks cross boundaries; cap stops before later episodes: ReplayClip carries episode_hash per frame and render_replay never anchors a chunk across episodes (test added); cap is still per key, now documented rather than applied per episode

This was referenced Sep 22, 2026

slurm: stop cancelled jobs from leaking whole GPUs #682

Open

trainer: DDP devices are per node; eval-mode val subsampling is single-rank #683

Closed

RyanPCo changed the base branch from ryanco/gpu-orphan-reap to graphite-base/684 September 22, 2026 18:51

RyanPCo force-pushed the ryanco/eval-video-wandb branch 2 times, most recently from 3a1c92d to fc272d2 Compare September 22, 2026 18:51

RyanPCo changed the base branch from graphite-base/684 to ryanco/ddp-world-size September 22, 2026 18:51

This was referenced Sep 23, 2026

eval: GT/prediction replay video, on by default #685

Closed

data: exclude episodes by hash; scan Mecka egomotion for failed head tracking #686

Draft

RyanPCo force-pushed the ryanco/eval-video-wandb branch from fc272d2 to 4a21de7 Compare September 24, 2026 03:19

aidang3019 mentioned this pull request Sep 29, 2026

eva: joint-space mode for the ABC YAM action space; robot image history #702

Draft

aidang3019 force-pushed the ryanco/ddp-world-size branch from e9a9545 to ec615c7 Compare September 30, 2026 01:06

aidang3019 force-pushed the ryanco/eval-video-wandb branch from 4a21de7 to 388ce59 Compare September 30, 2026 01:07

aidang3019 mentioned this pull request Sep 30, 2026

ponytail(u1): drop bench/ and duplicate viz scripts, reuse gpu_orphans in the reaper #703

Draft

RyanPCo force-pushed the ryanco/eval-video-wandb branch from 388ce59 to b649f4e Compare October 12, 2026 20:26

RyanPCo changed the base branch from ryanco/ddp-world-size to ryanco/gpu-orphan-reap October 12, 2026 20:27

RyanPCo force-pushed the ryanco/eval-video-wandb branch from b649f4e to d04cb33 Compare October 12, 2026 21:50

RyanPCo force-pushed the ryanco/eval-video-wandb branch from d04cb33 to ab61d1e Compare October 12, 2026 23:19

RyanPCo mentioned this pull request Oct 19, 2026

lance migration #716

Open

RyanPCo force-pushed the ryanco/gpu-orphan-reap branch from 7246780 to 222aa11 Compare October 19, 2026 19:51

RyanPCo force-pushed the ryanco/eval-video-wandb branch 2 times, most recently from b2dc225 to d99acaf Compare October 19, 2026 20:22

RyanPCo force-pushed the ryanco/eval-video-wandb branch from d99acaf to fe8933a Compare October 30, 2026 18:27

RyanPCo commented Oct 30, 2026

View reviewed changes

RyanPCo and others added 3 commits October 30, 2026 18:51

RyanPCo force-pushed the ryanco/gpu-orphan-reap branch from 26995d8 to 863ec5e Compare October 30, 2026 23:05

RyanPCo force-pushed the ryanco/eval-video-wandb branch from fe8933a to 1228626 Compare October 30, 2026 23:05

RyanPCo mentioned this pull request Oct 30, 2026

eval: overlay videos play at the source rate under DDP #666

Draft

Navigation Menu

Repository navigation

eval: log validation videos to wandb - #684

eval: log validation videos to wandb#684
RyanPCo wants to merge 3 commits into
ryanco/gpu-orphan-reapfrom
ryanco/eval-video-wandb

RyanPCo commented Sep 22, 2026 •

edited

Loading

Uh oh!

RyanPCo commented Sep 22, 2026 •

edited

Loading

Uh oh!

RyanPCo left a comment

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo commented Oct 30, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		return image


		def _draw_skeleton(image, keypoints, intrinsics, edges, color):

Conversation

RyanPCo commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RyanPCo commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RyanPCo left a comment

Choose a reason for hiding this comment

Uh oh!

RyanPCo Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

RyanPCo Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

RyanPCo Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

RyanPCo Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

RyanPCo Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

RyanPCo commented Oct 30, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

RyanPCo commented Sep 22, 2026 •

edited

Loading

RyanPCo commented Sep 22, 2026 •

edited

Loading