Repository navigation
Conversation
… share Co-Authored-By: Claude Fable 5.1 <[email protected]>
Co-Authored-By: Claude Fable 5.1 <[email protected]>
…/ starVLA port, torch primitives) Co-Authored-By: Claude Fable 5.1 <[email protected]>
…Model Co-Authored-By: Claude Fable 5.1 <[email protected]>
…nistic eval) Co-Authored-By: Claude Fable 5.1 <[email protected]>
…es for the QwenVLA vs HPT comparison
QwenVLA: qwenvla_bc_mecka_{6d,kp}_0p8b, qwenvla_bc_mecka_6d_2b,
qwenvla_cotrain_eva_mecka_6d, train_zarr_mecka_flagship_{6d,kp}_qwenvla.
Comparison arms (user, 2026-09-16): 144-D wrist-frame MANO keypoints on the
flagship top-3 split with default_prompt "fold clothes":
data/mecka_fold_flagship_{opsplit,top3}_hpt_keypoints and
train_zarr_mecka_flagship_kp_hpt; hpt_bc_keypoints_wrist_300M gains the
history_len / history_dropout stem knobs and backbone_lr_scale so it matches
hpt_bc_mecka_6d_300M's lanes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
…ression) QwenVLA needs find_unused_parameters (the interleaved DiT leaves the last VLM layer and final norm without a gradient) and, on the lab's NVLink-less 8 x A40 nodes, bf16 all-reduce: 6.4 -> 3.6 s/step at per-GPU batch 16. The hook is exposed through a zero-arg factory because Lightning logs its __qualname__, which a Hydra _partial_ does not carry. Co-Authored-By: Claude Fable 5.1 <[email protected]>
…once Lightning assigns the rank's GPU
Every DDP rank constructed the algo on torch.device('cuda') = cuda:0 before
Lightning had assigned per-rank devices, so 8 x 4.8 GB of fp32 VLM weights
sat on GPU 0 until on_fit_start. A fresh start just fit; a checkpoint resume
loads the state into those copies first and OOMs (job 3850634).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
_video_fps divided 30 fps by world_size to undo a DistributedSampler stride that never applies: the val heads are CombinedLoaders Lightning does not shard (every rank runs every val batch; 1 rank = 223 val steps, 8 ranks = 8 x 223 in the 2026-09-16 DDP check), so rank 0 renders every frame and the 8-GPU videos came out as 8x slow motion (416 frames at 4 fps). Co-Authored-By: Claude Fable 5.1 <[email protected]>
This was referenced Sep 17, 2026
RyanPCo
changed the title
This was referenced Sep 17, 2026
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports starVLA's QwenPI into EgoVerse as a new algo, so the HPT architecture campaign has a VLM-policy arm to compare against. Bottom to top:
bdf27905bDeterministicEvalMixin— the seeded-prompt / seeded-flow-sampling eval helpers lifted out of HPT so both algos share one contract.785b4153emulti-image fake Qwen 3.5 fixture — lets everything below be tested without loading a real checkpoint.9905b88b7layer-wise cross-attention DiT +LayerwiseFMHead— a GR00T N1.5 / starVLA port written in torch primitives.1aeeb5592QwenVLBackboneandQwenVLAModel— multi-frame Qwen 3.5 with all layers exposed, so the head can attend per block.45d199f45the QwenVLA algo — embodiment prompt, padded shared action width, deterministic eval.d7c81f253configs — QwenVLA model and recipe yamls, plus the flagship keypoint data and recipes for the QwenVLA-vs-HPT comparison.80609cdafddp_qwenvlatrainer —find_unused_parameters(the interleaved DiT leaves the last VLM layer and final norm without a gradient) and bf16 gradient compression, which took the lab's NVLink-less 8xA40 nodes from 6.4 to 3.6 s/step at per-GPU batch 16. The hook is exposed through a zero-arg factory because Lightning logs its__qualname__, which a Hydra_partial_does not carry.6acc4940bbuild on CPU, move once Lightning assigns the rank's GPU — every DDP rank used to construct the algo oncuda:0before per-rank devices were assigned, parking 8 x 4.8 GB of fp32 VLM weights on GPU 0. A fresh start just fit; a checkpoint resume loaded state into those copies first and OOMed (job 3850634).c3d071e11eval overlay fps —_video_fpsdivided 30 fps byworld_sizeto undo aDistributedSamplerstride that never applies (the val heads areCombinedLoadersLightning does not shard), so rank 0 rendered every frame and the 8-GPU videos came out as 8x slow motion.The comparison this feeds: 144-D wrist-frame keypoints, prompt "fold clothes", the same 8xA40 recipe as the HPT rerun.
Testing
test_layerwise_dit.py,test_qwenvla_algo.py,test_qwenvla_nets.pyandtest_qwenvla_configs.py; the config tests drive two optimizer steps through the realtrainHydra.train()path against the fake backbone.🤖 Generated with Claude Code