Repository navigation
Conversation
Contributor
Author
RyanPCo
force-pushed
the
ryanco/transformers-upgrade
branch
from
September 16, 2026 17:30
5ac62d2 to
ca14e40
Compare
RyanPCo
force-pushed
the
ryanco/qwen35-vlm
branch
from
September 16, 2026 17:30
e1d39ec to
d0e9f84
Compare
This was referenced Sep 16, 2026
RyanPCo
force-pushed
the
ryanco/qwen35-vlm
branch
from
September 17, 2026 14:52
d0e9f84 to
9432dfd
Compare
RyanPCo
force-pushed
the
ryanco/transformers-upgrade
branch
2 times, most recently
from
September 17, 2026 15:13
1ea2174 to
47ad99b
Compare
RyanPCo
force-pushed
the
ryanco/qwen35-vlm
branch
from
September 17, 2026 15:13
9432dfd to
bd59a39
Compare
New `Qwen35VLMEncoder` (encoder_specs.front_img_1) runs one Qwen 3.5 image+text forward per batch and splits the hidden states by token type: 220 visual tokens (640x352, patch 16 / merge 2) go to the existing image Perceiver stem (input_dim 840 -> 1024), and the text tokens of the same image-conditioned forward are read by the new `Qwen35TextStem` (shared_stem_specs.annotation) through a plain, unregistered reference, so the VLM weights exist once. The VLM is frozen, bfloat16 and runs under no_grad; `trainable_layers` > 0 unfreezes the last N text layers. It carries the `PretrainedWeights` mixin, so finalize_modules never re-initialises it and HPT.__init__ hashes it against its snapshot (the tied lm_head and the unused mtp.* head are reconciled inside the class). `HPTModel.stem_process` gains a six-line hook handing the batch's prompts to an encoder that wants them; HPT.__init__ wires the text stem to its encoder. The ResNet path is unchanged. New config model/hpt_bc_mecka_6d_300M_qwen35.yaml inherits hpt_bc_mecka_6d_300M and swaps only those two entries, the image stem width and the augmentations (the processor normalizes, so the ImageNet Normalize is dropped). flash-linear-attention is pinned: without it transformers falls back to a reference Gated DeltaNet kernel. Dependencies: flash-linear-attention 0.5.2 (with its fla-core), in pyproject and uv.lock. It provides the Triton kernels for Qwen 3.5's hybrid Gated DeltaNet attention; without it transformers falls back to a reference PyTorch implementation, correct but 1.27x slower on the measured forward. Pure-python wheel, no compilation. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011QXaT3JinWYQH8t1zoUhDU
HPT.__init__ defaults self.device to cuda whenever one is visible and copies it into the HPTModel it builds. The trainer's real device only arrives later, through ModelWrapper.on_fit_start / on_validation_start, which write it to the algo alone -- so the policy kept the construction-time device. HPTModel.compute_loss creates its loss accumulators on that stale device while HPT.compute_losses accumulates on the algo's, so a CPU-tier run on a machine with a visible GPU dies with 'Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!'. That is 7 of the unit tests (4 test_hpt_train_step, 2 test_norm_cache, 1 test_resolve_once) whenever the suite is run on a GPU node. HPT.device becomes a property whose setter also writes nets['policy'].device, so the two cannot diverge through the path the trainer uses. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
RyanPCo
force-pushed
the
ryanco/transformers-upgrade
branch
from
September 17, 2026 15:26
47ad99b to
60c5d29
Compare
RyanPCo
force-pushed
the
ryanco/qwen35-vlm
branch
from
September 17, 2026 15:26
bd59a39 to
07d8388
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New
Qwen35VLMEncoder(encoder_specs.front_img_1) runs one Qwen 3.5image+text forward per batch and splits the hidden states by token type:
220 visual tokens (640x352, patch 16 / merge 2) go to the existing image
Perceiver stem (input_dim 840 -> 1024), and the text tokens of the same
image-conditioned forward are read by the new
Qwen35TextStem(shared_stem_specs.annotation) through a plain, unregistered reference, so
the VLM weights exist once. The VLM is frozen, bfloat16 and runs under
no_grad;
trainable_layers> 0 unfreezes the last N text layers.It carries the
PretrainedWeightsmixin, so finalize_modules neverre-initialises it and HPT.init hashes it against its snapshot (the tied
lm_head and the unused mtp.* head are reconciled inside the class).
HPTModel.stem_processgains a six-line hook handing the batch's promptsto an encoder that wants them; HPT.init wires the text stem to its
encoder. The ResNet path is unchanged.
New config model/hpt_bc_mecka_6d_300M_qwen35.yaml inherits
hpt_bc_mecka_6d_300M and swaps only those two entries, the image stem
width and the augmentations (the processor normalizes, so the ImageNet
Normalize is dropped). flash-linear-attention is pinned: without it
transformers falls back to a reference Gated DeltaNet kernel.
Co-Authored-By: Claude Opus 5 (1M context) [email protected]
Claude-Session: https://claude.ai/code/session_011QXaT3JinWYQH8t1zoUhDU