Add GLM-5.3 Flash multimodal support - #713
Open
justinchuby wants to merge 1 commit into
Open
Conversation
Implement the pinned glm5_next NoPE KDA/pooled-DSA text core, mHC residual streams, clamped MoE, packed image/video tower, heterogeneous cache task, and shared-placeholder multimedia mixer. Add production-derived HTTP range-read real-weight fixtures with CUDA L4/L5 goldens and fail-closed runtime/FP8 limitations. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The routed-expert implementation in Glm5NextExperts.forward currently computes every expert each token (unrolled loop), which is a severe scalability/performance blocker for the production expert count.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR adds architecture-specific Mobius support for Hugging Face glm5_next (GLM‑5.3‑Flash), including a 3-model multimodal split (decoder / vision encoder / embedding), pinned reduced real-weight fixtures + goldens, and fail-closed ORT GenAI metadata behavior for GLM‑5.3’s heterogeneous cache state.
Changes:
- Introduces
Glm5NextConfig, GLM‑5.3 model/task implementations, and registersglm5_next/glm5_next_textin the registry. - Adds GLM‑5.3 vision components and config extraction hooks for the packed vision tower.
- Adds synthetic parity + model-specific integration/golden tests and pinned reduced fixtures/golden outputs.
File summaries
| File | Description |
|---|---|
| tests/synthetic_parity_test.py | Adds synthetic parity tolerance + HF config/model construction support for glm5_next_text. |
| tests/_test_configs.py | Adds tiny text and VL configs for glm5_next_text / glm5_next. |
| testdata/golden/vision-language/glm5-next-reduced.json | Adds pinned reduced real-weight L4 golden (logits/top-k). |
| testdata/golden/vision-language/glm5-next-reduced_generation.json | Adds pinned reduced real-weight L5 generation golden (token ids + per-step logits). |
| testdata/cases/vision-language/glm5-next.yaml | Adds an L4+L5 YAML case (CI-skipped) documenting the pinned production model/revision. |
| src/mobius/tasks/_glm5_next.py | Adds GLM‑5.3 text + vision-language tasks and heterogeneous cache I/O + metadata. |
| src/mobius/tasks/init.py | Exports/registers new GLM‑5.3 tasks. |
| src/mobius/models/glm5_next.py | Implements GLM‑5.3 text + multimodal modules, cache ABI, vision encoder, and weight preprocessing. |
| src/mobius/models/glm5_next_test.py | Adds contract, parity, reduced real-weight, CUDA semantics, and fail-closed metadata tests. |
| src/mobius/models/init.py | Exports GLM‑5.3 model classes. |
| src/mobius/integrations/transformers/_builder.py | Fail-closed weight-loading guard for the pinned 328GB block-FP8 checkpoint. |
| src/mobius/integrations/ort_genai/auto_export.py | Rejects ORT GenAI config emission when GLM‑5.3 heterogeneous cache inputs are detected. |
| src/mobius/components/_glm4v_vision.py | Adds GLM‑5.3 packed ViT vision model/blocks/attention variants. |
| src/mobius/components/_glm_ocr_vision.py | Replaces Compress with NonZero+Gather for position/cu_seqlens derivation. |
| src/mobius/components/init.py | Exports Glm5NextVisionModel. |
| src/mobius/_registry.py | Registers glm5_next / glm5_next_text model types and default IDs, including text-only mapping. |
| src/mobius/_configs/per_model/_glm5_next_vision.py | Adds per-model vision-config extraction hook for GLM‑5.3. |
| src/mobius/_configs/per_model/init.py | Imports the new GLM‑5.3 per-model vision hook for side-effect registration. |
| src/mobius/_configs/_sub_configs.py | Extends VisionConfig with swiglu_limit used by GLM‑5.3 vision tower. |
| src/mobius/_configs/_base.py | Adds Glm5NextConfig with strict validation and HF config extraction. |
| src/mobius/_configs/init.py | Exports Glm5NextConfig. |
| scripts/generate_glm5_next_reduced_fixture.py | Adds a script to create reduced production-derived fixture + goldens via bounded range reads. |
| README.md | Adds GLM‑5.3‑Flash to supported MoE and Multimodal examples list. |
Review details
- Files reviewed: 24/25 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+553
to
+566
| def _adapt_glm5_next_text_config(hf_kwargs: dict) -> None: | ||
| """Translate the typed Mobius GLM-5.3 fields to the upstream config.""" | ||
| num_heads = hf_kwargs.pop("linear_num_heads") | ||
| head_dim = hf_kwargs.pop("linear_head_dim") | ||
| conv_kernel = hf_kwargs.pop("linear_conv_kernel_dim") | ||
| lower_bound = hf_kwargs.pop("linear_lower_bound") | ||
| hf_kwargs["linear_attn_config"] = { | ||
| "num_heads": num_heads, | ||
| "head_dim": head_dim, | ||
| "short_conv_kernel_size": conv_kernel, | ||
| "gate_lower_bound": lower_bound, | ||
| "safe_gate": True, | ||
| } | ||
| hf_kwargs["n_routed_experts"] = hf_kwargs.pop("num_local_experts") |
Comment on lines
+91
to
+152
| def forward( | ||
| self, | ||
| op: OpBuilder, | ||
| hidden_states: ir.Value, | ||
| selected_experts: ir.Value, | ||
| routing_weights: ir.Value, | ||
| ) -> ir.Value: | ||
| original_shape = op.Shape(hidden_states) | ||
| flat_hidden = op.Reshape(hidden_states, [-1, self._hidden_size]) | ||
| flat_selected = op.Reshape(selected_experts, [-1, self._top_k]) | ||
| flat_weights = op.Reshape(routing_weights, [-1, self._top_k]) | ||
| result = None | ||
| for expert_index in range(self._num_experts): | ||
| gate_up_weight = op.Squeeze( | ||
| op.Gather( | ||
| self.gate_up_proj, | ||
| op.Constant(value_ints=[expert_index]), | ||
| axis=0, | ||
| ), | ||
| [0], | ||
| ) | ||
| projected = op.MatMul(flat_hidden, op.Transpose(gate_up_weight)) | ||
| gate, up = op.Split( | ||
| projected, | ||
| [self._intermediate_size, self._intermediate_size], | ||
| axis=-1, | ||
| _outputs=2, | ||
| ) | ||
| gate = op.Clip(gate, None, self._limit) | ||
| up = op.Clip(up, -self._limit, self._limit) | ||
| activated = op.Mul(op.Swish(gate), up) | ||
| down_weight = op.Squeeze( | ||
| op.Gather( | ||
| self.down_proj, | ||
| op.Constant(value_ints=[expert_index]), | ||
| axis=0, | ||
| ), | ||
| [0], | ||
| ) | ||
| expert_output = op.MatMul(activated, op.Transpose(down_weight)) | ||
| selected = op.Equal( | ||
| flat_selected, | ||
| op.Constant(value_int=expert_index), | ||
| ) | ||
| weight = op.ReduceSum( | ||
| op.Mul( | ||
| flat_weights, | ||
| op.CastLike(selected, flat_weights), | ||
| ), | ||
| [-1], | ||
| keepdims=True, | ||
| ) | ||
| contribution = op.CastLike( | ||
| op.Mul( | ||
| op.Cast(expert_output, to=ir.DataType.FLOAT), | ||
| weight, | ||
| ), | ||
| expert_output, | ||
| ) | ||
| result = contribution if result is None else op.Add(result, contribution) | ||
| assert result is not None | ||
| return op.Reshape(result, original_shape) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add production-shape Mobius support for Hugging Face
glm5_next, pinned tozai-org/GLM-5.3-Flash@03eb5366286afd40d2221b1d9c63a6dd1ba4832eand Transformers mergeeb4d9e2a64a013bec12289288b85d0b1210ba0aa.The implementation is architecture-specific rather than an alias:
<|image|>placeholder routing, disambiguating video by begin/end spansdecoder,vision_encoder, andembeddingModelPackage rolesPinned evidence
The committed 67,280-byte, 81-tensor reduced fixture is derived only from bounded HTTP range reads against the immutable production checkpoint. It selects production layer 0 (KDA+dense), layer 3 (pooled DSA+MoE), the complete reduced vision/embedding route, and records 115 source spans with shard, dtype, byte offsets, and SHA-256. BF16 and FP8 source ranges are covered; FP8 values are reconstructed with their pinned 128x128 block scales. Goldens were independently generated by the pinned Hugging Face implementation.
Validation
python -m pytest tests/build_graph_test.py tests/weight_alignment_test.py tests/arch_validation_test.py tests/synthetic_parity_test.py tests/yaml_schema_test.py -k "glm5_next or glm5-next" -q --tb=short— 17 passedpython -m pytest src/mobius/models/glm5_next_test.py -q --tb=short— 10 passed, 1 fp16 CUDA pass, 1 BF16 CUDA strict xfailpython -m pytest "src/mobius/models/glm5_next_test.py::test_reduced_real_weight_l4_l5_golden[cuda]" -q --tb=short— 1 passed; checks nonzero vision → embedding → full prefill, 24 cached decode steps, full logits, and exact tokenspython -m pytest src/mobius/models/glm_ocr_test.py src/mobius/models/glm_moe_dsa_test.py -q --tb=short— 26 passedpython -m pytest tests/quantization_integration_test.py -q --tb=short— 6 passedtest_ort_genai_metadata_fails_closed_for_heterogeneous_state— passedpython -m mobius build --model zai-org/GLM-5.3-Flash --revision 03eb5366286afd40d2221b1d9c63a6dd1ba4832e --no-weights --output <dir>— passed, 3 model roles / 6 files / 178,761,024 byteslintrunner f --output oneline --all-filesinvoked afterlintrunner init; Windows hit WinError 206 for the all-files Ruff command, so changed files were formatted directly with pinned Rufflintrunner -a— clean: no lint issuesWaivers
MemcpyTransformer: internalLessprovider type unset) before execution. FP32 and FP16 semantic parity pass; BF16 is a strict xfail with the exact runtime failure documented. The BF16 vision stage explicitly computes in float32 because ORT CUDA lacks a complete packed-ViT BF16 kernel set.Commit:
6697566bfdcb3a63e64b14d02c3c7e4b0ee1d023