Tied quantized lm_head fix - #671
Conversation
Signed-off-by: neilmsft <[email protected]>
|
|
There was a problem hiding this comment.
Pull request overview
Fixes Gemma4 text-only export when lm_head is tied to embed_tokens under block quantization (GGUF-style checkpoints that omit output.weight). The change aligns Gemma4CausalLMModel’s behavior with the base CausalLMModel by using a tied quantized head that shares the embedding’s packed table instead of creating an unfillable standalone quantized head.
Changes:
- Import and use
TiedQuantizedLMHeadfor Gemma4 text-only models when embeddings + head are both quantized and weights are tied. - Add
_tie_quantized_lm_head(...)helper to decide when the tied quantized head is valid (single-graph text-only path). - Update
_make_lm_headdocstring to clarify when the tied quantized head should be preferred.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| self.lm_head = _tie_quantized_lm_head(config, self.model.embed_tokens) or _make_lm_head( | ||
| config | ||
| ) |
Description of bug:
Gemma ties lm_head to embed_tokens , so the GGUF has no output.weight. But Gemma4CausalLMModel built a standalone quantized head anyway, declaring 3 initializers nothing could fill, which crashed the export and forced every build to silently fall back to fp32 embeddings.
Fix
Use TiedQuantizedLMHead in the text-only path when the weights are tied and both sides are quantized, matching the base class. The multimodal path is untouched. Its decoder and embedding live in separate ONNX graphs and genuinely cannot share a table