Skip to content

Tied quantized lm_head fix - #671

Open
neilmsft wants to merge 1 commit into
mainfrom
neilmsft/gemma4-tied-quant-lm-head
Open

Tied quantized lm_head fix#671
neilmsft wants to merge 1 commit into
mainfrom
neilmsft/gemma4-tied-quant-lm-head

Conversation

@neilmsft

@neilmsft neilmsft commented Aug 27, 2026

Copy link
Copy Markdown

Description of bug:

Gemma ties  lm_head  to  embed_tokens , so the GGUF has no  output.weight. But  Gemma4CausalLMModel  built a standalone quantized head anyway, declaring 3 initializers nothing could fill, which crashed the export and forced every build to silently fall back to fp32 embeddings.

Fix

Use TiedQuantizedLMHead in the text-only path when the weights are tied and both sides are quantized, matching the base class. The multimodal path is untouched. Its decoder and embedding live in separate ONNX graphs and genuinely cannot share a table

@neilmsft
neilmsft requested review from a team and a lite review from Copilot August 27, 2026 04:30
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes Gemma4 text-only export when lm_head is tied to embed_tokens under block quantization (GGUF-style checkpoints that omit output.weight). The change aligns Gemma4CausalLMModel’s behavior with the base CausalLMModel by using a tied quantized head that shares the embedding’s packed table instead of creating an unfillable standalone quantized head.

Changes:

  • Import and use TiedQuantizedLMHead for Gemma4 text-only models when embeddings + head are both quantized and weights are tied.
  • Add _tie_quantized_lm_head(...) helper to decide when the tied quantized head is valid (single-graph text-only path).
  • Update _make_lm_head docstring to clarify when the tied quantized head should be preferred.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +2252 to +2254
self.lm_head = _tie_quantized_lm_head(config, self.model.embed_tokens) or _make_lm_head(
config
)
@justinchuby justinchuby self-assigned this Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants