fix: Share the target lm_head for DSpark drafts without their own head - #200
Open
rain-sicoreai wants to merge 1 commit into
Open
fix: Share the target lm_head for DSpark drafts without their own head#200rain-sicoreai wants to merge 1 commit into
rain-sicoreai wants to merge 1 commit into
Conversation
RadixArk/Qwen3.8-27B-DSpark, a documented DSpark pair for RadixArk/Qwen3.8-27B-NVFP4, ships five proposal layers and no lm_head; the ONNX export path already shares the base model's head for it. The direct builder rejected the checkpoint with "DSpark draft checkpoint must provide lm_head", so tensorrt-edgellm-serve could not build the pair. Mirror the EAGLE3 draft: when the draft has no head, open the paired target checkpoint and build lm_head from the target's head in the target's precision (NVFP4 here), releasing the target weights after the graph is built. Verified on Jetson AGX Thor with Qwen3.8-27B NVFP4 + Qwen3.8-27B-DSpark (block 7): bundle builds and serves; coding prompts decode at 44-57 tok/s single-stream and 128 tok/s aggregate at 4 concurrent requests. Co-Authored-By: Claude Fable 5.1 <[email protected]> Signed-off-by: Rain Zhang <[email protected]> (cherry picked from commit 9847044edfb6de642d46b03c6323c6ce7dd7e98b)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Type of change: Bug fix (direct builder, DSpark speculative decoding)
Overview:
RadixArk/Qwen3.8-27B-DSpark is listed under DSpark Draft Models as the pair for
RadixArk/Qwen3.8-27B-NVFP4. The draft checkpoint ships five proposal layers and nolm_head; it shares the target model's head. The ONNX export path already handles this (tensorrt_edgellm/models/dspark/modeling_dspark_draft.pydocumentslm_headas "shared with base"), but the direct builder rejected the checkpoint:so
tensorrt-edgellm-servecould not build the pair.DSparkDraftModelnow mirrorsEagle3DraftModel: when the draft checkpoint has no head,from_configopens the paired target checkpoint (--target-model-dir, which the CLI already passes for paired drafts) as a secondWeightssource withcheckpoint_source="target", buildslm_headfrom the target's head in the target's precision (NVFP4 here), and releases the target weights inclose(). Drafts that carry their own head are unchanged.Verified on Jetson AGX Thor with the Qwen3.8-27B NVFP4 (ModelOpt) base and
Qwen3.8-27B-DSpark, block size 7, batch 4, max input 32768, KV capacity 65536:tensorrt-edgellm-servewith--speculative-config '{"method":"dspark","model":<draft>,"num_speculative_tokens":7}';num_speculative_tokensfrom 1 to 7 works at runtime;Usage
tensorrt-edgellm-serve RadixArk/Qwen3.8-27B-NVFP4 \ --cache-dir /data/edgellm-cache \ --speculative-config '{"method":"dspark","model":"RadixArk/Qwen3.8-27B-DSpark","num_speculative_tokens":7}'🚀 Pull Request Checklist
✅ Pre-commit Checks
pre-commitby runningpip install pre-commit.pre-commit install.pre-commit run --all-filesand fixed any reported issues.🧪 Tests
📄 Documentation
supported-models.md.⚙️ Compatibility
lm_headtake the unchanged path.Additional Information
Independent of #199; it only touches
experimental/builder/models/dspark/modeling_dspark_draft.py.🤖 Generated with Claude Code