Conversation
Remove the hybrid (mamba/SSM) dependency from GraniteSwitch. The switch model is attention-only and never constructs a mamba layer, so it does not need the GraniteMoeHybrid family — GraniteMoeShared provides every class it actually uses (shared MLP, MoE, RMSNorm, RoPE, PreTrainedModel base) minus the mamba machinery. HF backend: - GraniteSwitchConfig now extends GraniteMoeSharedConfig; modeling and core/lora imports repointed to the granitemoeshared twins. - layer_types / position_embedding_type become switch-owned attributes (the shared parent does not declare them, but internal readers still depend on them). shared_intermediate_size fix: the shared parent defaults it to 0, which is also the "no shared MLP" sentinel for pure sparse-MoE bases. The config now resolves it itself instead of inheriting a magic default — explicit values (including 0) are honored verbatim; when unset, dense resolves to intermediate_size and pure MoE keeps 0. This is a compose-time decision frozen into config.json; it also closes a latent bug where a bare dense config silently inherited the old 1024 default. Guarded by new unit tests. composer: granite_moe_hybrid_arch/_sr_arch renamed to granite_moe_shared_arch/_sr_arch. The "granitemoehybrid" registry key is retained (mapped to the shared arch) because real Granite 4.x dense checkpoints are still typed granitemoehybrid upstream; a "granitemoeshared" key is added alongside. vLLM backend: - Removed the vestigial IsHybrid / HasInnerState marker mixins (no hybrid contract was implemented). - The two borrowed upstream classes (GraniteMoeMoE, GraniteMoeSharedMLP) now load via a version-tolerant helper that prefers the non-hybrid granitemoe / granitemoeshared modules and falls back to granitemoehybrid, so a single codebase works across the pinned vLLM versions. Local CPU tests pass (unit, config sis-trap, composer arch skinning, HF granite4 equivalence, HF forward/lora/multi-switch). vLLM and GPU generation tests to run on the cluster. Signed-off-by: antonp <[email protected]>
eval/gen_smoke.py loads a composed Granite Switch checkpoint in vLLM and generates the same question with the base path and with each adapter's control token, printing the outputs so a reviewer can confirm the de-hybridized (granitemoeshared) backend both loads and routes adapters. Used by the Vela validation job (vela_yamls/dehybridize_vllm_gen.yaml). Signed-off-by: antonp <[email protected]>
Signed-off-by: antonp <[email protected]>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Collaborator
Author
|
/gpu-test-multi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.