Tracking: Nemotron 3.5 support, GGUF conversion, and customer prompt

[justinchuby]

Purpose

This issue is the asynchronous status channel for the remaining work. All material progress, blockers, decisions, PR links, and final results will be posted here.

Parallel workstreams

• Fix the unit-test and lint failures currently present on main; Fix merged unit and lint regressions #484 merged as 2073725840e007c1fba59ce3cb1f06e4ad9fe75a.
• Add support from the official BF16 checkpoint: Add NVIDIA Nemotron 3.5 Lightning export support #487 is published; FP16 CUDA is supported, BF16 execution is explicitly rejected by validated error evidence, and final review-fix/rebase CI is pending.
• Provide and validate an executable Olive recipe: Add NVIDIA Nemotron 3.5 Lightning export support #487 includes reduced-real Q4 quantization, assembled-package CUDA reload, and exact generation; automated GPU coverage is implemented locally and awaits final rebase CI.
• Evaluate direct conversion from unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF; direct conversion is not semantically supportable, and reusable fail-fast guards plus evidence-based limitations are published in Guard unsupported Nemotron 3.5 GGUF conversion #486.
• Refine repository skills using both trials: direct-GGUF guidance is in Guard unsupported Nemotron 3.5 GGUF conversion #486; MoE, reduced-real testing, ORT GenAI cache, and Olive guidance are in Add NVIDIA Nemotron 3.5 Lightning export support #487; final combined rebase is pending.
• Create a separate issue containing the final customer-facing prompt in Chinese (客户短 Prompt：Safetensors 与 GGUF 模型转换 #485).

Required implementation paths

Option A: official BF16 + Olive

Pin the source revision, implement faithful architecture/config/weight support, export with Mobius, provide an Olive quantization recipe, assemble the runtime package, and execute the quantized result.

Option B: direct GGUF conversion

Proceed only if GGUF architecture metadata, tensor mappings, quantization types, tokenizer/config provenance, Mobius weight loading, and the target runtime are all supported. Provide exact commands and post-conversion generation evidence. If unsupported, fail early with an actionable explanation rather than emitting a non-runnable package.

Acceptance principles

• Graph creation, export completion, config emission, and session loading are not semantic correctness evidence.
• Validate real nonzero input and final generated output.
• Include exact commands, artifact layout, dtype/EP/runtime, and provider assignment.
• Validate L1-L5 where feasible; document narrow resource/runtime waivers with evidence.
• Run initialized repository-pinned lint and automated review.
• Do not alter unrelated model code or tolerances to hide baseline failures.

Current status

CI repair #484 is merged. Option B guards and direct-GGUF skill guidance are published as #486 and are rebasing onto the repaired main. Option A official-checkpoint + Olive support is #487; its review fixes and rebase validation are in progress. The Chinese customer prompt is #485.

Reporting

All subsequent tracking updates on this issue will be in English. The final customer prompt will be delivered in a separate Chinese-language issue.

[justinchuby]

Baseline verification update

The CI repair agent started from the skills PR's final head 99cd247, while the BF16 agent started from the earlier e77fe76; the GGUF agent also required verification. The skills PR had merged into origin/main as 9b07f09. All three agents were instructed to rebase linearly onto latest origin/main and verify that 9b07f09 was an ancestor of HEAD before continuing. The CI agent was also explicitly directed to repair the real lint/unit failures already merged into main rather than skipping them because they were outside the skills diff.

[justinchuby]

Nemotron 3.5 blind-trial branch rebased linearly onto merged skills baseline 9b07f09. Verified merge-base ancestry. New HEAD: 9b07f09. No repository edits had been made before the rebase; implementation investigation is continuing on this baseline.

[justinchuby]

CI repair progress: rebased the focused child branch onto current origin/main (9b07f09). Updated the diffusers fallback assertion for execution_provider="default" and applied the exact Ruff format/RUF043 fixes reported by run 31757934212. Focused tests now pass (31 passed). Blocker: on Windows, the required full lintrunner f --output oneline --all-files invocation reaches Ruff but fails with WinError 206 (expanded command line too long); it still applied available patches. I am running lintrunner -a, targeted lintrunner checks, equivalent full pinned Ruff checks, and the canonical non-integration suite before publishing the follow-up PR.

[justinchuby]

Ancestry/scope update: the repair branch was retargeted directly onto current origin/main at 9b07f09 (the merged PR #482 commit), so history is linear and includes the original 10-file skills change only through main ancestry. The focused repair diff itself is 3 files: src/mobius/_builder_test.py updates the expected execution_provider="default" contract; src/mobius/_model_package.py applies the pinned Ruff formatting; and src/mobius/_model_package_test.py escapes the regex metacharacter flagged by RUF043. No model behavior, tolerances, skill files, or unrelated formatter output are changed.

[justinchuby]

Option B baseline update: rebased the isolated justinchuby-trial-nemotron-gguf-conversion branch linearly onto current origin/main (no merge). Commit 9b07f09 is now HEAD and is verified as an ancestor of the branch. The merged PR #482 skill files are present; direct-GGUF feasibility investigation is continuing from this baseline.

[justinchuby]

Repair published: PR #484 (#484), commit 27b5f5975a60b31cb0517ff0156e1da1a93eb299, based directly on origin/main/9b07f09. Root causes fixed: stale diffusers fallback expectation omitted execution_provider="default"; pinned Ruff required formatting plus escaped ir\.Model regexes; and 42 canonical unit failures came from ORT GenAI tests inserting MagicMock components after ModelPackage began enforcing real onnx_ir.Model instances. The tests now use minimal real IR fixtures without changing production model behavior or tolerances. Validation: targeted repair tests 137 passed; canonical non-integration suite 3923 passed, 58 skipped; lintrunner -a clean; all tracked Python files pass pinned Ruff check and format. The exact all-files lintrunner formatter command was attempted but Ruff adapter expansion exceeds Windows command-line limits (WinError 206); equivalent batched pinned Ruff validation passed.

[justinchuby]

Option B first concrete finding (pinned source): unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF@f2d3fe3694501008786e81e5f20360cbf715496a, file NVIDIA-Nemotron-3.5-Lightning-30B-A3B-UD-Q4_K_M.gguf (LFS SHA-256 edcb5d4650796ed2fb412498de6f83b585862312c747ddb74f0ea04b22206181, 25,266,255,936 bytes). Its GGUF v3 header reports architecture nemotron_h_moe, 53 blocks, 417 tensors. Current Mobius cannot consume it directly: GGUF_ARCH_TO_MODEL_TYPE has no nemotron_h_moe entry and the tensor mapper has no Nemotron-H hybrid mapping. An alias alone would still be wrong because GGUF block 52 is the MTP block while the deployable HF backbone has 52 layers (0-51), and layer types must be inferred (23 Mamba, 22 MoE, 7 attention in the base backbone). Quantization is not preserved end-to-end for this Q4_K_M preset: its actual tensor types are 129 Q8_0, 24 Q5_0, 20 Q5_1, 243 F32, and 1 BF16; Mobius directly repacks Q8_0 but has no Q5_0/Q5_1 repacker, so the large expert tensors would require dequantization/requantization. I am now evaluating whether another pinned variant (notably MXFP4_MOE or Q8_0) can use the existing native-block path and what reusable architecture/shard guards are required.

[justinchuby]

Nemotron 3.5 BF16 trial progress: pinned nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 at d468880b6ad3c6e0d21377ce7242adaea4cc884d. Root cause found and fixed: Transformers 5.14.1 normalizes layer names to linear_attention/full_attention, while Mobius only recognized legacy mamba/attention and therefore misclassified Mamba layers. Added explicit legacy/current vocabulary tests and unknown-type rejection. The checkpoint index has 6,513 keys across 14 shards, including 270 auxiliary mtp.* tensors; preprocessing now drops only that prefix, matching NemotronHForCausalLM's documented unexpected-key policy, with alignment tests preserving all decoder parameters and real per-expert naming. Corrected stale synthetic parity setup (it had mapped mlp to moe and used epsilon 1e-6 instead of 1e-5): both reduced dense and MoE variants now pass L3 (2/2). Promoted the exact 3.5 checkpoint to the L2 target with pinned YAML revision; full 52-layer config parse/graph/shape validation passes 3/3 in 37.9s. YAML schema: 247/247 passed.

[justinchuby]

Correction to the first Option B finding: after parsing the pinned official BF16 safetensors headers (nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16@d468880b6ad3c6e0d21377ce7242adaea4cc884d), the deployable 52-layer backbone is 23 Mamba + 23 MoE + 6 attention, not 23/22/7. The GGUF's apparent 24 MoE + 7 attention counts include block 52, which is the combined attention+MoE MTP block. This strengthens the original conclusion: block_count=53 cannot be aliased directly to the 52-layer nemotron_h backbone; block 52 and its nextn.* tensors require explicit MTP handling or an intentional skip backed by semantic generation.

[justinchuby]

Option B variant/shard decision (pinned evidence): all 19 GGUF artifacts were header-inspected at unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF@f2d3fe3694501008786e81e5f20360cbf715496a and compared with all 14 pinned official BF16 safetensors headers at nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16@d468880b6ad3c6e0d21377ce7242adaea4cc884d.

• Q8_0 (35,004,643,392 bytes) is the only practical single-file variant whose 32.904B quantized parameters are all Q8_0, a qtype Mobius can directly repack. UD-Q6_K_XL has the same tensor-type inventory despite its preset name; UD-Q8_K_XL mixes Q8_0 with BF16. Qtype compatibility alone is therefore not the blocker for Q8.
• MXFP4_MOE contains 14.687B MXFP4 parameters that the native block path can preserve, but also 12.772B Q5_1 parameters; UD-Q4_K_M contains 15.326B Q5_0 + 12.772B Q5_1 parameters. Mobius has no 5-bit runtime/repacker, so both require a quantization-changing float round-trip for large expert weights and are not exact direct conversions.
• The BF16 GGUF is split into two shards (split.count=2, 329 + 88 tensors, 417 total). Current GGUFReader/builder consumes one file and does not assemble shards, so accepting either shard would emit incomplete weights. I am adding an explicit local and remote shard guard.
• Independent semantic blockers remain even for Q8: the 53rd GGUF block is the combined attention+MoE MTP auxiliary block; current NemotronH synthetic parity is explicitly waived because the Mamba2 recurrence diverges; and ORT/ORT GenAI generation has no successful real-weight evidence. The embedded GGUF tokenizer also declares padding_token_id=999 (<SPECIAL_999>), while the pinned official special_tokens_map.json declares <|im_end|> (ID 11) as padding. The GGUF metadata/card names the BF16 base model but records no immutable upstream source commit, so exact source-weight provenance cannot be proven from the artifact.

Decision: no end-to-end keep-quantized Mobius path currently meets semantic generation acceptance, including Q8. I will add a narrow fail-fast nemotron_h_moe architecture guard (before remote multi-GB download when Hub metadata is available), plus pinned recipe/waiver documentation. The accepted alternatives are llama.cpp/Unsloth for exact GGUF execution, or Option A (official pinned BF16 -> Mobius -> Olive) only after NemotronH L4/L5 parity and generation pass.

[justinchuby]

Nemotron 3.5 runtime/Olive evidence: the validation host is an 8 GB RTX A1000, so I range-fetched 236 MiB of real tensors from the pinned 65.8 GB checkpoint (complete Mamba layer 0, MoE layer 1 with four real experts plus shared expert/router, attention layer 5, sliced real embeddings/lm_head, final norm) and retained production hidden/head/expert dimensions. Saved-package reload parity: FP32 CPU max_abs=9.54e-6; FP16 CUDA=0.00977; BF16 CUDA=0.04688. All three generated identical cached IDs [12,13,12,12]. CUDA profiles: FP16 833 CUDA/14 CPU node events with CausalConvWithState + LinearAttention + GQA; BF16 1141/49 using standard-ONNX fallback because ORT 1.28 cannot assign BF16 CausalConvWithState. Fixed mixed-dtype NemotronH routing by explicitly computing gate/bias in fp32 and casting routing weights back to expert dtype; enabled the known grouped-RMSNorm CUDA workaround. ORT GenAI 0.15.2 is now rejected before artifacts because it cannot bind sparse global KV plus conv_state/ssm_state caches. Olive 0.13 Q4_K_M completed in 56.7s after isolating its CPU-only pass from unrelated TensorRT DLL auto-registration. Quantized package: 73,190,965 bytes vs 247,380,256 FP16 (0.296x), 15 MatMulNBits nodes, successful CUDA reload/profile, identical generated IDs. Full 30B semantic L4/L5 and full quantization remain evidence-based resource waivers (65.8 GB checkpoint, 8 GB GPU); executable recipe and reduced-real validator are committed in the example diff.

[justinchuby]

The standalone customer-facing Chinese prompt has been created as Issue #485. It incorporates the validated Option A BF16-to-Mobius-to-Olive workflow and Option B GGUF architecture/qtype/shard/provenance/tokenizer feasibility gates. Final model and guard PR links will be added when those workstreams finish.

[justinchuby]

PR #484 validation update: every available check is green, including all six unit-test matrix jobs, lint, CodeQL, architecture diff, benchmarks, and L1. The only pending check is Integration (fast), queued on the self-hosted A10 pool. Repository-wide inspection shows older main and unrelated PR GPU jobs are also queued, so this is shared runner capacity rather than a branch failure; rerunning would only lose queue position.