Repository navigation
Conversation
export_compressed_tensors writes the asymmetric pack-quantized WNA16 layout (weight_packed, weight_scale, weight_zero_point, weight_shape) that vLLM and transformers + compressed-tensors load, with one config group per grid that targets the quantized layers by full name. The scale keeps the model's dtype, so nothing is cast; fractional zero-points, linear subclasses and tied weights are refused. The merged-layer lookup, integer zero-points, loader skeleton and checkpoint writing move from gptq.py into export/_checkpoint.py, shared by both exporters. The export test models move into tests/export_models.py and are built in their dtype like from_pretrained builds them (fp32 rotary inv_freq), so a loaded export matches them bit for bit.
…he checkpoint Loaders make weight_scale in the dtype that save_pretrained writes to config.json, so a scale in another dtype would be cast on load.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Add a compressed-tensors "pack-quantized" WNA16 asymmetric export (with zero-points, 2 to 8 bits) so vLLM can serve a merged qpeft model. Our GPTQ export cannot be served by vLLM, because vLLM's GPTQ path takes only symmetric 4/8 bit. Symmetric mode is deliberately NOT offered: it would throw away the trained zero-points, which are the point of qpeft.
Requirements from the user:
What Changed
qpeft.export.export_compressed_tensors(model, save_directory)writes a merged EfficientQAT or PEQA model as an asymmetricpack-quantizedWNA16 checkpoint for 2 to 8 bits.Per layer it writes
weight_packed(qweight.t()),weight_scale(kept in the model dtype),weight_zero_point(packed along the outputs) andweight_shape(int64).config.jsongets one config group per (bits, group_size), and each group names its layers in full.The export refuses fractional zero-points (QA-LoRA), scales in a different dtype than the checkpoint, unsupported bits, and quantized layers that the loader's config-built model does not hold as a plain, untied
nn.Linear(for example an MoE router or a tiedlm_head).qpeft/export/_checkpoint.py: finding merged layers, the integer zero-point check, the meta-device loader skeleton, and thesave_pretrained+quantization_configwrite.export_gptqnow uses it, and its behavior does not change, except that it now checks that every layer is merged before its other checks.tests/test_gptq_export.pytotests/export_models.py, and the builders now use_from_config(dtype=...).New
tests/test_compressed_tensors_export.pycovers the bit-exact decode and every refusal.New
tests/test_compressed_tensors_load.pydoes a real CPU load through transformers + compressed-tensors (skipped without compressed-tensors >= 0.18).README and AGENTS.md now document the format, the loader facts and the vLLM 0.29 results.
Risk Assessment
ignorelist may stop vLLM from serving partial-target or MoE exports (a loud failure, not a silent one).Testing
In a throwaway venv (compressed-tensors 0.19, transformers 5.19, gptqmodel + optimum), I ran the targeted export and load tests: compressed-tensors export, compressed-tensors load, GPTQ export, and the real GPTQ load that covers the refactor. All passed. I then ran the export as an end user would on the real SmolLM2-135M model. PEQA and EfficientQAT exports at bits 2, 3, 4, 5, 7 and 8, in bf16 and fp16, plus a partial-target export, reload through transformers with logits equal bit for bit, and greedy text is the same. All five refusal guards fire and write no files. For vLLM I used disposable docker containers. The arm64 CPU build of vLLM 0.31 accepted the 4-bit export and loaded every weight with its CompressedTensorsWNA16 kernel, but it cannot run the forward:
cpu_gemm_wna16is not compiled for arm64, and the CPU path supports only 4-bit. The x86_64 build hits SIGILL under emulation (no AVX-512). So vLLM generation itself stays untested here. I removed the disposable venv contents, the pulled images and the worktree build files. This is not a UI change, so there are no screenshots; the evidence is CLI transcripts and logs.pytest tests/test_compressed_tensors_load.pyin pytest-export-load.txtpytest tests/test_gptq_load.py(pytest-gptq-load.txt, 18 passed) andpytest tests/test_gptq_export.pyEvidence: Targeted pytest run: compressed-tensors export/load + GPTQ export
Evidence: Real GPTQ load through gptqmodel + optimum after the refactor
Evidence: SmolLM2-135M export -> transformers reload transcript (4-bit PEQA bf16, 3-bit EfficientQAT fp16, 8-bit PEQA bf16)
bits=4 peqa bf16: weight_packed (576,192) int32, weight_scale (576,24) bf16, weight_zero_point (72,24) int32, weight_shape int64; logits bit-for-bit equal: True; greedy 'The capital of France is Paris.\n\nThe French language is a very rich language' equal: True bits=3 efficient_qat fp16: logits bit-for-bit equal: True; greedy equal: True bits=8 peqa bf16: logits bit-for-bit equal: True; greedy equal: TrueEvidence: SmolLM2-135M 2/5/7-bit and partial-target exports reload bit for bit
Evidence: Refusal transcript on SmolLM2-135M (tied lm_head, QA-LoRA, unmerged, fp32 scale, bits=1): all refused, no dir written
Evidence: vLLM 0.31 arm64 CPU: 4-bit export accepted, CPUWNA16 kernel picked, weights loaded, forward fails on the missing cpu_gemm_wna16 op
~/.no-mistakes/evidence/01M4R72T4XAWXNMKNFKK2NZAED/vllm-0.31-cpu-arm64-8bit.log)Evidence: vLLM 0.31 x86_64 CPU under emulation: SIGILL (AVX-512 required)
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
qpeft/export/compressed_tensors.py:135- The export always writes"ignore": []. vLLM's CompressedTensorsConfig.get_scheme calls find_matched_target for every LinearBase layer that is not inignore, and that function raisesValueError: Unable to find matching target for <layer> in the compressed-tensors configwhen no target matches [unconfirmed: from memory of vLLM source, not checked in this environment; this is also the known failure for MoEmlp.gatelayers that are missing fromignore]. Concrete path: exportmerged_peqa(4, targets=["q_proj", "v_proj"]), or any MoE model whose router this exporter requires to stay unquantized (a ReplicatedLinear in vLLM), then callvllm.LLM(path). The unquantized o_proj / gate_up_proj / router is a LinearBase with no matching target, so vLLM refuses to load. The vLLM validation in README/AGENTS.md used all block linears on dense Qwen3-0.6B, so it did not cover this case. The failure is loud, not a silent approximation, but it blocks the stated goal ("so vLLM can serve a merged qpeft model") for every partial-target or MoE export. Likely fix: build the skeleton once and list, in full-name form, every plain nn.Linear that is not quantized inignore(llm-compressor checkpoints always list such layers, for examplelm_head). This changes the written quantization_config, so the author should confirm it against vLLM source and with a partial-target vLLM run before making the change. The sameignorelist is also read by transformers + compressed-tensors (tests/test_compressed_tensors_load.py), and that path must stay bitwise-equal.✅ **Test** - passed
✅ No issues found.
pytest tests/test_compressed_tensors_load.pyin pytest-export-load.txtpytest tests/test_gptq_load.py(pytest-gptq-load.txt, 18 passed) andpytest tests/test_gptq_export.pypytest tests/test_compressed_tensors_export.py tests/test_compressed_tensors_load.py tests/test_gptq_export.py(102 passed, compressed-tensors 0.19, transformers 5.19, CPU)pytest tests/test_gptq_load.pywith gptqmodel + optimum installed (18 passed): the GPTQ refactor still loads bit for bitReal-model flow on HuggingFaceTB/SmolLM2-135M: PEQA 4-bit bf16, EfficientQAT (Block-AP + E2E-QP) 3-bit fp16, PEQA 8-bit bf16. Each run trains, calls merge_and_unload and export_compressed_tensors, saves the tokenizer, then reloads with AutoModelForCausalLM. It compares logits and greedy generations and inspects config.json and tensor dtypes/shapesReal-model PEQA exports at 2-bit bf16, 5-bit fp16 and 7-bit bf16, plus a partial-target (q_proj, v_proj) export, each reloaded through transformers with a bit-for-bit logits checkRefusal checks on SmolLM2-135M: tied lm_head (targets=None), a QA-LoRA fold with fractional zero-points, an unmerged model, an fp32 scale in a bf16 model, and bits=1. Each one must refuse and write no output dirvLLM 0.31vllm/vllm-openai-cpu:latest-arm64in a throwaway docker container:LLM(export, dtype=...)on the 4-bit, 8-bit and 3-bit exportsvLLM 0.31vllm/vllm-openai-cpu:latest-x86_64under amd64 emulation on the 4-bit export✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.