Repository navigation
feat(export): write a merged qpeft model as a GPTQ checkpoint - #3
Merged
Merged
Conversation
qpeft.export.export_gptq(model, dir, checkpoint_format="gptq") writes the model's own save_pretrained files with GPTQ tensors for every QuantLinear (qweight as is, packed integer qzeros, fp16 scales (n_groups, out), g_idx) and a quantization_config in config.json, so transformers loads it with from_pretrained through optimum and gptqmodel. The export is exact or it refuses before writing anything: - fractional zero-points (a QA-LoRA fold, decision D2) - scales that fp16 does not hold exactly (an fp32 model) - zero-point 0 in the v1 format, which stores z - 1 and which loaders undo in different ways (gptq_v2 stores z and takes every zero-point) - quantized layers outside the decoder blocks, different targets per block, a suffix match that would make the loader replace an unquantized linear, bits other than 2/3/4/8, non-HF models Tests: the GPTQ dequant of the written file equals the merged layers bit for bit (PEQA and EfficientQAT, bits 2/3/4/8, both formats), every refusal, and a real load through transformers + gptqmodel on CPU (tests/test_gptq_load.py, skipped without gptqmodel).
…uter, reword README
…er keeps in float
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
qpeft step 5 of the axolotl integration plan (captain chose 'path A'): a GPTQ export for the merged qpeft model, so PEFT-cheap training yields a merged int2/3/4 model that is exact AND loadable by standard GPTQ tooling (axolotl docs say it does not support GPTQ export; qpeft fills that gap). Requirements from the brief: qpeft/export/gptq.py as an Ousterhout deep module with one small public entry point (export_gptq(model, save_directory, checkpoint_format='gptq')), all layout details hidden inside; qweight already in GPTQ layout; scales transposed to (n_groups, out) fp16; packed integer qzeros respecting the target loader's zero-point convention (the classic v1 'zeros - 1' offset, verified in gptqmodel source, not assumed); g_idx; quantization_config in config.json (quant_method gptq, bits, group_size, sym/desc_act true for the data). Invariant 2 refuse-instead-of-approximate: fractional zero-points (QA-LoRA merge, captain decision D2) are refused with a clear message; anything that cannot round-trip exactly is refused (a bf16->fp16 scale cast is checked by value; fp32 models are refused because fp16 scales would round). Decisions I made: (1) default checkpoint_format 'gptq' (v1, what tooling reads by default) refuses zero-point 0, because gptqmodel undoes the z-1 offset word-wise with carry for 2/4/8 bits and field-wise mod 2bits for 3 bits, so z==0 is not portable; 'gptq_v2' stores z unchanged and is exact for every z but only gptqmodel reads it - this optional keyword is deliberate. (2) sym is written true only when every zero-point equals 2(bits-1). (3) The export refuses quantized layers outside the decoder blocks (e.g. lm_head with target_modules=None), different targets per block, and suffix matches that would make optimum's plain str.endswith rule replace an unquantized linear; for a subset of block linears it writes modules_in_block_to_quantize. (4) The config.json quantization_config is added after save_pretrained(state_dict=...) instead of mutating the user's model config; export does not mutate the model. Tests (invariants 1 and 6, same step): tests/test_gptq_export.py decodes the written safetensors with an independent bit-stream decoder and checks GPTQ-formula dequant == merged QuantLinear weight and forward bit for bit for PEQA and EfficientQAT, bits 2/3/4/8, both formats, plus every refusal; tests/test_gptq_load.py loads the export with real transformers + optimum + gptqmodel on macOS CPU (skipped when gptqmodel/optimum are absent, by design, because axolotl's optimum pin breaks optimum.gptq so it must run in a separate venv without axolotl, report A2): per-layer dequantized weight bitwise equal, fp16 logits argmax 100% and within fp16 rounding (oracle: the merged model upcast to fp32; loaded error <= 2x qpeft's own half-precision error). Out of scope: do NOT touch qpeft/integrations/axolotl/plugin.py (a parallel task owns it); wiring 'export:' into the plugin is step 6. Findings documented in README/AGENTS.md: vLLM main accepts GPTQ only sym=True at 4/8 bits, so asymmetric qpeft exports are refused there; gptqmodel's default 4-bit CPU kernel TorchAtenLinear rounds scales to bf16 (backend='torch' is exact). All local, no GitHub Actions.
What Changed
Adds
qpeft.export.export_gptq(model, save_directory, checkpoint_format="gptq"). It writes a merged EfficientQAT or PEQA model as a Hugging Face GPTQ checkpoint. The checkpoint hasqweightin GPTQ layout, fp16 scales transposed to(n_groups, out), packedqzeros,g_idx, and aquantization_configadded toconfig.jsonaftersave_pretrained."gptq"(v1) storesz - 1and refuses a zero-point of 0."gptq_v2"storeszas it is.symistrueonly when every zero-point equals2**(bits-1). The export does not change the model.Before it writes a file, the export checks every layer and raises
UnsupportedSchemeErrorfor anything that would not round-trip exactly:nn.Linearsubclasses that the GPTQ loader keeps in floatWhen only some of a block's linears are quantized, the export writes
modules_in_block_to_quantize.Adds
tests/test_gptq_export.py. It decodes the written safetensors with an independent decoder and checks that the dequantized weights and forward match the merged model bit for bit (PEQA and EfficientQAT, bits 2/3/4/8, both formats). It also tests every refusal.Adds
tests/test_gptq_load.py. It does a real load through transformers + optimum + gptqmodel on CPU and is skipped when those are not installed.Documents the export and the loader findings in
README.mdandAGENTS.md:sym=Trueat 4/8 bitsRisk Assessment
✅ Low: The export is a new, self-contained module. The round-2 fix does what the user chose (option b): it builds a meta-device skeleton from a deep copy of root.config, uses that skeleton as the single loader simulation, and refuses quantized layers that are not exact nn.Linear before it writes anything. A test that fails without the fix covers this. I traced the full-block, subset, suffix-collision, router-subclass and outside-block paths, and none gives a wrong result without an error.
Testing
In a throwaway venv with real optimum + gptqmodel, I ran the two GPTQ test files (53 passed, none skipped). Then I drove the export as an end user on the cached real Qwen3-129m model. In fp16 with the torch kernel, the loader's per-layer weights matched qpeft's bit for bit and greedy text was the same for PEQA 4-bit v1, EfficientQAT 3-bit v2 and a 2-bit q/v subset. The bf16 default kernel stays as close to the exact model as qpeft itself (the documented bf16 rounding). Every refusal raised a clear UnsupportedSchemeError and wrote nothing: QA-LoRA, fp32, lm_head, per-block difference, quantized router subclass, unmerged, unknown format, and v1 zero-point 0. A v2 export with zero-point 0 loaded exactly, and the export left the model unchanged. No UI surface, so the evidence is CLI transcripts.
pytest tests/test_gptq_export.py::test_refuses_when_the_loader_suffix_match_would_hit_an_unquantized_linear(part of the 53 passed)Evidence: Targeted GPTQ pytest run (export + real gptqmodel load)
Evidence: Real-model export -> transformers load -> generate transcript
Evidence: Refusal / adversarial scenarios transcript
Evidence: Driver scripts
Evidence: Refusal driver script
Evidence: Key result lines
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 2 issues found → auto-fixed (2) ✅
qpeft/export/gptq.py:138-_loader_replacesis meant to mirror the layer types that optimum'sget_layersfinds (its docstring says so). But it usestype(module) is nn.Linear, while optimum'sget_layers(default[Conv1D, nn.Conv2d, nn.Linear]) usesisinstance, and qpeft's own_is_target(tuners_utils.py:254) also usesisinstance(module, nn.Linear). Concrete case: a decoder block holds an unquantized subclass of nn.Linear, and the user quantizes every plain nn.Linear in the block with a target list. Thenquantized == set(in_blocks)at gptq.py:123, so the export writes nomodules_in_block_to_quantize. optimum replaces the subclass too, and the checkpoint has no qweight/qzeros/scales for it, so the loader fills that layer with uninitialized GPTQ tensors. The subset path at gptq.py:127 has the same gap: a suffix collision with such a subclass is not detected. Fix: useisinstance(module, (QuantLinear, nn.Linear, nn.Conv2d))plus the Conv1D name check. Then the predicate matches optimum'sisinstancesemantics, and both theNonepath and the suffix-collision check at gptq.py:127-131 see the same set that the loader replaces.README.md:357- README says: 'The export is exact: the weight a GPTQ loader decodes equals the merged model's weight bit for bit.' For a bf16 model this is not literally true. tests/test_gptq_load.py:100 compares the loader's fp16 dequant with qpeft's dequant onscale.half(), not with the merged bf16 weight (_weight_in(bfloat16)). The torch kernel computes bf16 models in fp16 (see the test comment at line 103). The default 4-bit CPU kernel also rounds scales to bf16. The written tensors are exact (tests/test_gptq_export.py proves this with the GPTQ formula in the model's dtype). The README sentence should say what is exact: the stored codes, scale and zero-point, and the GPTQ dequant in the model's own dtype. The loader's result then depends on its compute dtype and kernel.🔧 Fix applied.
2 warnings still open:
qpeft/export/gptq.py:138-_loader_replacesis meant to mirror the layer types that optimum'sget_layersfinds (its docstring says so). But it usestype(module) is nn.Linear, while optimum'sget_layers(default[Conv1D, nn.Conv2d, nn.Linear]) usesisinstance, and qpeft's own_is_target(tuners_utils.py:254) also usesisinstance(module, nn.Linear). Concrete case: a decoder block holds an unquantized subclass of nn.Linear, and the user quantizes every plain nn.Linear in the block with a target list. Thenquantized == set(in_blocks)at gptq.py:123, so the export writes nomodules_in_block_to_quantize. optimum replaces the subclass too, and the checkpoint has no qweight/qzeros/scales for it, so the loader fills that layer with uninitialized GPTQ tensors. The subset path at gptq.py:127 has the same gap: a suffix collision with such a subclass is not detected. Fix: useisinstance(module, (QuantLinear, nn.Linear, nn.Conv2d))plus the Conv1D name check. Then the predicate matches optimum'sisinstancesemantics, and both theNonepath and the suffix-collision check at gptq.py:127-131 see the same set that the loader replaces.qpeft/export/gptq.py:140-_loader_replacescounts every QuantLinear as a layer the loader replaces. Its docstring says why: "a QuantLinear was an nn.Linear". That assumption is not always true. qpeft's_is_target(qpeft/tuners/tuners_utils.py:251) wraps anyisinstance(module, nn.Linear), so subclasses get wrapped too. Round 1 confirmed that optimum'sget_layerstakes nn.Linear by exact type. The fix round documented that rule for unquantized layers but left this quantized sibling behind.Concrete sequence: take the round-1 test model (each block holds
router = _Router(128, 4), a subclass of nn.Linear). Callget_quant_model(base, PEQAConfig(bits=4, group_size=64, target_modules=BLOCK_LINEARS + ["router"])), thenmerge_and_unload(), thenexport_gptq(...). Every in-block layer is now a QuantLinear, soquantized == set(in_blocks)at gptq.py:123. The export succeeds and writesrouter.qweight/qzeros/scales/g_idxwith nomodules_in_block_to_quantize. At load time, optimum rebuilds the model from config. It sees_Router, skips it because of the exact-type rule, and keeps it as a float layer. The checkpoint has norouter.weight, so transformers random-initializes the router and only warns about missing and unexpected keys. The result is a silently wrong model, which breaks invariant 2 (refuse, do not approximate).The export cannot detect this today: a QuantLinear does not record the class it replaced. Two possible fixes: (a) record the original type in QuantLinear, or (b) in export, build the model from
root.configon the meta device and refuse any quantized name whose fresh module is not exactlynn.Linear. Option (a) changes qpeft/tuners, which the round-1 human decision put off-limits. Option (b) adds new machinery to the export. Because the fix needs that authorization (the defect itself is mechanical), this is ask-user.No other site: the unquantized path at gptq.py:121-131 already uses the exact-type rule correctly.
🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
pytest tests/test_gptq_export.py::test_refuses_when_the_loader_suffix_match_would_hit_an_unquantized_linear(part of the 53 passed)pytest -p no:cacheprovider tests/test_gptq_export.py tests/test_gptq_load.pyin a throwaway venv (torch 2.14.1, transformers 5.19.0, optimum, gptqmodel 7.5.0): 53 passed, 0 skippedpython live_real_model.py: PEQA 4-bit fp16 v1 / PEQA 4-bit bf16 default kernel / EfficientQAT (Block-AP + E2E-QP) 3-bit fp16 gptq_v2 / PEQA 2-bit q_proj+v_proj subset on cached axolotl-ai-co/tiny-qwen3-129m; export_gptq + tokenizer, loaded with AutoModelForCausalLM.from_pretrained + AutoTokenizer, per-layer dequant compare, greedy generation comparepython live_refusals.py: QA-LoRA, fp32, lm_head, per-block difference, quantized nn.Linear subclass router, unmerged, unknown format, zero-point 0 in v1 (refused) vs v2 (loaded through transformers, bit for bit), router left unquantized exports, export does not mutate the model✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.