diff --git a/README-ZH.md b/README-ZH.md index 9106c4153..293d62a9f 100644 --- a/README-ZH.md +++ b/README-ZH.md @@ -17,6 +17,7 @@ mllm ## 最新动态 +- [2026 年 9 月 8 日] MLLM 现已支持 **[MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B)**！在手机上通过 ARM CPU 本地运行。[快速上手](./examples/minicpm5/README.md)。 - [2026 年 6 月 8 日] `pymllm` 已覆盖 Qwen3、Qwen3-VL 与 Qwen3.5 在 Jetson Orin 上的 W4A16 / W8A8 serving；Qwen3-VL-2B W8A8 在 AGX Orin 32GB 上最高达到 3.12x prefill 加速比，decode 吞吐整体与 llama.cpp 接近。 - [2026 年 4 月 30 日] `pymllm` 新增面向 Jetson 的 Qwen3 / Qwen3-VL BF16、W4A16 和 W8A8 serving 支持，覆盖 compressed-tensors AWQ 与 W8A8 INT8 路径。 - [2026 年 3 月 18 日] 🔥🔥🔥 `pymllm` 已支持在 Jetson Orin 和 Jetson Thor 设备上使用 CUDA（实验特性，仍在持续开发中）。 @@ -105,6 +106,8 @@ mllm 框架可以与主流社区框架的模型检查点无缝集成。通过 ml | [Qwen3-0.6B](https://github.com/QwenLM/Qwen3) | [✔️ w4a8](https://www.modelscope.cn/models/mllmTeam/Qwen3-0.6B-w4a32kai) | | | | [Qwen3-1.7B](https://github.com/QwenLM/Qwen3) | [✔️ w4a8](https://www.modelscope.cn/models/mllmTeam/Qwen3-1.7B-w4a8-i8mm-kai) | | [W4A16-SM8650](https://modelscope.cn/models/mllmTeam/Qwen3-1.7B-Qnn-AOT-SM8650/summary) | | [Qwen3-4B](https://github.com/QwenLM/Qwen3) | [✔️ w4a8](https://www.modelscope.cn/models/mllmTeam/Qwen3-4B-w4a8-i8mm-kai) | | | +| [MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) | [✔️ w4a8](./examples/minicpm5/README.md) | | | +| [MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) | [✔️ w4a8](./examples/minicpm5/README.md) | | | | Qwen3.5-2B | | ✔️ W4A16 / W8A8 | | | Qwen3.5-4B | | ✔️ W4A16 / W8A8 | | | Qwen3-VL-2B-Instruct | | ✔️ W4A16 / W8A8 | | diff --git a/README.md b/README.md index a511778c8..34893a80f 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,7 @@ mllm ## Latest News +- [2026 Sep 08] MLLM now supports **[MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B)**! Run it locally on your phone with ARM CPU inference. [Get started](./examples/minicpm5/README.md). - [2026 Jun 08] 🔥🔥🔥`pymllm` now covers Qwen3, Qwen3-VL, and Qwen3.5 on Jetson Orin with W4A16 / W8A8 serving; Qwen3-VL-2B W8A8 reaches up to 3.12x prefill speedup on AGX Orin 32GB, while decode throughput stays broadly close to llama.cpp. - [2026 May 02] 🔥🔥🔥 MLLM now supports the Ascend NPU backend, with ATB graph execution and Qwen3 W8A8 inference on Ascend devices. - [2026 Apr 30] 🔥🔥🔥`pymllm` adds Jetson-oriented Qwen3 / Qwen3-VL BF16, W4A16, and W8A8 serving support, including compressed-tensors AWQ and W8A8 INT8 paths. @@ -107,6 +108,7 @@ The mllm framework integrates seamlessly with popular community frameworks' chec | [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) | [✔️ w4a8](./examples/qwen3_5/README.md) | | | | [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) | [✔️ w4a8](./examples/qwen3_5/README.md) | | | | [MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) | [✔️ w4a8](./examples/minicpm5/README.md) | | | +| [MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) | [✔️ w4a8](./examples/minicpm5/README.md) | | | | [LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B) | [✔️ w4a8](./examples/lfm2/README.md) | | | | [DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) | [✔️ w4a8](https://www.modelscope.cn/models/mllmTeam/DeepSeek-OCR-w4a8-i8mm-kai) | | | | [SmolLM3](https://huggingface.co/blog/smollm3)| [✔️ w4a8](https://www.modelscope.cn/models/mllmTeam/SmolLM3-3B-w4a8-i8mm-kai) | | | diff --git a/examples/minicpm5/README.md b/examples/minicpm5/README.md index b1150a9a9..3ae104a05 100644 --- a/examples/minicpm5/README.md +++ b/examples/minicpm5/README.md @@ -1,6 +1,7 @@ -# MiniCPM5-1B on ARM CPU +# MiniCPM5 on ARM CPU -This example supports the text-only [`openbmb/MiniCPM5-1B`](https://huggingface.co/openbmb/MiniCPM5-1B) checkpoint. +This example supports the text-only [`openbmb/MiniCPM5-1B`](https://huggingface.co/openbmb/MiniCPM5-1B) and +[`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B/tree/0e9c66dce9fedde5ba8663bbcdd54b6810bb929a) checkpoints. The runtime contract is batch 1, a 2048-token mobile cache, 16 query heads, 2 native KV heads, and explicit `head_dim=128`. The implementation keeps KV history at KV-head count and uses a correctness-first eager GQA path; it does not materialize a persistent 16-head KV cache. @@ -8,6 +9,18 @@ it does not materialize a persistent 16-head KV cache. The runner implements the official no-tool chat-template branch, including optional system text and `enable_thinking=true|false`. Tool schemas, tool calls, and multi-turn history are outside this first product surface. +| Variant | Layers | Hidden / FFN width | Runtime config | Quantization config | +| --- | ---: | --- | --- | --- | +| 1B | 24 | 1536 / 4608 | `config_1B_w4a32_kai.json` | `quant_cfg_1B_w4a32_kai.json` | +| 2B | 42 | 2048 / 6144 | `config_2B_w4a32_kai.json` | `quant_cfg_2B_w4a32_kai.json` | + +Both variants reuse the same model graph, native KV cache and GQA kernels. At the supplied 2048-token +capacity, FP32 K/V payloads occupy 96 MiB for 1B and 168 MiB for 2B. The official 131072-position +RoPE limit does not enlarge the runner's supported mobile cache. +Use the matching checkpoint, runtime config and quantization config together; mixed 1B/2B geometry is rejected. +The 2B tokenizer vocabulary is unchanged. Its updated history template does not change the supported +single-user branch; multi-turn history and tool calling remain unsupported. + ## Convert Keep embeddings and norms in FP32; the supplied configuration packs transformer Linear weights and the independent @@ -26,6 +39,10 @@ python -m pymllm.mobile.utils.mllm_convertor \ ## Run +For 2B, use `MiniCPM5-2B` and `minicpm5-2b-w4a32-kai.mllm` in these commands and select +`quant_cfg_2B_w4a32_kai.json` during conversion and `config_2B_w4a32_kai.json` during inference. +The same `demo_prompt_200.txt` is used for both sizes. + ```bash mllm-minicpm5-runner \ --model_path /path/to/minicpm5-1b-w4a32-kai.mllm \ diff --git a/examples/minicpm5/config_2B_w4a32_kai.json b/examples/minicpm5/config_2B_w4a32_kai.json new file mode 100644 index 000000000..6cc3dfee5 --- /dev/null +++ b/examples/minicpm5/config_2B_w4a32_kai.json @@ -0,0 +1,28 @@ +{ + "architectures": [ + "LlamaForCausalLM" + ], + "model_type": "llama", + "bos_token_id": 0, + "eos_token_id": [ + 1, + 130073 + ], + "pad_token_id": 1, + "hidden_act": "silu", + "hidden_size": 2048, + "intermediate_size": 6144, + "max_position_embeddings": 131072, + "num_attention_heads": 16, + "num_hidden_layers": 42, + "num_key_value_heads": 2, + "head_dim": 128, + "rms_norm_eps": 1e-06, + "rope_theta": 5000000, + "rope_scaling": null, + "tie_word_embeddings": false, + "use_cache": true, + "vocab_size": 130560, + "max_cache_length": 2048, + "linear_impl_type": "KaiLinear_f32_qai8dxp_qsi4c32p_mxk_nxk_qai8dxp1x8_qsi4c32p8x8_1x8x32" +} diff --git a/examples/minicpm5/main.cpp b/examples/minicpm5/main.cpp index cf498f9dc..0b84f003a 100644 --- a/examples/minicpm5/main.cpp +++ b/examples/minicpm5/main.cpp @@ -216,7 +216,7 @@ MLLM_MAIN({ } if (exit_code == 0) { fmt::print("Benchmark records: {}\n", benchmark_jsonl.get()); } } else { - fmt::print("\n{:*^60}\n", prompt.isSet() ? " MiniCPM5-1B One-shot CLI " : " MiniCPM5-1B Interactive CLI "); + fmt::print("\n{:*^60}\n", prompt.isSet() ? " MiniCPM5 One-shot CLI " : " MiniCPM5 Interactive CLI "); if (!prompt.isSet()) fmt::print("Enter 'exit' or 'quit' to end the session\n\n"); while (true) { diff --git a/examples/minicpm5/quant_cfg_2B_w4a32_kai.json b/examples/minicpm5/quant_cfg_2B_w4a32_kai.json new file mode 100644 index 000000000..d9bfd6522 --- /dev/null +++ b/examples/minicpm5/quant_cfg_2B_w4a32_kai.json @@ -0,0 +1,62 @@ +{ + "^model\\.layers\\.\\d+\\.self_attn\\.q_proj\\.(bias|weight)": { + "hints": { + "quant_method": "kai", + "kai_matmul_triplet": "f32_qai8dxp_qsi4c32p", + "kai_matmul_layout": "mxk_nxk", + "kai_matmul_tile_cfg": "qai8dxp1x8_qsi4c32p8x8_1x8x32", + "shape": [2048, 2048], + "replace": true + } + }, + "^model\\.layers\\.\\d+\\.self_attn\\.(k_proj|v_proj)\\.(bias|weight)": { + "hints": { + "quant_method": "kai", + "kai_matmul_triplet": "f32_qai8dxp_qsi4c32p", + "kai_matmul_layout": "mxk_nxk", + "kai_matmul_tile_cfg": "qai8dxp1x8_qsi4c32p8x8_1x8x32", + "shape": [256, 2048], + "replace": true + } + }, + "^model\\.layers\\.\\d+\\.self_attn\\.o_proj\\.(bias|weight)": { + "hints": { + "quant_method": "kai", + "kai_matmul_triplet": "f32_qai8dxp_qsi4c32p", + "kai_matmul_layout": "mxk_nxk", + "kai_matmul_tile_cfg": "qai8dxp1x8_qsi4c32p8x8_1x8x32", + "shape": [2048, 2048], + "replace": true + } + }, + "^model\\.layers\\.\\d+\\.mlp\\.(gate_proj|up_proj)\\.(bias|weight)": { + "hints": { + "quant_method": "kai", + "kai_matmul_triplet": "f32_qai8dxp_qsi4c32p", + "kai_matmul_layout": "mxk_nxk", + "kai_matmul_tile_cfg": "qai8dxp1x8_qsi4c32p8x8_1x8x32", + "shape": [6144, 2048], + "replace": true + } + }, + "^model\\.layers\\.\\d+\\.mlp\\.down_proj\\.(bias|weight)": { + "hints": { + "quant_method": "kai", + "kai_matmul_triplet": "f32_qai8dxp_qsi4c32p", + "kai_matmul_layout": "mxk_nxk", + "kai_matmul_tile_cfg": "qai8dxp1x8_qsi4c32p8x8_1x8x32", + "shape": [2048, 6144], + "replace": true + } + }, + "^lm_head\\.(bias|weight)": { + "hints": { + "quant_method": "kai", + "kai_matmul_triplet": "f32_qai8dxp_qsi4c32p", + "kai_matmul_layout": "mxk_nxk", + "kai_matmul_tile_cfg": "qai8dxp1x8_qsi4c32p8x8_1x8x32", + "shape": [130560, 2048], + "replace": true + } + } +} diff --git a/mllm/models/minicpm5/configuration_minicpm5.hpp b/mllm/models/minicpm5/configuration_minicpm5.hpp index 3a5d782aa..138db7a17 100644 --- a/mllm/models/minicpm5/configuration_minicpm5.hpp +++ b/mllm/models/minicpm5/configuration_minicpm5.hpp @@ -95,23 +95,29 @@ struct MiniCPM5Config : protected ConfigFile { aops::LinearImplTypes linear_impl_type = aops::LinearImplTypes::kDefault; }; -inline auto matchesOfficialMiniCPM5_1BRuntimeContract(const MiniCPM5Config& config) -> bool { +inline auto matchesOfficialMiniCPM5RuntimeContract(const MiniCPM5Config& config) -> bool { constexpr auto kKaiLinearImpl = aops::LinearImplTypes::kKaiLinear_f32_qai8dxp_qsi4c32p_mxk_nxk_qai8dxp1x8_qsi4c32p8x8_1x8x32; - return config.vocab_size == 130560 && config.hidden_size == 1536 && config.intermediate_size == 4608 - && config.num_hidden_layers == 24 && config.num_attention_heads == 16 && config.num_key_value_heads == 2 - && config.head_dim == 128 && config.hidden_act == "silu" && config.max_position_embeddings == 131072 - && config.rms_norm_eps == 1.0e-6F && config.rope_theta == 5000000.0F && !config.attention_bias - && !config.tie_word_embeddings && config.bos_token_id == 0 && config.pad_token_id == 1 + const bool supported_dimensions = + (config.hidden_size == 1536 && config.intermediate_size == 4608 && config.num_hidden_layers == 24) + || (config.hidden_size == 2048 && config.intermediate_size == 6144 && config.num_hidden_layers == 42); + return supported_dimensions && config.vocab_size == 130560 && config.num_attention_heads == 16 + && config.num_key_value_heads == 2 && config.head_dim == 128 && config.hidden_act == "silu" + && config.max_position_embeddings == 131072 && config.rms_norm_eps == 1.0e-6F && config.rope_theta == 5000000.0F + && !config.attention_bias && !config.tie_word_embeddings && config.bos_token_id == 0 && config.pad_token_id == 1 && config.eos_token_ids == std::vector({1, 130073}) && config.max_cache_length == 2048 && config.linear_impl_type == kKaiLinearImpl; } +inline auto matchesOfficialMiniCPM5_1BRuntimeContract(const MiniCPM5Config& config) -> bool { + return matchesOfficialMiniCPM5RuntimeContract(config) && config.hidden_size == 1536; +} + inline void validateModelConfigMatch(const MiniCPM5Config& config, const ParameterFile::ptr_t& parameter_file) { constexpr auto kEmbeddingWeight = "model.embed_tokens.weight"; constexpr auto kLmHeadWeight = "lm_head.weight"; - if (!matchesOfficialMiniCPM5_1BRuntimeContract(config)) { + if (!matchesOfficialMiniCPM5RuntimeContract(config)) { throw std::invalid_argument( - "MiniCPM5 model/config mismatch: CPU runner supports only the official MiniCPM5-1B runtime contract"); + "MiniCPM5 model/config mismatch: CPU runner supports only the official MiniCPM5-1B and MiniCPM5-2B runtime contracts"); } if (parameter_file == nullptr || !parameter_file->has(kEmbeddingWeight) || !parameter_file->has(kLmHeadWeight)) { throw std::invalid_argument("MiniCPM5 model/config mismatch: checkpoint requires model.embed_tokens.weight and " @@ -132,7 +138,7 @@ inline void validateModelConfigMatch(const MiniCPM5Config& config, const Paramet const auto shape = embedding.shape(); if (shape.size() != 2 || shape[0] != config.vocab_size || shape[1] != config.hidden_size) { - throw std::invalid_argument("MiniCPM5 model/config mismatch: embedding shape must be [130560, 1536]"); + throw std::invalid_argument("MiniCPM5 model/config mismatch: embedding shape must match [vocab_size, hidden_size]"); } } diff --git a/tests/models/minicpm5/MiniCPM5ConfigTest.cpp b/tests/models/minicpm5/MiniCPM5ConfigTest.cpp index ae95d7ee2..f2ee20890 100644 --- a/tests/models/minicpm5/MiniCPM5ConfigTest.cpp +++ b/tests/models/minicpm5/MiniCPM5ConfigTest.cpp @@ -16,8 +16,8 @@ auto exampleDir() -> std::string { return example_dir_override == nullptr ? std::string(MINICPM5_EXAMPLE_DIR) : std::string(example_dir_override); } -auto loadConfig() -> mllm::models::minicpm5::MiniCPM5Config { - return mllm::models::minicpm5::MiniCPM5Config(exampleDir() + "/config_1B_w4a32_kai.json"); +auto loadConfig(const std::string& variant = "1B") -> mllm::models::minicpm5::MiniCPM5Config { + return mllm::models::minicpm5::MiniCPM5Config(exampleDir() + "/config_" + variant + "_w4a32_kai.json"); } auto parameterFile(mllm::ModelFileVersion version, const std::vector& embedding_shape, bool include_lm_head = true) @@ -55,6 +55,42 @@ TEST(MiniCPM5ConfigTest, RejectsRuntimeContractDrift) { EXPECT_FALSE(mllm::models::minicpm5::matchesOfficialMiniCPM5_1BRuntimeContract(config)); } +TEST(MiniCPM5ConfigTest, SupportsTwoBillionParameterContractAndRejectsMixedVariants) { + const auto config = loadConfig("2B"); + EXPECT_TRUE(mllm::models::minicpm5::matchesOfficialMiniCPM5RuntimeContract(config)); + EXPECT_FALSE(mllm::models::minicpm5::matchesOfficialMiniCPM5_1BRuntimeContract(config)); + EXPECT_EQ(config.hidden_size, 2048); + EXPECT_EQ(config.intermediate_size, 6144); + EXPECT_EQ(config.num_hidden_layers, 42); + EXPECT_EQ(config.num_attention_heads * config.head_dim, 2048); + EXPECT_EQ(config.num_key_value_heads * config.head_dim, 256); + for (int field = 0; field < 3; ++field) { + auto mixed = config; + if (field == 0) mixed.hidden_size = 1536; + if (field == 1) mixed.intermediate_size = 4608; + if (field == 2) mixed.num_hidden_layers = 24; + EXPECT_FALSE(mllm::models::minicpm5::matchesOfficialMiniCPM5RuntimeContract(mixed)); + } + auto drift = config; + drift.tie_word_embeddings = true; + EXPECT_FALSE(mllm::models::minicpm5::matchesOfficialMiniCPM5RuntimeContract(drift)); + drift = config; + drift.rope_theta = 10000; + EXPECT_FALSE(mllm::models::minicpm5::matchesOfficialMiniCPM5RuntimeContract(drift)); +} + +TEST(MiniCPM5ConfigTest, RejectsCrossVariantModelPairing) { + const auto one_b = loadConfig(); + const auto two_b = loadConfig("2B"); + for (const auto version : {mllm::ModelFileVersion::kV1, mllm::ModelFileVersion::kV2}) { + const auto parameters = parameterFile(version, {two_b.vocab_size, two_b.hidden_size}); + EXPECT_NO_THROW(mllm::models::minicpm5::validateModelConfigMatch(two_b, parameters)); + EXPECT_THROW(mllm::models::minicpm5::validateModelConfigMatch(one_b, parameters), std::invalid_argument); + const auto one_parameters = parameterFile(version, {one_b.vocab_size, one_b.hidden_size}); + EXPECT_THROW(mllm::models::minicpm5::validateModelConfigMatch(two_b, one_parameters), std::invalid_argument); + } +} + TEST(MiniCPM5ConfigTest, ValidatesEmbeddingAndIndependentLmHead) { const auto config = loadConfig(); const auto matching_v2 = parameterFile(mllm::ModelFileVersion::kV2, {config.vocab_size, config.hidden_size}); diff --git a/tests/models/minicpm5/MiniCPM5ModelTest.cpp b/tests/models/minicpm5/MiniCPM5ModelTest.cpp index 7a3552ddb..daad0f6fc 100644 --- a/tests/models/minicpm5/MiniCPM5ModelTest.cpp +++ b/tests/models/minicpm5/MiniCPM5ModelTest.cpp @@ -18,8 +18,12 @@ auto exampleDir() -> std::string { } // namespace -TEST(MiniCPM5ModelTest, BuildsNativeKVHeadLogicalSlotsAndResetsThem) { - mllm::initializeContext(); +class MiniCPM5ModelTest : public ::testing::Test { + protected: + static void SetUpTestSuite() { mllm::initializeContext(); } +}; + +TEST_F(MiniCPM5ModelTest, BuildsNativeKVHeadLogicalSlotsAndResetsThem) { const auto config = mllm::models::minicpm5::MiniCPM5Config(exampleDir() + "/config_1B_w4a32_kai.json"); auto model = mllm::models::minicpm5::MiniCPM5ForCausalLM(config); auto& cache = model.kvCache(); @@ -38,3 +42,17 @@ TEST(MiniCPM5ModelTest, BuildsNativeKVHeadLogicalSlotsAndResetsThem) { model.resetState(); EXPECT_EQ(cache.getCurrentSeqCnt(7), 0); } + +TEST_F(MiniCPM5ModelTest, TwoBillionParameterModelResetsLastLogicalSlot) { + const auto config = mllm::models::minicpm5::MiniCPM5Config(exampleDir() + "/config_2B_w4a32_kai.json"); + auto model = mllm::models::minicpm5::MiniCPM5ForCausalLM(config); + auto& cache = model.kvCache(); + EXPECT_EQ(cache.getLayerNums(), 42); + EXPECT_EQ(cache.getKCacheBuffer(41).shape(), (mllm::Tensor::shape_t{1, 2, 2048, 128})); + auto key = mllm::Tensor::zeros({1, 2, 1, 128}, mllm::kFloat32, mllm::kCPU); + cache.updateKVCache(41, key, key); + EXPECT_EQ(cache.getCurrentSeqCnt(41), 1); + EXPECT_EQ(cache.getCurrentSeqCnt(0), 0); + model.resetState(); + EXPECT_EQ(cache.getCurrentSeqCnt(41), 0); +}