From 378a71dd5ff9b27ad0563c6d6cbb267d77c44b57 Mon Sep 17 00:00:00 2001 From: Khanh Nguyen Date: Thu, 3 Sep 2026 09:23:14 +0700 Subject: [PATCH] init: report of llama.cpp commits working on supporting hexagon --- docs/backend/hexagon.md | 328 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 328 insertions(+) create mode 100644 docs/backend/hexagon.md diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md new file mode 100644 index 0000000..3e9acad --- /dev/null +++ b/docs/backend/hexagon.md @@ -0,0 +1,328 @@ +# `vla.cpp` on Snapdragon (Hexagon NPU) + +Notes on what it would take to run `vla.cpp` on Qualcomm's Hexagon NPU, read out +of llama.cpp's own Hexagon history up to `b10729` - the tag this tree pins. + +> **Status: read and planned, never run.** There is no Hexagon rung in +> `src/backend.h`, no `GGML_HEXAGON` in `CMakeLists.txt`, and no IQ-9 or IQ-10 +> board on any desk here. Every claim below is either checked against llama.cpp +> at `b10729` (op tables, env vars, commit hashes - go verify them) or is +> explicitly a **projection**. Nothing in this file is a measurement on Qualcomm +> hardware, and the projected latencies are arithmetic, not results. +> +> Two things upstream did in 2026 make this worth writing down now: the NPU +> backend learned `IM2COL` for patch-embedding convolutions - which is exactly +> and only what a VLA's vision tower needs - and it learned multi-NPU. Both are +> already inside the tag we pin. + +The previous version of this file was a verbatim copy of upstream's +[`docs/backend/snapdragon/linux.md`](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/linux.md). +That build recipe is still the right starting point and is kept below, but it +builds *llama.cpp*, not this engine; the gap between the two is the subject here. + +## What "IQ9" and "IQ10" refer to + +Two commits, both inside `b10729` (`458681e1`, 2026-09-01), and no Hexagon commit +lands after it as of this writing: + +| Commit | Date | What it actually did | +|---|---|---| +| `d81e63dc` `CI : support IOT device (IQ9) (#22987)` | 2026-05-14 | Added a third device to the Snapdragon CI matrix: `QCS9075M`, alongside the `SM8750` / `SM8850` phone parts. It is the only entry that is **Linux**, not Android - a separate build job (`linux-iot-snapdragon`, with `-DGGML_OPENCL=ON` on top of the preset), a BASH test framework instead of Appium/pytest, and `--retries 2 --retry-delay 300` because the IoT device in Qualcomm's Device Cloud is scarce. | +| `192067b7` `hexagon: support for multi-NPU devices (IQ9, IQ10) and fully asynchronous backend (#26501)` | 2026-08-26 | 5,314 insertions over 44 files, by Qualcomm. Physical vs virtual NPU sessions, an `ALLREDUCE` HTP kernel and sync tokens so a tensor split can span NPUs, a fully async backend with events, `GET_ROWS`/`SET_ROWS`, Q8_0 flash attention, op fusion, and the `build.py` / `run.py` scripts that replaced the old `adb/*.sh` and `windows/*.ps1` wrappers. | + +What the names do **not** tell you: neither `IQ9` nor `IQ10` appears anywhere in +the source, the docs, or the CI config as an identifier. The only board named in +code is `QCS9075M`, which places IQ-9 as the Dragonwing-class industrial/robotics +part rather than a phone SoC. IQ-10 exists in that commit title and nowhere else. +So "IQ9/IQ10" is upstream shorthand; what is real is the *capability* the commit +added, which is what the rest of this document reads against. + +The generational split visible in the code, rather than in the marketing: + +- **One physical NPU, N virtual sessions** (`HTP0:0`, `HTP0:1`, ...) - the IQ-9 + shape, and what every phone part does today. +- **N physical NPUs** (`HTP0:0`, `HTP1:0`) - discovered via `FASTRPC_GET_DOMAINS`, + with a static fallback that hardcodes physical 0 → CDSP domain 3 (`cdsp`) and + physical 1 → domain 4 (`cdsp1`). Ask for a third physical core without dynamic + discovery and the backend refuses by name: *"physical CDSP core %d not + supported without dynamic discovery"*. Two is what the fallback knows about; + the discovery path is open-ended. + +## What the backend gives you at `b10729` + +- **Two libraries.** `libggml-hexagon.so` on the CPU side, `libggml-htp-vNN.so` + on the NPU side. Skels are built for v68, v69, v73, v75, v79 and v81 and the + right one is picked at runtime from `htpdrv_get_arch`; a failed query falls + back to v73. `GGML_HEXAGON_ARCH` overrides. +- **Sessions are devices.** Each Hexagon process domain shows up to ggml as one + device and behaves like a GPU for offload and model splitting. + `GGML_HEXAGON_DEVICES` takes either a count or an explicit + `HTP:` list. `GGML_HEXAGON_NDEV` is the deprecated spelling. +- **~3.5 GB per session.** The backend now maps and unmaps execution buffers + during graph execution to fit larger models into one session, and layer- or + tensor-splitting across sessions is the alternative. +- **Repack buffers.** Q4_0, Q4_1, Q8_0, IQ4_NL and MXFP4 weights are repacked + into non-host buffers; since #26501 non-host is the default and + `GGML_HEXAGON_HOSTBUF=1` is the opt-out (needed to exercise `MUL_MAT` in + `test-backend-ops`). +- **VTCM is the real budget.** `supports_op` precomputes kernel params for + `MUL_MAT`, `FLASH_ATTN_EXT` and friends and returns false when the tile does + not fit VTCM. An op is not rejected by shape rules so much as by whether it + fits - which means coverage is a function of your tensor sizes, and has to be + measured on the board, not predicted from a table. +- **Fusion**, controlled by `GGML_HEXAGON_OPFUSION`: `RMS_NORM+MUL`, + `MUL_MAT+ADD`, N-way `MUL_MAT`, `ALLREDUCE+ADD`. + +The knobs worth knowing on day one: + +| Variable | Use | +|---|---| +| `GGML_HEXAGON_DEVICES=HTP0:0,HTP1:0` | which NPUs/sessions to open | +| `GGML_HEXAGON_VERBOSE=1` | log every op the NPU accepted, with dtypes and buffers | +| `GGML_HEXAGON_OPFILTER=` | force matching ops off the NPU - the bisection tool | +| `GGML_HEXAGON_PROFILE=1\|2` | per-op usecs/cycles (+PMU), pipe into `scripts/snapdragon/ggml-hexagon-profile.py -` | +| `GGML_HEXAGON_OPTRACE` | fine-grained HVX/HMX/DMA event trace | +| `GGML_HEXAGON_OPPOLL=1` | poll for op completion instead of sleeping - matters for many small graphs | +| `GGML_HEXAGON_HOSTBUF=1` | disable repack buffers (op testing) | + +## Op coverage against what `vla.cpp` actually builds + +This is the part that decides whether the port is a weekend or a month. The +column on the right is `ggml_backend_hexagon_device_supports_op` at `b10729`; +the middle column is a grep of `src/`. + +| ggml op | Where we build it | Hexagon at `b10729` | +|---|---|---| +| `MUL_MAT` / `MUL_MAT_ID` | 154 sites, everywhere | yes - Q4_0/Q4_1/Q8_0/IQ4_NL/MXFP4 (repacked), or F16/F32 direct. **No BF16.** Refuses `src0->ne[1] > 32768` (lm-heads), and anything whose tile misses VTCM | +| `IM2COL` | 6 `ggml_conv_2d`, every SigLIP/DINOv2 patch embed | yes, **partial** - added by `355303ed` (#26007) "targeting only patch-embedding convolutions": 2D only, F32 source, contiguous, **zero padding**. Our conv is kernel = stride = patch, pad 0, dilation 1. On paper it is precisely the supported case | +| `FLASH_ATTN_EXT` | 8 sites | yes - F16 or Q8_0 K/V, F16 mask, F32 sinks | +| `ROPE` / `ROPE_MULTI` | 14 sites | yes, including MROPE/IMROPE (`17d22a35`) and vision RoPE (`f2d1c2f3`) | +| `NORM`, `RMS_NORM`, `L2_NORM`, `SOFT_MAX`, `SCALE`, `CONCAT`, `SQR`, `GET_ROWS`, `ADD/SUB/MUL/DIV` | all over | yes | +| `CPY` / `CONT` (`ggml_cast`, 14 sites) | dtype hops between stages | yes, but **F32↔F16 only, and never a reshape and a conversion in the same op** | +| `SILU`, `GELU` (tanh), `GELU_QUICK`, `TANH`, `SIGMOID`, `NEG`, `ABS`, `EXP`, `SOFTPLUS` | 22 `ggml_silu`, 10 `ggml_gelu` | yes | +| **`RELU`** | **18 sites** - `action_expert.cpp` state encoder and action decoder, plus Evo-1, VLA-Adapter, VLA-JEPA, GR00T N1.7, OpenVLA-OFT, BitVLA | **not implemented.** `grep -i relu` over `ggml-hexagon/` returns nothing | +| **`GELU_ERF`** | **13 sites** - `ffn_gelu_erf` in `src/layers/ffn.h` and `dual_tower.h`, i.e. every tower trained with the exact erf form (DINOv2, SigLIP-so400m) | **not implemented.** `ggml_gelu` (tanh) is a *different op*, and substituting it silently changes numerics | +| `ggml_map_custom1` | BitVLA only | no, and BitVLA pins itself to CPU by design anyway | + +Two missing unary kernels, and one of them is in the vision tower of half the +model zoo. That is the whole gap. Note the shape of the mistake it invites: on +OpenVINO, "GELU is GELU" cost us a wrong arch and a day of bisection +([ov.md](ov.md)); here the op is simply absent, which is the *better* failure - +`supports_op` returns false and you find out at load time. + +**Except that in this engine, absent is fatal.** `src/backend.h` brings up one +backend and drives it through `gallocr` - there is no scheduler and no per-op CPU +fallback, by design, so an unsupported op asserts at predict time instead of +quietly limping. Every other backend we support (CUDA, Metal, SYCL, OpenVINO) +covers our op set completely, so this constraint has never bitten. Hexagon would +be the first backend where it does. + +So the port is gated on one of: + +1. **Two HVX kernels upstream.** `e70802a0` (#27786) added `ABS` and `LOG` as HTP + unary ops; `RELU` and `GELU_ERF` are the same shape of change, and `RELU` in + particular is a `vmax` against zero. This is the honest path and it is small. +2. **A scheduler in `vla.cpp`.** `ggml_backend_sched` with a CPU fallback, which + is an engine change with consequences for every backend, not a Hexagon one. +3. **Graph surgery** - swapping `gelu_erf` for `gelu` and re-checking fidelity. + Cheap, wrong, and it will cost more than it saves. Do not. + +## What would have to change in `vla.cpp` + +Nothing here has been written; this is the shape of the diff. + +**1. Build flags.** `GGML_HEXAGON` joins the mutually-exclusive accelerator list +at [CMakeLists.txt:15](../../CMakeLists.txt#L15) and defines `GGML_USE_HEXAGON` +the way the other four do. + +**2. A rung in the ladder** at [src/backend.h:176](../../src/backend.h#L176). +The Hexagon init API takes no ordinal - `ggml_backend_hexagon_init(void)` - so +device selection is environment-driven, and `VLA_DEVICE` (an integer ordinal for +CUDA and SYCL) does not map onto it. The right binding is `VLA_DEVICE=HTP0:0` +passed through to `GGML_HEXAGON_DEVICES`, which means the rung needs its own +parse rather than `backend_device_index()`. + +**3. Weights.** Our default resident dtype is BF16, which the NPU does not +accept in any op. Every Hexagon run needs a requantized checkpoint - +`python scripts/quantize_gguf.py --in model-bf16.gguf --out model-q8_0.gguf +--type Q8_0` (or `Q4_0`) - and the fidelity of *that* against the CPU reference +has to be established before any NPU number means anything. `--weight-dtype f32` +is the other option and doubles resident memory. + +**4. The build that is actually the most work: `vla-server`'s dependencies.** +`find_package(Protobuf REQUIRED)` and `pkg_check_modules(libzmq REQUIRED)` are +unconditional in our `CMakeLists.txt`, and the Snapdragon toolchain image carries +the Hexagon SDK and the OpenCL SDK but not protobuf, libzmq or cppzmq for arm64. +Configure fails before a single object compiles - including for `vla-cli` and +`vla-bench`, which link neither. Options, cheapest first: + +- Make the serving deps conditional so `vla-cli` / `vla-bench` configure without + them. Worth doing regardless of Hexagon; it is a dozen lines. +- Add the three to the cross sysroot. +- Build natively on the board, which for an IQ-9-class Linux part is plausible + and is how the first bring-up will probably actually happen. + +**5. Presets.** We have no `CMakeUserPresets.json`; upstream's +`arm64-linux-snapdragon-release` is the base to copy, with `GGML_HEXAGON=ON` and +`-march=armv8.2a+fp16+dotprod`. Note it sets `GGML_OPENCL=OFF` - upstream CI adds +`-DGGML_OPENCL=ON` on the command line for the IoT job. + +## Building llama.cpp for the board + +The upstream path, unchanged, and the thing to get working first - if +`llama-bench` will not run on the device, nothing downstream matters. Requires +only Docker on the host; `build.py` pulls +`ghcr.io/snapdragon-toolchain/arm64-linux:v0.7` itself. + +```bash +# cross-compile and deploy over SSH in one step +./scripts/snapdragon/build.py --target lnx:user@host --push + +# or by hand, inside the container +docker run -it --rm -u $(id -u):$(id -g) --volume $(pwd):/workspace \ + --platform linux/amd64 ghcr.io/snapdragon-toolchain/arm64-linux:v0.7 +[d]/workspace> cp docs/backend/snapdragon/CMakeUserPresets.json . +[d]/workspace> cmake --preset arm64-linux-snapdragon-release -B build-snapdragon +[d]/workspace> cmake --build build-snapdragon -j $(nproc) +[d]/workspace> cmake --install build-snapdragon --prefix pkg-linux +``` + +On the device, both library paths must be set - `ADSP_LIBRARY_PATH` is how the +FastRPC loader finds the HTP skel: + +```bash +export LD_LIBRARY_PATH=./lib +export ADSP_LIBRARY_PATH=./lib +./bin/llama-cli -m Llama-3.2-3B-Instruct-Q4_0.gguf --device HTP0 -ngl 99 -p "..." +``` + +`run.py` maps flags to those environment variables and can drive the device over +SSH or ADB from the host: + +```bash +./scripts/snapdragon/run.py --target lnx:user@host --devices HTP0 -- \ + llama-bench -m Llama-3.2-1B-Instruct-Q4_0.gguf -p 128 -n 64 + +# multi-NPU tensor split - the IQ-10 shape +./scripts/snapdragon/run.py --target lnx:user@host --devices HTP0:0,HTP1:0 -- \ + llama-completion -m gemma-2b-it-Q4_0.gguf -f prompt.txt --split-mode tensor +``` + +## Bring-up checklist, for the day a board arrives + +In order. Each step is cheap and each one has failed for somebody upstream. + +1. **Confirm the arch and session lines.** `ggml-hex: Hexagon Arch version v79` + and `allocating new session: HTP0:0`. A v73 line on a part you believe is + newer means `htpdrv_get_arch` failed and you are on the fallback skel. +2. **`test-backend-ops` before anything of ours.** + `run.py --hex-hostbuf 0 --devices HTP0:0 -- test-backend-ops -b HTP0:0 -o MUL_MAT`, + then `-o IM2COL`, `-o FLASH_ATTN_EXT`, `-o ROPE`. Filter to the dtypes we + would actually ship (`q8_0`, `q4_0`, `f16`). +3. **`llama-bench` on a 1B Q4_0** to get a throughput anchor on *this* board that + is comparable with upstream's published numbers, before any VLA is involved. +4. **Load a requantized VLA with `GGML_HEXAGON_VERBOSE=1`** and read which ops the + NPU accepted. This is where the two missing unaries will announce themselves. +5. **Fidelity before latency.** max|delta| against a CPU-backend reference on the + same checkpoint, the same 2.9e-3 bar SYCL and OpenVINO are held to + ([ov-progress.md](ov-progress.md)). A partially-failing run still prints a + wall-clock time; do not read a latency off a run whose actions did not come out. +6. **Then profile.** `GGML_HEXAGON_PROFILE=1 ... |& scripts/snapdragon/ggml-hexagon-profile.py -`, + and `GGML_HEXAGON_OPFILTER` to bisect anything that looks wrong. + +## Projections + +Everything in this section is arithmetic over numbers measured elsewhere. It is +here to set expectations and to be *falsified* by the first real run, not to be +quoted. + +Measured reference points, all ours, all `server_total_ms` per action chunk +(`ci/baselines/`, [sycl.md](sycl.md)): + +| Device | SmolVLA | π0 | GR00T N1.5 | Evo-1 | +|---|---:|---:|---:|---:| +| Ryzen 5 5500, CPU backend, 8 threads | 1,920 | - | - | 7,695 | +| Jetson Orin Nano | 510 | 1,485 | 1,183 | 3,552 | +| Apple M4 (Metal) | 324 | 1,129 | - | - | +| Arc A380 (SYCL) | 630 | - | - | 1,176 | +| RTX 3090 | 99 | 262 | 204 | 487 | + +The only Hexagon anchor that exists is upstream's own, on a v79 phone part: +Llama-3.2-1B Q4_0, **pp128 = 169 t/s**, **tg64 = 51.5 t/s** (single `HTP0` +session). Read as work rather than tokens: prefill at 169 t/s over 1.24 B params +is ≈ 0.42 TFLOP/s effective; decode at 51.5 t/s over 730 MiB of weights is +≈ 37 GB/s of weight traffic, which is a believable LPDDR5 read rate and a good +sign the published number is not cherry-picked. + +A VLA action chunk is **all prefill and no decode**: fixed shapes, a tiny KV +cache, no sampling loop. That is the regime the 0.42 TFLOP/s figure comes from, +which makes the extrapolation less dishonest than usual. SmolVLA is 1.07 GiB of +BF16 weights (≈ 0.54 B params), spent on three stages per prediction - a vision +tower over 2-3 views, a ~113-token prefix through the VLM, and 10 denoise steps +over a ~50-token suffix through the much smaller action expert. Multiplying that +out lands in the region of 300-400 GFLOP per chunk, so at the anchor rate: + +**SmolVLA, single v79-class HTP, Q4_0: 0.8-2.0 s per action chunk.** The band is +wide on purpose. The bottom needs the vision tower to land on the NPU via the new +`IM2COL` and the denoise loop to stay fed; the top is what you get if the loop is +dispatch-bound and the towers fall back. Either way that is somewhere between the +Orin Nano (510 ms) and a desktop CPU (1,920 ms) - useful for a 5-10 Hz policy on +a part that draws a few watts, not competitive with a discrete GPU. Expect the +first honest number to be nearer the top of the band and to improve with repack, +fusion and `OPPOLL`, exactly as upstream's own numbers did over the 107 commits +that have touched `ggml/src/ggml-hexagon/` since the backend landed in October +2025 (`63d2fc46`, "experimental"). + +Where the interesting risk is: + +- **The vision tower is the prize.** It is 58% of SmolVLA's CPU-backend time + (1,119 ms of 1,920 ms) and 7.1x faster on an A380. `IM2COL` landing for + patch-embed convolutions specifically is the single most encouraging thing in + this entire history, and it landed in July 2026 for reasons that had nothing to + do with us. +- **The denoise loop is the risk.** Ten sequential steps of small GEMMs is the + worst case for any offload engine - it is why SmolVLA gains least on SYCL + (3.0x when its own vision tower gains 7.1x). The fully-async backend, op + batching and graph reuse from #26501 are aimed at exactly this, which is + encouraging, but "many small ops over FastRPC" is a dispatch-latency problem + and NPUs do not usually win those. +- **The lm-head refusal does not touch us.** Hexagon bails on `src0->ne[1] > + 32768`; VLA action heads are small and no in-tree arch decodes a vocabulary on + the action path. + +### IQ-10, imagined + +Nothing about a second-generation part is knowable from this repository, so what +follows is a bet, not a forecast. The code in #26501 tells you what Qualcomm +built *for*: two or more physical NPUs, an `ALLREDUCE` kernel with a DMA solver +and fused `ALLREDUCE+ADD`, sync tokens to order work across devices, and a static +domain table that stops at two. + +If a dual-NPU IQ-10 lands, the obvious use - `--split-mode tensor` across +`HTP0:0,HTP1:0`, paying an allreduce per layer - is the *worse* fit for a VLA. +An action chunk has three stages that already hand off through host memory: the +vision tower, the VLM prefix, and the action expert. Putting the tower on one NPU +and the expert on the other pipelines across timesteps with **one** handoff per +stage instead of one collective per layer, and a policy server at 10 Hz has a +steady stream of chunks to pipeline. + +`vla.cpp` cannot do that today, for the same reason it cannot split across an +Intel iGPU and NPU ([ov.md](ov.md) reaches this conclusion from the other +direction): the core drives one backend for a whole prediction. It would need a +per-*stage* backend, not a per-op scheduler. The seam is already in the right +place. That is the one engine change this whole document argues for, and it pays +off on more than Hexagon. + +## What would falsify all of this + +- `RELU` and `GELU_ERF` turn out to be a 200-line HVX patch each → the port is a + weekend once the board exists, and everything above about schedulers is moot. +- `IM2COL`'s "partial" support turns out not to cover our patch shapes after all + (multi-view batching, a non-contiguous pixel buffer) → the vision tower stays + on CPU, the projection's lower bound disappears, and the whole exercise gets + much less interesting. +- Q8_0 requantization moves any arch past 2.9e-3 → the dtype story needs work + before the backend story does. +- VTCM budgets reject our GEMM shapes (which are wider and shorter than an LLM's) + → `supports_op` starts returning false for reasons no table here predicts, and + the answer is measurement, not more reading.