Field notes from running 235B-class open models on a small cluster of NVIDIA DGX Spark (GB10) boxes, using llama.cpp's RPC backend. Not a tutorial — a lab notebook: configs that worked, numbers measured, failures included.
Three DGX Spark nodes sit on a desk. Each GB10 has 128 GB unified memory
(~121 GiB in free -g reports) at ~273 GB/s. That is a lot for a single box
and not enough for a 180 GB Q6_K weight file. The cluster only makes sense as a
distributed layer split over RPC, with honest attention to link speed,
device enumeration order, and which node runs forty containers besides the
coordinator.
Aggregate unified memory is on the order of 363 GiB — not a transparent 384 GB GPU; cross-node sharding pays serialization and activation-hop tax on decode.
Every dispatch lists what broke before what worked. Launch snippets use
sanitized example addresses (169.254.x.x, 192.0.2.x); every flag is real
and load-bearing. If a number is not measured in the dispatch, it is labeled
as planning math or explicitly unbenched.
Start here — full story, seven failures, benchmark numbers:
Spark Dispatch #1 — Three DGX Sparks, one 235B
Measured on Qwen3-235B-A22B-Instruct-2507 Q6_K @ 131k context: 181 tok/s prefill, 5.8 tok/s decode, ~9 minute load.
| # | Date | Title | Topic |
|---|---|---|---|
| 1 | — | Three DGX Sparks, one 235B | llama.cpp RPC across three GB10s; full failure ladder and benchmarks |
| 2 | 2026-07-25 | Tensor-split order and RPC buffer sizing | Device order [rpc0, rpc1, local] and -b/-ub prefill buffers |
| 3 | 2026-07-25 | GB10 memory traps and context ceilings | Unified-memory env, --cache-ram 0, 262k KV limit over RPC |
Stable topology reference (hardware table, RPC roles, flag summary): fleet-manual.md
- llama.cpp build cited in dispatches:
11924d4c1, aarch64, CUDA on all nodes - MIT License — Copyright (c) 2026 Steven Haller
If you are running llama.cpp RPC on GB10s (or want to), compare notes — the failure modes are learnable.