A single-model inference engine for Qwen3.8-27B on the NVIDIA DGX Spark (GB10). v0 runs: it serves the production checkpoint end to end (docs/V0.md). M1 adds exact speculative decoding with the DFlash2 drafter: 45.5 tokens/s on replayed agent sessions, 97% of production's decode speed (reports/M1.md). Prefill came next (reports/M2-prefill.md): prompts token-identical to production's, prefill on block-scaled tensor cores at 2,100-2,600 tokens/s (3-5x M1's, 1.1-1.3x production's on long prompts) and shared-prefix checkpoints, so a replayed agent request now takes 4.3-4.4 s against M1's 8.1 s and production's 4.2 s.
Decoding Qwen3.8-27B on a DGX Spark is bound by its 273 GB/s memory bus, not by compute: every generated token streams the model's weights (17.6 GB per pass for the checkpoint Mightling serves), which caps plain decoding near 15 tokens/s. Speculation is the only multiplier, and removing everything that is not weight traffic is the other half.
The spec is built on Mightling's measured workload, not a chat benchmark: in 168 recorded agent sessions, 92% of the wall time is the model, a request writes 177 tokens on average (mostly tool-call arguments), and 46% of what the agent writes is copied from its own context. On that workload today's production server (SGLang with the DFlash2 drafter) decodes about 30 tokens/s at ~160 ms per speculative step, against a weight pass that should take about 90 ms.
Spec: specs/ (8 October 2026, split into documents on 9 October) · analysis scripts in bench/
Two programs: ling-compile, an offline compiler that fixes the precision map, folds the norms, lays the weights out on disk in kernel order, instantiates one fused CUDA kernel per matrix shape and captures each decode step as a CUDA graph; and ling-serve, a resident runtime that keeps every conversation's KV cache and DeltaNet state on the GPU and serves Mightling as its model server.
Six levers, the first four in order of payoff on one agent's session:
- Step time at the bandwidth bound: fused shape-specialized kernels, one CUDA graph per step, GPU-side sampling, FP8 KV and a DeltaNet replay scheme: ~160 ms per step to 76–90 ms.
- A hybrid draft: DFlash2's 16 draft tokens plus a context-lookup branch in the same verify, because agents copy: 4.87 accepted tokens per step to ≥ 5.8.
- Short requests stay short: FP4 prefill and no host work before the first pass: 0.68 s of fixed cost per request to ≤ 0.25 s.
- New sessions start warm: DeltaNet checkpoints at shared-prefix boundaries, so a new session's first answer comes in ≤ 1 s instead of 9–13 s.
- Several agents share one weight read: 2–4 sequences per pass for Night Shift, benchmarks, refine and subagents: ≥ 175 tokens/s aggregate.
- An agent-tuned drafter (stretch): fine-tuned on the user's own sessions, on the Spark, overnight.
Targets (estimates until M0 measures the bus and profiles production), single stream on a replay of recorded agent sessions:
| Production today | Target | Stretch | |
|---|---|---|---|
| Decode on agent output | ~30 tokens/s | ≥ 70 tokens/s (2.3×) | ≥ 100 |
| Mean model time per agent request | ~6.6 s | ≤ 2.8 s | ≤ 2.0 s |
| First answer of a new session | 9–13 s | ≤ 1 s | ≤ 0.5 s |
| Agent session wall time | 1× | ≥ 1.9–2.1× faster | ≥ 2.4–2.8× faster |
| Aggregate decode, 4 agents at once | measured in M0 | ≥ 175 tokens/s | ≥ 220 |
Greedy output stays identical to non-speculative decoding.
The drafter only proposes, so it may be lossy: by default its fc and conv kernel projections run as NVFP4 copies rounded at load, not in BF16 (since 2026-10-20, BACKLOG §19 item 33; --draft-bf16 turns them off). The target's weights are the checkpoint's.
This engine is planned as a model server for Mightling, Dreamference's local, confidential AI for DGX Spark, where Qwen3.8-27B is the default model.
GNU Affero General Public License v3.0 (LICENSE), the same as Mightling.
Contributions require a one-time Contributor License Agreement; see CONTRIBUTING.md.