[tawei-amd]

Summary

Serves d3:lite, vLLM Semantic Router's Decision 3.0 decision model (vllm-sr/d3-lite v3.1.3), on the NPU, next to or without a chat model, through the upstream package's own HTTP API:

flm serve --decision d3:lite                 # decision model only (-d)
flm serve qwen3.5:4b --decision d3:lite      # next to a chat model
curl -s localhost:52625/v1/systemone -d '{"state": "...", "questions": {"route": {"type": "choice", "criteria": {...}}}}'

Stacked on #815 (the contract library this uses).

• AutoDecisionModel (src/{include,common}/AutoDecisionModel). It checks the package's decision_config.json the way the upstream runtime does. It re-derives the 255 answer codes from the tokenizer and refuses to load on any mismatch. It cross-checks the chat template against the contract with minja. It runs the fp32 readout and warms up at load. The engine is libd3_npu (ROCm-Internal FastFlowLM_IRON#68), vendored for XRT and HRX; xclbins/D3-Lite-NPU2/mm.xclbin is Qwen3.5-0.8B's.
• Server.
  • POST /v1/systemone (alias /v1/decisions) parses the raw body with Python json semantics and replies exactly as d3_server.py: 422 detail texts, invalid_questions, and per-question max_length_exceeded inside a 200. It is NPU-exclusive like the other inference routes.
  • GET /v1/systemone/models is the list operation of vLLM-SR's vllm-sr/systemone@1 protocol.
  • Replies can now carry their own HTTP status (__http_status / __http_body).
• Catalog.
  • d3:lite → D3-Lite-NPU2 (family d3, label decision), hidden from /api/tags and /v1/models like embed-gemma.
  • flm run|serve|bench d3:lite explains how to serve it.
  • Prompts over 4,096 tokens get max_length_exceeded; nothing is ever truncated.
• src/test/d3_npu: AutoDecisionModel without HTTP (harness and engine-level perf driver).
• Docs: docs/docs/models/d3.md, docs/docs/benchmarks/d3_results.md.

Engine libraries come from FastFlowLM_IRON CI (#68's run): Linux XRT/HRX .so and Windows XRT/HRX d3_npu.dll + .lib. The CI-built Linux library is bit-identical in output to the locally validated build on all 311 golden prompts. The installers ship d3_npu.dll. CMake links d3 only where the platform's library is vendored (FLM_HAS_D3_NPU).

Test results (Linux, XRT and HRX builds, this NPU)

check	result
contract (test_d3_contract, + 10,000-request local fuzz run)	10,308 requests: prompts and token ids byte-identical, errors and answers identical
HTTP parity vs upstream d3_server.py (616 requests incl. 300 fuzz bodies)	same status, JSON shape and messages; only differences: images/videos refused (text-only v1) and prompts > 4,096 tokens (max_length_exceeded by design)
accuracy, 6,824 Decision Index public-suite questions	gold accuracy 60.6% (upstream BF16 61.2%), top-1 agreement 90.5%
sweep_decision.py	7/7 cases pass, XRT and HRX builds
soak: 1,000 mixed requests from 4 clients + concurrent chat streaming	0 crashes or hangs; every status equal to upstream's; 251 chat completions alongside, 0 errors
vLLM Semantic Router BackendInvoker replay (vllm-sr/systemone@1)	create, alias, 422 pass-through, list_models: all pass
latency over HTTP, one question of 330 tokens	164 ms p50; first request after ready 166 ms; serve -> ready 2.2 s
chat regression (sweep_llm.py, qwen3.5:0.8b)	passes

Weights: published at FastFlowLM/D3-Lite-NPU2 (commit cd6df2ea); a fresh download matches the package manifest, flm pull d3:lite into an empty model directory verifies all 9 catalog files, and sweep_decision.py passes on the pulled copy. ModelScope (amd/D3-Lite-NPU2) is not uploaded yet, so flm pull --modelscope d3:lite won't work until it is.

Not in this PR

• Images and video (the vision tower is in the package; images/videos lists are refused with upstream's 422 for now).
• A Windows NPU run (the Windows builds compile and link; no Windows machine was used for testing).

🤖 Generated with Claude Code

[tawei-amd]

Self-review (author). Four inline findings plus one on the vendored binaries, most important first. I'm happy to fix any of them here or in a follow-up.

Vendored engine binaries. The libd3_npu.so / d3_npu.dll / d3_npu.lib (XRT and HRX) here come from an IRON CI build of AMD-ROCm-Internal/FastFlowLM_IRON#68 at 6506b014. Since then #68 gained 0a9025d5 (review fixes; the default path's output is bit-identical, checked on 311 prompts). Before merging, re-vendor them from the IRON commit that actually lands on IRON main, so the binaries match a commit there. The Windows DLL has also never run on an NPU: a Windows smoke run (flm pull d3:lite, flm serve --decision d3:lite, one request) is still open.

🤖 Generated with Claude Code

[tawei-amd]

Decisions queue behind chat generations. /v1/systemone is an NPU route, so with flm serve <chat model> --decision d3:lite a decision waits for any in-flight chat completion to finish.

Measured with qwen3.5:0.8b next to d3:lite:

• alone, the docs example decision takes 146 ms;
• sent 1 s into a 600-token chat completion (13.6 s), it took 12.7 s.

For vLLM-SR this matters, because the router asks for the decision before it sends the chat. Options, cheapest first:

• Say it in docs/docs/models/d3.md: for latency-sensitive routing, serve the decision model on its own (flm serve --decision d3:lite, decision only).
• Let a decision request run between decode steps of a chat. The engines are independent, so they only need the device, not the chat's whole turn.
• Give decisions their own queue.

[tawei-amd]

Documented in 25f4d10. The scheduling itself is unchanged, and changing it is a call for the maintainers.

I tested the cheap way around it, a second flm serve for the decision model alone, and it does not help. The two processes contend for the NPU: a 600-token chat took 65 s instead of 13.6 s, and the first decision sent during it still waited 64 s. docs/models/d3.md now states the queueing with the numbers, and says a second server does not avoid it.

A real fix would be scheduler work in the server: let decision requests run between decode steps of an in-flight chat, or give them their own queue. I've left that open for you.

[tawei-amd]

On the vendored binaries from my self-review (no thread): fixed in 25f4d10. All six libraries are re-vendored from FastFlowLM_IRON#68's PR Check run 39445931404 on d51e5a85. The CI Linux library gives bit-identical rows to the locally validated build on all 311 text goldens, and the Windows XRT and HRX builds pass there. They still need re-vendoring from whichever IRON commit finally merges. The Windows NPU smoke run is still open.

🤖 Generated with Claude Code