From aaaa7e8af15ae78583b61ec54e491f6b6e253f95 Mon Sep 17 00:00:00 2001 From: tawei-amd Date: Tue, 20 Oct 2026 09:48:50 -0700 Subject: [PATCH 1/2] docs(d3): smoke test, the --decision launch method, catalog entry - models/d3.md: a four-command smoke test with the outputs a healthy install prints (taken from a run of this build). - instructions/server: `decision` is a model type; a "Decision Model (System One)" section with the launch commands, the models endpoint, and the NPU sharing caveat. - instructions/cli: `--decision`, server mode only. - models catalog page: Decision 3.0 (System One). Co-Authored-By: Claude --- docs/docs/instructions/cli.md | 14 ++++++++++++++ docs/docs/instructions/server/index.md | 16 ++++++++++++++++ docs/docs/models/d3.md | 24 ++++++++++++++++++++++++ docs/models.md | 4 +++- 4 files changed, 57 insertions(+), 1 deletion(-) diff --git a/docs/docs/instructions/cli.md b/docs/docs/instructions/cli.md index 8be86336b..62647a1ca 100644 --- a/docs/docs/instructions/cli.md +++ b/docs/docs/instructions/cli.md @@ -368,6 +368,20 @@ See the ASR guide [here](https://fastflowlm.com/docs/models/whisper/) --- +### 🧭 Decision Model (System One) + +`--decision ` loads a decision model (`d3:lite`) next to the chat model, or alone, and serves `POST /v1/systemone`. +Decision models run in **Server Mode only** (`flm run d3:lite` prints how to serve it). + +```shell +flm serve --decision d3:lite # decision model only +flm serve qwen3.5:4b --decision d3:lite # next to a chat model +``` + +See the d3 guide [here](https://fastflowlm.com/docs/models/d3/) + +--- + ## 💻 Commands Inside CLI Mode Once inside the CLI, use the following commands. System commands always start with `/` (e.g., `/help`). diff --git a/docs/docs/instructions/server/index.md b/docs/docs/instructions/server/index.md index 67ba55dce..f20460a1b 100644 --- a/docs/docs/instructions/server/index.md +++ b/docs/docs/instructions/server/index.md @@ -27,6 +27,7 @@ FLM can keep one NPU model loaded per type at a time: - `asr` - `llm` - `embedding` +- `decision` (`--decision d3:lite`, the System One API; see [Decision model](#decision-model-system-one) below) Different model types can run together (for example, one LLM and one embedding model). @@ -47,6 +48,21 @@ flm serve lfm2:1.2b --port 52628 +## Decision Model (System One) + +A decision model answers typed questions about a state (choices, yes / no, scores) with a probability for every +option, over `POST /v1/systemone`. It is served next to a chat model or on its own: + +```shell +flm serve --decision d3:lite # decision model only +flm serve qwen3.5:4b --decision d3:lite # next to a chat model +``` + +`GET /v1/systemone/models` lists the loaded decision model. Requests share the NPU with the chat model and run one +at a time, so a decision waits for a chat completion that is already running; a second `flm serve` for the decision +model does not avoid this (the two processes contend for the NPU). Request format, a smoke test and latency: +[Decision 3.0 (d3)](https://fastflowlm.com/docs/models/d3/). + ## Set Context Length at Launch The default context length for each model can be found [here](https://fastflowlm.com/docs/models/). diff --git a/docs/docs/models/d3.md b/docs/docs/models/d3.md index 9f3eda103..f05c283a1 100644 --- a/docs/docs/models/d3.md +++ b/docs/docs/models/d3.md @@ -65,6 +65,30 @@ The reply on a Ryzen AI MAX+ 395 (numbers rounded to 3 places here; the server s "usage":{"input_tokens":342,"output_tokens":0}} ``` +### Smoke test + +Four commands check a new install end to end (Linux or Windows): + +```shell +flm pull d3:lite # downloads D3-Lite-NPU2 (about 0.7 GB) +flm serve --decision d3:lite # wait for "Decision model ready: d3-lite (d3:lite), max input 4096 tokens" +``` + +In a second terminal: + +```shell +curl -s http://127.0.0.1:52625/v1/systemone/models +# {"object":"list","data":[{"id":"d3:lite","object":"model","api":"systemone"}]} + +curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \ + -d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice", + "criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}' +# {"model":"d3-lite","answers":{"route":{"type":"choice","choice":"returns","probabilities":{"returns":0.98,... +``` + +A healthy install answers `"choice":"returns"` with `returns` near 0.98 in about 0.1 s. `flm run d3:lite` does not +start a chat; it prints how to serve the model. + ### Request and response | Field | Meaning | diff --git a/docs/models.md b/docs/models.md index 618ea8863..93ea913a1 100644 --- a/docs/models.md +++ b/docs/models.md @@ -11,7 +11,8 @@ sections: FastFlowLM curates the most requested families and publishes tuned manifests under `flm pull `. We validate every build on Ryzen™ AI laptops and provide matching cards—see them [here](/docs/models/). The catalog already covers models spanning Qwen3.8 (27B, MTP speculative decoding), Qwen3.6 (MoE), Qwen3.5 (Vision), Gemma4 (omni), Qwen3, GPT-OSS-20B (MoE), DeepSeek-R1, Whisper - (Audio), Hy-MT2 (Translation), MiniCPM-V 4.7 (Vision), MedGemma (Medical), and other leading open families. + (Audio), Hy-MT2 (Translation), MiniCPM-V 4.7 (Vision), MedGemma (Medical), Decision 3.0 (d3-lite, the System One + decision model of vLLM Semantic Router), and other leading open families. ctas: - label: "View model docs" href: "/docs/models/" @@ -34,6 +35,7 @@ sections: - "Whisper (Audio)" - "Hy-MT2 (Translation)" - "SmolVLA (Robotics)" + - "Decision 3.0 (System One)" - "MedGemma (Medical)" - "DeepSeek R-1" - "LLaMA 3.x" From 1da24df24155ad32ff45ebebb70625268e4254f3 Mon Sep 17 00:00:00 2001 From: tawei-amd Date: Fri, 23 Oct 2026 12:38:17 -0700 Subject: [PATCH 2/2] docs(d3): FastFlowLM 1.0.11 -- images, 32,768 tokens, smoke test with an image - Catalog: d3:lite needs FastFlowLM 1.0.11 (images, 32,768-token input, the NPU attention kernel and the vision bitstreams ship with it). - models/d3.md: 'FastFlowLM 1.0.11 or later'; the smoke test shows this release's outputs (ready line with 32768, /v1/systemone/models with modalities and image_max_pixels) and adds an image request (panda.png from this repo; shell and PowerShell). Outputs from a run of main's CI build b2630871: panda 0.999, 404 input tokens, 0.42 s. - benchmarks/d3_results.md: 1.0.11; image accuracy on the Decision Index public vision suite (600 questions, 10 benchmarks: NPU 61.7% vs upstream BF16 60.2%, +1.50 pt, 95% CI -0.92 ... +3.92) and latency with and without an image from 512 to 32,000 input tokens. - instructions/server, cli: images, the input limit, the NPU hardware contexts the decision model takes. Co-Authored-By: Claude --- docs/docs/benchmarks/d3_results.md | 42 +++++++++++++++++++++++++- docs/docs/instructions/cli.md | 3 +- docs/docs/instructions/server/index.md | 11 ++++--- docs/docs/models/d3.md | 37 +++++++++++++++++++---- src/model_list.json | 2 +- 5 files changed, 82 insertions(+), 13 deletions(-) diff --git a/docs/docs/benchmarks/d3_results.md b/docs/docs/benchmarks/d3_results.md index d32885c1b..a4d788b9a 100644 --- a/docs/docs/benchmarks/d3_results.md +++ b/docs/docs/benchmarks/d3_results.md @@ -7,7 +7,7 @@ nav_order: 20 ## ⚡ d3-lite on the NPU -Accuracy and latency of `d3:lite` (vllm-sr/d3-lite v3.1.3, package D3-Lite-NPU2) served by FastFlowLM 1.0.10. +Accuracy and latency of `d3:lite` (vllm-sr/d3-lite v3.1.3, package D3-Lite-NPU2) served by FastFlowLM 1.0.11. > **Note:** > - AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FLM's default NPU power mode (Performance). @@ -44,3 +44,43 @@ First request after `flm serve` is ready: 191 ms (steady state 168 ms over HTTP) Rows of 2,048 tokens and more: median of 15 runs (32,768: 3). From 1,536 tokens attention runs on the NPU (below that the CPU is faster); the first such request also loads the attention kernel (about 0.1 s) and from then on the decision model holds two NPU hardware contexts instead of one. + +### Images + +Accuracy on the Decision Index public vision suite (the 10 benchmarks that can be rebuilt without gated data; 60 +questions each, images capped at 1.6 MP as the suite's `decide` engine sends them): + +| Benchmark | Upstream BF16 | NPU (FLM) | Same answer | +|---|---:|---:|---:| +| CV-Bench | 81.7% | 85.0% | 91.7% | +| BLINK | 51.7% | 58.3% | 78.3% | +| RealWorldQA | 68.3% | 71.7% | 88.3% | +| CharXiv | 65.0% | 70.0% | 90.0% | +| InfographicVQA | 78.3% | 80.0% | 90.0% | +| Mind2Web | 61.7% | 63.3% | 83.3% | +| KIE (CORD + FUNSD) | 91.7% | 93.3% | 98.3% | +| Moderation (Hateful Memes) | 55.0% | 53.3% | 95.0% | +| R-Bench-M | 31.7% | 26.7% | 71.7% | +| MMMU-Pro vision | 16.7% | 15.0% | 58.3% | +| **All 600** | **60.2%** | **61.7%** | **84.5%** | + +Paired gold accuracy, NPU minus upstream BF16: +1.50 pt (95% CI [−0.92, +3.92] pt). Image grids and input token +counts equal upstream's on every question; the processor's pixels are within 2 grey levels of upstream's. + +Latency with and without an image, at the same total input length (one question, over HTTP, median of 5; 8,192 +tokens: 3 runs; 16,384 and 32,000: 1): + +| Input tokens | Text only (ms) | Text + one image (ms) | Image | +|---:|---:|---:|---| +| 516 | 266 | 483 | 640 × 480 (300 tokens) | +| 1,028 | 448 | 658 | 640 × 480 (300 tokens) | +| 2,052 | 830 | 1,651 | 1.6 MP (1,600 tokens) | +| 4,100 | 1,803 | 2,641 | 1.6 MP | +| 8,196 | 3,798 | 4,641 | 1.6 MP | +| 16,386 | 8,733 | 9,560 | 1.6 MP | +| 31,994 | 21,466 | 22,477 | 1.6 MP | + +An image costs a fixed amount on top of its tokens (decoding, resizing and the vision tower): about 0.2 s at +640 × 480 and 0.8 s at 1.6 MP. All questions of a request share one image encoding: a 1.6 MP image with 8 questions +takes 6.4 s, not 8 × 1.5 s. The first request with images loads the vision tower (about 40 ms) and two more NPU +hardware contexts. diff --git a/docs/docs/instructions/cli.md b/docs/docs/instructions/cli.md index 62647a1ca..d02a0ea1d 100644 --- a/docs/docs/instructions/cli.md +++ b/docs/docs/instructions/cli.md @@ -370,7 +370,8 @@ See the ASR guide [here](https://fastflowlm.com/docs/models/whisper/) ### 🧭 Decision Model (System One) -`--decision ` loads a decision model (`d3:lite`) next to the chat model, or alone, and serves `POST /v1/systemone`. +`--decision ` loads a decision model (`d3:lite`) next to the chat model, or alone, and serves `POST /v1/systemone` +(text and images, up to 32,768 input tokens per question; FastFlowLM 1.0.11 or later). Decision models run in **Server Mode only** (`flm run d3:lite` prints how to serve it). ```shell diff --git a/docs/docs/instructions/server/index.md b/docs/docs/instructions/server/index.md index f20460a1b..c71b660e8 100644 --- a/docs/docs/instructions/server/index.md +++ b/docs/docs/instructions/server/index.md @@ -51,16 +51,19 @@ flm serve lfm2:1.2b --port 52628 ## Decision Model (System One) A decision model answers typed questions about a state (choices, yes / no, scores) with a probability for every -option, over `POST /v1/systemone`. It is served next to a chat model or on its own: +option, over `POST /v1/systemone`; requests may carry images (base64 data URLs). It is served next to a chat model or +on its own: ```shell flm serve --decision d3:lite # decision model only flm serve qwen3.5:4b --decision d3:lite # next to a chat model ``` -`GET /v1/systemone/models` lists the loaded decision model. Requests share the NPU with the chat model and run one -at a time, so a decision waits for a chat completion that is already running; a second `flm serve` for the decision -model does not avoid this (the two processes contend for the NPU). Request format, a smoke test and latency: +`GET /v1/systemone/models` lists the loaded decision model with its input limit and modalities. Requests share the +NPU with the chat model and run one at a time, so a decision waits for a chat completion that is already running; a +second `flm serve` for the decision model does not avoid this (the two processes contend for the NPU). The decision +model takes one NPU hardware context, two once a prompt of 1,536 tokens or more arrives, and two more on the first +request with images; mind this when the chat model and others already use most of the device's 16. Request format, a smoke test and latency: [Decision 3.0 (d3)](https://fastflowlm.com/docs/models/d3/). ## Set Context Length at Launch diff --git a/docs/docs/models/d3.md b/docs/docs/models/d3.md index f05c283a1..c912ca050 100644 --- a/docs/docs/models/d3.md +++ b/docs/docs/models/d3.md @@ -14,6 +14,7 @@ parent: Models - **Quantization:** Q4_K - **Max input:** 32,768 tokens per question (longer prompts get `max_length_exceeded`; nothing is truncated). One question takes about 0.4 s at 1,024 tokens, 1.6 s at 4,096 and 23 s at 32,768; the NPU serves one request at a time - **Inputs:** text and images (video: not supported) +- **FastFlowLM:** 1.0.11 or later d3 answers *System One* requests: a `state` (any JSON value) and one or more questions, each a `choice` (up to 255 options), a `noul` (no / yes) or a `score` (2–10 ordered levels). Every option gets a probability. It is the decision @@ -67,18 +68,19 @@ The reply on a Ryzen AI MAX+ 395 (numbers rounded to 3 places here; the server s ### Smoke test -Four commands check a new install end to end (Linux or Windows): +Five commands check a new install end to end, text and images: ```shell -flm pull d3:lite # downloads D3-Lite-NPU2 (about 0.7 GB) -flm serve --decision d3:lite # wait for "Decision model ready: d3-lite (d3:lite), max input 4096 tokens" +flm pull d3:lite # downloads D3-Lite-NPU2 (about 0.8 GB, vision weights included) +flm serve --decision d3:lite # wait for "Decision model ready: d3-lite (d3:lite), max input 32768 tokens" ``` In a second terminal: ```shell curl -s http://127.0.0.1:52625/v1/systemone/models -# {"object":"list","data":[{"id":"d3:lite","object":"model","api":"systemone"}]} +# {"object":"list","data":[{"id":"d3:lite","object":"model","api":"systemone","max_input_tokens":32768, +# "modalities":["text","image"],"image_max_pixels":1638400}]} curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \ -d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice", @@ -86,8 +88,31 @@ curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" # {"model":"d3-lite","answers":{"route":{"type":"choice","choice":"returns","probabilities":{"returns":0.98,... ``` -A healthy install answers `"choice":"returns"` with `returns` near 0.98 in about 0.1 s. `flm run d3:lite` does not -start a chat; it prints how to serve the model. +A healthy install answers `"choice":"returns"` with `returns` near 0.98 in about 0.1 s. + +Then an image (Linux shell; the image is a 639 × 480 PNG from this repository): + +```shell +curl -sLo panda.png https://raw.githubusercontent.com/ROCm/FastFlowLM/main/src/tb_files/panda.png +printf '{"questions": {"animal": {"type": "choice", "instructions": "Which animal is in the photo?", + "criteria": {"dog": "A dog", "panda": "A panda", "cat": "A cat"}}}, + "images": ["data:image/png;base64,%s"]}' "$(base64 -w0 panda.png)" > image_request.json +curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" -d @image_request.json +# {"model":"d3-lite","answers":{"animal":{"type":"choice","choice":"panda","probabilities":{"dog":0.0004,"panda":0.999,... +# "usage":{"input_tokens":404,"output_tokens":0}} +``` + +On Windows (PowerShell): + +```powershell +Invoke-WebRequest https://raw.githubusercontent.com/ROCm/FastFlowLM/main/src/tb_files/panda.png -OutFile panda.png +$img = [Convert]::ToBase64String([IO.File]::ReadAllBytes("panda.png")) +$body = '{"questions": {"animal": {"type": "choice", "instructions": "Which animal is in the photo?", "criteria": {"dog": "A dog", "panda": "A panda", "cat": "A cat"}}}, "images": ["data:image/png;base64,' + $img + '"]}' +Invoke-RestMethod http://127.0.0.1:52625/v1/systemone -Method Post -ContentType "application/json" -Body $body | ConvertTo-Json -Depth 5 +``` + +A healthy install answers `"choice":"panda"` with `panda` near 0.999 and 404 input tokens, in about 0.4 s (the first +image request also loads the vision tower). `flm run d3:lite` does not start a chat; it prints how to serve the model. ### Request and response diff --git a/src/model_list.json b/src/model_list.json index 2aee90b81..03abc52f1 100644 --- a/src/model_list.json +++ b/src/model_list.json @@ -831,7 +831,7 @@ "file_url": "https://huggingface.co/api/models/FastFlowLM/D3-Lite-NPU2/tree/main", "ms_url": "https://modelscope.cn/models/amd/D3-Lite-NPU2", "size": 800000000, - "flm_min_version": "1.0.10", + "flm_min_version": "1.0.11", "files": [ "config.json", "model.q4nx",