diff --git a/docs/docs/benchmarks/d3_results.md b/docs/docs/benchmarks/d3_results.md index d32885c1b..a4d788b9a 100644 --- a/docs/docs/benchmarks/d3_results.md +++ b/docs/docs/benchmarks/d3_results.md @@ -7,7 +7,7 @@ nav_order: 20 ## ⚡ d3-lite on the NPU -Accuracy and latency of `d3:lite` (vllm-sr/d3-lite v3.1.3, package D3-Lite-NPU2) served by FastFlowLM 1.0.10. +Accuracy and latency of `d3:lite` (vllm-sr/d3-lite v3.1.3, package D3-Lite-NPU2) served by FastFlowLM 1.0.11. > **Note:** > - AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FLM's default NPU power mode (Performance). @@ -44,3 +44,43 @@ First request after `flm serve` is ready: 191 ms (steady state 168 ms over HTTP) Rows of 2,048 tokens and more: median of 15 runs (32,768: 3). From 1,536 tokens attention runs on the NPU (below that the CPU is faster); the first such request also loads the attention kernel (about 0.1 s) and from then on the decision model holds two NPU hardware contexts instead of one. + +### Images + +Accuracy on the Decision Index public vision suite (the 10 benchmarks that can be rebuilt without gated data; 60 +questions each, images capped at 1.6 MP as the suite's `decide` engine sends them): + +| Benchmark | Upstream BF16 | NPU (FLM) | Same answer | +|---|---:|---:|---:| +| CV-Bench | 81.7% | 85.0% | 91.7% | +| BLINK | 51.7% | 58.3% | 78.3% | +| RealWorldQA | 68.3% | 71.7% | 88.3% | +| CharXiv | 65.0% | 70.0% | 90.0% | +| InfographicVQA | 78.3% | 80.0% | 90.0% | +| Mind2Web | 61.7% | 63.3% | 83.3% | +| KIE (CORD + FUNSD) | 91.7% | 93.3% | 98.3% | +| Moderation (Hateful Memes) | 55.0% | 53.3% | 95.0% | +| R-Bench-M | 31.7% | 26.7% | 71.7% | +| MMMU-Pro vision | 16.7% | 15.0% | 58.3% | +| **All 600** | **60.2%** | **61.7%** | **84.5%** | + +Paired gold accuracy, NPU minus upstream BF16: +1.50 pt (95% CI [−0.92, +3.92] pt). Image grids and input token +counts equal upstream's on every question; the processor's pixels are within 2 grey levels of upstream's. + +Latency with and without an image, at the same total input length (one question, over HTTP, median of 5; 8,192 +tokens: 3 runs; 16,384 and 32,000: 1): + +| Input tokens | Text only (ms) | Text + one image (ms) | Image | +|---:|---:|---:|---| +| 516 | 266 | 483 | 640 × 480 (300 tokens) | +| 1,028 | 448 | 658 | 640 × 480 (300 tokens) | +| 2,052 | 830 | 1,651 | 1.6 MP (1,600 tokens) | +| 4,100 | 1,803 | 2,641 | 1.6 MP | +| 8,196 | 3,798 | 4,641 | 1.6 MP | +| 16,386 | 8,733 | 9,560 | 1.6 MP | +| 31,994 | 21,466 | 22,477 | 1.6 MP | + +An image costs a fixed amount on top of its tokens (decoding, resizing and the vision tower): about 0.2 s at +640 × 480 and 0.8 s at 1.6 MP. All questions of a request share one image encoding: a 1.6 MP image with 8 questions +takes 6.4 s, not 8 × 1.5 s. The first request with images loads the vision tower (about 40 ms) and two more NPU +hardware contexts. diff --git a/docs/docs/instructions/cli.md b/docs/docs/instructions/cli.md index 8be86336b..d02a0ea1d 100644 --- a/docs/docs/instructions/cli.md +++ b/docs/docs/instructions/cli.md @@ -368,6 +368,21 @@ See the ASR guide [here](https://fastflowlm.com/docs/models/whisper/) --- +### 🧭 Decision Model (System One) + +`--decision ` loads a decision model (`d3:lite`) next to the chat model, or alone, and serves `POST /v1/systemone` +(text and images, up to 32,768 input tokens per question; FastFlowLM 1.0.11 or later). +Decision models run in **Server Mode only** (`flm run d3:lite` prints how to serve it). + +```shell +flm serve --decision d3:lite # decision model only +flm serve qwen3.5:4b --decision d3:lite # next to a chat model +``` + +See the d3 guide [here](https://fastflowlm.com/docs/models/d3/) + +--- + ## 💻 Commands Inside CLI Mode Once inside the CLI, use the following commands. System commands always start with `/` (e.g., `/help`). diff --git a/docs/docs/instructions/server/index.md b/docs/docs/instructions/server/index.md index 67ba55dce..c71b660e8 100644 --- a/docs/docs/instructions/server/index.md +++ b/docs/docs/instructions/server/index.md @@ -27,6 +27,7 @@ FLM can keep one NPU model loaded per type at a time: - `asr` - `llm` - `embedding` +- `decision` (`--decision d3:lite`, the System One API; see [Decision model](#decision-model-system-one) below) Different model types can run together (for example, one LLM and one embedding model). @@ -47,6 +48,24 @@ flm serve lfm2:1.2b --port 52628 +## Decision Model (System One) + +A decision model answers typed questions about a state (choices, yes / no, scores) with a probability for every +option, over `POST /v1/systemone`; requests may carry images (base64 data URLs). It is served next to a chat model or +on its own: + +```shell +flm serve --decision d3:lite # decision model only +flm serve qwen3.5:4b --decision d3:lite # next to a chat model +``` + +`GET /v1/systemone/models` lists the loaded decision model with its input limit and modalities. Requests share the +NPU with the chat model and run one at a time, so a decision waits for a chat completion that is already running; a +second `flm serve` for the decision model does not avoid this (the two processes contend for the NPU). The decision +model takes one NPU hardware context, two once a prompt of 1,536 tokens or more arrives, and two more on the first +request with images; mind this when the chat model and others already use most of the device's 16. Request format, a smoke test and latency: +[Decision 3.0 (d3)](https://fastflowlm.com/docs/models/d3/). + ## Set Context Length at Launch The default context length for each model can be found [here](https://fastflowlm.com/docs/models/). diff --git a/docs/docs/models/d3.md b/docs/docs/models/d3.md index 9f3eda103..c912ca050 100644 --- a/docs/docs/models/d3.md +++ b/docs/docs/models/d3.md @@ -14,6 +14,7 @@ parent: Models - **Quantization:** Q4_K - **Max input:** 32,768 tokens per question (longer prompts get `max_length_exceeded`; nothing is truncated). One question takes about 0.4 s at 1,024 tokens, 1.6 s at 4,096 and 23 s at 32,768; the NPU serves one request at a time - **Inputs:** text and images (video: not supported) +- **FastFlowLM:** 1.0.11 or later d3 answers *System One* requests: a `state` (any JSON value) and one or more questions, each a `choice` (up to 255 options), a `noul` (no / yes) or a `score` (2–10 ordered levels). Every option gets a probability. It is the decision @@ -65,6 +66,54 @@ The reply on a Ryzen AI MAX+ 395 (numbers rounded to 3 places here; the server s "usage":{"input_tokens":342,"output_tokens":0}} ``` +### Smoke test + +Five commands check a new install end to end, text and images: + +```shell +flm pull d3:lite # downloads D3-Lite-NPU2 (about 0.8 GB, vision weights included) +flm serve --decision d3:lite # wait for "Decision model ready: d3-lite (d3:lite), max input 32768 tokens" +``` + +In a second terminal: + +```shell +curl -s http://127.0.0.1:52625/v1/systemone/models +# {"object":"list","data":[{"id":"d3:lite","object":"model","api":"systemone","max_input_tokens":32768, +# "modalities":["text","image"],"image_max_pixels":1638400}]} + +curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \ + -d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice", + "criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}' +# {"model":"d3-lite","answers":{"route":{"type":"choice","choice":"returns","probabilities":{"returns":0.98,... +``` + +A healthy install answers `"choice":"returns"` with `returns` near 0.98 in about 0.1 s. + +Then an image (Linux shell; the image is a 639 × 480 PNG from this repository): + +```shell +curl -sLo panda.png https://raw.githubusercontent.com/ROCm/FastFlowLM/main/src/tb_files/panda.png +printf '{"questions": {"animal": {"type": "choice", "instructions": "Which animal is in the photo?", + "criteria": {"dog": "A dog", "panda": "A panda", "cat": "A cat"}}}, + "images": ["data:image/png;base64,%s"]}' "$(base64 -w0 panda.png)" > image_request.json +curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" -d @image_request.json +# {"model":"d3-lite","answers":{"animal":{"type":"choice","choice":"panda","probabilities":{"dog":0.0004,"panda":0.999,... +# "usage":{"input_tokens":404,"output_tokens":0}} +``` + +On Windows (PowerShell): + +```powershell +Invoke-WebRequest https://raw.githubusercontent.com/ROCm/FastFlowLM/main/src/tb_files/panda.png -OutFile panda.png +$img = [Convert]::ToBase64String([IO.File]::ReadAllBytes("panda.png")) +$body = '{"questions": {"animal": {"type": "choice", "instructions": "Which animal is in the photo?", "criteria": {"dog": "A dog", "panda": "A panda", "cat": "A cat"}}}, "images": ["data:image/png;base64,' + $img + '"]}' +Invoke-RestMethod http://127.0.0.1:52625/v1/systemone -Method Post -ContentType "application/json" -Body $body | ConvertTo-Json -Depth 5 +``` + +A healthy install answers `"choice":"panda"` with `panda` near 0.999 and 404 input tokens, in about 0.4 s (the first +image request also loads the vision tower). `flm run d3:lite` does not start a chat; it prints how to serve the model. + ### Request and response | Field | Meaning | diff --git a/docs/models.md b/docs/models.md index 618ea8863..93ea913a1 100644 --- a/docs/models.md +++ b/docs/models.md @@ -11,7 +11,8 @@ sections: FastFlowLM curates the most requested families and publishes tuned manifests under `flm pull `. We validate every build on Ryzen™ AI laptops and provide matching cards—see them [here](/docs/models/). The catalog already covers models spanning Qwen3.8 (27B, MTP speculative decoding), Qwen3.6 (MoE), Qwen3.5 (Vision), Gemma4 (omni), Qwen3, GPT-OSS-20B (MoE), DeepSeek-R1, Whisper - (Audio), Hy-MT2 (Translation), MiniCPM-V 4.7 (Vision), MedGemma (Medical), and other leading open families. + (Audio), Hy-MT2 (Translation), MiniCPM-V 4.7 (Vision), MedGemma (Medical), Decision 3.0 (d3-lite, the System One + decision model of vLLM Semantic Router), and other leading open families. ctas: - label: "View model docs" href: "/docs/models/" @@ -34,6 +35,7 @@ sections: - "Whisper (Audio)" - "Hy-MT2 (Translation)" - "SmolVLA (Robotics)" + - "Decision 3.0 (System One)" - "MedGemma (Medical)" - "DeepSeek R-1" - "LLaMA 3.x" diff --git a/src/model_list.json b/src/model_list.json index 2aee90b81..03abc52f1 100644 --- a/src/model_list.json +++ b/src/model_list.json @@ -831,7 +831,7 @@ "file_url": "https://huggingface.co/api/models/FastFlowLM/D3-Lite-NPU2/tree/main", "ms_url": "https://modelscope.cn/models/amd/D3-Lite-NPU2", "size": 800000000, - "flm_min_version": "1.0.10", + "flm_min_version": "1.0.11", "files": [ "config.json", "model.q4nx",