Repository navigation
feat(d3): image input for d3:lite (draft; stacked on #817) - #829
Merged
Merged
Conversation
A byte-exact C++ port of the d3 (Decision 3.0) System One contract from the upstream package vllm-sr/d3-lite v3.1.3 (d3_format.py, d3_runtime.py, d3_server.py): request checks and their 422 texts, question validation, the prompt the model reads, the answer codes, the 255-way readout and the answer objects (choice / noul / score, confidence, legend, usage). The prompt shows non-string values as Python's json.dumps text, so src/common/decision/py_json.* keeps exactly what json.loads builds (integers of any size, duplicate-key order) and reproduces json.dumps / repr (float repr switch points, ', ' separators) and Python 3.12's compensated sum(). src/test/d3_contract replays fixtures rendered by the upstream package itself (308 golden requests + 1,000 fixed-seed fuzz bodies): prompts byte-identical, errors identical, every answer number bit-identical; with the tokenizer, every token id and the 255 answer codes too. CPU only; runs in ubuntu-build. Co-Authored-By: Claude <[email protected]>
d3_server hands the raw body to json.loads, which detects the encoding first: a UTF-8 BOM is accepted, UTF-16/32 bodies are decoded, and invalid UTF-8 is reported as CPython's UnicodeDecodeError text with its byte positions. Found by the HTTP parity rig (a BOM body: 200 upstream, 422 here). The contract test checks 12 byte-level cases against CPython 3.12's messages. Co-Authored-By: Claude <[email protected]>
Object members were added with PyValue::set(), which scans the members already read, so an object with n keys cost n^2/2 string compares: 200,000 keys (a 2.7 MB body) took 24.6 s, while the System One route holds the NPU queue. Keys are now indexed while an object is read; a repeated key still keeps its first position and takes the last value, like Python's dict. 200,000 keys: 108 ms. The contract test still passes (1,308 requests incl. 598 fuzz bodies with duplicate keys, 5,519 checks, 0 failures). Co-Authored-By: Claude <[email protected]>
Co-Authored-By: Claude <[email protected]>
Serves vllm-sr/d3-lite (Decision 3.0) on the NPU next to, or without, a chat
model, over the upstream package's own HTTP API:
flm serve [<llm>] --decision d3:lite # -d
POST /v1/systemone (alias /v1/decisions)
- AutoDecisionModel: the package's decision_config checks, the answer codes
re-derived from the tokenizer, the chat template cross-checked against the
contract with minja, the fp32 readout, warmup; the engine is libd3_npu
(vendored XRT + HRX builds from FastFlowLM_IRON, mm.xclbin as Qwen3.5-0.8B).
- Server: the route parses the raw body with Python json semantics and replies
exactly as d3_server (422 detail texts, invalid_questions, per-question
max_length_exceeded in a 200); NPU-exclusive like the other inference routes.
Replies may now carry their own HTTP status (__http_status / __http_body).
- Catalog: d3:lite -> D3-Lite-NPU2 (family d3), hidden from /api/tags and
/v1/models like embed-gemma; flm run/serve/bench d3:lite explains how to serve
it. Prompts over 4,096 tokens are refused per question, never truncated.
- src/test/d3_npu: AutoDecisionModel without HTTP (harness + perf driver).
- Docs: docs/docs/models/d3.md.
Windows: linked only where the engine library is vendored; until d3_npu.dll
ships the decision model refuses to load there.
Co-Authored-By: Claude <[email protected]>
The list operation of vLLM Semantic Router's vllm-sr/systemone@1 protocol
(systemone-compatible providers declare create and list_models), in the
router's own format: {"object": "list", "data": [{"id", "object", "api"}]}.
Empty while no --decision model is loaded.
Co-Authored-By: Claude <[email protected]>
FastFlowLM_IRON d3_npu now packs up to four 512-row GEMM tiles per pass by default: 16 questions of 330 tokens 156 -> 133 ms per question, 4 questions 150 -> 122, bit-exact against one prompt per pass. XRT and HRX builds. Co-Authored-By: Claude <[email protected]>
The harness links libd3_npu directly, so it compiles AutoDecisionModel with FLM_HAS_D3_NPU like the CMake build does. Its replies for the 308 golden requests match FastFlowLM_IRON's engine harness: top-1 identical on every question, probabilities within 3e-6 (fp32 readout summation order). Co-Authored-By: Claude <[email protected]>
Generated from the 4.9 measurements (d3-lite plan): gold accuracy on 6,824 Decision Index public-suite questions (upstream BF16 61.2%, NPU 60.6%), top-1 agreement, and per-request latency at 128-4,096 tokens and 4/16 questions. Co-Authored-By: Claude <[email protected]>
lib/<runtime>/ holds both the Linux .so and the Windows .lib of every engine, so testing for either file made the Windows build link d3_npu.lib, which does not ship yet (LNK1181). Test the platform's own library. Co-Authored-By: Claude <[email protected]>
The engine libraries now come from IRON CI (FastFlowLM_IRON#68, run 39245613348): Linux XRT/HRX .so and Windows XRT/HRX d3_npu.dll + .lib. The CI-built Linux library gives bit-identical hidden states to the locally validated build on all 311 golden prompts, and passes sweep_decision.py. The installers (WiX, Inno) ship d3_npu.dll. Co-Authored-By: Claude <[email protected]>
The Hugging Face tree of FastFlowLM/D3-Lite-NPU2 (commit cd6df2ea), which flm pull verifies downloads against. Checked: flm pull d3:lite into an empty model directory downloads and verifies all 9 catalog files (sha256 equal to the package manifest), and sweep_decision.py passes on the pulled copy. Co-Authored-By: Claude <[email protected]>
- warmup() encodes one pass at 2,048, 1,024 and 512 rows. The engine now keeps each bucket's sequences for its lifetime, and the scratch of buckets that fit one 4,096-row pass together (these three do). The old comment said buckets were kept when the engine kept only the last one, so the first one-question request rebuilt everything after warming 700 tokens. - AutoDecisionModel and the d3_npu test harness no longer cite the local plan document. Co-Authored-By: Claude <[email protected]>
…umbers - Engine libraries from FastFlowLM_IRON#68 PR Check run 39445931404 (d51e5a85): Linux XRT/HRX .so, Windows XRT/HRX .dll + .lib. They add per-bucket sequence/scratch caching and the blocked host attention. The CI Linux library gives bit-identical rows to the locally validated build on all 311 text goldens. - docs/models/d3.md: the example reply is now the server's real reply (rounded, and said so); before, it was made up. The docs also note that a decision waits for an in-flight chat completion on the same NPU (146 ms alone, 12.7 s during a 600-token completion), and that a second flm serve does not help (the processes contend for the NPU). - docs/benchmarks/d3_results.md regenerated from the new measurements: 4,096 tokens 3,652 -> 2,135 ms, 1,024 tokens 518 -> 430 ms; gold accuracy 60.5% (was 60.6%, within noise: paired sign test p = 0.44). Co-Authored-By: Claude <[email protected]>
- Max input 4,096 -> 32,768 tokens per question (AutoDecisionModel and the catalog entry); longer prompts still get max_length_exceeded, untruncated. - Engine libraries from FastFlowLM_IRON#68 PR Check run 39840309905 (569a9617), all four (Linux/Windows, XRT/HRX): prompts of 1,536 tokens or more run attention on the NPU, shorter ones on the CPU as before (bit-identical below 1,536). Long-prompt buckets are freed after each call. - src/xclbins/D3-Lite-NPU2/attn.xclbin: Qwen3.5's prefill attention made bidirectional (xclbins-Qwen3_5_bidirectional_attention_prefill, same CI run). Loaded on the first long prompt, so the decision model holds a second NPU hardware context only after one arrives; if it cannot be loaded, attention stays on the CPU. - One question: 2,048 tokens 765 ms (CPU attention: 906), 4,096 1.6 s (2.1), 16,384 8.4 s (21.1), 32,768 22.7 s (107). Over HTTP with this build: a 32,000-token two-question request answers in 43 s; short requests return the same answers before and after. - Accuracy, 1,078 Decision Index questions of 1,024+ tokens, NPU vs CPU attention: same top answer 97.8%, top-1 agreement with upstream BF16 88.2% vs 87.6%, gold accuracy +0.46 pt (95% CI -0.08 ... +1.01). - docs: max input, latency rows to 32,768 tokens. Co-Authored-By: Claude <[email protected]>
Per decision model, served with --decision: 15 stored requests answered by the upstream BF16 model (shape exact, top-1 agreement and mean |dp| within limits), choice / noul / score in one request, a 27-option question (two-letter code), invalid question (422), body that is not JSON (422), a prompt over the limit (max_length_exceeded) and the example of docs/docs/models/d3.md. SweepTask gains needs_decision. Kept out of the model-sweep matrix with embedding and audio, per that workflow's rule. Co-Authored-By: Claude <[email protected]>
…ON bodies - Each further decision model restarts the server with its own --decision tag; before, every model was measured against the first model's server. --no-server with more than one model is refused. - The docs case also checks that the server decides what the documented reply decides (choice, noul side, most likely score level); the shape alone let the documented numbers drift. - A reply body that is not JSON (e.g. a proxy's 5xx page) fails its case instead of raising out of the sweep. Co-Authored-By: Claude <[email protected]>
… tokens)
The System One contract library now takes images as upstream's server does
(d3_server + d3_runtime @ v3.1.3), all in order and before anything else:
- strict data URLs: data:image/{png,jpeg,jpg,webp};base64, validated base64
within 8,000,000 bytes, a PNG/JPEG/WebP file (format and size read from its
header, as Pillow identifies them; MPO too) of at most 16,000,000 pixels;
422 'images[i]: ...' with upstream's texts. Decoding and the integrity
check stay with the server, through prepare()'s per-image decode callback
so a bad image is reported at its index, as upstream loads them.
- the processor's smart_resize (round half to even, like Python's round(),
and the aspect-ratio refusal) -> grid and input tokens per image;
- the image prompt: one <|vision_start|><|image_pad|><|vision_end|> per image
before the user text; a literal <|image_pad|>/<|video_pad|> in a question's
text is invalid_question;
- tokenize() expands each placeholder to its image's tokens; the length limit
counts them (upstream's '(N for k image(s))' message). Videos stay refused.
Fixtures: images.jsonl (57 requests, generated by the upstream runtime with
gen_fixtures.py --images-only) and inline checks for the size limits, the
video refusal and the decode order.
Co-Authored-By: Claude <[email protected]>
…ine payload) - AutoDecisionModel decodes each request image right after the contract checks its header (upstream's order), resizes it to the grid the processor chose (bicubic, antialiased), rescales/normalizes it (0.5/0.5), repeats the frame for the temporal patch and cuts 16x16 patches in 2x2 merge order -- once per request -- and passes them to the engine (d3_image_payload); D3_DUMP_PIXELS=<dir> writes them for the rig. The load-time chat-template check covers the image prompt too. - ImageReader decodes WebP (libwebp 1.6.0, decoder only, static: the version Pillow ships), besides JPEG and PNG. - Catalog: vision_weight.q4nx is downloaded with d3:lite. - GET /v1/systemone/models states max_input_tokens, modalities (text, image) and image_max_pixels, as upstream's /v1/models does. - Docs: images in the request table and an example; videos stay refused. - Sweep: 'image' (a PNG answered) and 'image-broken' (422 images[0]). - Engine header synced with FastFlowLM_IRON#81 (d3_image_payload). The engine libraries with the vision tower come from FastFlowLM_IRON#81's CI and are vendored in a follow-up commit. Co-Authored-By: Claude <[email protected]>
…r refused The contract library reads headers only; for a request upstream refuses while decoding image i, the test now stands in for the server's decoder and requires the refusal to name image i (later images are not looked at). CI caught it on bad-truncated-then-type, where the header check alone reached images[1]. Co-Authored-By: Claude <[email protected]>
… from FastFlowLM_IRON#81 CI PR Check run 39858857837 (eaed6cf2): libd3_npu.so with the vision tower and image tokens, and xclbins-MM_host7_mt_variant_bias_QWEN3_5_0_8B-VISION-MM / xclbins-VisionAttention_hostAttention_whoel_array_QWEN3_5_0_8B-VISION-ATTENTION as D3-Lite-NPU2/vision_mm.xclbin / vision_attn.xclbin. Windows libraries follow when that job finishes. Co-Authored-By: Claude <[email protected]>
tawei-amd
force-pushed
the
feat/d3-lite-images
branch
from
October 22, 2026 21:59
9ba13cb to
3cf1273
Compare
…c resample) FLM's resize_bicubic_antialias_rgb_planar_avx512 differs from the d3 processor's resize (torchvision, uint8, bicubic, antialias -- which is Pillow's Image.resize(BICUBIC), bit for bit): on the image goldens, mean |delta| 0.002-0.019 and up to 0.30 on the normalized pixels wherever the image is resampled (exact where it is not). AutoDecisionModel now resamples as Pillow's Resample.c does: separable, horizontal then vertical, 22-bit fixed-point coefficients of the a = -0.5 cubic widened by the downscale factor, each pass rounded and clipped to 8 bits; then (x / 255 - 0.5) / 0.5. A NumPy port of this code matches Pillow pixel for pixel on the goldens (upscaled, shrunk, portrait, 3000x60 strip, palette). Co-Authored-By: Claude <[email protected]>
…N#81 CI PR Check run 39858857837 (eaed6cf2), as the Linux libraries before it. Co-Authored-By: Claude <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, stacked on #817 (→ #816 → #815). Image input for the d3 decision model (
flm serve --decision d3:lite), asupstream
vllm-sr/d3-litev3.1.3 takes it:imagesis a list of base64 data URLs (PNG, JPEG, WebP) shared by everyquestion of a request. Engine: AMD-ROCm-Internal/FastFlowLM_IRON#81 (libraries and vision bitstreams vendored from its CI).
What it does:
src/common/decision): upstream's strict image checks and 422 texts, the processor's resize (grid, inputtokens; Python's half-even rounding), image prompts, placeholder expansion, the length limit. Fixtures from the
upstream runtime (
images.jsonl, 57 requests).AutoDecisionModel: decodes each image (WebP via libwebp 1.6.0, Pillow's version), resizes it exactly as the processordoes (Pillow's bicubic resample), normalizes and patches it once per request, and hands it to the engine.
vision_weight.q4nx;/v1/systemone/modelsstates modalities andimage_max_pixels; docs; sweepimage/image-brokencases.Results (Strix Halo, this PR's CI build):
pixels within 0.016 (≤ 2 grey levels); same top answer 79/81.
61.7% vs upstream BF16 60.2% (+1.5 pt, 95% CI −0.9 … +3.9), same top answer 84.5%, input tokens equal 600/600.
encode); text-only unchanged (109 ms). The vision tower loads on the first image request (two more NPU hardware
contexts).
🤖 Generated with Claude Code