ggml-openvino : make op support a property of the registry - #307
Draft
cavusmustafa wants to merge 24 commits into
Draft
cavusmustafa wants to merge 24 commits into
cavusmustafa wants to merge 24 commits into
Conversation
cavusmustafa
force-pushed
the
ov-supports-op-robust
branch
from
September 2, 2026 23:43
b800043 to
abf6caf
Compare
ravi9
force-pushed
the
dev_backend_openvino
branch
from
September 3, 2026 00:04
ec98377 to
fd9bc04
Compare
Assisted-by: Claude Sonnet
…state The stateful path seeds its KV state from ggml's cache when the decode position is ahead of what the state holds. That only works when ggml's cache is a plain prefix, where cell i holds position i. A sliding-window layer keeps just the last n_swa positions and drops the rest, so past the window cell i no longer holds position i and the seeded state is wrong. Slicing the state to the decode position also had no bounds check, so a position past the end surfaced as a bare ov::Exception from the ROI constructor (llama_decode ret = -3, with no reason given at default verbosity). Refuse both cases with a clear message instead, and refuse on the compile path too, where a new model starts with an empty state and so can only serve a sequence from its beginning. Reproducible with llama-bench -d, which restores a saved sequence state rather than recomputing the depth prefill. Assisted-by: Claude Opus 5
The stateful path reinterprets ggml's KV buffer [1, 1, seq, n_heads_kv * head_size] as [1, seq, n_heads_kv, head_size]. The head size is already taken from the tensor's own combined dim, because gemma-4 varies it per layer type, but the head count still came from a model-level scalar that compute_llm_params() overwrites per attention node, so it ended up holding whatever the last layer said. gemma-4 varies the head count per layer too: 12B has 8 x 256 sliding layers and 1 x 512 full layers, 31B has 16 x 256 and 4 x 512. So 40 of 12B's 48 layers were split as 1 x 2048 instead of 8 x 256, and attention read the state with the wrong head split - both models decoded garbage on CPU and GPU. E2B is unaffected, its head count is 1 everywhere. Record the count per layer instead and look it up by the cache_k_l<N> leaf name. Key it by layer, not by layer type: the sliding/full classification comes from cache extents, which tie at a small -c, while the head count does not. The stateful state trim now derives its sequence axis per state for the same reason, since pass::KVStateSeqAxis matches per state on the head count. Assisted-by: Claude Opus 5
pass::KVStateSeqAxis was limited to states with a single KV head, where moving the sequence axis from dim 1 to dim 2 is a pure metadata change. The limit was also based on a measurement showing no gain for a multi-head model, but that was taken at depth 0, which is the one depth where this change does nothing. With several heads the pass does more than move metadata: it drops the reader side transpose of the whole accumulated state, which the graph otherwise redoes every token at a cost that grows with the context length, and replaces it with a transpose of the single new row. Measured on GPU, tg128, alternating arms: gemma-4-12B 6.27 -> 9.11 t/s at depth 8192 (stateless is 7.69, so stateful now wins at depth instead of losing), Llama-3.2-1B 47.8 -> 59.6 t/s. Both are within noise at depth 0, which is why the earlier check saw nothing. The state refill needs the rows copied rather than reinterpreted now: ggml stores [seq][n_heads_kv * head_size], and a relayout state with several heads is a different element order. Without that, a refill would seed wrong data - it is reachable today through llama-bench -d. Assisted-by: Claude Opus 5
ravi9
force-pushed
the
dev_backend_openvino
branch
from
September 3, 2026 21:10
fd9bc04 to
33237ab
Compare
ci:ggml-openvino: disable failing tests
ggml-openvino : support ggml_rope_set_offset and simplify op support gating
Gemma4 MOE Fix / Qwen MOECompressed
cavusmustafa
force-pushed
the
ov-supports-op-robust
branch
from
September 4, 2026 23:47
abf6caf to
d50cef8
Compare
ravi9
force-pushed
the
dev_backend_openvino
branch
from
September 14, 2026 17:49
37b4e1d to
926ef68
Compare
wine99
force-pushed
the
dev_backend_openvino
branch
from
September 16, 2026 08:07
0365b00 to
37b53fd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Created this PR as an initial draft for suports_op changes. CIs and supported models should be tested. Below is the AI generated description:
supports_op() could accept a node the backend then failed to translate. The conditions lived in a switch in ggml-openvino.cpp that 19 of the 54 registered ops never reached, so those ops were accepted unchecked and failed later inside the translator - ARGSORT on an unknown sort order, TRI on an out-of-range triangle type, PAD on an empty input.
Move the per-op conditions into openvino/op_support.{h,cpp} and give every registry entry a support rule beside its translator:
The constructor is deliberate. Without it OpEntry is an aggregate and {translate_foo} compiles with supports silently null, which would defeat the only guarantee the type exists to provide. A registry entry now cannot be added without stating when the op may be used, and that is checked by the compiler rather than by review.
The 20 case groups of is_op_supported_case() are migrated unchanged, so behaviour is identical: test-backend-ops -b OPENVINO0 gives 3386/3386, 17783 not supported, 0 FAIL, exit 0, matching the base exactly. New rules for ARGSORT, PAD and TRI turn three translator throws into gate declines.
Two categories in op_support.cpp are marked for reviewers because they are not statements about what an op means: declines that depend on the device name, which work around specific plugin defects and should be deleted when those are fixed, and declines keyed on a tensor name or an exact test shape.
Also: ggml-openvino.cpp loses 427 lines and gains 36; the empty ops_not_support_view_input set was dead code and is removed.
Note GGML_OP_SSM_CONV's rule has its "return true" commented out, so its stated intent of keeping the op on CPU is not implemented. Migrated as-is rather than changed.
Assisted-by: Claude Opus 5