Repository navigation
Hold a closed Cohere2 MoE, Command R7B or Seed-OSS thought; stop holding Ministral and Solar Open - #1560
Merged
Conversation
…ing Ministral and Solar Open With thinking off, Cohere2 MoE's `reasoning: false` default closes a thought llama.cpp leaves open, and the Command R7B, Cohere2 MoE and Seed-OSS handlers close one a template opens, but their parse still takes text before the end tag as reasoning. They now mark the result, and a Cohere2 MoE hold also ends at `<|START_TEXT|>` or `<|START_ACTION|>`, where llama.cpp ends the thought. Ministral and Solar Open no longer mark: their parse takes a thought only after its start tag. Tests pin that template-closed and non-forced Hermes results stream unheld. Closes #1553. Closes #1557.
This was referenced Oct 28, 2026
leehack
marked this pull request as ready for review
October 28, 2026 21:25
Contributor
|
Chat app preview removed for |
13 tasks done
leehack
added a commit
that referenced
this pull request
Oct 29, 2026
…pen (#1569) * Match llama.cpp's Seed-OSS and Cohere2 MoE parse when no thought is open A Seed-OSS reply to a prompt that opens no thought now has one only when it starts with `<seed:think>`, kept as generated up to the first end tag; later thinking tags are content. A Cohere2 MoE reply whose thought a handler closed for thinking off parses as after the forced-open prompt llama.cpp renders, so `Plan<|START_TEXT|>Sunny.` gives reasoning Plan, and thinking tags inside a text or action are text. Response-schema Cohere2 MoE renders close the thought with thinking off, as the Cohere2 MoE handler does. `ChatTemplateEngine.parse` takes `thoughtClosedForThinkingOff` so a final parse matches the stream. Tests pin the stream hold release from #1560. Closes #1561. Closes #1566. * Keep the closed-thought parse flag internal ChatTemplateEngine.parse no longer takes thoughtClosedForThinkingOff: the mark it depends on is internal. The stream parser derives thinkingForcedOpen with the internal parsesThinkingForcedOpen instead. * Keep the public Seed-OSS parse of an opened thought; pin audit gaps ChatTemplateEngine.parse cannot see whether a template opened the thought, so a Seed-OSS reply that ends one it did not open parses as before, as llama.cpp parses it after such a template. The stream parses its renders through the internal parseRenderedOutput. Only ASCII whitespace may precede a leading <seed:think>, as in llama.cpp. Tests cover a stray end tag before a Cohere2 MoE body, an empty held reply, a bare start tag, and the response schema close on other formats and the multimodal path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
<|END_THINKING|>as content. llamadart renders the template withreasoning: false, which closes the thought, while llama.cpp620d2225sets noreasoning, keeps it open and parses the leading thought as reasoning. The Cohere2 MoE handler now marks that result (withThoughtClosedForThinkingOff), as do the Command R7B, Cohere2 MoE and Seed-OSS handlers where they append the end tag to a thought the template opened. Marks apply only where llamadart closed the thought: a caller's ownreasoning: falseorreasoning_effort: nonecloses it in llama.cpp too, and llama.cpp then parses no leading thought, so those stay unmarked.<|START_TEXT|>or<|START_ACTION|>, where llama.cpp ends the open thought (cohere2moe.cpp:until_one_of({THINK_END, TEXT_START, ACTION_START})). Without this, every tagless thinking-off Cohere2 MoE reply (the normal case, since the prompt closes the thought) would arrive only at the end.Plan[/THINK]Sunny.as content; Solar Open rejects it), so holding gained nothing. The registry test now parses with the render's own parser, which is how it missed this.Production-readiness scope
enableThinking: falseon Cohere2 MoE (North) templates, streamedchunk.textmatches the final message when the model still writes a thought: content waits for a thinking tag,<|START_TEXT|>or<|START_ACTION|>, then streams live. Custom Command R7B or Seed-OSS templates that open the thought with thinking off behave like the #1532 templates. Ministral and Solar Open templates that open[THINK]/<|think|>with thinking off stream as they arrive again (no on-disk template reaches this).Plan</seed:think>Sunny.as reasoning where llama.cpp keeps it as content (the stream already matches llama.cpp there). Cohere2 MoEPlan <|START_ACTION|>...<|END_ACTION|><|END_THINKING|>Done.and multi-end-tag replies stay stream != parse as on base.Completeness checklist
template-engine-internals.md,chat-template-and-parsing.md,skills/llamadart-chat-streaming/SKILL.md, both changelogs; README/examples/support matrix N/A, no capability change.)PR type guidance
reasoning: falsedefault makes the Cohere2 MoE template close the thought. Ministral and Solar Open marked unconditionally although their parse never takes an untagged leading thought; the registry test parsed without the render's parser. Tests:620d2225with and without the tool, streamed whole, bytewise, in 3-char chunks and at every 2-chunk split, thinking off and on; the mark survives required-tool grammar and token counting.reasoning: false/reasoning_effort: nonekwargs unmarked,reasoning: truemarked); the on-disk Command R7B template, which closes the thought itself as in llama.cpp, unmarked.<|START_TEXT|>streams as it arrives; Command R7B waits for a thinking tag; other formats do not release at Cohere openings.Test Plan
dart run tool/prepare_workspace.dartdart format --output=none --set-exit-if-changed .dart analyzedart test -p vm -j 1 --exclude-tags local-only(isolatedTMPDIR): +9664 ~5 at merged headc0449714a, all passed (+9522 ~123 before the merge)dart test -p chrome --exclude-tags local-only(CI). Locally:dart test -p chrome test/unit/core/template test/unit/core/engine/chat_completion_stream_parser_test.dart+1135, all passedcheck_platform_boundaries.dartOK;./tool/testing/run_template_parity_suites.sh(detection +74, diagnostics +93, unit template +1615, upstream chat +10, GBNF +58, all passed); oracle and mutation runs below.Oracle: the #1555 audit harness (llama.cpp
620d2225common_chat_session, 128,572 rows over 94 templates; every row streamed whole, paced bytewise and at every 2-split). Base49c90e45c, head lib3151c7a94(the merge ofmainadds only #1551, no template or stream code).Latency: content arrival moves only on held Cohere2 MoE rows (944); every move is a reply with no thinking tag or Cohere opening arriving at EOF, the #1555 trade-off. No row arrives later than upstream where upstream emits content.
Mutants (29, VM template/engine suites): R7B append does not mark; Cohere default close never marks / ignores the user
reasoningkwarg / ignoresreasoning_effort/ ignores the thinking flag / ignores the prompt end; Cohere2 MoE does not open by default; R7B opens by default; Seed-OSS does not mark; no Cohere release openings; Cohere openings for every format; scan misses a split marker; release yields only the new token; hold drops text at EOF; releases only at the end tag / only at the start tag; Ministral and Solar Open handler files reverted to base (mark again); audit X3, X4, X10, A-S1, A-S2; six handlers do not mark. 29 of 29 killed (X4 and X10 by the new template-closed and registry pins).Matrix Evidence
static-format-analyzeroot-chrometemplate-parityrun_template_parity_suites.shstructured-output-adversarial620d2225oraclehigh-risk-exact-head-independent-qa/opt/UnitySrc/personal/llama/llamadart-evidence/pr-1560/evidence-1560-c044971.jsonrelease-metadata-verificationlitert-lm-json-schemaReview Notes
c0449714a, CI pending.High-risk regression review
620d2225emissions (oracle and the new Cohere2 MoE fixture) stand in