Conversation
…mple Adds a per-backend reasoning/thinking-mode reference table (native Ollama, OpenAI/LiteLLM, HF) to configure-model-options.md, including Granite 4.2's "low"-only effort distinction and its default-on behaviour, and cross-links it from openai.md's existing Qwen3/vLLM extra_body section as the preferred portable entry point. Adds docs/examples/thinking_mode.py demonstrating THINKING=True/False/"low" against a local Ollama Granite model, verified live against granite4.2:3b. Fixes #1638 Assisted-by: Claude Code Signed-off-by: Nigel Jones <[email protected]>
Documents ModelOption.THINKING string levels and result.thinking as supported on LocalHFBackend too, matching the state after #1639 (HF string forwarding fix) and #1616 (HF <think> tag parsing) merge. Both are still open — this docs PR should merge after them, or the LocalHFBackend row will describe behaviour that isn't live yet. Assisted-by: Claude Code Signed-off-by: Nigel Jones <[email protected]>
|
Updated to document the `LocalHFBackend` row assuming #1639 (HF string-level forwarding) and #1616 (HF `` tag parsing) merge — both now describe HF as matching Ollama/OpenAI behaviour rather than "not supported". Merge-order note: this PR should merge after #1639 and #1616, otherwise the `LocalHFBackend` row describes behaviour that isn't live on `main` yet. |
- Qualify the OpenAIBackend/LiteLLM `False` cell: real OpenAI reasoning models and non-Ollama LiteLLM targets never receive `reasoning_effort= "none"` (openai.py's server-type guard, litellm.py's ollama-prefix guard), so `False` does not actually disable thinking there. - Add an inline callout marking the LocalHFBackend string-forwarding and result.thinking claims as contingent on #1639/#1616 merging, instead of only noting it in the PR description/comment. - Correct "same mechanism as the OpenAI backend" — HF forwards through a gated chat-template variable, OpenAI sends an ungated top-level param. - Note the runtime-forwarding dependency for the cross-backend Granite claim, add a `> Full example:` link, fix the unresolvable bare "#1617" reference, add a non-determinism note to the new code block, drop the unnecessary `qualitative` marker, and fix US-English spelling in new content per CONTRIBUTING_DOCS.md. - Add cheap assertions to docs/examples/thinking_mode.py pinning the documented per-arm behaviour; reran live against granite4.2:3b, all pass. Assisted-by: Claude Code Signed-off-by: Nigel Jones <[email protected]>
Code review — 3-reviewer panel, all findings addressedRan an independent tier-1/tier-2/tier-3 panel against this diff. All three reviewers verified and confirmed the same central finding independently, plus several narrower ones. Pushed a follow-up commit ( WARNING (all 3 reviewers, confirmed against source) — the WARNING — the LocalHFBackend claims (string forwarding, WARNING — the unresolvable bare WARNING — no SUGGESTIONs fixed: "same mechanism as the OpenAI backend" was inaccurate (HF forwards through a gated chat-template variable; OpenAI sends an ungated top-level param) — reworded; "This holds across all three backends" now notes the runtime-forwarding dependency; added a non-determinism note to the new code block per doc convention; added cheap assertions to the example pinning the documented per-arm behaviour (reran live against NITs fixed: dropped the unnecessary Not addressed (correctly scoped FOLLOW-UP, pre-existing, unrelated to this diff): missing |
ModelOption.THINKINGalready works for controlling Granite 4.2 reasoning on the native Ollama, OpenAI, and LiteLLM backends — including the"low"/"medium"/"high"string levels — but nothing in the docs said so. The one substantive treatment that existed (openai.md's thinking section) was framed entirely around Qwen3 on vLLM using rawextra_body/enable_thinking, which pointed readers at the lower-level workaround instead of the higher-levelModelOption.THINKINGAPI that's actually the preferred entry point when a backend honours it.This expands the vague one-line
ModelOption.THINKINGtable row inconfigure-model-options.mdinto a proper subsection with a per-backend behaviour table (native Ollama / OpenAI-compatible / HF), documents Granite 4.2's"low"-only effort distinction ("medium"/"high"are accepted but behave likeTrue) and its default-on-when-unset behaviour, and adds a working example (docs/examples/thinking_mode.py) exercisingTrue/False/"low"against a localgranite4.2:3bOllama model. It also cross-linksopenai.md's Qwen3/vLLM section to point atModelOption.THINKINGas the preferred portable mechanism, keeping the existingextra_bodycontent as the documented fallback for backends/params it doesn't cover.The
LocalHFBackendrow is documented as "not supported" (string values are silently dropped today) with a link to #1636, since the fix (#1639) hasn't merged yet. That row should be updated once #1639 lands.What's out of scope
configure-model-options.md/backends-and-configuration.mdduplication noted in the issue —backends-and-configuration.md's table row already links elsewhere and wasn't called out in the issue's proposed changes, so it's left as-is.result.thinkingparsing onLocalHFBackend(fix(hf): parse <think> tags into mot.thinking on LocalHFBackend #1616, open PR) and the HF string-drop fix (fix(hf): forward string THINKING levels as reasoning_effort, not low_effort #1639, open PR) are both separate in-flight PRs; this PR only documents current behaviour and links to them.Verification
uv run python docs/examples/thinking_mode.py— ran live againstgranite4.2:3bon local Ollama. ConfirmedTHINKING=Trueproduces a long reasoning trace,THINKING="low"a short one,THINKING=Falsenone, matching what's documented.npx markdownlint-cli "docs/docs/**/*.md" --config docs/docs/.markdownlint.json— clean on both edited files (one pre-existing, unrelated MD060 finding on a table this PR didn't touch).uv run ruff format/ruff check/mypyon the new example — clean.pre-commit runon the changed files — all hooks pass.Fixes #1638