Skip to content

docs: document ModelOption.THINKING behaviour for Granite - #1645

Draft
planetf1 wants to merge 3 commits into
mainfrom
issue-1638
Draft

planetf1 wants to merge 3 commits into
mainfrom
issue-1638

Conversation

@planetf1

Copy link
Copy Markdown
Contributor

ModelOption.THINKING already works for controlling Granite 4.2 reasoning on the native Ollama, OpenAI, and LiteLLM backends — including the "low"/"medium"/"high" string levels — but nothing in the docs said so. The one substantive treatment that existed (openai.md's thinking section) was framed entirely around Qwen3 on vLLM using raw extra_body/enable_thinking, which pointed readers at the lower-level workaround instead of the higher-level ModelOption.THINKING API that's actually the preferred entry point when a backend honours it.

This expands the vague one-line ModelOption.THINKING table row in configure-model-options.md into a proper subsection with a per-backend behaviour table (native Ollama / OpenAI-compatible / HF), documents Granite 4.2's "low"-only effort distinction ("medium"/"high" are accepted but behave like True) and its default-on-when-unset behaviour, and adds a working example (docs/examples/thinking_mode.py) exercising True/False/"low" against a local granite4.2:3b Ollama model. It also cross-links openai.md's Qwen3/vLLM section to point at ModelOption.THINKING as the preferred portable mechanism, keeping the existing extra_body content as the documented fallback for backends/params it doesn't cover.

The LocalHFBackend row is documented as "not supported" (string values are silently dropped today) with a link to #1636, since the fix (#1639) hasn't merged yet. That row should be updated once #1639 lands.

What's out of scope

Verification

  • uv run python docs/examples/thinking_mode.py — ran live against granite4.2:3b on local Ollama. Confirmed THINKING=True produces a long reasoning trace, THINKING="low" a short one, THINKING=False none, matching what's documented.
  • npx markdownlint-cli "docs/docs/**/*.md" --config docs/docs/.markdownlint.json — clean on both edited files (one pre-existing, unrelated MD060 finding on a table this PR didn't touch).
  • uv run ruff format/ruff check/mypy on the new example — clean.
  • pre-commit run on the changed files — all hooks pass.

Fixes #1638

…mple

Adds a per-backend reasoning/thinking-mode reference table (native Ollama,
OpenAI/LiteLLM, HF) to configure-model-options.md, including Granite 4.2's
"low"-only effort distinction and its default-on behaviour, and cross-links
it from openai.md's existing Qwen3/vLLM extra_body section as the preferred
portable entry point. Adds docs/examples/thinking_mode.py demonstrating
THINKING=True/False/"low" against a local Ollama Granite model, verified
live against granite4.2:3b.

Fixes #1638

Assisted-by: Claude Code
Signed-off-by: Nigel Jones <[email protected]>
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 10, 2026
Documents ModelOption.THINKING string levels and result.thinking as
supported on LocalHFBackend too, matching the state after #1639 (HF string
forwarding fix) and #1616 (HF <think> tag parsing) merge. Both are still
open — this docs PR should merge after them, or the LocalHFBackend row will
describe behaviour that isn't live yet.

Assisted-by: Claude Code
Signed-off-by: Nigel Jones <[email protected]>
@planetf1

Copy link
Copy Markdown
Contributor Author

Updated to document the `LocalHFBackend` row assuming #1639 (HF string-level forwarding) and #1616 (HF `` tag parsing) merge — both now describe HF as matching Ollama/OpenAI behaviour rather than "not supported".

Merge-order note: this PR should merge after #1639 and #1616, otherwise the `LocalHFBackend` row describes behaviour that isn't live on `main` yet.

- Qualify the OpenAIBackend/LiteLLM `False` cell: real OpenAI reasoning
  models and non-Ollama LiteLLM targets never receive `reasoning_effort=
  "none"` (openai.py's server-type guard, litellm.py's ollama-prefix guard),
  so `False` does not actually disable thinking there.
- Add an inline callout marking the LocalHFBackend string-forwarding and
  result.thinking claims as contingent on #1639/#1616 merging, instead of
  only noting it in the PR description/comment.
- Correct "same mechanism as the OpenAI backend" — HF forwards through a
  gated chat-template variable, OpenAI sends an ungated top-level param.
- Note the runtime-forwarding dependency for the cross-backend Granite
  claim, add a `> Full example:` link, fix the unresolvable bare "#1617"
  reference, add a non-determinism note to the new code block, drop the
  unnecessary `qualitative` marker, and fix US-English spelling in new
  content per CONTRIBUTING_DOCS.md.
- Add cheap assertions to docs/examples/thinking_mode.py pinning the
  documented per-arm behaviour; reran live against granite4.2:3b, all pass.

Assisted-by: Claude Code
Signed-off-by: Nigel Jones <[email protected]>
@planetf1

Copy link
Copy Markdown
Contributor Author

Code review — 3-reviewer panel, all findings addressed

Ran an independent tier-1/tier-2/tier-3 panel against this diff. All three reviewers verified and confirmed the same central finding independently, plus several narrower ones. Pushed a follow-up commit (f1a53834) fixing all confirmed items:

WARNING (all 3 reviewers, confirmed against source) — the OpenAIBackend/LiteLLM False cell claimed "Disables thinking" unconditionally. Verified against openai.py:747-749 (real OpenAI server type never gets reasoning_effort="none" — that branch is explicitly guarded out) and litellm.py:428-439 (only sent when "ollama" is in the model-id prefix). So False does not disable thinking for real OpenAI reasoning models or non-Ollama LiteLLM targets. Fixed: the cell now spells out the real-OpenAI/non-Ollama exception.

WARNING — the LocalHFBackend claims (string forwarding, result.thinking population) describe the state once #1639/#1616 merge, but the doc prose had zero inline signal of that — only the PR description said so. Fixed: added an inline > **Depends on #1639 and #1616** callout so the doc itself doesn't silently go stale if merged out of order.

WARNING — the unresolvable bare #1617 reference in openai.md's new callout (a published-site reader can't follow it). Fixed: reworded.

WARNING — no > **Full example:** link to docs/examples/thinking_mode.py, per CONTRIBUTING_DOCS.md's convention, leaving the inline snippet to drift from the real file. Fixed: added.

SUGGESTIONs fixed: "same mechanism as the OpenAI backend" was inaccurate (HF forwards through a gated chat-template variable; OpenAI sends an ungated top-level param) — reworded; "This holds across all three backends" now notes the runtime-forwarding dependency; added a non-determinism note to the new code block per doc convention; added cheap assertions to the example pinning the documented per-arm behaviour (reran live against granite4.2:3b, all pass).

NITs fixed: dropped the unnecessary qualitative pytest marker (no quality assertion existed at the time it was added); fixed US-English spelling (behaviourbehavior) in new content per CONTRIBUTING_DOCS.md's house style.

Not addressed (correctly scoped FOLLOW-UP, pre-existing, unrelated to this diff): missing **See also:** footer and a missing .md extension on an unrelated pre-existing link in configure-model-options.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: THINKING/reasoning-effort control is undocumented for Granite, and openai.md's example steers readers to the wrong mechanism

1 participant