Skip to content

fix: pass chat_template_kwargs to the chat template as sent - #260

Merged
solderzzc merged 2 commits into
SharpAI:mainfrom
HamedShams:fix/chat-template-kwargs-passthrough
Nov 6, 2026
Merged

solderzzc merged 2 commits into
SharpAI:mainfrom
HamedShams:fix/chat-template-kwargs-passthrough

Conversation

@HamedShams

@HamedShams HamedShams commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Closes #259.

chat_template_kwargs decoded as [String: Bool], so a string or a number anywhere in it answered the whole request with 400, and every key but enable_thinking was dropped before the template saw it. enable_thinking is the Qwen and Gemma name for the thinking switch; other families SwiftLM runs read their own: DeepSeek-V3.1 thinking, gpt-oss reasoning_effort and model_identity, MiniMax-M2 model_identity. None of those could be set per request.

What changes

  • chat_template_kwargs decodes as any JSON value, through the same AnyCodable the tool parameters use, and goes into the template context whole (chatTemplateContext). Nulls inside it are stripped by sanitizeForJinja on the way to the template, as they already are for tools.
  • enable_thinking keeps its precedence exactly as it was: the top-level field, then the kwarg, then --thinking. The Gemma 4 tool-call override is untouched.
  • enable_thinking inside the kwargs must still be a boolean. A string or a number there is answered with 400 invalid_request_error naming chat_template_kwargs.enable_thinking, which is what the whole field answered before.
  • README and docs/api-response-formats.md gain the chat_template_kwargs row.

Requests that send only enable_thinking, or no kwargs, render the same prompt as before.

Verified

  • ChatRequestParsingTests (4 new): a kwargs object with a boolean, a string, a number and a nested array decodes and keeps its types; an absent field stays nil; chatTemplateContext carries every kwarg and sets enable_thinking to the decided value; with no kwargs the context holds enable_thinking alone. The decode test is red on main (Expected to decode Bool but found a string instead).
  • tests/test-contract.sh gained block 17: string, number, boolean and nested kwargs are accepted (red on main: 400), and a non-boolean enable_thinking is still a 400 that names the param.
  • swift test --filter SwiftLMTests: 342 tests, 0 failures. tests/test-contract.sh against mlx-community/Qwen2.5-0.5B-Instruct-4bit (M1 Max, macOS 27.2): 28 passed, 0 failed, 2 skipped (the two capability skips for the tiny model).

Second commit: the CI flake in HFModelSearchTests

CI failed on the first push in SwiftBuddyTests.HFModelSearchTests.testStrictMLXFilterDisabled with Search failed: cancelled, the same failure as on main after #257, and it does the same on my machine. The cause is in HFModelSearchService: cancelling the debounce task does not stop a page fetch that is already awaiting the network (and a loadMore() fetch is not in that task at all), so the replaced fetch's request fails with URLError.cancelled, the generic catch records it as Search failed: cancelled, and the fetch goes on to clear isSearching and append its results, all over the search that replaced it. In the suite that happens whenever an earlier test leaves a fetch in flight, which the pagination test does when it skips under rate limiting.

The fix: each search() bumps a generation, a fetch publishes only while it is the current one, and a cancelled request is not an error to report. testSupersededSearchPublishesNothing pins the scenario, in the shape of its neighbours (live API, skip when rate limited). Locally testStrictMLXFilterDisabled failed in 2 of 5 runs on main before the rate limit turned the rest into skips, and passes 5 of 5 with the fix.

The field decoded as [String: Bool], so a string or a number anywhere
in it answered 400, and every key but enable_thinking was dropped
before the template saw it. Templates read other names and types:
DeepSeek-V3.1 `thinking`, gpt-oss `reasoning_effort` and
`model_identity`, MiniMax-M2 `model_identity`.

The kwargs now decode as any JSON value (AnyCodable, as tool parameters
already do) and go into the template context whole. enable_thinking
keeps its precedence (top-level field, then the kwarg, then --thinking)
and must still be a boolean; a request that sends it as anything else
gets a 400 naming the param, as the whole field did before.

Closes SharpAI#259.
Cancelling the debounce task does not stop a page fetch that is already
awaiting the network, and a loadMore() fetch is not in that task at all.
The replaced fetch's request fails with URLError.cancelled, which the
generic catch turned into errorMessage = "Search failed: cancelled",
and the fetch then set isSearching = false and appended its results,
all over the search that replaced it. That is the CI flake in
HFModelSearchTests.testStrictMLXFilterDisabled: whenever an earlier
test leaves a fetch in flight (the pagination test skips mid-fetch under
rate limiting), the next search inherits the stale error.

Each search bumps a generation; a fetch publishes only while it is the
current one, and a cancelled request is not an error to report.
@solderzzc

Copy link
Copy Markdown
Member

Thanks for the fix, and for the detailed write-up in #259. The reproduction table and the list of families that read their own kwarg names made the scope very clear. Reviewing it now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chat_template_kwargs only takes booleans, and only enable_thinking reaches the template

2 participants