[zaneni6]

Problem

Model Sweep run 39454907120 failed on sweep-windows-hrx (llm): qwen3-tk:4b round 2 timed out after 600 s, although the model decodes at ~20 tok/s.

The server log shows the reply ending in ### \xF0\x9F\x94, 3 of the 4 bytes of an emoji, at the 512-token cap. Then:

1. finalize() sent the leftover bytes, including the incomplete character.
2. json::dump() threw on the invalid UTF-8 (nlohmann's default strict mode).
3. ostream.finalize() was outside any try. The outer catch sent a plain error response, but the stream had already started, so the closing chunk was never sent and the socket stayed open.
4. The client waited out its full timeout.

This happens whenever a reply is cut off by the token limit or a cancel partway through a multi-byte character. That is why the sweep fails seemingly at random.

Changes

Server

• Streaming (/v1/chat/completions, /v1/completions, /api/chat, /api/generate): drop an incomplete trailing UTF-8 sequence in finalize().
  • Ollama streams now always end with the final chunk carrying done_reason and counts.
  • /v1/completions no longer ends the stream before finish_reason.
• OpenAI streaming handlers: an exception after the first chunk now goes out as an error event plus [DONE], which ends the stream. A failure while finishing the stream is also logged.
• Non-streaming chat (/v1/chat/completions, /api/chat): trim the incomplete tail from the generated text instead of failing the request with a 500. incomplete_utf8_tail moves from whisper/transcript_format.hpp to wstream_buf.hpp so both share it.

Model sweep

• Default max_completion_tokens goes from 512 to 1024. Thinking models such as qwen3:1.7b and qwen3:8b were spending the whole budget on reasoning.
• qwen3.8-mtp:27b gets a 1200 s request timeout through a new REQUEST_TIMEOUT_OVERRIDES table. It decodes at ~1.7 tok/s, so a full 1024-token reply takes about 600 s plus model load.

Testing

• flm target builds on Windows (MSVC, Release).
• Sweep timeout logic tested locally against a fake server:
  • the override model gets 1200 s and others 600 s;
  • --request-timeout above the override still wins;
  • the override model got its reply where a normal model timed out;
  • the whole-stream deadline still fires.
• Not yet run end-to-end on an NPU. The Model Sweep on this PR exercises it.

🤖 Generated with Claude Code