Shared chat parsing and constrained decoding for the OpenAI endpoints

.github/script/sweep_common.py

@@ -709,10 +709,8 @@ def collect_stream(self, response) -> tuple[str, str]:
                     f"stream exceeded the {self.args.request_timeout:.0f}s "
                     f"request timeout after {len(output_content)} characters"
                 )
-            # FLM attaches usage to the final chunk, which also carries a
-            # choices entry with a null delta. Read it before the choices
-            # guard below, so a server that instead sends usage on its own
-            # terminal chunk (the stock OpenAI shape) still gets picked up.
+            # With include_usage the usage arrives on its own terminal chunk
+            # with no choices, so read it before the choices guard below.
             if getattr(chunk, "usage", None):
                 usage = chunk.usage
             if not chunk.choices:
@@ -736,6 +734,8 @@ def chat(self, model: str, messages: list, stream: bool = True) -> tuple[str, st
             model=model,
             messages=messages,
             stream=stream,
+            # a stream reports usage only when asked, as OpenAI's does
+            **({"stream_options": {"include_usage": True}} if stream else {}),
             **self.completion_kwargs(),
         )
         if stream:

.gitmodules

@@ -1,3 +1,6 @@
 [submodule "third_party/tokenizers-cpp"]
 	path = third_party/tokenizers-cpp
 	url = https://github.com/mlc-ai/tokenizers-cpp
+[submodule "third_party/xgrammar"]
+	path = third_party/xgrammar
+	url = https://github.com/mlc-ai/xgrammar

README.md

@@ -134,6 +134,7 @@ No model rewrites, no tuning — it just works.
 - Inspired by the widely adopted [llama.cpp](https://github.com/ggml-org/llama.cpp) and [Ollama](https://github.com/ollama/ollama)
 - Tokenization accelerated with [MLC-ai/tokenizers-cpp](https://github.com/mlc-ai/tokenizers-cpp)
 - Chat formatting via [Google/minja](https://github.com/google/minja)
+- Constrained decoding (tool calls, structured output) powered by [MLC-ai/xgrammar](https://github.com/mlc-ai/xgrammar)
 - Low-level kernels optimized using the powerful [IRON](https://github.com/amd/iron)+[AIE-MLIR](https://github.com/Xilinx/mlir-aie)
 
 ---

THIRD_PARTY_NOTICES.txt

@@ -44,6 +44,9 @@ Components
   29. HRX
   30. flatcc
   31. pkgconf
+  32. XGrammar
+  33. PicoJSON
+  34. DLPack
 
 ==============================================================================
 1. libcurl
@@ -1146,6 +1149,72 @@ This software is provided 'as is' and without any warranty, express or
 implied.  In no event shall the authors be liable for any damages arising
 from the use of this software.
 
+==============================================================================
+32. XGrammar
+==============================================================================
+
+License:  Apache License 2.0
+Homepage: https://github.com/mlc-ai/xgrammar
+Used in:  Statically linked into flm (v0.2.8).
+
+Copyright (c) 2024 by XGrammar Contributors
+
+Licensed under the Apache License, Version 2.0. The license text is
+reproduced at the end of this file.
+
+XGrammar's NOTICE file reads:
+
+    XGrammar
+
+    Copyright (c) 2024 by XGrammar Contributors
+
+==============================================================================
+33. PicoJSON
+==============================================================================
+
+License:  BSD 2-Clause License
+Homepage: https://github.com/kazuho/picojson
+Used in:  Compiled into flm (via XGrammar).
+
+Copyright 2009-2010 Cybozu Labs, Inc.
+Copyright 2011-2014 Kazuho Oku
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice,
+   this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+==============================================================================
+34. DLPack
+==============================================================================
+
+License:  Apache License 2.0
+Homepage: https://github.com/dmlc/dlpack
+Used in:  Compiled into flm (via XGrammar, header only).
+
+Copyright 2017 by Contributors
+
+Licensed under the Apache License, Version 2.0. The license text is
+reproduced at the end of this file.
+
 ==============================================================================
 Full license texts
 ==============================================================================

docs/_config.yml

@@ -87,6 +87,12 @@ docs_nav:
         url: "/docs/instructions/server/webui/"
       - label: Tool Calling
         url: "/docs/instructions/server/tool_calling/"
+      - label: Reasoning & Tool Calls
+        url: "/docs/instructions/server/reasoning_and_tools/"
+      - label: Structured Output
+        url: "/docs/instructions/server/structured_output/"
+      - label: Output Parsing Internals
+        url: "/docs/instructions/server/chat_output_internals/"
       - label: LangChain RAG
         url: "/docs/instructions/server/rag_LangChain/"
       - label: LangChain Web Search