[chat] perf: 聊天流式返回提速与 SSE 全链路修复 - #85
Merged
Merged
Conversation
ReAct LLM 已被 vector_search 工具描述指示自行优化检索 query(指代消解/ 上下文补全/多角度搜索),外层强制改写只增加首 token 前的一次 LLM 往返, 对召回无增益。rewritten_queries 保留为空列表,下游 multi-path 契约不变。
新增 app/routers/streaming.py(SSE_HEADERS + sse_streaming_response 工厂), 四个 SSE 端点统一走它:nginx/APISIX 认 X-Accel-Buffering: no,逐 token 下发 不再被代理攒成一次 bulk 投递。session_summary 与 task-quiz/chat 此前无任何 防缓冲头。去掉冗余的 hop-by-hop 头 Connection: keep-alive(nginx 会剥掉、 HTTP/2 语义非法)。
nginx_config.http.proxy_buffering 不是 APISIX 支持的键:ngx_tpl.lua 不渲染 该键,apisix test 通过但生成的 nginx.conf 里没有指令(3.11.0 实测)。 改用 http_server_location_configuration_snippet 落 proxy_buffering off 进 每个 location 块,已实测渲染成功。与后端 X-Accel-Buffering: no 形成双保险。
Next.js rewrites 代理会缓冲 SSE(frontendv2 代码注释已记录的已知问题), 浏览器经 :3000 访问时逐 token 流式被攒成一次投递。改动: - client.ts: API_BASE_URL 拆分浏览器/SSR——浏览器用 NEXT_PUBLIC_API_URL (绝对地址直连 nginx:80),SSR 固定走内部 http://nginx:80(原逻辑设值后 SSR 会误指容器自身) - main.py: CORS 头白名单补 X-Requested-With(frontendv2 每个 API fetch 都 携带,跨域预检声明它,缺了预检 400) - default.yaml: CORS origin 白名单补 http://127.0.0.1:3000 链路仍为 浏览器 -> nginx -> apisix -> backend,仅绕开 frontend 容器内 Next.js 服务端转发这一跳。
容器 flex items-end 下,24px 单行 textarea 与 32px 发送按钮底对齐, 文字下方 8px、上方 16px,视觉偏下约 4px。改为 leading-6 + py-1, MIN_HEIGHT 32px 与按钮等高:单行文字绝对居中,多行时按钮仍钉在右下。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
用户感知聊天"不是流式返回"(token 一次性整块到达)。排查发现首 token 延迟与 SSE 传输链路两层问题,且 APISIX 配置写法静默无效。
排查结论(逐层实证)
inject_context强制 query rewrite 增加首 token 前一次 LLM 往返nginx_config.http.proxy_buffering不是支持的键,静默无效改动
rewritten_queries保留空列表,下游 multi-path 契约不变app/routers/streaming.py(SSE_HEADERS + 工厂函数),4 个 SSE 端点统一防缓冲头;去掉 hop-by-hop 的Connection: keep-alivehttp_server_location_configuration_snippet(已用一次性容器实测渲染进 nginx.conf)client.ts拆分浏览器/SSR base URL:浏览器直连网关(nginx:80),绕过 Next 代理;CORS 配套放行X-Requested-With头 +http://127.0.0.1:3000origin链路保持 浏览器 → nginx → apisix → backend,仅绕开 frontend 容器内 Next.js 服务端转发一跳。
验证
event_counts={'on_chat_model_stream': 275}确认逐 token 产出apisix test+ grep 确认指令渲染py_compile/ router import / YAML 解析全过影响范围
chat agent 图、4 个 SSE 路由、APISIX http 级配置(全局 proxy_buffering off,对普通 JSON 响应无实质影响)、frontendv2 API base URL 解析、CORS 白名单。不涉及 DB schema / Milvus。