Repository navigation
fix(mcp): a slow log stream can no longer stop the server - #64
Merged
Merged
Conversation
Log writes ran on the event loop, so a blocked stdout froze every request. On Nov 4 a container whose Railway logs lagged 13 minutes dropped to 10-20% of its calls and timed out on /health until a redeploy replaced it. - Write all logs from a background thread through a bounded queue. When the stream stays blocked, records are dropped and counted instead of blocking. - Stop logging client disconnects as error tracebacks, which fed the backlog. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
SIGTERM killed the process with records still queued, so every redeploy lost its last log lines. Drain for up to 2 seconds before exiting on SIGTERM or SIGINT, which also removes the Ctrl-C traceback. The drop notice went through the last handler used. Most often that is uvicorn's access handler, whose format rejects a record without its arguments, so the notice never appeared. It now has its own handler. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Log writes ran on the event loop, so a blocked stdout froze every request. On Nov 4 from 13:03 to 13:28 UTC, a container whose Railway logs lagged up to 13 minutes dropped to 10-20% of its normal calls and timed out on
/healthuntil a redeploy replaced it. Client disconnects then wrote long tracebacks, which added to the backlog.What
Dropped N log recordsline appears once it recovers.zernio-mcpand never reached this repo.1.2.3. STDIO mode is unchanged.Verified
Installed as the Dockerfile does, real server, local stub API, mix of successful calls, tool errors, unknown tools, bad arguments and 401s.
/healthtimes out/healthat most 0.68 s, no timeouts/healthtimeoutsDropped 35961 log recordsappears, no logging errorsApplication shutdown completeis logged, exit 143develop, about 1,000 log lines)developAlso 26 of 26 end-to-end checks, including
/chatgpt, legacy SSE and Sentry envelopes.Not verified on Railway. The log lag and the stall coincide, but the pipe blocking there is inferred, not observed. After merging, a log lag should show up as
Dropped N log recordsinstead of a stall. If the server stalls with no such line, the cause is something else.Follow-ups
SENTRY_DSNis missing in Railway production (onlyzernio-mcp-devhas it).developtoo, and the frequent regen redeploys hide it.🤖 Generated with Claude Code