The symptom
TestSupervisorPTYCapture (added by #245) fails intermittently on the ubuntu CI runner. Twice on kanban so far, always the same shape — the two final stdout lines missing from the log, stderr and the earlier PTY output intact:
supervisor_pty_test.go:60: the log is missing "a plain line\n"
supervisor_pty_test.go:60: the log is missing "no_color=1"
(log shown: a stderr line / a line only a terminal gets / Downloading 1% / Downloading 100%)
Failing runs: 36205308529 and 36206469398. go test -race -count=10 on macOS passes every time, so it is timing-shaped and platform-shaped, not a logic break in any PR touching it.
Hypothesis
The stub engine exits immediately after its last echo. On Linux a read on the PTY master returns EIO the moment every slave-side fd closes — the child's exit — and bytes written just before the exit but not yet drained can be lost. The truncation point in both failures is exactly after printf '\rDownloading m.gguf 100%%\r': the pump took everything up to there, then the read errored and the remaining \na plain line\nno_color=1\n never arrived. The macOS PTY appears to keep the data readable across that window, which is why local runs never see it.
What a fix would look like
Worth deciding deliberately rather than papering over:
- the pump's error path on
EIO/EOF — does it flush what the column still holds, and can it distinguish "engine gone, data possibly buffered" from a real fault? (e.g. an exit-then-drain with FIONREAD, or reading until EIO after the child is reaped)
- whether
waitForState(StateStopped) can honestly claim "the log is whole" on Linux under this race — today the test relies on exactly the promise the flake breaks.
The e2e chain matters here: the unit tests cover ptylog in pieces, so this test is the only thing proving the tail of a download survives to the log.
The symptom
TestSupervisorPTYCapture(added by #245) fails intermittently on the ubuntu CI runner. Twice onkanbanso far, always the same shape — the two final stdout lines missing from the log, stderr and the earlier PTY output intact:Failing runs: 36205308529 and 36206469398.
go test -race -count=10on macOS passes every time, so it is timing-shaped and platform-shaped, not a logic break in any PR touching it.Hypothesis
The stub engine exits immediately after its last
echo. On Linux a read on the PTY master returnsEIOthe moment every slave-side fd closes — the child's exit — and bytes written just before the exit but not yet drained can be lost. The truncation point in both failures is exactly afterprintf '\rDownloading m.gguf 100%%\r': the pump took everything up to there, then the read errored and the remaining\na plain line\nno_color=1\nnever arrived. The macOS PTY appears to keep the data readable across that window, which is why local runs never see it.What a fix would look like
Worth deciding deliberately rather than papering over:
EIO/EOF — does it flush what the column still holds, and can it distinguish "engine gone, data possibly buffered" from a real fault? (e.g. an exit-then-drain withFIONREAD, or reading untilEIOafter the child is reaped)waitForState(StateStopped)can honestly claim "the log is whole" on Linux under this race — today the test relies on exactly the promise the flake breaks.The e2e chain matters here: the unit tests cover
ptylogin pieces, so this test is the only thing proving the tail of a download survives to the log.