Skip to content

runtime: a background loop starts again after a panic - #13

Merged
cpoder merged 1 commit into
masterfrom
fix/background-loops
Nov 6, 2026
Merged

cpoder merged 1 commit into
masterfrom
fix/background-loops

Conversation

@cpoder

@cpoder cpoder commented Nov 6, 2026

Copy link
Copy Markdown
Owner

ci is red on master at the v0.4.0 commit: e2e/http-in-bound.sh found a panic in about one run in two. It is a real defect, and a narrow one.

  • The defect. version_watch runs on a bare thread, and nats::connect panics when the OS has no thread to give. A burst of connections landing on its first connection, under a thread limit, ended it for good with nothing to show for it, and a promoted version no longer reached that instance until a restart. With the burst 5 s after boot: 0 panics in 10 runs. control::supervise has the same shape.
  • The fix. Both now run through background(): a named thread, restarted after a panic.
  • The test. Its burst landed during boot. It now waits for the boot to finish, and checks directly that the version watch is alive after the burst. Two pipelines used grep -q under pipefail, which reads an early exit of grep as a failure: they no longer do.

The check that wants no panic at all runs after boot, so a burst during boot is covered by a unit test and a repeated local run, where the thread panicked and stayed alive, not by this script.

Local: 48 unit tests, http-in-bound nine times in a row, and the transport, governed, emit-subject and doc-example suites.

🤖 Generated with Claude Code

https://claude.ai/code/session_018wXgxT2UoFSZfqhAE1Tb5V

nats::connect spawns threads and panics when the OS has none. version_watch runs
on a bare thread, so a burst of connections during boot, under a thread limit,
ended it for good with nothing to show for it, and a promoted version no longer
reached that instance until a restart. control::supervise has the same shape.
Both now run through background(): a named thread, restarted after a panic.

e2e/http-in-bound.sh turned master's CI red on the v0.4.0 commit, about one run in
two, because its burst landed during boot. It now waits for the boot to finish, and
checks directly that the version watch is alive after the burst. Two pipelines
there used grep -q under pipefail, which reads an early exit of grep as a failure:
they no longer do.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018wXgxT2UoFSZfqhAE1Tb5V
@cpoder
cpoder merged commit fca57c9 into master Nov 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant