Skip to content

Fix endless Nginx reload loop after failed observer restart - #77

Merged
tdgroot merged 1 commit into
masterfrom
Fix_loop
Oct 12, 2026
Merged

tdgroot merged 1 commit into
masterfrom
Fix_loop

Conversation

@hpowernl

Copy link
Copy Markdown
Contributor

On server nmsbva-****-magweb-tbbm, the nginx_config_reloader gets stuck in a loop after an observer fails to start correctly.

On every check, it tries to restart the observer again, but stop_observer() attempts to join a thread that was never started. This results in the following error roughly every 5 seconds:

RuntimeError: cannot join thread before it is started

Despite the error, Applying new config is still triggered, causing Nginx to be reloaded continuously.

This matches the known issue in the symlink rearm logic where a single failed observer start can lead to an endless Nginx reload loop.

When a symlink target under the watched dir changes, after_loop()
restarts the observer. If Observer.start() fails (e.g. the inotify
watch limit is reached or the target disappears mid-deploy), watchdog
raises before the observer thread is started. The next loop sees the
still stale symlink snapshot and tries to restart again, but joining
the never-started thread raises "cannot join thread before it is
started". start_observer() is never reached again, the tree is marked
dirty and nginx is reloaded on every loop, forever, while nothing is
being watched.

Only join the observer when its thread is alive, so a failed start can
be retried on the next loop and the reloader heals itself. Also only
mark the tree dirty once the restart succeeded, so a failing restart no
longer reloads nginx on every iteration.
@tdgroot
tdgroot merged commit b3b35d6 into master Oct 12, 2026
5 checks passed
@tdgroot
tdgroot deleted the Fix_loop branch October 12, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants