fix: make the daemon fail fast and put it under a systemd watchdog - #3
Merged
Merged
Conversation
This was referenced Sep 1, 2026
richardkiene
force-pushed
the
fix/daemon-fail-fast-and-watchdog
branch
from
September 1, 2026 16:15
bae0254 to
ca6e89d
Compare
The daemon used to swallow every error: an unparseable config was treated as "all GPUs auto", a missing or corrupt curve fell back to a hardcoded default, an unknown mode fell back to that default too, and a failed nvmlDeviceSetFanSpeed_v2 was ignored so the service sat "active" while logging to stderr every 5 s. A daemon that hung inside NVML was never restarted, leaving the fans pinned at the last speed it set. Daemon: - Every error is fatal. The reason goes to syslog, the fans are reset to auto, and the process exits non-zero so Restart=on-failure takes over. This covers: unreadable config or curve file, manual mode without a valid speed, unknown mode, temperature read failure, and any fan command NVML rejects. - The curve file is re-read every poll, like config.json already was, so `nvfd curve <temp> <speed>` takes effect within 5 s. SIGHUP is kept as a logged no-op so a stray HUP does not kill the daemon; ExecReload is gone and the README updated. - sd_notify READY=1 / WATCHDOG=1 / STOPPING=1 via a 40-line AF_UNIX datagram shim (src/notify.c), no libsystemd dependency. The unit is Type=notify with WatchdogSec=30 against a 5 s poll; the daemon honours WATCHDOG_USEC and refuses to start if it is under twice the poll. - The first poll hands every GPU not in manual/curve mode back to the driver, so an instance that died uncleanly cannot leave those fans pinned. A GPU that cannot be reset on that startup sweep is logged and skipped rather than fatal: this instance never touched it, and a card without controllable fans must not stop the others being managed. - StartLimitIntervalSec=60 / StartLimitBurst=5 so persistent failure parks the unit in "failed" instead of restarting forever. Readers: - config_read() distinguishes "no file yet" (empty object) from "file exists but is unusable" (NULL + config_last_error()). config_write_gpu refuses to overwrite a config it could not parse. Both files are loaded with JSON_REJECT_DUPLICATES. - curve_load() validates the file: object at top level, integer keys 0-100 with a leading digit, integer values 0-100 (checked as json_int_t before narrowing), no duplicate temperatures (which divided by zero in curve_interpolate), at most MAX_CURVE_POINTS, at least one point. curve_require() is the shared "load or explain why not"; curve_read() stays as a wrapper for the TUI. - curve_edit/curve_reset return status instead of void; curve_edit refuses to overwrite an invalid file. - Legacy plain-text migration refuses anything but auto, curve, or a speed in FAN_SPEED_MIN..MAX instead of writing a config the daemon would reject on every start, and a migration failure is fatal. - fan_get_count returns -1 on NVML error, so fan_reset_to_auto cannot reset zero fans and report success. CLI / TUI: - `nvfd curve` and `nvfd <n> curve` verify the curve loads before switching a GPU into curve mode. The TUI does the same before writing "curve" and exits with the reason rather than falling back to a built-in curve the daemon does not have; curve_default_interpolate has no callers left and is removed. - Exit status is non-zero when a command fails. - The TUI and `nvfd status` exit with the parse error instead of showing a blank config; the curve editor refuses to open on an invalid file rather than overwriting it from the default on save, and the dashboard reports that error after leaving curses. - display_fan_curve distinguishes missing (suggest reset) from invalid (print the error, do not suggest overwriting it); display_list_gpus says "fan count unavailable" instead of "-1 fans".
richardkiene
force-pushed
the
fix/daemon-fail-fast-and-watchdog
branch
from
September 1, 2026 16:30
ca6e89d to
49b1377
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The daemon swallowed every error: an unparseable
config.jsonwas treated as "all GPUs auto", a missing or corruptcurve.jsonfell back to a hardcoded default, an unknown mode fell back to that default too, and a failednvmlDeviceSetFanSpeed_v2was ignored — the service satactivewhile writing to stderr every 5 s. A daemon that hung inside NVML was never restarted, which in curve mode leaves the fan pinned at whatever speed was last set (possibly the 30% floor) while load arrives.What
Daemon
Restart=on-failuretakes over. Covers unreadable config/curve, manual mode without a valid speed, unknown mode, temperature read failure, and any fan command NVML rejects.curve.jsonis re-read every poll (asconfig.jsonalready was), sonvfd curve <temp> <speed>takes effect within 5 s without a reload. SIGHUP is kept as a logged no-op so a stray HUP does not kill the daemon;ExecReloadis removed and the README updated.sd_notifyREADY=1/WATCHDOG=1/STOPPING=1through a 40-line AF_UNIX datagram shim (src/notify.c) — no libsystemd link dependency. Unit isType=notify,WatchdogSec=30against the 5 s poll; the daemon refuses to start ifWATCHDOG_USECis under twice the poll interval.StartLimitIntervalSec=60/StartLimitBurst=5so persistent failure parks the unit infailedinstead of restarting forever.Readers
config_read()distinguishes "no file yet" (empty object) from "file exists but unusable" (NULL+config_last_error()).config_write_gpurefuses to overwrite a config it could not parse.JSON_REJECT_DUPLICATESon both files.curve_load()validates: top-level object, integer keys 0–100 with a leading digit, integer values 0–100, no duplicate temperatures ("30"and"030"divided by zero incurve_interpolate), at mostMAX_CURVE_POINTS, at least one point.curve_require()is the shared "load or explain why not" for callers that cannot proceed without a curve.fan_get_countreturns -1 on error sofan_reset_to_autocannot reset zero fans and report success.CLI / TUI
nvfd curveandnvfd <n> curveverify the curve loads before switching a GPU into curve mode; the TUI does the same before writing "curve", and exits with the reason rather than falling back to a built-in curve the daemon does not have.curve_default_interpolateis removed.nvfd statusexit with the parse error instead of rendering a blank config; the curve editor refuses to open on an invalid file rather than overwriting it from the default on save.