Skip to content

fix: make the daemon fail fast and put it under a systemd watchdog - #3

Merged
richardkiene merged 1 commit into
mainfrom
fix/daemon-fail-fast-and-watchdog
Sep 1, 2026
Merged

fix: make the daemon fail fast and put it under a systemd watchdog#3
richardkiene merged 1 commit into
mainfrom
fix/daemon-fail-fast-and-watchdog

Conversation

@richardkiene

@richardkiene richardkiene commented Sep 1, 2026

Copy link
Copy Markdown

Why

The daemon swallowed every error: an unparseable config.json was treated as "all GPUs auto", a missing or corrupt curve.json fell back to a hardcoded default, an unknown mode fell back to that default too, and a failed nvmlDeviceSetFanSpeed_v2 was ignored — the service sat active while writing to stderr every 5 s. A daemon that hung inside NVML was never restarted, which in curve mode leaves the fan pinned at whatever speed was last set (possibly the 30% floor) while load arrives.

What

Daemon

  • Every error is fatal: reason to syslog, fans reset to auto, exit non-zero so Restart=on-failure takes over. Covers unreadable config/curve, manual mode without a valid speed, unknown mode, temperature read failure, and any fan command NVML rejects.
  • curve.json is re-read every poll (as config.json already was), so nvfd curve <temp> <speed> takes effect within 5 s without a reload. SIGHUP is kept as a logged no-op so a stray HUP does not kill the daemon; ExecReload is removed and the README updated.
  • sd_notify READY=1 / WATCHDOG=1 / STOPPING=1 through a 40-line AF_UNIX datagram shim (src/notify.c) — no libsystemd link dependency. Unit is Type=notify, WatchdogSec=30 against the 5 s poll; the daemon refuses to start if WATCHDOG_USEC is under twice the poll interval.
  • First poll hands every GPU not in manual/curve mode back to the driver, so an instance that died uncleanly cannot leave those fans pinned either.
  • StartLimitIntervalSec=60 / StartLimitBurst=5 so persistent failure parks the unit in failed instead of restarting forever.

Readers

  • config_read() distinguishes "no file yet" (empty object) from "file exists but unusable" (NULL + config_last_error()). config_write_gpu refuses to overwrite a config it could not parse. JSON_REJECT_DUPLICATES on both files.
  • curve_load() validates: top-level object, integer keys 0–100 with a leading digit, integer values 0–100, no duplicate temperatures ("30" and "030" divided by zero in curve_interpolate), at most MAX_CURVE_POINTS, at least one point. curve_require() is the shared "load or explain why not" for callers that cannot proceed without a curve.
  • Legacy plain-text migration refuses values the daemon would reject instead of writing them; a migration failure is fatal.
  • fan_get_count returns -1 on error so fan_reset_to_auto cannot reset zero fans and report success.

CLI / TUI

  • nvfd curve and nvfd <n> curve verify the curve loads before switching a GPU into curve mode; the TUI does the same before writing "curve", and exits with the reason rather than falling back to a built-in curve the daemon does not have. curve_default_interpolate is removed.
  • Non-zero exit status when a command fails.
  • The TUI and nvfd status exit with the parse error instead of rendering a blank config; the curve editor refuses to open on an invalid file rather than overwriting it from the default on save.

The daemon used to swallow every error: an unparseable config was
treated as "all GPUs auto", a missing or corrupt curve fell back to a
hardcoded default, an unknown mode fell back to that default too, and a
failed nvmlDeviceSetFanSpeed_v2 was ignored so the service sat "active"
while logging to stderr every 5 s. A daemon that hung inside NVML was
never restarted, leaving the fans pinned at the last speed it set.

Daemon:
- Every error is fatal. The reason goes to syslog, the fans are reset to
  auto, and the process exits non-zero so Restart=on-failure takes over.
  This covers: unreadable config or curve file, manual mode without a
  valid speed, unknown mode, temperature read failure, and any fan
  command NVML rejects.
- The curve file is re-read every poll, like config.json already was, so
  `nvfd curve <temp> <speed>` takes effect within 5 s. SIGHUP is kept as
  a logged no-op so a stray HUP does not kill the daemon; ExecReload is
  gone and the README updated.
- sd_notify READY=1 / WATCHDOG=1 / STOPPING=1 via a 40-line AF_UNIX
  datagram shim (src/notify.c), no libsystemd dependency. The unit is
  Type=notify with WatchdogSec=30 against a 5 s poll; the daemon honours
  WATCHDOG_USEC and refuses to start if it is under twice the poll.
- The first poll hands every GPU not in manual/curve mode back to the
  driver, so an instance that died uncleanly cannot leave those fans
  pinned. A GPU that cannot be reset on that startup sweep is logged and
  skipped rather than fatal: this instance never touched it, and a card
  without controllable fans must not stop the others being managed.
- StartLimitIntervalSec=60 / StartLimitBurst=5 so persistent failure
  parks the unit in "failed" instead of restarting forever.

Readers:
- config_read() distinguishes "no file yet" (empty object) from "file
  exists but is unusable" (NULL + config_last_error()). config_write_gpu
  refuses to overwrite a config it could not parse. Both files are
  loaded with JSON_REJECT_DUPLICATES.
- curve_load() validates the file: object at top level, integer keys
  0-100 with a leading digit, integer values 0-100 (checked as json_int_t
  before narrowing), no duplicate temperatures (which divided by zero in
  curve_interpolate), at most MAX_CURVE_POINTS, at least one point.
  curve_require() is the shared "load or explain why not"; curve_read()
  stays as a wrapper for the TUI.
- curve_edit/curve_reset return status instead of void; curve_edit
  refuses to overwrite an invalid file.
- Legacy plain-text migration refuses anything but auto, curve, or a
  speed in FAN_SPEED_MIN..MAX instead of writing a config the daemon
  would reject on every start, and a migration failure is fatal.
- fan_get_count returns -1 on NVML error, so fan_reset_to_auto cannot
  reset zero fans and report success.

CLI / TUI:
- `nvfd curve` and `nvfd <n> curve` verify the curve loads before
  switching a GPU into curve mode. The TUI does the same before writing
  "curve" and exits with the reason rather than falling back to a
  built-in curve the daemon does not have; curve_default_interpolate
  has no callers left and is removed.
- Exit status is non-zero when a command fails.
- The TUI and `nvfd status` exit with the parse error instead of
  showing a blank config; the curve editor refuses to open on an
  invalid file rather than overwriting it from the default on save, and
  the dashboard reports that error after leaving curses.
- display_fan_curve distinguishes missing (suggest reset) from invalid
  (print the error, do not suggest overwriting it); display_list_gpus
  says "fan count unavailable" instead of "-1 fans".
@richardkiene
richardkiene force-pushed the fix/daemon-fail-fast-and-watchdog branch from ca6e89d to 49b1377 Compare September 1, 2026 16:30
@richardkiene
richardkiene merged commit f1270ec into main Sep 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant