Skip to content

Add hourly health check with external dead-man's-switch alerting - #31

Merged
byjpr merged 2 commits into
claude/admiring-planck-25cvzffrom
claude/fix-7-monitoring
Oct 9, 2026
Merged

byjpr merged 2 commits into
claude/admiring-planck-25cvzffrom
claude/fix-7-monitoring

Conversation

@byjpr

@byjpr byjpr commented Oct 9, 2026

Copy link
Copy Markdown
Owner

Closes #7

Changes

  • modules/monitoring.nix: an hourly mail-health-check service. It checks:
    • Services: postfix, dovecot, rspamd, redis, nginx, kresd and fail2ban are active, and no systemd unit has failed. That last part catches failed certificate renewals and backup runs.
    • TLS certificate: present, not expiring within 14 days, and not the self-signed placeholder left when Let's Encrypt issuance fails. With MTA-STS in enforce mode, an expired certificate stops inbound mail.
    • Disk: usage below diskWarnPercent (85).
    • Mail queue: fewer than queueWarn (100) messages.
    • Outbound SMTP: can connect to gmail-smtp-in.l.google.com:25, which catches a provider block or OVH's anti-spam block.
    • Blocklists: the public IPv4 address on Spamhaus ZEN or SpamCop. Queries go through the local resolver, and "query refused" answers (127.255.255.x) are ignored.
  • Reporting:
    • With monitoring.healthcheckUrl set (healthchecks.io, self-hosted Healthchecks, an Uptime Kuma push monitor), the result is POSTed there, to /fail on problems. The service alerts on failures and on missing reports, so it also covers the server being down.
    • Without a URL, problems are mailed to adminEmail, at most once a day.
  • just health runs the check now and shows the output. settings.nix and the README explain the setup.

Testing

  • I ran the generated script with stubbed systemctl, postqueue, curl and dig. It reported:

    • a stopped service;
    • a failed unit;
    • a missing certificate;
    • a 150-message queue;
    • blocked outbound SMTP;
    • a Spamhaus listing.

    It POSTed the list to …/fail.

  • Both report variants (URL and mail fallback) render correctly. The generated script passes shellcheck.

  • All 5 provider example systems build. nixfmt, shellcheck and tofu validate pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G22o7QQ17GRtkCqpJNFr9B


Generated by Claude Code

modules/monitoring.nix checks services and failed units, the TLS
certificate (expiry, self-signed placeholder), disk usage, queue size,
outbound SMTP and the public IP on Spamhaus ZEN/SpamCop, and reports to
settings.monitoring.healthcheckUrl (healthchecks.io, Uptime Kuma push,
...), which also alerts when reports stop. Without a URL, problems are
mailed to adminEmail at most once a day. just health runs it on demand.

Closes #7

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01G22o7QQ17GRtkCqpJNFr9B
@byjpr

byjpr commented Oct 9, 2026

Copy link
Copy Markdown
Owner Author

@copilot resolve the merge conflicts in this pull request

…nto claude/fix-7-monitoring

# Conflicts:
#	flake.nix

Co-authored-by: byjpr <[email protected]>

Copilot AI commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

Resolved the conflict by retaining both monitoring and release modules in flake.nix, and merged the default branch. Merge commit: dba50f3.

@byjpr
byjpr merged commit d492bec into claude/admiring-planck-25cvzf Oct 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[High] No monitoring or alerting

3 participants