[ProjectZero02]

Phase 3 of the recovery layer plan: the hub tells a hung machine apart from one that's booting, asleep, installing updates or just missing its helper, says why, and tells whoever listens.

What changes

Hub watch (every 30 s per machine with a KVM): evidence gathered in parallel, 8 s at most:

• helper status (two tries), TCP to the helper's port (refuses / accepts / no reply), tailscale ping --icmp, three power-light readings 0.5 s apart (on / off / blinking / unreadable), Tailscale's online flag (shown only).
• A pure detector (machines.Track.Step) turns the history into on, off, hung or unknown + reason. Hung = 3 min (owner setting 2–30) of silence, port not refusing, no ping reply, a wired light steadily on, and no excuse: booting (10 min after a hub action or the light coming on), a shutdown/restart note or countdown (10 min; 90 with Windows updates pending → updating), or possibly asleep. No helper or an untrusted light is never hung.

Notes from helpers (POST /v2/note, peer keys only, machine taken from the caller's address): Windows sends sleep/resume from PowerRegisterSuspendResumeNotification (a callback Windows waits for) and shutdown/restart from the service's pre-shutdown notice; Linux sends shutdown when systemd is stopping. Windows helpers also report sleep facts (S3, Modern Standby, idle timeout on AC). After a sleep note the hub learns what the light does asleep.

Events:

• GET /v2/machines/events: SSE, snapshot or Last-Event-ID replay (last 500), 25 s heartbeat, a per-write deadline so the server's 150 s write timeout doesn't end it, 16 streams max, filtered like GET /v2/machines.
• Webhooks (owner devices): https, or http to Tailscale/loopback only; secret shown once and sealed; HMAC-SHA256 over timestamp.body; retries at 10 s, 1 min, 5 min; no redirects.
• hung / recovered also go to the activity log; the phone now notifies about them.

App: the device page says Frozen since … and shows what the hub saw; starting up, installing updates and may-be-asleep show as such.

Docs: docs/protocol.md "Machine states and events" (with a signature known-answer), plan text brought in line with what was built, CHANGELOG 2.3.0.

Tests

• Detector table tests on a fake clock (contradictions, unreadable light, boot grace, notes, updates, Modern Standby, light-on sleep, gaps).
• Watcher end to end against the fake NanoKVM, a fake helper, a fake pinger and real loopback listeners (one that accepts and never answers, one closed).
• Server: notes only from the machine's own peer key; SSE on a real server with 150 ms read/write timeouts (fails without the per-write deadline), replay, unknown ID, per-key filtering, stream limit; webhook routes owner-only, URL rules, secret shown once, signed test delivery, limit; hungAfterMinutes validation.
• Webhook signature known-answer from Python and openssl; retries; redirects; tampered secret.
• fleet.Tell reaches every keyed peer and gives up on a slow one in 1.5 s.
• Windows CI: sleep facts read and logged, suspend registration works, power-event mapping.
• Android: hub verdict parsing and wording, notification rule, device page screenshot test.

A real sleep, shutdown and hang on hardware is for the hardware checklist (phase 5).

🤖 Generated with Claude Code