Phase 3: hang detection and events

.github/workflows/build.yml

@@ -152,8 +152,8 @@ jobs:
       - uses: actions/checkout@v7
       - uses: actions/setup-go@v7
         with: { go-version-file: helper/go.mod, cache-dependency-path: helper/go.sum }
-      - name: Unit tests for Windows-only code (DPAPI sealing, NanoKVM client, window access, PowerShell)
-        run: go test -count=1 -v -run . ./internal/secret/ ./internal/nanokvm/... ./internal/winsvc/ ./internal/setup/
+      - name: Unit tests for Windows-only code (DPAPI sealing, NanoKVM client, window access, PowerShell, sleep settings, suspend notices)
+        run: go test -count=1 -v -run . ./internal/secret/ ./internal/nanokvm/... ./internal/winsvc/ ./internal/sleepinfo/ ./internal/setup/
       - name: Build PowerDeck.exe
         run: |
           go install github.com/tc-hib/[email protected]

CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## 2.3.0
+
+The hub now notices when a computer freezes, and says why it thinks so.
+
+**Update the helper on the hub (the always-on computer that drives your KVMs), the helpers on your computers, and the phone app.** A computer's helper before 2.3 still works, but can't tell the hub it's about to sleep or restart, so the hub is more careful before calling it frozen.
+
+- **Frozen or just busy?** Every 30 seconds the hub checks each computer with a KVM: does its PowerDeck answer, does it answer a ping, does its PowerDeck port turn connections away or swallow them, and is its power light on, off or blinking. A computer is called frozen only when, for 3 minutes in a row, nothing answers and the light stays on, and it isn't starting up (10 minutes after it's turned on, restarted or reset), shutting down or restarting (10 minutes; 90 when Windows is installing updates), or possibly asleep. A computer without the helper, or whose power light isn't known to be connected, is never called frozen.
+- **Sleep is learned.** A Windows helper tells the hub how the PC sleeps (Modern Standby or classic, and after how long), and tells it just before it sleeps, shuts down or restarts, and when it wakes. The first time a computer sleeps, the hub learns what its power light does then (off, blinking or on).
+- **App:** a frozen computer's page says **Frozen since** with the time, and shows what the hub saw. **Reset** is still one tap away. A computer that's starting up, installing updates or may be asleep says so instead of "On".
+- **Events for assistants:** `GET /v2/machines/events` streams every computer's state as it changes (Server-Sent Events, with replay after a reconnect), and owners can add **webhooks** that receive `hung` and `recovered` events, signed with HMAC-SHA256 and retried. Machines in `GET /v2/machines` gain `hung`, a `reason`, `since` and the `evidence`. `hungAfterMinutes` (2 to 30) changes the 3 minutes. Frozen and recovered computers are also in the hub's activity log, and the phone notifies about both (it checks about every 15 minutes). See [docs/protocol.md](docs/protocol.md#machine-states-and-events).
+
 ## 2.2.3
 
 - **PC window setup shows once.** The first-run setup was only marked done by **Done** on its last step, so closing the window part-way (the main screen has the same QR codes) started it over on every open. It's now marked done as soon as it's shown.

android/app/src/main/java/net/odysseylabs/powerdeck/data/Model.kt

@@ -96,18 +96,42 @@ data class KvmWiring(val led: String, val power: String, val reset: String) {
     }
 }
 
-/** A machine as a hub sees it (GET /v2/machines). [led] null: not read. */
-data class HubMachine(val name: String, val state: String, val led: Boolean?, val ledError: String?, val kvm: String?, val wiring: KvmWiring?, val actions: List<String>) {
+/**
+ * A machine as a hub sees it (GET /v2/machines). [led] null: not read. From 2.3 the hub's watch adds
+ * [state] "hung", a [reason] for "unknown", [since] and its last look ([evidence]).
+ */
+data class HubMachine(
+    val name: String, val state: String, val led: Boolean?, val ledError: String?, val kvm: String?, val wiring: KvmWiring?, val actions: List<String>,
+    val reason: String? = null, val asleep: Boolean = false, val since: Instant? = null, val evidence: HubEvidence? = null,
+) {
+    val hung get() = state == HUNG
+
     companion object {
+        const val HUNG = "hung"
         fun from(o: JSONObject): HubMachine {
             val k = o.optJSONObject("kvm")
             return HubMachine(o.optString("name"), o.optString("state", "unknown"), if (o.has("led") && !o.isNull("led")) o.optBoolean("led") else null,
                 o.optStringOrNull("ledError"), k?.optStringOrNull("name"), k?.optJSONObject("wiring")?.let(KvmWiring::from),
-                o.optJSONArray("actions")?.strings() ?: emptyList())
+                o.optJSONArray("actions")?.strings() ?: emptyList(),
+                o.optStringOrNull("reason"), o.optBoolean("asleep"), o.instant("since"), o.optJSONObject("evidence")?.let(HubEvidence::from))
         }
     }
 }
 
+/**
+ * The hub's last look at a machine: its helper ("answers", "silent"), the helper's port ("refuses",
+ * "accepts", "no_reply"), a ping ("replies", "no_reply", "unavailable") and the power light ("on",
+ * "off", "blinking", "unreadable"). Null: not looked at (no helper).
+ */
+data class HubEvidence(val at: Instant?, val helper: String?, val port: String?, val ping: String?, val led: String?, val ledTrusted: Boolean) {
+    companion object {
+        fun from(o: JSONObject) = HubEvidence(o.instant("at"), o.optStringOrNull("helper"), o.optStringOrNull("port"), o.optStringOrNull("ping"),
+            o.optStringOrNull("led"), o.optBoolean("ledTrusted"))
+    }
+}
+
+private fun JSONObject.instant(k: String) = optStringOrNull(k)?.let { runCatching { Instant.parse(it) }.getOrNull() }
+
 /** What a hub did for an action: its message, and each step with the power light read. */
 data class HubResult(val ok: Boolean, val code: String?, val message: String, val steps: List<String>) {
     companion object {
@@ -239,7 +263,6 @@ data class HelperUpdate(
     val autoAt: Instant? = null, val pausedUntil: Instant? = null,
 ) {
     companion object {
-        private fun JSONObject.instant(k: String) = optStringOrNull(k)?.let { runCatching { Instant.parse(it) }.getOrNull() }
         fun from(o: JSONObject) = HelperUpdate(o.optBoolean("available"), o.optStringOrNull("latest")?.ifEmpty { null },
             o.optBoolean("installing"), o.optBoolean("auto", true), o.optStringOrNull("error")?.ifEmpty { null },
             o.instant("autoAt"), o.instant("pausedUntil"))
@@ -275,7 +298,8 @@ data class Schedule(val id: String, val action: String, val target: String?, val
 /** Live state of a device, as shown on its card. */
 sealed interface DeviceState {
     data object Checking : DeviceState
-    data class On(val status: HelperStatus?) : DeviceState // status null when only a KVM LED says so
+    /** [status] null when only a KVM LED says so; [hub] is then what the hub that drives the KVM makes of it. */
+    data class On(val status: HelperStatus?, val hub: HubMachine? = null) : DeviceState
     data object Off : DeviceState
     data class Problem(val message: String) : DeviceState
 }

android/app/src/main/java/net/odysseylabs/powerdeck/data/PowerDeck.kt

@@ -367,7 +367,8 @@ class PowerDeck(
         when (ledOn(d)) {
             true -> {
                 if (!d.hasHelper) seeLed(d) // a helper device is confirmed in helperState, with the helper agreeing
-                DeviceState.On(null)
+                // The hub watches it: frozen, booting, asleep… (read with the light, so no extra call)
+                DeviceState.On(null, if (it.hub != null) runCatching { hubMachine(d) }.getOrNull() else null)
             }
             false -> if (it.ledSeen || d.hasHelper) DeviceState.Off
                 else DeviceState.Problem("Turn it on by hand once so PowerDeck can check its power light")

android/app/src/main/java/net/odysseylabs/powerdeck/ui/Components.kt

@@ -62,8 +62,12 @@ import kotlinx.coroutines.delay
 import kotlinx.coroutines.launch
 import net.odysseylabs.powerdeck.data.Device
 import net.odysseylabs.powerdeck.data.DeviceState
+import net.odysseylabs.powerdeck.data.HubMachine
 import java.time.Duration
 import java.time.Instant
+import java.time.ZoneId
+import java.time.format.DateTimeFormatter
+import java.time.format.FormatStyle
 
 /** "Hold for 3 s to activate": how long Shut down and Restart have to be held. */
 const val HOLD_MILLIS = 3000
@@ -110,7 +114,7 @@ fun describe(state: DeviceState?, now: Long): Pair<String, Tone> = when (state)
                 val left = Duration.between(Instant.ofEpochMilli(now), p.at).seconds.coerceAtLeast(0)
                 "${if (p.action == "restart") "Restarting" else "Shutting down"} in ${left / 60}:${"%02d".format(left % 60)}" to Tone.Busy
             }
-            s == null -> "On" to Tone.On
+            s == null -> hubNote(state.hub, now) ?: ("On" to Tone.On)
             s.presence?.active == true -> "On · in use${s.presence.user?.let { " by $it" } ?: ""}" to Tone.On
             else -> "On · up ${uptime(s.uptimeSeconds)}" to Tone.On
         }
@@ -128,13 +132,40 @@ fun note(state: DeviceState?, now: Long): Pair<String, Tone>? = when (state) {
         val p = s?.pending
         when {
             p != null -> describe(state, now)
+            s == null -> hubNote(state.hub, now)
             s?.presence?.active == true -> "In use${s.presence.user?.let { " by $it" } ?: ""}" to Tone.On
             else -> null
         }
     }
     else -> null
 }
 
+/**
+ * What the hub that drives a machine's KVM makes of it while its helper is silent and its light is
+ * on, or null when it has nothing to add (see docs/protocol.md, Machine states and events).
+ */
+fun hubNote(m: HubMachine?, now: Long): Pair<String, Tone>? = when {
+    m == null -> null
+    m.hung -> (m.since?.let { "Frozen since " + clockTime(it, now) } ?: "Frozen") to Tone.Warn
+    else -> when (m.reason) {
+        "booting" -> "Starting up" to Tone.Busy
+        "restarting" -> "Restarting" to Tone.Busy
+        "shutting_down" -> "Shutting down" to Tone.Busy
+        "updating" -> "Installing updates" to Tone.Busy
+        "may_be_asleep" -> "May be asleep" to Tone.Unknown
+        "helper_stopped" -> "Running, but PowerDeck on it isn't answering" to Tone.Warn
+        "checking" -> "Not answering" to Tone.Unknown
+        else -> null
+    }
+}
+
+private val shortTime = DateTimeFormatter.ofLocalizedTime(FormatStyle.SHORT).withZone(ZoneId.systemDefault())
+private val shortDateTime = DateTimeFormatter.ofLocalizedDateTime(FormatStyle.SHORT).withZone(ZoneId.systemDefault())
+
+/** A time of day, with the date when it isn't within the last 20 hours. */
+fun clockTime(at: Instant, now: Long): String =
+    if (Duration.between(at, Instant.ofEpochMilli(now)).toHours() < 20) shortTime.format(at) else shortDateTime.format(at)
+
 @Composable
 fun toneColor(tone: Tone): Color = when (tone) {
     Tone.On -> OnGreen

android/app/src/main/java/net/odysseylabs/powerdeck/ui/DeviceScreen.kt

@@ -75,6 +75,7 @@ import net.odysseylabs.powerdeck.data.Device
 import net.odysseylabs.powerdeck.data.DeviceState
 import net.odysseylabs.powerdeck.data.Event
 import net.odysseylabs.powerdeck.data.HelperStatus
+import net.odysseylabs.powerdeck.data.HubMachine
 import net.odysseylabs.powerdeck.data.KeyInfo
 import net.odysseylabs.powerdeck.data.Peer
 import net.odysseylabs.powerdeck.data.KvmWiring
@@ -145,6 +146,35 @@ private fun Row2(title: String, sub: String? = null, icon: (@Composable () -> Un
     )
 }
 
+/** The hub's last look at [name]: each signal in words, so the owner can see why it's called frozen. */
+@Composable
+private fun HubEvidenceSection(name: String, hubName: String, m: HubMachine) {
+    val e = m.evidence ?: return
+    Section("What $hubName sees") {
+        e.helper?.let {
+            Row2("PowerDeck on $name", if (it == "answers") "Answers" else "Not answering")
+        }
+        val network = listOfNotNull(
+            when (e.ping) { "replies" -> "Answers ping"; "no_reply" -> "No reply to ping"; else -> null },
+            when (e.port) {
+                "refuses" -> "turns PowerDeck's connection away (it's up, PowerDeck isn't running)"
+                "accepts" -> "takes PowerDeck's connection but says nothing"
+                "no_reply" -> "nothing on PowerDeck's port"
+                else -> null
+            },
+        )
+        if (network.isNotEmpty()) Row2("Network", network.joinToString("; ").replaceFirstChar(Char::uppercase))
+        Row2("Power light", when (e.led) {
+            "on" -> "On"
+            "off" -> "Off"
+            "blinking" -> "Blinking (asleep)"
+            else -> "Can't be read"
+        } + if (!e.ledTrusted) " · not known to be connected yet" else "")
+        // The card above already says what it makes of it.
+        e.at?.let { at -> Row2("Last checked " + clockTime(at, System.currentTimeMillis()), if (m.hung) "A reset brings it back; unsaved work is lost" else null) }
+    }
+}
+
 @OptIn(ExperimentalMaterial3Api::class, ExperimentalLayoutApi::class)
 @Composable
 fun DeviceScreen(d: Device, state: UiState, detail: Detail, a: DeviceActions, snackbar: SnackbarHostState) {
@@ -238,6 +268,11 @@ fun DeviceScreen(d: Device, state: UiState, detail: Detail, a: DeviceActions, sn
                 }
             }
 
+            // ---- What the hub sees, while the helper is silent and the light is on (frozen, booting…)
+            (st as? DeviceState.On)?.hub?.takeIf { status == null && it.evidence != null && (it.hung || it.reason != null) }?.let { m ->
+                HubEvidenceSection(d.name, hub?.name ?: "The hub", m)
+            }
+
             // ---- PowerDeck helper update
             status?.update?.takeIf { it.available || it.installing }?.let { u ->
                 Section("Update") {

android/app/src/main/java/net/odysseylabs/powerdeck/work/EventsWorker.kt

@@ -88,6 +88,7 @@ class EventsWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ctx
     }
 
     private fun notable(e: Event, d: Device, helpers: Set<String>): Boolean {
+        if (hubWatch(e)) return true
         if (d.me != null && (e.who == d.me || e.who.startsWith("${d.me} via "))) return false
         val verb = e.action.substringBefore(' ').lowercase()
         val target = e.action.substringAfter(' ', "").lowercase()
@@ -97,6 +98,7 @@ class EventsWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ctx
     }
 
     private fun describe(e: Event, d: Device): String {
+        if (hubWatch(e)) return e.action + "." // already a sentence: "Workstation looks frozen: …"
         val verb = e.action.substringBefore(' ')
         val target = e.action.substringAfter(' ', "").ifBlank { d.name }
         val what = when (verb) {
@@ -113,6 +115,10 @@ class EventsWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ctx
     }
 
     companion object {
+        /** A hub (2.3+) saw a machine with a KVM freeze, or answer again after that. */
+        internal fun hubWatch(e: Event) =
+            e.result == "detected" && e.action.contains(" looks frozen") || e.result == "recovered" && e.action.endsWith(" answers again")
+
         private const val NAME = "events"
         private const val CHANNEL = "activity"
         private const val UPDATES = "updates"

android/app/src/test/java/net/odysseylabs/powerdeck/ScreenshotTest.kt

@@ -52,6 +52,8 @@ import net.odysseylabs.powerdeck.data.DeviceState
 import net.odysseylabs.powerdeck.data.Event
 import net.odysseylabs.powerdeck.data.HelperStatus
 import net.odysseylabs.powerdeck.data.HelperUpdate
+import net.odysseylabs.powerdeck.data.HubEvidence
+import net.odysseylabs.powerdeck.data.HubMachine
 import net.odysseylabs.powerdeck.data.KvmLink
 import net.odysseylabs.powerdeck.data.KvmWiring
 import net.odysseylabs.powerdeck.data.Peer
@@ -269,6 +271,25 @@ class ScreenshotTest {
         compose.onRoot().captureRoboImage("build/outputs/roborazzi/device-frozen.png")
     }
 
+    /** The hub calls it hung: the device page says so, with what the hub saw, and still offers Reset. */
+    @Test fun deviceHungOnHub() {
+        val ws = Device("ws", "Workstation", "100.101.1.11", pin = "w", os = "windows", mac = "02:00:5e:00:00:01",
+            kvm = KvmLink("nanokvm", "kvm-2.tail1234.ts.net", "kvm-2", "admin", ledSeen = true, hub = "pve", machine = "workstation",
+                wiring = KvmWiring("wired", "wired", "unknown")))
+        val seen = java.time.Instant.now().minusSeconds(20)
+        val m = HubMachine("workstation", "hung", true, null, "kvm-2", ws.kvm!!.wiring, listOf("reset", "force-off"),
+            since = seen.minusSeconds(200), evidence = HubEvidence(seen, "silent", "accepts", "no_reply", "on", true))
+        val state = all.copy(devices = all.devices + ws, states = all.states + ("ws" to DeviceState.On(null, m)))
+        shot("device-hung") { DeviceScreen(ws, state, Detail(events = emptyList()), DeviceActions(), SnackbarHostState()) }
+        compose.onNodeWithText("Frozen since", substring = true).assertIsDisplayed()
+        compose.onNodeWithText("A reset brings it back; unsaved work is lost").performScrollTo().assertIsDisplayed()
+        compose.onNodeWithText("Reset").assertIsDisplayed()
+        compose.onNodeWithText("What pve sees").performScrollTo().assertIsDisplayed()
+        compose.onNodeWithText("No reply to ping; takes PowerDeck's connection but says nothing").assertExists()
+        compose.onNodeWithText("Not answering").assertExists()
+        compose.onRoot().captureRoboImage("build/outputs/roborazzi/device-hung.png")
+    }
+
     @Test fun settingsHub() {
         shot("settings-hub") { SettingsScreen(Settings(), SettingsActions(), hubs = listOf(pc, pve), showHub = true) }
         compose.onNodeWithText("Automatic: pve").assertExists()