fix(execution): remove CC's SCM dependency edges so the monitor watchdog's restart isn't refused (ALP-945)

scripts/RUNBOOK_command_center.md

@@ -56,19 +56,21 @@ per-step PASS / FAIL report.
    `install_pipeline_scheduler_service.ps1` /
    `install_monitor_service.ps1`. NSSM ships as a single .exe; drop into a
    directory on PATH.
-6. **Pipeline + monitor services already installed.** The command center
-   declares both as service dependencies (see
-   `install_command_center_service.ps1`); a missing dependency causes the
-   Windows SCM to refuse the start.
+6. **No service-ordering prerequisite.** The command center declares no SCM
+   service dependencies (ALP-945) — it installs and starts whether or not
+   the scheduler / monitor services exist yet. Its SSE consumers reconnect
+   with capped backoff until the upstreams come up, and the control proxy
+   reaches them per-request.
 
-The full prod runtime is **six** NSSM services (see `RUNBOOK_production.md`'s
+The full prod runtime is **seven** NSSM services (see `RUNBOOK_production.md`'s
 service table): `alphamind-collector`, `alphamind-scheduler`,
-`alphamind-monitor`, `alphamind-safety-core`, `alphamind-safety-core-watchdog`,
-and `AlphaMindCommandCenter`. The `alphamind-safety-core` service is the isolated
-breach + price-staleness safety core (ADR-0004 / ALP-857); its dedicated
-out-of-process watchdog `alphamind-safety-core-watchdog` restarts it on heartbeat
-staleness. The command center does **not** depend on the safety-core services and
-does not surface their state — they are monitored via their logs
+`alphamind-monitor`, `alphamind-monitor-watchdog`, `alphamind-safety-core`,
+`alphamind-safety-core-watchdog`, and `AlphaMindCommandCenter`. The
+`alphamind-safety-core` service is the isolated breach + price-staleness safety
+core (ADR-0004 / ALP-857); each supervised service's dedicated out-of-process
+watchdog (`alphamind-monitor-watchdog` / `alphamind-safety-core-watchdog`,
+ALP-941) restarts its target on heartbeat staleness. The command center does
+not surface the safety-core services' state — they are monitored via their logs
 (`safety_core.*.log`) and the heartbeat file, per `RUNBOOK_production.md` §7/§9.
 
 ## Initial bring-up
@@ -120,7 +122,7 @@ The install script wires:
 - `python -m alphamind.command_center` as the entry point.
 - AppExit Default Restart with a 60 s throttle.
 - Stdout / stderr → `%USERPROFILE%\AlphaMind\logs\command_center.{out,err}.log`.
-- Dependency on `alphamind-scheduler` + `alphamind-monitor`.
+- No SCM service dependencies (ALP-945; see prerequisite 6).
 
 The `ObjectName` step is manual — NSSM prompts for the account password
 interactively, which keeps it out of shell history.
@@ -378,11 +380,13 @@ The command center's three owned tables (`alerts`, `webauthn_credentials`,
 pipeline + monitor's state. The canonical recovery is the same as for the
 pipeline / monitor:
 
-1. Stop all six services (reverse-dependency order; stop watchdog before core):
+1. Stop all seven services (reverse start order — operational convention,
+   not SCM-enforced; stop each watchdog before the service it supervises):
    ```powershell
    nssm stop AlphaMindCommandCenter
    nssm stop alphamind-safety-core-watchdog
    nssm stop alphamind-safety-core
+   nssm stop alphamind-monitor-watchdog
    nssm stop alphamind-monitor
    nssm stop alphamind-scheduler
    nssm stop alphamind-collector
@@ -391,11 +395,13 @@ pipeline / monitor:
    `archive/` directory carries periodic snapshots; pick the most recent
    verified-good copy).
 3. `uv run alembic upgrade head` to bring the restored DB's schema current.
-4. Restart all six services in dependency order (core before watchdog):
+4. Restart all seven services in the conventional start order (each
+   supervised service before its watchdog):
    ```powershell
    nssm start alphamind-collector
    nssm start alphamind-scheduler
    nssm start alphamind-monitor
+   nssm start alphamind-monitor-watchdog
    nssm start alphamind-safety-core
    nssm start alphamind-safety-core-watchdog
    nssm start AlphaMindCommandCenter

scripts/RUNBOOK_production.md

@@ -117,13 +117,17 @@ uv run alembic heads       # what the codebase expects
 If the two revisions match, **skip to § 1.6**. Nothing to migrate; just
 restart any services whose code changed.
 
-### 1.4 Stop services in reverse dependency order
-
-The command center depends on scheduler + monitor; stop it first so its
-loopback consumers don't reconnect mid-migration. Collector is independent of
-the trading DB schema but stop it too to keep the snapshot quiet during the
-migration. Stop each **watchdog before the service it supervises** so the
-watchdog does not `nssm restart` its target while you are taking it down.
+### 1.4 Stop services (reverse of the start order)
+
+The stop order is operational convention — no service declares an SCM
+dependency on another, so the SCM enforces no ordering (the topology
+invariant; § 7). Stop the
+command center first so its loopback consumers sit quiet through the
+migration window rather than reconnect-looping against stopped upstreams.
+Collector is independent of the trading DB schema but stop it too to keep
+the snapshot quiet during the migration. Stop each **watchdog before the
+service it supervises** so the watchdog does not `nssm restart` its target
+while you are taking it down.
 
 ```powershell
 nssm stop AlphaMindCommandCenter
@@ -151,11 +155,14 @@ If the upgrade fails, **do not restart services.** Read the traceback,
 restore the DB from the most recent backup (see `RUNBOOK_command_center.md`
 § Recovery → DB corruption restore), then re-apply.
 
-### 1.6 Restart services in dependency order
+### 1.6 Restart services (conventional start order)
 
-Start each supervised service **before its watchdog** so the watchdog finds a
-fresh heartbeat on its first probe (and does not restart a service that is
-still coming up).
+The start order is operational convention, not SCM-enforced. Start each
+supervised service **before its watchdog** so the watchdog finds a fresh
+heartbeat on its first probe (and does not restart a service that is still
+coming up). The command center comes last only by symmetry with § 1.4 — it
+starts fine before its upstreams (its consumers reconnect with capped
+backoff until they come up).
 
 ```powershell
 nssm start alphamind-collector
@@ -1052,7 +1059,15 @@ per daemon; the scheduler supervisor uses a manually-bounded wait rather than
 in the window are abandoned and NSSM/the SCM force-terminates the process; there
 is no separate SIGTERM "fallback" the supervisor itself fires.
 
-**Two restart gotchas that cost real diagnosis time (2026-06-02 monitor wedge):**
+A plain `nssm restart <service>` works for any single service with all the
+others running: no service declares an SCM dependency on another, so the SCM
+never refuses the stop half of the restart. That is a deliberate topology
+invariant (ALP-945, the 2026-06-10 incident — a `DependOnService` edge from
+the command center made the SCM refuse every monitor-watchdog restart,
+wedging the monitor for 42 minutes): **never declare an SCM dependency on a
+watchdog-supervised service.**
+
+**One restart gotcha that cost real diagnosis time (2026-06-02 monitor wedge):**
 
 - **`Get-Service … Running` does NOT mean healthy.** NSSM reports `Running`
   while the *wrapper* process is alive. Two liveness layers now apply:
@@ -1093,11 +1108,6 @@ is no separate SIGTERM "fallback" the supervisor itself fires.
   (Get-Process -Id $p).StartTime
   ```
   See § 8.9 for the fill-stream and underlying-stream auto-recovery details.
-- **Restarting `alphamind-scheduler` or `alphamind-monitor` while
-  `AlphaMindCommandCenter` is up can be silently refused** (the command center
-  depends on both). Stop CC first, restart the target, then start CC — stop
-  reverse / start forward: `nssm stop AlphaMindCommandCenter` →
-  `nssm restart <target>` → `nssm start AlphaMindCommandCenter`.
 
 **When the operator can restart a single service without coordinating the
 others:**
@@ -1242,10 +1252,9 @@ Both the fill (trade-updates) stream and the underlying-price stream now have
 
 **Manual § 7 restart is the fallback** when auto-recovery is itself suspect (e.g.
 the StartTime is stale, no `8766` heartbeat is appearing, or `monitor.err.log`
-shows a `reconnect budget exhausted` entry). Use the CC-dependency order:
-`nssm stop AlphaMindCommandCenter` → `nssm stop` + `nssm start alphamind-monitor`
-→ `nssm start AlphaMindCommandCenter`. Confirm by StartTime + heartbeat, not
-`Get-Service`.
+shows a `reconnect budget exhausted` entry): `nssm stop alphamind-monitor` +
+`nssm start alphamind-monitor` — no other service needs touching. Confirm by
+StartTime + heartbeat, not `Get-Service`.
 
 #### Pre-fix history (2026-06-02 — for reference only)
 

scripts/install_command_center_service.ps1

@@ -10,11 +10,11 @@
     - Automatic (Delayed Start) start type
     - Stdout / stderr redirected to %USERPROFILE%\AlphaMind\logs\
     - AppDirectory set to the project root
-    - DependsOnService = AlphaMindPipeline + AlphaMindMonitor so the
-      command center starts only after the pipeline + monitor processes
-      have come up (the SSE consumer tasks fail-fast when their
-      upstreams are unreachable, so a too-eager start would cycle
-      through restarts; the dependency makes the boot ordering explicit).
+    - No SCM service-ordering edges: the command center starts on its
+      own, before or after its upstreams. Its SSE consumers reconnect
+      with capped backoff until the scheduler / monitor come up
+      (command_center/events/multiplexer.py), and the control proxy
+      reaches upstreams per-request (ALP-945).
 
     The ObjectName (service account) configuration is left as a manual
     final step documented in RUNBOOK_command_center.md to avoid
@@ -33,9 +33,6 @@
       src/alphamind/command_center/frontend/dist/ (run "bun install
       --frozen-lockfile && bun run build" before this script — see
       RUNBOOK_command_center.md § Initial bring-up).
-    - The pipeline + monitor services must already be installed (this
-      script declares them as dependencies; absent services cause
-      Windows to refuse to start the command center).
     - Tested on Windows 10/11 with PowerShell 5.1+
 #>
 
@@ -48,13 +45,6 @@ $ErrorActionPreference = 'Stop'
 
 $ServiceName = 'AlphaMindCommandCenter'
 
-# The two services this one depends on. Their install scripts:
-#   - install_pipeline_scheduler_service.ps1  -> alphamind-scheduler
-#   - install_monitor_service.ps1             -> alphamind-monitor
-# The dependency names below match those service names verbatim.
-$PipelineDependency = 'alphamind-scheduler'
-$MonitorDependency = 'alphamind-monitor'
-
 # Resolve the project root as the directory containing this script's parent.
 # Script lives in <repo>\scripts\; project root is one level up.
 $ScriptDir  = Split-Path -Parent $MyInvocation.MyCommand.Path
@@ -114,7 +104,6 @@ Write-Host "  Python:      $PythonExe"
 Write-Host "  ProjectRoot: $ProjectRoot"
 Write-Host "  Stdout log:  $StdoutLog"
 Write-Host "  Stderr log:  $StderrLog"
-Write-Host "  Depends on:  $PipelineDependency, $MonitorDependency"
 Write-Host ""
 
 # Install: nssm install <service> <exe> <args...>
@@ -141,11 +130,13 @@ nssm set $ServiceName AppStderr $StderrLog
 nssm set $ServiceName AppStdoutCreationDisposition 4
 nssm set $ServiceName AppStderrCreationDisposition 4
 
-# Declare service dependencies — Windows will only start
-# AlphaMindCommandCenter after alphamind-scheduler + alphamind-monitor
-# have entered the Running state. The SCM honors the dependency on
-# both Start and Stop (cmd center stops first on a system shutdown).
-nssm set $ServiceName DependOnService $PipelineDependency $MonitorDependency
+# Deliberately NO SCM service-ordering edge on the scheduler / monitor:
+# the command center starts on its own — its SSE consumers reconnect with
+# capped backoff until the upstreams come up (events/multiplexer.py), and
+# the control proxy reaches them per-request. An SCM edge pointing at a
+# watchdog-supervised service would also make the SCM refuse the watchdog's
+# `nssm restart` (stop + start) while this service runs — the 2026-06-10
+# 42-minute monitor wedge (ALP-945).
 
 # ---------------------------------------------------------------------------
 # Summary and next steps

scripts/install_monitor_service.ps1

@@ -59,6 +59,12 @@ $ErrorActionPreference = 'Stop'
 $MonitorServiceName  = 'alphamind-monitor'
 $WatchdogServiceName = 'alphamind-monitor-watchdog'
 
+# Topology invariant (ALP-945, the 2026-06-10 incident): no service may
+# declare an SCM dependency (DependOnService) on alphamind-monitor. The
+# watchdog's `nssm restart` is stop + start, and the SCM refuses the stop
+# while a dependent service runs — a wedged monitor would then be
+# unrestartable for as long as the dependent stays up.
+
 # Resolve the project root as the directory containing this script's parent.
 # Script lives in <repo>\scripts\; project root is one level up.
 $ScriptDir   = Split-Path -Parent $MyInvocation.MyCommand.Path

scripts/install_safety_core_service.ps1

@@ -55,6 +55,12 @@ $ErrorActionPreference = 'Stop'
 $CoreServiceName     = 'alphamind-safety-core'
 $WatchdogServiceName = 'alphamind-safety-core-watchdog'
 
+# Topology invariant (ALP-945, the 2026-06-10 incident): no service may
+# declare an SCM dependency (DependOnService) on alphamind-safety-core. The
+# watchdog's `nssm restart` is stop + start, and the SCM refuses the stop
+# while a dependent service runs — a wedged safety core would then be
+# unrestartable for as long as the dependent stays up.
+
 # Resolve the project root as the directory containing this script's parent.
 # Script lives in <repo>\scripts\; project root is one level up.
 $ScriptDir   = Split-Path -Parent $MyInvocation.MyCommand.Path

src/alphamind/execution/process_supervision/process_control.py

@@ -24,8 +24,16 @@
 
 
 def _checked_run(cmd: list[str]) -> None:
-    """Default runner: ``subprocess.run`` with ``check=True`` (raises on failure)."""
-    subprocess.run(cmd, check=True)
+    """Default runner: ``subprocess.run`` with ``check=True`` (raises on failure).
+
+    Output is captured so the raised :class:`subprocess.CalledProcessError`
+    carries the child's stderr — the SCM's refusal reason lives there, not in
+    the exit status (ALP-945). ``errors="replace"`` keeps a non-decodable byte
+    in that output from raising ``UnicodeDecodeError`` instead of the
+    ``CalledProcessError`` the caller logs. On success the captured output is
+    discarded (it no longer passes through to the watchdog's own stdio).
+    """
+    subprocess.run(cmd, check=True, capture_output=True, text=True, errors="replace")
 
 
 class NssmServiceController:
@@ -52,6 +60,13 @@ def restart(self) -> None:
         log.critical("watchdog: restarting service %r", self._service_name)
         try:
             self._run(list(cmd))
+        except subprocess.CalledProcessError as exc:
+            # The SCM's refusal reason lives in the child's stderr (ALP-945).
+            log.exception(
+                "watchdog: nssm restart of %r failed (stderr: %s); will retry next tick",
+                self._service_name,
+                exc.stderr or "<no stderr captured>",
+            )
         except Exception:
             log.exception(
                 "watchdog: nssm restart of %r failed; will retry next tick", self._service_name

tests/execution/process_supervision/test_process_control.py

@@ -8,8 +8,15 @@
 
 from __future__ import annotations
 
+import logging
+import subprocess
+import sys
+
+import pytest
+
 from alphamind.execution.process_supervision.process_control import (
     NssmServiceController,
+    _checked_run,
 )
 
 
@@ -36,3 +43,52 @@ def _boom(cmd: list[str]) -> None:
     controller = NssmServiceController(service_name="alphamind-monitor", run=_boom)
 
     controller.restart()  # must not raise
+
+
+def test_nssm_controller_failure_log_carries_the_shell_outs_stderr(
+    caplog: pytest.LogCaptureFixture,
+) -> None:
+    """A ``CalledProcessError``'s stderr reaches the failure log line.
+
+    The 2026-06-10 incident (ALP-945): sixteen refused restarts logged only
+    "exit status 1" while the SCM's refusal reason sat in the child's stderr.
+    The log line must carry that reason — and the error is still swallowed so
+    the watchdog keeps probing.
+    """
+    refusal = (
+        "alphamind-monitor: STOP: A stop control has been sent to a service "
+        "that other running services are dependent on."
+    )
+
+    def _refused(cmd: list[str]) -> None:
+        raise subprocess.CalledProcessError(1, cmd, stderr=refusal)
+
+    controller = NssmServiceController(service_name="alphamind-monitor", run=_refused)
+
+    with caplog.at_level(
+        logging.ERROR, logger="alphamind.execution.process_supervision.process_control"
+    ):
+        controller.restart()  # must not raise
+
+    assert refusal in caplog.text
+
+
+def test_checked_run_attaches_the_childs_stderr_to_the_raised_error() -> None:
+    """The default runner captures the child's stderr onto ``CalledProcessError``.
+
+    Exercised against a real subprocess — the runner IS the OS-process
+    boundary, so there is nothing to fake below it. The child's stderr
+    includes a byte no text encoding need accept: a non-decodable byte must
+    degrade to a replacement character, not raise ``UnicodeDecodeError`` in
+    place of the ``CalledProcessError`` the watchdog logs.
+    """
+    cmd = [
+        sys.executable,
+        "-c",
+        "import sys; sys.stderr.buffer.write(b'stop control refused \\xff'); sys.exit(1)",
+    ]
+
+    with pytest.raises(subprocess.CalledProcessError) as excinfo:
+        _checked_run(cmd)
+
+    assert "stop control refused" in excinfo.value.stderr