diff --git a/scripts/RUNBOOK_command_center.md b/scripts/RUNBOOK_command_center.md index f69d29d8..9ef35997 100644 --- a/scripts/RUNBOOK_command_center.md +++ b/scripts/RUNBOOK_command_center.md @@ -56,19 +56,21 @@ per-step PASS / FAIL report. `install_pipeline_scheduler_service.ps1` / `install_monitor_service.ps1`. NSSM ships as a single .exe; drop into a directory on PATH. -6. **Pipeline + monitor services already installed.** The command center - declares both as service dependencies (see - `install_command_center_service.ps1`); a missing dependency causes the - Windows SCM to refuse the start. +6. **No service-ordering prerequisite.** The command center declares no SCM + service dependencies (ALP-945) — it installs and starts whether or not + the scheduler / monitor services exist yet. Its SSE consumers reconnect + with capped backoff until the upstreams come up, and the control proxy + reaches them per-request. -The full prod runtime is **six** NSSM services (see `RUNBOOK_production.md`'s +The full prod runtime is **seven** NSSM services (see `RUNBOOK_production.md`'s service table): `alphamind-collector`, `alphamind-scheduler`, -`alphamind-monitor`, `alphamind-safety-core`, `alphamind-safety-core-watchdog`, -and `AlphaMindCommandCenter`. The `alphamind-safety-core` service is the isolated -breach + price-staleness safety core (ADR-0004 / ALP-857); its dedicated -out-of-process watchdog `alphamind-safety-core-watchdog` restarts it on heartbeat -staleness. The command center does **not** depend on the safety-core services and -does not surface their state — they are monitored via their logs +`alphamind-monitor`, `alphamind-monitor-watchdog`, `alphamind-safety-core`, +`alphamind-safety-core-watchdog`, and `AlphaMindCommandCenter`. The +`alphamind-safety-core` service is the isolated breach + price-staleness safety +core (ADR-0004 / ALP-857); each supervised service's dedicated out-of-process +watchdog (`alphamind-monitor-watchdog` / `alphamind-safety-core-watchdog`, +ALP-941) restarts its target on heartbeat staleness. The command center does +not surface the safety-core services' state — they are monitored via their logs (`safety_core.*.log`) and the heartbeat file, per `RUNBOOK_production.md` §7/§9. ## Initial bring-up @@ -120,7 +122,7 @@ The install script wires: - `python -m alphamind.command_center` as the entry point. - AppExit Default Restart with a 60 s throttle. - Stdout / stderr → `%USERPROFILE%\AlphaMind\logs\command_center.{out,err}.log`. -- Dependency on `alphamind-scheduler` + `alphamind-monitor`. +- No SCM service dependencies (ALP-945; see prerequisite 6). The `ObjectName` step is manual — NSSM prompts for the account password interactively, which keeps it out of shell history. @@ -378,11 +380,13 @@ The command center's three owned tables (`alerts`, `webauthn_credentials`, pipeline + monitor's state. The canonical recovery is the same as for the pipeline / monitor: -1. Stop all six services (reverse-dependency order; stop watchdog before core): +1. Stop all seven services (reverse start order — operational convention, + not SCM-enforced; stop each watchdog before the service it supervises): ```powershell nssm stop AlphaMindCommandCenter nssm stop alphamind-safety-core-watchdog nssm stop alphamind-safety-core + nssm stop alphamind-monitor-watchdog nssm stop alphamind-monitor nssm stop alphamind-scheduler nssm stop alphamind-collector @@ -391,11 +395,13 @@ pipeline / monitor: `archive/` directory carries periodic snapshots; pick the most recent verified-good copy). 3. `uv run alembic upgrade head` to bring the restored DB's schema current. -4. Restart all six services in dependency order (core before watchdog): +4. Restart all seven services in the conventional start order (each + supervised service before its watchdog): ```powershell nssm start alphamind-collector nssm start alphamind-scheduler nssm start alphamind-monitor + nssm start alphamind-monitor-watchdog nssm start alphamind-safety-core nssm start alphamind-safety-core-watchdog nssm start AlphaMindCommandCenter diff --git a/scripts/RUNBOOK_production.md b/scripts/RUNBOOK_production.md index 53b0885d..6a2ce294 100644 --- a/scripts/RUNBOOK_production.md +++ b/scripts/RUNBOOK_production.md @@ -117,13 +117,17 @@ uv run alembic heads # what the codebase expects If the two revisions match, **skip to § 1.6**. Nothing to migrate; just restart any services whose code changed. -### 1.4 Stop services in reverse dependency order - -The command center depends on scheduler + monitor; stop it first so its -loopback consumers don't reconnect mid-migration. Collector is independent of -the trading DB schema but stop it too to keep the snapshot quiet during the -migration. Stop each **watchdog before the service it supervises** so the -watchdog does not `nssm restart` its target while you are taking it down. +### 1.4 Stop services (reverse of the start order) + +The stop order is operational convention — no service declares an SCM +dependency on another, so the SCM enforces no ordering (the topology +invariant; § 7). Stop the +command center first so its loopback consumers sit quiet through the +migration window rather than reconnect-looping against stopped upstreams. +Collector is independent of the trading DB schema but stop it too to keep +the snapshot quiet during the migration. Stop each **watchdog before the +service it supervises** so the watchdog does not `nssm restart` its target +while you are taking it down. ```powershell nssm stop AlphaMindCommandCenter @@ -151,11 +155,14 @@ If the upgrade fails, **do not restart services.** Read the traceback, restore the DB from the most recent backup (see `RUNBOOK_command_center.md` § Recovery → DB corruption restore), then re-apply. -### 1.6 Restart services in dependency order +### 1.6 Restart services (conventional start order) -Start each supervised service **before its watchdog** so the watchdog finds a -fresh heartbeat on its first probe (and does not restart a service that is -still coming up). +The start order is operational convention, not SCM-enforced. Start each +supervised service **before its watchdog** so the watchdog finds a fresh +heartbeat on its first probe (and does not restart a service that is still +coming up). The command center comes last only by symmetry with § 1.4 — it +starts fine before its upstreams (its consumers reconnect with capped +backoff until they come up). ```powershell nssm start alphamind-collector @@ -1052,7 +1059,15 @@ per daemon; the scheduler supervisor uses a manually-bounded wait rather than in the window are abandoned and NSSM/the SCM force-terminates the process; there is no separate SIGTERM "fallback" the supervisor itself fires. -**Two restart gotchas that cost real diagnosis time (2026-06-02 monitor wedge):** +A plain `nssm restart ` works for any single service with all the +others running: no service declares an SCM dependency on another, so the SCM +never refuses the stop half of the restart. That is a deliberate topology +invariant (ALP-945, the 2026-06-10 incident — a `DependOnService` edge from +the command center made the SCM refuse every monitor-watchdog restart, +wedging the monitor for 42 minutes): **never declare an SCM dependency on a +watchdog-supervised service.** + +**One restart gotcha that cost real diagnosis time (2026-06-02 monitor wedge):** - **`Get-Service … Running` does NOT mean healthy.** NSSM reports `Running` while the *wrapper* process is alive. Two liveness layers now apply: @@ -1093,11 +1108,6 @@ is no separate SIGTERM "fallback" the supervisor itself fires. (Get-Process -Id $p).StartTime ``` See § 8.9 for the fill-stream and underlying-stream auto-recovery details. -- **Restarting `alphamind-scheduler` or `alphamind-monitor` while - `AlphaMindCommandCenter` is up can be silently refused** (the command center - depends on both). Stop CC first, restart the target, then start CC — stop - reverse / start forward: `nssm stop AlphaMindCommandCenter` → - `nssm restart ` → `nssm start AlphaMindCommandCenter`. **When the operator can restart a single service without coordinating the others:** @@ -1242,10 +1252,9 @@ Both the fill (trade-updates) stream and the underlying-price stream now have **Manual § 7 restart is the fallback** when auto-recovery is itself suspect (e.g. the StartTime is stale, no `8766` heartbeat is appearing, or `monitor.err.log` -shows a `reconnect budget exhausted` entry). Use the CC-dependency order: -`nssm stop AlphaMindCommandCenter` → `nssm stop` + `nssm start alphamind-monitor` -→ `nssm start AlphaMindCommandCenter`. Confirm by StartTime + heartbeat, not -`Get-Service`. +shows a `reconnect budget exhausted` entry): `nssm stop alphamind-monitor` + +`nssm start alphamind-monitor` — no other service needs touching. Confirm by +StartTime + heartbeat, not `Get-Service`. #### Pre-fix history (2026-06-02 — for reference only) diff --git a/scripts/install_command_center_service.ps1 b/scripts/install_command_center_service.ps1 index f5769b9c..4a136588 100644 --- a/scripts/install_command_center_service.ps1 +++ b/scripts/install_command_center_service.ps1 @@ -10,11 +10,11 @@ - Automatic (Delayed Start) start type - Stdout / stderr redirected to %USERPROFILE%\AlphaMind\logs\ - AppDirectory set to the project root - - DependsOnService = AlphaMindPipeline + AlphaMindMonitor so the - command center starts only after the pipeline + monitor processes - have come up (the SSE consumer tasks fail-fast when their - upstreams are unreachable, so a too-eager start would cycle - through restarts; the dependency makes the boot ordering explicit). + - No SCM service-ordering edges: the command center starts on its + own, before or after its upstreams. Its SSE consumers reconnect + with capped backoff until the scheduler / monitor come up + (command_center/events/multiplexer.py), and the control proxy + reaches upstreams per-request (ALP-945). The ObjectName (service account) configuration is left as a manual final step documented in RUNBOOK_command_center.md to avoid @@ -33,9 +33,6 @@ src/alphamind/command_center/frontend/dist/ (run "bun install --frozen-lockfile && bun run build" before this script — see RUNBOOK_command_center.md § Initial bring-up). - - The pipeline + monitor services must already be installed (this - script declares them as dependencies; absent services cause - Windows to refuse to start the command center). - Tested on Windows 10/11 with PowerShell 5.1+ #> @@ -48,13 +45,6 @@ $ErrorActionPreference = 'Stop' $ServiceName = 'AlphaMindCommandCenter' -# The two services this one depends on. Their install scripts: -# - install_pipeline_scheduler_service.ps1 -> alphamind-scheduler -# - install_monitor_service.ps1 -> alphamind-monitor -# The dependency names below match those service names verbatim. -$PipelineDependency = 'alphamind-scheduler' -$MonitorDependency = 'alphamind-monitor' - # Resolve the project root as the directory containing this script's parent. # Script lives in \scripts\; project root is one level up. $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path @@ -114,7 +104,6 @@ Write-Host " Python: $PythonExe" Write-Host " ProjectRoot: $ProjectRoot" Write-Host " Stdout log: $StdoutLog" Write-Host " Stderr log: $StderrLog" -Write-Host " Depends on: $PipelineDependency, $MonitorDependency" Write-Host "" # Install: nssm install @@ -141,11 +130,13 @@ nssm set $ServiceName AppStderr $StderrLog nssm set $ServiceName AppStdoutCreationDisposition 4 nssm set $ServiceName AppStderrCreationDisposition 4 -# Declare service dependencies — Windows will only start -# AlphaMindCommandCenter after alphamind-scheduler + alphamind-monitor -# have entered the Running state. The SCM honors the dependency on -# both Start and Stop (cmd center stops first on a system shutdown). -nssm set $ServiceName DependOnService $PipelineDependency $MonitorDependency +# Deliberately NO SCM service-ordering edge on the scheduler / monitor: +# the command center starts on its own — its SSE consumers reconnect with +# capped backoff until the upstreams come up (events/multiplexer.py), and +# the control proxy reaches them per-request. An SCM edge pointing at a +# watchdog-supervised service would also make the SCM refuse the watchdog's +# `nssm restart` (stop + start) while this service runs — the 2026-06-10 +# 42-minute monitor wedge (ALP-945). # --------------------------------------------------------------------------- # Summary and next steps diff --git a/scripts/install_monitor_service.ps1 b/scripts/install_monitor_service.ps1 index c23fc7a6..8b4214b5 100644 --- a/scripts/install_monitor_service.ps1 +++ b/scripts/install_monitor_service.ps1 @@ -59,6 +59,12 @@ $ErrorActionPreference = 'Stop' $MonitorServiceName = 'alphamind-monitor' $WatchdogServiceName = 'alphamind-monitor-watchdog' +# Topology invariant (ALP-945, the 2026-06-10 incident): no service may +# declare an SCM dependency (DependOnService) on alphamind-monitor. The +# watchdog's `nssm restart` is stop + start, and the SCM refuses the stop +# while a dependent service runs — a wedged monitor would then be +# unrestartable for as long as the dependent stays up. + # Resolve the project root as the directory containing this script's parent. # Script lives in \scripts\; project root is one level up. $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path diff --git a/scripts/install_safety_core_service.ps1 b/scripts/install_safety_core_service.ps1 index 1d1a70e3..04ba217c 100644 --- a/scripts/install_safety_core_service.ps1 +++ b/scripts/install_safety_core_service.ps1 @@ -55,6 +55,12 @@ $ErrorActionPreference = 'Stop' $CoreServiceName = 'alphamind-safety-core' $WatchdogServiceName = 'alphamind-safety-core-watchdog' +# Topology invariant (ALP-945, the 2026-06-10 incident): no service may +# declare an SCM dependency (DependOnService) on alphamind-safety-core. The +# watchdog's `nssm restart` is stop + start, and the SCM refuses the stop +# while a dependent service runs — a wedged safety core would then be +# unrestartable for as long as the dependent stays up. + # Resolve the project root as the directory containing this script's parent. # Script lives in \scripts\; project root is one level up. $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path diff --git a/src/alphamind/execution/process_supervision/process_control.py b/src/alphamind/execution/process_supervision/process_control.py index ea98b19f..84cc29d4 100644 --- a/src/alphamind/execution/process_supervision/process_control.py +++ b/src/alphamind/execution/process_supervision/process_control.py @@ -24,8 +24,16 @@ def _checked_run(cmd: list[str]) -> None: - """Default runner: ``subprocess.run`` with ``check=True`` (raises on failure).""" - subprocess.run(cmd, check=True) + """Default runner: ``subprocess.run`` with ``check=True`` (raises on failure). + + Output is captured so the raised :class:`subprocess.CalledProcessError` + carries the child's stderr — the SCM's refusal reason lives there, not in + the exit status (ALP-945). ``errors="replace"`` keeps a non-decodable byte + in that output from raising ``UnicodeDecodeError`` instead of the + ``CalledProcessError`` the caller logs. On success the captured output is + discarded (it no longer passes through to the watchdog's own stdio). + """ + subprocess.run(cmd, check=True, capture_output=True, text=True, errors="replace") class NssmServiceController: @@ -52,6 +60,13 @@ def restart(self) -> None: log.critical("watchdog: restarting service %r", self._service_name) try: self._run(list(cmd)) + except subprocess.CalledProcessError as exc: + # The SCM's refusal reason lives in the child's stderr (ALP-945). + log.exception( + "watchdog: nssm restart of %r failed (stderr: %s); will retry next tick", + self._service_name, + exc.stderr or "", + ) except Exception: log.exception( "watchdog: nssm restart of %r failed; will retry next tick", self._service_name diff --git a/tests/execution/process_supervision/test_process_control.py b/tests/execution/process_supervision/test_process_control.py index 0336eaed..44f65cfe 100644 --- a/tests/execution/process_supervision/test_process_control.py +++ b/tests/execution/process_supervision/test_process_control.py @@ -8,8 +8,15 @@ from __future__ import annotations +import logging +import subprocess +import sys + +import pytest + from alphamind.execution.process_supervision.process_control import ( NssmServiceController, + _checked_run, ) @@ -36,3 +43,52 @@ def _boom(cmd: list[str]) -> None: controller = NssmServiceController(service_name="alphamind-monitor", run=_boom) controller.restart() # must not raise + + +def test_nssm_controller_failure_log_carries_the_shell_outs_stderr( + caplog: pytest.LogCaptureFixture, +) -> None: + """A ``CalledProcessError``'s stderr reaches the failure log line. + + The 2026-06-10 incident (ALP-945): sixteen refused restarts logged only + "exit status 1" while the SCM's refusal reason sat in the child's stderr. + The log line must carry that reason — and the error is still swallowed so + the watchdog keeps probing. + """ + refusal = ( + "alphamind-monitor: STOP: A stop control has been sent to a service " + "that other running services are dependent on." + ) + + def _refused(cmd: list[str]) -> None: + raise subprocess.CalledProcessError(1, cmd, stderr=refusal) + + controller = NssmServiceController(service_name="alphamind-monitor", run=_refused) + + with caplog.at_level( + logging.ERROR, logger="alphamind.execution.process_supervision.process_control" + ): + controller.restart() # must not raise + + assert refusal in caplog.text + + +def test_checked_run_attaches_the_childs_stderr_to_the_raised_error() -> None: + """The default runner captures the child's stderr onto ``CalledProcessError``. + + Exercised against a real subprocess — the runner IS the OS-process + boundary, so there is nothing to fake below it. The child's stderr + includes a byte no text encoding need accept: a non-decodable byte must + degrade to a replacement character, not raise ``UnicodeDecodeError`` in + place of the ``CalledProcessError`` the watchdog logs. + """ + cmd = [ + sys.executable, + "-c", + "import sys; sys.stderr.buffer.write(b'stop control refused \\xff'); sys.exit(1)", + ] + + with pytest.raises(subprocess.CalledProcessError) as excinfo: + _checked_run(cmd) + + assert "stop control refused" in excinfo.value.stderr