From b5d78bbe26f762fd53bb42cefef19038b1b48737 Mon Sep 17 00:00:00 2001 From: Jackson Atassi Date: Wed, 10 Jun 2026 18:05:16 -0600 Subject: [PATCH 1/3] fix(execution): remove CC's SCM dependency edges; surface nssm stderr at the watchdog restart seam (ALP-945) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-06-10 incident: the monitor's event loop froze (ALP-946) and the out-of-process watchdog detected staleness and issued sixteen restarts over 38 minutes — every one refused by the Windows SCM, because AlphaMindCommandCenter declared a DependOnService edge on alphamind-monitor and the SCM refuses a stop control on a service with running dependents. The ~150 s wedge bound ALP-941 built was in practice unbounded. - install_command_center_service.ps1: delete both dependency edges and all supporting prose. The edge's rationale (fail-fast SSE consumers) was never true of the shipped code — the consumers reconnect forever with capped backoff (multiplexer.py) and the control proxy hits upstreams per-request. - process_control.py: _checked_run captures output so CalledProcessError carries the child's stderr; restart() logs that stderr distinctly (the SCM's refusal reason was invisible in the watchdog's own log — it showed only "exit status 1"). Swallow-and-retry behavior unchanged. - install_monitor_service.ps1 + install_safety_core_service.ps1: topology invariant comment — no SCM dependency may target a watchdog-supervised service. - RUNBOOK_production.md §1.4/§1.6/§7/§8.9 + RUNBOOK_command_center.md: no-dependency topology; stop/start orderings reframed as operational convention; §8.9 manual fallback is now a plain stop+start of the monitor with no CC choreography. Also fixed ALP-941 staleness in the CC runbook (seven services, monitor-watchdog in the recovery lists). Deploy note: the live SCM config requires the one-time operator step `sc.exe config AlphaMindCommandCenter depend= /` on the prod box (Scope E). Co-Authored-By: Claude Fable 5 --- scripts/RUNBOOK_command_center.md | 35 +++++++----- scripts/RUNBOOK_production.md | 48 ++++++++++------- scripts/install_command_center_service.ps1 | 33 +++++------- scripts/install_monitor_service.ps1 | 6 +++ scripts/install_safety_core_service.ps1 | 6 +++ .../process_supervision/process_control.py | 18 ++++++- .../test_process_control.py | 53 +++++++++++++++++++ 7 files changed, 142 insertions(+), 57 deletions(-) diff --git a/scripts/RUNBOOK_command_center.md b/scripts/RUNBOOK_command_center.md index f69d29d8f..69aa99ea5 100644 --- a/scripts/RUNBOOK_command_center.md +++ b/scripts/RUNBOOK_command_center.md @@ -56,19 +56,21 @@ per-step PASS / FAIL report. `install_pipeline_scheduler_service.ps1` / `install_monitor_service.ps1`. NSSM ships as a single .exe; drop into a directory on PATH. -6. **Pipeline + monitor services already installed.** The command center - declares both as service dependencies (see - `install_command_center_service.ps1`); a missing dependency causes the - Windows SCM to refuse the start. +6. **No service-ordering prerequisite.** The command center declares no SCM + service dependencies (ALP-945) — it installs and starts whether or not + the scheduler / monitor services exist yet. Its SSE consumers reconnect + with capped backoff until the upstreams come up, and the control proxy + reaches them per-request. -The full prod runtime is **six** NSSM services (see `RUNBOOK_production.md`'s +The full prod runtime is **seven** NSSM services (see `RUNBOOK_production.md`'s service table): `alphamind-collector`, `alphamind-scheduler`, -`alphamind-monitor`, `alphamind-safety-core`, `alphamind-safety-core-watchdog`, -and `AlphaMindCommandCenter`. The `alphamind-safety-core` service is the isolated -breach + price-staleness safety core (ADR-0004 / ALP-857); its dedicated -out-of-process watchdog `alphamind-safety-core-watchdog` restarts it on heartbeat -staleness. The command center does **not** depend on the safety-core services and -does not surface their state — they are monitored via their logs +`alphamind-monitor`, `alphamind-monitor-watchdog`, `alphamind-safety-core`, +`alphamind-safety-core-watchdog`, and `AlphaMindCommandCenter`. The +`alphamind-safety-core` service is the isolated breach + price-staleness safety +core (ADR-0004 / ALP-857); each supervised service's dedicated out-of-process +watchdog (`alphamind-monitor-watchdog` / `alphamind-safety-core-watchdog`, +ALP-941) restarts its target on heartbeat staleness. The command center does +not surface the safety-core services' state — they are monitored via their logs (`safety_core.*.log`) and the heartbeat file, per `RUNBOOK_production.md` §7/§9. ## Initial bring-up @@ -120,7 +122,8 @@ The install script wires: - `python -m alphamind.command_center` as the entry point. - AppExit Default Restart with a 60 s throttle. - Stdout / stderr → `%USERPROFILE%\AlphaMind\logs\command_center.{out,err}.log`. -- Dependency on `alphamind-scheduler` + `alphamind-monitor`. +- No SCM service dependencies — the SSE consumers reconnect with capped + backoff until the scheduler / monitor come up (ALP-945). The `ObjectName` step is manual — NSSM prompts for the account password interactively, which keeps it out of shell history. @@ -378,11 +381,13 @@ The command center's three owned tables (`alerts`, `webauthn_credentials`, pipeline + monitor's state. The canonical recovery is the same as for the pipeline / monitor: -1. Stop all six services (reverse-dependency order; stop watchdog before core): +1. Stop all seven services (reverse start order — operational convention, + not SCM-enforced; stop each watchdog before the service it supervises): ```powershell nssm stop AlphaMindCommandCenter nssm stop alphamind-safety-core-watchdog nssm stop alphamind-safety-core + nssm stop alphamind-monitor-watchdog nssm stop alphamind-monitor nssm stop alphamind-scheduler nssm stop alphamind-collector @@ -391,11 +396,13 @@ pipeline / monitor: `archive/` directory carries periodic snapshots; pick the most recent verified-good copy). 3. `uv run alembic upgrade head` to bring the restored DB's schema current. -4. Restart all six services in dependency order (core before watchdog): +4. Restart all seven services in the conventional start order (each + supervised service before its watchdog): ```powershell nssm start alphamind-collector nssm start alphamind-scheduler nssm start alphamind-monitor + nssm start alphamind-monitor-watchdog nssm start alphamind-safety-core nssm start alphamind-safety-core-watchdog nssm start AlphaMindCommandCenter diff --git a/scripts/RUNBOOK_production.md b/scripts/RUNBOOK_production.md index 53b0885d1..7f5ef67a3 100644 --- a/scripts/RUNBOOK_production.md +++ b/scripts/RUNBOOK_production.md @@ -117,13 +117,16 @@ uv run alembic heads # what the codebase expects If the two revisions match, **skip to § 1.6**. Nothing to migrate; just restart any services whose code changed. -### 1.4 Stop services in reverse dependency order +### 1.4 Stop services (reverse of the start order) -The command center depends on scheduler + monitor; stop it first so its -loopback consumers don't reconnect mid-migration. Collector is independent of -the trading DB schema but stop it too to keep the snapshot quiet during the -migration. Stop each **watchdog before the service it supervises** so the -watchdog does not `nssm restart` its target while you are taking it down. +The stop order is operational convention — no service declares an SCM +dependency on another (ALP-945), so the SCM enforces no ordering. Stop the +command center first so its loopback consumers sit quiet through the +migration window rather than reconnect-looping against stopped upstreams. +Collector is independent of the trading DB schema but stop it too to keep +the snapshot quiet during the migration. Stop each **watchdog before the +service it supervises** so the watchdog does not `nssm restart` its target +while you are taking it down. ```powershell nssm stop AlphaMindCommandCenter @@ -151,11 +154,14 @@ If the upgrade fails, **do not restart services.** Read the traceback, restore the DB from the most recent backup (see `RUNBOOK_command_center.md` § Recovery → DB corruption restore), then re-apply. -### 1.6 Restart services in dependency order +### 1.6 Restart services (conventional start order) -Start each supervised service **before its watchdog** so the watchdog finds a -fresh heartbeat on its first probe (and does not restart a service that is -still coming up). +The start order is operational convention, not SCM-enforced. Start each +supervised service **before its watchdog** so the watchdog finds a fresh +heartbeat on its first probe (and does not restart a service that is still +coming up). The command center comes last only by symmetry with § 1.4 — it +starts fine before its upstreams (its consumers reconnect with capped +backoff until they come up). ```powershell nssm start alphamind-collector @@ -1052,7 +1058,15 @@ per daemon; the scheduler supervisor uses a manually-bounded wait rather than in the window are abandoned and NSSM/the SCM force-terminates the process; there is no separate SIGTERM "fallback" the supervisor itself fires. -**Two restart gotchas that cost real diagnosis time (2026-06-02 monitor wedge):** +A plain `nssm restart ` works for any single service with all the +others running: no service declares an SCM dependency on another, so the SCM +never refuses the stop half of the restart. That is a deliberate topology +invariant (ALP-945, the 2026-06-10 incident — a `DependOnService` edge from +the command center made the SCM refuse the monitor watchdog's restarts for +42 minutes): **never declare an SCM dependency on a watchdog-supervised +service.** + +**One restart gotcha that cost real diagnosis time (2026-06-02 monitor wedge):** - **`Get-Service … Running` does NOT mean healthy.** NSSM reports `Running` while the *wrapper* process is alive. Two liveness layers now apply: @@ -1093,11 +1107,6 @@ is no separate SIGTERM "fallback" the supervisor itself fires. (Get-Process -Id $p).StartTime ``` See § 8.9 for the fill-stream and underlying-stream auto-recovery details. -- **Restarting `alphamind-scheduler` or `alphamind-monitor` while - `AlphaMindCommandCenter` is up can be silently refused** (the command center - depends on both). Stop CC first, restart the target, then start CC — stop - reverse / start forward: `nssm stop AlphaMindCommandCenter` → - `nssm restart ` → `nssm start AlphaMindCommandCenter`. **When the operator can restart a single service without coordinating the others:** @@ -1242,10 +1251,9 @@ Both the fill (trade-updates) stream and the underlying-price stream now have **Manual § 7 restart is the fallback** when auto-recovery is itself suspect (e.g. the StartTime is stale, no `8766` heartbeat is appearing, or `monitor.err.log` -shows a `reconnect budget exhausted` entry). Use the CC-dependency order: -`nssm stop AlphaMindCommandCenter` → `nssm stop` + `nssm start alphamind-monitor` -→ `nssm start AlphaMindCommandCenter`. Confirm by StartTime + heartbeat, not -`Get-Service`. +shows a `reconnect budget exhausted` entry): `nssm stop alphamind-monitor` + +`nssm start alphamind-monitor` — no other service needs touching. Confirm by +StartTime + heartbeat, not `Get-Service`. #### Pre-fix history (2026-06-02 — for reference only) diff --git a/scripts/install_command_center_service.ps1 b/scripts/install_command_center_service.ps1 index f5769b9c9..4a1365884 100644 --- a/scripts/install_command_center_service.ps1 +++ b/scripts/install_command_center_service.ps1 @@ -10,11 +10,11 @@ - Automatic (Delayed Start) start type - Stdout / stderr redirected to %USERPROFILE%\AlphaMind\logs\ - AppDirectory set to the project root - - DependsOnService = AlphaMindPipeline + AlphaMindMonitor so the - command center starts only after the pipeline + monitor processes - have come up (the SSE consumer tasks fail-fast when their - upstreams are unreachable, so a too-eager start would cycle - through restarts; the dependency makes the boot ordering explicit). + - No SCM service-ordering edges: the command center starts on its + own, before or after its upstreams. Its SSE consumers reconnect + with capped backoff until the scheduler / monitor come up + (command_center/events/multiplexer.py), and the control proxy + reaches upstreams per-request (ALP-945). The ObjectName (service account) configuration is left as a manual final step documented in RUNBOOK_command_center.md to avoid @@ -33,9 +33,6 @@ src/alphamind/command_center/frontend/dist/ (run "bun install --frozen-lockfile && bun run build" before this script — see RUNBOOK_command_center.md § Initial bring-up). - - The pipeline + monitor services must already be installed (this - script declares them as dependencies; absent services cause - Windows to refuse to start the command center). - Tested on Windows 10/11 with PowerShell 5.1+ #> @@ -48,13 +45,6 @@ $ErrorActionPreference = 'Stop' $ServiceName = 'AlphaMindCommandCenter' -# The two services this one depends on. Their install scripts: -# - install_pipeline_scheduler_service.ps1 -> alphamind-scheduler -# - install_monitor_service.ps1 -> alphamind-monitor -# The dependency names below match those service names verbatim. -$PipelineDependency = 'alphamind-scheduler' -$MonitorDependency = 'alphamind-monitor' - # Resolve the project root as the directory containing this script's parent. # Script lives in \scripts\; project root is one level up. $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path @@ -114,7 +104,6 @@ Write-Host " Python: $PythonExe" Write-Host " ProjectRoot: $ProjectRoot" Write-Host " Stdout log: $StdoutLog" Write-Host " Stderr log: $StderrLog" -Write-Host " Depends on: $PipelineDependency, $MonitorDependency" Write-Host "" # Install: nssm install @@ -141,11 +130,13 @@ nssm set $ServiceName AppStderr $StderrLog nssm set $ServiceName AppStdoutCreationDisposition 4 nssm set $ServiceName AppStderrCreationDisposition 4 -# Declare service dependencies — Windows will only start -# AlphaMindCommandCenter after alphamind-scheduler + alphamind-monitor -# have entered the Running state. The SCM honors the dependency on -# both Start and Stop (cmd center stops first on a system shutdown). -nssm set $ServiceName DependOnService $PipelineDependency $MonitorDependency +# Deliberately NO SCM service-ordering edge on the scheduler / monitor: +# the command center starts on its own — its SSE consumers reconnect with +# capped backoff until the upstreams come up (events/multiplexer.py), and +# the control proxy reaches them per-request. An SCM edge pointing at a +# watchdog-supervised service would also make the SCM refuse the watchdog's +# `nssm restart` (stop + start) while this service runs — the 2026-06-10 +# 42-minute monitor wedge (ALP-945). # --------------------------------------------------------------------------- # Summary and next steps diff --git a/scripts/install_monitor_service.ps1 b/scripts/install_monitor_service.ps1 index c23fc7a63..8b4214b51 100644 --- a/scripts/install_monitor_service.ps1 +++ b/scripts/install_monitor_service.ps1 @@ -59,6 +59,12 @@ $ErrorActionPreference = 'Stop' $MonitorServiceName = 'alphamind-monitor' $WatchdogServiceName = 'alphamind-monitor-watchdog' +# Topology invariant (ALP-945, the 2026-06-10 incident): no service may +# declare an SCM dependency (DependOnService) on alphamind-monitor. The +# watchdog's `nssm restart` is stop + start, and the SCM refuses the stop +# while a dependent service runs — a wedged monitor would then be +# unrestartable for as long as the dependent stays up. + # Resolve the project root as the directory containing this script's parent. # Script lives in \scripts\; project root is one level up. $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path diff --git a/scripts/install_safety_core_service.ps1 b/scripts/install_safety_core_service.ps1 index 1d1a70e3f..04ba217c5 100644 --- a/scripts/install_safety_core_service.ps1 +++ b/scripts/install_safety_core_service.ps1 @@ -55,6 +55,12 @@ $ErrorActionPreference = 'Stop' $CoreServiceName = 'alphamind-safety-core' $WatchdogServiceName = 'alphamind-safety-core-watchdog' +# Topology invariant (ALP-945, the 2026-06-10 incident): no service may +# declare an SCM dependency (DependOnService) on alphamind-safety-core. The +# watchdog's `nssm restart` is stop + start, and the SCM refuses the stop +# while a dependent service runs — a wedged safety core would then be +# unrestartable for as long as the dependent stays up. + # Resolve the project root as the directory containing this script's parent. # Script lives in \scripts\; project root is one level up. $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path diff --git a/src/alphamind/execution/process_supervision/process_control.py b/src/alphamind/execution/process_supervision/process_control.py index ea98b19f2..b488c7c2d 100644 --- a/src/alphamind/execution/process_supervision/process_control.py +++ b/src/alphamind/execution/process_supervision/process_control.py @@ -24,8 +24,13 @@ def _checked_run(cmd: list[str]) -> None: - """Default runner: ``subprocess.run`` with ``check=True`` (raises on failure).""" - subprocess.run(cmd, check=True) + """Default runner: ``subprocess.run`` with ``check=True`` (raises on failure). + + Output is captured so the raised :class:`subprocess.CalledProcessError` + carries the child's stderr — the SCM's refusal reason lives there, not in + the exit status (ALP-945). + """ + subprocess.run(cmd, check=True, capture_output=True, text=True) class NssmServiceController: @@ -52,6 +57,15 @@ def restart(self) -> None: log.critical("watchdog: restarting service %r", self._service_name) try: self._run(list(cmd)) + except subprocess.CalledProcessError as exc: + # Surface the child's stderr: the exit status alone hides the SCM's + # refusal reason (ALP-945 — sixteen restarts logged only "exit + # status 1" while the reason sat in nssm's stderr). + log.exception( + "watchdog: nssm restart of %r failed (stderr: %s); will retry next tick", + self._service_name, + exc.stderr, + ) except Exception: log.exception( "watchdog: nssm restart of %r failed; will retry next tick", self._service_name diff --git a/tests/execution/process_supervision/test_process_control.py b/tests/execution/process_supervision/test_process_control.py index 0336eaed1..47b39ba31 100644 --- a/tests/execution/process_supervision/test_process_control.py +++ b/tests/execution/process_supervision/test_process_control.py @@ -8,8 +8,15 @@ from __future__ import annotations +import logging +import subprocess +import sys + +import pytest + from alphamind.execution.process_supervision.process_control import ( NssmServiceController, + _checked_run, ) @@ -36,3 +43,49 @@ def _boom(cmd: list[str]) -> None: controller = NssmServiceController(service_name="alphamind-monitor", run=_boom) controller.restart() # must not raise + + +def test_nssm_controller_failure_log_carries_the_shell_outs_stderr( + caplog: pytest.LogCaptureFixture, +) -> None: + """A ``CalledProcessError``'s stderr reaches the failure log line. + + The 2026-06-10 incident (ALP-945): sixteen refused restarts logged only + "exit status 1" while the SCM's refusal reason sat in the child's stderr. + The log line must carry that reason — and the error is still swallowed so + the watchdog keeps probing. + """ + refusal = ( + "alphamind-monitor: STOP: A stop control has been sent to a service " + "that other running services are dependent on." + ) + + def _refused(cmd: list[str]) -> None: + raise subprocess.CalledProcessError(1, cmd, stderr=refusal) + + controller = NssmServiceController(service_name="alphamind-monitor", run=_refused) + + with caplog.at_level( + logging.ERROR, logger="alphamind.execution.process_supervision.process_control" + ): + controller.restart() # must not raise + + assert refusal in caplog.text + + +def test_checked_run_attaches_the_childs_stderr_to_the_raised_error() -> None: + """The default runner captures the child's stderr onto ``CalledProcessError``. + + Exercised against a real subprocess — the runner IS the OS-process + boundary, so there is nothing to fake below it. + """ + cmd = [ + sys.executable, + "-c", + "import sys; sys.stderr.write('stop control refused'); sys.exit(1)", + ] + + with pytest.raises(subprocess.CalledProcessError) as excinfo: + _checked_run(cmd) + + assert "stop control refused" in excinfo.value.stderr From f201a6b4485af905427af35cf60a8916c5a3b56c Mon Sep 17 00:00:00 2001 From: Jackson Atassi Date: Wed, 10 Jun 2026 18:12:55 -0600 Subject: [PATCH 2/3] fix: address code-review findings on ALP-945 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - _checked_run decodes with errors="replace" so a non-decodable byte in nssm's output degrades to a replacement character instead of raising UnicodeDecodeError in place of the CalledProcessError the watchdog logs (confirmed by repro: strict text=True decode raises before check=True). The real-subprocess test now writes an invalid byte to stderr to guard it. - restart() logs "" when CalledProcessError.stderr is None (injected runners that don't capture), instead of a bare "None". - Trimmed the stderr rationale to one canonical site (_checked_run docstring). - De-duplicated the no-dependency prose (RUNBOOK_production §1.4 now points at §7's canonical invariant; CC runbook wiring bullet points at prereq 6). Co-Authored-By: Claude Fable 5 --- scripts/RUNBOOK_command_center.md | 3 +-- scripts/RUNBOOK_production.md | 3 ++- .../execution/process_supervision/process_control.py | 12 ++++++------ .../process_supervision/test_process_control.py | 7 +++++-- 4 files changed, 14 insertions(+), 11 deletions(-) diff --git a/scripts/RUNBOOK_command_center.md b/scripts/RUNBOOK_command_center.md index 69aa99ea5..9ef359970 100644 --- a/scripts/RUNBOOK_command_center.md +++ b/scripts/RUNBOOK_command_center.md @@ -122,8 +122,7 @@ The install script wires: - `python -m alphamind.command_center` as the entry point. - AppExit Default Restart with a 60 s throttle. - Stdout / stderr → `%USERPROFILE%\AlphaMind\logs\command_center.{out,err}.log`. -- No SCM service dependencies — the SSE consumers reconnect with capped - backoff until the scheduler / monitor come up (ALP-945). +- No SCM service dependencies (ALP-945; see prerequisite 6). The `ObjectName` step is manual — NSSM prompts for the account password interactively, which keeps it out of shell history. diff --git a/scripts/RUNBOOK_production.md b/scripts/RUNBOOK_production.md index 7f5ef67a3..12a4eabc8 100644 --- a/scripts/RUNBOOK_production.md +++ b/scripts/RUNBOOK_production.md @@ -120,7 +120,8 @@ restart any services whose code changed. ### 1.4 Stop services (reverse of the start order) The stop order is operational convention — no service declares an SCM -dependency on another (ALP-945), so the SCM enforces no ordering. Stop the +dependency on another, so the SCM enforces no ordering (the topology +invariant; § 7). Stop the command center first so its loopback consumers sit quiet through the migration window rather than reconnect-looping against stopped upstreams. Collector is independent of the trading DB schema but stop it too to keep diff --git a/src/alphamind/execution/process_supervision/process_control.py b/src/alphamind/execution/process_supervision/process_control.py index b488c7c2d..0fc332760 100644 --- a/src/alphamind/execution/process_supervision/process_control.py +++ b/src/alphamind/execution/process_supervision/process_control.py @@ -28,9 +28,11 @@ def _checked_run(cmd: list[str]) -> None: Output is captured so the raised :class:`subprocess.CalledProcessError` carries the child's stderr — the SCM's refusal reason lives there, not in - the exit status (ALP-945). + the exit status (ALP-945). ``errors="replace"`` keeps a non-decodable byte + in that output from raising ``UnicodeDecodeError`` instead of the + ``CalledProcessError`` the caller logs. """ - subprocess.run(cmd, check=True, capture_output=True, text=True) + subprocess.run(cmd, check=True, capture_output=True, text=True, errors="replace") class NssmServiceController: @@ -58,13 +60,11 @@ def restart(self) -> None: try: self._run(list(cmd)) except subprocess.CalledProcessError as exc: - # Surface the child's stderr: the exit status alone hides the SCM's - # refusal reason (ALP-945 — sixteen restarts logged only "exit - # status 1" while the reason sat in nssm's stderr). + # The SCM's refusal reason lives in the child's stderr (ALP-945). log.exception( "watchdog: nssm restart of %r failed (stderr: %s); will retry next tick", self._service_name, - exc.stderr, + exc.stderr or "", ) except Exception: log.exception( diff --git a/tests/execution/process_supervision/test_process_control.py b/tests/execution/process_supervision/test_process_control.py index 47b39ba31..44f65cfe6 100644 --- a/tests/execution/process_supervision/test_process_control.py +++ b/tests/execution/process_supervision/test_process_control.py @@ -77,12 +77,15 @@ def test_checked_run_attaches_the_childs_stderr_to_the_raised_error() -> None: """The default runner captures the child's stderr onto ``CalledProcessError``. Exercised against a real subprocess — the runner IS the OS-process - boundary, so there is nothing to fake below it. + boundary, so there is nothing to fake below it. The child's stderr + includes a byte no text encoding need accept: a non-decodable byte must + degrade to a replacement character, not raise ``UnicodeDecodeError`` in + place of the ``CalledProcessError`` the watchdog logs. """ cmd = [ sys.executable, "-c", - "import sys; sys.stderr.write('stop control refused'); sys.exit(1)", + "import sys; sys.stderr.buffer.write(b'stop control refused \\xff'); sys.exit(1)", ] with pytest.raises(subprocess.CalledProcessError) as excinfo: From 021c9a1f90c4fdb868304596bb61af47b5ab9a12 Mon Sep 17 00:00:00 2001 From: Jackson Atassi Date: Wed, 10 Jun 2026 18:15:34 -0600 Subject: [PATCH 3/3] fix: address /review findings on ALP-945 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - RUNBOOK §7: the 42-minute figure is the wedge duration, not the refusal window — reworded so the shipped docs agree. - _checked_run docstring notes the success-path output is discarded. Co-Authored-By: Claude Fable 5 --- scripts/RUNBOOK_production.md | 6 +++--- .../execution/process_supervision/process_control.py | 3 ++- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/scripts/RUNBOOK_production.md b/scripts/RUNBOOK_production.md index 12a4eabc8..6a2ce294a 100644 --- a/scripts/RUNBOOK_production.md +++ b/scripts/RUNBOOK_production.md @@ -1063,9 +1063,9 @@ A plain `nssm restart ` works for any single service with all the others running: no service declares an SCM dependency on another, so the SCM never refuses the stop half of the restart. That is a deliberate topology invariant (ALP-945, the 2026-06-10 incident — a `DependOnService` edge from -the command center made the SCM refuse the monitor watchdog's restarts for -42 minutes): **never declare an SCM dependency on a watchdog-supervised -service.** +the command center made the SCM refuse every monitor-watchdog restart, +wedging the monitor for 42 minutes): **never declare an SCM dependency on a +watchdog-supervised service.** **One restart gotcha that cost real diagnosis time (2026-06-02 monitor wedge):** diff --git a/src/alphamind/execution/process_supervision/process_control.py b/src/alphamind/execution/process_supervision/process_control.py index 0fc332760..84cc29d48 100644 --- a/src/alphamind/execution/process_supervision/process_control.py +++ b/src/alphamind/execution/process_supervision/process_control.py @@ -30,7 +30,8 @@ def _checked_run(cmd: list[str]) -> None: carries the child's stderr — the SCM's refusal reason lives there, not in the exit status (ALP-945). ``errors="replace"`` keeps a non-decodable byte in that output from raising ``UnicodeDecodeError`` instead of the - ``CalledProcessError`` the caller logs. + ``CalledProcessError`` the caller logs. On success the captured output is + discarded (it no longer passes through to the watchdog's own stdio). """ subprocess.run(cmd, check=True, capture_output=True, text=True, errors="replace")