From 42b1cc6ffc3ca8965af3175fe26253c389467e13 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Keilyn=20Rodr=C3=ADguez=20Perojo?= <73491750+keilynrp@users.noreply.github.com> Date: Tue, 29 Sep 2026 16:35:57 -0600 Subject: [PATCH] docs(ops): paging woke the phone; close #377 and publish sev1-paging MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both paths passed on the operator's phone on 2026-09-29: - The Pushover channel's test (21:36 UTC) rang through Do Not Disturb, repeated, and was acknowledged. - With the monitor switched off (UKIP_OPS_MONITOR_ENABLED=0; the app kept serving), Healthchecks.io paged after 15 minutes, the 5-minute period plus 10-minute grace, again through Do Not Disturb. Re-enabling the monitor brought the check back up within one cycle. The runbook suggested stopping the container; switching the monitor off sends Healthchecks.io the same silence without taking the API down. Recorded in the tabletop's corrective action status, whose other rows were stale too: #375, #376 and #378 closed on 2026-09-24, with the note that addresses name the client only from 2026-09-29 (#413). Plan §11 gap 2 is closed and §4 says paging is live; what remains is one person receiving every page (gap 1) and no paging on access (gap 6). The privacy controls overview and the Mexico annex no longer list paging as pending. sev1-paging is archived with all tasks done, and its five requirements are published as openspec/specs/sev1-paging. Closes #377. Co-Authored-By: Claude Opus 5.5 --- docs/legal/MEXICO_ANNEX.md | 6 +- docs/legal/PRIVACY_CONTROLS_OVERVIEW.md | 2 +- docs/operating/INCIDENT_RESPONSE_PLAN.md | 23 +++--- .../operating/INCIDENT_TABLETOP_2026-09-22.md | 8 +- .../2026-09-29-sev1-paging}/.openspec.yaml | 0 .../2026-09-29-sev1-paging}/design.md | 0 .../2026-09-29-sev1-paging}/proposal.md | 0 .../specs/sev1-paging/spec.md | 0 .../2026-09-29-sev1-paging}/tasks.md | 12 ++- openspec/specs/sev1-paging/spec.md | 82 +++++++++++++++++++ 10 files changed, 112 insertions(+), 21 deletions(-) rename openspec/changes/{sev1-paging => archive/2026-09-29-sev1-paging}/.openspec.yaml (100%) rename openspec/changes/{sev1-paging => archive/2026-09-29-sev1-paging}/design.md (100%) rename openspec/changes/{sev1-paging => archive/2026-09-29-sev1-paging}/proposal.md (100%) rename openspec/changes/{sev1-paging => archive/2026-09-29-sev1-paging}/specs/sev1-paging/spec.md (100%) rename openspec/changes/{sev1-paging => archive/2026-09-29-sev1-paging}/tasks.md (80%) create mode 100644 openspec/specs/sev1-paging/spec.md diff --git a/docs/legal/MEXICO_ANNEX.md b/docs/legal/MEXICO_ANNEX.md index 955f4b0b..5fd0a5f4 100644 --- a/docs/legal/MEXICO_ANNEX.md +++ b/docs/legal/MEXICO_ANNEX.md @@ -68,9 +68,9 @@ and maintain administrative, technical, and physical security measures. The implemented measures and their evidence are documented in [PRIVACY_CONTROLS_OVERVIEW.md](PRIVACY_CONTROLS_OVERVIEW.md), including an honest register of measures not yet in place. Of the corrective actions from -the first incident-response exercise of 2026-09-22, read auditing (#375) and -attributable request logs (#376) are in place; a paging path is implemented -(#377) and awaits its installation and test on the operator's phone. Still +the first incident-response exercise of 2026-09-22, read auditing (#375), +attributable request logs (#376) and a paging path (#377, tested on the +operator's phone on 2026-09-29) are in place. Still open: retention of request logs beyond the container, an external pentest, data residency, and a passing restore drill. diff --git a/docs/legal/PRIVACY_CONTROLS_OVERVIEW.md b/docs/legal/PRIVACY_CONTROLS_OVERVIEW.md index 5c0373d2..4898a0ba 100644 --- a/docs/legal/PRIVACY_CONTROLS_OVERVIEW.md +++ b/docs/legal/PRIVACY_CONTROLS_OVERVIEW.md @@ -31,7 +31,7 @@ prefer an honest register over an impressive one. | Open item | Tracking ID | Status | |-----------|-------------|--------| -| Incident response capability (incl. breach-notification SLA backing) | ER-IR-001 | Partial — the plan is published, its authority and 72-hour notification commitment were decided on 2026-09-22, and it was first exercised on 2026-09-22. Of that exercise's corrective actions, read auditing and attributable request logs are in place (#375, #376); paging is implemented but not yet installed and tested (#377); request logs are still not retained beyond the container (plan §11 gap 3, `central-log-retention`) | +| Incident response capability (incl. breach-notification SLA backing) | ER-IR-001 | Partial — the plan is published, its authority and 72-hour notification commitment were decided on 2026-09-22, and it was first exercised on 2026-09-22. Of that exercise's corrective actions, read auditing, attributable request logs and a paging path are in place and tested (#375, #376, #377); request logs are still not retained beyond the container (plan §11 gap 3, `central-log-retention`) | | External penetration test | ER-ASSURE-001 | Open — no third-party assessment report available yet | | Data residency commitments | ER-DEP-001 | Open — residency follows hosting region; no contractual commitment defined | | A passing isolated restore drill | US-073 / ER-BCP-001 | Pending — provider configured and two backup cycles evidenced; the first isolated restore drill (2026-09-21) is recorded as failed and its dossier was approved on 2026-09-21; a passing drill remains | diff --git a/docs/operating/INCIDENT_RESPONSE_PLAN.md b/docs/operating/INCIDENT_RESPONSE_PLAN.md index a719bad3..462a5702 100644 --- a/docs/operating/INCIDENT_RESPONSE_PLAN.md +++ b/docs/operating/INCIDENT_RESPONSE_PLAN.md @@ -94,10 +94,9 @@ What actually reaches a human today: | Dokploy | Deploy failures, container restarts | Its own UI | | Customer report | Everything nobody instrumented | Email | -**Known detection gaps.** Paging exists in code (#377) but pages nobody until -the Pushover channel and the Healthchecks.io check are set up and each has been -tested on the phone ([PAGING_RUNBOOK.md](PAGING_RUNBOOK.md)); until then a SEV1 at 03:00 still waits -for someone to look at Slack. Only two checks page, and nothing detects +**Known detection gaps.** Paging is live since 2026-09-29, and both paths have +woken the phone in a test ([PAGING_RUNBOOK.md](PAGING_RUNBOOK.md)). One person +receives every page (gap 1). Only two checks page, and nothing detects unauthorized access on its own (gap 6). No central log retention: the backend's container log is ephemeral and disappears with the container, which is why capturing it is the first evidence step below; Traefik's access log survives @@ -338,14 +337,14 @@ Recorded here so nobody discovers them mid-incident: Accepted by the owner on 2026-09-22 as what the current team can commit to, and revisited when funding and staffing allow a second responder. It is also why the notification commitment is 72 hours rather than 24. -2. **No paging** (#377). Alerts reach Slack and the container log; nothing - wakes anyone up. Measured in the 2026-09-22 tabletop: **6 h 22 min** from - exposure to detection, overnight, and the report came from a third party. - **In code since 2026-09-26** (§4): Pushover emergency pages for `database` - and `migrations` critical, and a Healthchecks.io dead man's switch for the - monitor itself. It stays open until both are installed and have woken the - phone in a test. What remains after that is narrower: one person receives - every page (gap 1), and only availability pages, not access (gap 6). +2. ~~**No paging** (#377).~~ **Closed 2026-09-29.** The 2026-09-22 tabletop + measured **6 h 22 min** from exposure to detection, overnight, reported by a + third party. Now `database` and `migrations` critical page through Pushover + at emergency priority, and a Healthchecks.io dead man's switch pages when + the monitor itself goes silent (§4). Both woke the phone through Do Not + Disturb in a test (INCIDENT_TABLETOP_2026-09-22.md, corrective action + status). What remains is narrower: one person receives every page (gap 1), + and only availability pages, not access (gap 6). 3. **No central log retention.** The backend's container log is ephemeral, so early capture is the only way to keep it, and for ordinary reads outside the audited classes (gap 9) it is the only record that the read happened at all. diff --git a/docs/operating/INCIDENT_TABLETOP_2026-09-22.md b/docs/operating/INCIDENT_TABLETOP_2026-09-22.md index 0503956d..fc758323 100644 --- a/docs/operating/INCIDENT_TABLETOP_2026-09-22.md +++ b/docs/operating/INCIDENT_TABLETOP_2026-09-22.md @@ -100,10 +100,10 @@ Added after the exercise; the record above is left as it was written. | Gap | Status | |---|---| | Per-session revocation (#368 phase C.3) | **Closed 2026-09-23.** Tokens carry a `sid` naming their session, every request resolves the user through it, and a session can be revoked alone — including the operator's own, which deactivation refuses. Plan §5.3 | -| Reads are not audited (#375) | Open | -| Request log has no identity (#376) | Open | -| Nothing pages (#377) | Open | -| No `ip_address` filter on the audit log (#378) | Open | +| Reads are not audited (#375) | **Closed 2026-09-24** (#395). Exports, audit-log reads, API-key reads and bulk reads are audited with their session or key. Ordinary first-page reads are not, by design (plan §7) | +| Request log has no identity (#376) | **Closed 2026-09-24** (#387). Each authenticated line names the user and the session or API key. Its `client_ip` names the client only since 2026-09-29 (#413) | +| Nothing pages (#377) | **Closed 2026-09-29.** Both paths woke the operator's phone on 2026-09-29: the Pushover channel's test at 21:36 UTC rang through Do Not Disturb, repeated and was acknowledged; with the monitor switched off (`UKIP_OPS_MONITOR_ENABLED=0`, the app kept serving), Healthchecks.io paged after 15 minutes (5-minute period plus 10-minute grace), again through Do Not Disturb (reported 22:29 UTC). Re-enabling the monitor brought the check back up within one cycle. | +| No `ip_address` filter on the audit log (#378) | **Closed 2026-09-24** (#384). The addresses it filters on are the client's only from 2026-09-29 03:57 UTC (#413); earlier rows carry a proxy's | | Blast radius undeterminable (ER-BCP-001 residual risk 1) | Open | ## What was verified to work diff --git a/openspec/changes/sev1-paging/.openspec.yaml b/openspec/changes/archive/2026-09-29-sev1-paging/.openspec.yaml similarity index 100% rename from openspec/changes/sev1-paging/.openspec.yaml rename to openspec/changes/archive/2026-09-29-sev1-paging/.openspec.yaml diff --git a/openspec/changes/sev1-paging/design.md b/openspec/changes/archive/2026-09-29-sev1-paging/design.md similarity index 100% rename from openspec/changes/sev1-paging/design.md rename to openspec/changes/archive/2026-09-29-sev1-paging/design.md diff --git a/openspec/changes/sev1-paging/proposal.md b/openspec/changes/archive/2026-09-29-sev1-paging/proposal.md similarity index 100% rename from openspec/changes/sev1-paging/proposal.md rename to openspec/changes/archive/2026-09-29-sev1-paging/proposal.md diff --git a/openspec/changes/sev1-paging/specs/sev1-paging/spec.md b/openspec/changes/archive/2026-09-29-sev1-paging/specs/sev1-paging/spec.md similarity index 100% rename from openspec/changes/sev1-paging/specs/sev1-paging/spec.md rename to openspec/changes/archive/2026-09-29-sev1-paging/specs/sev1-paging/spec.md diff --git a/openspec/changes/sev1-paging/tasks.md b/openspec/changes/archive/2026-09-29-sev1-paging/tasks.md similarity index 80% rename from openspec/changes/sev1-paging/tasks.md rename to openspec/changes/archive/2026-09-29-sev1-paging/tasks.md index a351382e..d833685c 100644 --- a/openspec/changes/sev1-paging/tasks.md +++ b/openspec/changes/archive/2026-09-29-sev1-paging/tasks.md @@ -59,6 +59,16 @@ TDD throughout. Sections 1–3 are backend and can ship as one PR; 4 is the UI; to `ops.check_failed`, create the Healthchecks.io check (period = monitor interval, grace = 2 intervals, Pushover integration at emergency), set `UKIP_OPS_HEARTBEAT_URL`. -- [ ] 5.3 **Operator:** test the channel with the phone in Do Not Disturb and +- [x] 5.3 **Operator:** test the channel with the phone in Do Not Disturb and confirm it rings; stop the backend container and confirm Healthchecks.io pages; record both in the tabletop follow-up. Then close #377. + Both paths woke the operator's phone on 2026-09-29: the Pushover + channel's test at 21:36 UTC rang through Do Not Disturb, repeated + and was acknowledged; with the monitor switched off + (`UKIP_OPS_MONITOR_ENABLED=0`, the app kept serving), + Healthchecks.io paged after 15 minutes (5-minute period plus + 10-minute grace), again through Do Not Disturb (reported 22:29 UTC). + Re-enabling the monitor brought the check back up within one cycle. + The heartbeat test switched the monitor off instead of stopping the + container: the same silence reaches Healthchecks.io, and the app stayed + up. Recorded in `INCIDENT_TABLETOP_2026-09-22.md`. diff --git a/openspec/specs/sev1-paging/spec.md b/openspec/specs/sev1-paging/spec.md new file mode 100644 index 00000000..79442690 --- /dev/null +++ b/openspec/specs/sev1-paging/spec.md @@ -0,0 +1,82 @@ +# sev1-paging Specification + +## Purpose +A SEV1 wakes a person: ops alerts carry a severity derived from which checks are critical, Pushover pages at emergency priority through Do Not Disturb once per incident, and a Healthchecks.io dead man's switch pages when the monitor itself goes silent. Created by archiving change sev1-paging. +## Requirements +### Requirement: Pushover is an alert channel type + +The system SHALL accept `pushover` as an alert channel type, store its +application token and user key encrypted, never return them, and deliver alerts +to Pushover's messages API without raising on failure. + +#### Scenario: Credentials are validated and never returned + +- **WHEN** an admin creates a `pushover` channel without a user key +- **THEN** the request is rejected with 422 +- **AND WHEN** the channel is created with both +- **THEN** no response ever contains the token or the user key + +### Requirement: Ops alerts carry a severity from the critical checks + +The system SHALL classify each ops alert as `page` when `database` or +`migrations` is critical, `urgent` when `backup_freshness` or `secrets` is +critical (and no paging check is), and `info` otherwise, and SHALL map them to +Pushover priorities 2, 1 and 0. + +#### Scenario: The database going down pages + +- **WHEN** the database check becomes critical +- **THEN** a Pushover channel receives a priority-2 message with `retry` and + `expire` set + +#### Scenario: A stale backup does not wake anyone + +- **WHEN** only `backup_freshness` is critical +- **THEN** the Pushover message has priority 1 + +#### Scenario: Slack sees the same classification + +- **WHEN** an alert is `page` +- **THEN** its Slack message starts with `[PAGE]` + +### Requirement: A page is sent once per incident + +The system SHALL send severity `page` only when the set of critical paging +checks grows relative to the previous observation, and SHALL send reminders and +unrelated changes during that incident as `info`. + +#### Scenario: A reminder during an outage does not page again + +- **WHEN** the database has been critical for longer than the reminder interval +- **THEN** the reminder is sent at priority 0 + +#### Scenario: A second paging check failing pages again + +- **WHEN** migrations become critical while the database already was +- **THEN** a new priority-2 page is sent + +### Requirement: The monitor reports that it is alive + +The system SHALL, when `UKIP_OPS_HEARTBEAT_URL` is set, ping it after every +monitor cycle, ping `/fail` when the evaluation raised, never raise, and +never log the URL. + +#### Scenario: A failed evaluation reports failure + +- **WHEN** the checks raise during a cycle +- **THEN** the heartbeat request goes to the `/fail` path + +#### Scenario: An unreachable heartbeat service does not break the cycle + +- **WHEN** the heartbeat request times out +- **THEN** the cycle completes and the next one runs + +### Requirement: The paging path is testable end to end + +The system SHALL, for a Pushover channel's test action, send a real +emergency-priority message that expires on its own within a minute. + +#### Scenario: Testing a Pushover channel + +- **WHEN** an admin tests a Pushover channel +- **THEN** a priority-2 message with `expire` of 60 seconds is sent