Refresh internal KMS trust and deliver durable monitor alerts

deploy/crypto-agility-alerts@.service

@@ -0,0 +1,25 @@
+# Configure EnvironmentFile in a host-specific drop-in to read the existing mail
+# runtime environment in place. Never place its values in this unit or a copy.
+[Unit]
+Description=Crypto-agility alert delivery (%i)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+User=%i
+Group=%i
+WorkingDirectory=/opt/crypto-agility
+ExecStart=/opt/crypto-agility/.venv/bin/python -m tools.alert_delivery --config /etc/crypto-agility/%i/delivery.json --state-directory /var/lib/crypto-agility-consumer/%i
+TimeoutStartSec=120
+UMask=0077
+NoNewPrivileges=true
+PrivateTmp=true
+PrivateDevices=true
+ProtectSystem=strict
+ProtectHome=true
+RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
+ReadWritePaths=/var/lib/crypto-agility-consumer/%i
+ReadOnlyPaths=/etc/crypto-agility/%i
+StandardOutput=journal
+StandardError=journal

deploy/crypto-agility-alerts@.timer

@@ -0,0 +1,10 @@
+[Unit]
+Description=Retry pending crypto-agility alert delivery (%i)
+
+[Timer]
+OnBootSec=30
+OnUnitInactiveSec=60
+Unit=crypto-agility-alerts@%i.service
+
+[Install]
+WantedBy=timers.target

docs/consumer-deployment.md

@@ -54,8 +54,46 @@ The state directory contains `initialized.json`, `replay.json`, `state.json`, `c
 
 The long-running process retains the existing bounded in-memory authenticated-input cache and reevaluates expiration every cycle. After restart, a previously consumed token is rejected as replay; provision a new fresh observation and JTI. An unchanged token cannot silently become current assurance again. Historical trusted measurements and pending alerts survive, allowing later valid classical evidence to reveal a hybrid regression even after rejected inputs.
 
-Reports go to stdout/journald; they are not alert-delivery receipts. The durable `state.json` outbox retains each occurrence's `deliveryId` until a separately configured delivery component confirms success. After confirmed delivery, acknowledge only that exact occurrence with `python -m tools.monitor --state /absolute/state/state.json --ack-alert <deliveryId>`. Never acknowledge merely because a report was printed or read. No delivery adapter or destination is selected here. Input failures continue producing reports and durable pending alerts; failures that prevent persistence explicitly include `deliveryPersistenceFailure`. State-directory startup failures or stdout errors stop the CLI so the supervisor can restart it. The CLI does not exit on an assessment finding, avoiding replay-producing restart loops.
+Reports go to stdout/journald; they are not alert-delivery receipts. The durable `state.json` outbox retains each occurrence's `deliveryId` until a separately configured delivery component confirms success. After confirmed delivery, acknowledge only that exact occurrence with `python -m tools.monitor --state /absolute/state/state.json --ack-alert <deliveryId>`. Never acknowledge merely because a report was printed or read. The optional Resend adapter below handles the approved destination and provider receipt before acknowledgment. Input failures continue producing reports and durable pending alerts; failures that prevent persistence explicitly include `deliveryPersistenceFailure`. State-directory startup failures or stdout errors stop the CLI so the supervisor can restart it. The CLI does not exit on an assessment finding, avoiding replay-producing restart loops.
 
 `deploy/[email protected]` is an installation template, not discovered production coordinates. Review its paths and existing service account, install a pinned checkout and virtual environment at the chosen location, provision approved configuration/bundles, and retain private writable state. Install dependencies from `requirements.txt` with the supported Python version; ML-DSA verification additionally needs the compatible OpenSSL runtime described in the attestation documentation. The template grants no network access and exposes no signing credentials. Validate the unit on the actual Linux host with `systemd-analyze verify` before installing; no enable/start command is authorized by the mere presence of this template.
 
 Before activation, run the zero-skip acceptance suite, authenticate a fresh exact-bound bundle, exercise a selector switch, restart/replay rejection, expiry, missing configuration, durable failure recovery and confirmed alert delivery at the operator's destination. Verify actual service ownership, filesystem permissions and persistence across a service restart. Hosted CI must cover the final committed tree; local fixture tests alone do not establish production success.
+
+## Automated internal KMS status
+
+An operator can replace the manually asserted `keyStatus` object with the following internal policy. The existing `trust` entry must use the pinned ML-DSA-65 SPKI DER key; a matching key identifier alone is insufficient:
+
+```json
+{
+  "mode": "aws-kms",
+  "receipt": "/absolute/private/config/kms-status.json",
+  "withdrawal": "/absolute/private/config/withdrawal.json",
+  "keyArn": "arn:aws:kms:REGION:ACCOUNT:key/UUID",
+  "spkiSha256": "<independently approved 64-hex SPKI digest>"
+}
+```
+
+The independently provisioned withdrawal file contains `{"version":1,"withdrawn":false,"reason":"<operator decision>"}`. Changing `withdrawn` to `true` immediately fails the next consumer check; a missing/malformed file fails closed. The refresher never writes this override. Retain the local override independently of downloaded receipts, bundles and deployment releases.
+
+Run `python -m tools.kms_status --policy /private/kms-policy.json --output /private/kms-status.json` in the existing authorized signing environment. The policy contains exactly `version: 1`, `keyArn`, `spkiSha256` and `ttlSeconds`. Use an explicit key ARN, never an alias. A 15-minute workflow interval with `ttlSeconds: 1800` leaves one refresh interval of margin; the maximum accepted receipt lifetime is one hour. Optional `--profile qk-attest` selects an already configured local profile. Actions should inherit its existing AWS authentication and omit the profile. No signing API is called and no credentials are read out, copied, exported to the consumer or installed on its host.
+
+The bounded CLI invokes authenticated `DescribeKey` and `GetPublicKey` against the fixed regional HTTPS KMS endpoint. It requires the exact ARN, enabled state, `ML_DSA_65`, `SIGN_VERIFY`, `ML_DSA_SHAKE_256`, valid ML-DSA-65 SPKI framing and the approved SHA-256 digest. AWS documents the SPKI and algorithm fields in [GetPublicKey](https://docs.aws.amazon.com/kms/latest/APIReference/API_GetPublicKey.html). Requests that fail or disagree replace the receipt with `unknown` and return a nonzero exit code. Transfer that unknown receipt too; do not conceal a disabled key by publishing only successful refreshes. Invalid local policy or an unwritable output path stops publication; an old receipt expires naturally and cannot remain valid indefinitely.
+
+Publish the receipt over the independently authenticated SSH connection into the private configuration directory, with atomic replacement, file/directory fsync and owner-only permissions. Its exact fields are `version`, `status`, `keyArn`, `spkiSha256`, `checkedAt`, `validUntil`, and `source` (`aws-kms:DescribeKey+GetPublicKey`). This JSON is not self-authenticating: trust depends on the authorized refresher, verified SSH host identity and protected local file. The consumer checks the receipt against both the configured ARN/pin and actual trusted public key, requires current bounded timestamps, and persists the highest valid KMS check time to reject rollback, including unknown-status receipts. After an unknown receipt, only an active check with a strictly later timestamp can restore acceptance. Receipt absence, expiry, unknown status or local withdrawal queues a durable monitor failure without replacing the last trusted baseline.
+
+This internal policy establishes that the authorized refresh observed the expected KMS key enabled, subject to the receipt's freshness window and AWS eventual consistency. It does not establish public, root-signed nonrevocation, third-party measurement independence or PQC TLS authentication. Public revocation publication and independent interoperability remain distinct acceptance work.
+
+## Durable email delivery
+
+The approved QuantaKrypto alert destination is `[email protected]`. Provision owner-private `delivery.json` with `{"version":1,"recipient":"[email protected]","sender":"quantakrypto <[email protected]>"}` after confirming the existing sender configuration. Honor a configured production `EMAIL_FROM` when provisioning this sender instead of assuming the default. The adapter uses the existing `RESEND_API_KEY` from its runtime environment; it does not create, copy or print the key. Configure the service to read the existing host environment in place using a reviewed host-specific `EnvironmentFile` drop-in or an existing launcher that passes only the needed variable. Protect that existing source file with owner-only permissions before activation. The public contact form and its automatic confirmation are not a monitor transport.
+
+Explicitly initialize delivery once with `python -m tools.alert_delivery --initialize-delivery --state-directory /absolute/private/state`, after initializing the consumer state. This adds `delivery.json` and its stable lock inside the persistent state directory. Initialization refuses previous or interrupted delivery state. Missing delivery state after initialization is an error; never delete it to retry a send. Run `python -m tools.alert_delivery --config /absolute/private/config/delivery.json --state-directory /absolute/private/state` from the existing mail runtime environment. The service/timer templates run independently from the offline consumer, so mail failures do not stop measurement assessment.
+
+Each alert occurrence gets a durable sending intent and immutable payload before any network call. The Resend idempotency key is derived from its `deliveryId`; a returned provider message ID is persisted before polling. The message names the subject, occurrence, finding and repair action. Up to three least-recently-checked pending occurrences are processed per run, so bounced or ambiguous messages cannot starve later alerts. The provider transport uses fixed HTTPS endpoints, rejects redirects and bounds response bytes and total request time. Errors expose only fixed diagnostics, never API credentials or response bodies.
+
+API acceptance (`sent` or `queued`) is not delivery. The adapter retains the pending alert until the provider's matching message record reports `delivered`, `opened` or `clicked`; it durably records that result before acknowledging the exact occurrence. This is provider-reported delivery to the receiving mail system, not proof that a human read or acted on the alert. The provider ID and delivery event remain in the private journal for acceptance evidence. Bounces, suppression, failure and complaints remain pending and report `delivery-failed` for operator action. Unknown or inaccessible provider status cannot acknowledge an alert; the existing key must permit message retrieval as well as sending.
+
+Resend retains [idempotency keys for 24 hours](https://resend.com/changelog/idempotency-keys). An ambiguous submission can retry the same immutable request for at most 23 hours. After that it reports `reconciliation-required` and never blindly resends. Reconcile the occurrence tag/provider activity before any operator recovery; the tool deliberately cannot guess whether a message was sent when both acknowledgment and provider ID were lost. Delivered receipts can be retried locally after an outbox-write failure without sending another email. Retain receipt history across releases; reaching the bounded journal capacity requires reviewed archival, not silent deletion.
+
+For production acceptance, send a clearly labeled monitor test occurrence to the approved recipient through this same durable path. Verify the provider-reported delivery event, persistent receipt and exact outbox acknowledgment, then exercise a send/status failure and recovery without losing its occurrence ID. Do not claim delivery from the CLI exit code alone. Hosted CI covers fixture provider behavior; the real host receipt establishes actual delivery.

tests/test_alert_delivery.py

@@ -0,0 +1,134 @@
+"""Durable provider receipts, bounded idempotency and precise outbox acknowledgment."""
+import datetime as dt
+import json
+from pathlib import Path
+import tempfile
+import unittest
+from unittest.mock import patch
+from tools.consumer import initialize_state
+from tools.state import enqueue_failure
+from test_consumer import NOW, put
+
+
+class Provider:
+    def __init__(self):
+        self.sent = []
+        self.event = 'sent'
+        self.payload = None
+        self.failure = False
+    def request(self, method, path, payload=None, idempotency_key=None):
+        if method == 'POST':
+            self.sent.append(idempotency_key)
+            self.payload = payload
+            if self.failure: raise RuntimeError('provider timeout with sensitive detail')
+            return {'id': 'receipt-1'}
+        return {'id': 'receipt-1', 'to': self.payload['to'], 'from': self.payload['from'], 'subject': self.payload['subject'], 'last_event': self.event}
+
+
+class DeliveryTests(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        self.root = Path(self.tmp.name).resolve()
+        self.state = self.root/'state'
+        self.state.mkdir(mode=0o700)
+        initialize_state(self.state, NOW)
+        self.config = self.root/'delivery.json'
+        put(self.config, b'{"version":1,"recipient":"[email protected]","sender":"quantakrypto <[email protected]>"}')
+        self.alert = {'id': 'test-alert', 'type': 'tls-regression', 'severity': 'error', 'action': 'Inspect the TLS deployment and restore approved hybrid key exchange.'}
+        self.pending = enqueue_failure(self.state/'state.json', self.alert)[0]
+        self.provider = Provider()
+        from tools.alert_delivery import initialize_delivery
+        initialize_delivery(self.state, NOW)
+
+    def deliver(self, now=NOW):
+        from tools.alert_delivery import deliver_pending
+        return deliver_pending(self.state, self.config, self.provider, now=now)
+
+    def pending_alerts(self):
+        return json.loads((self.state/'state.json').read_bytes())['pendingAlerts']
+
+    def test_acceptance_is_not_delivery_and_restart_polls_saved_receipt(self):
+        self.deliver()
+        self.assertTrue(self.pending_alerts())
+        self.assertEqual(len(self.provider.sent), 1)
+        self.provider.event = 'delivered'
+        self.deliver()
+        self.assertEqual(len(self.provider.sent), 1)
+        self.assertEqual(self.pending_alerts(), [])
+        self.assertIn('restore approved hybrid', self.provider.payload['text'])
+        self.assertEqual(self.provider.payload['to'], ['[email protected]'])
+
+    def test_failed_send_retains_same_occurrence_and_idempotency_key(self):
+        self.provider.failure = True
+        self.deliver()
+        self.deliver(NOW+dt.timedelta(seconds=60))
+        self.assertEqual(self.provider.sent[0], self.provider.sent[1])
+        self.assertEqual(self.pending_alerts()[0]['deliveryId'], self.pending['deliveryId'])
+        self.assertNotIn('sensitive detail', (self.state/'delivery.json').read_text())
+
+    def test_ambiguous_send_beyond_idempotency_window_is_not_blindly_resent(self):
+        self.provider.failure = True
+        self.deliver()
+        results = self.deliver(NOW+dt.timedelta(hours=24))
+        self.assertEqual(len(self.provider.sent), 1)
+        self.assertEqual(results[0]['status'], 'reconciliation-required')
+        self.assertTrue(self.pending_alerts())
+
+    def test_receipt_persisted_before_ack_and_old_receipt_cannot_ack_recurrence(self):
+        self.provider.event = 'delivered'
+        with patch('tools.alert_delivery.acknowledge_alerts', side_effect=OSError('disk failure')):
+            self.deliver()
+        self.assertTrue(self.pending_alerts())
+        self.deliver()
+        self.assertEqual(self.pending_alerts(), [])
+        self.assertEqual(len(self.provider.sent), 1)
+        new = enqueue_failure(self.state/'state.json', self.alert)[0]
+        self.assertNotEqual(new['deliveryId'], self.pending['deliveryId'])
+        self.provider.event = 'sent'
+        self.deliver()
+        self.assertEqual(self.pending_alerts()[0]['deliveryId'], new['deliveryId'])
+        self.assertEqual(len(self.provider.sent), 2)
+
+    def test_bounce_and_mismatched_provider_receipt_never_acknowledge(self):
+        self.provider.event = 'bounced'
+        result = self.deliver()
+        self.assertEqual(result[0]['status'], 'delivery-failed')
+        self.assertTrue(self.pending_alerts())
+        self.provider.event = 'delivered'
+        with patch.object(self.provider, 'request', return_value={'id': 'wrong', 'last_event': 'delivered'}):
+            self.deliver()
+        self.assertTrue(self.pending_alerts())
+
+    def test_missing_receipt_journal_or_clock_rollback_never_resets_delivery(self):
+        self.deliver()
+        with self.assertRaises(ValueError): self.deliver(NOW-dt.timedelta(seconds=1))
+        (self.state/'delivery.json').unlink()
+        with self.assertRaises(ValueError): self.deliver()
+        self.assertFalse((self.state/'delivery.json').exists())
+        self.assertTrue(self.pending_alerts())
+
+    def test_no_network_without_api_key(self):
+        from tools.alert_delivery import ResendClient
+        with self.assertRaises(ValueError): ResendClient('')
+
+    def test_failed_first_batch_does_not_starve_later_occurrences(self):
+        for number in range(3):
+            enqueue_failure(self.state/'state.json', dict(self.alert, id='other-'+str(number)))
+        self.provider.failure = True
+        self.deliver()
+        first = set(self.provider.sent)
+        self.deliver(NOW+dt.timedelta(seconds=60))
+        self.assertEqual(len(set(self.provider.sent)), 4)
+        self.assertEqual(len(first), 3)
+
+    def test_corrupted_delivery_clock_is_not_accepted(self):
+        path = self.state/'delivery.json'
+        raw = json.loads(path.read_bytes())
+        raw['lastNow'] = float('nan')
+        put(path, json.dumps(raw).encode())
+        with self.assertRaises(ValueError): self.deliver()
+        self.assertEqual(self.provider.sent, [])
+
+
+if __name__ == '__main__': unittest.main()