[runck014]

devmon MSI publisher: ingest key from Secret Manager (FRI 2026-10-16)

MSI never holds the ingest token. Each run the publisher reads it from Secret Manager with a
service-account key that can read the two token secrets and nothing else. Nothing below has been
run against GCP or MSI; the patch is not pushed.

Flow

scrontab (agate, runck014) -> scripts/scheduled_device_monitoring.sh
  -> parse credentials/.rtgs_credentials as data (allowlist; DEVICEMON_INGEST_URL, GEMS DB,
     Particle, Gmail)
  -> checkout the tag+SHA in ~/.config/devmon/release (git hooks off)
  -> refuse if the script, its dir, the checkout, or the venv is group/world-writable
  -> rtgs device-monitoring check-ingest   (preflight: token fetch + /v1/status)
  -> rtgs device-monitoring monitor -> batch -> logs/batches/ -> POST /v1/batches
token fetch: ~/.config/devmon/sa.json (600, ours) -> device-monitoring-msi@rtgslab
  -> GET secretmanager .../secrets/<secret>/versions/latest:access -> memory only
secret follows the URL: test host -> devicemon-ingest-token-test,
                        prod -> devicemon-ingest-token.
The token is sent only to the 3 known ingest hostnames.

What MSI holds

• ~/.config/devmon/sa.json: SA key, 600, Bryan's home. Can read the 2 token secrets, nothing
  else (secret-level bindings; no project roles).
• ~/.config/devmon/release: devmon-vX.Y.Z <40-hex sha>, 600. Editing this line deploys.
• shared credentials/.rtgs_credentials (660): URL + GEMS/Particle/Gmail. No token. A
  DEVICEMON_INGEST_TOKEN line is ignored and logged as "delete it".
• code + venv + scripts/: Bryan-owned 2750 dirs, 640/750 files. logs/ + logs/batches/: 2770.

Order (Bryan)

1. Here: review 03-output-20261016-devmon-rtgs-lab-tools.patch (7 commits on 7098865); push
   the branch, PR, merge, tag devmon-v0.1.0; note the tag's commit SHA.
2. Here: bash 06-output-20261016-devmon-msi-sa-setup.sh (dry run), then
   DRY_RUN=0 bash 06-output-20261016-devmon-msi-sa-setup.sh (y/N per change).
3. Run the ssh / scp / chmod lines it prints (Duo on each).
4. agate: (umask 077; printf 'devmon-v0.1.0 <sha>\n' > ~/.config/devmon/release)
5. agate, shared workspace (dir 04's install): move the radli009-owned scripts/ and
   rtgs-lab-tools-cron/ aside, re-create them as yours 2750; chmod 2770 logs logs/batches.
   The script refuses to run until the code side is yours and not group-writable.
6. agate: set DEVICEMON_INGEST_URL to the TEST host in the credentials file, delete any token
   line, check: scripts/scheduled_device_monitoring.sh --check-creds credentials/.rtgs_credentials
7. agate: test sequence below; one manual run; then the prod URL; then scrontab.

Test sequence (agate shell, venv active, against test)

T=https://device-monitoring-ingest-test-685278787484.us-central1.run.app
curl -s -o /dev/null -w '%{http_code}\n' $T/health                        # 200
curl -s -o /dev/null -w '%{http_code}\n' -X POST $T/v1/batches \
     -H 'Content-Type: application/json' -d '{}'                         # 401
DEVICEMON_INGEST_URL=$T rtgs device-monitoring check-ingest
  # token from projects/685278787484/secrets/devicemon-ingest-token-test/versions/1
  # /health HTTP 200; /v1/status HTTP 200
DEVICEMON_INGEST_URL=$T python -c '
import datetime as d; from rtgs_lab_tools.device_monitoring import publisher as p
n = d.datetime.now(d.timezone.utc); day = n.strftime("%Y-%m-%d")
print(p.publish(p.build_batch([], [], {}, n.strftime("%Y-%m-%d %H:%M"), day, day,
                              full_fleet=False)))'
  # 201 {"status": "stored", "monitoring_rows": 0, ...}; same minute again -> 200 duplicate

The empty batch writes one row to ingest_batches in device_monitoring_test (shows in
/v1/status); full_fleet=False so no logger is marked inactive. Token never on a command line.

Rotate the API key

Services pin token versions by number (checked 14:35: test -> devicemon-ingest-token-test:1,
prod -> devicemon-ingest-token:1; 05-context). MSI reads latest. So a version must never
be added without a redeploy, or MSI gets 401 until the service is redeployed.

• Use workflow devmon-rotate-ingest-token.yml (rtgs-infrastructure): phase=add adds a
  version AND redeploys accepting new + previous. MSI picks up the new one next run; nothing to
  do on MSI. After one green run: phase=retire.
• Its "Next step" summary still says to paste the token on MSI: stale, a one-line infra fix.

Rotate the SA key

gcloud iam service-accounts keys create ~/.config/devmon/sa.json.new \
  --iam-account=device-monitoring-msi@rtgslab.iam.gserviceaccount.com --project=rtgslab
(run under umask 077) ; scp it to agate:.config/devmon/sa.json.new ; on agate:
chmod 600 ~/.config/devmon/sa.json.new && mv ~/.config/devmon/sa.json{.new,}
rtgs device-monitoring check-ingest        # must say "token from ..."
gcloud iam service-accounts keys delete <old key id> --iam-account=... --project=rtgslab

gcloud iam service-accounts keys list --iam-account=... --managed-by=user shows the IDs.

Open

• The credentials file's variable names were never read; --check-creds lists what it accepts.
• LOGGING_INSTANCE_CONNECTION_NAME and the direct Cloud SQL path are not accepted (removed).
• Local ~/.config/devmon/msi.env still holds the prod token from dir 02; not needed on MSI.

🤖 Generated with Claude Code

https://claude.ai/code/session_0149VSJ7dmCkXd4UcUjQbG9R