device_monitoring: publish to ingest API; token from Secret Manager; hardened MSI cron

.github/workflows/devmon-publisher.yml

@@ -0,0 +1,44 @@
+name: Device monitoring publisher
+
+# Checks the MSI cron side on every PR that touches it and on every release tag the cron can be
+# pinned to (devmon-v*). The cron itself runs on MSI; GitHub cannot reach MSI.
+
+on:
+  pull_request:
+    paths:
+      - "src/rtgs_lab_tools/device_monitoring/**"
+      - "tests/device_monitoring/**"
+      - ".github/workflows/devmon-publisher.yml"
+  push:
+    tags:
+      - "devmon-v*"
+
+permissions:
+  contents: read
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Cron script parses
+        run: bash -n src/rtgs_lab_tools/device_monitoring/scheduled_device_monitoring.sh
+
+      - name: Install
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e . pytest pytest-cov jsonschema
+
+      - name: Publisher tests
+        run: pytest -q --no-cov tests/device_monitoring
+
+      - name: CLI takes the flags the cron passes
+        run: |
+          rtgs device-monitoring monitor --help | grep -q -- '--start-date'
+          rtgs device-monitoring publish --help >/dev/null
+          rtgs device-monitoring check-ingest --help >/dev/null

pyproject.toml

@@ -28,7 +28,7 @@ dependencies = [
     'geopandas',
     "earthengine-api>=0.1.375,<0.2",
     "numpy>=1.21.0",
-    "sqlalchemy>=1.4.0",
+    "sqlalchemy>=1.4.0,<2.1",
     "psycopg2-binary>=2.9.0",
     "python-dotenv>=0.19.0",
     "requests>=2.25.0",

src/rtgs_lab_tools/device_monitoring/README.md

@@ -37,123 +37,100 @@ The device monitoring system runs automatically on MSI (Minnesota Supercomputing
 scrontab -e
 
 # Add this line to run monitoring daily at 4:27 AM
-27 4 * * 1-5 /home/$USER/code/scripts/scheduled_device_monitoring.sh
+# The line lives in scrontab.msi (versioned); install it with
+scrontab /projects/standard/runck014/shared/device_monitoring/rtgs-lab-tools-cron/rtgs-lab-tools/src/rtgs_lab_tools/device_monitoring/scrontab.msi
 ```
 
 #### Script Location and Setup
 
-The monitoring script is located at:
+The monitoring script runs from the lab's shared workspace on Bryan's scrontab:
 ```
-/home/$USER/code/scripts/scheduled_device_monitoring.sh
+/projects/standard/runck014/shared/device_monitoring/scripts/scheduled_device_monitoring.sh
 ```
+Paths are overridable from the scrontab environment (never from the credentials file):
+DEVICEMON_BASE_DIR, DEVICEMON_WORK_DIR, DEVICEMON_LOG_DIR, DEVICEMON_CREDS_FILE,
+DEVICEMON_RELEASE_FILE, DEVICEMON_SA_KEY, DEVICEMON_TOKEN_SECRET.
 
-This script automatically:
-- Loads required MSI modules (Python, Git)
-- Clones/updates the rtgs-lab-tools repository
-- Sets up the virtual environment
-- Loads credentials from the MSI home directory
-- Runs the device monitoring
-- Logs all activities to `~/logs/device-monitoring-logs/`
-
-### MSI Credentials Configuration
-
-#### Credentials File Setup
+Ownership (the script refuses to run if the code side is group- or world-writable):
+```
+scripts/, rtgs-lab-tools-cron/ (checkout + venv)   Bryan, 2750 dirs, 640/750 files
+logs/, logs/batches/                               Bryan, group runck014, 2770 (the one
+                                                   group-writable place)
+credentials/.rtgs_credentials                      660, group runck014 -- data, never sourced
+~/.config/devmon/ (700): sa.json (600), release (600)    Bryan's home, not shared
+```
 
-Create a secure credentials file in your MSI home directory:
+Each run:
+- loads MSI modules (Python, Git)
+- parses the credentials file against an allowlist (no `source`)
+- checks out the release pinned in `~/.config/devmon/release` (`<tag> <commit sha>`; both
+  checked; git runs with hooks disabled; an existing tag is never force-moved)
+- fetches the ingest token from Secret Manager with `~/.config/devmon/sa.json`
+  (`rtgs device-monitoring check-ingest` preflight), runs the monitor, publishes the batch
+- logs to `logs/device_monitoring_<stamp>_<random>.log`; the lock is in `~/.cache/devmon/`
 
-```bash
-# Create credentials file with restricted permissions
-touch ~/.rtgs_creds
-chmod 600 ~/.rtgs_creds
-```
+### MSI Credentials Configuration
 
-#### Required Credentials
+#### Shared credentials file (data only)
 
-Edit `~/.rtgs_creds` with your actual credentials:
+`credentials/.rtgs_credentials` holds KEY=VALUE lines (an `export ` prefix and one pair of
+outer quotes are accepted; `$(...)` and `$VAR` are taken literally). Accepted names:
+DEVICEMON_INGEST_URL, PARTICLE_ACCESS_TOKEN, DB_HOST, DB_PORT, DB_NAME, DB_USER, DB_PASSWORD,
+LOGGING_DB_HOST, LOGGING_DB_PORT, LOGGING_DB_NAME, LOGGING_DB_USER, LOGGING_DB_PASSWORD,
+POSTGRES_LOGGING_STATUS, GMAIL_USER, GMAIL_APP_PASSWORD, GMAIL_RECIPIENT. Everything else is
+ignored and named in the log -- in particular DEVICEMON_INGEST_TOKEN (now from Secret
+Manager), DEVICEMON_RELEASE, *_DIR, *_FILE, PATH, LD_*, PYTHON*, GOOGLE_*.
 
 ```bash
 # GEMS Database Configuration
-export DB_HOST=sensing-0.msi.umn.edu
-export DB_PORT=5433
-export DB_NAME=gems
-export DB_USER=your_username
-export DB_PASSWORD=your_password
-
-# Logging Database (optional)
-export LOGGING_DB_HOST=34.170.80.6
-export LOGGING_DB_PORT=5432
-export LOGGING_DB_NAME=logs
-export LOGGING_DB_USER=your_logging_username
-export LOGGING_DB_PASSWORD=your_logging_password
-
-# Device Monitoring Database (Cloud SQL Postgres - backs the web dashboard)
-export DEVICEMON_INSTANCE_CONNECTION_NAME=sustained-edge-501900-p8:us-central1:device-monitoring-database
-export DEVICEMON_DB_NAME=device_monitoring
-export DEVICEMON_DB_USER=devicemon_app
-export DEVICEMON_DB_PASSWORD=your_devicemon_db_password
-# Path to a service account key with only the `roles/cloudsql.client` role,
-# used to authenticate the Cloud SQL connector from MSI (which has no gcloud
-# login of its own). Get this from whoever manages the GCP project.
-export GOOGLE_APPLICATION_CREDENTIALS=/path/to/device-monitoring-writer-key.json
+DB_HOST=sensing-0.msi.umn.edu
+DB_PORT=5433
+DB_NAME=gems
+DB_USER=your_username
+DB_PASSWORD=your_password
+
+# Publish target: device-monitoring-ingest-prod (or -test). The token is NOT here.
+DEVICEMON_INGEST_URL=https://device-monitoring-ingest-prod-685278787484.us-central1.run.app
 
 # Particle Cloud API
-export PARTICLE_ACCESS_TOKEN=your_particle_token
+PARTICLE_ACCESS_TOKEN=your_particle_token
 
 # Email Notifications (optional)
-export [email protected]
-export GMAIL_APP_PASSWORD="your_gmail_app_password"
-export [email protected]
+[email protected]
+GMAIL_APP_PASSWORD="your_gmail_app_password"
+[email protected]
 ```
 
-#### Security Notes
-
-- **File permissions**: The credentials file has `600` permissions (readable only by you)
-- **No VPN required**: MSI infrastructure has direct access to the database
-- **Secure storage**: Credentials are stored locally on MSI, not in version control
-- **Alternative names**: The script also checks for `~/.rtgs_credentials` as a fallback
-- **Service account key**: `GOOGLE_APPLICATION_CREDENTIALS` should also point to a file with `600` permissions, kept outside version control like the credentials file itself
+Check what a run would accept (names and value lengths only, never values):
+```bash
+scripts/scheduled_device_monitoring.sh --check-creds credentials/.rtgs_credentials
+```
 
-### MSI Monitoring Workflow
+#### Ingest token (Secret Manager)
 
-#### Daily Execution Process
+The token is read each run from `devicemon-ingest-token` (prod URL) or
+`devicemon-ingest-token-test` (test URL) in project rtgslab, as the service account
+`[email protected]`, whose only grant is
+secretAccessor on those two secrets. Its key is `~/.config/devmon/sa.json`, owned by the
+running user, mode 600 -- anything else is refused. The token is sent only to the known
+ingest hosts and is never logged or written to disk.
 
-1. **Before 5:00 AM daily**: SLURM crontab triggers the monitoring script
-2. **Environment setup**: Script loads Python and Git modules
-3. **Code update**: Repository is updated with `git pull`
-4. **Credential loading**: Secure credentials are loaded from `~/.rtgs_creds`
-5. **Monitoring execution**: Device monitoring runs across all projects
-6. **Logging**: Results and logs are saved to `~/logs/device-monitoring-logs/`
-7. **Cleanup**: Virtual environment is properly deactivated
+#### Release pin
 
-#### Log File Management
+`~/.config/devmon/release`, one line: `devmon-v0.1.0 <40-hex commit sha of that tag>`.
+Changing this line is the deploy.
 
-Monitoring logs are stored in:
-```
-~/logs/device-monitoring-logs/device_monitoring_YYYYMMDD_HHMMSS.log
-```
+### Log File Management
 
-Each log file contains:
-- Execution timestamps
-- Git repository updates
-- Credential loading status
-- Monitoring command output
-- Error messages and debugging information
-- Cleanup status
+Logs: `logs/device_monitoring_YYYYMMDD_HHMMSS_<random>.log`; batches:
+`logs/batches/<run>Z-<batch id>.json` (contain no credentials).
 
 #### Manual Execution
 
-You can also run monitoring manually on MSI:
-
 ```bash
-# Load required modules
-module load python
-module load git
-
-# Run the monitoring script directly
-/home/$USER/code/scripts/scheduled_device_monitoring.sh
-
-# Or run specific monitoring with parameters
-/home/$USER/code/scripts/scheduled_device_monitoring.sh 2024-01-01 2024-01-31 "e00fce68c7f2c2b634bb4b7d" LCCMR
+/projects/standard/runck014/shared/device_monitoring/scripts/scheduled_device_monitoring.sh
+# or with parameters
+.../scripts/scheduled_device_monitoring.sh 2024-01-01 2024-01-31 "e00fce68c7f2c2b634bb4b7d" LCCMR
 ```
 
 ## Python API Usage

src/rtgs_lab_tools/device_monitoring/cli.py

@@ -96,5 +96,55 @@ def monitor_cmd(ctx, start_date, end_date, node_ids, project, no_email):
     )
 
 
+@device_monitoring_cli.command("publish")
+@click.argument("batch_file", type=click.Path(exists=True, dir_okay=False))
+def publish_cmd(batch_file):
+    """Re-send a saved batch file to the ingest service (a stored batch is a no-op)."""
+    from .publisher import publish_file
+
+    publish_file(batch_file)
+
+
+@device_monitoring_cli.command("check-ingest")
+def check_ingest_cmd():
+    """Fetch the ingest token from Secret Manager and call /health and /v1/status.
+
+    Prints the secret version used and the HTTP statuses, never the token. Exit 0 ok,
+    2 setup is wrong (key file, IAM, URL), 3 transient (network, 5xx): worth a retry.
+    """
+    import os
+
+    import requests
+
+    from . import ingest_token
+
+    url = os.getenv("DEVICEMON_INGEST_URL", "")
+    try:
+        base, _ = ingest_token.check_ingest_url(url)
+        secret = ingest_token.secret_for(base)
+        token, version = ingest_token.fetch_from_secret_manager(secret)
+    except ingest_token.TokenError as e:
+        click.echo(f"check-ingest: FAILED: {e}", err=True)
+        sys.exit(3 if e.transient else 2)
+    click.echo(f"check-ingest: token from {version}")
+    try:
+        h = requests.get(f"{base}/health", timeout=(10, 30))
+        s = requests.get(f"{base}/v1/status", timeout=(10, 30),
+                         headers={"Authorization": f"Bearer {token}"})
+    except requests.RequestException as e:
+        click.echo(f"check-ingest: FAILED: {base} unreachable: {type(e).__name__}", err=True)
+        sys.exit(3)
+    finally:
+        del token
+    click.echo(f"check-ingest: {base}/health HTTP {h.status_code}; /v1/status HTTP "
+               f"{s.status_code} {s.text[:200] if s.status_code == 200 else ''}".rstrip())
+    if s.status_code == 401:
+        click.echo("check-ingest: FAILED: the service rejects this token (it pins secret "
+                   "versions; redeploy it after adding a version)", err=True)
+        sys.exit(2)
+    if h.status_code != 200 or s.status_code != 200:
+        sys.exit(3 if max(h.status_code, s.status_code) >= 500 else 2)
+
+
 if __name__ == "__main__":
     device_monitoring_cli()

src/rtgs_lab_tools/device_monitoring/config.py

@@ -79,6 +79,7 @@
 STEP_TIMEOUT_DATABASE_WRITE = 600  # Cloud SQL connect + commits
 STEP_TIMEOUT_MESSAGE_BUILD = 600  # includes Particle API lookups
 STEP_TIMEOUT_NOTIFY = 300  # SMTP handshake + send
+STEP_TIMEOUT_PUBLISH = 540  # POST to device-monitoring-ingest, retries included
 
 # Socket timeout for the outgoing SMTP connection (seconds).
 SMTP_TIMEOUT = 60

src/rtgs_lab_tools/device_monitoring/core.py

@@ -22,6 +22,7 @@
     STEP_TIMEOUT_DATA_RETRIEVAL,
     STEP_TIMEOUT_MESSAGE_BUILD,
     STEP_TIMEOUT_NOTIFY,
+    STEP_TIMEOUT_PUBLISH,
 )
 from .data_analyzer import analyze_data
 from .data_formatter import format_data_with_parser
@@ -32,7 +33,11 @@
 from .watchdog import step_timeout
 
 from .web_app.models import app
-from .web_app.produce_db import init_db, build_db, build_logger_info, build_app_config
+from .web_app.produce_db import (
+    init_db, build_db, build_logger_info, build_app_config,
+    monitoring_records, logger_info_records, app_config_defaults,
+)
+from .publisher import build_batch, publish, save_batch
 
 
 def _run_timestamp():
@@ -83,15 +88,34 @@ def monitor(
         analysis_dict = analyze_data(formatted_data)
     print(f"--Data analysis complete--")
 
-    # Step 3.5: Add data to SQLite database
-    print(f"--Beginning adding data to SQLite database--")
-    with step_timeout(STEP_TIMEOUT_DATABASE_WRITE, "Database write"):
-        with app.app_context():
-            init_db()
-            build_db(analysis_dict, monitoring_timestamp)
-            build_logger_info(analysis_dict)
-            build_app_config()
-    print(f"--Addition to SQLite database complete--")
+    # Step 3.5: Store the results. On MSI they are published to device-monitoring-ingest
+    # (rtgs-infrastructure); the direct database write remains for local use.
+    if os.getenv("DEVICEMON_INGEST_URL"):
+        print("--Beginning batch build (Particle lookups)--")
+        with step_timeout(STEP_TIMEOUT_DATABASE_WRITE, "Batch build"):
+            batch = build_batch(
+                monitoring_records(analysis_dict),
+                logger_info_records(analysis_dict),
+                app_config_defaults(),
+                monitoring_timestamp,
+                start_date,
+                end_date,
+                full_fleet=node_ids is None,
+            )
+            batch_path = save_batch(batch)
+        print(f"--Batch saved: {batch_path}--")
+        with step_timeout(STEP_TIMEOUT_PUBLISH, "Publish"):
+            publish(batch)
+        print("--Publish complete--")
+    else:
+        print(f"--Beginning adding data to SQLite database--")
+        with step_timeout(STEP_TIMEOUT_DATABASE_WRITE, "Database write"):
+            with app.app_context():
+                init_db()
+                build_db(analysis_dict, monitoring_timestamp)
+                build_logger_info(analysis_dict)
+                build_app_config()
+        print(f"--Addition to SQLite database complete--")
 
     # Step 4: Build notification messages
     print(f"--Beginning message building--")