v2.27.0: admin — central cron-health dashboard (live Cloud Scheduler per tenant)

[MyAlterLego]

What

A central Cron Health dashboard on platform admin (/admin/modules/cron-health) so Triarch can see, per tenant, whether each project's scheduled jobs are registered, enabled, and succeeding.

The platform queries Google Cloud Scheduler directly per tenant project via the list API and diffs the actual jobs against a canonical expected-cron suite. This catches missing / disabled crons even when nothing is running, which is the exact gap today: Revolution Cyber and Eve currently have zero scheduler jobs and will light up red.

How it authenticates (ADC note)

The task spec called for Application Default Credentials. This repo does not depend on google-auth-library / googleapis; the established pattern for calling Google REST APIs here is mintFahAccessToken() in src/lib/fah-rollout.ts — it signs an RS256 JWT with the platform service-account key (from the secrets vault: FAH_PROMOTER_SA_KEY), exchanges it at oauth2.googleapis.com/token for a cloud-platform-scoped access token, and caches it 50 min. I reused that helper rather than introducing a new auth dependency. cloud-platform scope covers Cloud Scheduler. (Deviation from "ADC" is documented below.)

Reader calls, per tenant:
GET https://cloudscheduler.googleapis.com/v1/projects/{project}/locations/{location}/jobs with Authorization: Bearer <token>.

Files

• src/lib/cron-health/tenants.ts — tenant registry + EXPECTED_CRON_JOBS suite
• src/lib/cron-health/diff.ts — pure expected-vs-actual diff (no I/O)
• src/lib/cron-health/diff.test.ts — 16 unit tests for the diff logic
• src/lib/cron-health/reader.ts — per-tenant Cloud Scheduler reader, error-isolated
• src/app/admin/modules/cron-health/page.tsx — staff-gated dashboard
• src/app/admin/page.tsx — registers the Cron Health module in the admin grid

Tenant registry (extend in tenants.ts)

slug	GCP project	location
triarch	triarchsecurity-atlas	us-central1
revolutioncyber	revolutioncyber-triarchcrm	us-central1
evesecurity	evesecurity-triarchcrm	us-central1

Expected cron suite

connector-sync, event-dispatcher, workflow-dispatcher, workflow-enroll, workflow-tick, workflow-enrich, embedding-refresh, pipeline-snapshot, invoice-overdue-sweep, retention-purge, foundry-ledger-scan, llm-usage-report.

Matched loosely by job-name suffix: an expected key matches an actual scheduler job id if the id equals the key or ends with -<key> (case-insensitive). So connector-sync, triarch-connector-sync, and atlas-prod-connector-sync all match connector-sync.

Diff / health logic

Per expected job → a status row:

• missing (red): no scheduler job matches the key
• paused / disabled (amber)
• last run failed (red) — status.code non-zero
• enabled + recent success (green) — last attempt within 48h, no failure
• enabled but stale (>48h) or never ran (amber)

Tenant rolls up to red if any missing/failing, amber if any paused/stale, else green. The top summary flags tenants with dark crons.

Distinct per-tenant "scheduler not enabled / no access" state: a 403/404 from the list API is rendered as a dedicated red banner (this is the RC/Eve signal today), separate from an unexpected query error. Per-tenant isolation: one project failing never breaks the page (auth fail-closed, per-project queries fail-open).

Operator steps (required before data shows for a tenant)

For each tenant project (triarchsecurity-atlas, revolutioncyber-triarchcrm, evesecurity-triarchcrm):

1. Enable the Cloud Scheduler API on the project:
   gcloud services enable cloudscheduler.googleapis.com --project <PROJECT>
2. Grant the platform runtime SA roles/cloudscheduler.viewer (or a custom role with cloudscheduler.jobs.list / .get) on the project:
   gcloud projects add-iam-policy-binding <PROJECT> --member="serviceAccount:<PLATFORM_RUNTIME_SA>" --role="roles/cloudscheduler.viewer"

The <PLATFORM_RUNTIME_SA> is the identity behind FAH_PROMOTER_SA_KEY (the SA used by the existing FAH rollout client). Until both are done, the tenant section shows the "scheduler not enabled / no access" state — which is itself useful signal.

Verification

• tsc --noEmit: zero new errors in changed files (6 pre-existing errors remain in packages/triarch-shared/src/internal-hmac.test.ts, unrelated).
• vitest run src/lib/cron-health/diff.test.ts: 16/16 pass (missing detection, loose matching, all health classifications, extra-job surfacing).
• next build: compiled successfully; /admin/modules/cron-health builds as a dynamic route.
• Version triad bumped to 2.27.0 (package.json, package-lock.json, src/lib/version.ts).

Deviations

• ADC → SA-JWT reuse: used the repo's existing mintFahAccessToken() (SA key from vault, cloud-platform scope) instead of google-auth-library ADC, to match conventions and avoid a new dependency. Functionally equivalent for reaching the Scheduler API.
• llm-usage sibling page: the llm-usage dashboard referenced as a look model is on an unmerged sibling branch and is not on dev, so I mirrored ci-cd/page.tsx (same zinc/amber design language) instead.

Do not merge/deploy — operator IAM + API-enable steps land first.

🤖 Generated with Claude Code