Monthly Climate Data Update workflow fails every scheduled run (four stacked defects); add dry-run QA + auto-file-issue-on-failure

[NewGraphEnvironment]

Problem

The Monthly Climate Data Update workflow (.github/workflows/climate-update.yml) has failed on every scheduled run (2026-05-01, 06-01, 07-01, 08-01). The only green runs were manual workflow_dispatch back in April. So the automated producer pipeline has effectively never worked on schedule, and the S3 climate data has not been auto-updated since April.

This is not the benign "no new data this month" case — the job dies at package load, before it ever checks for new data.

Reconciled 2026-08-26 (at merge of #80). This body originally named two bugs and proposed granting contents: write. The work found four independent defects, and contents: write turned out to be the wrong fix — see "What was actually found" below. The original Bug 1 / Bug 2 descriptions are kept intact for the record; the proposed-fix and acceptance sections have been corrected to describe what shipped.

Bug 1 — devtools is not installed on the runner

scripts/pipeline_update_edh.R:31:

if (requireNamespace("cd", quietly = TRUE)) library(cd) else devtools::load_all()

CI installs cd's dependencies but not cd itself, so the else branch runs → devtools::load_all() →

Error in loadNamespace(x) : there is no package called 'devtools'
Execution halted

The pipeline halts immediately; no EDH fetch, no catalog read, nothing.

Bug 2 — the workflow token is read-only

The Commit log step (if: always()) then runs and cannot push:

remote: Permission to NewGraphEnvironment/cd.git denied to github-actions[bot].
fatal: ... The requested URL returned error: 403   (exit code 128)

There is no permissions: block, so the default GITHUB_TOKEN is read-only.

What was actually found

Four defects, stacked. Bugs 3 and 4 were discovered during the work and are the reason the fix does not match the original proposal.

Bug 3 — the Commit log step was dead code, so contents: write was the wrong fix. .gitignore carries logs/*.log, and the pipeline writes logs/update_$(date +%Y%m%d).log. git add logs/ therefore staged nothing, the git diff --cached --quiet guard short-circuited the commit, and only the doomed git push remained — and git push 403s during git-receive-pack ref advertisement even with nothing to push. Granting contents: write would have "repaired" a step that could never commit anything. The step was deleted and replaced with actions/upload-artifact@v4; the token stays contents: read (plus issues: write for the failure-alarm step only).

Bug 4 — the EDH_TOKEN repo secret was stale. Found by the new dry-run credential probe: the secret had not been updated since 2026-04-14T17:04:21Z, exactly the date of the last green run, and returned HTTP 403 (authenticates, but forbidden). Even with Bugs 1–3 fixed, a live run would still have died — six hours in at the EDH fetch rather than four seconds in at the probe. Rotated by the repo owner on 2026-08-26; the next dispatch went green.

Related gotcha worth keeping: EDH's HTTP Basic auth needs httpauth = 1L on the curl handle, because libcurl otherwise waits for a WWW-Authenticate challenge EDH never sends. Credentials go on the handle, not in the URL — the token is not URL-encoded, and this keeps it out of anything loggable.

Fix as shipped

• Bug 1: setup-r-dependencies@v2 gets extra-packages: local::. so cd itself installs and the script takes the library(cd) branch. Chosen over any::devtools — it exercises the installed-package path CI should be testing and avoids pulling the whole dev-tooling tree. The devtools::load_all() fallback now fails with a readable message instead of erroring inside loadNamespace.
• Bug 2 + 3: permissions: { contents: write } — rejected. The log-commit step was removed in favour of actions/upload-artifact@v4, and the job declares contents: read + issues: write. Nothing in this workflow legitimately needs write access to the repo; it publishes to S3.
• Bug 4: EDH_TOKEN rotated. The weekly heartbeat is what catches this next time.

QA / monitoring — dry-run + auto-file-issue-on-failure

Two complementary mechanisms: a dry-run mode to actively confirm the plumbing (now and on a weekly heartbeat), and an auto-filed GitHub issue on failure as the durable, team-visible alarm.

• dry_run path in pipeline_update_edh.R (--dry-run flag or CD_DRY_RUN env): probes credentials, reads the STAC catalog, computes the target year, logs what a live run would fetch, then exits 0 before any EDH pull / COG rebuild / S3 push.
  • Deliberately not a no-write mode: it round-trips a sentinel object under s3://stac-era5-land/_healthcheck/<run-id>. aws sts get-caller-identity proves the keys parse and says nothing about whether the bucket is writable — which is the exact class of failure that has taken this pipeline down.
  • The probe block runs before Step 1, so it executes on the live path too, not just in dry-run.
• workflow_dispatch input dry_run (default true) — manual triggers are cheap and safe by default.
• Weekly cron '0 6 * * 1' in dry-run mode alongside the monthly '0 6 1 * *' live run, distinguished via github.event.schedule. Gives a regular "plumbing still works" signal and keeps the scheduled workflow active so GitHub does not auto-disable it after 60 days of repo inactivity.
• Auto-file a GitHub issue on failure: final if: failure() step searches for an open issue labelled climate-update-failure and comments on it if one exists, otherwise creates one. Body carries run URL, event name, resolved dry-run mode, and the last 50 log lines; the full log is attached as a workflow artifact (30d retention).

Confirming the fix (acceptance)

• workflow_dispatch with dry_run=true completes green: package loads, secrets verified, EDH probe HTTP 200, AWS identity + bucket write both proven, STAC read, target year computed, exits before the EDH fetch. Run 33006696386 — the first non-failing run since April.
• The push path is exercised without a 403 — moot. There is no push path any more (Bug 3); the log is uploaded as an artifact and the token is contents: read.
• Auto-file-on-failure verified organically rather than by deliberate breakage: run 31204565836 opened Monthly Climate Data Update failed (2026-08-07) #79, and run 31204944259 commented on it instead of opening a duplicate — the dedup path, exercised end to end. The alarm's first real customer was Bug 4 itself.
• Both run URLs + outcomes recorded in planning/archive/2026-08-issue-78-climate-update-workflow/findings.md.

References

• Failing runs: 30688521648 (2026-08-01), 28501189058 (2026-07-01)
• Workflow: .github/workflows/climate-update.yml
• Entry point: scripts/pipeline_update_edh.R