Fix Monthly Climate Data Update workflow; add dry-run QA and auto-file-on-failure

.github/workflows/climate-update.yml

@@ -2,14 +2,30 @@ name: Monthly Climate Data Update
 
 on:
   schedule:
-    - cron: '0 6 1 * *'  # 1st of each month, 6am UTC
-  workflow_dispatch:       # manual trigger
+    - cron: '0 6 1 * *'  # 1st of each month, 6am UTC — live run
+    # Weekly dry-run heartbeat. Proves the plumbing (package load, EDH auth,
+    # AWS read+write, catalog read) between live runs, and keeps the workflow
+    # active so GitHub does not auto-disable the cron after 60 days of repo
+    # inactivity — a silent failure no notification would ever catch.
+    - cron: '0 6 * * 1'  # Mondays, 6am UTC — dry run
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: 'Dry run — probe credentials and catalog, skip fetch/publish'
+        type: boolean
+        default: true
 
 jobs:
   update:
     runs-on: ubuntu-latest
     timeout-minutes: 360   # 6 hours max
 
+    # The pipeline publishes to S3, not to this repo — it never needs to push.
+    # issues:write is for the auto-file-on-failure step only.
+    permissions:
+      contents: read
+      issues: write
+
     env:
       EDH_TOKEN: ${{ secrets.EDH_TOKEN }}
       AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
@@ -23,7 +39,13 @@ jobs:
         with:
           use-public-rspm: true
 
+      # local::. installs cd itself, so pipeline_update_edh.R takes the
+      # library(cd) branch. Without it only cd's dependencies install, the
+      # script falls through to devtools::load_all(), and devtools is absent
+      # on the runner — the failure that killed every scheduled run (#78).
       - uses: r-lib/actions/setup-r-dependencies@v2
+        with:
+          extra-packages: local::.
 
       - name: Install uv (runs the Python backfill)
         uses: astral-sh/setup-uv@v5
@@ -39,6 +61,21 @@ jobs:
             exit 1
           fi
 
+      # Plain if/elif/else rather than a nested ${{ a && b || c }} expression:
+      # that idiom mis-evaluates when the middle term is falsy, which is exactly
+      # the case here (dry_run can legitimately be false).
+      - name: Resolve run mode
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            DRY="${{ inputs.dry_run }}"
+          elif [ "${{ github.event.schedule }}" = "0 6 * * 1" ]; then
+            DRY=true      # weekly heartbeat
+          else
+            DRY=false     # monthly live run
+          fi
+          echo "CD_DRY_RUN=$DRY" >> "$GITHUB_ENV"
+          echo "event=${{ github.event_name }} schedule='${{ github.event.schedule }}' -> dry_run=$DRY"
+
       - name: Run EDH update pipeline
         # pipefail so Rscript's exit code propagates through `tee`; default
         # bash -e alone would let tee swallow a non-zero R exit.
@@ -48,14 +85,73 @@ jobs:
           mkdir -p logs
           Rscript scripts/pipeline_update_edh.R 2>&1 | tee logs/update_$(date +%Y%m%d).log
 
-      # Only commit logs when running on main. workflow_dispatch from a
-      # feature branch should not rebase onto or push to main.
-      - name: Commit log
-        if: always() && github.ref == 'refs/heads/main'
+      # Logs are gitignored (.gitignore: logs/*.log), so the old "commit the
+      # log back to main" step could never stage anything — it only ever
+      # reached `git push`, which 403s on the default read-only token even
+      # with nothing to push. Upload as an artifact instead (#78).
+      - name: Upload run log
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: climate-update-log-${{ github.run_id }}
+          # Scoped to this pipeline's logs — logs/ also holds three stale
+          # backfill logs tracked in the repo from before logs/*.log was
+          # gitignored, which have nothing to do with this run.
+          path: logs/update_*.log
+          retention-days: 30
+          if-no-files-found: warn
+
+      # Primary failure alarm. Team-visible and durable, unlike watch-emails,
+      # which go only to the actor and depend on per-user notification settings.
+      # Runs after the artifact upload so the log is preserved either way.
+      # Deduped by label: a run of red months yields one thread, not four.
+      - name: File or update failure issue
+        if: failure()
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
-          git config user.name "github-actions[bot]"
-          git config user.email "github-actions[bot]@users.noreply.github.com"
-          git pull --rebase origin main
-          git add logs/ || true
-          git diff --cached --quiet || git commit -m "Monthly climate update $(date +%Y-%m-%d)"
-          git push
+          set -euo pipefail
+
+          # Static parts first. The log tail is appended separately so backticks
+          # and $ in R/AWS output are never expanded by this shell.
+          cat > /tmp/failure-body.md <<EOF
+          The **Monthly Climate Data Update** workflow failed.
+
+          | | |
+          |---|---|
+          | Run | ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} |
+          | Trigger | \`${{ github.event_name }}\` |
+          | Cron | \`${{ github.event.schedule }}\` |
+          | Mode | \`dry_run=${CD_DRY_RUN:-unknown}\` |
+          | Ref | \`${{ github.ref_name }}\` |
+          | Commit | ${{ github.sha }} |
+
+          Full log is attached to the run as artifact \`climate-update-log-${{ github.run_id }}\`.
+
+          <details><summary>Last 50 log lines</summary>
+
+          \`\`\`
+          EOF
+
+          tail -n 50 logs/update_*.log >> /tmp/failure-body.md 2>/dev/null \
+            || echo "(no log file — the job failed before the pipeline step)" >> /tmp/failure-body.md
+
+          cat >> /tmp/failure-body.md <<'EOF'
+          ```
+
+          </details>
+          EOF
+
+          EXISTING=$(gh issue list --label climate-update-failure --state open \
+            --limit 1 --json number --jq '.[0].number // empty')
+
+          if [ -n "$EXISTING" ]; then
+            echo "Commenting on existing tracking issue #${EXISTING}"
+            gh issue comment "$EXISTING" --body-file /tmp/failure-body.md
+          else
+            echo "Opening a new tracking issue"
+            gh issue create \
+              --title "Monthly Climate Data Update failed ($(date -u +%Y-%m-%d))" \
+              --label climate-update-failure \
+              --body-file /tmp/failure-body.md
+          fi

CLAUDE.md

@@ -133,6 +133,10 @@ The two-vignette template is the foundation for additional regional reporting ap
 
 - `data-raw/*.R` runs **locally only** — never on CI. Outputs land in `inst/extdata/` or `inst/vignette-data/` and are committed.
 - `bcsnowdata` (used by ASWS QA scripts) is GitHub-only — **never** add it to DESCRIPTION Suggests; pak can't resolve it on the pkgdown runner. Same for any other GitHub-only package.
+- **CI cannot commit a gitignored path.** `logs/*.log` is ignored, so the old "commit the run log back to main" step in `climate-update.yml` staged nothing, skipped the commit, and only ever reached `git push` — which 403s during `git-receive-pack` ref advertisement even with nothing to push. Four months of red runs. Use `actions/upload-artifact@v4` for run output and keep the token `contents: read` (#78).
+- **`setup-r-dependencies@v2` installs dependencies, not the package.** Any workflow running a `scripts/*.R` entry point that calls `library(cd)` needs `extra-packages: local::.`, or the script falls through to a `devtools::load_all()` that isn't installed on the runner (#78).
+- **Probe credentials at the top of long jobs.** `pipeline_update_edh.R` STEP 0 checks EDH auth and round-trips an S3 sentinel object before any work, turning a stale `EDH_TOKEN` from a six-hour failure into a four-second one. `aws sts get-caller-identity` alone is not enough — it proves the keys parse, not that the bucket is writable. EDH's HTTP Basic auth needs `httpauth = 1L` on the curl handle; libcurl otherwise waits for a `WWW-Authenticate` challenge EDH never sends.
+- **A weekly dry-run cron is cheap insurance.** It proves the plumbing between monthly live runs *and* keeps the workflow active — GitHub auto-disables scheduled workflows after 60 days of repo inactivity, a silent failure no notification catches.
 
 
 <!-- BEGIN SOUL CONVENTIONS — DO NOT EDIT BELOW THIS LINE -->

planning/archive/2026-08-issue-78-climate-update-workflow/README.md

@@ -0,0 +1,43 @@
+## Outcome
+
+The Monthly Climate Data Update workflow had failed on every scheduled run since
+April, and the cause turned out to be four independent defects stacked on top of
+each other. `setup-r-dependencies@v2` had no `extra-packages`, so cd's
+dependencies installed but cd did not, and `pipeline_update_edh.R` fell through
+to a `devtools::load_all()` that wasn't on the runner — the job died at package
+load, before it ever checked for new data. The default `GITHUB_TOKEN` was
+read-only, so the log-commit step 403'd. That step was dead code regardless:
+`logs/*.log` is gitignored, so it staged nothing and only ever reached `git push`
+— which 403s during `git-receive-pack` ref advertisement even with nothing to
+push. Granting `contents: write` would have "repaired" a step that could never
+commit anything, so it was replaced with an artifact upload and the token stayed
+`contents: read`. Then the new dry-run probe found the fourth: the `EDH_TOKEN`
+repo secret has been stale since 2026-04-14, exactly the date of the last green
+run. Even with the first three fixed, the live run would still have died — six
+hours later at the EDH fetch instead of four seconds in at the probe.
+
+The QA layer is the durable part. `--dry-run` / `CD_DRY_RUN` runs credential
+probes, reads the STAC catalog and computes the target year, then exits before
+any fetch or publish. Deliberately not a no-write mode: it round-trips a sentinel
+object under `_healthcheck/`, because `aws sts get-caller-identity` proves the
+keys parse and says nothing about whether the bucket is writable, which is the
+exact class of failure that took this workflow down. A weekly cron runs it as a
+heartbeat, which also keeps GitHub from auto-disabling the schedule after 60 days
+of repo inactivity. Any failure opens a tracking issue, or comments on the open
+one, deduped by label — team-visible and durable, unlike watch-emails that go
+only to the actor. Two acceptance dispatches confirmed the whole chain including
+the dedup path, and the alarm's first real customer was Bug 4 itself.
+
+Worth remembering: EDH's HTTP Basic auth needs `httpauth = 1L` on the curl
+handle, since libcurl waits for a `WWW-Authenticate` challenge EDH never sends;
+and credentials belong on the handle rather than in the URL, both because the
+104-character token isn't URL-encoded and because it keeps the token out of
+anything loggable.
+
+**Left open for the repo owner:** rotate `EDH_TOKEN` (`gh secret set EDH_TOKEN`).
+The sandbox refused to let the agent overwrite shared CI credential material,
+correctly. Until that lands, the live publish path stays unproven — it is
+unchanged by this work, but cannot be exercised. If DestinE tokens are
+short-lived this will recur, and the weekly heartbeat is what catches it.
+
+Closed by: PR for #78 (commits 62256d5, 041f16e, ca4e41e, 0f4a3cc)