Fix Monthly Climate Data Update workflow; add dry-run QA and auto-file-on-failure - #80
Merged
NewGraphEnvironment merged 6 commits intoAug 26, 2026
Conversation
Three defects kept every scheduled run of Monthly Climate Data Update red since April: 1. setup-r-dependencies@v2 had no extra-packages, so cd's dependencies installed but cd did not. pipeline_update_edh.R fell through to devtools::load_all(), and devtools is absent on the runner: "there is no package called 'devtools'". The pipeline died at load, before it ever checked for new data. Fixed with extra-packages: local::. which installs cd itself and takes the library(cd) branch. 2. No permissions: block, so GITHUB_TOKEN was read-only and the log commit step's git push got a 403. 3. The log commit step was dead code anyway. logs/*.log is gitignored, so git add logs/ staged nothing and the git diff --cached --quiet guard short-circuited the commit. Only git push remained, and git push 403s during git-receive-pack ref advertisement even with nothing to push. Granting contents: write would have repaired a step that could never commit. The pipeline publishes to S3, not git, so the step is replaced with actions/upload-artifact@v4 and the token stays contents: read. issues: write is added for the auto-file-on-failure step in Phase 4. Both pipeline scripts now fail with a readable message when neither cd nor devtools is available, instead of a bare loadNamespace error. Relates to #78 Co-Authored-By: Claude Opus 4.8 <[email protected]>
--dry-run (or CD_DRY_RUN=true) runs the credential probes, reads the STAC
catalog and computes the target year, then exits 0 before STEP 3. No EDH
pull, no COG rebuild, no S3 publish. Mirrors the --dry-run flag already on
pipeline_stage3_edh.R; the env var lets the GitHub Action pick the mode
without rewriting the command line.
New STEP 0 runs on every path, live runs included. STEP 1/2 can exit 0 early
when already current, and a live run does not touch S3 until STEP 5 — six
hours in. Probing up front turns a credential problem into an immediate,
legible failure instead of one buried at the end of a long job.
Three probes:
- EDH: HEAD the consolidated Zarr metadata, distinguishing a bad token
(401/403) from an unreachable host.
- AWS identity: sts get-caller-identity, so a missing or expired key
reports as such rather than as an opaque S3 error.
- AWS write: round-trip a sentinel object under _healthcheck/<run-id> and
delete it. get-caller-identity only proves the credentials parse; it
says nothing about whether this principal may write to the bucket, and
that is exactly the class of failure this issue is about. Keyed by run
id so concurrent runs cannot clobber each other. A failed delete warns
rather than aborts, since the write is what STEP 5 actually needs.
The EDH probe needs credentials on the curl handle plus httpauth = 1L
(CURLAUTH_BASIC). Embedding them in the URL the way the Python fsspec calls
do yields a spurious 401 in R: the 104-character token is not URL-encoded,
and libcurl otherwise waits for a WWW-Authenticate challenge that EDH never
sends. Handle-based credentials also keep the token out of any loggable
string. Full matrix in findings.md.
Verified locally: exit 0 in ~4 s, sentinel cleaned up, both --dry-run and
CD_DRY_RUN=true take the same path.
Relates to #78
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Dry-run wiring:
- workflow_dispatch gains a boolean dry_run input defaulting to true, so a
manual trigger is cheap and safe by default and the fix can be confirmed
without waiting for the Sept 1 cron.
- A second weekly cron (Mondays 06:00 UTC) runs the dry run as a heartbeat.
Beyond proving the plumbing between live runs, it keeps the workflow
active so GitHub does not auto-disable the cron after 60 days of repo
inactivity, which is a silent failure no notification would catch.
- A Resolve run mode step maps event to CD_DRY_RUN via plain if/elif/else
on github.event_name and github.event.schedule. Deliberately not the
nested ${{ a && b || c }} idiom, which mis-evaluates when the middle term
is falsy, and dry_run can legitimately be false.
Failure alarm:
- A final if: failure() step opens a tracking issue, or comments on the
open one, deduped by the climate-update-failure label so a run of red
months yields one thread rather than four. Team-visible and durable,
unlike watch-emails, which go only to the actor and depend on per-user
notification settings. Covers the weekly dry run too, so a broken
heartbeat self-reports.
- Body carries run URL, trigger, cron, resolved mode, ref, commit and the
last 50 log lines. Static parts come from an interpolating heredoc; the
log tail is appended separately so backticks and $ in R/AWS output are
never expanded by the step's shell.
- The label did not exist (repo had only the 9 GitHub defaults) and has
been created.
Both the artifact path and the log tail are scoped to logs/update_*.log
rather than logs/, which also holds three stale backfill logs tracked in the
repo from before logs/*.log was gitignored.
Relates to #78
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two code-check findings, both in pipeline_update_edh.R.
The mode banner claimed "no write" while STEP 0 round-trips a sentinel object
against the production bucket on every run, weekly heartbeat included. That
write is the point of the probe, but the wording would send someone debugging
a bucket-policy or object-lock issue down the wrong path. Banner and header
comment now say what actually happens.
run_id used Sys.getenv("GITHUB_RUN_ID", unset = <pid>), but unset= only fires
when the variable is absent. A set-but-empty value would yield the bare prefix
s3://<bucket>/_healthcheck/ — writable, but not removable by the paired rm, so
a zero-length orphan would accumulate. Guarded with nzchar instead. Latent
rather than live, since Actions always populates the variable.
Verified with GITHUB_RUN_ID="" — falls back to the pid, exits 0, no orphan
left under _healthcheck/.
Relates to #78
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two dry-run dispatches on this branch confirmed everything in scope: cd installs, the log path no longer 403s, CD_DRY_RUN resolves from the dispatch input, the failure alarm opens an issue and the second failure comments on it rather than opening a duplicate. They also exposed a fourth defect no static review would have found. The EDH_TOKEN repo secret has been stale since 2026-04-14, the date of the last green run. The probe returns 403 on the runner and 200 locally. Even with the three known bugs fixed, the live run would still have died, six hours in at the EDH fetch instead of four seconds in at STEP 0. Rotating the secret is left to the repo owner; overwriting shared CI credential material is not the agent's call. CLAUDE.md picks up four CI lessons: gitignored paths cannot be committed by CI, setup-r-dependencies installs dependencies rather than the package, probe credentials at the top of long jobs, and a weekly dry-run cron doubles as protection against GitHub auto-disabling the schedule. Fixes #78 Co-Authored-By: Claude Opus 4.8 <[email protected]>
11 tasks
NewGraphEnvironment
deleted the
78-monthly-climate-data-update-workflow-fa
branch
August 26, 2026 20:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Monthly Climate Data Update workflow has failed on every scheduled run
since April. Four independent defects were stacked on top of each other; three
are fixed here, and the fourth was found by the fix.
setup-r-dependencies@v2had noextra-packages— cd's dependenciesinstalled but cd did not, so
pipeline_update_edh.Rfell through to adevtools::load_all()that isn't on the runner. The job died at package load,before it ever checked for new data. Fixed with
extra-packages: local::..GITHUB_TOKEN— nopermissions:block, so the log-commit step403'd.
logs/*.logisgitignored, so
git add logs/staged nothing, thegit diff --cached --quietguard skipped the commit, and only
git pushremained — which 403s duringgit-receive-packref advertisement even with nothing to push. Grantingcontents: writewould have repaired a step that could never commit anything.Replaced with
actions/upload-artifact@v4; the token stayscontents: read.QA layer
--dry-run/CD_DRY_RUNinpipeline_update_edh.R: run credentialprobes, read the STAC catalog, compute the target year, exit 0 before any fetch
or publish. Deliberately not a no-write mode — it round-trips a sentinel
object under
_healthcheck/, becauseaws sts get-caller-identityproves thekeys parse and says nothing about whether the bucket is writable, which is
exactly the class of failure that took this workflow down.
hours in. Probing up front turns a credential problem into an immediate,
legible failure.
runs, and keeps GitHub from auto-disabling the schedule after 60 days of repo
inactivity — a silent failure no notification would catch.
workflow_dispatchdry_runinput, defaulttrue.climate-update-failurelabel:opens one, comments on it thereafter. Team-visible and durable, unlike
watch-emails which go only to the actor and depend on per-user settings.
Verification
Two dispatches on this branch:
31204565836,
31204944259.
devtoolserrorUpload run log✓ both runsCD_DRY_RUN: true; "DRY RUN" banner loggedevent=workflow_dispatch schedule='' -> dry_run=trueLocally:
Rscript scripts/pipeline_update_edh.R --dry-runexits 0 in ~4 s withall three probes green and no sentinel left behind.
devtools::test()— 214PASS / 0 FAIL.
/code-checkfound 2 issues (misleading "no write" banner, anempty-
GITHUB_RUN_IDhole in the sentinel key); both fixed in 0f4a3cc.EDH_TOKENThe new probe found a fourth defect. The
EDH_TOKENrepo secret has beenstale since 2026-04-14 — exactly the date of the last green run.
authenticates, the principal is forbidden. Revoked or expired, not malformed.
died — six hours in at the EDH fetch instead of four seconds in at STEP 0.
I did not do this; overwriting shared CI credential material isn't the agent's
call, and the sandbox blocked it. Until it's rotated the live publish path
stays unproven — it's unchanged by this work, but can't be exercised. If
DestinE tokens are short-lived this will recur, and the weekly heartbeat is what
catches it next time, within days rather than at the next monthly run.
Issue #79 is a real open failure, not a test artifact — it should close when the
token is rotated and a dry-run goes green.
Related Issues
Test plan
devtools::test()— 214 PASS / 0 FAIL/code-check— 2 findings, both fixed--dry-runexits 0, sentinel cleaned upworkflow_dispatchruns confirm every step except the stale tokenEDH_TOKENrotationNotes
Two things worth carrying forward, now in CLAUDE.md: EDH's HTTP Basic auth needs
httpauth = 1Lon the curl handle (libcurl waits for aWWW-Authenticatechallenge EDH never sends), and credentials belong on the handle rather than the
URL — the 104-character token isn't URL-encoded, and it keeps the token out of
anything loggable.
Generated with Claude Code