Provenanced rerun of all WSGs (4 hosts) + pre-flight gates so inputs and hosts can't drift unnoticed

[NewGraphEnvironment]

Provenanced rerun of all WSGs (4 hosts) + pre-flight gates so inputs and hosts can't drift unnoticed

Problem

fresh holds 93 WSGs / 2,865,775 rows accumulated across runs spanning
2026-05 to 2026-08, on several link and fresh versions — and it has no log
/ log_input tables at all (only fresh_default does). Not one row can be
traced to a run, a config hash, or a date. The floodplain rebuild downstream
reads this schema.

Three gaps produced that, and all three are still open.

1. A run today would silently skip 80 of 119 WSGs

The cypher image is cypher-20260512-warm, which bakes link 0.35.0 + fresh
0.31.0 (env/do/dev/cypher/terraform.tfvars). cypher_prep.sh reinstalls
link from the branch but the installed fresh stays whatever the image carried.
Line 66 of prep echoes it and asserts nothing.

Correction (2026-08-30): the root cause is a declaration, not a missing
install line. DESCRIPTION already pins NewGraphEnvironment/fresh@v0.33.0
in Remotes: — but that pin is never resolved, because fresh is declared in
Suggests and pak::local_install() defaults to dependencies = NA, i.e.
hard dependencies only (verified: formals(pak::local_install)$dependencies is
NA). Meanwhile 20 files under R/ call fresh:: with zero
requireNamespace() guards, including a default argument on an exported
function (lnk_wsg_downstream_check(outlets = fresh::frs_wsg_outlets())). So
fresh is a hard runtime dependency mis-declared as optional, and library(link)
loads cleanly on a host whose fresh cannot run the pipeline.

Moving it to Imports therefore fixes this with no change to the pak call:
upgrade = FALSE suppresses gratuitous upgrades, not required ones, and 0.31.0
does not satisfy the >= 0.33.0 floor.

Correction: the version boundary is v0.33.0, not "newer than 0.31.0".
Checked against the tags — frs_wsg_drainage arrived in v0.32.0, but
frs_wsg_outlets only in v0.33.0. Both are absent from 0.31.0, so the
blocker stands, but 0.32.0 is not sufficient.

link 0.46.0's wsg_run_one.R calls lnk_wsg_downstream_check(), which needs
frs_wsg_outlets(). That errors → the tryCatch does quit(status = 1) → the
bucket loop's || echo "[WARN]" continues to the next WSG.

Result: every WSG on every cypher fails, the run exits 0, and only the
dispatcher's 39 WSGs land. This is the failure mode CLAUDE.md already records
("cyphers hard-fail and WSGs get skipped, which lnk_access(merge=TRUE)
cannot repair").

2. Nothing prevents input staleness

cypher_prep.sh runs snapshot_bcfp.sh on every cypher, but
study_area_run.sh never runs or verifies it on the dispatcher. M1's
primitives are 2026-05-23/26 — three months behind what cyphers load. A run
started today models one area on May data and three on August data, and nothing
says so until afterwards.

3. Credentials fail serially, with a false-green probe

Both DO credentials expired on the same day and had to be replaced (2026-08-30):

credential	used by
access-token in ~/Library/Application Support/doctl/config.yaml	doctl — pre-flight, reserved-IP recovery in cypher_up.sh
do_token in rtj/env/do/dev/cypher/terraform.tfvars	the tofu DO provider — what actually spins droplets

Both were minted 2026-05-18; DO now requires an expiry on PATs. The pre-flight
probes only the first, so the second would have surfaced mid-spin. Worse, the
obvious tofu probe is a false green: tofu plan on a workspace with zero
resources returns Plan: 2 to add without ever calling the DO API. Only a
workspace with existing state (default, holding the reserved IP) forces the
refresh that returns 401. Same class as CLAUDE.md's "an empty result set is not
a pass".

Goal

One clean, uniformly-provenanced rebuild of fresh across 4 hosts,
consolidated to a single table set on the M1, with the pre-flight hardened so
none of the above can recur silently.

Scope — 119 modelable WSGs

96 focal (the 93 persisted + Columbia's KOTL/LARL/SLOC, from
rtj/scripts/gis/projects/nelson/project.yml) close to 125 WSGs, of which
119 are modelable (6 dropped by species presence: LNRS, LEUT, LFRT, MFRT,
UFRT, LKEC). Coverage grows 93 → 119 because closure pulls in the Liard system
(LIAR, LMUS, ULRD, DEAR, FROG, BEAV, DUNE) and others.

Bucketing — derive from closure, not from wscode roots

Partitioning by wscode_ltree root is wrong and reproduces the
sliver-misclassification RUNBOOK §8b documents: a first attempt put NATR in
Fraser (it drains to the Peace) and SPAT with Skeena (it drains the Stikine),
producing 7 overlapping WSGs.

Correct method — union-find over per-WSG fresh::frs_wsg_drainage() closures:
two focal WSGs share a bucket iff their closures intersect. That yields 22
drainage-independent components, LPT-packed into 4 hosts with zero
overlap, so nothing is computed twice and consolidate has no last-writer-wins
ambiguity.

host	focal	raw closure	modelable	content
M1 (dispatcher)	33	41	39	Fraser
job1	25	28	28	Peace + Nass/Taku/coastal
job2	14	28	24	Liard / Fort Nelson / Stikine
job3	24	28	28	Skeena + Columbia

Largest bucket on the dispatcher (free, fast). Exact --focal= strings and
DS-first orders land in research/study_areas.md.

M4 is deferred, not included. It has fresh 0.33.0 and link 0.40.2 but no
database running (docker ps empty, nothing on :5432), so it needs its fwapg
brought up and possibly an FWA restore. It also needs a code change:
study_area_run.sh is 1 dispatcher + N cyphers, with spin/prep/burn
cypher-specific, so a pre-existing host needs a --host= flag that skips those
three but joins run + consolidate. Worth doing; not worth blocking this run.

Work

Status 2026-08-30 — Phases 1-2 shipped in link v0.47.0. Phases 1 and 2 (plus
the bucketing derivation, which was asserted here but uncommitted) landed on
246-preflight-gates-provenanced-rerun. Phases 3-5 — the destructive multi-host
wipe, the paid 4-host run and the provenance audit — are deliberately a separate
session, so the run is driven by merged, reviewed code and the gates get
exercised on a real run rather than being their own test. Three corrections from
that work are folded into the text above.

The bucketing is now derived rather than asserted, by
data-raw/study_area_buckets.R (union-find over per-WSG frs_wsg_drainage()
closures, LPT over the resulting components, writing research/study_areas.md).
It reproduces the numbers claimed below from first principles — 96 focal, 125 in
closure, 119 modelable, 22 components, 39 on the dispatcher, and the
same six groups dropped by species presence. The only difference is the cypher
split, 27/26/27 rather than 28/24/28, because weight is stream-segment count
rather than WSG count; the components are identical and only the packing moves.

Two properties Phase 4 depends on are now asserted in the generator, not
inferred: every host bucket is drainage-closed, and DS-first order holds within
each host across concatenated components. Audited on the derived split — 0
overlapping component closures, 0 closure violations, 0 ordering violations —
and both assertions verified by restoring the bug. Also checked the one gap the
species filter opens: LEUT, LFRT, LKEC, LNRS, MFRT and UFRT sit in the closure
but are never modelled, so they never persist barriers. None of the six
contains a dam, so no WSG above them can be modelled with a barrier missing.

Phase 1 — unblock (correctness cannot depend on image age)

• cypher_prep.sh installs fresh from a pinned ref alongside link, so
  the baked image version is never load-bearing — same reasoning that
  already led prep to reinstall link rather than trust the image
• Assert the installed fresh exports frs_wsg_drainage / frs_wsg_outlets
  before any WSG runs

Phase 2 — pre-flight gates (study_area_run.sh)

Each must fail toward stop, not skip (CLAUDE.md, "A guard must not fail
toward skip"), and each is tested against both known answers before use.

• Sibling-host version parity (Pre-flight: sibling-host parity hook (link+fresh version+SHA) with auto-install option #183) — collect link + fresh
  version + git SHA + dirty from dispatcher and every cypher before any
  WSG runs; abort on mismatch, --auto-install escape
• Credential probe — exercise both DO paths, and force a real API
  call (a workspace with existing state, or doctl compute droplet list);
  a plan against an empty workspace is a false green
• Primitive vintage — assert the oldest of the four primitive tables is
  within N days on every host; abort otherwise. Correction: max(last_analyze)
  is unimplementable as written — measured 2026-08-30, last_analyze is
  NULL on all ten input primitives and only last_autoanalyze is
  populated, so the check would compare an empty string and read as
  "nothing to see" (the anti-pattern CLAUDE.md records). Use
  GREATEST(last_analyze, last_autoanalyze), and treat an absent table or
  a NULL timestamp as a failure rather than a neutral.
• Branch pushed — cyphers git reset --hard origin/$BRANCH; assert
  git rev-list --count @{u}..HEAD is 0
• Persist-schema guard — both bcfishpass and default resolve
  pipeline$schema to fresh, so --config=default without --schema=
  silently clobbers the bcfp persist. Require explicit --schema= when
  config is not bcfishpass.
• Export FWAPG_GIT_SHA on the ssh leg — cyphers have no
  ~/Projects/repo/fwapg, so log.fwapg_sha records NA today

Phase 3 — clean baseline

• state_clean.sh on all hosts: drops 30 orphan working_* schemas,
  fresh_* bundle schemas, DROP SCHEMA fresh CASCADE (takes the 45
  zz_lnk_mc_scratch_* tables with it), re-runs snapshot_bcfp.sh --force
• Confirm primitives reload on every host

Phase 4 — the run

• study_area_run.sh --cy-workspaces=job1,job2,job3 with the four
  --focal= sets
• Post-consolidate lnk_access(merge=TRUE) recompute across all 119
• Compare CSV + lnk_parity_annotate against fresh.streams_vw_bcfp

Phase 5 — verify provenance

• fresh.log has one row per WSG-run, all 119, with non-NULL
  config_hash, link_sha, fresh_sha, fwapg_sha
• fresh.log_input carries per-primitive row counts and source_at
• config_drift FALSE everywhere
• log_parameters_fresh / log_dimensions populated — these carry no
  watershed_group_code, so schema_consolidate.R (which keys off that
  column) skips them; the dispatcher writes them, sufficient only while
  every host runs the same config. Confirm, don't assume.
• Run record in research/ with all four hosts' stamps

Notes

• Image residue is real. lnk_persist_init(force_recreate = TRUE) only
  drops tables carrying unexpected GENERATED columns — a baked fresh.streams
  with correct DDL is kept, rows and all. What contains it is
  bucket-filtered consolidate (Step 0 pre-clean should be additive-by-default; --reset-schema opts into full wipe #180/schema_consolidate: enumerate wgc_tables per-source, not from destination #185), which copies only the bucket's
  WSGs. That holds, but it is the only mitigation. Root fix is
  NewGraphEnvironment/rtj#145 (fwa-dump-only image) — upstream, hours of
  Stage-1 work, not blocking once Phase 1 lands.
• link's 14 bcfishpass override CSVs are not a staleness axis: they ship in
  the package, follow the git branch (dispatcher pkgload, cyphers
  pak::local_install from the same pushed ref), last synced 2026-08-12
  (b139b49). fresh::crossings.csv is vestigial — crossings are built from DB
  primitives (.lnk_crossings_union), so crossings freshness is Phase 3.
• Installed at filing: link 0.45.1 (repo v0.46.0), fresh 0.33.0, crate 0.1.0.
• Relates to Pre-flight: sibling-host parity hook (link+fresh version+SHA) with auto-install option #183 (sibling-host parity hook — this absorbs it), Add surgical table-level drops to state_clean.sh (--tables=schema.table,...) #181 (surgical
  table drops in state_clean.sh), Cheap access-only post-consolidate recompute (bulletproof cross-WSG mapping_code parity, efficiently) #205 (cheap access-only recompute, in the
  run path), NewGraphEnvironment/rtj#145 (image hygiene).