Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ Experimental package — breaking all the time and loving the learning curve. St

**Key correction (do NOT re-rabbit-hole):** the pipeline builds `<schema>.crossings` **from DB primitives** (`lnk_pipeline_crossings` → `.lnk_crossings_union`: PSCIS + `fresh.modelled_stream_crossings` + CABD), **not** from `crossings.csv`. The CSV read at `lnk_pipeline_load.R:100` is **vestigial** — the union drops + rebuilds the table before break/classify/mapping_code touch it. So #231 ("consume weekly crossings.csv; repoint pipeline off fresh") was **closed as misdirected**; lessons in `planning/archive/2026-07-issue-231-crossings-from-primitives/README.md` (+ memory `crossings-built-from-primitives`). Freshness lever = `data-raw/snapshot_bcfp.sh` reloading the primitives into the **`fwapg`** DB (last load ~2026-05-26; `lnk_db_conn()` defaults to a `bcfishpass` DB that LACKS them). Opened **#232** — confirm link's built crossings ≈ bcfp's complete `crossings_vw` (the parity reference). Aside: `crossings.csv` was published to `s3://newgraph` (db_newgraph#15, smnorris PR #57) before we realized the models don't consume it — **db_newgraph#16** tracks reconsidering that dump.

## Status (2026-07-31) — v0.44.3 shipped (#233 config dictionaries + ownership boundary)

Both config CSVs now have data dictionaries: `configs/dictionary_dimensions.csv` (renamed from `dimensions_columns.csv`) and the new `configs/dictionary_parameters_fresh.csv` (19 rows — type, group, `owner`, `consumed_by`, default, description). **The point was not documentation, it was stopping the re-derivation:** the fresh↔link `parameters_fresh` column-ownership split had been settled long ago by [fresh#129](https://github.com/NewGraphEnvironment/fresh/issues/129) (fresh 0.12.7 *removed* `observation_*` — "fish passage interpretation belongs in link, not the network engine") but was only findable by archaeology through two repos' planning archives, so it kept getting re-worked from scratch. It is now the `owner` column — **14 fresh-owned engine params, 5 link-owned `observation_*`** — read by `audit_configs.R` §3b instead of a hardcoded `grepl("^observation_", ...)`, and written up in [`RUNBOOK.md`](RUNBOOK.md) §7 "Who owns which `parameters_fresh` column". Adding a link-owned column is now a dictionary edit, not a regex edit.

**Two findings from machine-verifying every `consumed_by` file:line (24/24) rather than inferring them:** link never reads the nine `cluster_*` columns at all — it only passes the frame through (`lnk_pipeline_connect.R:107`) to fresh's `.frs_run_connectivity()`; and **`rear_gradient_min` is read by no code in either package** (recorded as unused, not dropped — fresh owns that schema). **Gotcha worth knowing:** the bundles carry *different* column subsets — bcfishpass `dimensions.csv` has 30 columns to the three `default*` bundles' 32 — so any dictionary/coverage check must assert against the **union**, never a single bundle. Guarded in two layers because `data-raw/` is `.Rbuildignore`d and never runs for an installed package: `tests/testthat/test-dictionaries.R` (+23, the CI-side guard) and the audit's coverage / reverse-consistency / missing-dictionary flags (negative-tested — dropping one row exits 1). Also removed `audit_configs.R`'s hardcoded `setwd("/Users/airvine/...")`: the script now derives its repo root from its own location and resolves paths via `repo_path()`, so it runs from any cwd and mutates none. Open follow-ups unchanged: **#224**, **#225**, **#227** (the `public.wsg_outlet` builder — its absence is the one standing test failure, `test-lnk_wsg_resolve.R:138`).

## Status (2026-07-04) — v0.44.1 shipped (#226 vignette accessible_km)

Extended the PARS vignette with an **Accessible habitat (km)** section proving `accessible_km` bcfp-equivalence (link 6,822.5 vs bcfp 6,822.9 km BT, **−0.01%**; table from cached `inst/vignette-data/pars_accessible.rds`). **Gotcha that bit hard:** "regenerate the vignette artifacts" was NOT docs-only — the two persist configs drift in segmentation because only WSGs re-modelled post-#223 are dense. `fresh` (bcfp config) had PARS at 97,538 segs but `fresh_default` (default/grayling) was still pre-#223 (48,558); the gpkg's single `streams` layer joins `fresh` geometry to `fresh_default` `mapping_code_gr` on `id_segment`, so a naive regen attaches grayling tokens to mismatched geometry → corrupt GR map. Fix: re-model the lagging config (`data-raw/wsg_run_one.R` + `merge=TRUE` recompute via `wsg_recompute_one.R` for cross-WSG `;DAM`) so both share segmentation; `wsg_vignette_data.R` now carries a **segmentation-parity guard** that refuses a mixed build. Any cross-config artifact joined on `id_segment` must verify both sides share segmentation first. mapping_code parity refreshed 99.04%→98.91% (denser post-#223). Open follow-ups unchanged: **#224**, **#225**, **#227**.
Expand Down Expand Up @@ -253,7 +259,7 @@ To run the entire province: loop over watershed groups. Or pass any AOI with `sp
- #45 — Gradient classes cleanup (derive from `loaded$parameters_fresh$access_gradient_max`)
- #52 — Channel-class break positions vs gradient thresholds (research)
- #53 — Distribute tar_make across M4 + M1 + db_newgraph
- #75 — `dimensions_columns.csv` as source-of-truth: auto-gen README + `lnk_rules_build()` validation (CSV seeded in v0.17.0)
- #75 — `dictionary_dimensions.csv` as source-of-truth: auto-gen README + `lnk_rules_build()` validation (CSV seeded in v0.17.0)

## Recently closed

Expand Down
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: link
Title: Stream Network Habitat Interpretation (Experimental)
Version: 0.44.2
Version: 0.44.3
Date: 2026-07-06
Authors@R: c(
person("Allan", "Irvine", , "[email protected]",
Expand Down
4 changes: 4 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# link 0.44.3

Data dictionaries for both config CSVs, and an end to re-deriving the fresh↔link column-ownership boundary ([#233](https://github.com/NewGraphEnvironment/link/issues/233)). `inst/extdata/configs/dimensions_columns.csv` is renamed `dictionary_dimensions.csv` (pure rename — no code read it; the only live reference was the #75 entry in `CLAUDE.md`), establishing a `dictionary_<file>` convention for its new sibling `dictionary_parameters_fresh.csv`. The new file documents all 19 `parameters_fresh.csv` columns — type, group, `owner`, `consumed_by`, default-when-absent, description — where previously the 9 `cluster_*` columns were undocumented outside fresh's source and the rest lived in scattered prose. Every `consumed_by` is a machine-verified `file:line` (24/24 resolve), which corrected two things worth knowing: link does not read the `cluster_*` columns at all, it only passes the frame through (`R/lnk_pipeline_connect.R:107`) to fresh's `.frs_run_connectivity()`, and `rear_gradient_min` is read by **no code in either package** — recorded as unused rather than dropped, since fresh owns that schema. The `owner` column encodes, rather than restates, the partition settled by [fresh#129](https://github.com/NewGraphEnvironment/fresh/issues/129) (shipped fresh 0.12.7, which removed `observation_*` from fresh: "fish passage interpretation belongs in link, not the network engine"): 14 fresh-owned engine params, 5 link-owned `observation_*` interpretation params. `data-raw/audit_configs.R` §3b now reads that column instead of hardcoding `grepl("^observation_", ...)`, so a new link-owned column is documented once rather than taught to a regex, and gains three guards — undocumented-column coverage, a reverse check (dictionary claims link-owned but fresh ships it = stale dictionary), and a missing-dictionary guard; negative-tested by dropping a row, which exits 1. Because `data-raw/` is `.Rbuildignore`d and never runs for an installed package, the durable guard is the new `tests/testthat/test-dictionaries.R` (+23 tests), which checks coverage in both directions against the **union** of bundle columns — bundles legitimately carry different subsets, bcfishpass's `dimensions.csv` having 30 columns to the defaults' 32. `RUNBOOK.md` §7 gains the ownership section so the boundary is findable without archaeology through two repos' planning archives. Separately, `audit_configs.R` no longer `setwd()`s to a hardcoded `/Users/airvine/...` path — it derives the repo root from its own script location and resolves every path through a new `repo_path()` helper, so the audit runs from any working directory (verified from `/tmp`), works for anyone's checkout, and mutates no global state when `source()`d.

# link 0.44.2

Reader-facing prose cleanup on the PARS vignette (`vignettes/pars-habitat-connectivity.Rmd`), from review — no model, artifact, or numbers change. Dropped internal issue numbers and build-infrastructure ("tunnel") references, and cut the per-segment `mapping_code` disagreement detail (the top-diff-pattern columns and the "remaining disagreements" paragraph) since bull-trout parity is ~99%. The accessible / spawning / rearing km table stays but is reframed around `lnk_rollup_wsg()` as a standalone habitat-totals roll-up — link's own numbers, no reference required — that happens to match bcfishpass, rather than a proof-of-parity exercise. Removed the `mapping_code` symbology paragraph entirely: the colour semantics were easy to mis-state (e.g. red marks an accessible reach whose downstream PSCIS crossing is ranked a *barrier*, not merely "field-assessed"), and each map already carries its own colour legend.
Expand Down
36 changes: 33 additions & 3 deletions RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -417,9 +417,39 @@ direction. Not yet scoped; candidate issue.
| Habitat dimensions (spawn/rear by gradient, channel width, lake/stream, …) | `configs/<name>/dimensions.csv` → `lnk_rules_build()` → `rules.yaml` | `frs_habitat_classify()` (token1 habitat) |
| Species residence (resident/anadromous/spawn-only) | **hardcoded** defaults in `lnk_pipeline_mapping_code()` | which mc_barrier flavor + spawn-only token1 |
| Dam / anthropogenic blocking | **nowhere** — universal `all species` in `lnk_barriers_unify` | `blocks_species` (§2a). Not rules-driven. |

Two gaps worth knowing: **species residence** is hardcoded (data-drive is
follow-up #189), and **dam blocking is not rules-driven** at all (universal).
| What each config column means | `configs/dictionary_dimensions.csv`, `configs/dictionary_parameters_fresh.csv` | data dictionaries — per-column type, group, default, description, and (for `parameters_fresh`) `owner` + `consumed_by` file:line |

### Who owns which `parameters_fresh` column

**Do not re-derive this.** `parameters_fresh.csv` is co-owned, and the split is
settled:

- **fresh owns the 14 network-engine columns** — `species_code`,
`access_gradient_max`, the two `*_gradient_min`, and the nine `cluster_*`.
fresh ships them in its own `inst/extdata/parameters_fresh.csv`; link's
bundles are seeded from it.
- **link owns the 5 `observation_*` columns** — fish-passage interpretation
(counts, thresholds, date windows, species pooling, control veto).

The boundary was decided in
[fresh#129](https://github.com/NewGraphEnvironment/fresh/issues/129) (shipped
fresh 0.12.7), which *removed* `observation_*` from fresh after fresh#69 had
added them: "fish passage interpretation belongs in link, not the network
engine." Values may diverge freely per bundle — link tunes them — but the
**column set** is contractual.

Enforced in two places, both of which read `dictionary_parameters_fresh.csv`'s
`owner` column rather than hardcoding the rule:
`data-raw/audit_configs.R` §3b (pre-trifecta gate) and
`tests/testthat/test-dictionaries.R` (runs in CI, since `data-raw/` is
`.Rbuildignore`d). Direction of travel is opposite for the two shared artifacts:
`rules.yaml` flows **link → fresh** (link owns the generator, `lnk_rules_build()`);
the `parameters_fresh` column schema flows **fresh → link**.

Three gaps worth knowing: **species residence** is hardcoded (data-drive is
follow-up #189), **dam blocking is not rules-driven** at all (universal), and
`rear_gradient_min` is carried in the schema but **read by no code in either
package** — it is fresh-owned, so removing it is a fresh-side call.
If dam blocking should ever become species-specific, it's a new
per-source-per-species column + `lnk_barriers_unify` change — not a tweak.

Expand Down
Loading