Release v0.47.2: stop shipping internal dirs; record the cypher pilots

DESCRIPTION

@@ -1,6 +1,6 @@
 Package: link
 Title: Stream Network Habitat Interpretation (Experimental)
-Version: 0.47.1
+Version: 0.47.2
 Date: 2026-08-31
 Authors@R: c(
     person("Allan", "Irvine", , "[email protected]",

NEWS.md

@@ -1,3 +1,11 @@
+# link 0.47.2
+
+Stops shipping `comms/` and `research/` in the built package ([#235](https://github.com/NewGraphEnvironment/link/pull/235)). `R CMD build` includes every top-level directory not named in `.Rbuildignore`, so `pak::pak("NewGraphEnvironment/link")` was installing 15 files of cross-repo coordination notes and 22 of research working files into the user's library. `R CMD check` reports this only as a NOTE and `.gitignore` does not cover it. Verified against the built tarball rather than the config, because the regex is easy to get subtly wrong: both directories go 15/22 → 0. The gap opens over time rather than at scaffold — `planning`, `dev`, `.claude` and `CLAUDE.md` were already excluded, and these two were added to the repo later.
+
+Also packs distributed work by **finish time rather than segment count** ([#253](https://github.com/NewGraphEnvironment/link/pull/253)). The hosts are not interchangeable: measured on identical work, the dispatcher runs at 0.0391 min per 1000 persisted segments and a cypher at 0.0872 — **2.23× slower**. Balancing raw segments therefore balances *work* and unbalances *time*, and the makespan ends up set by a host that was given less to do; on the provincial set the dispatcher finished at 95 min while every cypher ran to ~190. Assigning each component to the host that would complete it earliest gives 149/149/150/150, cutting the provincial modelling phase by 41 minutes (21%) and taking the end-to-end estimate from 5.0 h to 4.3 h. `--host-speeds=` overrides the measured default. The relabelling step is gone rather than adapted — it packed into anonymous equal-speed bins and renamed them by load, which is meaningless once host 1 *is* the dispatcher and cannot be swapped for a cypher. Drainage-closure and downstream-first assertions were re-verified against the new assignment by restoring the bug, not inherited from the old one.
+
+Adds `research/run_record_2026_08_31_cypher_pilots.md` and `data-raw/logs/README.md` — the first record under [soul#129](https://github.com/NewGraphEnvironment/soul/issues/129), covering the four cypher pilots that produced the rates above and the four defects they exposed, including the ones where a correct diagnosis was retracted on a bad inference and where two gates passed tests that could not fail.
+
 # link 0.47.1
 
 Fixes two defects in the v0.47.0 pre-flight gates, both found by piloting them against a real cypher for the first time ([#246](https://github.com/NewGraphEnvironment/link/issues/246)). Both would have blocked the provincial run, and neither could have been caught by the tests as written.

data-raw/logs/README.md

@@ -1,55 +1,45 @@
-# `data-raw/logs/`
-
-Run artifacts from pipeline drivers (`compare_bcfishpass_wsg.R`, `run_provincial_parity.R`, the trifecta scripts) plus operational outputs (pg_dumps, methodology-delta queries, regression logs).
-
-## Subdirectories
-
-Per-run output is keyed by topic. Each subdir holds `<WSG>.rds` per-WSG rollup tibbles plus `<TS>_per_wsg_times.csv` host-tagged timing rows.
-
-| Subdir | Source script | Contents |
-|--------|---------------|----------|
-| `provincial_parity/` | `run_provincial_parity.R --config=bcfishpass` | bcfishpass-bundle rollups (link vs bcfp tunnel) |
-| `provincial_default/` | `run_provincial_parity.R --config=default` | default-bundle rollups |
-| `provincial_default_extrabreaks/` | `run_provincial_parity.R --config=default_extrabreaks` | orphan-class break-source experiment (v0.28.0) |
-| `methodology_delta/` | `query_schema_delta.R` | schema-vs-schema delta RDS snapshots |
-| `dumps_<schema>/` | `consolidate_schema.R` (manual) | pg_dump custom-format files for cross-host consolidation |
-| `baseline_pre_*/` | hand-archived | Pre-change baselines kept for regression diffs |
-
-## Top-level files
-
-### `bcfp_baselines.csv` — bcfp build inventory per run
-
-Records which `bcfishpass.*` schema rebuild each provincial run was compared against. Critical for paper trail because:
-
-- The tunnel's `bcfishpass.*` schema rebuilds **weekly Tuesdays ~20:00 PDT** via `smnorris/db_newgraph`'s scheduled GHA workflow.
-- Today's rollups in `provincial_*/` carry `bcfishpass_value` columns sourced from whichever build was live at the moment of comparison.
-- Without recording the build, tomorrow's same-config rerun produces shifts that look like methodology change but are actually upstream-rebuild change (`bcfishpass.streams_habitat_*` repopulated from new code / new input data).
-
-Columns:
-
-- `run_started_pdt` — local time the provincial dispatch fired
-- `run_label` — directory name where rollup RDS files landed
-- `link_schema` — persistent target schema for `lnk_pipeline_persist`
-- `bcfp_model_run_id` — primary key from `bcfishpass.log`
-- `bcfp_model_version` — `<tag>-<commits>-g<short-sha>` string
-- `bcfp_date_completed` — when Simon's rebuild finished
-- `notes` — anything else (orphan-branch experiments, partial reruns, etc.)
-
-How to query the current bcfp baseline (run before any provincial dispatch):
-
-```sql
--- localhost:63333 / dbname=bcfishpass / user=newgraph / password=PG_PASS_SHARE
-SELECT model_run_id, date_completed, model_version
-FROM bcfishpass.log
-ORDER BY model_run_id DESC LIMIT 1;
+# Run logs
+
+**These are retained deliberately as contemporaneous evidence of measurement
+runs, not accumulated by accident.** They are tracked in git, not gitignored,
+and should stay that way — see NewGraphEnvironment/soul#129.
+
+Every number quoted in a `research/` run record, a NEWS entry or a PR body
+should be traceable to a file here.
+
+## What is here
+
+| pattern | produced by | holds |
+|---|---|---|
+| `study_area_run/<TS>_up_<ws>.log` | `cypher_up.sh` via `study_area_run.sh` | droplet spin, tofu apply, cloud-init wait |
+| `study_area_run/<TS>_prep_<ws>.log` | `cypher_prep.sh` | git reset, package install, snapshot, persist_init |
+| `study_area_run/<TS>_stamps.tsv` | `host_stamp.R` | one provenance line per host — the parity gate's input |
+| `study_area_run/<TS>_vintage.log` | `host_vintage.R` | primitive freshness per host |
+| `study_area_run/<TS>_run_{local,<ws>}.log` | `wsg_run_one.R` | per-WSG modelling, `done in N min` |
+| `study_area_run/<TS>_consolidate.log` | `schema_consolidate.R` | cross-host COPY |
+| `study_area_run/<TS>_recompute.log` | `wsg_recompute_one.R` | post-consolidate access rebuild |
+| `study_area_run/<TS>_compare.{log,csv}` | `study_area_compare.R` | bcfishpass parity |
+| `study_area_run/<TS>_burn_<ws>.log` | `cypher_down.sh` | teardown + verification |
+| `bcfp_baselines.csv` | `snapshot_bcfp.sh` | which upstream bcfp build each host loaded |
+| `provincial_*/`, `methodology_delta/` | earlier orchestrators | historical runs |
+
+`<TS>` is UTC `YYYYMMDD_HHMMSS` and is shared by every file from one run, so a
+single run's artifacts sort together.
+
+## Reading a run
+
+Phase durations are not logged as such — reconstruct them from file mtimes,
+which is how the timings in `research/` were derived:
+
+```bash
+python3 -c "
+import glob,os,datetime
+fs=sorted(glob.glob('data-raw/logs/study_area_run/<TS>_*'),key=os.path.getmtime)
+t0=os.path.getmtime(fs[0])
+for f in fs: print('%6.1f min  %s' % ((os.path.getmtime(f)-t0)/60, os.path.basename(f)))"
 ```
 
-### Future automation
-
-The csv-sync rewrite ([link#117](https://github.com/NewGraphEnvironment/link/issues/117)) will append to this CSV at sync time, recording which bcfp build the bundle CSVs are now SHA-pinned to. That closes the loop: every comparison rollup has both the bcfp build AND the matching bundle CSV state on file.
-
-Until then, manually append a row at the start of each provincial run.
-
-## Naming convention for log files
-
-Run logs follow `<TS>_<topic>_<host>.txt` where `<TS>` is `YYYYMMDDHHMM`. See `data-raw/README.md` (parent) for the broader conventions.
+Per-WSG runtimes are better read from the run log itself
+(`grep 'done in' <TS>_run_*.log`) or, for any run after v0.45.0, from
+`<persist>.log` in Postgres, which records `date_start` / `date_end` per WSG
+alongside the software SHAs.

data-raw/logs/study_area_run/20260831_190558_burn_job1.log

@@ -0,0 +1,140 @@
+[cypher_down] destroying workspace 'job1' (no reserved IP)
+module.droplet.digitalocean_droplet.this: Refreshing state... [id=596689967]
+module.droplet.digitalocean_firewall.this[0]: Refreshing state... [id=2b99f569-a20d-49bf-a516-a39d795c2779]
+
+OpenTofu used the selected providers to generate the following execution
+plan. Resource actions are indicated with the following symbols:
+  - destroy
+
+OpenTofu will perform the following actions:
+
+  # module.droplet.digitalocean_droplet.this will be destroyed
+  - resource "digitalocean_droplet" "this" {
+      - backups              = false -> null
+      - created_at           = "2026-08-31T19:06:08Z" -> null
+      - disk                 = 400 -> null
+      - graceful_shutdown    = false -> null
+      - id                   = "596689967" -> null
+      - image                = "228350154" -> null
+      - ipv4_address         = "147.182.228.14" -> null
+      - ipv4_address_private = "10.124.0.2" -> null
+      - ipv6                 = false -> null
+      - locked               = false -> null
+      - memory               = 32768 -> null
+      - monitoring           = false -> null
+      - name                 = "cypher-job1" -> null
+      - price_hourly         = 0.25 -> null
+      - price_monthly        = 168 -> null
+      - private_networking   = true -> null
+      - public_networking    = true -> null
+      - region               = "sfo3" -> null
+      - resize_disk          = true -> null
+      - size                 = "s-8vcpu-32gb-amd" -> null
+      - ssh_keys             = [
+          - "53795424",
+          - "56433424",
+        ] -> null
+      - status               = "active" -> null
+      - tags                 = [
+          - "compute",
+          - "cypher",
+          - "dev",
+          - "job1",
+        ] -> null
+      - urn                  = "do:droplet:596689967" -> null
+      - user_data            = (sensitive value) -> null
+      - vcpus                = 8 -> null
+      - volume_ids           = [] -> null
+      - vpc_uuid             = "0503d836-77ab-45e3-a50f-52542155755f" -> null
+    }
+
+  # module.droplet.digitalocean_firewall.this[0] will be destroyed
+  - resource "digitalocean_firewall" "this" {
+      - created_at      = "2026-08-31T19:08:27Z" -> null
+      - droplet_ids     = [
+          - 596689967,
+        ] -> null
+      - id              = "2b99f569-a20d-49bf-a516-a39d795c2779" -> null
+      - name            = "cypher-job1-fw" -> null
+      - pending_changes = [] -> null
+      - status          = "succeeded" -> null
+      - tags            = [] -> null
+
+      - inbound_rule {
+          - port_range                = "22" -> null
+          - protocol                  = "tcp" -> null
+          - source_addresses          = [
+              - "0.0.0.0/0",
+            ] -> null
+          - source_droplet_ids        = [] -> null
+          - source_kubernetes_ids     = [] -> null
+          - source_load_balancer_uids = [] -> null
+          - source_tags               = [] -> null
+        }
+      - inbound_rule {
+          - port_range                = "41641" -> null
+          - protocol                  = "udp" -> null
+          - source_addresses          = [
+              - "0.0.0.0/0",
+              - "::/0",
+            ] -> null
+          - source_droplet_ids        = [] -> null
+          - source_kubernetes_ids     = [] -> null
+          - source_load_balancer_uids = [] -> null
+          - source_tags               = [] -> null
+        }
+
+      - outbound_rule {
+          - destination_addresses          = [
+              - "0.0.0.0/0",
+              - "::/0",
+            ] -> null
+          - destination_droplet_ids        = [] -> null
+          - destination_kubernetes_ids     = [] -> null
+          - destination_load_balancer_uids = [] -> null
+          - destination_tags               = [] -> null
+          - protocol                       = "icmp" -> null
+        }
+      - outbound_rule {
+          - destination_addresses          = [
+              - "0.0.0.0/0",
+              - "::/0",
+            ] -> null
+          - destination_droplet_ids        = [] -> null
+          - destination_kubernetes_ids     = [] -> null
+          - destination_load_balancer_uids = [] -> null
+          - destination_tags               = [] -> null
+          - port_range                     = "1-65535" -> null
+          - protocol                       = "tcp" -> null
+        }
+      - outbound_rule {
+          - destination_addresses          = [
+              - "0.0.0.0/0",
+              - "::/0",
+            ] -> null
+          - destination_droplet_ids        = [] -> null
+          - destination_kubernetes_ids     = [] -> null
+          - destination_load_balancer_uids = [] -> null
+          - destination_tags               = [] -> null
+          - port_range                     = "1-65535" -> null
+          - protocol                       = "udp" -> null
+        }
+    }
+
+Plan: 0 to add, 0 to change, 2 to destroy.
+
+Changes to Outputs:
+  - droplet_id   = "596689967" -> null
+  - droplet_ip   = "147.182.228.14" -> null
+  - tailnet_name = "cypher-job1" -> null
+module.droplet.digitalocean_firewall.this[0]: Destroying... [id=2b99f569-a20d-49bf-a516-a39d795c2779]
+module.droplet.digitalocean_firewall.this[0]: Destruction complete after 0s
+module.droplet.digitalocean_droplet.this: Destroying... [id=596689967]
+module.droplet.digitalocean_droplet.this: Still destroying... [id=596689967, 10s elapsed]
+module.droplet.digitalocean_droplet.this: Still destroying... [id=596689967, 20s elapsed]
+module.droplet.digitalocean_droplet.this: Destruction complete after 22s
+
+Destroy complete! Resources: 2 destroyed.
+
+[cypher_down] complete. Workspace 'job1' state empty (still listed).
+Delete the workspace with: cd /Users/airvine/Projects/repo/rtj/env/do/dev/cypher && tofu workspace select default && tofu workspace delete job1

data-raw/logs/study_area_run/20260831_190558_compare.csv

@@ -0,0 +1,12 @@
+"wsg","species","total_segs","match_pct","n_diffs","top_pattern","top_pattern_count"
+"COWN","BT",19564,99.25,146,"REAR;MODELLED | ACCESS;MODELLED",32
+"COWN","CH",19564,99.74,51,"ACCESS;NONE | SPAWN;NONE",38
+"COWN","CM",19564,99.06,183,"ACCESS;NONE | SPAWN;NONE",169
+"COWN","CO",19564,99.34,129,"ACCESS;NONE | SPAWN;NONE",40
+"COWN","PK",19564,99.95,9,"ACCESS;MODELLED;INTERMITTENT | ",3
+"COWN","ST",19564,99.83,33,"ACCESS;NONE | SPAWN;NONE",10
+"TSIT","CH",15898,99.94,9,"ACCESS;MODELLED | ACCESS;REMEDIATED",2
+"TSIT","CM",15898,99.96,6,"SPAWN | ",3
+"TSIT","CO",15898,99.95,8,"ACCESS;MODELLED | ACCESS;REMEDIATED",2
+"TSIT","PK",15898,99.94,9,"ACCESS;NONE | SPAWN;NONE",3
+"TSIT","ST",15898,99.76,38,"REAR;MODELLED | ACCESS;MODELLED",7

data-raw/logs/study_area_run/20260831_190558_compare.log

@@ -0,0 +1,15 @@
+[study_area_compare] 11 rows across 2/2 WSGs -> /Users/airvine/Projects/repo/link/data-raw/logs/study_area_run/20260831_190558_compare.csv
+# A tibble: 11 × 7
+   wsg   species total_segs match_pct n_diffs top_pattern      top_pattern_count
+ * <chr> <chr>        <int>     <dbl>   <int> <chr>                        <int>
+ 1 COWN  BT           19564      99.2     146 "REAR;MODELLED …                32
+ 2 COWN  CH           19564      99.7      51 "ACCESS;NONE | …                38
+ 3 COWN  CM           19564      99.1     183 "ACCESS;NONE | …               169
+ 4 COWN  CO           19564      99.3     129 "ACCESS;NONE | …                40
+ 5 COWN  PK           19564     100.0       9 "ACCESS;MODELLE…                 3
+ 6 COWN  ST           19564      99.8      33 "ACCESS;NONE | …                10
+ 7 TSIT  CH           15898      99.9       9 "ACCESS;MODELLE…                 2
+ 8 TSIT  CM           15898     100.0       6 "SPAWN | "                       3
+ 9 TSIT  CO           15898     100.0       8 "ACCESS;MODELLE…                 2
+10 TSIT  PK           15898      99.9       9 "ACCESS;NONE | …                 3
+11 TSIT  ST           15898      99.8      38 "REAR;MODELLED …                 7