34199 Addition of delta restore script for tracking table dump

[argush3]

Issue #: /bcgov/entity#34199

What changed

Replaces the old, unsupported DELTA_MODE=true ./restore_extract.sh prototype with a proper,
standalone data-tool/delta_restore_extract.sh that can merge a preserved-table dump into a
developer's existing local COLIN extract without truncating the local tracking/migration/auth
side-tables.

restore_extract.sh now aborts immediately if DELTA_MODE=true is passed, with a pointer to the
new script.

---

Key additions

Path	What
data-tool/delta_restore_extract.sh	New entry-point script — preview / apply / cleanup modes
data-tool/scripts/restore/delta/10_functions.sql	All PL/pgSQL classification & apply logic
data-tool/scripts/restore/preserved_tables.conf	Centralised list of 17 tracking/auth tables; shared by backup, full-restore, and delta scripts
data-tool/scripts/restore/repair_sequences.sql	Post-apply sequence repair
data-tool/scripts/restore/README_delta_restore.md	Full usage docs, class semantics, exit codes
data-tool/tests/delta_restore/	Shell/AWK static checks + Postgres-backed smoke tests
data-tool/Makefile	make test-delta-restore target

Modified:

• backup_extract_tables.sh — reads preserved_tables.conf (was hardcoded)
• restore_extract.sh — dead DELTA_MODE branch removed; now errors cleanly
• scripts/colin_corps_extract_postgres_ddl — minor column additions for tracking tables

---

Row classification (reviewer orientation)

Preview mode classifies every staged row before anything is written:

Class	Meaning	Applied by default?
NEW	No local match	✅
CHANGED	Natural-key match, payload differs (dump wins)	✅
CHANGED_LOCAL_NEWER	Local last_modified is newer	❌ (opt-in)
UNCHANGED	Identical	❌
BLOCKED_FK / BLOCKED_PARENT / AMBIGUOUS_NK	Cannot be safely applied	❌

---

Happy path — preview → apply

cd data-tool

# 1. Dump preserved tables from the source extract
PGDATABASE=source_extract BACKUP_DIR=/tmp/preserved ./backup_extract_tables.sh

# 2. Preview (no DB writes — default mode)
PGDATABASE=local_extract ./delta_restore_extract.sh \
  --dump /tmp/preserved/keep_2026-07-13.dump --mode preview

scripts/generated/delta_restore/<run>/preview.txt (truncated):

Delta restore preview
=====================
Dump:    /tmp/preserved/keep_2026-07-13.dump
Target:  localhost:5432/local_extract as postgres

Class counts
------------
table                                       NEW    CHANGED  UNCHANGED  CHANGED_LOCAL_NEWER  BLOCKED_FK
email_domain_groups                           0          0         24                    0           0
corp_processing                              12          4       3421                    1           0
colin_tracking                               89         11      18203                    0           0
auth_processing                               7          2        892                    0           0
auth_component_operation                     14          0        456                    0           0

Samples
-------
── corp_processing ──
CHANGED 4 · CHANGED_LOCAL_NEWER 1 · LOCAL_ONLY 2 · NEW 12 · UNCHANGED 3421

CHANGED — showing up to 20 of 4
▸ staged row 12 · staged id 881 · local id 881
    changed: last_filing_dt, status

Selection: edit selection.conf or pass apply selection flags, then run --mode apply.

selection.conf (edit before applying — e.g. exclude a table or opt-in changed_local_newer):

# classes: new | changed | changed_local_newer   (others are never applyable)
[*]                        include=new,changed
[corp_processing]          include=new,changed         # new=12 changed=4 changed_local_newer=1
[colin_tracking]           include=new,changed         # new=89 changed=11 changed_local_newer=0
[auth_processing]          include=new,changed         # new=7  changed=2  changed_local_newer=0
[auth_component_operation] include=new,changed         # new=14 changed=0  changed_local_newer=0
[email_domain_groups]      include=                    # ← exclude entirely (set include= blank)

# 3. Apply with the (optionally edited) selection file
PGDATABASE=local_extract ./delta_restore_extract.sh \
  --dump /tmp/preserved/keep_2026-07-13.dump --mode apply \
  --selection-file scripts/generated/delta_restore/<run>/selection.conf --yes

apply_summary.txt:

Delta restore apply summary
===========================
Dump:   /tmp/preserved/keep_2026-07-13.dump
Target: localhost:5432/local_extract as postgres

Selected counts
---------------
table                                class                  selected
corp_processing                      CHANGED                       4
corp_processing                      NEW                          12
colin_tracking                       CHANGED                      11
colin_tracking                       NEW                          89
auth_processing                      CHANGED                       2
auth_processing                      NEW                           7
auth_component_operation             NEW                          14

Affected counts
---------------
table                                class       action   expected   affected
corp_processing                      CHANGED     UPDATE          4          4
corp_processing                      NEW         INSERT         12         12
colin_tracking                       CHANGED     UPDATE         11         11
colin_tracking                       NEW         INSERT         89         89
auth_processing                      CHANGED     UPDATE          2          2
auth_processing                      NEW         INSERT          7          7
auth_component_operation             NEW         INSERT         14         14

# Cleanup delta schemas if a run was interrupted
PGDATABASE=local_extract ./delta_restore_extract.sh --cleanup

Run tests

make -C data-tool test-delta-restore

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the lear license (Apache 2.0).

[sonarqubecloud]

Quality Gate passed

Issues
196 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.7% Duplication on New Code

See analysis details on SonarQube Cloud

[ketaki-deodhar]

sonarqube shows 196 new issues. Not entirely sure if too critical to resolve but just noticed it.

[argush3]

sonarqube shows 196 new issues. Not entirely sure if too critical to resolve but just noticed it.

i'll see if i can knock some of these down