feat: safe database upgrades from any release (PRD-016 Phase 0)

.github/workflows/ci.yml

@@ -91,6 +91,50 @@ jobs:
       - name: The workflow files themselves (actionlint)
         run: go run github.com/rhysd/actionlint/cmd/[email protected] -shellcheck= .github/workflows/*.yml
 
+  upgrade:
+    # PRD-016: the latest release, built from its tag, and this commit on the same database. This commit
+    # upgrades what the release wrote, and the release then refuses the upgraded file instead of misreading
+    # it. Upgrades from every release's real data are TestUpgradeFromEveryRelease, in the test job.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+      - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
+        with:
+          go-version-file: go.mod
+          check-latest: true
+      - name: Build this commit and the latest release
+        run: |
+          # the newest release in this commit's history, other than this commit itself (a release run)
+          here="$(git tag --points-at HEAD | head -1)"
+          prev="$(git tag -l 'v*' --sort=-v:refname --merged HEAD | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | grep -vxF -e "${here:-none}" | head -1)"
+          test -n "$prev" || { echo "no earlier release tag"; exit 1; }
+          echo "latest release: $prev"
+          CGO_ENABLED=0 go build -o /tmp/ch-new ./cmd/cipherhorizon
+          git worktree add -q --detach /tmp/prev-src "$prev"
+          (cd /tmp/prev-src && CGO_ENABLED=0 go build -o /tmp/ch-old ./cmd/cipherhorizon)
+      - name: Upgrade a database the release made, then hand it back to the release
+        run: |
+          set -x
+          /tmp/ch-old db stats --db /tmp/u.db >/dev/null       # the release creates its database
+          before="$(/tmp/ch-new db status --json --db /tmp/u.db | jq '.pending | length')"
+          /tmp/ch-new db migrate --db /tmp/u.db
+          /tmp/ch-new db check --db /tmp/u.db
+          if [ "$before" -gt 0 ]; then
+            if /tmp/ch-old db stats --db /tmp/u.db > /tmp/old.out 2>&1; then
+              echo "the release opened a database this commit upgraded"; cat /tmp/old.out; exit 1
+            fi
+            grep -q "newer than this program" /tmp/old.out || { cat /tmp/old.out; exit 1; }
+            # and the backup taken before the upgrade gets it back
+            bak="$(/tmp/ch-new db status --json --db /tmp/u.db | jq -r '.backups[0].path')"
+            /tmp/ch-new db restore --db /tmp/u.db "$bak"
+            /tmp/ch-old db stats --db /tmp/u.db >/dev/null
+          else
+            /tmp/ch-old db stats --db /tmp/u.db >/dev/null       # same schema: both read it
+          fi
+
   vuln:
     # Known vulnerabilities in the dependencies and the standard library, limited to code that is reachable.
     runs-on: ubuntu-latest

.gitignore

@@ -24,6 +24,9 @@ Thumbs.db
 *.db
 *.db-shm
 *.db-wal
+*.db.lock
+# except the databases released versions wrote, kept for the upgrade tests (PRD-016 U-02)
+!/internal/store/testdata/upgrade/v*.db
 
 # release build output and local release scratch
 /RELEASE_NOTES.md

CHANGELOG.md

@@ -10,6 +10,32 @@ the CBOM/report/eval schema versions, the database) can change in a minor releas
 
 ## [Unreleased]
 
+### Added
+
+- **Safer upgrades** ([PRD-016](docs/product/PRD-016-upgrades-and-updates.md) Phase 0). The server still upgrades
+  its database at launch, from any earlier version, after a backup; now it also:
+  - refuses to run two servers on one database, and refuses to upgrade a database another CipherHorizon process
+    is using (a `<database>.lock` file next to it; the OS releases the lock if the process dies);
+  - checks there is room for the backup first, and changes nothing if there is not;
+  - checks the database after upgrading it, and if a step or the check fails, puts the file back exactly as it
+    was, keeps the failed copy for diagnosis, and refuses to start;
+  - records each upgrade (from which schema to which, by which version, and the backup) in the database.
+- `cipherhorizon db status` (schema, what an upgrade would do, backups, upgrade history, whether a server is
+  using it), `db check` (SQLite integrity check, references, structure; exit 2 on a problem), `db migrate`
+  (`--dry-run` to rehearse) and `db restore BACKUP-FILE` (the server must be stopped; the replaced database
+  is kept beside it).
+- `serve --migrate refuse`: for change-controlled sites, stop instead of upgrading an older database, so the
+  upgrade is run as its own step with `db migrate`.
+
+### Upgrade notes
+
+- Schema 15 → 16: adds the `upgrade_log` table. Nothing existing is changed; the upgrade takes under a second.
+- From this version on, a second `serve` on the same database exits with a message naming the process using
+  it. If you ran two servers on one file before, give each its own `--db`.
+- A database upgraded by this version is refused by 0.4.0 and earlier ("newer than this program
+  understands"). To go back, restore the backup made before the upgrade with `cipherhorizon db restore`
+  (this version) and then run the older one.
+
 ## [0.4.0] - 2026-10-22
 
 ### Security

CONTRIBUTING.md

@@ -47,6 +47,31 @@ rest. `go vet ./...` and `gofmt -l .` (expect no output) should be clean before
    `go test ./internal/app -update` and read the diff.
 5. **Run the checks** in "Development setup" above.
 
+## Changing the database
+
+Every install upgrades its own database when a new version starts, from whatever version it ran before,
+however many releases it skipped. That only works if these rules hold (the tests enforce most of them; see
+[PRD-016](docs/product/PRD-016-upgrades-and-updates.md) §5.2):
+
+- **Add a new step to `migrations` in `internal/store/store.go`; never edit, reorder or remove one that has
+  shipped.** Shipped steps are listed in `internal/store/migrations.lock` (`TestMigrationsLock`). Fix a
+  shipped mistake with a new step.
+- **Additive in a minor release:** new tables, nullable columns or columns with a default, new indexes.
+  Dropping or renaming a column, or changing what one means, goes in two releases: add the new one and write
+  both; then, a release later, stop using and drop the old one.
+- **A step's Go back-fill (`postMigrations`) uses only frozen code:** SQL, the standard library and helpers
+  named `…V<step>` that are copied for it. No `Store` methods, no other package of this module, no shared
+  package variables (`TestPostMigrationsUseOnlyFrozenCode`). `recordConfirmedLinkV6` is an example.
+- **Stored evidence is never rewritten.** Scan reports (`report_json`) and host inventories stay as observed;
+  a new report shape gets a new `schemaVersion` and readers keep reading the old ones.
+- **Regenerate the schema description** and review its diff:
+  `go test ./internal/store -run TestSchemaFingerprint -update-schema` (writes `schema_fingerprint.go` and
+  `internal/store/testdata/schema.txt`).
+- **Write upgrade notes:** while a step is unreleased, `CHANGELOG.md` `[Unreleased]` needs a
+  `### Upgrade notes` section saying what the upgrade does and roughly how long it takes on a large database.
+- `TestUpgradeFromEveryRelease` upgrades a database written by every past release (`internal/store/testdata/upgrade`)
+  and checks nothing was lost and the result matches a fresh install.
+
 ## Evidence and citations
 
 This is the one rule the codebase enforces in tests, not just in review: **every PQC verdict, every TLS

docs/deploy.md

@@ -89,11 +89,13 @@ docker compose exec cipherhorizon /cipherhorizon db backup --db /data/cipherhori
 ```
 
 A backup is a consistent copy taken while the server runs, readable by its owner only. It holds
-password and session hashes: protect it like the database. To restore, stop the server, put the copy
-in place of the database file and start it again (`serve --db` also opens a copy directly).
+password and session hashes: protect it like the database. To restore, stop the server, run
+`cipherhorizon db restore --db FILE BACKUP` (it keeps the database it replaces) and start it again
+(`serve --db` also opens a copy directly).
 
-Upgrades migrate the database on start, after keeping a copy next to it; the newest three such
-copies are kept.
+Upgrades migrate the database on start, after keeping a copy next to it (the newest three are kept); a failed
+upgrade puts the file back as it was. `cipherhorizon db status` shows the schema, the backups and the upgrade
+history. Details: [INSTALL.md](engineering/INSTALL.md#upgrading).
 
 ## Checking the inventory scripts
 

docs/engineering/INSTALL.md

@@ -184,13 +184,17 @@ docker exec cipherhorizon /cipherhorizon db backup --db /data/cipherhorizon.db /
 
 On Windows (administrator): `cipherhorizon db backup --db C:\ProgramData\CipherHorizon\cipherhorizon.db D:\backup\ch.db`.
 
-To restore:
-1. Stop the server.
-2. Put the backup in place of `cipherhorizon.db`.
-3. Delete the stale `cipherhorizon.db-wal` and `cipherhorizon.db-shm` files next to it.
-4. Start the server.
+To restore, stop the server, then:
 
-A backup opens like any database (`cipherhorizon serve --db backup.db`), so you can check it first.
+```sh
+cipherhorizon db restore --db /var/lib/cipherhorizon/cipherhorizon.db /root/cipherhorizon-2026-10-24.db
+docker run --rm -v cipherhorizon-data:/data ghcr.io/offsetsecurity/cipherhorizon:<version> db restore --db /data/cipherhorizon.db /data/backup.db
+```
+
+It checks the backup is a sound CipherHorizon database first, refuses while a server is using the database,
+and keeps the database it replaces beside it (`cipherhorizon.db.replaced-<time>`, with its `-wal` and `-shm`)
+rather than deleting it. Then start the server. `cipherhorizon db check --db FILE` examines any database,
+backup or live, without changing it.
 
 ## Upgrading
 
@@ -201,9 +205,36 @@ A backup opens like any database (`cipherhorizon serve --db backup.db`), so you
 | container | pulling the new tag, then recreating the container; the volume keeps the data |
 | binary | replacing the file |
 
-On start, the server migrates the database forward. Before each migration it saves a consistent copy
-next to the file, `cipherhorizon.db.bak-v<schema>-<time>`, and keeps the newest three. 0.x releases do
-not promise a downgrade path: to go back, restore that copy (as above) and run the older version.
+You can skip versions: any release upgrades straight from any earlier one. Every release is tested
+upgrading databases written by each earlier release.
+
+**What happens at start.** The new version upgrades the database itself, the first time it opens it (as
+`serve`, or any `db` or `user` command):
+
+1. It takes the database's lock (`cipherhorizon.db.lock`), so it never changes a file another CipherHorizon
+   process is using. A second server on the same database is refused, naming the one that holds it. Stop the
+   old version before starting the new one; Compose, Quadlet and the packages already do.
+2. It checks there is room for a backup, then saves a consistent copy next to the file,
+   `cipherhorizon.db.bak-v<schema>-<time>`, keeping the newest three.
+3. It applies each change the database lacks, in order, then checks the result.
+4. If anything fails, it puts the file back exactly as it was, keeps the failed copy as
+   `cipherhorizon.db.failed-upgrade-<time>` and refuses to start, saying why. Your data is never left
+   half-upgraded.
+
+**Seeing it before it happens.**
+
+```sh
+cipherhorizon db status --db FILE           # schema, what this version would do, backups, history
+cipherhorizon db migrate --dry-run --db FILE
+```
+
+Run the new version's binary against the file before replacing the old one, or in a throwaway container with the
+volume mounted. **Change-controlled sites** can start the server with `--migrate refuse`: it then stops instead of
+upgrading, and the upgrade is its own step, `cipherhorizon db migrate --db FILE`.
+
+**Going back.** An older version refuses a database a newer one has upgraded. To go back, stop the server,
+restore the backup made before the upgrade (`cipherhorizon db status` lists them) with `cipherhorizon db restore`,
+and run the older version. Scans made since the upgrade are not in that backup.
 
 ## Removing it
 

docs/engineering/RELEASING.md

@@ -77,11 +77,13 @@ Prerequisites: you are a maintainer; `main` is green; Actions are enabled; you h
 2. **Prepare the changelog** on a branch:
    ```sh
    git switch main && git pull && git switch -c chore/release-0.1.0
-   scripts/prepare-release.sh 0.1.0        # turns [Unreleased] into [0.1.0] - <today>, fixes compare links
+   scripts/prepare-release.sh 0.1.0        # turns [Unreleased] into [0.1.0] - <today>, fixes compare links,
+                                           # and locks the database migrations it ships (migrations.lock)
    ```
    Read the section critically: users read it, and it becomes the release notes. Every user-visible change,
-   every changed default, every new flag; **Security** and **Breaking** items first. Also confirm `SECURITY.md`
-   "Supported versions" is right.
+   every changed default, every new flag; **Security** and **Breaking** items first. If the release changes
+   the database, its **Upgrade notes** say what the upgrade does (CI refuses unreleased migrations without
+   them). Also confirm `SECURITY.md` "Supported versions" is right.
 3. **Open a PR** titled `chore(release): 0.1.0`. CI runs (including the GoReleaser and container dry run).
    Merge (squash).
 4. **Rehearse locally** (optional, needs `goreleaser`): `make snapshot` builds all six archives into `dist/`
@@ -105,6 +107,14 @@ Prerequisites: you are a maintainer; `main` is green; Actions are enabled; you h
 7. **Verify** the published release exactly as a user would (commands in [DISTRIBUTION.md](DISTRIBUTION.md#verifying-a-download)),
    and start the binary: `cipherhorizon version` shows the tag, the commit, and the build date.
 8. **Announce**: nothing to do for 0.1.0 beyond the release page; for later releases, link the changelog.
+9. **Record the release's database for the upgrade tests** (PRD-016 U-02). From `main`, after the tag exists:
+   ```sh
+   git fetch --tags && git switch -c test/upgrade-fixture-0.5.0
+   testing/upgrade/make-fixture.sh v0.5.0   # builds the tag, seeds it through its own API, writes
+                                            # internal/store/testdata/upgrade/v0.5.0.db
+   ```
+   Commit the file and open a PR. From then on every change is tested against what installs of this version
+   hold (`TestUpgradeFromEveryRelease`). A release that shipped a migration *must* have one; CI says so.
 
 ### If the workflow fails
 
@@ -154,12 +164,14 @@ patch, and if it is a vulnerability publish a GitHub Security Advisory.
 - [ ] `main` is green, no open PR I intend to include is waiting
 - [ ] Version chosen by the rules above: `___`
 - [ ] `scripts/prepare-release.sh` run; CHANGELOG section read and complete, Breaking/Security first
+- [ ] Database changed? Upgrade notes present; `migrations.lock` updated by the script and committed
 - [ ] `SECURITY.md` supported-versions text is current
 - [ ] README install instructions match reality for this version
 - [ ] Merged; tag created on the merge commit and pushed
 - [ ] `release.yml` green through `publish`
 - [ ] Downloaded an archive from the release page, verified checksum + signature + attestation, ran `version`
 - [ ] Pulled the container image, verified the signature, ran it
+- [ ] Upgrade fixture made with `testing/upgrade/make-fixture.sh` and its PR opened
 - [ ] Any follow-up issues opened for things found while releasing
 
 ## What can go wrong here, and why the pipeline is shaped this way