diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b5ede6b..81022a9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -91,6 +91,50 @@ jobs: - name: The workflow files themselves (actionlint) run: go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 -shellcheck= .github/workflows/*.yml + upgrade: + # PRD-016: the latest release, built from its tag, and this commit on the same database. This commit + # upgrades what the release wrote, and the release then refuses the upgraded file instead of misreading + # it. Upgrades from every release's real data are TestUpgradeFromEveryRelease, in the test job. + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + fetch-depth: 0 + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version-file: go.mod + check-latest: true + - name: Build this commit and the latest release + run: | + # the newest release in this commit's history, other than this commit itself (a release run) + here="$(git tag --points-at HEAD | head -1)" + prev="$(git tag -l 'v*' --sort=-v:refname --merged HEAD | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | grep -vxF -e "${here:-none}" | head -1)" + test -n "$prev" || { echo "no earlier release tag"; exit 1; } + echo "latest release: $prev" + CGO_ENABLED=0 go build -o /tmp/ch-new ./cmd/cipherhorizon + git worktree add -q --detach /tmp/prev-src "$prev" + (cd /tmp/prev-src && CGO_ENABLED=0 go build -o /tmp/ch-old ./cmd/cipherhorizon) + - name: Upgrade a database the release made, then hand it back to the release + run: | + set -x + /tmp/ch-old db stats --db /tmp/u.db >/dev/null # the release creates its database + before="$(/tmp/ch-new db status --json --db /tmp/u.db | jq '.pending | length')" + /tmp/ch-new db migrate --db /tmp/u.db + /tmp/ch-new db check --db /tmp/u.db + if [ "$before" -gt 0 ]; then + if /tmp/ch-old db stats --db /tmp/u.db > /tmp/old.out 2>&1; then + echo "the release opened a database this commit upgraded"; cat /tmp/old.out; exit 1 + fi + grep -q "newer than this program" /tmp/old.out || { cat /tmp/old.out; exit 1; } + # and the backup taken before the upgrade gets it back + bak="$(/tmp/ch-new db status --json --db /tmp/u.db | jq -r '.backups[0].path')" + /tmp/ch-new db restore --db /tmp/u.db "$bak" + /tmp/ch-old db stats --db /tmp/u.db >/dev/null + else + /tmp/ch-old db stats --db /tmp/u.db >/dev/null # same schema: both read it + fi + vuln: # Known vulnerabilities in the dependencies and the standard library, limited to code that is reachable. runs-on: ubuntu-latest diff --git a/.gitignore b/.gitignore index 90b438c..91b388f 100644 --- a/.gitignore +++ b/.gitignore @@ -24,6 +24,9 @@ Thumbs.db *.db *.db-shm *.db-wal +*.db.lock +# except the databases released versions wrote, kept for the upgrade tests (PRD-016 U-02) +!/internal/store/testdata/upgrade/v*.db # release build output and local release scratch /RELEASE_NOTES.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 0aebe1a..d79ded9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,32 @@ the CBOM/report/eval schema versions, the database) can change in a minor releas ## [Unreleased] +### Added + +- **Safer upgrades** ([PRD-016](docs/product/PRD-016-upgrades-and-updates.md) Phase 0). The server still upgrades + its database at launch, from any earlier version, after a backup; now it also: + - refuses to run two servers on one database, and refuses to upgrade a database another CipherHorizon process + is using (a `.lock` file next to it; the OS releases the lock if the process dies); + - checks there is room for the backup first, and changes nothing if there is not; + - checks the database after upgrading it, and if a step or the check fails, puts the file back exactly as it + was, keeps the failed copy for diagnosis, and refuses to start; + - records each upgrade (from which schema to which, by which version, and the backup) in the database. +- `cipherhorizon db status` (schema, what an upgrade would do, backups, upgrade history, whether a server is + using it), `db check` (SQLite integrity check, references, structure; exit 2 on a problem), `db migrate` + (`--dry-run` to rehearse) and `db restore BACKUP-FILE` (the server must be stopped; the replaced database + is kept beside it). +- `serve --migrate refuse`: for change-controlled sites, stop instead of upgrading an older database, so the + upgrade is run as its own step with `db migrate`. + +### Upgrade notes + +- Schema 15 → 16: adds the `upgrade_log` table. Nothing existing is changed; the upgrade takes under a second. +- From this version on, a second `serve` on the same database exits with a message naming the process using + it. If you ran two servers on one file before, give each its own `--db`. +- A database upgraded by this version is refused by 0.4.0 and earlier ("newer than this program + understands"). To go back, restore the backup made before the upgrade with `cipherhorizon db restore` + (this version) and then run the older one. + ## [0.4.0] - 2026-10-22 ### Security diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b77bf51..ce0193d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -47,6 +47,31 @@ rest. `go vet ./...` and `gofmt -l .` (expect no output) should be clean before `go test ./internal/app -update` and read the diff. 5. **Run the checks** in "Development setup" above. +## Changing the database + +Every install upgrades its own database when a new version starts, from whatever version it ran before, +however many releases it skipped. That only works if these rules hold (the tests enforce most of them; see +[PRD-016](docs/product/PRD-016-upgrades-and-updates.md) §5.2): + +- **Add a new step to `migrations` in `internal/store/store.go`; never edit, reorder or remove one that has + shipped.** Shipped steps are listed in `internal/store/migrations.lock` (`TestMigrationsLock`). Fix a + shipped mistake with a new step. +- **Additive in a minor release:** new tables, nullable columns or columns with a default, new indexes. + Dropping or renaming a column, or changing what one means, goes in two releases: add the new one and write + both; then, a release later, stop using and drop the old one. +- **A step's Go back-fill (`postMigrations`) uses only frozen code:** SQL, the standard library and helpers + named `…V` that are copied for it. No `Store` methods, no other package of this module, no shared + package variables (`TestPostMigrationsUseOnlyFrozenCode`). `recordConfirmedLinkV6` is an example. +- **Stored evidence is never rewritten.** Scan reports (`report_json`) and host inventories stay as observed; + a new report shape gets a new `schemaVersion` and readers keep reading the old ones. +- **Regenerate the schema description** and review its diff: + `go test ./internal/store -run TestSchemaFingerprint -update-schema` (writes `schema_fingerprint.go` and + `internal/store/testdata/schema.txt`). +- **Write upgrade notes:** while a step is unreleased, `CHANGELOG.md` `[Unreleased]` needs a + `### Upgrade notes` section saying what the upgrade does and roughly how long it takes on a large database. +- `TestUpgradeFromEveryRelease` upgrades a database written by every past release (`internal/store/testdata/upgrade`) + and checks nothing was lost and the result matches a fresh install. + ## Evidence and citations This is the one rule the codebase enforces in tests, not just in review: **every PQC verdict, every TLS diff --git a/docs/deploy.md b/docs/deploy.md index 34d8968..0801b3a 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -89,11 +89,13 @@ docker compose exec cipherhorizon /cipherhorizon db backup --db /data/cipherhori ``` A backup is a consistent copy taken while the server runs, readable by its owner only. It holds -password and session hashes: protect it like the database. To restore, stop the server, put the copy -in place of the database file and start it again (`serve --db` also opens a copy directly). +password and session hashes: protect it like the database. To restore, stop the server, run +`cipherhorizon db restore --db FILE BACKUP` (it keeps the database it replaces) and start it again +(`serve --db` also opens a copy directly). -Upgrades migrate the database on start, after keeping a copy next to it; the newest three such -copies are kept. +Upgrades migrate the database on start, after keeping a copy next to it (the newest three are kept); a failed +upgrade puts the file back as it was. `cipherhorizon db status` shows the schema, the backups and the upgrade +history. Details: [INSTALL.md](engineering/INSTALL.md#upgrading). ## Checking the inventory scripts diff --git a/docs/engineering/INSTALL.md b/docs/engineering/INSTALL.md index 1d114c1..41f5dd9 100644 --- a/docs/engineering/INSTALL.md +++ b/docs/engineering/INSTALL.md @@ -184,13 +184,17 @@ docker exec cipherhorizon /cipherhorizon db backup --db /data/cipherhorizon.db / On Windows (administrator): `cipherhorizon db backup --db C:\ProgramData\CipherHorizon\cipherhorizon.db D:\backup\ch.db`. -To restore: -1. Stop the server. -2. Put the backup in place of `cipherhorizon.db`. -3. Delete the stale `cipherhorizon.db-wal` and `cipherhorizon.db-shm` files next to it. -4. Start the server. +To restore, stop the server, then: -A backup opens like any database (`cipherhorizon serve --db backup.db`), so you can check it first. +```sh +cipherhorizon db restore --db /var/lib/cipherhorizon/cipherhorizon.db /root/cipherhorizon-2026-10-24.db +docker run --rm -v cipherhorizon-data:/data ghcr.io/offsetsecurity/cipherhorizon: db restore --db /data/cipherhorizon.db /data/backup.db +``` + +It checks the backup is a sound CipherHorizon database first, refuses while a server is using the database, +and keeps the database it replaces beside it (`cipherhorizon.db.replaced-