Repository navigation
feat: safe database upgrades from any release (PRD-016 Phase 0) - #29
Open
khalid-offset wants to merge 2 commits into
Open
khalid-offset wants to merge 2 commits into
khalid-offset wants to merge 2 commits into
Conversation
The server already upgrades its database at launch; this makes that provably safe and gives operators the tools around it. - Lock released migrations (internal/store/migrations.lock, built from the v0.1.0 and v0.4.0 sources); TestMigrationsLock refuses an edited, reordered or removed step. - Databases written by the real v0.1.0, v0.2.0, v0.3.0 and v0.4.0 binaries (testing/upgrade/make-fixture.sh) and TestUpgradeFromEveryRelease: every table keeps its rows, evidence, accounts and triage are unchanged, and the result matches a fresh install's structure. - Upgrade flow: database lock (one server per file, no upgrade under a running process), free-space check, backup, steps, verification; a failed step or check restores the pre-upgrade backup and refuses to start. Each upgrade is recorded in upgrade_log (migration 16). - Schema fingerprint compiled in, with a readable dump (testdata/schema.txt) for review. - db status, db check, db migrate [--dry-run], db restore; serve --migrate auto|refuse. - Migration Go steps may use only frozen code (TestPostMigrationsUseOnlyFrozenCode); migration 6 and 10 now use frozen copies (recordConfirmedLinkV6, timestampColumnsV10), with identical output. - New platform layer internal/osfile (file lock, free space) so store stays OS-free. - CI: an upgrade job checks the latest release refuses, then after db restore reads, a database this commit upgraded. Unreleased migrations need "### Upgrade notes" in the changelog. - prepare-release.sh locks a release's migrations; RELEASING, CONTRIBUTING, INSTALL and deploy docs updated.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
CodeQL js/file-access-to-http flagged seed.mjs: it read the sample host inventory and sent it over HTTP. That was only ever to the throwaway local server make-fixture.sh starts, but nothing enforced it. seed.mjs now refuses any --base that is not http:// on loopback, and the inventory is uploaded by make-fixture.sh (curl to the same local server), so the script reads no files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 0 of PRD-016: make database upgrades provably safe, from any earlier release, however many versions were skipped. The server still upgrades its database at launch; this adds the guard rails and the operator commands around it.
What changes for users
serveholds<db>.lockfor its life; a second server on the same file is refused with the holder's pid and version. Other commands take the lock only when they would upgrade, sodb backupstill works while a server runs.quick_check, no new foreign-key violations). If a step or the check fails, the pre-upgrade backup is put back, one failed copy is kept for diagnosis, and the server refuses to start.upgrade_logtable (migration 16).db status,db check(exit 2 on a problem),db migrate [--dry-run],db restore BACKUP(the replaced database is kept).serve --migrate auto|refusefor change-controlled sites.How skipped versions are proven
internal/store/migrations.lock: steps 1–14 hashed from the v0.1.0 tag, 15 from v0.4.0.TestMigrationsLockrefuses an edited, reordered or removed step;prepare-release.shlocks a release's steps.internal/store/testdata/upgrade/v0.{1,2,3,4}.0.db: written by the real released binaries viatesting/upgrade/make-fixture.sh(TLS, PQ cert, SSH, STARTTLS, dead port, two scans each, triage, link, host inventory, admin account).TestUpgradeFromEveryReleasechecks every table keeps its rows, evidence, accounts and triage are byte-identical, and the result matches a fresh install's structure. Mutation-checked: a step that deletes data fails it.testdata/schema.txt(regen with-update-schema).upgradeCI job: the latest release (built from its tag) refuses a database this commit upgraded, and reads it again afterdb restore.Latent bugs fixed
timestampColumnsmap: adding a later table to it would break upgrades from before step 10. NowtimestampColumnsV10.recordManualLinkTx(usesinternal/estate). NowrecordConfirmedLinkV6, verified to write identical rows.TestPostMigrationsUseOnlyFrozenCodestops this pattern returning.Notes for review
internal/osfile(flock / LockFileEx, free space): archtest only lets platform layers importx/sys/windows..gitignoreexception); test passwords and loopback scans only.db restore.Testing
go test ./...,golangci-lint run ./...clean; Linux/macOS: build, vet and lint clean locally, tests run in CI.