Skip to content

A node never says in sync with a main that isn't the one it synced with - #331

Merged
udaychandra merged 4 commits into
mainfrom
agent/sail-main-rebuilt
Oct 21, 2026
Merged

udaychandra merged 4 commits into
mainfrom
agent/sail-main-rebuilt

Conversation

@udaychandra

@udaychandra udaychandra commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Spec sail-main-rebuilt.

  • M1 DatabaseCatalog: one migration (CREATE TABLE catalog AS SELECT lower(hex(randomblob(16))) AS id, NULL AS main) gives every database a random id at creation (or at upgrade, for an existing one). Upgrades and copies keep it. Main's welcome names it (catalog; an older main sends none), and a node records the first one it sees.
  • M2/M3 NodeRound.admit runs at session open in SyncOperations, before the demo unseed and NodeRound.begin. It refuses a different catalog, or none once one is recorded, with exit 1: "Main at is not the database this box synced with (catalog , now ). Restore main's database (/var/lib/sail/sail.db) from a copy, then sync again." The check uses the catalog, not the box id, so a fresh database under the old host.yaml is refused too.
  • M4 Every join prints "Pinned main's host key SHA256:…. Check it on main: sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub". rebuiltRecovery now covers a rebuild from a copy.
  • M5 The re-join probe (welcomedAs) prints nothing of its own.
  • M6 README and ARCHITECTURE.md: main's database is the org, with the backup and restore steps. New invariant I5. CHANGELOG entry.

Tests: DatabaseCatalogTest, SchemaManagerTest (a database from before catalogs), SyncWireTest, NodeRoundAdmitTest, MainRebuiltRoundTest (scenarios 1–5 through a real _sync server; a refusal that survives main hanging up), JoinCommandTest (scenarios 6–7), and NativeFleetIT (scenarios 1 and 3 over real ssh). Scenario 1 failed in both the unit and native lanes when admit was disabled.

mvn -o -Pintegration -Dsail.it.requireIncus=false clean verify passes locally.

Takeover (third commit, bf37dff): the independent delta review found one HIGH and three MEDIUMs, fixed red-then-green.

  • A main restored in place from an older copy keeps its box id, so its log sat below the node's checkpoint: the node paged nothing and said "Already in sync" while main held rows written after the restore. A log below the checkpoint now forgets the checkpoints (LocalReplica.forgetCheckpoints) and walks every head again (MainRebuiltRoundTest).
  • A box that leaves the node role kept main's catalog and would refuse any other main forever with the wrong remedy; HandleChange and the migrate-time repair forget it with the scope (HandleChangeTest, SyncCliTest).
  • README and ARCHITECTURE no longer say a restore recovers work main took and then lost; only unpushed work is pushed.
  • A blank catalog is refused at the record; the catalog-less refusal names sudo sail upgrade as the protocol refusal does; main's path comes from SailPaths; the docs say a node records the first welcome that names a catalog.

Spec sail-main-rebuilt. M1: every database is minted a random catalog id
(one migration, kept by upgrades and copies); main's welcome names it and a
node records the first one at session open. M2/M3: NodeRound.admit refuses
any other catalog, or none once one is recorded, before anything is pushed,
adopted or recorded, with exit 1 and the restore sentence, so "Already in
sync" follows only a round with the recorded main. M4: every join prints the
pinned fingerprint and how to check it on main; rebuiltRecovery names a
rebuild from a copy. M5: the re-join probe prints nothing of its own. M6:
README and ARCHITECTURE.md say main's database is the org, how to back it up
and restore it; invariant I5 names its tests. NativeFleetIT rebuilds main
empty and from a copy over real ssh.
…nd a box leaving the node role forgets main's catalog

The delta review found two gaps. A main restored in place keeps its box id,
so its log sat below the node's checkpoint: the node paged nothing and said
it was in sync while main held rows written after the restore. A log below
the checkpoint now forgets the checkpoints and walks every head again.
A box promoted to main, or migrated as main or standalone, kept the catalog
it recorded as a node and would refuse any other main forever with the
wrong remedy; it forgets it with its scope. A blank catalog is refused at
the record, the catalog-less refusal names the upgrade command as the
protocol refusal does, main's database path comes from SailPaths, and the
docs no longer promise that a restore recovers work main took and lost.
@udaychandra
udaychandra merged commit f6a1a4f into main Oct 21, 2026
5 checks passed
@udaychandra
udaychandra deleted the agent/sail-main-rebuilt branch October 21, 2026 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant