docs: replicating the system database with a constrained topology (5.2) - #583
docs: replicating the system database with a constrained topology (5.2)#583kriszyp wants to merge 9 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the replication documentation and release notes for version 5.2.0, introducing the ability to replicate the system database under controlled flow with a constrained topology and scope replication flow per database. A review comment correctly identifies that the YAML configuration example uses the invalid property 'host' instead of 'hostname'.
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-583 This preview will update automatically when you push new commits. |
Controlled-flow replication can now include the `system` database while keeping a constrained topology (directional self-record). Update the replication overview (Controlling Replication Flow) — the previous "avoid replicating system" note no longer applies for directional routes — and add a 5.2 release note. Documents per-database directional routes, the add_node database-scoping behavior change, and the central-visibility caveat. Companion to the harper-pro core change (branch kris/systemdb-routing-repro). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
4ecc02d to
a290435
Compare
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-583 This preview will update automatically when you push new commits. |
- Fix heading-hierarchy skip (h4 -> h3) and hostname/host key inconsistency in the controlled-flow example. - Extend the per-database routing example to actually route `system`, matching the section it introduces. - Soften "discovered non-neighbor nodes therefore do not open direct connections" to the actual guarantee (no subscription/replication connection), and call out that on-demand residency/retrieval connections are a separate, unaffected mechanism. - Document add_node/set_node's sendsTo/receivesFrom fields in the Clustering reference (previously undocumented) and note the config-authority limitation: dynamic Operations API scoping doesn't make a node advertise a directional self-record — that's derived only from harper-config.yaml routes. Co-Authored-By: Claude Sonnet 5 <[email protected]>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-583 This preview will update automatically when you push new commits. |
Ethan-Arrowood
left a comment
Ethan-Arrowood
left a comment
There was a problem hiding this comment.
The core 5.2 constrained-topology content is accurate — I traced every major claim to merged harper-pro#572 and its source (computeSelfReplicates, shouldReplicateFromNode, setNode.ts) and it holds up. The host:→hostname: fix the Gemini bot asked for is in at this head too.
The problem is concentrated in the new add_node parameter bullet, which has a factual error serious enough to break a user's setup silently: a bare-string sendsTo entry is matched as a peer name, not a database name, so the documented "sendsTo": ["cardata"] authorizes nothing and replication just doesn't happen. Details inline, along with a direction-perspective problem and an overstated containment claim.
One of these may actually be an implementation-intent question rather than a docs bug — see the sendsTo direction comment. Since no test pins add_node's direction, it'd be worth confirming with you directly which behavior is intended; the doc needs to state a direction explicitly either way, and a pinning integration test would settle it permanently.
sent with Claude Opus 5
Ethan-Arrowood's review traced the actual behavior in harper-pro's knownNodes.ts/setNode.ts: a bare-string sendsTo/receivesFrom entry is matched as a peer name, not a database, and silently authorizes nothing; and add_node's sendsTo/receivesFrom describe the *added* node's perspective (opposite of a config route's local-node perspective) with no peer/target scoping, so the resulting hdb_nodes record is visible cluster-wide rather than confined to one connection. Corrects both the clustering.md reference and the matching 5.2 release note bullet to describe this accurately. Co-Authored-By: Claude Sonnet 5 <[email protected]>
…-docs # Conflicts: # release-notes/v5-lincoln/5.2.md
…esFrom
Independent review (codex/grok) caught that the previous wording said
entries "carry no peer scoping" and must be `{ database, excludeTables? }`
objects — both incomplete. harper-pro's RouteEntry type and matcher
(knownNodes.ts routeEntriesIncludePeer) support target/source fields that
do scope an entry to one peer; the docs just never mentioned them. Also
clarifies that the per-database YAML example pushes system/config
downstream, whereas the roadside-to-core narrative right after it
describes the opposite (upstream) direction, and how to flip it.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
…gregation example Independent review (codex/gemini/grok) flagged two more issues: - add_node's sendsTo/receivesFrom are new in v5.2 (harper-pro#572, commit 0b49587f) but the parameter list had no VersionBadge, and the "must be an object" wording incorrectly implied a bare string is never valid (it is — it just matches a peer name across all databases, not a database name). - The "aggregate upstream" guidance didn't say which node's config route needs the receivesFrom entry, so it could be misread as "configure receivesFrom on the leaf's route to the core" (which would do the opposite of what's intended). Spelled out the roadside/middle example explicitly. Co-Authored-By: Claude Sonnet 5 <[email protected]>
…by review - The roadside-to-middle upstream aggregation example only described the middle tier's receivesFrom entry. replicationConnection.ts's send-side authority gate (harper-pro#498) independently checks the SENDER's own directional config route, so roadside also needs a matching sendsTo entry or the subscription is rejected as unauthorized. Documented both sides. - The 5.2 release note still said a bare-string entry "authorizes nothing," contradicting the corrected clustering.md wording (a string names a peer and authorizes all databases for it). Aligned. - Added a caveat that routing `system` upstream from an edge node propagates hdb_user/hdb_role along with everything else, so a compromised or careless edge node's changes reach every node downstream of it. Co-Authored-By: Claude Sonnet 5 <[email protected]>
…ken) Traced setNode.ts's reciprocal add_node_back registration: it swaps sendsTo/receivesFrom wholesale onto the peer (targetAddNodeObj.sendsTo = req.receivesFrom, etc.) without rewriting each entry's target/source for the new direction. A target/source value that's correct for the calling side ends up wrong on the peer's copy, so the send-authority gate (replicationConnection.ts's shouldCloseSendAuthWatch) rejects the subscription — replication silently doesn't happen. The docs previously told users to set target/source to scope an add_node entry to one peer; that's not a reliable fix given this behavior. Removed the recommendation and pointed to config routes instead, which don't have this problem (computeSelfReplicates stamps target/source correctly on both sides). Also fixed an overstated "rejected as unauthorized" claim for the roadside/middle example: omitting the receiver's half means no subscription is attempted at all, not a rejection. This is a real harper-pro implementation gap, not just a docs issue — noting it in the dispatch Findings for a follow-up. Co-Authored-By: Claude Sonnet 5 <[email protected]>
…ion bug as target/source
Traced the failure Grok/codex flagged: add_node's bare-string form
(entry === peerName) has the identical swap-without-rewrite problem as
target/source — a string that's correct for the caller's own send
authorization becomes wrong once add_node_back copies it unmodified
onto the peer's record, so the peer's send-authority gate rejects the
subscription. Consolidated the guidance: only the unscoped object form
{ database?, excludeTables? } is reliable; steered both the string
form and target/source into the same "don't rely on this" warning
instead of presenting the string as a working (if blunt) alternative.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-583 This preview will update automatically when you push new commits. |
|
Reviewed — |
Ethan-Arrowood
left a comment
Ethan-Arrowood
left a comment
There was a problem hiding this comment.
My earlier CHANGES_REQUESTED at 3ac6fee is stale — all four of those findings are genuinely fixed, and I re-traced each against merged harper-pro#572 rather than taking the wording at face value:
- Bare-string entries are now correctly documented as peer names, not database names — matches
routeEntriesIncludePeer. - The
add_nodedirection is now stated explicitly as the added node's perspective. I re-derived it fromsetNodewritingreplicates.sendsToonto the peer's row and the receive gate reading a row'ssendsToas "this peer sends to me." As documented. - The "scopes only the connection to this node" containment claim is gone, replaced with the accurate opposite.
subscriptionsprecedence is now described as behavior rather than enforcement.
The unprompted additions — the VersionBadge tags, the Gossip Discovery full-mesh caveat, and especially the users/roles trust-boundary bullet — are all good and correct.
Re-requesting changes for one thing that is new in these commits, not a leftover: the two-sided-config paragraph added in 9c07b2b9/8b8328e4 states a hard requirement the implementation doesn't have, and it contradicts the one-sided YAML example three lines above it. The send-authority half is right; the receive half isn't. shouldReplicateFromNode only consults a config route when configRouteReplicates exists and is the directional object form — a missing route, or a plain - hostname: roadside route (which iterateRoutes normalizes to boolean true), both fall through to the peer's advertised hdb_nodes record. Since computeSelfReplicates stamps if (!entry.target) entry.target = peer, roadside advertises sendsTo: [{ database: 'system', target: 'middle' }] and the fallback matches it exactly. That fallback is the 5.2 feature — roadside dials middle, middle knows roadside only by gossip, and no receivesFrom on middle is needed. Suggestion inline.
The sharper risk is the second-order read: this paragraph invites treating a missing receivesFrom as a constraint. It isn't. A reader who omits it believing that blocks inbound system replication gets it anyway — immediately after a new trust-boundary warning about exactly that data.
Two smaller inline suggestions on other newly added lines, both non-blocking.
Worth noting this hinges on one undefined fallback branch, so if you read shouldReplicateFromNode differently I'd rather resolve it here than land either version. And the real question underneath: is the two-sided requirement the intended contract — should the discovered-peer fallback be tightened so an explicit receivesFrom really is required? If so that's a harper-pro change, not prose.
Two unrelated asks before merge, neither blocking the inline fix:
- The
add_nodebullet inclustering.mdis now the sole record of an unfixed harper-pro defect (add_node_backcarries the entry array over without rewriting per-entry peer references). I traced it and the claim is real, but I searched harper-pro and there's no issue tracking it and no test coveringadd_nodewithsendsTo/receivesFromat all. A prose warning with no owner and no expiry silently becomes wrong the day someone fixes the code. Please file the harper-pro issue and link it from the bullet so the caveat has something to retire it. Related: the bullet leans ontarget/source, which aren't documented anywhere in this repo — either define them there or just say peer-scoping doesn't work throughadd_node, use config routes. - The PR body still says the harper-pro change is on branch
kris/systemdb-routing-reprowith a "feature PR to be opened." That's harper-pro#572, merged. Please update the link.
sent with Claude Opus 5
| - database: cardata # aggregate telemetry upstream | ||
| ``` | ||
|
|
||
| `sendsTo` / `receivesFrom` are declared from the perspective of the node whose `harper-config.yaml` they're in, for its route to that one peer, and — because a directional route also gates what it's willing to send — both sides normally need a matching entry. To aggregate a database upstream instead of pushing it downstream — for example, so a role created on a roadside node reaches a middle-tier node — the **roadside** node's route to middle needs `sendsTo: [{ database: system }]`, and the **middle-tier** node's route to roadside needs the matching `receivesFrom: [{ database: system }]`. If the middle tier is missing its `receivesFrom` half, it never attempts the subscription; if roadside is missing its `sendsTo` half, middle's subscription attempt is rejected as unauthorized. |
There was a problem hiding this comment.
A matching receivesFrom is not required when the peer has no directional route for that neighbor — and this contradicts the YAML example directly above, which shows sendsTo on one side only and presents it as working (it is).
shouldReplicateFromNode gates on the config route only when it's present and object-shaped:
const configRoute = (node as any).configRouteReplicates; // undefined when no route matches this peer
if (configRoute && typeof configRoute === 'object') {
peerFeedsUs = configRoute.receives || routeEntriesIncludePeer(configRoute.receivesFrom, node.name, databaseName);
} else {
peerFeedsUs = node.replicates?.sends || node.replicates?.sendsTo?.some?.(…); // the peer's ADVERTISED record
}configRouteReplicates is matchingRoute ? matchingRoute.replicates : undefined, and a plain - hostname: roadside route normalizes to boolean true in iterateRoutes — also not the object form. Either way middle falls through to roadside's advertised hdb_nodes record, which computeSelfReplicates has already fully qualified with target, so the fallback matches and middle subscribes with no receivesFrom of its own. That's the canonical 5.2 shape.
The send-authority half is correct — I confirmed roadside closes with 1008 Unauthorized database subscription.
| `sendsTo` / `receivesFrom` are declared from the perspective of the node whose `harper-config.yaml` they're in, for its route to that one peer, and — because a directional route also gates what it's willing to send — both sides normally need a matching entry. To aggregate a database upstream instead of pushing it downstream — for example, so a role created on a roadside node reaches a middle-tier node — the **roadside** node's route to middle needs `sendsTo: [{ database: system }]`, and the **middle-tier** node's route to roadside needs the matching `receivesFrom: [{ database: system }]`. If the middle tier is missing its `receivesFrom` half, it never attempts the subscription; if roadside is missing its `sendsTo` half, middle's subscription attempt is rejected as unauthorized. | |
| `sendsTo` / `receivesFrom` are declared from the perspective of the node whose `harper-config.yaml` they're in, for its route to that one peer. Because a directional route also gates what a node is willing to send, the **sending** side always needs the matching `sendsTo` entry. To aggregate a database upstream instead of pushing it downstream — for example, so a role created on a roadside node reaches a middle-tier node — the **roadside** node's route to middle needs `sendsTo: [{ database: system }]`; without it, middle's subscription attempt is rejected as unauthorized. The **receiving** side needs a matching `receivesFrom` only when it has its own directional route for that peer: a middle-tier node with a directional route to roadside is gated by that route, so omitting `receivesFrom` there means it never attempts the subscription. If middle has no route to roadside at all — or only a plain, non-directional one — it falls back to roadside's advertised registry record and subscribes on the strength of roadside's `sendsTo` alone. Omitting `receivesFrom` is therefore not a way to block inbound replication. |
| - This constrains replication subscriptions only. On-demand residency/retrieval connections (for example, sharded or invalidated-cache reads) use a separate mechanism governed by data residency, not by this registry record, and can still open a direct socket to a non-neighbor node. | ||
| - Central visibility of every node is not guaranteed: an aggregation node may not list every distant leaf in its `hdb_nodes` registry (the registry relay differs from data relay). This does not open a connection either way. | ||
| - Route changes to a node's own directionality take effect on restart. | ||
| - Replicating `system` upstream (edge → core) propagates `hdb_user`/`hdb_role` along with everything else in the database: a role or user created — or a compromised edge node's route table altered — anywhere on the upstream path reaches every node it flows to. Weigh this against your trust boundary for edge nodes before routing `system` upstream from them. |
There was a problem hiding this comment.
Minor, on an otherwise good addition: "route table" conflates two things. Routes come from harper-config.yaml and aren't replicated — what replicates is the hdb_nodes registry. A node's own row gets rewritten from its config on restart, so the poisonable surface is peers' copies of other nodes' rows.
| - Replicating `system` upstream (edge → core) propagates `hdb_user`/`hdb_role` along with everything else in the database: a role or user created — or a compromised edge node's route table altered — anywhere on the upstream path reaches every node it flows to. Weigh this against your trust boundary for edge nodes before routing `system` upstream from them. | |
| - Replicating `system` upstream (edge → core) propagates `hdb_user`/`hdb_role` along with everything else in the database: a role or user created — or a compromised edge node's `hdb_nodes` registry rows altered — anywhere on the upstream path reaches every node it flows to. Weigh this against your trust boundary for edge nodes before routing `system` upstream from them. |
| **Destructive schema operations are not replicated**: `drop_database`, `drop_table`, and `drop_attribute` must be run on each node independently. | ||
|
|
||
| Users and roles are not replicated across the cluster. | ||
| Users and roles are not replicated across the cluster by default. As of v5.2, they do propagate when the `system` database (where `hdb_user` and `hdb_role` live) is included in replication — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). |
There was a problem hiding this comment.
This version-gates the wrong half. Replicating system propagated hdb_user/hdb_role before 5.2 too — the old guidance was to disable system replication for a tight topology, which also lost global users/roles/schema propagation. What 5.2 changed is that you can now do it without collapsing to a full mesh.
| Users and roles are not replicated across the cluster by default. As of v5.2, they do propagate when the `system` database (where `hdb_user` and `hdb_role` live) is included in replication — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). | |
| Users and roles are not replicated across the cluster by default. They do propagate when the `system` database (where `hdb_user` and `hdb_role` live) is included in replication; as of v5.2 this no longer forces a full mesh — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). |
3 participants
Summary
Documents the 5.2 replication change that lets the
systemdatabase replicate while keeping a constrained (non-mesh) topology.reference/replication/overview.md(Controlling Replication Flow): the previous "avoid replicating thesystemdatabase" note no longer applies when a node uses directional routes. Adds a per-databasesendsTo/receivesFromexample and a<VersionBadge type="changed" version="v5.2.0" />subsection explaining constrainedsystemreplication, its opt-in nature, and the central-visibility caveat.release-notes/v5-lincoln/5.2.md: adds a "Replicating the System Database with a Constrained Topology" section (also documents theadd_nodedatabase-scoping behavior change).Companion PR
Documents the harper-pro core change on branch
kris/systemdb-routing-repro(feature PR to be opened; this docs PR will be cross-linked once it is).Where to look
The behavior-change framing in the reference doc and the
add_nodedatabase-scoping note in the release note — please sanity-check the wording against the intended product behavior.Drafted by an LLM (Claude Opus 4.8).