diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 05bc4c1a..234e9a90 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -15,7 +15,7 @@ All clustering operations require `super_user` role. ### Add Node -Adds a new Harper instance to the cluster. If `subscriptions` are provided, it creates the specified replication relationships between the nodes. Without `subscriptions`, a fully replicating system is created (all data in all databases). +Adds a new Harper instance to the cluster. If `subscriptions` or `sendsTo`/`receivesFrom` are provided, they create the specified (or database-scoped) replication relationships between the nodes. Without any of these, a fully replicating system is created (all data in all databases). **Parameters**: @@ -32,6 +32,9 @@ Adds a new Harper instance to the cluster. If `subscriptions` are provided, it c - `table` — table name - `subscribe` — if `true`, transactions on the remote table are replicated locally - `publish` — if `true`, transactions on the local table are replicated to the remote node +- `sendsTo` / `receivesFrom` _(optional)_ — database-scoped controlled-flow entries for this node (see [Controlling Replication Flow](./overview.md#controlling-replication-flow)). Each entry is an object `{ database?, excludeTables? }`, or (less usefully) a bare string naming a peer, intended to authorize all databases for that peer rather than match a database name. **Only the unscoped object form — `{ database }`, with no `target`/`source` — is reliable today.** Harper's reciprocal `add_node_back` registration to the added peer carries the array over without rewriting each entry's peer reference for the new direction, so a bare string or a `target`/`source` value that's correct for your side of the connection ends up wrong on the peer's copy, and the peer's send-authority check silently rejects the subscription — replication doesn't happen. Use [config routes](./overview.md#controlling-replication-flow) for anything that needs to be scoped to one peer. Unlike a config route's `replicates.sendsTo` / `replicates.receivesFrom`, which describes the _local_ node's own direction, these describe the **added node's** perspective: `sendsTo` lists what the added node sends (what this node receives from it), and `receivesFrom` lists what the added node receives (what this node sends to it). If both `subscriptions` and `sendsTo`/`receivesFrom` are provided, `subscriptions` takes precedence and `sendsTo`/`receivesFrom` are ignored. To replicate all databases while excluding specific tables, use a wildcard entry (`excludeTables` with no `database`). + + > **Note**: Because these entries aren't scoped to one peer, the resulting `hdb_nodes` record for the added node isn't restricted to the connection that created it — any other node that also holds the listed database(s) can match against it too. This also doesn't change how the local node advertises itself to the rest of the cluster: a node's own directional (non-mesh) `hdb_nodes` self-record is derived solely from its `harper-config.yaml` routes, so replicating `system` without collapsing to a full mesh requires directional routes in config, not just `add_node`/`set_node` scoping (see [Replicating the `system` database with controlled flow](./overview.md#replicating-the-system-database-with-controlled-flow)). **Request**: diff --git a/reference/replication/overview.md b/reference/replication/overview.md index a9526c82..a1b4f656 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -51,6 +51,8 @@ You can also manage nodes dynamically through the [Operations API](./clustering. Harper automatically replicates node information to other nodes in the cluster using [gossip-style discovery](https://highscalability.com/gossip-protocol-explained/). This means you only need to connect to one existing node in a cluster, and Harper will automatically detect and connect to all other nodes bidirectionally. +This full-mesh, bidirectional auto-connect behavior applies to nodes with no directional routes. A node configured with [directional routes](#controlling-replication-flow) advertises a constrained registry record instead, so discovered non-neighbor nodes do not receive a replication connection — see [Controlling Replication Flow](#controlling-replication-flow). + ### Data Selection By default, Harper replicates all data in all databases. You can narrow replication to specific databases: @@ -198,7 +200,41 @@ replication: In this example, the local node only receives from `node-two` (one-way inbound) and only sends to `node-three` (one-way outbound). -> **Note**: When using controlled flow replication, avoid replicating the `system` database. The `system` database contains node configurations, so replicating it would cause all nodes to have identical (and incorrect) route configurations. +You can also scope flow per database, so different databases flow in different directions between the same two nodes. Use `sendsTo` / `receivesFrom` entries with a `database`: + +```yaml +replication: + databases: + - cardata + - config + - system + routes: + - hostname: node-two + replicates: + sendsTo: + - database: config # push central config downstream + - database: system # push central config (users, roles, schemas) downstream + receivesFrom: + - database: cardata # aggregate telemetry upstream +``` + +`sendsTo` / `receivesFrom` are declared from the perspective of the node whose `harper-config.yaml` they're in, for its route to that one peer, and — because a directional route also gates what it's willing to send — both sides normally need a matching entry. To aggregate a database upstream instead of pushing it downstream — for example, so a role created on a roadside node reaches a middle-tier node — the **roadside** node's route to middle needs `sendsTo: [{ database: system }]`, and the **middle-tier** node's route to roadside needs the matching `receivesFrom: [{ database: system }]`. If the middle tier is missing its `receivesFrom` half, it never attempts the subscription; if roadside is missing its `sendsTo` half, middle's subscription attempt is rejected as unauthorized. + +### Replicating the `system` database with controlled flow + + + +Before v5.2, replicating the `system` database under controlled flow was discouraged: because `hdb_nodes` (the node registry) lives in `system` and each node advertised itself as a full-mesh participant, replicating `system` caused every node to discover and directly connect to every other node — collapsing a constrained topology into a full mesh. + +As of v5.2 you can replicate `system` while keeping a constrained topology. When a node has directional routes, it advertises a **directional** registry record derived from those routes (which neighbors it sends to / receives from) instead of a blanket "connect to everyone." A discovered non-neighbor node therefore is not subscribed to and does not receive a replication connection. This lets central configuration — users, roles, and schemas — propagate transitively across the whole cluster while user-database connections stay on the routes you configured. For example, in a `roadside → middle → core` aggregation tree, a role created on a roadside node reaches the core through the middle tier, yet the core never opens a direct replication subscription to a roadside node. + +Notes and current limitations: + +- This applies only when a node has **directional** routes (`replicates` with `sends`/`receives` or `sendsTo`/`receivesFrom`). A node with no directional routes keeps the legacy full-mesh advertisement. +- This constrains replication subscriptions only. On-demand residency/retrieval connections (for example, sharded or invalidated-cache reads) use a separate mechanism governed by data residency, not by this registry record, and can still open a direct socket to a non-neighbor node. +- Central visibility of every node is not guaranteed: an aggregation node may not list every distant leaf in its `hdb_nodes` registry (the registry relay differs from data relay). This does not open a connection either way. +- Route changes to a node's own directionality take effect on restart. +- Replicating `system` upstream (edge → core) propagates `hdb_user`/`hdb_role` along with everything else in the database: a role or user created — or a compromised edge node's route table altered — anywhere on the upstream path reaches every node it flows to. Weigh this against your trust boundary for edge nodes before routing `system` upstream from them. ### Explicit Subscriptions @@ -282,7 +318,7 @@ The following data operations are replicated across the cluster: **Destructive schema operations are not replicated**: `drop_database`, `drop_table`, and `drop_attribute` must be run on each node independently. -Users and roles are not replicated across the cluster. +Users and roles are not replicated across the cluster by default. As of v5.2, they do propagate when the `system` database (where `hdb_user` and `hdb_role` live) is included in replication — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). Certain management operations — including component deployment and rolling restarts — can also be replicated across the cluster. diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 54bed7e3..af163346 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -37,3 +37,24 @@ Components can now declare `host` and `urlPath` in `config.yaml`, or pass them t ### Web Application Firewall Harper Pro now includes a Web Application Firewall that evaluates rule-based IP/CIDR, method, path, header, and query conditions before authentication and application routing. Rules support block, log, and score actions; cluster-wide monitor and off modes; per-rule shadowing; node activation gates; live replicated updates; and RE2-backed regular expressions. See [Web Application Firewall](/reference/v5/web-application-firewall/overview). + +## Replication + +### Replicating the System Database with a Constrained Topology + +Controlled-flow replication can now include the `system` database while keeping a constrained (non-mesh) topology. + +Previously, replicating `system` was discouraged when using [controlled replication flow](/reference/v5/replication/overview#controlling-replication-flow): because the node registry (`hdb_nodes`) lives in `system` and each node advertised itself as a full-mesh participant, replicating `system` caused every node to discover and directly connect to every other node — defeating the point of a constrained topology. + +As of 5.2, a node with directional routes advertises a **directional** registry record derived from those routes (the neighbors it sends to / receives from) instead of a blanket "connect to everyone." A discovered non-neighbor node therefore is not subscribed to and does not receive a replication connection. This lets central configuration — users, roles, and schemas — propagate transitively across the entire cluster while user-database connections stay on the routes you configured. + +For example, in a `roadside → middle → core` aggregation tree, replicating `system` now lets a role created on a roadside node reach the core (through the middle tier) without the core ever opening a direct replication subscription to a roadside node. + +Behavior notes: + +- This applies only to nodes that have **directional** routes. A node with no directional routes keeps the legacy full-mesh advertisement, so existing full-mesh clusters are unaffected. +- This constrains replication subscriptions only; on-demand residency/retrieval connections (e.g. sharded or invalidated-cache reads) are a separate mechanism, governed by data residency, and are unaffected by this record. +- `add_node` / `set_node` accept database-scoped `sendsTo` / `receivesFrom` entries to restrict replication with the added node to specific databases (see [Add Node](/reference/v5/replication/clustering#add-node)). Only the unscoped object form (`{ database?, excludeTables? }`) is reliable today: Harper's reciprocal `add_node_back` registration to the added peer doesn't rewrite bare peer-name strings or `target`/`source` fields for the new direction, so attempts to scope an entry to one peer can silently fail to authorize replication. These describe the **added node's** perspective, the opposite direction from a config route's `replicates.sendsTo` / `replicates.receivesFrom`, and (unlike config routes) aren't confined to the connection that created them. They also do not make the local node advertise a directional (non-mesh) `hdb_nodes` self-record, which is derived solely from the node's own config routes. Constraining `system` replication cluster-wide still requires directional routes in `harper-config.yaml`. +- Central visibility of every node is not guaranteed: an aggregation node may not list every distant leaf in its `hdb_nodes` registry. This does not open a connection either way. + +See [Controlling Replication Flow](/reference/v5/replication/overview#controlling-replication-flow) for configuration details.