Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions docs/beacon-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,14 +88,38 @@ curl http://<your-public-ip>:8400/node-info

Beacons federate with each other via **real-time WebSocket connections** — sharing their node registries so every beacon has the complete grid view.

Federation requires a shared secret, `EARTHGRID_FEDERATION_KEY`. **Every beacon
that federates must set the same value.**

```bash
# Generate one secret and copy it to every federated beacon
openssl rand -hex 32

# On each beacon:
export EARTHGRID_FEDERATION_KEY=<the same value everywhere>

# Add a peer beacon
export EARTHGRID_BEACON_PEERS=http://other-beacon.example.com:8400
export EARTHGRID_BEACON_PEERS=https://other-beacon.example.com:8400

# Or add multiple (comma-separated)
export EARTHGRID_BEACON_PEERS=http://beacon1.example.com:8400,http://beacon2.example.com:8400
export EARTHGRID_BEACON_PEERS=https://beacon1.example.com:8400,https://beacon2.example.com:8400
```

Notes:

- **Without the key, federation is off.** `/api/beacon/ws` returns 503 and no
outbound connections are attempted. This is deliberate: a peer on that socket
can add nodes to your registry *and delete them*, so an unconfigured beacon
federates with nobody rather than with anybody.
- **The federation key is separate from `EARTHGRID_API_KEY` on purpose.** A
federated peer gets registry sync and nothing else — it cannot write to your
node's API. Do not reuse your grid key here.
- **Use `https://` peer URLs.** The key travels in a request header on the
WebSocket handshake; over plain `http://` it is sent in the clear.
- Mismatched keys are not silent: the dialer logs
`Federation: failed to connect to <url>` once a minute and retries with
backoff.

### How it works

1. On startup, each beacon connects to its peers via WebSocket (`/beacon/ws`)
Expand Down
2 changes: 1 addition & 1 deletion earthgrid-core/src/auth.rs
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ impl AuthConfig {
}

/// Constant-time string comparison to prevent timing attacks on key checks.
fn constant_time_eq_str(a: &str, b: &str) -> bool {
pub(crate) fn constant_time_eq_str(a: &str, b: &str) -> bool {
if a.len() != b.len() {
return false;
}
Expand Down
3 changes: 3 additions & 0 deletions earthgrid-core/src/beacon.rs
Original file line number Diff line number Diff line change
Expand Up @@ -975,6 +975,9 @@ pub struct BeaconState {
pub registry: Arc<Mutex<BeaconRegistry>>,
pub federation: Option<FederationState>,
pub auth: AuthConfig,
/// Credential for beacon-to-beacon federation. Separate from `auth` on
/// purpose: a federated peer gets registry sync only, not the node API.
pub federation_auth: crate::beacon_federation::FederationAuth,
}

async fn register_node(
Expand Down
176 changes: 175 additions & 1 deletion earthgrid-core/src/beacon_federation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ use axum::{
State,
ws::{Message, WebSocket, WebSocketUpgrade},
},
http::{HeaderMap, StatusCode},
response::IntoResponse,
};
use futures_util::{SinkExt, StreamExt};
Expand Down Expand Up @@ -84,12 +85,87 @@ impl FederationState {
// WebSocket handler (inbound connections from peer beacons)
// ---------------------------------------------------------------------------

/// Header carrying the federation credential on the WebSocket handshake.
pub const FEDERATION_KEY_HEADER: &str = "x-earthgrid-federation-key";

/// The credential two beacons share in order to federate.
///
/// Deliberately *not* the node's `EARTHGRID_API_KEY`. A federated peer needs
/// exactly one capability — beacon registry sync — and nothing else. Reusing
/// the grid key would hand every peer write access to the whole node API
/// (`/api/fetch`, `/api/replicate`, the fetch queue, …), because the dialer
/// must transmit whatever key the listener checks.
#[derive(Debug, Clone, Default)]
pub struct FederationAuth {
key: String,
}

impl FederationAuth {
/// Read `EARTHGRID_FEDERATION_KEY` from the environment.
pub fn from_env() -> Self {
Self {
key: std::env::var("EARTHGRID_FEDERATION_KEY").unwrap_or_default(),
}
}

/// Whether a federation key has been configured.
pub fn is_configured(&self) -> bool {
!self.key.is_empty()
}

/// The configured key, for the outbound dialer.
pub fn key(&self) -> &str {
&self.key
}

/// Whether `presented` matches the configured key.
///
/// Returns false when nothing is configured: federation **fails closed**,
/// so an unconfigured beacon refuses to federate rather than accepting
/// everyone. Comparison is constant-time.
pub fn verify(&self, presented: Option<&str>) -> bool {
if !self.is_configured() {
return false;
}
presented.is_some_and(|p| crate::auth::constant_time_eq_str(p, &self.key))
}
}

/// GET /beacon/ws — upgrade to WebSocket for federation sync.
///
/// A peer on this socket is not a passive reader: `apply_remote_event` lets it
/// upsert nodes into the registry and, via `NodePruned`, **delete** any node by
/// ID. Previously the upgrade was unauthenticated, so anyone who could reach
/// the port could wipe a beacon's registry or fill it with fabricated nodes.
///
/// Fails closed. With no `EARTHGRID_FEDERATION_KEY` configured the endpoint is
/// disabled outright — an unconfigured beacon federates with nobody instead of
/// with everybody.
pub async fn ws_handler(
ws: WebSocketUpgrade,
State(state): State<BeaconState>,
headers: HeaderMap,
) -> impl IntoResponse {
if !state.federation_auth.is_configured() {
warn!("Federation: WS refused — EARTHGRID_FEDERATION_KEY is not set");
return (
StatusCode::SERVICE_UNAVAILABLE,
"beacon federation is disabled: set EARTHGRID_FEDERATION_KEY to enable it",
)
.into_response();
}

let presented = headers
.get(FEDERATION_KEY_HEADER)
.and_then(|v| v.to_str().ok());

if !state.federation_auth.verify(presented) {
warn!("Federation: rejecting WS upgrade with missing or invalid federation key");
return (StatusCode::UNAUTHORIZED, "invalid federation key").into_response();
}

ws.on_upgrade(move |socket| handle_peer_connection(socket, state))
.into_response()
}

async fn handle_peer_connection(socket: WebSocket, state: BeaconState) {
Expand Down Expand Up @@ -222,6 +298,15 @@ fn upsert_if_newer(registry: &crate::beacon::BeaconRegistry, node: &BeaconNode)

/// Spawn background tasks that connect to each peer beacon via WebSocket.
pub fn spawn_peer_connections(state: BeaconState, peer_urls: Vec<String>) {
if !state.federation_auth.is_configured() {
warn!(
"Federation: {} peer(s) configured but EARTHGRID_FEDERATION_KEY is not set — \
not connecting. Set the same key on every federated beacon.",
peer_urls.len()
);
return;
}

for url in peer_urls {
let state = state.clone();
tokio::spawn(async move {
Expand All @@ -230,6 +315,30 @@ pub fn spawn_peer_connections(state: BeaconState, peer_urls: Vec<String>) {
}
}

/// Build the WebSocket handshake request for a peer beacon, attaching the
/// shared federation key.
fn build_federation_request(
ws_url: &str,
federation_key: &str,
) -> std::result::Result<tokio_tungstenite::tungstenite::handshake::client::Request, String> {
use tokio_tungstenite::tungstenite::client::IntoClientRequest;

if federation_key.is_empty() {
return Err("no federation key configured".to_string());
}

let mut request = ws_url
.into_client_request()
.map_err(|e| format!("invalid websocket url: {e}"))?;

let value = federation_key
.parse()
.map_err(|_| "federation key is not a valid header value".to_string())?;
request.headers_mut().insert(FEDERATION_KEY_HEADER, value);

Ok(request)
}

/// Connect to a single peer beacon, reconnecting with exponential backoff.
async fn connect_to_peer_loop(state: BeaconState, peer_url: String) {
let ws_url = peer_url
Expand All @@ -244,7 +353,17 @@ async fn connect_to_peer_loop(state: BeaconState, peer_url: String) {
loop {
info!("Federation: connecting to peer {}", ws_url);

match tokio_tungstenite::connect_async(&ws_url).await {
// Present the shared federation key so the peer's `ws_handler` accepts
// us. Rebuilt each attempt because `connect_async` consumes the request.
let request = match build_federation_request(&ws_url, state.federation_auth.key()) {
Ok(r) => r,
Err(e) => {
warn!("Federation: cannot build request for {}: {}", ws_url, e);
return;
}
};

match tokio_tungstenite::connect_async(request).await {
Ok((ws_stream, _)) => {
info!("Federation: connected to {}", ws_url);
backoff = Duration::from_secs(1); // Reset backoff on success
Expand Down Expand Up @@ -416,4 +535,59 @@ mod tests {
assert!(json.contains("full_sync"));
assert!(json.contains("beacon-1"));
}

#[test]
fn federation_request_carries_federation_key() {
let req = build_federation_request("ws://beacon.example/api/beacon/ws", "fed-secret")
.expect("request should build");
assert_eq!(
req.headers()
.get(FEDERATION_KEY_HEADER)
.map(|v| v.to_str().unwrap()),
Some("fed-secret"),
"dialer must present the key the peer's ws_handler requires"
);
assert!(
req.headers().get("x-api-key").is_none(),
"the node's grid API key must never be sent to a federated peer"
);
}

#[test]
fn federation_request_requires_a_key() {
assert!(
build_federation_request("ws://beacon.example/api/beacon/ws", "").is_err(),
"dialing without a federation key must fail rather than connect anonymously"
);
}

#[test]
fn federation_request_rejects_bad_url() {
assert!(build_federation_request("not a url", "k").is_err());
}

/// Federation must fail closed: an unconfigured beacon federates with
/// nobody, rather than accepting every anonymous client.
#[test]
fn unconfigured_federation_auth_rejects_everything() {
let auth = FederationAuth::default();
assert!(!auth.is_configured());
assert!(!auth.verify(None));
assert!(!auth.verify(Some("")));
assert!(!auth.verify(Some("anything")));
}

#[test]
fn configured_federation_auth_accepts_only_the_key() {
let auth = FederationAuth { key: "correct-horse".to_string() };
assert!(auth.is_configured());
assert!(auth.verify(Some("correct-horse")));

assert!(!auth.verify(None));
assert!(!auth.verify(Some("")));
assert!(!auth.verify(Some("wrong")));
assert!(!auth.verify(Some("correct-horse ")), "no trimming");
assert!(!auth.verify(Some("correct-hors")), "prefix must not pass");
assert!(!auth.verify(Some("correct-horse-battery")), "extension must not pass");
}
}
Loading