Skip to content

Bound cluster waits by deadline and close the Calvin gaps from #414 - #415

Merged
farhan-syah merged 5 commits into
mainfrom
fix/calvin-gaps
Oct 12, 2026
Merged

farhan-syah merged 5 commits into
mainfrom
fix/calvin-gaps

Conversation

@farhan-syah

@farhan-syah farhan-syah commented Oct 11, 2026 •

Copy link
Copy Markdown
Member

What changed

Breaking

Change Where
quic_keep_alive_secs and quic_idle_timeout_secs are removed. Both derive from rpc_timeout_secs. nodedb-types/src/config/tuning/network.rs
Array tiles route by fmix64(tile_key). ShardBucket replaces the Hilbert-range shard filter. nodedb-types/src/array_route.rs
The WAL logs resolved rows for KV and vector predicate writes. The old KV predicate record is removed. nodedb/src/control/write_resolve

Cluster waits

  • nodedb-cluster/src/transport: QUIC idle timeout = rpc_timeout_secs. Keep-alive = a fifth of it.
  • Startup refuses an RPC timeout below MIN_RPC_TIMEOUT (500 ms).
  • One statement deadline runs through the gateway, Raft forwarding, Calvin submit, CRDT admission, DDL and sync.
  • The gateway retries a leader change until the statement deadline.
  • A remote read gets the full statement budget.

Correctness

  • A held redo entry stops its group's apply lane until a snapshot covers it.
  • A replica that owes a snapshot cannot lead. Membership keeps one voter that can lead.
  • A panicking apply holds its group and answers its waiter. Other groups keep applying.
  • Data-group snapshots carry backup cut barriers and cut floors (nodedb/src/types/snapshot_calvin.rs).
  • Calvin barrier rows cost one fsync per apply pass, not one per event.
  • A departed vShard's Calvin state is retried until removed, after boot recovery finishes.
  • Row reads take LockMode::IntentShared on their collection. TRUNCATE invalidates point and index reads.
  • The Calvin validator orders collection-level conflicts (nodedb-cluster/src/calvin/sequencer/validator/).

Behaviour you will notice

Before After
A silent peer kept its connections for 30 s. It loses them after the RPC timeout (default 5 s). In-flight writes report an unknown outcome.
Background metadata proposals waited 5 s. They wait up to the statement deadline (default 30 s).
A read on a partitioned node was refused at 0.75 s. It is refused at the RPC timeout.

Why

Test Cause
crash_redo_chunk_stream A fixed 750 ms read wait. The boot gate opened before replay finished.
calvin_multi_part_node_delete The same 750 ms read wait.
sql_transactions_kv_overlay A fixed 5 s metadata wait, reported as XX000.

How to check it

cargo nextest run --workspace --exclude nodedb-cluster-tests --all-features --no-fail-fast --retries 0
cargo nextest run -p nodedb-cluster-tests --all-features --no-fail-fast --retries 0
  • Stage 1: 20782 of 20782 pass. Stage 2: 502 of 502 pass.
  • Clippy is clean with all features and with none. Every scripts/ci gate passes.

Not covered here:

  • nodedb-lite must drop the MetaOp::CalvinFlush arm (adapter/meta.rs).
  • nodedb-lite must rename hilbert_range: None to shard_bucket: None (query/visitor/array/query.rs, two sites).
  • cluster_pitr_restore_point once failed on CI with 57014 after restore. The root cause is unconfirmed. It passed in the final run.

Tile keys of a full-range array share their top bits, so routing by the
raw top bits sent every tile to the same few buckets. Routing now takes
the top bits of a mixed hash of the key, shared by the planner, the
cluster coordinator and the Data Plane.
A KV predicate UPDATE/DELETE and a vector-primary write with a row
predicate or patch expression scan state that depends on the clock or on
sidecars newer than the checkpoint being replayed. The proposer now
resolves them and the log record names the rows and their values, so
replay never evaluates a predicate again.
…eadline

Raft, the multi-raft layer, the applier, the gateway and the Calvin
submit path now carry one statement deadline end to end. Election and
snapshot-debt handling move into their own Raft modules, configuration
changes pass a gate, and the apply loop holds, settles and reports an
entry it cannot apply instead of stalling the group. The cluster test
harness and cases cover a hung node, a held replica and slow catalog
DDL.
A data-group snapshot now covers the backup cut barriers and cut
floors the builder applied in the group, so a receiver that never
replays those log entries still holds the state a waiting cut needs.
Install restores them alongside the Calvin applied positions and
dependent-read barrier logs.

Forgetting the Calvin state of vShards a node left now reports its
error and stays pending for retry on the next reconcile pass, and is
held back until boot recovery has replayed every data group.

Add a cluster test that installs a snapshot across a backup cut.
…line through CRDT and gateway paths

The QUIC idle timeout is now the RPC timeout, with keep-alives at a
fifth of it, so a silent peer closes its connection and fails every
pending read on it, while a live peer holds a slow handler open. The
separate keep-alive and idle-timeout tunables are removed, and startup
refuses an RPC timeout below 500 ms.

The transport endpoint moves into its own module. A build with the
failpoints feature can blackhole a node's datagrams to simulate a
partition.

CRDT admission becomes a directory module and, with the DDL, sync,
Raft dispatch and gateway paths, takes a statement deadline instead of
a fixed timeout. Gateway retries are reworked around that deadline, so
the per-route attempts-left parameter goes away.

The cluster test harness gains a convergence wait and the cluster cases
are updated for the new liveness and deadline behaviour.
@farhan-syah farhan-syah added the run-ci Opt this PR into the full test suite; re-add to force a re-run label Oct 11, 2026
@farhan-syah
farhan-syah merged commit 885edc1 into main Oct 12, 2026
13 checks passed
@farhan-syah
farhan-syah deleted the fix/calvin-gaps branch October 12, 2026 00:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci Opt this PR into the full test suite; re-add to force a re-run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant