Repository navigation
Bound cluster waits by deadline and close the Calvin gaps from #414 - #415
Merged
Merged
Conversation
Tile keys of a full-range array share their top bits, so routing by the raw top bits sent every tile to the same few buckets. Routing now takes the top bits of a mixed hash of the key, shared by the planner, the cluster coordinator and the Data Plane.
A KV predicate UPDATE/DELETE and a vector-primary write with a row predicate or patch expression scan state that depends on the clock or on sidecars newer than the checkpoint being replayed. The proposer now resolves them and the log record names the rows and their values, so replay never evaluates a predicate again.
…eadline Raft, the multi-raft layer, the applier, the gateway and the Calvin submit path now carry one statement deadline end to end. Election and snapshot-debt handling move into their own Raft modules, configuration changes pass a gate, and the apply loop holds, settles and reports an entry it cannot apply instead of stalling the group. The cluster test harness and cases cover a hung node, a held replica and slow catalog DDL.
A data-group snapshot now covers the backup cut barriers and cut floors the builder applied in the group, so a receiver that never replays those log entries still holds the state a waiting cut needs. Install restores them alongside the Calvin applied positions and dependent-read barrier logs. Forgetting the Calvin state of vShards a node left now reports its error and stays pending for retry on the next reconcile pass, and is held back until boot recovery has replayed every data group. Add a cluster test that installs a snapshot across a backup cut.
…line through CRDT and gateway paths The QUIC idle timeout is now the RPC timeout, with keep-alives at a fifth of it, so a silent peer closes its connection and fails every pending read on it, while a live peer holds a slow handler open. The separate keep-alive and idle-timeout tunables are removed, and startup refuses an RPC timeout below 500 ms. The transport endpoint moves into its own module. A build with the failpoints feature can blackhole a node's datagrams to simulate a partition. CRDT admission becomes a directory module and, with the DDL, sync, Raft dispatch and gateway paths, takes a statement deadline instead of a fixed timeout. Gateway retries are reworked around that deadline, so the per-route attempts-left parameter goes away. The cluster test harness gains a convergence wait and the cluster cases are updated for the new liveness and deadline behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Breaking
quic_keep_alive_secsandquic_idle_timeout_secsare removed. Both derive fromrpc_timeout_secs.nodedb-types/src/config/tuning/network.rsfmix64(tile_key).ShardBucketreplaces the Hilbert-range shard filter.nodedb-types/src/array_route.rsnodedb/src/control/write_resolveCluster waits
nodedb-cluster/src/transport: QUIC idle timeout =rpc_timeout_secs. Keep-alive = a fifth of it.MIN_RPC_TIMEOUT(500 ms).Correctness
nodedb/src/types/snapshot_calvin.rs).LockMode::IntentSharedon their collection. TRUNCATE invalidates point and index reads.nodedb-cluster/src/calvin/sequencer/validator/).Behaviour you will notice
Why
crash_redo_chunk_streamcalvin_multi_part_node_deletesql_transactions_kv_overlayHow to check it
scripts/cigate passes.Not covered here:
MetaOp::CalvinFlusharm (adapter/meta.rs).hilbert_range: Nonetoshard_bucket: None(query/visitor/array/query.rs, two sites).cluster_pitr_restore_pointonce failed on CI with 57014 after restore. The root cause is unconfirmed. It passed in the final run.