vicrds - A Peer-To-Peer Collaborative Text Editor using CRDTs

We plan to implement a Peer-To-Peer collaborative text editor leveraging Conflict Free Replicated Data Types (CRDTs). The foundation of the project is a custom Rust CRDT library that will handle the core logic of collaborative editing.

Our CRDT library implementation will handle:

• Decentralized communication between the clients using gRPC protocol.
• Implementation of the CRDT algorithm and conflict resolution.
• Broadcast CRDT updates to other clients.

For text editing, we'll use a sequence CRDT to represent the indices of the characters of the text.

We want to provide a universal API to use in various frontends. The first interface will be a very simple web-based editor. The frontend will talk to the backend over simple web protocols like websockets and JSON.

Implementation Details

We will implement Replicated-Growth Array (RGA), a CRDT that relies on a logical linked list. The idea is that each user types a new character, we send

• ID: globally unique identifier for this new character
• Left Origin: id of the character to its left
• Value: The character
• Is Deleted: tombstone boolean

This gives a total causal ordering via a logical clock for each node that is at minimum the max of all previously seen clock values from other nodes. Concurrent inserts at the same logical location are sorted by a clock value, breaking ties by unique node identifier (ip + port). When deleting text, mark it as deleted (called a tombstone) to preserve the stability of reference anchors.

Once we have the core functionality working we plan to expand it with the following distributed systems features:

1. Dynamic Group Membership (SWIM based)

   • Gossip-based failure detection algorithm where the nodes periodically send pings to random nodes, updating the internal list to suspect and dead after agreed timeout periods.
   • The original paper used UDP, but we will run this over gRPC.
   • Link to the SWIM paper

2. State Transfer for Joining Nodes

   • When a node joins the network (i.e. found another node through autodiscovery) it will request for the entire log (or CRDT snapshot) to update itself.
   • Once it has done so, the node will then behave as normal, receiving normal updates from the rest of the nodes.
   • We get causal ordering by exchanging a lamport clock during message transfer.

3. Distributed Tombstone Compaction

   • We will use a two-phase commit (initiated e.g. by a node with lowest IP) to propose a set of tombstones to remove. Once nodes agree on the compaction the metadata for the previous operations, effectively acting as a garbage collection for the system.

Our proposed set of experiments that define the range of functionality we expect to cover:

• Three (or more) users connect and try use our system to take lecture notes in real time
• User joins after the file/log has grown in size
• Log gets successfully compacted as the system progresses
• Measure latency to join the system and receive all updates as the log of operations increases, measure if tombstone compaction improves this
• Conflicting writes on the same line produce a consistent ordering
• Conflicting writes within the same word produce a consistent ordering
• Create a network partition, edit on both sides of the partition, heal thepartition and see if everything converges to a unified output.

How to Run

Web Editor (Browser Demo)

Each running node automatically spins up an Axum-based web server serving a live collaborative web editor UI. The web server port is calculated as grpc_port + 3000.

To run a multi-user collaborative browser demo:

1. Start two or more nodes:

cargo run --release --bin vicrds -- 1 127.0.0.1:5000
cargo run --release --bin vicrds -- 2 127.0.0.1:5001 127.0.0.1:5000

2. Open your browser and navigate to:

• Node 1 Editor: http://127.0.0.1:8000
• Node 2 Editor: http://127.0.0.1:8001

3. Any edits made in one browser tab will sync in real-time to the other, resolving concurrent conflicts automatically via the underlying RGA CRDT.

Local Cluster

Run a 3-node cluster:

./run_cluster.sh

Or run manually:

cargo run --release --bin vicrds -- <id> [listen_address] [peers]...

Compaction Demo

To demo distributed tombstone compaction, open three terminals and run:

cargo run --release --bin vicrds -- 1 --cli 0.0.0.0:5700                  # terminal 1: founds the document
cargo run --release --bin vicrds -- 2 --cli 0.0.0.0:5701 127.0.0.1:5700   # terminal 2: joins live
cargo run --release --bin vicrds -- 3 --cli 0.0.0.0:5702 127.0.0.1:5700   # terminal 3: joins live

Type with i 0 hello world, delete with d <index>, and watch t (node and tombstone counts) drain to zero on every terminal within a couple of seconds. q retires a node gracefully and compaction continues without it. kill -9 a node instead and t freezes on the survivors while editing keeps working; the frozen count is the tier-1 failure mode and clears only if the dead node returns. kill -STOP / kill -CONT simulates a partition: counts freeze, then drain on heal.

Tests

Run all tests:

cargo test --release

Useful CLI Testing Options:

• <substring>: Filter by test name (e.g., cargo test --release test_swim)
• --test <file_name>: Target a specific integration test file (e.g., cargo test --release --test convergence)
• --test <file_name> -- <substring>: Target a specific test within a specific file (e.g., cargo test --release --test convergence -- test_late_joiner)
• -- --nocapture: Print console output (stdout) from passing tests (e.g., cargo test --release -- --nocapture)
• -- --test-threads=1: Limit concurrent test threads to avoid port conflicts or network congestion (e.g., cargo test --release -- --test-threads=1)
• (Omit --release): Run in debug mode for faster compilation but slower execution (e.g., cargo test)

Distributed GC Testing (YAML Scenario):

To run the automated Docker-container-based distributed garbage collection and tombstone compaction scenario:

cargo run --release --bin container_test -- --scenario scenarios/gc_compaction.yaml

Benchmarks

Run the default benchmark suite:

cargo run --release --bin bench

Configure parameters using CLI options:

• --nodes <NODES>: Comma-separated node counts (default: 3,10,30,60)
• --writers <WRITERS>: Comma-separated writer counts (default: 2,4,8,16,32)
• --doc-sizes <DOC_SIZES>: Comma-separated character counts (default: 100,1000,10000,100000,1000000)
• --gossip-ms <MS>: Gossip interval in ms (default: 150)
• --reconnect-ms <MS>: Reconnect sleep in ms (default: 300)
• --suspect-secs <SECS>: SWIM suspect timeout in seconds (default: 5)
• --append: Append results to the output file instead of overwriting (useful to compare configurations)
• --output <PATH>: Destination path (default: results/benchmark_results.json)
• --gc-secs <SECS>: Seconds of mixed churn for the GC space benchmark; set to 0 to skip (default: 20)
• --gc-output <PATH>: Output path for GC space results (default: results/gc_space.json)

Example running custom sizes:

cargo run --release --bin bench -- --nodes 3,10 --doc-sizes 100,1000,10000

Generate plots (requires matplotlib):

./scripts/plot_results.py ./scenarios/[name].json   # general benchmark plots
python3 scripts/plot_gc.py results/gc_space.json    # GC space plots

YAML-Driven Orchestrator & Scenarios

For advanced validation, failover, and scaling verification, the workspace features a YAML-driven orchestration harness (container_test). It supports running both local-loopback scenarios and fully containerized environments using Docker to emulate real-world WAN conditions (latency, jitter, packet loss) along with concurrent operations (chaos, network partitions, and fuzz edits).

To run a specific scenario:

cargo run --release --bin container_test -- --scenario scenarios/<scenario_name>.yaml

Available Scenarios (located in scenarios/):

• local_benchmark.yaml: The primary scaling benchmark sweeping nodes (from 3 up to 100), writers, and document sizes in local loopback mode.
• fuzz.yaml: Spawns a 5-node cluster under WAN latency/loss, performs concurrent seed-based fuzz inserts/deletes, heals impairments, and asserts total document consistency.
• comprehensive_stress.yaml: A high-intensity fuzzing and chaos sequence with larger network size and operation count.
• partition_heal.yaml: Splits a 6-node cluster into two disconnected partitions ([1, 2, 3] and [4, 5, 6]), allows independent concurrent edits on both sides to force divergence, heals the partition, and verifies the CRDT successfully merges to a single, identical document state.
• swim_convergence.yaml: Evaluates failure detection times and cluster topology synchronization when nodes join or fail under the SWIM gossip protocol.
• wan_latency.yaml: Benchmarks CRDT convergence and data transfer under a static, uniform WAN latency.
• wan_latency_sweep.yaml: Sweeps WAN latency from 0ms up to 300ms in a containerized cluster to measure convergence times.
• wan_loss_sweep.yaml: Sweeps packet loss from 0% to 20% in a containerized cluster to verify RGA durability and convergence under lossy networks.
• wan_scaling.yaml: Sweeps cluster sizes under fixed WAN network conditions.
• gc_compaction.yaml: Verifies distributed garbage collection and tombstone compaction in a 3-node containerized cluster, asserting that tombstones are cleared and nodes are fully garbage-collected after convergence.

Running the Entire Suite

To compile and execute all scenarios (local scaling followed by container sweeps) and generate plots in one step, run either:

./scripts/run_all.sh

or:

./scenarios/run_all.sh

The script runs each scenario, saving JSON data and compiling performance plots (e.g., node scaling, WAN latency sweeps, and WAN loss sweeps) inside the results/ directory.