fix(replication): drain priority sync queue and recover from routing-event lag#165
Open
mickvandijke wants to merge 39 commits into
Open
fix(replication): drain priority sync queue and recover from routing-event lag#165mickvandijke wants to merge 39 commits into
mickvandijke wants to merge 39 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR improves replication convergence under churn by (a) ensuring neighbor sync drains queued priority peers without waiting for periodic ticks and (b) recovering from lagged DHT routing-table broadcast events. It also expands verification/fetch pipeline behavior (batching caps, retry/defer semantics, paid-list edge-voter quorum handling) and adds an e2e scenario covering paid-list-authorized repair below storage quorum.
Changes:
- Neighbor-sync loop now drains the priority queue back-to-back and resyncs close peers on
RecvError::Lagged. - Verification/fetch pipeline enhancements: bounded verification batches, fetch→verification retry metadata, and deferred re-verification scheduling.
- Paid-list quorum evaluation updated with “edge voter” handling; adds a deterministic paid-list repair e2e scenario and bumps crate version.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/poc_d1_bounded_queues.rs | Updates test VerificationEntry construction for new timing fields. |
| tests/poc_bootstrap_stall.rs | Updates test VerificationEntry construction for new timing fields. |
| tests/e2e/replication.rs | Adds deterministic e2e scenario for paid-list-majority repair under low storage quorum. |
| src/replication/types.rs | Adds next_verify_at, fetch retry metadata, and NeighborSyncState::has_priority_peers + test. |
| src/replication/scheduling.rs | Adds deferred pending scheduling, fetch retry→verification requeue, and returns evicted keys. |
| src/replication/quorum.rs | Adds edge-aware paid-list vote summary and splits verification requests into capped batches. |
| src/replication/mod.rs | Implements lag recovery, neighbor-sync drain-before-park, verification request caps, and retry/defer integration. |
| src/replication/config.rs | Introduces PAID_LIST_FLEX_EDGE_COUNT and MAX_VERIFICATION_KEYS_PER_REQUEST. |
| src/replication/bootstrap.rs | Updates test VerificationEntry construction for new timing fields. |
| src/replication/admission.rs | Adjusts cross-set dedup/admission so paid hints can survive replica rejection under churn. |
| Cargo.toml | Bumps version to 0.14.3. |
| Cargo.lock | Updates locked crate version to 0.14.3. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
2543
to
2547
| results.push(protocol::KeyVerificationResult { | ||
| key: *key, | ||
| present, | ||
| present: cached.present.unwrap_or(false), | ||
| paid, | ||
| }); |
| } | ||
| continue; | ||
| } | ||
| Err(RecvError::Closed) => continue, |
Comment on lines
+495
to
+497
| let handles = | ||
| spawn_verification_batch_tasks(targets, p2p_node, config.verification_request_timeout); | ||
| collect_verification_batch_results(handles, targets, &mut evidence).await; |
Comment on lines
+43
to
+47
| /// Number of furthest paid-list close-group peers treated as churny edge | ||
| /// voters. | ||
| /// | ||
| /// Once the paid-list close group reaches [`PAID_LIST_CLOSE_GROUP_SIZE`], edge | ||
| /// peers are queried, but a negative edge paid-list response does not count |
mickvandijke
force-pushed
the
fix/neighbor-sync-drain-window
branch
from
July 2, 2026 12:59
1a6fe63 to
fc8d724
Compare
Comment on lines
+1284
to
+1288
| warn!( | ||
| "Prune audit challenge for {} against {peer} could not acquire coordinator slot", | ||
| hex::encode(key) | ||
| ); | ||
| return PruneAuditChallengeResult::MalformedResponse; |
Comment on lines
+2837
to
2841
| if cached.present.is_none() && paid.is_none() { | ||
| continue; | ||
| } | ||
|
|
||
| results.push(protocol::KeyVerificationResult { |
Comment on lines
18
to
21
| pub mod audit; | ||
| pub mod audit_coordinator; | ||
| pub(crate) mod audit_metrics; | ||
| pub mod bootstrap; |
Comment on lines
+132
to
+134
| fn fresh_offer_key_lock_index(key: &XorName) -> usize { | ||
| usize::from(key[0]) % FRESH_OFFER_KEY_LOCK_SHARDS | ||
| } |
Comment on lines
+1283
to
+1289
| let Some(_slot) = audit_challenge_coordinator.acquire(*peer).await else { | ||
| warn!( | ||
| "Prune audit challenge for {} against {peer} could not acquire coordinator slot", | ||
| hex::encode(key) | ||
| ); | ||
| return PruneAuditChallengeResult::MalformedResponse; | ||
| }; |
mickvandijke
force-pushed
the
fix/neighbor-sync-drain-window
branch
from
July 14, 2026 09:39
74c990e to
b8d490d
Compare
Comment on lines
+631
to
+633
| self.insert_pending_unchecked(key, verification); | ||
| self.release_retry_slot(&sender); | ||
| true |
Comment on lines
+53
to
+54
| #[cfg(feature = "logging")] | ||
| impl AuditType { |
Comment on lines
+77
to
+78
| #[cfg(feature = "logging")] | ||
| impl AuditResponderClass { |
Comment on lines
+1283
to
+1289
| let Some(_slot) = audit_challenge_coordinator.acquire(*peer).await else { | ||
| warn!( | ||
| "Prune audit challenge for {} against {peer} could not acquire coordinator slot", | ||
| hex::encode(key) | ||
| ); | ||
| return PruneAuditChallengeResult::MalformedResponse; | ||
| }; |
Comment on lines
+204
to
+209
| if let Some(retry) = &mut candidate.retry_verification { | ||
| retry | ||
| .replica_hint_sources | ||
| .extend(entry.replica_hint_sources); | ||
| retry.hint_sources.extend(entry.hint_sources); | ||
| } |
Comment on lines
+222
to
+227
| if let Some(retry) = &mut in_flight.retry_verification { | ||
| retry | ||
| .replica_hint_sources | ||
| .extend(entry.replica_hint_sources); | ||
| retry.hint_sources.extend(entry.hint_sources); | ||
| } |
Comment on lines
+1283
to
+1289
| let Some(_slot) = audit_challenge_coordinator.acquire(*peer).await else { | ||
| warn!( | ||
| "Prune audit challenge for {} against {peer} could not acquire coordinator slot", | ||
| hex::encode(key) | ||
| ); | ||
| return PruneAuditChallengeResult::MalformedResponse; | ||
| }; |
Comment on lines
+110
to
+114
| enum PruneAuditChallengeResult { | ||
| Response(Box<ReplicationMessage>), | ||
| NoResponse(AuditFailureClass), | ||
| MalformedResponse, | ||
| } |
Comment on lines
+1183
to
+1187
| PruneAuditChallengeResult::Response(decoded) => *decoded, | ||
| PruneAuditChallengeResult::NoResponse(class) => { | ||
| // No response means an immediate audit failure, but keep the local | ||
| // class split so timeout metrics are not polluted by pre-delivery | ||
| // failures. |
Comment on lines
+1283
to
+1289
| let Some(_slot) = audit_challenge_coordinator.acquire(*peer).await else { | ||
| warn!( | ||
| "Prune audit challenge for {} against {peer} could not acquire coordinator slot", | ||
| hex::encode(key) | ||
| ); | ||
| return PruneAuditChallengeResult::MalformedResponse; | ||
| }; |
Comment on lines
+321
to
+326
| if let Some(keys) = self.pending_keys_by_source.get_mut(source) { | ||
| keys.remove(key); | ||
| if keys.is_empty() { | ||
| self.pending_keys_by_source.remove(source); | ||
| } | ||
| } |
Comment on lines
+115
to
+121
| let Some(entry) = targets.get_mut(&peer) else { | ||
| return; | ||
| }; | ||
| entry.references = entry.references.saturating_sub(1); | ||
| if entry.references == 0 { | ||
| targets.remove(&peer); | ||
| } |
Comment on lines
+53
to
+54
| #[cfg(feature = "logging")] | ||
| impl AuditType { |
Comment on lines
+77
to
+78
| #[cfg(feature = "logging")] | ||
| impl AuditResponderClass { |
Comment on lines
+153
to
157
| // Cross-set precedence: the replica pass already ruled on this key, | ||
| // under the same gate this pass would apply. Re-asking cannot change | ||
| // the answer, and re-recording it would double-count the rejection. | ||
| if seen_replica.contains(&key) { | ||
| continue; |
Comment on lines
+1283
to
+1289
| let Some(_slot) = audit_challenge_coordinator.acquire(*peer).await else { | ||
| warn!( | ||
| "Prune audit challenge for {} against {peer} could not acquire coordinator slot", | ||
| hex::encode(key) | ||
| ); | ||
| return PruneAuditChallengeResult::MalformedResponse; | ||
| }; |
Size bootstrap capacity-rejection expiry from the full configured sync cycle, await aborted engine tasks before claiming shutdown quiescence, and apply paid-list edge flexibility against the runtime group width. Also fixes the all-target Clippy gate and adds regression coverage.\n\nSemVer: patch
Comment on lines
560
to
564
| let keys = self | ||
| .blocking_tracker | ||
| .spawn_blocking(move || -> Result<Vec<XorName>> { | ||
| let rtxn = env | ||
| .read_txn() |
Comment on lines
+605
to
+609
| let value = self | ||
| .blocking_tracker | ||
| .spawn_blocking(move || -> Result<Option<Vec<u8>>> { | ||
| let rtxn = env | ||
| .read_txn() |
Comment on lines
+1031
to
+1033
| let config = ReplicationConfig::default(); | ||
| let batches = config.neighbor_sync_scope / config.neighbor_sync_peer_count; | ||
| let cycle_cadence = config.neighbor_sync_interval_max * u32::try_from(batches).unwrap(); |
Comment on lines
+1283
to
+1287
| let Some(_slot) = audit_challenge_coordinator.acquire(*peer).await else { | ||
| warn!( | ||
| "Prune audit challenge for {} against {peer} could not acquire coordinator slot", | ||
| hex::encode(key) | ||
| ); |
Comment on lines
+279
to
+283
| fn paid_hint_survives_duplicate_replica_rejection() { | ||
| // With churned local views, a sender may believe we are in the storage | ||
| // close group while our own view rejects the replica hint. If the same | ||
| // key also arrives as a paid hint, paid-list admission must still run. | ||
| let key = xor_name_from_byte(0xEE); |
mickvandijke
force-pushed
the
fix/neighbor-sync-drain-window
branch
from
July 16, 2026 14:09
8d73642 to
e0c0385
Compare
Comment on lines
+1544
to
+1549
| let Some(_slot) = audit_challenge_coordinator.acquire(*peer).await else { | ||
| warn!( | ||
| "Prune audit challenge with {key_count} keys against {peer} could not acquire coordinator slot" | ||
| ); | ||
| return PruneAuditChallengeResult::MalformedResponse; | ||
| }; |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 23 out of 24 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
src/replication/admission.rs:283
- The unit test
paid_hint_survives_duplicate_replica_rejectionmodels behavior thatadmit_hintsno longer implements. In production, the paid-hints loop unconditionally skips any key that appeared inreplica_hints(seen_replica.contains(&key)), and both loops use the sameis_relevantgate, so a “replica rejected but paid admitted” cross-set outcome is currently unrepresentable. This makes the test (and its doc comment) misleading and likely to confuse future changes.
#[test]
fn paid_hint_survives_duplicate_replica_rejection() {
// With churned local views, a sender may believe we are in the storage
// close group while our own view rejects the replica hint. If the same
// key also arrives as a paid hint, paid-list admission must still run.
let key = xor_name_from_byte(0xEE);
src/replication/pruning.rs:1559
audit_challenge_coordinator.acquire(*peer).awaitonly returnsNoneif the underlying semaphore was closed (it otherwise waits), but this branch treats it as an inability to get a “slot” and returnsMalformedResponse.receive_prune_audit_responsethen reports a prune-audit trust failure for every challenged key, which would incorrectly penalize the remote peer for a local internal error/closure. Consider adding a distinct result for local coordinator failure (handled as “skip audit attempt” without reporting trust), and reserveMalformedResponsefor actual malformed remote replies.
let Some(_slot) = audit_challenge_coordinator.acquire(*peer).await else {
warn!(
"Prune audit challenge with {key_count} keys against {peer} could not acquire coordinator slot"
);
return PruneAuditChallengeResult::MalformedResponse;
};
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 23 out of 24 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
src/replication/pruning.rs:1559
send_prune_audit_challengetreats a failure to acquire anAuditChallengeCoordinatorpermit asMalformedResponse. That result is later interpreted as an audit failure and can trigger a trust penalty (viareceive_prune_audit_response→report_prune_audit_failure_once), even though no request was sent and the peer did nothing wrong. This is also inconsistent with the responsible audit and possession check paths, which treat coordinator-acquire failure as a local/inconclusive condition (no penalty).
Consider introducing a distinct result for local admission failure (or returning early from receive_prune_audit_response without reporting) so coordinator issues never get misattributed to the remote peer.
let Some(_slot) = audit_challenge_coordinator.acquire(*peer).await else {
warn!(
"Prune audit challenge with {key_count} keys against {peer} could not acquire coordinator slot"
);
return PruneAuditChallengeResult::MalformedResponse;
};
all_keys() and get_raw() opened LMDB read transactions without holding the shared env_lock, while try_resize() takes the exclusive env_lock and calls unsafe env.resize() on the assumption that no transactions are active. A map-full write could therefore resize the memory map out from under an active audit/pruning/commitment raw read, violating LMDB's documented safety precondition. Acquire env_lock.read() at the top of both raw-read blocking closures, matching get()/exists()/delete()/current_chunks()/try_put(). This keeps the resize-safety invariant that ADR-0005 relies on and composes cleanly with the shutdown wait_idle() drain (the closures are still tracked; the guard lives entirely within the tracked closure). Add resize_waits_for_in_flight_raw_read, which parks a detached get_raw holding the shared lock and proves try_resize() blocks until the read releases. Backed by a test-only test_read_gate mirroring test_put_gate. Verified the test fails without the guard (raw read never takes the shared lock) and passes with it. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Use elastic max-min admission and sender-fair verification service while allowing peers to borrow otherwise unused global capacity. Track displaced bootstrap hints for redelivery and cover adversarial flooding, large bootstrap batches, retries, and ownership transfer.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 23 out of 24 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
src/replication/admission.rs:157
- The paid-hint loop currently skips any key that appeared in
replica_hints(seen_replica.contains(&key)), even if that replica hint was rejected. That makes a duplicate paid hint unable to “rescue” a rejected replica hint, and it can also drop work if the relevance fast-paths (local storage / paid list) change between the two passes. Cross-set precedence should only suppress the paid hint when the key was actually admitted as a replica hint, and overlapping rejections should not be double-counted.
// Cross-set precedence: the replica pass already ruled on this key,
// under the same gate this pass would apply. Re-asking cannot change
// the answer, and re-recording it would double-count the rejection.
if seen_replica.contains(&key) {
continue;
Comment on lines
+211
to
+226
| let now = Instant::now(); | ||
| let mut state = bootstrap_state.write().await; | ||
| let before = state.capacity_rejected_sources.len(); | ||
| state | ||
| .capacity_rejected_sources | ||
| .retain(|source, rejected_at| { | ||
| let expired = now.duration_since(*rejected_at) >= max_age; | ||
| if expired { | ||
| warn!( | ||
| "Bootstrap: expiring capacity-rejection record for {source} — the source \ | ||
| abandoned re-delivery (or departed mid-admission) and the hints it still \ | ||
| owed are forfeited" | ||
| ); | ||
| } | ||
| !expired | ||
| }); |
Comment on lines
+12
to
+16
| /// This intentionally matches the per-source digest admission cap enforced by | ||
| /// the already-deployed fleet. Waiting here happens before the response | ||
| /// deadline starts, so excess local bursts are serialized instead of being | ||
| /// converted into guaranteed remote admission drops and false timeouts. | ||
| pub(crate) const MAX_CONCURRENT_AUDIT_CHALLENGES_PER_TARGET: usize = 2; |
Comment on lines
+1554
to
+1559
| let Some(_slot) = audit_challenge_coordinator.acquire(*peer).await else { | ||
| warn!( | ||
| "Prune audit challenge with {key_count} keys against {peer} could not acquire coordinator slot" | ||
| ); | ||
| return PruneAuditChallengeResult::MalformedResponse; | ||
| }; |
Extract the audit responder's global-permit and per-peer-slot admission into a reusable helper while preserving the existing audit call sites. Keep the provisional peer slot under RAII so guard drops and cancelled admission waits cannot leak capacity.
Admit fetch requests through a flood-fair bounded window, then serve them on three bandwidth-limited workers. Shed requests at worker dequeue once the caller's fetch deadline has elapsed, so stale 4 MiB reads and uploads cannot sustain overload.
Serve inbound verification batches on two bounded workers with a single outstanding batch allowed per source. Drop expired batches at worker dequeue so synchronous LMDB point lookups cannot monopolize the serial replication lane after the requester deadline.
Serve inbound neighbor sync on two bounded workers and admit at most one request per source, preserving the sync-history-before-repair-proof invariant while allowing different peers to progress concurrently. Shed work at dequeue against the existing verification request deadline.\n\nVerified cross-peer interleaving is safe: admit_and_queue_hints publishes through the queues RwLock, publish_bootstrap_admission_outcomes applies each completed outcome under the bootstrap-state RwLock, and sync history, repair proofs, and cycle state are independently lock-guarded. No cross-peer call ordering is assumed.
Never execute replication handlers inline when the bounded serial handoff is full or closed. Classify and warn for the dropped message with queue depth, relying on each protocol class's existing retry or convergence path so the P2P event receiver remains responsive.
Add relaxed atomic counters for fetch, verification, and neighbor-sync admission drops and staleness sheds. Count serial-queue drops independently for every existing replication wire variant, reusing the protocol's variant index and preserving queue-depth/class logging.
Log logical audit issuers and responder source peers, capacity decisions, stage timings, and minute-window top-origin summaries for the shared audit response pool. SemVer: minor
Carry the exact responsive possession validation failure through the probe verdict and emit it as the structured possession_failure_reason field. Cover stable labels and each existing validation branch without changing trust or penalty behavior. SemVer: minor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 25 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
src/replication/storage_commitment_audit.rs:161
- The doc comment says this is a "gossip-triggered" audit, but
run_subtree_auditnow delegates torun_subtree_audit_with_origin(..., SubtreeAuditOrigin::Manual). This is misleading for callers (including the test-onlyaudit_peer_now) and contradicts the new origin tagging.
Update the doc comment to describe the function generically (or note that it defaults to Manual) and point gossip/first-monetized callers at run_subtree_audit_with_origin.
/// Run one gossip-triggered subtree audit against `challenged_peer`, pinned to
/// the commitment hash the peer just gossiped (`expected_commitment_hash`).
///
src/replication/pruning.rs:1569
send_prune_audit_challengetreats failure to acquire anAuditChallengeCoordinatorslot asMalformedResponse. Inreceive_prune_audit_response,MalformedResponseleads toreport_prune_audit_failure_once(...)withfailure_class=None, which trust-penalizes the remote peer.
If coordinator acquisition can fail (e.g. if the semaphore is ever closed during shutdown or future refactors), this would incorrectly produce a prune audit failure and a trust penalty even though no request was sent. Consider introducing a distinct result variant (e.g. LocalDrop/NotSent) and having receive_prune_audit_response return early without reporting/penalizing in that case.
let Some(_slot) = audit_challenge_coordinator.acquire(*peer).await else {
warn!(
"Prune audit challenge with {key_count} keys against {peer} could not acquire coordinator slot"
);
return PruneAuditChallengeResult::MalformedResponse;
};
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR hardens replication repair under churn, load, queue pressure, event-stream failure, and shutdown. It expands the original neighbor-sync drain fix into a broader set of paid-list repair, verification, bootstrap accounting, audit concurrency, fresh-offer dispatch, admission gating, and async lifecycle fixes so legitimate repair work is not silently dropped, converted into false audit failures, delayed behind stale peers, left spinning on closed broadcasts, serialized behind a saturated worker pool and dropped through broadcast lag, used to conscript out-of-range nodes into storing arbitrary keys, downloaded and stored under responsibility decisions that topology churn had already invalidated, left holding LMDB/P2P resources — including detached LMDB blocking transactions — after engine shutdown, or wedged in permanent bootstrap by a peer-removal race that orphans capacity-rejection accounting.
The branch contains thirty-nine commits:
4049325-fix(replication): harden paid-list verification repair3d2a22c-fix(replication): tolerate paid-list edge churn24fefa4-fix(replication): drain priority sync queue and recover from routing-event lagb214c8d-fix(replication): preserve verification retry capacityc6abce2-fix(replication): eliminate false audit challenge timeouts71dc001-fix(replication): preserve dequeued retry reservations02032a0-chore(replication): fix audit admission clippy6ea12a5-fix(replication): release cancelled async work63f4b22-fix(replication): silence no-logging audit label warnings37f8e34-fix(replication): unblock bootstrap when rejected peer leaves8f38d4c-refactor(replication): prioritize source-aware hints94705cf-refactor(replication): aggregate bootstrap hint batchesf9263de-fix(replication): aggregate verification per peer8d3834b-fix(replication): drain fresh offers through LMDB writesb577e8e-fix(replication): track detached audit workab848a2-fix(replication): stop message handler when event streams close3e8860c-fix(replication): prune departed peers during DHT lag recoveryfc9dff0-fix(replication): penalize rejected singleton replica hints44bd01d-fix(replication): keep fresh offers off the serial message loop0cda93b-fix(replication): gate replica downloads on storage responsibility92339a1-refactor(replication): admit hints through a single relevance gateb5c0367-perf(replication): merge duplicate hints without rebuilding the fetch heap3a00fef-fix(replication): drain detached LMDB blocking ops on shutdown9cef69f-fix(replication): expire orphaned capacity-rejection records to unstall bootstrap4c8f821-fix(replication): recheck storage responsibility at the point of download14dfd46-docs(adr): record replication repair hardening decisions (PR #165)e0c0385-fix(replication): address deep review findings9951312-fix(replication): resolve all-target clippy failures4ce8e72-fix(docs): avoid private rustdoc link172e97f-fix(storage): guard raw LMDB reads against concurrent map resize86c4910-fix(replication): fairly share verification capacitya40da7b-fix(replication): generalize bounded responder admissionec8eb6a-fix(replication): isolate fetch responders864a5b4-fix(replication): isolate verification responders57873d0-fix(replication): isolate neighbor sync responders7b5ee55-fix(replication): drop serial queue overflow48db2d5-fix(replication): count responder load shedding4ef753b-feat(replication): trace audit request origins and latency339f44b-feat(replication): log possession proof failure reasonsProblems Addressed
Paid-list repair could become terminal too early. Duplicate replica/paid hints were deduplicated before their actual admission outcome was known, and verified repair work could lose its retry context after transient no-holder or no-source rounds.
Paid-list edge peers were too strict under churn. Boundary disagreement could reject a valid majority formed by the stable core of the paid close group.
Neighbor sync could stall after topology bursts.
Notifycoalescing allowed queued priority peers to wait for later 10-20 minute periodic ticks, while lagged DHT broadcasts could hide entrants entirely or leave departed peers ahead of current neighbors, each consuming a request timeout.Verification retry capacity could be stolen. Promoted verified keys stopped counting against pending capacity, allowing unrelated hints to consume the capacity required to requeue them.
Verification work lacked source-aware prioritisation. Duplicate hints discarded useful corroborating-source information, and large ready queues could schedule weak singleton claims ahead of better-supported repair work.
Bootstrap hint batches were published incrementally. Verification could race a partially admitted neighbor-sync batch and miss the complete source picture.
Verification fan-out was unnecessarily fragmented. Per-peer key sets were split into many requests despite already having a bounded verification cycle.
Bootstrap could remain blocked by departed peers. A peer that caused an admission-capacity rejection could leave while its rejection marker continued preventing drain.
Local audit bursts could look like honest-peer timeouts. Independent audit issuers could exceed the responder's per-source admission limit and interpret the resulting drops as remote failure.
Cancelled fresh offers could detach LMDB writes. Dropping an async
spawn_blockingwaiter does not cancel the blocking transaction, so shutdown could report drained while LMDB still owned the environment.Detached responder/audit work was outside engine lifecycle tracking. Digest, subtree, byte, possession-check, and audit-launch tasks could outlive engine shutdown while retaining storage or P2P state.
Closed event streams could spin the replication loop. Closed Tokio broadcast receivers remain immediately ready forever; continuing after
RecvError::Closedcould consume a core, flood P2P warnings, and prevent the replication pipeline from shutting down cleanly.Sole-source replica hints could avoid trust penalties. A definitive close-group rejection was only penalized when the advertising peer also explicitly denied possession, allowing unsupported free-replication claims to escape punishment.
Fresh-offer dispatch could serialize on the non-audit loop. Once all four worker permits were held, dispatch handled the next offer — an on-chain payment verification plus a multi-MiB LMDB write — inline on the serial message loop. Per-key shard locks (
key[0] % 64) collapsed the close-prefix accepted-key set onto a single shard, making the inline path the steady state, backing up the 256-slot inbound queue, and dropping replication messages wholesale once the P2P event loop lagged its broadcast receiver.Replica hints could conscript out-of-range nodes into storage.
HintPipeline::Replicaskipped theis_responsible(storage_admission_width)gate that paid keys had to pass, and a stored pipeline tag let a second replica message escalate a queued paid entry through thealready_pendingfast path, so a peer could name any key and force nodes ranked 10-20 for it to fetch and store it.Duplicate hint merges rebuilt the whole fetch heap. Merging a re-advertised key's advertiser only touches a field the heap never orders on, but the old path took the whole heap, scanned it linearly, and re-heapified — O(n) per duplicate and O(m·n) for a batch under the global queue write lock, which a neighbor could trigger deliberately by re-hinting queued keys.
Shutdown could return while detached LMDB transactions were still running. Several paths race a
select!on the shutdown token against futures awaiting aspawn_blockingLMDB transaction — the fetch worker'sstorage.put, the prune pass'sstorage.delete/paid_list.remove_batch, the verification worker'spaid_list.insert. Dropping the losing future does not cancel the blocking closure, which keeps running with a clonedEnv; per-fetch tasks were also bare-spawned outside both trackers, so a droppedin_flightset could leakArc<LmdbStorage>pastshutdown(). Reopening the same environment afterwards was undefined behavior.Bootstrap drain could stall permanently on a peer-removal race. A capacity-rejection record for source P was cleared only by P's next clean admission cycle or by P's
PeerRemovedcleanup — but the note sites and the removal handler run on different tokio tasks, with await points between the last "P is live" observation and the insert. A removal fully processed inside that window made its clear a no-op, and the subsequent note recorded an entry no future event could retire:check_bootstrap_drainedreturned false forever, audits stayed disabled (Invariant 19), and the node advertisedbootstrapping: trueindefinitely, drawing network-wide bootstrap-claim trust penalties — reachable deliberately by overflowingpending_verifyduring a victim's bootstrap and disconnecting. Adjacent liveness gap: every drain check was event-driven and a clean-cycle clear never re-checked, so a quiet node could satisfy the drain condition with nothing left to observe it.Fetch decisions could go stale between promotion and download. Storage responsibility was checked once, at verification-completion time, and never again before the chunk was downloaded and stored. The fetch queue holds up to 131,072 entries and dequeues nearest-first, so a far candidate can wait unboundedly long while closer keys jump ahead, and every per-source retry reused the same stale answer — topology churn after promotion still ended in a download, a disk write, and fetch→store→prune churn, violating the contract documented in
types.rsandadmission.rsthat responsibility is decided against live routing state at the point of download.Raw LMDB reads could race concurrent map resize.
all_keys()andget_raw()opened read transactions without the shared environment lock, allowing an exclusiveenv.resize()to change the memory map while those transactions were active.One routing peer could monopolize verification capacity. The global-only 131,072-entry admission bound let one peer fill the queue and continuously consume the 8,192-key verification cycle, rejecting honest hints before source-count prioritization could help; restoring a fixed 8,192-per-peer quota would instead truncate legitimate bootstrap batches containing tens of thousands of hints.
Bulk replication responders could monopolize the serial message loop. Fetch responses perform large LMDB reads and uploads, while verification and neighbor-sync responses perform synchronous point-lookups and queue publication; handling them directly let one source delay unrelated protocol traffic.
Serial-queue backpressure could defeat its own bound. When the bounded handoff filled or closed, the fallback executed the rejected handler inline on the P2P event receiver, allowing overload to propagate into broadcast lag and wider message loss.
Responder load shedding and audit latency lacked actionable attribution. Admission drops, stale dequeues, logical audit issuers, remote sources, and stage timings were not available as bounded counters and structured summaries.
Responsive possession-proof failures lost their precise cause. Validation branches collapsed into a generic failed verdict, making malformed proofs, missing records, and commitment mismatches indistinguishable in logs even though trust behavior correctly remained the same.
Changes
Paid-list verification and repair
pending_verify, fetch queue, and in-flight fetch state.Paid-list edge churn
Neighbor sync and bootstrap lifecycle
HashMap<PeerId, Instant>instead of a bare set) and expires records only after a runtime-derived full-cycle window covering every neighbor batch, per-peer request deadlines, cooldown, and one slow-cadence interval of slack (125 minutes with defaults), forfeiting abandoned/departed-source debt consistently with peer-removal cleanup.pending_peer_requestsearly-returns: pending requests legitimately block the drain check itself but no longer block expiry, and a drain condition that became true without a triggering event is now observed within one tick.Source-aware bounded verification
Fresh-offer dispatch, admission gating, and fetch-queue merges
handle_fresh_offerhas a single caller and no inline verification/LMDB path on the serial loop.key[0] % 64shard locks with an exact per-key in-flight set behind an RAII guard, so unrelated keys never contend and concurrent duplicates collapse onto the first claimant instead of repeating its verification.replica_hint_sourcesinstead of storing the tag, deleting the paid→replica escalation and both demotion sites.is_responsible(storage_admission_width)at both fetch sites (local paid-list fast path and post-verification path), matching pruning width; fresh PUTs keep their wider accept window.paid_list_close_group_size(20), so mislabelling a hint gains nothing and the two-gate rescue dance (rejected_replicarescued byadmitted_paid) is removed; the admissible key set is unchanged.FetchCandidateintoFetchOrder(key + distance, the only fields theOrdimpl reads) held in the heap andFetchPayload(sources + retry metadata) held in a key-indexed map that also serves as the membership index, making a duplicate-source merge an O(1) map lookup and rebuilding the heap only when a departed peer actually orphans a candidate.is_responsible(storage_admission_width)on every fetch attempt insideexecute_single_fetch— before spending bandwidth (per-source retries re-enter there, so they are covered) and once more beforestorage.put, so bytes arriving after responsibility lapsed mid-round-trip are not written. A lapsed attempt resolves asFetchResult::NoLongerResponsible, which sharesStored's terminal path (retry-slot release plus bootstrap accounting) and reports no trust event; the verification-time check remains as a cheap pre-filter that keeps never-responsible keys out of the fetch queue.apply_fetch_result, so eachFetchResultvariant's queue transition is unit-testable without a live network.Audit concurrency and observability
AuditChallengeCoordinatoracross responsible, prune-confirmation, and possession audits.Bounded responder isolation and diagnostics
possession_failure_reasonfield without changing evidence, trust, or penalty semantics.Storage resize safety
env_lock.read()inside the tracked blocking closures forall_keys()andget_raw(), matching the other raw storage operations.env.resize()operation while preserving cancellation and shutdown tracking semantics.Event-stream and detached-task lifecycle
LmdbStorageandPaidListroute everyspawn_blockingthrough per-instanceTaskTrackers and exposewait_idle(), so a dropped async awaiter no longer untracks a live transaction (constructor-time opens stay untracked — they cannot outlive the constructor).shutdown()awaits timed-out long-lived tasks after requesting abort, then awaitswait_idle()on both environments after the detached-task drain; when it returns, producer tasks have actually exited, no LMDB blocking operation is still running, and no engine-spawned task holdsArc<LmdbStorage>/Arc<PaidList>.tokio::spawn; thein_flightplumbing and prompt network-I/O cancellation are unchanged — only the bounded in-flight LMDB transaction is awaited.Latest Commit Details
e0c0385- address deep review findingsCloses four review findings before merge. Bootstrap capacity-rejection expiry is now derived from the runtime neighbor scope, batch width, slowest cadence, per-peer request deadline, cooldown, and one cadence interval of slack; with defaults the window is 125 minutes rather than 70, so a live source near the end of a five-batch cycle is not forfeited before its next legitimate re-hint.
shutdown()now awaits a long-lived task after requestingabort, ensuring synchronous sections have actually exited and dropped storage/P2P clones before detached work and LMDB trackers are declared quiescent. Paid-list flexible-edge activation now usesReplicationConfig::paid_list_close_group_size, preserving strict majority for an undersized 20/24 group and enabling the intended four-edge tolerance for a full non-default 16-peer group. The commit also fixes the all-target/all-feature Clippy gate and adds focused regression coverage. Validation: 738 all-feature library tests, 434 no-default-feature replication tests, the shutdown-LMDB PoC, formatting, diff checks, and the exact CI Clippy command all pass.14dfd46- record replication repair hardening decisionsAdds ADR-0005, documenting the repair-hardening decisions, invariants, operational trade-offs, and lifecycle guarantees implemented by this PR.
4c8f821- recheck storage responsibility at the point of download0cda93bgated replica downloads onis_responsible(storage_admission_width), but only at verification-completion time — the answer was never re-asked before the chunk was actually downloaded and stored. The fetch queue is nearest-first and up to 131,072 entries deep, so a far candidate can wait unboundedly long while closer keys jump ahead, and the per-source retry path reused the same stale decision for every fallback source. Topology churn between promotion and download therefore still ended in a download, a disk write, and later fetch→store→prune churn once pruning evicted the record at the same width — whiletypes.rsandadmission.rsboth documented that responsibility is decided against live routing state at the point of download. Responsibility is now rechecked per fetch attempt insideexecute_single_fetch, where no queue lock is held: once at the top before spending bandwidth (retries re-enter there), and once more beforestorage.putso bytes that arrive after responsibility lapsed mid-round-trip are not written; edge flapping is dampened by the marginstorage_admission_widthalready adds overclose_group_size. A lapsed attempt resolves as the newFetchResult::NoLongerResponsible, which deliberately sharesStored's terminal path —complete_fetchreleases the verification retry-slot reservation and the worker's terminal handling shrinks the bootstrap pending set, so a declined key cannot stall bootstrap drain — and reports no trust event, since the source did nothing wrong. The promotion-time check stays as a cheap pre-filter; event-driven purging of the fetch queue on routing events was considered and rejected (O(queue) scans per churn event, racy, strictly more complex than the lazy recheck). The worker's result handling is extracted intoapply_fetch_resultfor direct unit coverage, and a new e2e drives a live 12-node network through a test-only seam that enqueues a fetch candidate for a key the target is not responsible for — the exact state a stale promotion leaves behind, since live topology cannot be shifted deterministically inside the millisecond promotion→dequeue window — asserting the chunk is never stored and the key exits the pipeline terminally, with an in-responsibility positive control through the same seam and holder. With the rechecks disabled the e2e fails by storing the stale chunk, confirming it discriminates.9cef69f- expire orphaned capacity-rejection records to unstall bootstrap37f8e34madePeerRemovedretire a departed source's capacity-rejection record, but the record's lifecycle was still purely event-driven and the note/removal paths run on different tokio tasks. Each note path has await points (LMDB reads, lock acquisitions) between its last "source is live" observation and the insert, so a removal fully processed inside that window cleared nothing and the subsequent note recorded an entry that no later admission cycle or removal event could ever retire — permanently blocking drain, disabling all audits (Invariant 19), and leaving the node claiming bootstrap network-wide, where other peers' bootstrap-claim abuse logic trust-penalizes it. An attacker could reach this deliberately by overflowing a victim'spending_verifyduring bootstrap and disconnecting immediately. The record now carries its most recent rejection time and expires after a runtime-derived full neighbor-cycle window (all configured batches and request deadlines, the cooldown floor, plus one slow-cadence interval of slack; 125 minutes with defaults), with expiry and a drain re-check run from the verification worker tick ahead of the pending-request early-returns. That same tick closes the adjacent liveness gap where a clean-cycle clear satisfied the drain condition with no event left to observe it. Expiry forfeits the keys the departed source still owed — consistent withupdate_bootstrap_after_peer_removed, which already forfeits a departed source's owed work wholesale; post-bootstrap neighbor sync and audit/repair recover them. Deliberately not a routing-table membership check at the note sites (only shrinks the TOCTOU window) and not a global bootstrap deadline (would change Invariant 19 semantics for genuinely busy bootstraps). Regression coverage reproduces the exact race ordering: removal cleanup first as a no-op, the rejection recorded after, drain asserted blocked, then TTL expiry drains it.44bd01d- keep fresh offers off the serial message loopOnce all four worker permits were held,
dispatch_fresh_offerhandled the next offer — an on-chain payment verification and a multi-MiB LMDB write — inline on the serial non-audit loop, backing up the 256-slot inbound queue until the P2P event loop also handled messages inline and its broadcast receiver lagged and dropped replication messages wholesale. The per-key shard locks (key[0] % 64) made that the steady state rather than the exception: a node only receives fresh offers for keys close to its own ID, so the accepted key set shares a long prefix and nearly every offer landed on one shard. The ordering those locks preserved has no meaning here, since the key is the content address of the data. Dispatch now only claims the key, takes an admission permit, and spawns; the worker permit is awaited inside the task sohandle_fresh_offerhas a single caller and no inline fallback exists. A 16-permit admission semaphore bounds outstanding offers at a 64 MiB ceiling, and an exact per-key in-flight set behind an RAII guard replaces the shard locks so unrelated keys never contend and concurrent duplicates collapse onto the first claimant. Past the bound an offer is refused — not stored, and therefore penalized as absent by the delayed possession check — rather than queued or handled inline.0cda93b- gate replica downloads on storage responsibilityA replica hint is a possession claim, but the receiver treated it as authorization:
HintPipeline::Replicaskipped theis_responsible(storage_admission_width)check thatPaidOnlykeys had to pass, so whoever labelled the hint decided what we store. Two messages were enough to exploit it — a paid hint queued key K asPaidOnly, then a replica re-advertisement hit thealready_pendingfast path inadmit_hintsand escalated the live entry toReplica, and both fetch paths downloaded without ever asking whether this node was in the top-9 for K. The tag is now derived fromreplica_hint_sourcesinstead of stored (the tag is "did any peer claim to hold this"), which deletes the escalation and both demotion sites, and downloads are gated on live routing state at both diverging fetch sites. This narrows replica repair from the sender's view to our own — a transiently skewed routing table now declines to repair a key it ranks outside the top-9, matching pruning, which already evicts at the same width — while fresh PUTs keep their wider accept window.92339a1- admit hints through a single relevance gateAdmission previously asked two questions depending on the sender's label — replica hints gated at
storage_admission_width(9), paid hints atpaid_list_close_group_size(20) — letting the sender choose its own gate. Admission now decides only relevance (should we learn this key exists and is paid for) at the 20-wide paid close group, since that is the width across which nodes track payment validity; storage responsibility stays at download time against live routing state. Mislabelling a hint gains an attacker nothing because both labels reach the same gate; the admissible key set is unchanged (a paid hint for any key a replica hint could carry was already admissible at 20), and the two-gate rescue dance is removed. It does recover information the old gate discarded: a replica hint for a key we rank 10-20 for was previously rejected outright even though that band exists precisely to track payment validity.b5c0367- merge duplicate hints without rebuilding the fetch heapA hint for a key already in the fetch queue only needs its advertiser merged into that candidate's source set — a field the heap never orders on — but the old path took the whole heap, scanned it linearly for the key, and re-heapified, costing O(n) per duplicate; under source aggregation a batch of m duplicates ran O(m·n) while holding the global queue write lock, which a neighbor could trigger by re-hinting queued keys.
FetchCandidateis split intoFetchOrder(key + distance, the only fields theOrdimpl reads) held in the heap andFetchPayload(sources + retry metadata) held in a key-indexed map that also serves as the membership index; merging a source is now an O(1) map lookup that never touches the heap, and the departed-peer path edits payloads in place and rebuilds only when a peer actually orphaned a candidate. Measured per-key merge cost goes from 302µs at a 50k-deep queue to a flat ~400ns independent of depth.3a00fef- drain detached LMDB blocking ops on shutdownReplicationEngine::shutdown()promises that when it returns no background work still holds the LMDB environment, but several paths race aselect!on the shutdown token against futures awaiting aspawn_blockingLMDB transaction — the fetch worker'sstorage.put, the neighbor-sync prune pass'sstorage.deleteandpaid_list.remove_batch(thePaidListis a second LMDB environment with the same pattern), and the verification worker'spaid_list.insert. Dropping the losing future does not cancelspawn_blocking: the closure keeps running on the blocking pool with a clonedEnv, so reopening the same environment after shutdown was undefined behavior. Rather than shielding every call site, the blocking tasks are now tracked at the storage layer:LmdbStorageandPaidListroute everyspawn_blockingthrough their ownTaskTrackerand exposewait_idle(), whichshutdown()awaits after its existing detached-task drain. Per-fetch tasks are additionally spawned on the detached tracker, so an aborted worker's droppedin_flightset can no longer leakArc<LmdbStorage>past shutdown. Cancellation semantics are unchanged — network I/O still aborts promptly, and shutdown waits only for the bounded in-flight transaction (milliseconds). Regression coverage parks a write inside its blocking closure, drops the awaiter, and proveswait_idle()/shutdown()block until it commits and both environments reopen cleanly.Preceding Follow-up Commit Details
ab848a2- stop the message handler when event streams closeMakes P2P lag remain recoverable but treats a closed P2P broadcast as terminal, matching the new terminal handling for a closed DHT broadcast. The replication loop now exits instead of spinning indefinitely on an immediately ready closed receiver; dropping its sender also lets the serial handler finish. The commit additionally hoists imports required by the Rust conventions hook.
3e8860c- prune departed peers during DHT lag recoveryBrings lag recovery in line with the normal
KClosestPeersChangedpath by retaining only the current close-peer set before requeueing its members. Stale peers from missed departure events no longer sit ahead of genuine entrants and burn one request timeout each.fc9dff0- penalize rejected singleton replica hintsPenalizes a sole replica advertiser when either the close group definitively rejects the key or that advertiser explicitly denies possessing it. This closes the free-replication abuse path while keeping inconclusive rounds without a direct contradiction, paid-only advertisements, and corroborated replica hints non-penalizing. Trust reports remain bounded per peer and verification cycle.
Earlier Follow-up Commit Details
37f8e34- unblock bootstrap when a rejected peer leavesRetires capacity-rejection debt on
PeerRemovedand immediately reruns bootstrap drain detection, so progress no longer depends on an unrelated later pipeline event.8f38d4c- prioritise source-aware hintsReplaces the single hint sender with live source sets, preserves replica claimants separately, prioritises corroborated work, bounds verification cycles and network concurrency, and simplifies stale proof-of-concept tests around the production queue implementation.
94705cf- aggregate bootstrap hint batchesMakes a bootstrap neighbor batch an atomic source-aggregation unit: sync requests run concurrently, response metadata is processed first, admitted hints are queued together, and verification waits until batch publication and drain accounting are complete.
f9263de- aggregate verification per peerSends one bounded request per target peer for the cycle, aligns the incoming limit with the cycle bound, and rejects oversized batches rather than processing a misleading prefix.
8d3834b- drain fresh offers through LMDB writesRemoves cancellation around a started fresh-offer handler and makes its tracker drain unconditional, preventing a dropped async waiter from detaching a live blocking LMDB transaction.
b577e8e- track detached audit workGeneralises the fresh-offer tracker into a shared detached-task lifecycle barrier and applies it to storage/P2P-capable audit responders, delayed possession checks, and audit launches.
Latest Follow-up Commit Details
172e97f- guard raw LMDB reads against concurrent map resizeall_keys()andget_raw()now acquire the shared LMDB environment lock inside their tracked blocking closures before opening a read transaction. This restores the invariant assumed bytry_resize(): its exclusive lock means no transaction can still reference the old memory map whenenv.resize()runs. A deterministic regression parks an in-flight raw read and proves resize waits for it to release.86c4910- fairly share verification capacityReplaces the removed hard per-peer admission quota with elastic max-min accounting under the unchanged global cap. A peer may use all otherwise idle capacity, preserving large legitimate bootstrap batches, but under contention an under-served sender can displace a low-priority borrowed entry held by an over-served sender. Verification service is independently fair across capacity owners via persistent round-robin scheduling, with slack redistribution and source-count/retry priority within each owner. Capacity ownership survives fetch retries, transfers when a source departs, and is separate from evidence provenance; bootstrap explicitly records displaced hints for redelivery. The wire protocol and serialized/public compatibility surface are unchanged.
Adversarial coverage includes a paid-only attacker filling all 131,072 slots before an honest source submits 10,000 hints, the reverse arrival order, a sole legitimate 50,000-hint source, full-cycle fairness, retry protection, owner transfer, duplicate-bookkeeping bounds, and bootstrap displacement accounting. Validation passes all 766 library tests, the seven bounded-queue PoCs, formatting, diff checks, and all-target/all-feature Clippy with warnings denied.
Newest Follow-up Commit Details
a40da7b- generalize bounded responder admissionExtracts the audit responder's global permit and per-peer slot into a reusable admission primitive. A provisional peer slot is RAII-owned while waiting for global capacity, so both normal guard drops and cancelled waits restore capacity.
ec8eb6a,864a5b4, and57873d0- isolate bulk respondersMoves fetch, verification, and neighbor-sync response work off the serial replication lane and onto bounded worker pools. Fetch uses three bandwidth-limited workers; verification and neighbor sync use two each and permit one outstanding batch per source. Requests that have already exceeded the caller's existing deadline are shed at dequeue, before stale LMDB or upload work begins.
7b5ee55and48db2d5- make overload bounded and visibleRemoves the inline execution fallback when the serial handoff is full or closed, preserving responsiveness of the P2P event receiver. Relaxed atomic counters record admission drops, stale dequeues, and per-variant serial drops without adding contention to the overloaded paths.
4ef753band339f44b- add audit and possession diagnosticsAdds structured audit issuer/source attribution, capacity decisions, stage latency, and bounded minute summaries. Responsive possession failures now retain a stable reason label for every existing validation branch; the wire protocol and trust decisions are unchanged.
Coverage Added Or Updated
Bounded responder guards release global and per-source slots on ordinary drop and cancelled admission waits.
Neighbor-sync admission serializes one source while allowing a different source to progress concurrently.
Expired responder work is shed at dequeue while fresh requests are served.
A full serial handoff drops the message instead of invoking its handler inline.
Bulk-responder counters remain separated by class and admission-versus-staleness outcome.
Audit summaries order top logical origins and preserve per-stage timings.
Possession validation branches map to stable structured failure labels.
LMDB resize waits for an in-flight
get_raw()transaction holding the shared environment lock.Elastic max-min admission under attacker-first and honest-first arrival orders, including large sole-source bootstrap batches and the unchanged global capacity bound.
Fair per-owner verification service with slack redistribution, retry protection, peer-removal ownership transfer, bounded duplicate bookkeeping, and displaced-bootstrap redelivery accounting.
Paid-hint admission after duplicate replica rejection.
Verification retry, reservation transfer, discard, exhaustion, and per-sender capacity behavior.
Paid-list edge vote behavior for negative, positive, unresolved, self-inclusive, undersized, and non-default runtime-configured group widths.
Neighbor-sync priority drain/termination and lag recovery.
Bootstrap completion after a capacity-rejected peer departs.
Capacity-rejection TTL derivation across the full configured sync cycle and runtime settings, plus semantics: a within-TTL rejection still blocks drain, an expired record unblocks it, expiry is per-source (a stale source's expiry does not forfeit a fresh source's owed re-delivery), and a repeat rejection refreshes the timestamp.
The peer-removal race ordering itself: removal cleanup runs first as a no-op, the rejection is recorded after for the now-departed peer, drain is blocked, then TTL expiry retires the orphaned record and drain completes.
The verification-tick self-heal helper: an orphaned record survives a within-TTL tick and a past-TTL tick expires it and completes bootstrap.
Duplicate hint source aggregation and source-aware scheduling.
Singleton replica-hint penalties for definitive close-group rejection and explicit source denial, including neutral inconclusive, paid-only, and corroborated cases.
Atomic bootstrap batch publication and full-cycle verification request bounds.
Fresh-offer admission bounding, per-key in-flight collapse of concurrent duplicates, and the refusal-past-bound possession penalty.
Replica-download responsibility gating at both fetch sites, including rejection of out-of-range keys and the removed paid→replica escalation path.
Single-gate hint admission parity across replica and paid labels for the unchanged admissible key set.
O(1) duplicate-source merge and heap rebuild only on genuine candidate orphaning.
Audit coordinator per-target serialization, cross-peer parallelism, and cancellation cleanup.
Closed P2P event handling now explicitly verifies terminal control flow; lagged P2P event handling still verifies continuation and metric accounting.
E2E paid-list majority repair below storage quorum.
Storage- and paid-list-level
wait_idleregression tests: a write parked inside its blocking closure with a dropped awaiter keepswait_idleblocked, commits after release, and leaves the store usable.Engine-level shutdown drain (
poc_shutdown_lmdb_drain):shutdown()blocks until a detached LMDB write commits, after which both environments reopen cleanly.Per-attempt responsibility recheck: worker disposition of
NoLongerResponsible(terminal exit, retry-slot release, no verification requeue), terminal-path parity withStored, and preserved source-failure retry/requeue transitions.E2E stale-fetch-candidate driver on a live 12-node network: a seam-enqueued candidate for an out-of-responsibility key is never stored and exits the pipeline terminally, with an in-responsibility positive control through the same seam and holder; the test fails when the rechecks are disabled.
Existing prune, fetch-retry, and replication tests updated for shared coordination and reservation-aware APIs.
Expected Effect
Fetch, verification, and neighbor-sync floods cannot monopolize the serial replication lane; bounded, per-source-fair workers isolate their LMDB, queue, and upload costs.
Serial-queue saturation remains local to the rejected message instead of propagating inline work and lag into the P2P receiver.
Operators can distinguish admission pressure, expired queued work, audit-origin hotspots, slow response stages, and exact responsive possession-proof failures.
Raw LMDB reads cannot overlap an unsafe memory-map resize, closing the remaining resize-safety gap for audit, pruning, and commitment scans.
A routing peer can use idle verification capacity but cannot monopolize admission or service once honest peers contend; large genuine bootstrap batches remain admissible when capacity is available.
Partition heals and mass joins drain priority neighbor-sync work in seconds-scale rounds instead of waiting through periodic ticks or stale-peer timeouts.
Repair remains live through transient routing disagreement, no-holder/no-source rounds, queue pressure, and fetch exhaustion.
Better-corroborated hints are verified first without allowing verification rounds or request fan-out to grow without bound.
Sole peers cannot advertise unacknowledged replicas to offload storage for free without incurring bounded trust penalties.
Fresh-offer verification and storage never run inline on the serial loop, so worker saturation no longer backs up the inbound queue or drops replication messages through broadcast lag.
A peer cannot conscript out-of-range nodes into fetching and storing arbitrary keys by labelling hints as replicas.
Duplicate-hint bursts no longer scale the fetch-queue write-lock hold time with queue depth.
Topology churn between fetch promotion and download no longer costs a download, a disk write, and a prune cycle — stale candidates are declined at download time, terminally and without stalling bootstrap drain or penalizing an innocent source.
Bootstrap cannot be held indefinitely by partial batch publication or departed capacity-rejected peers — including one whose removal races the rejection record — and a drain condition that becomes true without a triggering event is observed within one worker tick, so audits cannot be disabled permanently nor the node trust-penalized for a perpetual bootstrap claim.
Local audit concurrency no longer manufactures false remote timeout verdicts.
Closed replication event streams terminate cleanly without CPU spin or repeated warnings.
Graceful shutdown does not return while tracked detached storage or P2P work is still active.
When
shutdown()returns, timed-out long-lived tasks have been aborted and joined, no LMDB blocking operation is still running on either environment, and no engine-spawned task holds the storage, so the same LMDB files can be reopened safely.SemVer
Minor.