Replace full-chunk audit responses with BLAKE3 verified slices

[grumbach]

What

Round 2 of the storage-commitment audit (ADR-0002) made the audited node return the complete original bytes of the nonce-sampled chunks, up to two full 4 MiB chunks per response. Traffic accounting showed this subtree_byte_response was the fleet's second-largest replication bandwidth consumer: ~4.2 TB out + 3.3 TB in per day, ~7.5 TB/day of chunk bytes moved purely to prove storage on a network holding ~18.6 TB.

This replaces the full-byte round 2 with a verified slice: the response for one opened 1 KiB block is a few KB instead of a full chunk (~1000x reduction on the dominant message), while proving more than the old check.

How

A chunk's address is BLAKE3(content), and BLAKE3 is internally a Merkle tree over 1 KiB blocks, so a Bao verified slice proves a block is the real content at a given offset against the address the auditor already knows (content authenticity for free). But the address is public, so authenticity alone is delegable: a node storing nothing could pass by fetching the block on demand from an honest holder (the FetchRequest path serves full chunk bytes with no auth). Possession must therefore be bound separately.

So round 1 additionally commits, per leaf, a fresh nonced block tree: a Merkle root over the same 1 KiB blocks whose leaves are BLAKE3(nonce ‖ peer ‖ key ‖ block_index ‖ block_len ‖ block_bytes). Because the fresh per-audit nonce enters every block leaf, building the correct root requires all of a chunk's bytes under that nonce, and the block the auditor opens is drawn with fresh randomness after the roots are committed (cut-and-choose). Round 2 verifies both chains over the same block bytes: the Bao slice against the chunk address (authenticity) and the nonced-tree opening against the round-1 root (possession).

A responder that did not hold the bytes at commit time cannot have committed a correct nonced root, and cannot fold an after-the-fact-fetched block to a garbage committed root without a preimage break. Putting the nonce in every block also closes a preprocessing gap in the old flat nonced_hash (where only the first BLAKE3 chunk mixed in the nonce, letting an adversary retain ~1.3 KiB per chunk and recompute the hash for any nonce).

Binding the content length

content_len and nonced_root travel in the round-1 leaf but are not covered by the signed commitment root (it hashes only key ‖ bytes_hash), so a malicious responder can claim any length. Left unchecked this forges possession two ways:

• Deflation: claiming a tiny content_len collapses the block count to 1, so only block 0 is ever opened, and a Bao prefix slice for [0, 1024) verifies fine against the true full-content address. A node storing ~1 KiB passes an audit for the whole chunk.
• Header rewrite: Bao does not bind the slice length header for a prefix slice that never reaches EOF, so a responder can claim a short length just past a subtree boundary, hand over the true right-subtree hash as the opaque root sibling, and pass any opening in the retained left half while storing only that half.

Both are closed on the auditor side: verify_block_slice requires the slice's own length header to equal the claimed content_len, and the auditor also opens the claimed final block per sampled leaf. The final-block decode reaches EOF, where Bao authenticates the encoded length against the address, so a forged-short length fails there. Signing content_len into the commitment alone would not help (the attacker just commits the false length from the start); only anchoring it against the address does.

Shape of the change

• New replication::slice module: nonced block tree (build/open/verify) + Bao slice extract/verify helpers (bao 0.13.1, shares the existing blake3).
• Round-1 SubtreeLeaf now carries content_len + nonced_root instead of the flat nonced_hash.
• Round 2 SubtreeByteChallenge/Response → SubtreeSliceChallenge/Response: opens a fresh-random block plus the claimed final block per sampled leaf, and verifies both chains on each. No batching (the reply is KB-scale); MAX_SLICE_OPENINGS caps the sample (raised to cover two openings per leaf).
• REPLICATION_PROTOCOL_ID advances to v3 for a clean cutover (matches the v1→v2 pattern; v2 and v3 nodes simply don't exchange replication traffic during the short auto-upgrade window).

Why not the literal proposal

The ticket's original wording (verify the slice only against the chunk address, with a nonce-derived offset) is unsound: it drops the possession binding entirely, so a storage-less relay farms audit passes. That defeats the point of proof-of-storage and lets one warehouse back many replica identities, breaking replication multiplicity. The offset also must be fresh post-proof randomness, not nonce-derived, or the cut-and-choose collapses. This PR implements the corrected dual-chain design.

Testing

• replication::slice unit tests: Bao root equals the BLAKE3 address across lengths, slice roundtrip for every block, tamper / wrong-address / wrong-block rejection, nonced-tree binding to nonce/peer/key, relay-cannot-open-against-a-foreign-root, plus regressions for the deflation and header-rewrite length forgeries.
• poc_commitment_audit_attacks (ported to the slice model): relay-holds-addresses, predict-and-fetch under fresh sampling, fabricated-fraction, deleter, plus the substitution / replay / structural attacks.
• poc_audit_handler_live: the production responder against real LMDB storage, including an end-to-end test that opens a deep block of a multi-block chunk and verifies both chains.
• tests/e2e/subtree_audit_testnet: real multi-node QUIC + LMDB. Honest node passes, data-deleting node is caught, and repeated audits produce no false positives.
• Full cfd clean (clippy -D warnings, fmt, cargo doc --deny=warnings); all replication lib tests + both PoC suites green.

Commits

1. Replace full-chunk audit responses with BLAKE3 verified slices: the dual-chain slice audit above.
2. Authenticate slice-audit content_len to defeat possession forgery: the content-length binding (header check + final-block open).

Linear

Tracked in V2-685.

Tier

Tier 3: this changes the possession-proof representation itself, and a weak slice proof would let under-storing nodes pass audits. Evidence plan (V2-685 validation steps): DEV testnet A/B on subtree_byte_response byte rates, corrupted-node detection parity vs the full-byte baseline, mixed-version interop. Review status: changes requested at bae97e9c; rework in progress.

Semver

Intended feature/minor: additive wire variants with full-byte fallback for older peers. Public API surface to be re-confirmed at cut time.

[Copilot]

Pull request overview

This PR updates ant-node’s storage-commitment audit (ADR-0002) to replace round-2 “serve full chunk bytes” proofs with Bao/BLAKE3 verified 1 KiB slice openings plus a nonced per-block Merkle commitment, drastically reducing audit bandwidth while strengthening possession binding. It also wires in ADR-0005 audit-tally reporting so quote responses can carry a signed, nonce-bound audit report.

Changes:

• Replace round-2 full-chunk byte challenges with slice challenges (SubtreeSliceChallenge/Response), including block-index sampling and dual-chain verification (Bao slice + nonced block-tree opening).
• Introduce replication::slice (Bao slice helpers + nonced block-tree build/open/verify) and update subtree leaf commitments to carry content_len + nonced_root.
• Add ADR-0005 audit tally plumbing: record subtree-audit outcomes, persist them, and attach a signed audit report to quote responses; bump replication protocol ID to v3.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/poc_commitment_audit_attacks.rs	Updates attack PoC to the slice + nonced-root audit model.
tests/poc_audit_handler_live.rs	Live-handler tests for slice openings, including deep-block proof on multi-block chunks and cap enforcement.
src/storage/handler.rs	Adds optional signed audit-report sidecar to quote responses and test coverage for report verification.
src/replication/subtree.rs	Changes SubtreeLeaf to include content_len and nonced_root; removes old nonced_hash helper.
src/replication/storage_commitment_audit.rs	Implements slice round-trip request/response, random block selection, and responder slice-opening handler.
src/replication/slice.rs	New module: Bao slice extract/verify + nonced block-tree root/open/verify helpers and tests.
src/replication/protocol.rs	Wire types updated for slice challenges/responses; adds SubtreeSliceOpening; adjusts sizing test.
src/replication/mod.rs	Exposes new modules; integrates ADR-0005 tally persistence and recording hooks.
src/replication/config.rs	Adds MAX_SLICE_OPENINGS, slice response timeout, and bumps replication protocol ID to v3.
src/replication/audit.rs	Extends AuditTickResult::Passed to carry optional commitment_key_count for ADR-0005 tally.
src/replication/audit_tally.rs	New module: per-peer tally, snapshot tagging, report-row building, and quote-source adapter.
src/payment/quote.rs	Adds AuditReportSource and builds/sizes/signs nonce-bound audit reports for quote responses.
src/node.rs	Wires replication engine’s tally into the quote generator as an AuditReportSource.
docs/adr/ADR-0005-earned-reward-eligibility.md	New ADR describing earned eligibility model and report semantics.
Cargo.toml	Adds bao dependency and updates ant-protocol version/pinning.
Cargo.lock	Locks bao and updates ant-protocol resolution.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

src/replication/slice.rs:257

• extract_block_slice copies the entire chunk (Cursor::new(content.to_vec())) before extracting a small Bao slice. For multi‑MiB chunks this adds an avoidable allocation and extra memcpy per opening; Cursor<&[u8]> works here and avoids duplicating the content buffer.

pub fn extract_block_slice(content: &[u8], index: u32) -> std::io::Result<Vec<u8>> {
    let (start, end) = block_range(content.len() as u64, index);
    let len = end - start;
    // The outboard carries the BLAKE3 tree hashes separately from the content, so
    // the extractor reads the real chunk bytes plus just the parent hashes on the
    // block's path — no need to materialise a full Bao encoding of the chunk.
    let (outboard, _hash) = bao::encode::outboard(content);
    let mut extractor = bao::encode::SliceExtractor::new_outboard(
        Cursor::new(content.to_vec()),
        Cursor::new(outboard),

src/replication/storage_commitment_audit.rs:1207

• handle_subtree_slice_challenge reads chunk bytes from LMDB once per opening (get_raw_retrying inside the openings loop). Because the auditor can request two openings per sampled leaf (random + final), the same key can appear twice and will be read twice, doubling disk I/O and work for the common case. Consider de-duplicating openings by key (read each chunk once, then build all requested openings for that key while preserving response order).

    let mut items = Vec::with_capacity(challenge.openings.len());
    for opening in &challenge.openings {
        let key = opening.key;
        // Open ONLY keys committed under this pin. A key not in the pinned tree
        // is `Absent` — never served from local storage just because we happen to
        // hold it (§15: serving an uncommitted-but-held key would let a forged
        // challenge harvest data and muddy the possession proof for THIS commit).
        if built.proof_for(&key).is_none() {
            items.push(SubtreeSliceItem::Absent { key });
            continue;
        }
        match get_raw_retrying(storage, &key).await {

src/replication/storage_commitment_audit.rs:821

• This success log still says "byte-checked", but checked is now the number of slice openings (block proofs) verified. The message is misleading when diagnosing audit behavior.

        "Audit: peer {challenged_peer} passed subtree audit ({} leaves, {checked} \
         byte-checked)",

Cargo.toml:207

• ant-protocol = "3.0.0" is still being overridden via [patch.crates-io] to a Git rev, with a comment saying "STRIP BEFORE MERGE". This makes the build source ambiguous and can accidentally ship a git override to production. Either remove the patch when 3.0.0 is available on crates.io, or switch the dependency itself to the git source (and drop the patch) until the crates.io release exists.

# An immutable rev (not a mutable branch) keeps the build reproducible; the
# committed Cargo.lock records the exact resolution. STRIP BEFORE MERGE —
# replace with the published ant-protocol 3.0.0 once it lands on crates.io.
[patch.crates-io]
ant-protocol = { git = "https://github.com/grumbach/ant-protocol", rev = "2015113270bbde8204a1740674bed2ada52d2ca6" }

[Copilot]

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/replication/slice.rs:257

• extract_block_slice clones the entire chunk with content.to_vec() just to create a Cursor, doubling memory and adding an extra copy per opening. Since Cursor<&[u8]> works, pass the slice directly to avoid the allocation/copy (especially important when openings require full 4 MiB chunks).

    let mut extractor = bao::encode::SliceExtractor::new_outboard(
        Cursor::new(content.to_vec()),
        Cursor::new(outboard),

src/replication/storage_commitment_audit.rs:1199

• handle_subtree_slice_challenge reads the same chunk from LMDB once per opening. Since the auditor can legitimately request two openings per sampled leaf (random + final block), this causes redundant reads for the same key and makes disk-read amplification easier for a forged auditor (even within MAX_SLICE_OPENINGS). Consider grouping openings by key and reading each chunk at most once, while still returning items in the original requested order.

        match get_raw_retrying(storage, &key).await {
            // Committed key, bytes present → build the two-chain opening (or a
            // terminal reject for an out-of-range index / surprise extraction error).
            Ok(Some(bytes)) => {
                match build_slice_item(challenge, key, opening.block_index, &bytes) {

[dirvine]

Reviewed exact head bae97e9c867b5b1e3b00159cf316ef68a0708eaf with a six-seat panel (two groups of three), then independently reproduced the blocking cryptographic finding and ran the focused checks locally.

Verdict: changes requested. The bandwidth direction is good, and the Bao length-forgery fix is well tested, but the current construction does not yet establish the possession property claimed by the PR.

Blocking findings

1. High — nonce-independent BLAKE3 CV still permits deterministic under-storage

src/replication/slice.rs:51,99-115

The leaf input is:

domain(38) || nonce(32) || peer(32) || key(32) || index(4) || len(4) || block(1024)

The 142-byte prefix plus the first 882 block bytes fill BLAKE3 chunk 0. The final 142 block bytes form chunk 1 and are nonce-independent. A node can retain that chunk's 32-byte chaining value instead of the 142 original bytes, saving 110 bytes per full block / 10.7421875%, while reconstructing the exact round-one leaf/root for every fresh nonce.

I reproduced this against the production layout using BLAKE3's low-level subtree API: merge_subtrees_root(chunk0_cv, precomputed_chunk1_cv) exactly matched blake3::hash(prefix || block).

Round two still needs the missing suffixes, but at most ten openings require only about 1.4 KiB of missing data. The generous slice deadline therefore becomes security-relevant again, contrary to the comments claiming the preprocessing gap is closed and all bytes are required.

A fresh nonce-derived BLAKE3 key (new_keyed(nonce) or a domain-separated key derived from the nonce) makes every BLAKE3 chunk nonce-dependent and closes this specific attack. Merely prefixing the nonce does not.

2. High — round-two requests amplify full-chunk reads and synchronous hashing

src/replication/storage_commitment_audit.rs:1148-1203; src/replication/slice.rs:144-207,248-263; src/replication/mod.rs:2527-2594

Each opening independently:

1. reads the full chunk from LMDB;
2. creates a full Bao outboard and clones the full content;
3. rebuilds the complete nonced tree.

The normal random-plus-final pair does this twice for the same key. Duplicate openings are accepted, so one ten-opening request can cause up to 40 MiB of reads plus repeated full scans/copies. Sixteen globally admitted responder tasks bound concurrency, but there is no rate/work budget or binding of round two to responder-held round-one state; repeated requests can keep the pool and Tokio workers busy.

Please coalesce by key, read each chunk once, reuse Bao/nonced tree state for all openings, reject/coalesce duplicates, and add byte/work-aware admission or bind the request to live round-one state. The CPU-heavy construction should not run unbounded on Tokio worker threads.

3. Medium — correlated mid-audit Bootstrapping preserves holder credit

src/replication/storage_commitment_audit.rs:351-357,782-785; src/replication/mod.rs:4471-4507

After serving a valid round-one proof, a responder can answer round two with a matching Bootstrapping response. It is mapped to Timeout; subtree timeouts are fully graced and preserve existing recent_provers credit. This responsive contradiction bypasses both confirmed-failure handling and bootstrap-claim-abuse accounting.

Treat it as a protocol failure, or at minimum revoke the pinned commitment's credit and route it through bootstrap-claim accounting.

4. Rollout gate — v3 partitions all replication operations

src/replication/config.rs:247-262; src/replication/mod.rs:1237-1257

The single global protocol ID is used for fresh replication, neighbour sync, verification, fetch, repair/pruning/possession audits and commitment fetches—not just the changed subtree audit. During a staggered v2/v3 rollout, DHT routing still mixes cohorts while cross-version peers exchange no replication traffic at all.

This needs either a dedicated/dual-registered audit protocol or a quantified and tested coordinated-rollout plan. The current test only asserts the string constant; it does not test mixed-version bootstrap, fresh replication or repair.

Design documentation

ADR-0002 is unchanged and still specifies full chunk responses, flat freshness hashes and deadline-based possession. Several code comments also still describe full bytes/one opening or incorrectly imply the Bao header check alone authenticates prefix-slice length. These are material security-invariant mismatches, not wording nits; please update the ADR and comments with the implementation.

Verified locally

• cargo fmt --all -- --check — pass
• cargo test -q replication::slice --lib --no-fail-fast — 15/15 pass
• cargo test -q storage_commitment_audit --lib --no-fail-fast — 16/16 pass
• cargo test -q --features test-utils --test poc_commitment_audit_attacks — 19/19 pass
• cargo test -q --features test-utils --test poc_audit_handler_live — 9/9 pass
• cargo clippy --all-targets --all-features -- -D warnings — pass
• Max-chunk measurement: Bao slice 1,800 bytes + 12 nonced siblings/384 bytes per opening; about 22 KiB for ten openings before small envelope overhead.

Review-team consensus

All six seats completed. Both cryptographic reviewers independently reproduced the BLAKE3 preprocessing gap. Both runtime reviewers confirmed duplicate full-chunk work amplification. Both rollout/accounting passes confirmed the global v3 split and round-two bootstrap-credit path. No dissent on the blockers; the main qualification is that the CV attack still needs retrieval/delegation for the few opened suffixes.

CI is green except the Windows test job is still pending at review time.