[tisonkun]

Adds immutable XOR filters for complete input sets, with 8-, 16-, and 32-bit fingerprints behind the xor feature. Construction uses bounded retries and compact scratch storage without adding runtime dependencies.

API

XorFilterBuilder follows the library's consuming configuration pattern: choose a fingerprint type, optionally set seed(...), then consume the builder with build(values), build_from_hashes(hashes), or build_from_distinct_hashes(&hashes). It does not retain input or expose incremental updates. The first two paths deduplicate; the distinct path borrows without copying or sorting. Callers can retain borrowed input to retry with another seed. Construction goes through the builder only.

Construction makes at most 100 attempts. Size violations, duplicate input, and exhausted attempts return InvalidArgument. Duplicate hashes on the distinct-input path prevent peeling; when a failed attempt leaves two equal hashes in one slot, construction reports duplicate input immediately instead of consuming the remaining attempts. This detection is not guaranteed for every duplicate configuration. Valid input can exhaust the attempt budget, and reseeding does not guarantee success. With consistent hashing, a negative query proves absence and a positive result may be a false positive.

Compatibility

Standard serialization supports 8/16/32-bit fingerprints using DataSketches family 22, version 1. The 8/16-bit images use the format of the merged Java implementation. For the same inputs and base seed, Rust and Java choose the same construction seed and layout, so headers match; fingerprint slots can differ because the peeling order differs. The 32-bit images use the same header with fingerprint width 32 and little-endian u32 slots; interoperability requires readers that support this width. Serialization requires a known item count and signed 32-bit format bounds. Application-defined storage uses XorFilter::into_parts / XorFilter::from_parts, which exchange XorFilterParts built with XorFilterParts::new(seed, segment_length, fingerprints) and optionally with_num_items(count). from_parts validates the three-segment layout. Tests check that raw parts round-trip with xorf 0.13 for all three widths.

Imported parts can omit cardinality; num_items, is_empty, and bits_per_item then return None. Known-empty filters reject all queries, while unknown counts preserve ordinary fingerprint comparisons.

Ordinary Rust values use xxHash64 with seed 0, but Rust and Java value encodings can differ. The module documents integer and raw-byte encodings for interoperability. Externally computed hashes must use the precomputed-hash APIs throughout. Rust deserialization owns its data; independently constructed filters need not have identical fingerprint bytes.

Construction performance

The synthetic xor::matched_build workloads use base seed 1 for both implementations and verify matching winning seeds, ensuring equal attempt counts. Input generation is outside timing; neither distinct-hash path includes deduplication. Mapping visits one segment at a time and reuses the peel stack for mixed hashes, improving locality without another scratch allocation.

Xor8 results for eight million distinct synthetic hashes on an Apple M4 Max, arm64 macOS, rustc 1.101.0-nightly (2026-10-17), optimized bench profile:

Hashes	DataSketches	xorf 0.13.0	Elapsed-time change
8,000,000	289.45 ms	307.50 ms	-5.9%

These are medians of four round medians, alternating builder order, with 20 single-iteration samples per builder per round. Peak live heap allocation is approximately 263.3 MB versus 1.082 GB, excluding preexisting input and allocator overhead; it is not process RSS. Both fingerprint payloads occupy 9.84 MB. Results are specific to this workload and machine, not a universal speed guarantee.

The matched suite covers all three widths and a synthetic case requiring a retry. Automatic retry after a failed peel is checked against a fresh build starting at the next seed.

cargo bench --package benchmarks --bench benchmarks -- 'xor::matched_build::(datasketches_8|xorf_8).*8000000' --sample-count 20 --sample-size 1

[tisonkun]

fyi @pmcgleenon @freakyzoidberg @notfilippo @ariesdevil

You may mainly review:

• datasketches/src/xor/filter.rs
• datasketches/src/xor/serialization.rs

[ariesdevil]

Great!