Repository navigation
refactor(xor): remove item counts and simplify construction - #329
Conversation
This reduces the 64-bit filter header from 56 to 48 bytes. It is a memory tradeoff, not a lookup speedup: the imported-filter query benchmark is mostly 2–5% slower than main. The unknown-count path now loads and compares one count instead of branching around the count check based on the Option tag. Before:
Reproduce each revision with The empty-set check remains necessary: a known-empty filter must reject every hash, while an imported filter with unknown cardinality must preserve the fingerprint comparison. No additional query special cases or packed metadata types were added. |
tisonkun
changed the title
tisonkun
changed the title
tisonkun
changed the title
|
Retain the cached segment length: deriving Before: For
The final multi-filter query loops match the baseline instructions apart from addresses. Measurements still include host noise: ordinary-release All 45 final workloads: arithmetic means and all three samples, in millisecondsApple M4 Max, macOS 26.6.2, aarch64, rustc 1.100.0-nightly ( Build each revision with: cargo +nightly-2026-08-28 bench -p benchmarks --bench benchmarks --no-run
CARGO_PROFILE_BENCH_LTO=thin cargo +nightly-2026-08-28 bench -p benchmarks --bench benchmarks --no-runInvoke each saved binary for each registered XOR scenario with
Validation: |
tisonkun
changed the title
|
Before: All 45 workloads were run in optimized release and ThinLTO builds, with three samples each. Median changes range from -3.4% to +2.6% in release and -9.5% to +1.7% with ThinLTO. No workload shows a median slowdown above 3%. Apple M4 Max, 36 GiB, macOS 26.6.2; rustc 1.101.0-nightly (8006091eb). Commands: All measurements (milliseconds; three samples in measurement order)
|
| @@ -22,16 +22,15 @@ | |||
| //! performs exactly three fingerprint reads per query. Construction needs temporary space linear | |||
| //! in the number of distinct hashes. | |||
There was a problem hiding this comment.
Could we clarify which part of construction this space bound applies to? XorFilterBuilder stores one u64 per input until build() deduplicates the hashes. Passing one million copies of the same hash to update_hashes allocated 8,000,000 bytes just to collect them. I’d suggest saying that the builder buffers one hash per input and that, after deduplication, constructing the layout needs additional space proportional to the number of distinct hashes.
There was a problem hiding this comment.
We may also use a hashset to collect hashes. I'd leave it as an internal improvement.
Perhaps we should hide this details from user documents.
There was a problem hiding this comment.
Agreed, users should not know the details.
Remove item-count metadata and the count-dependent APIs from
XorFilter. Membership queries need only the seed, layout, and fingerprints; imported filters can now use standard serialization without supplying a count. Segment length stays cached internally and has no public accessor. Remove redundant sizing accessors while retainingestimated_size.Use
XorFilter::from_hashesfor distinct hashes orfrom_fingerprints(seed, fingerprints)for imports.XorFilterBuilderreplacesfrom_valuesand accepts duplicate inputs. Add values or precomputed hashes individually, or useupdate_valuesandupdate_hashesfor batches. The initial seed selects the layouts to try; imports take the seed returned by the built filter.Reduce the serialization header from 24 to 16 bytes, matching the Java proposal. XOR has not been released in either library, so serialization stays at version 1. Reference fixtures are generated from the Java change.
Validation:
cargo x check,cargo x test(810 tests including 147 doctests), andcargo x lint. Release and ThinLTO results compare 45 XOR workloads at12dbf18andc294258. At1dd2c25, building 2,000 Xor8 filters from 16,384 values each (4,096 distinct) took a median of 340 ms with individualupdatecalls and 324 ms withupdate_valuesover five release runs.