Repository navigation
Peel xor filters through one queue per segment - #786
Conversation
XorFilter construction peeled singleton slots through one LIFO queue over all slots. The reference C implementation (FastFilter xor_singleheader), the Rust xorf crate, and datasketches-rust keep one queue per segment and drain segment 0, then 1, then 2. Whether peeling succeeds does not depend on the order, so every implementation selects the same construction seed, but the order decides which slot each key owns. The same items and base seed therefore produced different fingerprint payloads in Java and Rust, and different non-member keys were false positives. Peel through one queue region per segment within the existing queue array, in the reference order. A peeled key is removed only from its slots in the other two segments, because its own slot holds no other key. Images now match datasketches-rust byte for byte, and draining one segment at a time keeps its reads close to sequential order, which speeds up construction of large filters. The serialized format is unchanged, so existing images remain readable and answer queries as before.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
leerho
left a comment
leerho
left a comment
There was a problem hiding this comment.
Thanks, this looks good. Matching the reference peeling order before XorFilter's first release means the first released images will match datasketches-rust.
I checked the queue logic and found no problems. Each slot is queued at most once and counts never go back up, so each segment's region of queue can't overflow, and the count == 0 stale check is equivalent to the old count != 1. Leaving the peeled key's own slot untouched is safe, because nothing else maps to it and assign() uses only the stack.
I also ran the new map() against the one on main over 120 random configurations (8 and 16 bits, 1 to 300,000 keys). Both picked the same construction seed and wrote the same header every time. Every member was found, both before and after heapify.
One small suggestion: CodeQL flags a possible NumberFormatException in the test's fromHex helper. You could drop the helper and use java.util.HexFormat.of().parseHex(...), which clears the alert and saves a few lines. It's not a blocker.
|
@leerho Thanks for your review! The minor comment has been resolved. |
Problem
XorFilterconstruction peels singleton slots through a single LIFO queue that spans all three segments. The reference C implementation (FastFilter xor_singleheaderxor8_populate), the Rust xorf crate, and the datasketches-rust implementation in apache/datasketches-rust#309 instead keep one queue per segment and drain segment 0, then 1, then 2, repeating until all queues are empty.Whether peeling succeeds does not depend on the order, because the 2-core of the hypergraph is unique. Both orders therefore select the same construction seed and write the same header. The order does decide which slot each key owns, though, so the fingerprint payloads differ. For the same items and base seed, Java and Rust produced different images, and a different set of non-member keys tested as false positives.
Change
map()splits the existingqueuearray into one region per segment and peels in the reference order. A slot is queued at most once, so each region ofsegmentLengthentries cannot overflow, and no memory is added.int[]buffers.referenceImplementationImageTestchecks 8- and 16-bit images against bytes produced by datasketches-rust for the same items and base seed. The test fails with the previous peeling order.Compatibility
The serialized format does not change. Existing images remain readable and answer every query exactly as before, because queries depend only on the stored seed, segment length, and fingerprints. Only newly built images differ.
XorFilterhas not been released yet.Verification
XorFilterTest: 25 tests pass.The gain grows once the working set exceeds the CPU caches. Draining one segment at a time keeps its reads close to sequential order, because peeling a key only creates new singletons in the other two segments. With a single LIFO queue, each new singleton is popped immediately, which follows chains of slots across the whole table.