Skip to content

Support big-endian targets - #64

Open
pkubaj wants to merge 1 commit into
tower120:mainfrom
pkubaj:be-support
Open

pkubaj wants to merge 1 commit into
tower120:mainfrom
pkubaj:be-support

Conversation

@pkubaj

@pkubaj pkubaj commented Nov 12, 2026

Copy link
Copy Markdown

The big-endian branches were never compiled and no longer build:
direct_bitset.rs uses copy_nonoverlapping without importing it and
gates lvl_get_item to little-endian while its callers are unconditional,
bitset/serialization.rs calls a non-existent chunked_read with a
non-existent data_blocks_len. The remaining branches are
unimplemented!().

Implement them so that the serialized format is the same on both byte
orders (every u64 lane stored little-endian, lanes in array order):

  • BitBlock::to_le_bytes/from_le_bytes for the wide SIMD blocks convert
    lane-wise. BitBlock::to_le now reinterprets the little-endian bytes
    natively (from_ne_bytes) instead of round-tripping back to the
    original value; BufferedWriter relies on that.
  • Writer::write_primitives/write_masks and
    Reader::read_primitives/read_masks/foreach_read_mask convert per
    element on big-endian. foreach_read_mask needs T: BitBlock for that.
  • BitSet::deserialize reads the data blocks through foreach_read_mask on
    big-endian.
  • DirectBitset::read_mask reads the byte array and converts it. The old
    ALIGNED branch also passed size_of::() as a count of masks, not
    bytes. lvl_get_item needs no index correction: sub-masks are stored as
    little-endian u64 in array order and read_mask:: already converts
    them.

Tested on FreeBSD/powerpc64 (big-endian, POWER9): the library tests
pass, and data serialized on x86-64 deserializes identically through
BitSet, ImmutableBitset and DirectBitset (aligned and unaligned), and
vice versa.

The big-endian branches were never compiled and no longer build:
direct_bitset.rs uses copy_nonoverlapping without importing it and
gates lvl_get_item to little-endian while its callers are unconditional,
bitset/serialization.rs calls a non-existent chunked_read with a
non-existent data_blocks_len. The remaining branches are
unimplemented!().

Implement them so that the serialized format is the same on both byte
orders (every u64 lane stored little-endian, lanes in array order):

- BitBlock::to_le_bytes/from_le_bytes for the wide SIMD blocks convert
  lane-wise. BitBlock::to_le now reinterprets the little-endian bytes
  natively (from_ne_bytes) instead of round-tripping back to the
  original value; BufferedWriter relies on that.
- Writer::write_primitives/write_masks and
  Reader::read_primitives/read_masks/foreach_read_mask convert per
  element on big-endian. foreach_read_mask needs T: BitBlock for that.
- BitSet::deserialize reads the data blocks through foreach_read_mask on
  big-endian.
- DirectBitset::read_mask reads the byte array and converts it. The old
  ALIGNED branch also passed size_of::<Mask>() as a count of masks, not
  bytes. lvl_get_item needs no index correction: sub-masks are stored as
  little-endian u64 in array order and read_mask::<u64> already converts
  them.

Tested on FreeBSD/powerpc64 (big-endian, POWER9): the library tests
pass, and data serialized on x86-64 deserializes identically through
BitSet, ImmutableBitset and DirectBitset (aligned and unaligned), and
vice versa.
@tower120

Copy link
Copy Markdown
Owner

Do you REALLY need big-endian? How do you test it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants