Skip to content

[python] Introduce rbm32 aggregator function - #8724

Open
PyRSA wants to merge 1 commit into
apache:masterfrom
PyRSA:feature/pypaimon-roaring-bitmap-agg-impl
Open

[python] Introduce rbm32 aggregator function#8724
PyRSA wants to merge 1 commit into
apache:masterfrom
PyRSA:feature/pypaimon-roaring-bitmap-agg-impl

Conversation

@PyRSA

@PyRSA PyRSA commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Purpose

This PR introduces the Python FieldRoaringBitmap32Agg implementations and aligns the Python roaring bitmap aggregation behavior with the Java implementation.

The changes include:

  • Add rbm32 field aggregation support.
  • Support roaring bitmap merge using serialized bytes representation.
  • Add VARBINARY type validation for roaring bitmap aggregation.
  • Add unit tests for aggregation and invalid input types.

Tests

  • python -m pytest pypaimon/tests/test_field_aggregators.py::FieldRoaringBitmap32AggTest -q
  • python -m pytest pypaimon/tests/test_aggregation_e2e.py -q
  • python -m pytest pypaimon/tests/test_aggregation_merge_function.py -q
  • git diff --check

@JingsongLi

Copy link
Copy Markdown
Contributor

rbm64 is serialized using pyroaring.BitMap64, but it is incompatible with the Java Roaring64Bitmap format. Bidirectional verification fails in both cases: Java reads an EOF when accessing the Python datagram, and Python cannot deserialize the Java datagram; this breaks cross-engine reading between Java and PyPaimon.

@PyRSA

PyRSA commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

rbm64 is serialized using pyroaring.BitMap64, but it is incompatible with the Java Roaring64Bitmap format. Bidirectional verification fails in both cases: Java reads an EOF when accessing the Python datagram, and Python cannot deserialize the Java datagram; this breaks cross-engine reading between Java and PyPaimon.

Thanks for the feedback. I confirmed that pyroaring.BitMap64 is not compatible with Java's Roaring64Bitmap serialization format, so cross-engine interoperability cannot be guaranteed. I'll remove the rbm64 aggregator from this PR and revisit it when a compatible implementation is available.

@PyRSA PyRSA changed the title [python] Introduce rbm32/rbm64 aggregator function [python] Introduce rbm32 aggregator function Jul 26, 2026
@JingsongLi

Copy link
Copy Markdown
Contributor

Please rebase master.

@PyRSA
PyRSA force-pushed the feature/pypaimon-roaring-bitmap-agg-impl branch from c01c529 to 0c972b6 Compare July 26, 2026 15:07
@PyRSA PyRSA closed this Jul 26, 2026
@PyRSA
PyRSA force-pushed the feature/pypaimon-roaring-bitmap-agg-impl branch from 0c972b6 to bce7744 Compare July 26, 2026 15:25
@PyRSA PyRSA reopened this Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants