Apache Paimon Vector Index is a pure Rust vector indexing library designed for Apache Paimon and data lake storage such as S3, HDFS, and OSS. Its seek-based readers load only the index pages needed by each query.
The library supports IVF-FLAT, IVF-SQ, IVF-PQ, IVF-RQ, and DiskANN through
shared Rust, C, C++, Java/JNI, and Python APIs. IVF-RQ stores deterministic
rotated residuals as 1–8 bit planes (4 by default), uses a bounded sign-plane
coarse pass before full refinement, and exposes the persisted width as
rq_bits metadata. DiskANN combines
a Vamana graph, resident PQ codes, paged adjacency/raw vectors, and F16 or F32
reranking for indexes queried from local SSD or object storage. Its parallel
builder uses contiguous fixed-capacity adjacency, adaptive dense/sparse visited
state, heap frontiers, and reusable prune scratch; query-local adjacency
retention is bounded and the shared cold cache is sharded for concurrent hits.
- Index selection and architecture: compare all index families and open the detailed page for each implementation.
- DiskANN positioning and tuning: decide when to use DiskANN and configure build, search, local-SSD, and object-store parameters.
- API and language bindings: lifecycle, query parameters, warm-up, Rust, C, C++, Java, Python, and metadata filter pushdown.
- Development and benchmarks: workspace layout, build and test commands, ANN benchmarks, and storage compatibility checks.
- Storage format specification: normative v1 binary layout and compatibility policy.
GitHub shows committed HTML files as source. To view the styled documentation,
clone the repository and serve docs/ from the repository root:
python3 -m http.server --directory docs 8000Then open http://localhost:8000/. All documentation assets are local, so no separate build step is required.
Public implementations live in core, ffi, include,
jni, java, and python. See the
development guide for responsibilities and
verification commands.
- Read the Contributing Guide.
- Create an issue for a bug report or feature request.
- Join the dev mailing list (subscribe / unsubscribe / archives).
- Talk to the community in the Slack #paimon channel.
Submit an issue or ask a question in GitHub Discussions.
Licensed under the Apache License, Version 2.0.