Skip to content

[rust] Check in generated protobuf code and drop the protoc build requirement - #662

Closed
seokjin0414 wants to merge 3 commits into
apache:mainfrom
seokjin0414:commit-generated-proto
Closed

[rust] Check in generated protobuf code and drop the protoc build requirement#662
seokjin0414 wants to merge 3 commits into
apache:mainfrom
seokjin0414:commit-generated-proto

Conversation

@seokjin0414

@seokjin0414 seokjin0414 commented Aug 4, 2026

Copy link
Copy Markdown

Fixes #661.

fluss_api.proto is static, but build.rs reran prost-build on every build, making a system protoc a requirement for every consumer of fluss-rs and for most workflows in this repo. This checks the generated code in, following the arrow-flight layout (committed prost output + a small gen crate + regen.sh):

  • crates/fluss/src/proto/proto.rs: checked-in prost output, byte-identical to what build.rs produced (same prost-build version, same bytes config), with the ASF header and a do-not-edit notice prepended
  • crates/fluss/gen: a tiny bin crate that regenerates it. prost-build is a normal dependency there, so the workspace compiles without protoc; protoc is only needed when actually running crates/fluss/regen.sh
  • build.rs and the [build-dependencies] on prost-build are removed; cargo package -p fluss-rs --list confirms proto.rs ships in the crate and the nested gen crate is excluded automatically
  • a new proto-up-to-date CI job reruns the regeneration and fails on any diff, so proto edits cannot land with stale generated code. The output is deterministic: prost-build is version-locked through Cargo.lock, and generation is byte-identical across protoc 27.1, 29.3 and 35.1
  • the two Rust CI workflows stop installing protoc otherwise, so CI itself now proves the protoc-free build. The installs in the python/cpp/elixir/release workflows and the protoc gate in bindings/cpp/BUILD.bazel also become unnecessary, left for a follow-up to keep this reviewable
  • DEVELOPMENT.md: protobuf is now only a prerequisite for regenerating

The first commit fixes the two clippy::for_kv_map errors that Rust 1.97 surfaced in accumulator.rs; they have kept check_license_and_formatting red on main since Jul 15, and this PR cannot go green without them.

Verified locally with no protoc installed: cargo build, cargo clippy --all-targets --workspace -- -D warnings, cargo test --all-targets --workspace with the CI excludes (572 passed), cargo fmt --all -- --check, cargo doc --workspace --no-deps --exclude fluss_python, and cargo check -p fluss-rs --features integration_tests --all-targets. Rerunning regen.sh reproduces the committed file byte-for-byte, and cargo deny check licenses plus the skywalking-eyes header check pass unchanged.

Context: apache/iggy#3688 (Fluss source connector for Apache Iggy; the protoc build requirement is the blocker there). If this lands, a 0.1.1 patch release would remove the protoc requirement for 0.1.0 consumers too.

Rust 1.97 stable flags these two loops in accumulator.rs under
-D warnings, which has kept check_license_and_formatting red on
main since Jul 15.

Signed-off-by: seokjin0414 <[email protected]>
…uirement

fluss_api.proto is static, but build.rs reran prost-build codegen on
every build, forcing a system protoc onto every consumer of fluss-rs
and onto most CI workflows in this repo. Check the generated proto.rs
in and replace build.rs with a small gen crate plus regen.sh, following
the arrow-flight layout: prost-build is a normal dependency of the gen
tool, so building never invokes protoc, only regenerating does. The
checked-in file is byte-identical to the previous build.rs output.

Signed-off-by: seokjin0414 <[email protected]>
Regenerates src/proto/proto.rs in CI and fails if the result differs
from the checked-in file, so proto edits cannot land without rerunning
regen.sh. The output is deterministic: prost-build is version-locked
through Cargo.lock, and the generated code is byte-identical across
protoc 27.1, 29.3 and 35.1.

Signed-off-by: seokjin0414 <[email protected]>
@seokjin0414

Copy link
Copy Markdown
Author

The Rust client moved into the apache/fluss monorepo, so this is re-targeted there: apache/fluss#3874. The monorepo version regenerates from the canonical FlussApi.proto (package fluss) instead of the old vendored subset, and drops the clippy commit since that is already fixed there. Closing.

@seokjin0414 seokjin0414 closed this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Check in the prost-generated code so builds don't require protoc

1 participant