feat(connectors): add Apache Fluss source connector - #3799
Open
seokjin0414 wants to merge 4 commits into
Open
Conversation
|
Thanks for the PR. It is labeled Slash commands (own line, regular comment) move it around the queue:
See CONTRIBUTING.md for details. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3799 +/- ##
============================================
+ Coverage 75.96% 76.31% +0.35%
- Complexity 969 1046 +77
============================================
Files 1320 1335 +15
Lines 157608 165084 +7476
Branches 130990 137638 +6648
============================================
+ Hits 119720 125990 +6270
- Misses 34297 35279 +982
- Partials 3591 3815 +224
🚀 New features to boost your workflow:
|
seokjin0414
force-pushed
the
3688-fluss-source
branch
from
August 2, 2026 12:17
e8d3b85 to
67c05d3
Compare
8 tasks
Apache Fluss keeps streams as schema-aware columnar log tables, so feeding one into Apache Iggy so far meant writing a bespoke client. This connector reads a Fluss log table and publishes each row as JSON, tracking offsets per bucket through the runtime state API so a restart resumes where the previous run stopped. Buckets already present in the restored state keep their offset, which keeps a widened bucket count from rewinding buckets that were already consumed. Scope follows the released fluss-rs 0.1.0. Primary-key changelog scanning is absent from that release, and arrow_ipc payloads need a separate scanner with its own offset-tracking path, so both are rejected at startup instead of being silently downgraded. Resolving `starting_offset = "latest"` needs an offset spec type the client does not export, so that value is rejected with the reason. Column projection is pushed down to the server when `columns` is set, and temporal values keep their Fluss-native integer form because Fluss carries no timezone that a formatted string could honour. Signed-off-by: seokjin0414 <[email protected]>
Exercises the whole path rather than the connector in isolation: rows are appended to a real Fluss log table, and the test asserts on what comes back out of the Apache Iggy topic, including the offsets and bucket the connector attached as metadata. The cluster runs as a single container. The image ships local-cluster.sh, which starts the same embedded ZooKeeper, coordinator server and tablet server, but it rewrites the tablet server's bind port to 0 so it cannot collide with the coordinator, and a random port inside the container cannot be published to the host. Giving the tablet server an explicit second port instead keeps both reachable, and lets both servers advertise localhost, which then resolves the same way inside the container and from the test process. The table is created during fixture setup because the connectors runtime starts before the test body and the connector resolves the table schema while opening. Writes retry because bucket leadership is assigned shortly after the tablet server registers, so the first attempts can still be rejected. Signed-off-by: seokjin0414 <[email protected]>
Registers the connector as a regular workspace member so it inherits the shared dependency versions and stays inside cargo sort, the version bump script and the DAG-based test scoping, the way every other connector does. The alternative is the exclude list, which would keep the workspace build untouched but costs the workspace dependency inheritance, so every dependency would be pinned locally and the crate would fall outside that tooling. Signed-off-by: seokjin0414 <[email protected]>
The Fluss client compiles its protocol definitions in a build script, so protoc has to be on the PATH for anything that builds the workspace. It is not present on the runner images and prost-build no longer vendors one, so there is nothing to fall back on. The shared Rust setup action already installs system dependencies and is used by every workflow that compiles Rust, so one entry there covers all of them. Signed-off-by: seokjin0414 <[email protected]>
seokjin0414
force-pushed
the
3688-fluss-source
branch
from
August 5, 2026 15:38
67c05d3 to
26bf766
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a source connector that reads an Apache Fluss
log table and publishes each row into an Apache Iggy stream as JSON. This is
phase 1 of #3688: log tables and
Schema::Json.How it works
The connector resolves the table schema while opening, so a missing table, a
primary-key table, a partitioned table or a column type with no JSON form all
fail at startup rather than once per batch. The log scanner is created once in
open()and reused, which works because it owns its client state rather thanborrowing the connection.
Offsets are tracked per bucket and returned in every
ProducedMessages,including empty polls, so the runtime can persist them. On restart, buckets
already present in the state keep their offset and only new ones fall back to
starting_offset, which keeps a widened bucket count from rewinding bucketsthat were already consumed. Each message carries an id derived from its bucket
and offset so Apache Iggy can deduplicate after an at-least-once replay.
Column projection is pushed down to the server when
columnsis set. Temporalvalues keep their Fluss-native integer form, because Fluss carries no timezone
that a formatted string could honour.
The packaging question I asked in #3688
fluss-rscompiles its protocol definitions in a build script, so the workspacenow needs a system
protoc. I raised this in#3688 (comment) and said I
leaned towards registering the crate as a regular workspace member, so that is
what the last two commits do:
protobuf-compilergoes into the shared Rustsetup action, which every Rust workflow already uses, and CONTRIBUTING gets the
prerequisite.
If you would rather keep it out of the workspace build via
exclude, onlybuild: register the Apache Fluss source in the workspaceandci: install protoc ...need reworking. The connector and its test do notchange. Happy to flip it.
Testing
Unit tests cover config validation, the state round-trip, offset resolution and
the row-to-JSON mapping (including nulls, base64 for binary and non-finite
floats).
The integration test appends rows to a real Fluss log table and asserts on what
comes back out of the Apache Iggy topic, offsets and bucket included. The
cluster runs as a single container: the image ships
local-cluster.sh, whichstarts the same embedded ZooKeeper, coordinator and tablet server, but it
rewrites the tablet server's bind port to 0 so it cannot collide with the
coordinator, and a random port inside the container cannot be published to the
host. Giving the tablet server an explicit second port keeps both reachable and
lets both advertise
localhost, which then resolves the same way inside thecontainer and from the test process.
Notes on scope
Two limits come from the released
fluss-rs0.1.0 rather than from theconnector, and each is rejected at startup with the reason rather than silently
ignored:
table_type = "primary_key"is refused.payload_format = "arrow_ipc"is doable (0.1.0 does expose an ArrowRecordBatchscanner) but it is a separate scanner with its ownoffset-tracking path, so I left it for a follow-up.
starting_offset = "latest"is supported: buckets without persisted state gettheir tail offset from
FlussAdmin::list_offsets(.., OffsetSpec::Latest)atopen time (verified against a real 0.9.1 cluster: an empty bucket reports 0 and
the offset moves to the row count after appends, so subscribing there yields
only new records).
Refs #3688