Skip to content

feat(connectors): add Apache Fluss source connector - #3799

Open
seokjin0414 wants to merge 4 commits into
apache:masterfrom
seokjin0414:3688-fluss-source
Open

feat(connectors): add Apache Fluss source connector#3799
seokjin0414 wants to merge 4 commits into
apache:masterfrom
seokjin0414:3688-fluss-source

Conversation

@seokjin0414

@seokjin0414 seokjin0414 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

What

Adds a source connector that reads an Apache Fluss
log table and publishes each row into an Apache Iggy stream as JSON. This is
phase 1 of #3688: log tables and Schema::Json.

How it works

The connector resolves the table schema while opening, so a missing table, a
primary-key table, a partitioned table or a column type with no JSON form all
fail at startup rather than once per batch. The log scanner is created once in
open() and reused, which works because it owns its client state rather than
borrowing the connection.

Offsets are tracked per bucket and returned in every ProducedMessages,
including empty polls, so the runtime can persist them. On restart, buckets
already present in the state keep their offset and only new ones fall back to
starting_offset, which keeps a widened bucket count from rewinding buckets
that were already consumed. Each message carries an id derived from its bucket
and offset so Apache Iggy can deduplicate after an at-least-once replay.

Column projection is pushed down to the server when columns is set. Temporal
values keep their Fluss-native integer form, because Fluss carries no timezone
that a formatted string could honour.

The packaging question I asked in #3688

fluss-rs compiles its protocol definitions in a build script, so the workspace
now needs a system protoc. I raised this in
#3688 (comment) and said I
leaned towards registering the crate as a regular workspace member, so that is
what the last two commits do: protobuf-compiler goes into the shared Rust
setup action, which every Rust workflow already uses, and CONTRIBUTING gets the
prerequisite.

If you would rather keep it out of the workspace build via exclude, only
build: register the Apache Fluss source in the workspace and
ci: install protoc ... need reworking. The connector and its test do not
change. Happy to flip it.

Testing

Unit tests cover config validation, the state round-trip, offset resolution and
the row-to-JSON mapping (including nulls, base64 for binary and non-finite
floats).

The integration test appends rows to a real Fluss log table and asserts on what
comes back out of the Apache Iggy topic, offsets and bucket included. The
cluster runs as a single container: the image ships local-cluster.sh, which
starts the same embedded ZooKeeper, coordinator and tablet server, but it
rewrites the tablet server's bind port to 0 so it cannot collide with the
coordinator, and a random port inside the container cannot be published to the
host. Giving the tablet server an explicit second port keeps both reachable and
lets both advertise localhost, which then resolves the same way inside the
container and from the test process.

Notes on scope

Two limits come from the released fluss-rs 0.1.0 rather than from the
connector, and each is rejected at startup with the reason rather than silently
ignored:

  • Primary-key changelog scanning is not in the release, so table_type = "primary_key" is refused.
  • payload_format = "arrow_ipc" is doable (0.1.0 does expose an Arrow
    RecordBatch scanner) but it is a separate scanner with its own
    offset-tracking path, so I left it for a follow-up.

starting_offset = "latest" is supported: buckets without persisted state get
their tail offset from FlussAdmin::list_offsets(.., OffsetSpec::Latest) at
open time (verified against a real 0.9.1 cluster: an empty bucket reports 0 and
the offset moves to the row count after appends, so subscribing there yields
only new records).

Refs #3688

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 2, 2026
@codecov

codecov Bot commented Aug 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 79.87013% with 93 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.31%. Comparing base (102484a) to head (26bf766).
⚠️ Report is 25 commits behind head on master.

Files with missing lines Patch % Lines
core/connectors/sources/fluss_source/src/lib.rs 83.74% 46 Missing and 7 partials ⚠️
...ore/connectors/sources/fluss_source/src/mapping.rs 70.58% 32 Missing and 8 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3799      +/-   ##
============================================
+ Coverage     75.96%   76.31%   +0.35%     
- Complexity      969     1046      +77     
============================================
  Files          1320     1335      +15     
  Lines        157608   165084    +7476     
  Branches     130990   137638    +6648     
============================================
+ Hits         119720   125990    +6270     
- Misses        34297    35279     +982     
- Partials       3591     3815     +224     
Components Coverage Δ
Rust Core 76.34% <79.87%> (+0.37%) ⬆️
Java SDK 63.67% <ø> (+0.95%) ⬆️
C# SDK 71.13% <ø> (-1.13%) ⬇️
Python SDK 88.93% <ø> (-4.17%) ⬇️
PHP SDK 84.52% <ø> (ø)
Node SDK 96.34% <ø> (+1.11%) ⬆️
Go SDK 43.08% <ø> (ø)
Files with missing lines Coverage Δ
...ore/connectors/sources/fluss_source/src/mapping.rs 70.58% <70.58%> (ø)
core/connectors/sources/fluss_source/src/lib.rs 83.74% <83.74%> (ø)

... and 178 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Apache Fluss keeps streams as schema-aware columnar log tables, so feeding one
into Apache Iggy so far meant writing a bespoke client. This connector reads a
Fluss log table and publishes each row as JSON, tracking offsets per bucket
through the runtime state API so a restart resumes where the previous run
stopped. Buckets already present in the restored state keep their offset, which
keeps a widened bucket count from rewinding buckets that were already consumed.

Scope follows the released fluss-rs 0.1.0. Primary-key changelog scanning is
absent from that release, and arrow_ipc payloads need a separate scanner with
its own offset-tracking path, so both are rejected at startup instead of being
silently downgraded. Resolving `starting_offset = "latest"` needs an offset spec
type the client does not export, so that value is rejected with the reason.

Column projection is pushed down to the server when `columns` is set, and
temporal values keep their Fluss-native integer form because Fluss carries no
timezone that a formatted string could honour.

Signed-off-by: seokjin0414 <[email protected]>
Exercises the whole path rather than the connector in isolation: rows are
appended to a real Fluss log table, and the test asserts on what comes back out
of the Apache Iggy topic, including the offsets and bucket the connector
attached as metadata.

The cluster runs as a single container. The image ships local-cluster.sh, which
starts the same embedded ZooKeeper, coordinator server and tablet server, but it
rewrites the tablet server's bind port to 0 so it cannot collide with the
coordinator, and a random port inside the container cannot be published to the
host. Giving the tablet server an explicit second port instead keeps both
reachable, and lets both servers advertise localhost, which then resolves the
same way inside the container and from the test process.

The table is created during fixture setup because the connectors runtime starts
before the test body and the connector resolves the table schema while opening.
Writes retry because bucket leadership is assigned shortly after the tablet
server registers, so the first attempts can still be rejected.

Signed-off-by: seokjin0414 <[email protected]>
Registers the connector as a regular workspace member so it inherits the shared
dependency versions and stays inside cargo sort, the version bump script and the
DAG-based test scoping, the way every other connector does.

The alternative is the exclude list, which would keep the workspace build
untouched but costs the workspace dependency inheritance, so every dependency
would be pinned locally and the crate would fall outside that tooling.

Signed-off-by: seokjin0414 <[email protected]>
The Fluss client compiles its protocol definitions in a build script, so protoc
has to be on the PATH for anything that builds the workspace. It is not present
on the runner images and prost-build no longer vendors one, so there is nothing
to fall back on.

The shared Rust setup action already installs system dependencies and is used by
every workflow that compiles Rust, so one entry there covers all of them.

Signed-off-by: seokjin0414 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant