Skip to content

ref(producer): Use arroyo KafkaProducer for commit log - #8585

Open
lvthanh03 wants to merge 1 commit into
masterfrom
tony/commit-log-arroyo
Open

lvthanh03 wants to merge 1 commit into
masterfrom
tony/commit-log-arroyo

Conversation

@lvthanh03

Copy link
Copy Markdown
Member

Ports the producer that sends commit log entries from confluent_kafka.Producer to arroyo's KafkaProducer.

This changes:

  • Swaps to arroyo producer in MultistorageCollector and ProcessedMessageBatchWriter
  • close() on each batch now waits on the produce futures, so a failed commit log write raises before the batch's offsets are committed
  • ConsumerBuilder.flush() closes the producer at shutdown, which also sends anything still queued
  • Adds a test for commit log production and for failed delivery

Refs STREAM-1930

@lvthanh03
lvthanh03 requested a review from a team as a code owner October 20, 2026 23:51
@linear-code

linear-code Bot commented Oct 20, 2026

Copy link
Copy Markdown

STREAM-1930

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant