rocketmq-replicator 同步消息异常

[LittleBoy18]

1、readme 指导操作有误
2、同步消息到白名单设置的topic

[LittleBoy18]

1、优化了 消息同步
2、优化了 消费进度同步

[tsunghanjacktsai]

Hey @LittleBoy18, instead of RmqSourceTask and MetaSourceTask, how about trying the newer version of rocketmq-replicator, which applies the ReplicatorSourceConnector and ReplicatorSourceTask and provides more complete features?

[RockteMQ-AI]

⚠️ Merge conflict detected

This PR has conflicts with the base branch and cannot be merged. Please rebase or merge the base branch into your branch and resolve the conflicts:

git fetch origin
git checkout fix-syncmessage
git rebase origin/main
# resolve conflicts, then:
git push --force-with-lease

This is a one-time reminder. Feel free to @mention me for a re-review after conflicts are resolved.

---

Automated notification by github-manager-bot

[RockteMQ-AI]

Summary

This PR modifies 8 file(s) with 455 lines of diff. No test changes detected — consider adding test coverage.

---

Automated review by github-manager-bot

[RockteMQ-AI]

No test changes detected alongside source modifications. Consider adding tests to cover the changes.

[RockteMQ-AI]

This PR has been open since May 2022 (over 4 years) and currently has merge conflicts with the master branch.

Status check: Is this replicator sync fix still relevant? The changes to MetaSourceTask, RmqSourceTask, and WorkerSourceTask appear to address real issues, but the code needs to be rebased.

If this is still being worked on, please rebase and update. If abandoned, consider closing.

---

Automated review by github-manager-bot

[RockteMQ-AI]

Summary

PR received and logged for review. This PR requires detailed code review by a maintainer.

Diff size: 455 lines
Author: LittleBoy18 (CONTRIBUTOR)

---

Automated review by RockteMQ-AI

[RockteMQ-AI]

Summary

Review of PR #145: rocketmq-replicator 同步消息异常

Findings: 13 issue(s) identified (3 critical).
CLA: unknown

Please address the inline comments above.

---

Automated review by github-manager-bot

[RockteMQ-AI]

NPE risk: clusterAddrTable.get(this.config.getTargetCluster()) can return null if the target cluster name is misconfigured or not yet registered in the target cluster topology. The subsequent clusterNameSet.iterator() would throw NullPointerException with no useful error message. Add a null check and log a meaningful error identifying the missing cluster.

[RockteMQ-AI]

NPE risk: brokerAddrTable.get(clusterName) may return null if a broker listed in the cluster table is not yet in the broker address table (race during broker add/remove). The subsequent brokerData.getBrokerAddrs() would NPE. Additionally, brokerAddrs.get(new Long(0)) may return null if no master (brokerId=0) is available, causing updateConsumeOffset(null, ...) to silently fail inside the catch block. Add null checks for brokerData, brokerAddrs, and brokerAddresMaster.

[RockteMQ-AI]

NPE risk: sourceDataEntry.getExtensions().getInt("queueId") assumes getExtensions() is non-null and contains the "queueId" key. For any record not produced by RmqSourceTask (which is the only place queueId is set), this throws NPE or NoSuchElementException. Should null-check getExtensions() and provide a safe default or skip the MessageQueue-specific send path.

[RockteMQ-AI]

Performance: this.tarMQAdminExt.examineBrokerClusterInfo() is called inside the per-group loop. This is an expensive RPC that returns the same target cluster topology for every consumer group. Hoist this call outside the for-loop and reuse the result across all groups.

[RockteMQ-AI]

NPE risk in stop(): tarMQAdminExt.shutdown() is called unconditionally. If start() fails after creating srcMQAdminExt but before assigning tarMQAdminExt (e.g., startTarMQAdminTool throws MQClientException), stop() will throw NPE on a null tarMQAdminExt. Should null-check before calling shutdown.

[RockteMQ-AI]

Correctness: new MessageQueue(targetTopic, brokerName, queueId) is constructed unconditionally, but targetTopic and brokerName may be null — the null check at line 316 only guards sourceMessage.setTopic(), not the MessageQueue construction. Sending to a MessageQueue with null topic or broker name will fail at the broker. Validate these fields before entering the RocketMQConverter branch or ensure they are always set by the source task.

[RockteMQ-AI]

The early return (not continue) for RocketMQMetaConverter skips the toSendRecord = null cleanup at the end of sendRecord(). If the framework does not reset toSendRecord each cycle, records could accumulate across poll cycles for meta tasks. Use continue or ensure toSendRecord is cleared outside this method for meta tasks.

[RockteMQ-AI]

Maintainability: The SendCallback implementation (onSuccess/onException with stats tracking and position storage) is duplicated verbatim between the RocketMQConverter branch and the else branch (~40 lines each). Extract this into a helper method to avoid divergence bugs when one copy is updated but not the other.

[RockteMQ-AI]

Offset tracking concern: poll() now always returns an empty ConnectRecord list, and sendRecord() returns early for RocketMQMetaConverter, bypassing positionStorageWriter entirely. The framework's position/offset tracking is completely skipped for meta tasks. On task restart or reassignment, no progress is recorded. Verify this is intentional — the direct updateConsumeOffset approach may need at-least-once delivery guarantees that the framework no longer provides for this path.

[RockteMQ-AI]

No test coverage for the substantially changed meta offset sync logic (direct updateConsumeOffset, new startTarMQAdminTool, ClusterInfo traversal) or the RocketMQConverter send-to-specific-queue path in WorkerSourceTask. These are significant behavioral changes that warrant integration tests, especially given the NPE risks identified above.