PHOENIX-7977 Stage forwarded replication log files in a per-shard staging subdirectory before atomic rename - #2590
Open
tkhurana wants to merge 1 commit into
Open
Conversation
…ging subdirectory before atomic rename The forwarder copied a fallback-cluster log file straight onto its final <ts>_<origin>.plog name on the peer via FileUtil.copy (overwrite=false). Because FileUtil.copy publishes the file at that replay-eligible name before close() completes, the standby replay consumer could pick up the half-written file, force-recover the HDFS lease, and cause the forwarder's close() to fail with LeaseExpiredException. That threw out of processFile before the STORE_AND_FORWARD -> SYNC_AND_FORWARD throughput check, delaying the mode transition. Fix: copy into a per-shard .staging subdirectory (<shard>/.staging/<ts>_<origin>.plog, overwrite=true to reclaim orphans) and atomically rename it up to the final .plog in the shard directory once fully written. The staged file keeps its real .plog name but is invisible to replay because every listing gates on FileStatus.isFile(), which skips the subdirectory and anything inside it. Replay only ever sees the atomically published file. Delivery remains at-least-once (unchanged); replay idempotency makes a re-publish safe. Tests: forwarder staging/rename/retry/orphan cases in ReplicationLogDiscoveryForwarderTest; .staging-subdirectory exclusion across the listing entry points in ReplicationLogTrackerTest.
Contributor
Author
|
@Himanshu-g81 can you take a look |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The forwarder copied a fallback-cluster log file straight onto its final _.plog name on the peer via FileUtil.copy (overwrite=false). Because FileUtil.copy publishes the file at that replay-eligible name before close() completes, the standby replay consumer could pick up the half-written file, force-recover the HDFS lease, and cause the forwarder's close() to fail with LeaseExpiredException. That threw out of processFile before the STORE_AND_FORWARD -> SYNC_AND_FORWARD throughput check, delaying the mode transition.
Fix: copy into a per-shard .staging subdirectory (/.staging/_.plog, overwrite=true to reclaim orphans) and atomically rename it up to the final .plog in the shard directory once fully written. The staged file keeps its real .plog name but is invisible to replay because every listing gates on FileStatus.isFile(), which skips the subdirectory and anything inside it. Replay only ever sees the atomically published file. Delivery remains at-least-once (unchanged); replay idempotency makes a re-publish safe.
Tests: forwarder staging/rename/retry/orphan cases in ReplicationLogDiscoveryForwarderTest; .staging-subdirectory exclusion across the listing entry points in ReplicationLogTrackerTest.