Fix race when falling back from RDMA to TCP#3406
Open
legionxiong wants to merge 1 commit into
Open
Conversation
Make the handshake state atomic and publish RDMA_OFF before FALLBACK_TCP. This prevents TCP event handling from observing a partially published fallback state. Use a relaxed load for the completion-path check because it does not consume related data. Signed-off-by: Lijin Xiong <[email protected]>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a concurrency correctness issue in the RDMA transport handshake/fallback path by making the RDMA handshake state thread-safe and ensuring the TCP fallback state is published in a safe order, preventing TCP event handling from observing a partially published fallback.
Changes:
- Change
rdma::RdmaEndpoint::_statefrom a plain enum tobutil::atomic<State>to eliminate a C++ data race across handshake/TCP/completion paths. - Reorder fallback publication in
RdmaConnect::StartConnect()to setRdmaTransport::_rdma_state = RDMA_OFFbefore publishingRdmaEndpoint::_state = FALLBACK_TCP. - Use a relaxed atomic load for the completion-path invariant check in
RdmaEndpoint::HandleCompletion()to avoid adding unnecessary ordering overhead on a hot path.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| src/brpc/rdma/rdma_endpoint.h | Makes handshake state (_state) atomic to remove data races between concurrent RDMA/TCP/completion execution contexts. |
| src/brpc/rdma/rdma_endpoint.cpp | Fixes fallback publication ordering and updates the completion-path state check to use an explicit relaxed atomic load. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
chenBright
requested changes
Jul 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make the handshake state atomic and publish RDMA_OFF before FALLBACK_TCP. This prevents TCP event handling from observing a partially published fallback state. Use a relaxed load for the completion-path check because it does not consume related data.
What problem does this PR solve?
Issue Number: #3405
Problem Summary:
RdmaEndpoint::_stateis concurrently accessed by the RDMA handshake,TCP event handling, and completion handling paths, but it is not atomic.
This causes a C++ data race.
In addition,
RdmaConnect::StartConnect()publishesFALLBACK_TCPbefore setting
RdmaTransport::_rdma_statetoRDMA_OFF. A concurrentTCP event may therefore observe a partially published fallback state and
start TCP message processing while the transport still appears to use
RDMA.
What is changed and the side effects?
Changed:
RdmaEndpoint::_statetobutil::atomic<State>.RdmaTransport::_rdma_statetoRDMA_OFFbefore publishingRdmaEndpoint::_stateasFALLBACK_TCP._statecheck inRdmaEndpoint::HandleCompletion(), because the check does not consumedata published by the state transition.
Side effects:
Performance effects:
The handshake and TCP fallback paths use the default sequentially
consistent ordering. These paths are not hot after the handshake
completes. The completion hot path uses a relaxed load, so it does not
introduce unnecessary ordering overhead.
Breaking backward compatibility:
None. This change does not modify any public API, wire protocol, or
externally visible behavior.
Check List: