Hand the replication-ack wait to a background thread (port of #1963, with measurements) - #6144
Draft
emelialei88 wants to merge 1 commit into
Draft
Hand the replication-ack wait to a background thread (port of #1963, with measurements)#6144emelialei88 wants to merge 1 commit into
emelialei88 wants to merge 1 commit into
Conversation
Port of bloomberg#1963 onto current main. Block processors commit locally and then block in trans_wait_for_seqnum_int until every replicant acks, so with the default 8 writer threads only 8 commits can be in flight regardless of client concurrency. This hands the wait to a dedicated seqnum-wait thread and returns the block processor to the queue immediately; the ack is still performed, so replicants stay bounded. Off by default (async_dist_commit). The thread always starts but stays idle until work is queued, so the tunable can be flipped at runtime. Rather than carry bloomberg#1963's copy of bdb_wait_for_seqnum_from_all_int, which predated durable-majority accounting, gbl_2pc, non_durable_retry and set_durable_lsn, the shared logic is factored out of main into bdb_wait_for_seqnum_finish and bdb_wait_for_seqnum_mark_incoherent. Signed-off-by: Emelia Lei <[email protected]>
roborivers
approved these changes
Aug 17, 2026
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Success ✓.
Regression testing: Success ✓.
The first 10 failing tests are:
reco-ddlk-sql **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
tunables
sc_downgrade [timeout] **quarantined**
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of #1963 (3126 commits behind main, hand-ported rather than rebased) onto current main, plus an A/B measurement of what it actually buys.
Why. Block processors commit locally and then block in
trans_wait_for_seqnum_intuntil every replicant acks. With the default 8 writer threads, that caps in-flight commits at 8 no matter how many clients are connected.What. A dedicated seqnum-wait thread takes the wait; the block processor returns to the queue immediately. The ack still happens, so replicants stay bounded — unlike
rep-sync none, which drops it and drove two nodes permanently incoherent inside one 5-minute run at 64 clients. Off by default (async_dist_commit), togglable at runtime.Rather than carry #1963's copy of
bdb_wait_for_seqnum_from_all_int— which predated durable-majority accounting,gbl_2pc,non_durable_retryandset_durable_lsn— the shared logic is factored out of main intobdb_wait_for_seqnum_finishandbdb_wait_for_seqnum_mark_incoherent.Measured. 8-node btda cluster, 300s runs, single-row autocommit INSERTs, one table per client, inserts/sec:
Scope confirmed reachable: 726,674 commits enqueued vs 726,658 rows, zero inline fallbacks.
Risk / why this is a draft. The gain is +2.4–3.6% and flat across client counts, so writer-thread starvation was not the binding constraint — the working hypothesis going in. Raising
async_dist_commit_max_outstanding_trans8→32→128 does not help either.rep-sync noneshows 12–35% of real headroom, but this is not the lever that reaches it. Opening for the port and the numbers; I would not merge it on a 3% return for ~500 lines of concurrent code that has to be kept in step withbdb_wait_for_seqnum_from_all_intforever.Three latent bugs found and fixed during the port, which would have hit the original too:
thrman_register(THRTYPE_GENERIC)madebegin_clean_exit()wait forever on a thread that never exits; the waiter tookBDB_READLOCKwith no bdb lock slot and killed the master with[FATAL] bdb lock not inited in this thread; andgbl_set_coherent_state_tracedefaults to 1, whose per-node-per-poll trace is catastrophic for a polling waiter.