Repository navigation
compute: don't forward implied capabilities while rehydrating a replica - #40418
Merged
Merged
Conversation
`rehydrate_replica` removes a replica and adds it back. If it is the cluster's only replica, the replica map is empty in between, and `remove_replica` called `forward_implied_capabilities`, which treats that as a paused cluster. It advanced the read holds of indexes that had not hydrated past their as-of, and the storage controller let their inputs compact. The replica process is still running those dataflows, and on reconnect compute reconciliation keeps them at the old as-of. A dataflow whose persist reader had not registered yet then halts the process with "cannot serve requested as_of". `maintain` already forwards implied capabilities on every tick, after rehydrating failed replicas, so clusters that really have no replicas still get the downgrade. Refs: CPU-228
DAlperin
force-pushed
the
dov/compute-rehydrate-read-holds
branch
from
November 2, 2026 17:16
ce2c549 to
d439ff0
Compare
Member
Author
|
This unwinds some of the work in #27363, but I think it's worth it for now in lieu of some greater refactoring. |
DAlperin
marked this pull request as ready for review
November 2, 2026 18:26
antiguru
approved these changes
Nov 2, 2026
antiguru
left a comment
antiguru
left a comment
Member
There was a problem hiding this comment.
Looks good, thanks for figuring this out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rehydrate_replicaremoves a replica and adds it back. If it is the cluster's only replica, the replica map is empty in between, andremove_replicacalledforward_implied_capabilities, which treats an empty replica map as a paused cluster. It advanced the read holds of indexes that had not hydrated past their as-of, and the storage controller then let their inputs compact. The replica process is still running those dataflows, and on reconnect compute reconciliation keeps them at their old as-of, because their output frontier is at or below the new one. A dataflow whose persist reader had not registered yet then halts the process withunhandled error in compute_import: ... cannot serve requested as_of, and the replay repeats it.An Antithesis run under heavy network faults hit this. The full timeline, confirmed from persist state history, is in CPU-228.
The fix drops the call from
remove_replica.maintainalready forwards implied capabilities on every tick, after rehydrating failed replicas, so a cluster that really has no replicas still gets the downgrade within a tick.Tests
test/clustergetstest-replica-reconnect-keeps-read-holds. It puts an index on a cluster with one unmanaged replica, pauses the replica's container until the controller times out and reconnects, and asserts that the index's read frontier never passes its write frontier. The reconciliation side of the halt (a kept dataflow whose reader has not registered) is not covered. Reproducing it needs a stalled persist reader registration, which no failpoint provides today.Refs: CPU-228
🤖 Generated with Claude Code