[#1061] Ask for the session restart under an owner as well, and leave it to the state checkpointer - #1062
Open
vharseko wants to merge 2 commits into
Conversation
…wound parked as dependencies A change which waits for another one is parked and stays owned by the replay thread which parked it: getNextUpdate() is what hands it out again, to whichever thread clears the change it was waiting for. A replay which is unwound leaves that thread without the road back - it takes the next delivery off the shared queue - so the change stayed owned by a thread which never came back to it, while every redelivery of a change a replay thread owns is refused as a duplicate. On a domain which then went quiet that change was where this replica's ServerState, and every change behind it from every master, stopped. RemotePendingChanges gives back the changes the calling thread parked and unparks them in the same step, under both locks, so that only one road can hand a change out: a change released while it is still listed as waiting would be handed to the thread getNextUpdate() gives it to and to the thread which takes over the delivery which follows. The changes another thread parked are left alone, as everywhere else. The list of what it released is the one allocation of the method past the locks, sized for every change which is waiting before anything is taken out of the set - the rule getNextUpdate() states for itself, on a road out of a JVM which has just refused an allocation - and the second lock is taken inside the try of the first, since taking a lock another thread holds allocates too. replay() gives them back before the road of the change it was replaying runs, since that road restarts the session and a change which is still owned when the replication server sends it again is turned down. They are handed back without a failure counted against them - they were never applied here - and the session is restarted for them, without the backoff on the roads the owned road skips it on: a thread which is stopping, and one an OutOfMemoryError is ending. On a domain whose session has an owner - the domain itself, going away, or a total update into it from the moment it is asked for - they are released and nothing more, the way abandonReplay() hands a change back there: no session of this thread's is left to restart, and the restart it would ask for is refused where it runs. Which change this thread owns is read before they are given back, and not after. The read is a plain map lookup which allocates nothing, and the give-back below it allocates - it builds the list of what it released and the line which reports each one. A throw from it on the road it exists for would otherwise reach the last resort of replay() with nothing read, and the change this thread was replaying would be left listed, uncommitted and owned by a thread which is ending: the wedge OpenIdentityPlatform#922 is about, one road over. That last resort runs the restart outside the guard on the change this thread owned: a give-back which released the parked changes, asked for the restart and then threw reporting them, or a restart the parked road ran and which threw, both leave a request standing on a thread which may own no change of its own. It asks for the restart of the change it owned after releasing it, as every road which releases a change asks, and runs nothing on a domain whose session has an owner: the request stands for the state checkpointer rather than being spent on a restart which is refused. SessionRestartRequests links the call sites of its lambda and of its atomic reference once, at construction: their first execution allocates, and every request is made on a road where an allocation may be what has just failed. The end-to-end test unwinds the replay of a change which is applied and whose ack runs out of memory: the one road which leaves replay() with no change of this thread's to ask for again, so the restart the give-back asks for is the only one and the test is red without it. The change it parks travels the replication server, and nothing but the restarted session brings it back. The road with an owner is pinned where a test can hold it open, a total update whose request is on its way: the changes are released, and nothing is asked for, counted or reported for them. Who runs the restart is pinned with the replay on the thread of the test and the restart asked to fail once: the thread which gave the changes back meets the failure itself, a stopping thread leaves it to the state checkpointer to meet and report. The stopping thread is stopped by the ack of the change it applied, not before the replay: stopped before it, the change is abandoned unapplied at the top of its first attempt and the abandon road asks for a restart of its own next to the give-back's, so the give-back's request would be pinned by nothing - the case asserts that the change was applied, and a third case runs that other road, which pins the abandon arm of the catch. The count of the restarts in a row pins that the give-back asks for the restart without the backoff: only the restart run again after the failure moves it.
…r as well, and leave it to the state checkpointer A change released while the session of its domain had an owner - the domain shutting down or disabled, or a total update into this replica from the moment it is asked for - was released and nothing more: recoverFromReplayFailure(), abandonReplay() and, with OpenIdentityPlatform#954, giveBackParkedChanges() all returned before asking for the session restart. Two of the three owners forget the pending changes and every request with them, disable() with enable() and the import at its end, so nothing was lost there. The third does not: a total update which is asked for and never begins - initializeFromRemote() refused at publish, an ErrorMsg from the exporter, the watchdog of InitializeTask, or the second attempt initialize() makes after a failed import - ends in releaseIEContext() alone, and a change released under it stayed listed, uncommitted and owned by nobody until the next failed replay of this domain restarted the session. On a domain which then went quiet that was for good: a commit moves the ServerState no further than the oldest uncommitted change, so the state in memory, and the one persisted from it, stopped at the change - the shape of OpenIdentityPlatform#954, one road over. OpenIdentityPlatform#968 left it that way because at the time the request was a flag only the thread of a failed replay ran: a flag set under an owner was served by the next failed replay, which would have set it itself. Since OpenIdentityPlatform#981 the state checkpointer runs the request nobody ran, and holds it for as long as the session has an owner - which the last resort of replay() relies on already: its request is made under an owner too, and only its run is gated. The three roads now ask on every road, as one policy: the request is made once the change is released, before the owner is read, and the owner arm returns with it standing. The owners which forget the pending changes forget the request with them; the one which does not leaves it to the checkpointer, which runs it within its tick of the owner letting go and has the replication server send the change again. The lines which say the change is being asked for again, and the count of the deliveries given back, stay off the owner roads: a server which is shutting down abandons every change in flight, and none of them is delivered again before it is started back. The gates on the two runs of replay() - the parked road and the last resort - are load-bearing on every owner road with this: a run there would spend the request on restartSession()'s refusal, where the request left standing is what the checkpointer runs. Pinned in ReplayDuringImportTest on the one owner a test can hold open and then take away: the total update is asked for, the exporter holds the request, and in the place of the answer the request is given up through abortStalledInitializeFromRemote(0). One case per road: a change whose attempts in place are spent under the request, one a stopping thread abandons under it, and one a replay which is unwound had parked - behind a change another thread of the test holds before its operation is built, so that nothing has failed and the give-back's request is the only one standing. Each change travels the replication server, and each is delivered again over the session the checkpointer restarts once the request is given up, replayed and covered. The parked case also pins the gate of the parked road: with the import forgotten by the gate, the run under the owner spends the give-back's request and nothing comes back. TestSynchronousReplayQueue is backed by a ConcurrentLinkedDeque, as the redelivery is offered by the listener thread of the domain and taken by the thread of the test.
vharseko
added
concurrency
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1061.
Stacked on #985. The branch is master (
d07bb31897) +[#954]as #985 stands after its fifth round (f8d55997bb, on that master) + the one[#1061]commit, which is the diff to review. The third of the three roads this closes,giveBackParkedChanges(), is #985's, so the change cannot go on master alone as one policy; if #985 takes another round, this is restacked on its head.What was wrong
recoverFromReplayFailure(),abandonReplay()and - with #954 -giveBackParkedChanges()release the change they hand back and, while the session has an owner (sessionHasAnOwner(): the domain shutting down or disabled, or a total update into this replica from the moment it is asked for), asked for no session restart. Two of the three owners forget the pending changes and every request with them -disable()/enable()and the import'sfinallyclear both - so nothing was lost there. The third does not: a total update which is asked for and never begins -initializeFromRemote()refused at publish, anErrorMsgfrom the exporter, the two-minute watchdog ofInitializeTask(abortStalledInitializeFromRemote()), or the second attemptinitialize()makes after a failed import, which re-publishes the request and keeps the context - ends inreleaseIEContext()alone, and a change released under it stayed listed, uncommitted and owned by nobody until the next failed replay of this domain restarted the session. On a domain which then goes quiet that is for good:commit()moves the ServerState no further than the oldest uncommitted change, so the state in memory, and the one persisted from it, stop at the change - the shape of #954, one road over.#968 left it that way on purpose: the request was a flag which only the thread of a failed replay ran, so a flag set under an owner was served by the next failed replay, which would have set it itself. #981 changed that premise - the state checkpointer runs a request nobody ran, and holds it for as long as
ieRunning()- and the last resort ofreplay()relies on it already: its request is made under an owner as well, and only its run is gated. Found in the fourth review of #985, filed as #1061.What changed
remotePendingChanges.replayFailed(csn)/releaseParkedChangesOwnedByCurrentThread(), thensessionRestarts.request(…), thensessionHasAnOwner()- and the owner arm returns with it standing. What is asked for is what was asked for before:NOWon a thread which is stopping or which anOutOfMemoryErroris ending,AFTER_BACKOFFotherwise;abandonReplay()asksNOWas it did. The owners which forget the pending changes forget the request with them (disable():5221,enable():5401, the import'sfinally:5916); the one which does not leaves it to the checkpointer, which runs it within its tick ofreleaseIEContext()and has the replication server send the change again.WARN_REPLAY_RETRYING_CHANGE(through the throttle of Replication: the replay retry warning is logged once per delivery, with nothing bounding it once the give-up budget is raised #942),NOTE_REPLAY_ABANDONED_CHANGEandNOTE_REPLAY_PARKED_CHANGE_GIVEN_BACKare not logged under an owner, and the parked deliveries are not counted as processed, as before: a server which is shutting down abandons every change in flight, and none of them is delivered again before it is started back, so a line which says the replication server sends the change again would not hold on every owner road. On the never-begun road the change is now asked for again without a line saying so, and a parked delivery given back there is counted by the delivery which replays it; said here rather than papered over with a line of its own.replay()become load-bearing - the parked road (:2826) and the last resort (:2910): a request now stands under every owner, and a run there would spend it onrestartSession()'s refusal, where the request left standing is what the checkpointer runs. The javadoc ofsessionHasAnOwner(), the comments of the three arms, of the parked road, ofenable()(whose account of thedisabled = falseordering rested onabandonReplay()asking for nothing) and of the test hookrequestSessionRestart()say what holds now.What this does not close. A no-owner thread which read no owner, made its request and reached
runRequestedSessionRestarts()while the owner arrived spends the merged flag - its own and any request made under the owner in between - onrestartSession()'s refusal. That window is the few statements between the read and the lock, the same class as the last resort's today, and it is not a regression (before this, nothing was asked for under the owner at all); closing it for good meansrestartSession()saying it refused and the caller giving the request back, which is a change to the loop ofrunRequestedSessionRestarts()and belongs to a follow-up. #1049 closes the import direction of it in passing - itsrunRequestedSessionRestarts()leaves the requests standing whileieRunning()- and #1045 turns the owner read ofrestartSession()into a claim. With #1049 in, the import part of the parked road's gate is no longer load-bearing (the request is not taken), and the mutant below which pins it would survive; whichever lands second re-checks.Tests
All in
ReplayDuringImportTest, on the one owner a test can hold open and then take away:initializeFromRemote()with the exporter holding theInitializeRequestMsg, and in the place of the answerabortStalledInitializeFromRemote(0)- the watchdog's road, and the one road out of an unanswered request a test can take at a time of its choosing (anErrorMsgreleases the context only through the task, which the test does not have; production always has one). Each change travels the replication server - published by the exporter's broker, taken off the domain's synchronous queue by the test - which is what has it to deliver again; a message handed to the domain, as the class's other cases do, is one the replication server never had.aChangeReleasedUnderARequestWhichIsNeverAnsweredIsDeliveredAgain(new)recoverFromReplayFailure()- the attempts in place spent on an entryUUID search which does not run!cover(csn); after the abort the change is delivered again within the checkpointer's tick + the 1 s backoff, replayed, applied, coveredaChangeAbandonedUnderARequestWhichIsNeverAnsweredIsDeliveredAgain(new)abandonReplay()- replayed with the stopping flag set, abandoned at the top of its first attemptaParkedChangeGivenBackUnderARequestWhichIsNeverAnsweredIsDeliveredAgain(new)giveBackParkedChanges()- the parked change waits for one another thread of the test holds before its operation is built (ModifyMsgWhoseOperationWaitsToBeBuilt, new), so nothing has failed and the give-back's request is the only one standing; the replay unwound is the OOME-on-ack fixture of #954dependent-changes-size1 → 0, held change committed when let go, parked change delivered again after the abort and coveredaParkedChangeGivenBackWhileTheRequestIsOnItsWayLeavesTheSessionToTheOwner(renamed from…IsNotAskedForAgain, assertions unchanged)The three new cases were run first against the class as it stands on #985's head (
git show HEAD:…LDAPReplicationDomain.javacompiled ahead oftarget/classes): 3/3 red, each on the wait for the redelivery - "was not delivered again once the request gave up … within 30000 ms".Mutants, each compiled in place of the class and run against the case meant to catch it:
recoverFromReplayFailure()asks only when the session has no owner (the request moved back below the owner arm)abandonReplay()asks only when the session has no ownergiveBackParkedChanges()asks only when the session has no owner!shutdown.get() && !disabledin place of!sessionHasAnOwner()(:2826)TestSynchronousReplayQueueis backed by aConcurrentLinkedDequerather than aLinkedList- the redelivery is offered by the listener thread of the domain and taken by the thread of the test - byte for byte the change #1049 makes to the same file, so the two merge in either order.Run on this head, one JVM per class:
ReplayDuringImportTest9/9,ParkedChangeGiveBackTest3/3 (as of #985's fifth round),SessionRestartBackoffTest3/3,SessionRestartTest2/2,DisabledDomainServerStateTest2/2,RemotePendingChangesTest24/24,SessionRestartRequestsTest7/7,ServerStateFlushTest4/4,LDAPReplicationDomainConfigChangeTest9/9,DependencyTest3/3,UpdateOperationTest40/40,InitOnLineTest10/10,GenerationIdTest4/4,ReSyncTest2/2,NamingConflictTest21/21,AssuredReplicationPluginTest14/14,ReplicationDomainTest12/12 - the classes which restart a session, drive a total update or read the requests, 169 tests,Skipped: 0.