[#956] Tell a failed entryUUID search apart from an entry which is not there

[vharseko]

solveNamingConflict() decides an entry is gone by searching for its entryUUID and getting nothing back, and getFirstResult() answers the same thing for a search which found nothing and for a search which never ran:

private static SearchResultEntry getFirstResult(InternalSearchOperation search)
{
  if (search.getResultCode() == ResultCode.SUCCESS) { ... }
  return null;                       // no entry, or the search never ran
}

Every caller in conflict resolution read that null as "the entry has been deleted" and answered NOTHING_TO_DO, and that branch commits the CSN unconditionally - it never got the guard #892 added two lines below it, on case FAILED. So a change which was never applied was recorded as replayed, the replication server never sent it again because this replica reported itself past that CSN, and no alert was raised: the #889 failure mode, through a branch #892 did not harden.

What this changes

• findEntryDN() reports a search which did not run instead of answering "no entry" out of it: a non-SUCCESS result code is a SearchFailedException. One result code is looked at twice: a backend which serves the base DN and has no base entry answers NO_SUCH_OBJECT on every route (EntryContainer.searchIndexed fetches the base entry before it returns success, MemoryBackend.search checks it first) - and so does a backend which is not there. The backend itself tells the two apart, baseEntryIsAbsentFromALiveBackend(): the search ran over a backend which is there and empty, and it did not run over one which is gone or fails to answer. Without that, the base entry of a domain replayed into an empty replica - two empty replicas share the generation ID of an empty backend, so no initialization is needed and the base entry is the first change - would be retried until the give-up budget skipped it.
• The entryUUID is looked up as the value it is rather than read as part of a filter string: it comes off the wire and nothing validates it as one, and "entryuuid=" + uuid made a value which does not parse - a dangling escape - a search which never runs, retried as a transient failure for as long as the change was asked for. Looked up as a value, such an entryUUID names no entry, which is what a search which ran and found nothing says. The // never happens because the filter is always valid comment is true now.
• The replay takes the failure as the failure of the server it is: a new ConflictResolution.SEARCH_FAILED gets the in-place attempts a storage which failed gets, and the change is left out of the ServerState once they are spent, so the replication server delivers it again. The result code of the attempt says nothing of it - it is the conflict the operation failed on - so the attempt keeps the search failure and the exhaustion exit reports it, in the ERR_ERROR_REPLAYING_OPERATION line it already logs, in place of the error of the operation. Nothing is logged per attempt in place, as nothing is for a storage which failed to serve the operation: a backend which is down for a while fails every attempt of every change delivered meanwhile. What the exit reports is the attempt which spent the last of them: the mark an attempt leaves is reset at the top of every attempt, so one the server refused before it reached the data reports that refusal rather than the search of an earlier attempt (review round 2).
• The entryUUID searches which check a replayed Add for a conflict before it runs (handleConflictResolution(PreOperationAddOperation)) are treated the same way: the operation is stopped with UNAVAILABLE and the search which did not run as its error message, which is what the exhaustion exit reports. Reading the first of them as "not replayed here yet" adds an entry a second time when it was renamed since; reading the second as a parent which is gone hands the Add to conflict resolution as the naming conflict it is not, and renames the entry under the base DN as a conflicting entry when the search conflict resolution makes fails as well.
• A total update into this replica owns the session of the domain, and the replay stays out of the data it replaces (review round 3). preBackendImport() takes the backend away without disabling the domain - it can not stop the session it is about to import over - so a change queued for replay before the InitializeTargetMsg arrived was replayed into no backend: NO_SUCH_OBJECT from the workflow element, then a search which did not run, SEARCH_FAILED ten times, the change given back and the session restarted for it - which stopped the broker the import was reading. receiveEntryBytes() records no exception for a broker which is shutting down, so the import ended on the entries which had arrived and was reported as finished, a truncated dataset under the exporter's generationId. On the base the same change was NOTHING_TO_DO, and its CSN wiped by loadDataState(). Now importBackend() sets importingData and drains the attempts in flight before the backend is taken away, the way disable() does for an import run on this server; the replay threads read the flag where they read disabled and give the change back without a restart; sessionHasAnOwner() - ownsItsSession() || importInProgress() - refuses the restart in recoverFromReplayFailure(), abandonReplay() and restartSession() for the whole of the total update, from the request rather than from the first entry, since the InitializeTargetMsg which answers the request arrives over that session too; and once the state is loaded from the imported data, the pending changes, the restart request and the backoff are reset, as disable() resets them. Without that reset a change given back during the import stays listed and uncommitted, commit() never moves the ServerState past the oldest uncommitted change, and the imported state already covers it, so the replication server never sends it again - the ServerState of the replica would stop for good. The guard after the backoff in restartSession() stays on ownsItsSession(): the session that thread stopped is the one an import would stream over, so none is streaming, and a total update asked for meanwhile needs the session started back to be answered at all.

findEntryUUID() is deliberately left alone: a search which fails there leaves a locally originated ModifyDN published without the entryUUID of its new superior, which is a bug on what this server sends rather than on what it records. It deserves an issue of its own.

Rebased on master

The branch sits on master as it is now, which carries #948, #965 and - since the review round - #935, #959, #969, #970, #972, #973, #975, #976, and - since round 2 - #958 and #964, and - since round 3 - #974: the commits rebased without a conflict each time, git range-diff reads them as the same text on both bases, and what #958 adds to LDAPReplicationDomain sits above the replay loop and below its verdict, with the loop body between them as it was. #974 moved serviceStateLock and the session generation into ReplicationDomain and put the guards of restartSession() on ownsItsSession(); the round-3 change is written on that shape - sessionHasAnOwner() is ownsItsSession() with the total update added, and the post-backoff guard is #974's as it is. The only conflict of the rebase before it was replication.properties, where #959 and #972 added 326 and 327 next to this branch's 322; both sides are kept. The Java merged on its own, and #972 reads the same situation this change lets through: a suffix whose base entry is not in the backend is "what a suffix waiting to be initialized looks like", and its generationId is now left unstored rather than written to the configuration entry - StateWithoutBaseEntryTest is 3/3 on this branch.

#948 put the in-place attempt of a replayed change under the replay read lock it introduced, and
the SEARCH_FAILED retry is inside it: its 50 ms are waited where the FAILED retry of #892
already waits - a domain on its way down takes that lock exclusively and waits out the attempt in
flight, as it does for every other in-place retry.

#965 answers a ModifyDN whose entry is gone before the new superior is looked up, so
solveNamingConflict(ModifyDNOperation) reads the entryUUID search first and returns on it. A
search which did not run now leaves that method rather than being read as an entry which is gone -
it declares throws Exception, so the SearchFailedException reaches the catch in the replay
loop, which is where it is turned into SEARCH_FAILED.

Tests

Ten cases in NamingConflictTest, driven by ShortCircuitPlugin on SEARCH/PreParse. The plugin grew a registerShortCircuit(..., letThroughFirst, maxTimes) so that a failure can start part way through the searches of an attempt - the second search failing while the first ran - and every case which registers a bounded short circuit asserts, before it is deregistered (which drops the count), that the budget was spent and the search after it ran. Round 3 added an overload with a Predicate<PluginOperation>, for a short circuit which is for the replayed operation only - the ones the predicate does not accept are neither refused nor counted - because the ServerState flush thread writes the base entry with a Modify of its own on its tick, and a MODIFY short circuit which counted it took the let-through, or a refusal, meant for the replayed operation.

• modifyIsRetriedWhileTheEntryUUIDSearchCanNotRun, deleteIsRetriedWhileTheEntryUUIDSearchCanNotRun, modifyDnIsRetriedWhileTheEntryUUIDSearchCanNotRun - a change on an entry which was renamed here, so that only the entryUUID search finds it; the search fails twice and is served on the third attempt. Without the fix: NOTHING_TO_DO after a single search, the change dropped and the CSN recorded as replayed.
• addIsNotReplayedTwiceWhileTheEntryUUIDSearchCanNotRun - an Add delivered a second time whose entry was renamed since; the first search of the first attempt fails. Without the fix the entry is added a second time under its former DN, one entryUUID twice in the data.
• addIsRetriedWhileTheParentEntryUUIDSearchCanNotRun - the parent check fails while the search before it ran; pinned by the monitor: no naming conflict is counted for a search which read nothing. Without the fix conflict resolution counts one and rewrites the message to the DN it already carries.
• addIsRetriedWhileTheConflictResolutionSearchCanNotRun - the parent was renamed here, so the Add fails on a genuine conflict, and the search conflict resolution reads the data with fails. Without the fix the entry is renamed under the base DN as a conflicting entry; with it, the conflict is counted once, when it is solved.
• modifyIsLeftOutOfTheServerStateWhenTheEntryUUIDSearchNeverRuns - every attempt in place fails its search: the change is not in the ServerState and the entry untouched. This is the case which pins the exhaustion exit: with || searchFailedResolvingConflict != null removed, the exit falls into the ERR_LOOP branch and commits the CSN. Since round 3 it also pins what the exit reports: every ERR_ERROR_REPLAYING_OPERATION record of the CSN names the entryUUID, which only the search failure carries - a mutant which always reports op.getErrorMessage() fails here.
• theExhaustionExitReportsTheAttemptWhichSpentTheLastOfThem - the first attempt ends on a search which did not run, and the Modify itself is refused with UNAVAILABLE on the nine after it (letThroughFirst=1 on MODIFY, for the replayed operation only, named by its CSN): the ERR_ERROR_REPLAYING_OPERATION record of the CSN must not name the entryUUID, which only the search failure carries. Without the reset it reads error Unavailable Could not read the data to check change ... the search of the entry with entryUUID ... did not run - two causes on one line. The flush thread's Modify of the base entry is made inside the window, by the case itself rather than left to the tick, and the case asserts it was not counted: without the predicate it is, and the tick landing before the first attempt or between two of them is a red under load.
• baseEntryIsAddedToAnEmptyReplica - the base entry replayed into a backend without one. Fails on the previous head of this branch: ten UNAVAILABLE attempts, the change left out of the ServerState, no base entry.
• anEntryUUIDWhichIsNotOneNamesNoEntry - an entryUUID no filter string parses; the change is resolved as one on an entry which is not in the data, and recorded.

Three cases in ReplayDuringImportTest, a class of its own because a total update needs the userRoot backend - the memory backend of o=test loses its data when it is disabled and enabled back, which is what an import does to the backend it replaces. The exporter is a broker of the test, so that the test says when the entries arrive: it publishes the InitializeTargetMsg, waits for the backend of the domain to be deregistered, replays a Modify on a stale DN through the synchronous replay queue while the import waits for its entries, and only then sends them and the DoneMsg. For the request window it is the domain which asks (initializeFromRemote(), no task, so no stalled-request watchdog), and the exporter holds the InitializeRequestMsg until the change has been replayed.

• aReplayDuringTheImportLeavesTheSessionToTheImport - every entry the exporter sent is in the backend once the import ends, no WARN_REPLAY_RETRYING_CHANGE names the change, and - since round 4 - no ERR_ERROR_REPLAYING_OPERATION does either: the hold-off gives the change back before an attempt is made, and that record is the one thing which tells it from the owner guard saving the session after ten attempts into no backend. On 03ed5cf it fails on the second entry: the log reads the exhaustion exit, the retry line, then Processed 0 entries, imported 0. With || importingData deleted from both goingDown reads it fails on the exhaustion record, the other cases green.
• aRequestOnItsWayOwnsTheSessionTheAnswerArrivesOver (round 4) - the total update asked for and its answer held by the exporter, the backend live, a Modify whose entryUUID search never runs (ShortCircuitPlugin on SEARCH) spent in place: the exhaustion exit reports it, no WARN_REPLAY_RETRYING_CHANGE follows, the domain is still connected, and the import then runs to its end over that session. This is the case which reaches sessionHasAnOwner() in recoverFromReplayFailure() - the other two leave at the hold-off - and with that guard on ownsItsSession() alone it fails on the retry warning, the other cases green.
• aChangeGivenBackDuringTheImportDoesNotHoldTheServerStateBack - a Modify on an entry the import brought, replayed once the import is over, is applied and covered by the ServerState. With the reset after loadDataState() removed it is applied and not covered: the change given back during the import is the barrier.

Mutation runs, each on the final tests: with findEntryDN() answering null for a search which did not run and the exhaustion term removed, seven of the nine round-1 cases fail (baseEntryIsAddedToAnEmptyReplica and the filter case are the two that behaviour does not reach); with the two catches of the Add hook reverted to "no entry" and everything else kept, addIsNotReplayedTwice... fails on the duplicate entry and addIsRetriedWhileTheParent... on the conflict counted.

NamingConflictTest is 18/18 and ReplayDuringImportTest 3/3 on the round-4 head (the two classes rerun there; the round-4 change to production code is two comments), and ReplayDuringImportTest 2/2 at 3368a1d, and the replication/plugin package with UpdateOperationTest, AssuredReplicationPluginTest, InitOnLineTest, GenerationIdTest and ReSyncTest - the suites which drive a total update - is 314/314 on the rebased tree, nothing skipped. Before round 3 the replication/plugin package with UpdateOperationTest and AssuredReplicationPluginTest was 294/294 on the tree with #958 and #964 - UpdateOperationTest 31/31, StateWithoutBaseEntryTest 3/3, nothing skipped. The org.opends.server.replication.** package ran on the review-round commit before the rebase: 3595 tests with 2 failures, both setUp of an embedded server which did not get the admin port it binds (Address already in use on 65534 and 65530 - test servers of other checkouts on the same machine), which took the 65 methods of ProtocolCompatibilityTest and FileChangeNumberIndexDBTest into skipped; on the rebased tree the two classes are 58/58 and 5/5, and NamingConflictTest 17/17 again.

Not in this change

• A change which never applied is recorded as replayed when server-error-result-code is set to NO_OPERATION or SUCCESS #953 - SUCCESS, NO_OPERATION and BUSY being read before either guard is consulted.
• NPE in solveNamingConflict(ModifyDNOperation) when both the moved entry and its new parent are gone #955 - the NPE in solveNamingConflict(ModifyDNOperation) when both the moved entry and its new parent are gone: fixed on master by [#955] Answer a ModifyDN whose entry is gone before the new superior is looked up #965, which this branch sits on. What this change adds there is that a search which did not run no longer reaches that decision at all.
• The second half of A failed entryUUID search reads as a deleted entry, and conflict resolution records the change as replayed #956: case NOTHING_TO_DO refusing to commit the CSN while the result code is the configured server-error-result-code. With the search telling a failure from an empty answer, NOTHING_TO_DO is only reached when the search did run and the entry really is not in the data, and the guard would refuse legitimate no-ops - a Modify on an entry genuinely deleted elsewhere would be retried until the give-up budget raised a false "this replica diverged" alert.
• Replication: the session restart a configuration change asks for cuts a total update into this replica #1040 - restartService(), the restart a configuration change asks for, cuts an import into this replica the way the replay did: ownsItsSession() reads "disabled for the length of a total update", and the total update into this replica never sets disabled. Pre-existing and [#926] Restart the session of a replication domain in one place, under the lock and the generation #974's shape, on a road rarer than a replay - a change of the domain's configuration while it is being initialized, or of its external changelog entry, whose listener refuses nothing. sessionHasAnOwner() is the predicate it would take.
• Replication: a total update whose session stops before the DoneMsg ends as a finished import, with the exporter's generationId over partial data #1039 - nothing turns a truncated import into a reported failure: entryLeftCount only drives the progress of the task, nothing compares it with zero on the way out, and a broker which stopped ends the stream the way the exporter's DoneMsg does. This change keeps the replay from stopping that broker; what would make the import say it was cut is that issue.
• Replication: the owner read of a session restart and the listener's claim of a remote-initiated import share no lock #1041 - the owner read in restartSession() and the listener's acquireIEContext() for a remote-initiated InitializeTargetMsg share no lock: a restart decided between the dequeue of the message and the CAS stops the broker the import is about to read - the round-3 shape, a few statements wide instead of the whole import. A design change on the listener side, and rare: it needs the ten attempts of a change spent inside that window.
• An export-direction case. importInProgress() is direction-aware on purpose - an export is not an owner, a failed replay during one restarts the session the EntryMsgs go out over, and the exporter reports the cut; owning the session for an export would leave the change given back with nothing to restart the session for it, since an export reloads no state. Pre-existing, and pinned by nothing yet: a case with the export held in flight (a window smaller than the entry count and the ack withheld) asserting the restart happens is fixture work of its own.
• A domain whose base entry is a smart referral. The search findEntryDN() makes carries no ManageDsaIT - none of the internal searches of the domain do - so the backend answers REFERRAL before it streams an entry. That is a SearchFailedException now, the give-up an administrator sees, where it was a CSN committed for a change never applied; the topology is buildable and unsupported in practice, and reading through the referral there is not what this change is about.

Fixes #956

Ordinal

ERR_REPLAY_ENTRYUUID_SEARCH_FAILED_322, moved off 310 in 4110d0c and renamed from WARN_ once it stopped being logged on its own: it is the error message of the operation the Add hook stops, and the error the exhaustion exit reports. Six open branches had each read
310 as the first ordinal free in master and taken it, and git merges those additions without
reporting a conflict - they land in different places in the file - so the duplicate would only have
surfaced afterwards, as two unrelated messages sharing one support ID. The generator does not check
either: it keys on name and ordinal together, so both compile. The open PRs which add to
replication.properties now hold 310-325 with nothing claimed twice.

[vharseko]

Ordinal moved: WARN_REPLAY_ENTRYUUID_SEARCH_FAILED 310 → 322 (4110d0c).

Six open branches had each read 310 as the first ordinal free in master and taken it - #935, #945,
#959, #964, #968, #977.

Nothing catches this on the way in. The additions land in different parts of the file, so git merges
every pair of them without reporting a conflict - I merged all ten onto master to check, and the
result carried 310, 311, 315 and 316 twice each. The generator does not check either: it loads the
file into a Properties and keys on name and ordinal (MessagePropertyKey.compareTo), so both
sides compile. What comes out is two unrelated messages carrying one support ID, found by whoever
reads a log rather than by CI.

The open PRs which add to the file now hold 310-325 with nothing claimed twice:

310-313 #935 · 314 #959 · 315-317 #958, #985 · 318 #985, #988 · 319-320 #945 · 321 #964 ·
322 #968 · 323-324 #977 · 325 #981

No Java moved with it: the generated constant is the key name without its ordinal, so the rename is
confined to replication.properties. #935, #958 and #985 keep what they had.

[vharseko]

Rebased on master; the conflict was #948 moving the block this change rewrites (bdb4f49)

Master carries #948 now, which put the whole in-place attempt of a replayed change - building the
operation, running it, and the conflict resolution which follows a failure - under the replay read
lock that change introduced. This branch rewrites the same block where it used to be, so git kept
both copies side by side rather than merging them: the same code, in two places, neither of them
wrong on its own.

Resolved by dropping the copy this branch carried and applying its three edits to the block where
#948 put it:

• the dispatch to solveNamingConflict() wrapped in try / catch (SearchFailedException), which
  logs WARN_REPLAY_ENTRYUUID_SEARCH_FAILED and answers SEARCH_FAILED;
• case SEARCH_FAILED ahead of case FAILED, which marks the attempt and waits 50 ms;
• default clearing that mark, so the verdict below the loop reads the attempt which ended it.

The 50 ms are now waited holding the replay read lock, which is where the FAILED retry of #892
already waits: a domain on its way down takes that lock exclusively and waits out the attempt in
flight, whichever retry it is.

replication.properties conflicted for a plainer reason - #945 and #948 added 319 and 320 where
this branch adds its message. Both sides are kept and 322 is still nothing else's.

The diff is the same 270 insertions / 37 deletions over the same three files as before the rebase,
and NamingConflictTest is 9/9 on it. The org.opends.server.replication.** package is being run
again on the rebase; the description carries the numbers of the run made before it until then.

[vharseko]

Rebased again, on master with #965 (d68a79c)

#965 landed while this was waiting, and it reads the same entryUUID search this change is about:
solveNamingConflict(ModifyDNOperation) now answers a ModifyDN whose entry is gone before the new
superior is looked up. The two fit without either giving anything up - the Java merged on its own -
and the shape is the one this change wants: the search is read first, and a search which did not
run leaves the method rather than being answered as an entry which is gone. solveNamingConflict
declares throws Exception, so the SearchFailedException reaches the catch in the replay loop,
which turns it into SEARCH_FAILED - the retry, not the NOTHING_TO_DO which would have recorded
the change as replayed.

The conflict was in NamingConflictTest: #965 and this branch each added a test at the same place
in the file. Both are kept - modifyDnOnAnEntryAndANewSuperiorWhichAreBothGone from #965, then the
two from this branch - and the class is 10/10.

The diff is unchanged at 270 insertions / 37 deletions over the same three files. The
org.opends.server.replication.** package is being run again on this base; the description carries
the numbers of the run made before #965 until it finishes.

The #955 line under "Not in this change" is updated: that NPE is fixed on master now, by #965.
What this change adds there is that a search which did not run no longer reaches the decision at
all.

[maximthomas]

praise: the fix is small and lands where the read happens.

SearchFailedException is thrown at the one site that reads the data (findEntryDN),
every caller either answers SEARCH_FAILED or stops the pre-op with UNAVAILABLE, and the
replay loop needs one flag on top of the server-failure retry it already had. NamingConflictTest
is 10/10 at HEAD, and the OOME / alert contract below the loop is untouched. The comments at
LDAPReplicationDomain.java:2876-2894 and the PR body state the deliberate choices (the parse
failure retried, the error naming the conflict code) instead of leaving the reader to guess, and
ShortCircuitPlugin with a bounded maxTimes is the right tool for "fails, then serves again".

---

opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:3524

issue (blocking): findEntryDN() now throws on NO_SUCH_OBJECT, so a replayed Add of the
domain base entry into an empty replica never lands.

Two empty replicas share EMPTY_BACKEND_GENERATION_ID (48), so no initialize is needed and the
first change replayed is the base entry itself. Its pre-op hook runs findEntryDN(uuid) at
:1856, before the parentEntryUUID == null short-cut at :1875. A backend that serves the base
DN but has no base entry answers NO_SUCH_OBJECT on every route (EntryContainer.fetchBaseEntry);
there is no "SUCCESS with zero entries" path. At BASE that was null -> the Add went through. At
HEAD it is SearchFailedException -> UNAVAILABLE -> 10 attempts -> redeliveries until
replay-give-up-delay (300 s) -> the change is skipped with a "replica diverged" alert.

The same code comes back when no backend serves the DN (SearchOperationBasis:1211,
backend offline or being rebuilt), which is the case this PR must keep catching — so a bare
NO_SUCH_OBJECT whitelist would reopen #956. Ask the backend instead:

if (search.getResultCode() != ResultCode.SUCCESS)
{
  if (search.getResultCode() == ResultCode.NO_SUCH_OBJECT && baseEntryIsAbsentFromALiveBackend())
  {
    // The backend serves the base DN and has no base entry yet: the search ran, and nothing
    // is below a base entry which is not there. This is the empty replica about to receive it.
    return null;
  }
  throw new SearchFailedException(uuid, ...);
}

private boolean baseEntryIsAbsentFromALiveBackend()
{
  final LocalBackend<?> backend =
      getServerContext().getBackendConfigManager().findLocalBackendForEntry(getBaseDN());
  if (backend == null)
  {
    return false; // nothing serves the DN: offline or being rebuilt - the search did not run
  }
  try
  {
    return !backend.entryExists(getBaseDN());
  }
  catch (DirectoryException e)
  {
    return false; // the storage failed to answer - the search did not run
  }
}

And a test that would have caught it — no test in src/test/.../replication replays the
base-entry AddMsg into a backend without one:

@Test
public void baseEntryIsAddedToAnEmptyReplica() throws Exception
{
  TestCaseUtils.initializeTestBackend(false); // the backend, without its base entry
  final Entry base = TestCaseUtils.makeEntry("dn: " + TEST_ROOT_DN_STRING,
      "objectClass: top", "objectClass: organization", "o: test");
  final CSN csn = gen.newCSN();

  replayMsg(addMsg(base, csn, null, "7c1a0d2e-4b6f-4c8a-9e1d-3f5b7a9c1e2d"));

  assertTrue(DirectoryServer.entryExists(base.getName()),
      "the base entry of an empty replica must land: its search found nothing, it did not fail");
  assertTrue(domain.getServerState().cover(csn));
}

---

opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:2899

issue (blocking): the exhaustion exit — || searchFailedResolvingConflict, the half of the
fix that closes #889 — is pinned by no test.

Both new cases fail the search 2 and 3 times against a budget of 10 and then succeed, so the loop
leaves on replayDone and the gate is never reached with this flag as the deciding term. Measured:
with the term removed, NamingConflictTest is still 10/10; with the whole case SEARCH_FAILED
arm deleted (falls into default:), still 10/10. A regression of the exact #889 shape — attempts
spent, CSN committed, change lost — passes CI.

@Test
public void modifyIsLeftOutOfTheServerStateWhenTheEntryUUIDSearchNeverRuns() throws Exception
{
  final Entry entry = createAndAddEntry("modifyWhoseSearchNeverRuns");
  final String entryUUID = getEntryUUID(entry.getName());
  final DN staleDN = DN.valueOf("cn=movedAway," + TEST_ROOT_DN_STRING);
  final CSN csn = gen.newCSN();

  // No maxTimes: every attempt in place fails its search.
  ShortCircuitPlugin.registerShortCircuit(OperationType.SEARCH, "PreParse", ResultCode.UNAVAILABLE.intValue());
  try
  {
    replayMsg(new ModifyMsg(csn, staleDN, generatemods("telephonenumber", "01 02 45"), entryUUID));
  }
  finally
  {
    ShortCircuitPlugin.deregisterShortCircuit(OperationType.SEARCH, "PreParse");
  }

  assertFalse(domain.getServerState().cover(csn),
      "a change whose search never ran is not in the data and must not advance the ServerState");
  assertTrue(ShortCircuitPlugin.getShortCircuitCount(OperationType.SEARCH, "PreParse") >= 10,
      "every attempt in place must have made its search");
}

This assert kills the mutant: without the term the gate falls into the ERR_LOOP branch at
:2913-2924, which commits the CSN. (The exit requests a session restart; the fixture has no
replication server, so if that blocks, replay with a shutdown flag set to true so
runRequestedSessionRestarts(false) returns at once.)

---

opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/NamingConflictTest.java:328

issue (blocking): this comment, and the PR's "searches ... are treated the same way",
describe a flow the test does not run; the parent-search catch at
LDAPReplicationDomain.java:1884-1887 is never reached.

Every attempt makes the first findEntryDN(uuid) at :1856; under the short circuit it fails and
the hook returns before :1882. maxTimes=3 is three attempts, one failed search each; the
fourth attempt passes every search. Measured: reverting the parent catch to BASE semantics
(parentDnFromCtx = null) keeps the class 10/10. Of the three new production lines in the hook the
test sees one.

Minimum: fix the comment (one search per attempt, the first one) and the PR text, and say the
parent catch is pinned by symmetry with :1858, not by a test. ShortCircuitPlugin cannot select
by filter, so pinning the parent search itself needs a skip-first-N short circuit or a filter-aware
one — worth it only if the plugin grows that anyway.

---

opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:2765

suggestion (non-blocking): WARN_REPLAY_ENTRYUUID_SEARCH_FAILED is logged once per attempt
in place — up to 10 lines per delivery, plus 10 more for every redelivery
(WARN_REPLAY_RETRYING_CHANGE at :3296 is one per delivery). The sibling isServerFailure arm
at :2692 logs nothing per attempt. A backend offline for the whole 300 s budget on a busy domain
is changes x 10 x redeliveries lines. Log once per delivery — on the first SEARCH_FAILED, or at
the exhaustion exit where the failure is already reported.

---

opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/NamingConflictTest.java:291

suggestion (non-blocking): neither new case checks that the short circuit fired. A run where
it never does (plugin not loaded in the fixture, search routed elsewhere) stays green on the
sibling assertions. One line per case, after deregisterShortCircuit:

// The count includes the searches let through once maxTimes was spent: > 2 says the
// budget was used and the search after it ran.
assertTrue(ShortCircuitPlugin.getShortCircuitCount(OperationType.SEARCH, "PreParse") > 2,
    "the short circuit must have been spent by the attempts in place");

---

opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:2756

suggestion (non-blocking): the SearchFailedException -> SEARCH_FAILED switch serves all
four conflict resolutions, but only Modify and Add have a case; Delete and ModifyDN are pinned by
nothing. Same mechanism, so low risk — either one case each (a DeleteMsg / ModifyDNMsg on a
stale DN with maxTimes=2, same assertions as the Modify case), or a sentence in the PR saying
they ride on the shared switch.

---

opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:3517

note (non-blocking): a filter that does not parse is a permanent condition retried as a
transient one — 10 attempts, then redeliveries until the give-up budget, for an entryUUID that
will never parse. The PR body says this is deliberate and the cost is bounded, so nothing to
change; only noting that this branch is reached by no test, and that the unescaped
"entryuuid=" + uuid it guards is pre-existing.

[vharseko]

Review round 1 addressed (68cff84)

• findEntryDN() throws on NO_SUCH_OBJECT, the base entry of an empty replica never lands - confirmed, and the shape is as described: EntryContainer.searchIndexed fetches the base entry before it returns success with nothing sent, MemoryBackend.search checks it first, and LocalBackendWorkflowElement.execute answers the same code when no backend serves the DN. Fixed with baseEntryIsAbsentFromALiveBackend() in findEntryDN(), asked of the backend as suggested (getBackend() was already there). baseEntryIsAddedToAnEmptyReplica fails on d68a79c - ten UNAVAILABLE attempts, no base entry - and passes now.

• The exhaustion exit is pinned by no test - modifyIsLeftOutOfTheServerStateWhenTheEntryUUIDSearchNeverRuns added; with the term removed it fails on the CSN committed, as measured. Two things in the sketch did not survive contact with the fixture: deregisterShortCircuit() drops the count with the registration, so the count is asserted before the finally; and a shutdown flag set to true is read before the first attempt (replay() abandons the change at once), so the test runs with SHUTDOWN false and takes the 1 s session-restart wait - 2 s for the method.

• The Add test comment describes a flow it does not run - it did, and worse: with maxTimes short of the conflict-resolution search the old test passed with findEntryDN() answering null for a failed search, because conflict resolution's own search re-read the parent. ShortCircuitPlugin grew registerShortCircuit(..., letThroughFirst, maxTimes), and the Add is now three cases, each pinning one search: addIsNotReplayedTwiceWhileTheEntryUUIDSearchCanNotRun (the first search; the harm is a second copy of a renamed entry), addIsRetriedWhileTheParentEntryUUIDSearchCanNotRun (the parent check; pinned by the monitor - no naming conflict counted for a search which read nothing, which is the one thing that tells the hook's catch from conflict resolution re-reading the parent), and addIsRetriedWhileTheConflictResolutionSearchCanNotRun (the search after a genuine conflict; the parent was renamed here). A run with both hook catches reverted and everything else kept fails the first two.

• One WARN per attempt in place - nothing is logged per attempt now, as nothing is for a storage which failed to serve the operation. The attempt keeps the SearchFailedException, and the exhaustion exit reports it in the ERR_ERROR_REPLAYING_OPERATION line it already logs, in place of the error of the operation - which for this case only named the conflict. The Add hook puts the same text on the operation it stops, so that path reaches the exit line the same way. The message is ERR_REPLAY_ENTRYUUID_SEARCH_FAILED_322 now that it is never logged on its own; the ordinal stays.

• Neither new case checks that the short circuit fired - every case with a bounded short circuit asserts, before deregistering, that the count went past the budget: the searches were made and the one after the budget ran.

• Delete and ModifyDN pinned by nothing - deleteIsRetriedWhileTheEntryUUIDSearchCanNotRun and modifyDnIsRetriedWhileTheEntryUUIDSearchCanNotRun added, the ModifyDN one because solveNamingConflict(ModifyDNOperation) throws through a throws Exception rather than a declaration which names the failure.

• A filter that does not parse is retried as transient - taken further than noting it: the entryUUID is looked up as a value now (SearchFilter.createEqualityFilter), so there is no filter string to parse and no branch left. anEntryUUIDWhichIsNotOneNamesNoEntry replays a change whose entryUUID carries a dangling escape - the one thing a simple filter string refuses - and expects it resolved as a change on an entry which is not in the data; on d68a79c it was retried until the attempts were spent. For the record, the wildcard case is not a hole: UUID syntax has no substring matching rule, so entryuuid=abcd* matched nothing either way.

[vharseko]

Review round 3 addressed, and rebased on master with #974 (3368a1d)

• A change replayed while this replica is the target of a total update restarts the session the import streams over - confirmed, end to end: ReplayDuringImportTest.aReplayDuringTheImportLeavesTheSessionToTheImport drives it with a broker as the exporter (InitializeTargetMsg, then a Modify on a stale DN replayed through the synchronous queue while the import waits for its entries, then the entries and the DoneMsg), and on 03ed5cf the log reads the exhaustion exit, WARN_REPLAY_RETRYING_CHANGE, then Processed 0 entries, imported 0 - the second entry is not there.

  Taken as the first of the two shapes, with two things the sketch did not have:

  • The predicate is importInProgress(), as suggested, and it goes into sessionHasAnOwner() - ownsItsSession() || importInProgress() - which recoverFromReplayFailure(), abandonReplay() and the first guard of restartSession() read; the post-backoff guard stays on ownsItsSession(), since the session that thread stopped is the one an import would stream over, and a total update asked for meanwhile needs the session back to be answered at all. It owns the session from the request rather than from the first entry: the InitializeTargetMsg which answers the request arrives over that session, so a restart made while it is on its way loses it.
  • The replay is held off for the length of the import as well, the way disable() holds it off: importBackend() sets importingData, drains the attempts in flight (awaitReplayDrained(), the bounded wait disable() and shutdown() already make) and only then takes the backend away; the replay threads read the flag where they read disabled, so a change replayed meanwhile is given back at the top of its first attempt - no ten attempts, no error line, no NOTE_REPLAY_ABANDONED_CHANGE per change of a deep queue.
  • What a guard alone leaves behind: a change given back is listed, uncommitted and owned by nobody, commit() stops at the oldest uncommitted change, and the state the import loads is the exporter's, which covers it already - nothing would ever send it again, and the ServerState of this replica would never move past it. So the end of importBackend(), after loadDataState(), resets the pending changes, the restart request and the backoff, as disable() resets them. Pinned by aChangeGivenBackDuringTheImportDoesNotHoldTheServerStateBack: with the reset removed, a Modify replayed after the import is applied and not covered.

  The "no import follows" road is the one thing the guard leaves standing: a change given back while the request was on its way and never answered stays listed until the next failed replay restarts the session, which has the replication server send it again with everything after it - said in the javadoc of sessionHasAnOwner().

  On the open question: nothing turns a truncated import into a reported failure. entryLeftCount only drives the progress of the task, nothing compares it with zero on the way out, and a stopped broker ends the stream the way the DoneMsg does. Left under "Not in this change" as an issue of its own, next to restartService(): the restart a configuration change asks for cuts an import the same way, pre-existing and [#926] Restart the session of a replication domain in one place, under the lock and the generation #974's shape, on a rarer road.

• letThroughFirst = 1 on MODIFY competes with the flush thread - confirmed, and pinned rather than said: ShortCircuitPlugin.registerShortCircuit grew an overload with a Predicate<PluginOperation>, the operations it does not accept being neither refused nor counted, and the case registers it for op -> csn.equals(OperationContext.getCSN(op)). The flush thread's Modify of the base entry is then made inside the window by the case itself (flushLikeTheStateFlushThread(), the internal synchronization Modify PersistentServerState makes, on a harmless attribute) and the case asserts the count is still 0: without the predicate it is 1, the let-through is gone, and the case is red the way the tick makes it red under load.

• The exhaustion text has no positive assertion - modifyIsLeftOutOfTheServerStateWhenTheEntryUUIDSearchNeverRuns now asserts that every ERR_ERROR_REPLAYING_OPERATION record of the CSN names the entryUUID, which only the 322 text carries; a mutant reporting op.getErrorMessage() unconditionally fails on that line, the other 17 green.

The rebase: #974 landed since round 3 and moved serviceStateLock and the generation into ReplicationDomain, with the guards of restartSession() on ownsItsSession(); the four commits rebased without a conflict, git range-diff reads them as the same text on both bases, and the round-3 commit is written on that shape.

NamingConflictTest 18/18 and ReplayDuringImportTest 2/2 at 3368a1d; the replication/plugin package with UpdateOperationTest, AssuredReplicationPluginTest, InitOnLineTest, GenerationIdTest and ReSyncTest - the suites which drive a total update - is 314/314 on the rebased tree, nothing skipped.

[maximthomas]

praise: the blocker is closed for real, and pinned.

• ReplayDuringImportTest run over the previous head's production code is red 2/2 (case 1 at :159, on its own assertion); at head it is green, NamingConflictTest 18/18.
• The fix has the right shape: the total update owns the session from the request (sessionHasAnOwner()), and importingData gives a queued change back on its first attempt instead of its tenth — the half the round-4 sketch did not have.
• ShortCircuitPlugin.registerShortCircuit(..., Predicate<PluginOperation>) with csn.equals(OperationContext.getCSN(op)): the flusher's Modify has no CSN, so the short circuit is blind to it by construction, not by timing.
• The 322 wording now has its positive twin (NamingConflictTest.java:585-590: isNotEmpty() and every record names the entryUUID).
• The rebase over #974 is textually identical to the reviewed head (range-diff = on all four commits).
• The silent truncation named in round 4 is confirmed and filed as its own issue.

---

issue (blocking): the importingData hold-off is pinned by no test.

opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/ReplayDuringImportTest.java:156-164

With || importingData deleted from both goingDown reads (opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:2818, :2831) the class is green 2/2: each replay ran its ten attempts into no backend and left through ERR_ERROR_REPLAYING_OPERATION; the session was saved by sessionHasAnOwner() at :3638 alone. The WARN_REPLAY_RETRYING_CHANGE check is negative by construction — the :3638 guard returns before that warn — so it is green under any mutant of this shape. One assertion pins the mechanism the commit message names:

assertThat(errorLogRecordsOf(ERR_ERROR_REPLAYING_OPERATION.ordinal(), csn))
    .as("the change was attempted into no backend instead of being given back at once")
    .isEmpty();

Zero records at head, one under the mutant.

---

issue (blocking): two comments describe the road before this commit.

opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:3327-3328

// The ack has been published and the change, still owned by the replication
// server, is being delivered again: there is nothing left to replay here.

On the owner road nobody delivers it again until the import restarts the session. Suggested:

// The ack has been published and the change is given back: the replication server
// delivers it again, now or - while a total update owns the session - after the
// import restarts it. There is nothing left to replay here.

opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/ReplayDuringImportTest.java:127-135

The javadoc says the attempts are spent and the restart truncates the import. At head the case never makes an attempt — the hold-off gives the change back at the top — and the truncation is what the case asserts does not happen. Describe the head road: the change is given back at once, the session is left to the import, every exported entry arrives.

---

suggestion (non-blocking): a request-window case would pin sessionHasAnOwner() at :3638 and the stated "from the request rather than from the first entry".

opendj-server-legacy/src/test/java/org/opends/server/replication/plugin/ReplayDuringImportTest.java

Neither case reaches :3638: with the hold-off up the replay leaves at :2818, so swapping sessionHasAnOwner() for ownsItsSession() there alone also survives. Shape: the ieCtx acquired before the InitializeTargetMsg arrives (a locally requested initialize, or its hook), backend live, the replay short-circuited to exhaustion — then assert the broker is still connected and sessionRestartRequested is false. Fixture work; a follow-up is fine.

---

issue (non-blocking): the owner read and the listener's acquireIEContext share no lock — a few-statement race, remote-initiated import only.

opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:3787
opendj-server-legacy/src/main/java/org/opends/server/replication/service/ReplicationDomain.java:2561

A replay thread on the recovery road reads ieCtx == null at :3787, then disableService() → broker.stop(), while the listener CASes the ieCtx for an InitializeTargetMsg and enters importBackend: receiveEntryBytes returns null (ReplicationDomain.java:2130) and the import ends truncated — the round-4 shape, now a few statements wide instead of the whole import. Stopped before the dequeue, the listener exits and the InitializeTargetMsg is dropped on this replica. Needs a ten-attempt exhaustion timed at the message; rare. Not for this PR — a design change (the listener re-checks shuttingDown() after its CAS and records the stop as the import's exception, or the CAS takes the lock the restart guard holds). Please file it.

---

note (non-blocking): restartService() (the #974 configuration road, LDAPReplicationDomain.java:5486, :5558) still restarts under ownsItsSession() — an import in flight is not an owner there. isConfigurationChangeAcceptable refusing while ieRunning() (:5624-5627) covers it, except an import starting between acceptable and apply. "Not in this change" acknowledged; worth its own issue.

---

suggestion (non-blocking): an export-direction case.

importInProgress() is direction-aware by design: an export is not an owner, so a replay failure during an export restarts the session the EntryMsgs go out over. Pre-existing shape and the exporter reports the cut — but nothing pins that the split is intended. One case with acquireIEContext(false) (ReplicationDomain.java:1614) asserting the restart happens.

---

suggestion (if-minor): one sentence in the sessionHasAnOwner() javadoc (LDAPReplicationDomain.java:5544-5546): a change left listed also freezes the ServerState — RemotePendingChanges.commit() stops at the first uncommitted entry — and the persisted state with it, until that next restart.

[vharseko]

Review round 4 addressed (00798ab)

• The importingData hold-off is pinned by no test - confirmed as measured: with || importingData deleted from both goingDown reads the class was 3/3 green here as well, the attempts spent into no backend and the session saved by sessionHasAnOwner() in recoverFromReplayFailure(), which returns before the retry warning. aReplayDuringTheImportLeavesTheSessionToTheImport asserts the ERR_ERROR_REPLAYING_OPERATION records of the change are empty now; under that mutant it is red on that line (one record, the other cases green).

• Two comments describe the road before this commit - both rewritten: the one after recoverFromReplayFailure() in your words, the javadoc of the first case on the head road - the change given back at the top of its first attempt, nothing attempted, nothing reported, the session left to the import - with the old road named as what happens without the hold-off and the owner.

• A request-window case - taken now rather than as a follow-up, since the fixture had the pieces: aRequestOnItsWayOwnsTheSessionTheAnswerArrivesOver has the domain ask (initializeFromRemote() with no task, so no stalled-request watchdog), the exporter hold the InitializeRequestMsg, and a Modify whose entryUUID search never runs spent in place with the backend live: the exhaustion exit reports it, no WARN_REPLAY_RETRYING_CHANGE follows, the domain is still connected, and the import then runs to its end over that session - the InitializeTargetMsg carries this replica as requestor, so it runs in the context the request acquired. With the guard at :3638 on ownsItsSession() alone it is red on the warning, the other two green. sessionRestartRequested is private, and the warning is the one thing that guard decides: with :3638 mutated alone the session is still saved by the guard in restartSession(), so the broker being connected pins nothing on its own.

• The owner read and acquireIEContext() share no lock - confirmed and filed as Replication: the owner read of a session restart and the listener's claim of a remote-initiated import share no lock #1041. Not touched here.

• restartService() under ownsItsSession() - filed as Replication: the session restart a configuration change asks for cuts a total update into this replica #1040. One thing more than "between acceptable and apply": the external changelog entry has no gate at all - ExternalChangelogDomain.isConfigurationChangeAcceptable() returns true unconditionally, and changeConfig(eclIncludes, ...) restarts the session from there.

• An export-direction case - agreed on the split, and left as a follow-up: the export has to be held in flight (a window smaller than the entry count, the ack withheld), which is fixture work of its own. Said under "Not in this change".

• The sessionHasAnOwner() javadoc - the sentence is there: a change left listed holds the ServerState back, in memory and as persisted, until the next restart.

• The silent truncation - one correction to the praise: it was named as deserving an issue, not filed. It is now, as Replication: a total update whose session stops before the DoneMsg ends as a finished import, with the exporter's generationId over partial data #1039, with the three roads which still reach it (the configuration change, the race above, and the server shutdown itself) and the fix shape - the stop recorded as the import's exception, and what arrived compared with what was announced.

ReplayDuringImportTest 3/3 and NamingConflictTest 18/18 on the round-4 head; the change to production code is two comments, so the package numbers in the description stand.