PHOENIX-7973 HA client can adopt a stale (lower-version) ClusterRoleRecord when one endpoint lags the peer

[lokiore]

PHOENIX-7973

JIRA: https://issues.apache.org/jira/browse/PHOENIX-7973

Targets the PHOENIX-7562-feature-new (consistent failover) feature branch. Client-side only, no API or wire-format change.

The Phoenix HA client fetches a ClusterRoleRecord (CRR) from a RegionServer endpoint of each of the two paired clusters. CRR admin-version propagation across a cluster's RegionServers is not synchronized, and the client picks an endpoint at effectively random per fetch, so at startup or during an in-flight transition one endpoint can momentarily serve a lower admin version (or an UNKNOWN role) than a peer already advertises. This PR hardens the client against adopting such a staler view, in two complementary places.

1. Reconcile both endpoints on fetch (getClusterRoleRecordFromEndpoint)

Previously the fetch queried cluster 1 first and returned its record immediately whenever it had no UNKNOWN role, without ever consulting cluster 2.

It now always fetches the CRR from both cluster endpoints and reconciles them via a new package-private static helper reconcileClusterRoleRecords(r1, r2):

1. A record without an UNKNOWN role beats one with an UNKNOWN role (an UNKNOWN role means that endpoint could not resolve the cluster roles — e.g. a ZooKeeper problem — and is not usable for routing, so it must never win on version alone).
2. Among records in the same category, the higher admin version wins.
3. On a tie, the peer (cluster 2) record is returned; both are equivalent for routing.

This is a strict superset of the previous UNKNOWN-only handling. If the peer endpoint is unreachable, cluster 1's record is used as-is (exception consumed with a WARN).

2. Guard the refresh path against version rollback (refreshClusterRoleRecord)

The refresh path applied any non-equals() fetched record with no version comparison — so a lagging endpoint serving an older admin version could silently revert the client to a stale cluster-role view even after a newer version had already been applied in the same JVM.

A new package-private static helper shouldApplyRefreshedRecord(current, fetched) now gates the apply: the current record is kept whenever it is strictly newer than the fetched one (equivalently !current.isNewerThan(fetched)).

Crucially, an equal admin version is still applied. The admin version only advances on an operator-driven change; an autonomous state-machine transition changes the cluster roles while keeping the same version. That legitimate same-version role change must still take effect — so only a strictly lower version is rejected, and a strict > guard is deliberately avoided (it would strand failover on autonomous transitions).

This wires in the existing ClusterRoleRecord.isNewerThan() comparison, which the refresh path did not previously use.

Performance

Adds one endpoint RPC on the CRR refresh path only. CRR is fetched on connect/refresh and cached in roleRecord; it is not on the per-query hot path, so there is no meaningful performance impact.

Testing

Two new unit tests in HighAvailabilityGroupTest (mirroring the existing package-private shouldCountFailover unit-test pattern — no mini-cluster needed):

• testReconcileClusterRoleRecords: higher version wins when both usable (regression guard); order-independence; non-UNKNOWN beats UNKNOWN even at lower version (both orders); UNKNOWN vs UNKNOWN → higher version.
• testShouldApplyRefreshedRecord: (a) reject a strictly lower version (the stale-rollback guard); (b) apply a strictly higher version; (c) apply a same-version record with changed roles (the autonomous-transition safety property a strict > guard would break).

Both touched files are spotless:check clean.

Note: this branch has a pre-existing test-compile break in the unrelated org.apache.phoenix.replication package (ReplicationLogGroup*), so the helper logic was verified by compiling and running the assertions directly against a freshly built phoenix-core-client; all pass.