feat: extract reusable member-traffic reconciliation trigger (Scala)

[sadiq1971]

Stacked on #14. The reconciliation logic is SV-shaped today: it asks SynchronizerNodeService for its own sequencer, so there is no way to point it at another synchronizer. This factors the logic out so the Sync Operator Node can run it against a dedicated synchronizer. No behavior change for the SV.

What this does

• New ReconcileSequencerLimitWithMemberTrafficTriggerBase (apps/common/automation), following the existing PackageVettingTrigger pattern: shared logic in an abstract trigger, app-specific dependencies as abstract defs (targetSynchronizerId, sequencerAdminConnection, getTotalPurchasedMemberTraffic, trafficLimitOffset).
• The SV trigger becomes a thin subclass, 175 lines down to 78, binding those hooks to SvDsoStore, SynchronizerNodeService and the synchronizer id already held by SvDsoAutomationService.
• Contracts for other synchronizers are now skipped on a local comparison, before any sequencer RPC. Previously every foreign contract cost a getStatus round trip.

The SV subclass deliberately keeps its name and package. identifyTriggerClassByName returns getCanonicalName and that string keys automationConfig.pausedTriggers, while metrics use getSimpleName as trigger_name. Moving the concrete class would silently change both, including in dashboards and cluster configs outside this repo. Only the reusable logic moves out of singlesv.

One deliberate behavior change

A mismatch between the configured target and the sequencer the connection actually serves is now Status.INTERNAL rather than a skip. After this change those are different conditions: a contract naming another synchronizer is normal and still skips, but the configured target disagreeing with the connection would credit the wrong sequencer, so it should be loud rather than silently granting nothing. This cannot fire from an LSU switchover, which preserves the logical synchronizer id.

How it's verified

apps-sv/compile, scalafmtCheck, headerCheck and scalafix --check pass locally on both apps-common and apps-sv.

No new tests: apps/sv holds 79 triggers and no trigger unit tests. Equivalence rests on the target being the same value the old code compared against, which holds by type (PhysicalSynchronizerId.logical : SynchronizerId) and because SvDsoAutomationService receives the decentralized synchronizer id.

Tracked in

Implements E2-1 (ChainSafe/canton-extending-mainnet#31); unblocks E2-2 (#32).

Note for E2-2: getTotalPurchasedMemberTraffic filters on migration_id, and MemberTraffic ingestion filters on payload.migrationId == domainMigrationId, so an operator node built on this path would inherit #60 and grant nothing on a network past migration 0.

[timwu20]

A mismatch between the configured target and the sequencer the connection actually serves is now Status.INTERNAL rather than a skip.

The code ships the better version of this — FAILED_PRECONDITION with retryable semantics, so a connection mid-switchover recovers on its own (the in-code comment makes that argument well). Worth updating this paragraph to match so the merged description doesn't contradict the implementation.

[timwu20]

This comment says the permanently-wrong case "is reported by warnOnceIfTargetNotServed", but this branch never calls it — the warn-once only fires from the foreign-contract skip path. The two miswiring shapes therefore diverge: a wrong target makes every contract look foreign and the warn fires; a wrong connection (contracts match the target) lands here every task and produces only retry noise, no one-line diagnosis. Since warnOnceIfTargetNotServed is already CAS-guarded and best-effort, invoking it from this branch too is a one-liner that makes the comment true. (Same point promised from #14's tripwire thread.)

[sadiq1971]

Fixed in 946c4a9. You were right that the comment was false. Rather than call warnOnceIfTargetNotServed() here, which would re-fetch the status this branch already has, I split out a shared at-most-once warnOnceTargetNotServed(served) and call it directly before failing.

One thing that fell out while doing it: the single targetCheckSettled flag was conflating "already warned" with "confirmed correctly wired", so a successful early check would have consumed the one warning a later mismatch is entitled to. Now two flags, targetMismatchWarned and targetConfirmed.

[timwu20]

Your fix is better than what I suggested, for two reasons. Reusing the sync id the code already has in hand beats asking the sequencer again. And splitting the flag in two fixes a bug my version would have had: once the wiring check passed, the flag would already be used up, so a later mismatch would never get its warning.

[timwu20]

canton-network#31 asks that the shape "not hard-wire a single sequencer (keep BFT open)". The one-admin-connection-per-reconcile shape is right for MVP, but a BFT operator will need to reconcile the limit on every sequencer of the synchronizer. One sentence here sketching the intended extension (the connection hook returns a set, reconcile loops) keeps that door visibly open rather than implicitly closed.

[sadiq1971]

Added to the class docstring in 946c4a9: one instance reconciles one sequencer, a BFT operator runs one instance per sequencer each with its own sequencerAdminConnection, and nothing here assumes the single-sequencer case beyond that hook returning one connection.

Worth recording why it has to be per-sequencer rather than one instance looping: a traffic grant is submitted with an AggregationRule over the active sequencers and only commits once the group threshold is met (TrafficPurchasedSubmissionHandler), so the grant genuinely needs threshold-many operators each submitting through their own sequencer.

[timwu20]

One trigger per sequencer is simpler than what I had in mind (one trigger looping over all of them) — each instance gets its own connection and nothing is shared. The docstring now covers what canton-network#31 asked for.