Skip to content

WIRE-419: recover batch operator startup when outposts are unavailable - #679

Merged
huangminghuang merged 6 commits into
masterfrom
fix/wire-419-batch-operator-startup
Oct 24, 2026
Merged

huangminghuang merged 6 commits into
masterfrom
fix/wire-419-batch-operator-startup

Conversation

@huangminghuang

@huangminghuang huangminghuang commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Use configured Ethereum chain IDs without an endpoint-ID startup query. Discover omitted IDs asynchronously with independent retries; pending clients cannot sign.
  • Refresh the relay/underwriter factory and cranker client snapshots so late outpost startup recovers without a restart. Preserve local signer, policy and registry-ID checks, bounded requests and responsive shutdown; make the diagnostic RPC tool wait for readiness.
  • Add startup/recovery regression coverage using BindConfigProvider ports and a pinned published test dependency.
  • Remove root and OPP pnpm lockfiles, use non-frozen installs, align Node setup with Tools workflows, and pin the five shared AWS SDK clients to 3.1102.0 through root resolutions.
  • Supply NODE_AUTH_TOKEN only to the gated OPP publish step through setup-node’s registry configuration.

Why

WIRE-419: a five-second synchronous Ethereum identity check aborted batch-operator initialization when remote outposts started later. Configured signing IDs are authoritative locally; clients needing discovery remain pending and retry until recovery or shutdown. Callers fetch a fresh immutable snapshot to observe readiness.

Interrupted database/block-log startup diagnostics were split into WIRE-420 / PR #681, now included in the refreshed target. This PR adds no persistence repair or state clearing. WireQL/query timeout configuration is separate.

Validation

  • Clang 18 Release build passed against refreshed master 47ade7dd67; all 52 generated contract artifacts match the tracked versions.
  • Focused startup and integration checks passed: 90 Ethereum cases, 81 batch-operator cases, cranker startup recovery, and shared outpost/chain-plugin coverage. The base refresh also passed the changed Solana suite, selected dispute/producer contract tests, bootstrap ordering, and interrupted-startup/snapshot regressions.
  • Both real-Anvil recovery scenarios passed using the native production client and Ethereum 867bcf0c1e: initialization took 18/19 ms during an eight-second outage, then each case sent three whole-envelope transactions, delivered three attestations with zero failures, finalized the epoch and advanced the next-epoch cursor to 3 without restarting. Configured startup made no chain-ID query; omitted-ID discovery queried four times. This focused check does not replace full multi-node cluster coverage or validate older deployed contracts.
  • Retained dependency/auth checks: non-frozen pnpm installation resolved the shared AWS clients to 3.1102.0; a local dummy registry rejected the old npm auth setup and accepted NODE_AUTH_TOKEN. No package was published.
  • Coordinated full-platform E2E passed all 14 discovered flows, including syndication-plugin-underwriting, on Sysio 4f673a0d7b, Ethereum 0b5184cec0, Solana 73f69330a7, Tools d28e094dfa, and Libraries 4245dad44c. The actual checkout revisions, full build/artifact checks, flow inventory and Tools SDK resolution were verified from the completed run’s reports.

Change-Id: I4aa6e9a8d649fbb84335296ccc69602c40e584c0
@huangminghuang
huangminghuang marked this pull request as ready for review October 14, 2026 21:27
@huangminghuang
huangminghuang requested a review from a team October 14, 2026 21:27
jglanz
jglanz previously requested changes Oct 15, 2026

@jglanz jglanz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Don't use pnpm-lock.yaml in this project, NO frozen lockfile.
  • Use the node setup from the tools project in the release workflows
  • move the block file changes to another PR

When those are done please have @heifner review

@huangminghuang
huangminghuang marked this pull request as draft October 15, 2026 14:33
Change-Id: Ia8247b1db92b66900ac4fd98ff1a26b66e22b101
Change-Id: I4207dc9322f73efba93a9d9d4d84671fab76eee4
@huangminghuang
huangminghuang marked this pull request as ready for review October 16, 2026 20:59

@heifner heifner left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two fixes needed before merge (inline).

with:
package_json_file: libraries/opp/tools/package.json
node-version: 24
registry-url: https://registry.npmjs.org

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This breaks the npm publish. With registry-url, setup-node writes $RUNNER_TEMP/.npmrc with _authToken=${NODE_AUTH_TOKEN}, points NPM_CONFIG_USERCONFIG at it, and sets NODE_AUTH_TOKEN to a placeholder. npm then ignores ~/.npmrc, so the token the publish step writes there is never used. The push run on merge (this PR touches libraries/opp/**) would fail the npm publishes, and tag releases waiting on a successful OPP Bundles run would block. PR CI doesn't run the publish step, so it can't catch this.

Fix: set NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} on the publish step, as wire-tools-ts does, and drop the ~/.npmrc line. Or remove registry-url here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7689737 (retained in the current PR). The gated publish step now receives NODE_AUTH_TOKEN from secrets.NPM_TOKEN and uses setup-node’s registry configuration; the ~/.npmrc write is removed. A local dummy registry test reproduced the previous authentication failure and verified the corrected token setup. No package was published during validation.

Comment thread package.json
"@types/lodash": "^4.17.21",
"@types/node": "^24.10.13",
"@types/shelljs": "^0.10.0",
"@wireio/cluster-tool": "0.1.20",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wireio/cluster-tool brings in @aws-sdk/* (client-ssm, plus client-sts and client-firehose through @wireio/shared), and with no lockfile their transitive ranges float. The platform keeps one pinned @aws-sdk version through a resolutions block in every root package.json; without it an upstream publish race (like the 2026-08-04 ERR_PNPM_NO_MATCHING_VERSION on @aws-sdk/credential-provider-http) fails every CI install. Please add the same block wire-tools-ts and wire-libraries-ts carry:

"resolutions": {
  "@aws-sdk/client-firehose": "3.1102.0",
  "@aws-sdk/client-kms": "3.1102.0",
  "@aws-sdk/client-sns": "3.1102.0",
  "@aws-sdk/client-ssm": "3.1102.0",
  "@aws-sdk/client-sts": "3.1102.0"
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7689737 (retained in the current PR). The root package.json now pins client-firehose, client-kms, client-sns, client-ssm, and client-sts to 3.1102.0 through resolutions. A non-frozen pnpm install verified those resolved versions. No JavaScript lockfile was added.

@huangminghuang
huangminghuang marked this pull request as draft October 20, 2026 14:12
Change-Id: Ice688673a4d0c1ba091e3839984c6805e5d2c629
@huangminghuang
huangminghuang marked this pull request as ready for review October 20, 2026 19:31
@huangminghuang
huangminghuang marked this pull request as draft October 21, 2026 19:30
Change-Id: Id507ede0effb871b5a1d8a158204517921c8860d
@huangminghuang
huangminghuang marked this pull request as ready for review October 21, 2026 23:03
@huangminghuang
huangminghuang marked this pull request as draft October 22, 2026 19:04
Change-Id: I353dff398707e2a891fe94e380c3cae99ed1c9d9
@huangminghuang

Copy link
Copy Markdown
Contributor Author

Refreshed this branch by merging current master 47ade7dd67 in merge commit 4f673a0d7b; GitHub reports the PR mergeable. The complete startup behavior and both inline-review fixes remain intact.

The no-lockfile/non-frozen-install and Tools-style Node setup requests are addressed; the persistence changes were split into WIRE-420 / #681, which is now merged into master. The two inline replies above cover publishing auth and AWS SDK pins.

The refreshed build, 52 contract artifact comparisons, selected startup/base-integration tests, and both real-Anvil startup recovery scenarios passed. Independent architecture and correctness/security/compliance reviews found no actionable PR issues. The previous E2E underwriting failure points to Solana yield-account decoding; the refreshed base includes #691’s fix.

Started a new full-flow E2E run using the refreshed PR branch: https://github.com/Wire-Network/wire-platform-build-system/actions/runs/39839997376 . Results are pending; the PR remains draft during validation. The existing review request to @heifner is retained.

@huangminghuang
huangminghuang marked this pull request as ready for review October 22, 2026 21:10

@heifner heifner left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both fixes verified at 4f673a0. jglanz's three asks are addressed: no pnpm lockfile or frozen installs, Node setup matches wire-tools-ts (the release workflows use no Node; opp-bundles matches the tools publish job), and the persistence changes moved to #681. e2e 39839997376 passed on this head.

@heifner
heifner dismissed jglanz’s stale review October 24, 2026 13:55

Getting this merged

@huangminghuang
huangminghuang merged commit 4214e6a into master Oct 24, 2026
14 checks passed
heifner added a commit that referenced this pull request Oct 24, 2026
Brings in #679 and #592. sysio.msgch.wasm is rebuilt from the merged sources, and every other contract matches its committed wasm and ABI. The sources merged without a hunk both sides edited.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants