Skip to content

Fix interrupted transaction recovery catchup - #450

Merged
huangminghuang merged 1 commit into
masterfrom
fix/interrupt-trx-recovery
Jun 27, 2026
Merged

huangminghuang merged 1 commit into
masterfrom
fix/interrupt-trx-recovery

Conversation

@huangminghuang

@huangminghuang huangminghuang commented Jun 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes the interrupted transaction recovery path that could leave a restarted producer stuck in LIB catchup after it had already reached the peer's known fork DB root.

Failure

Failure CI run: https://github.com/Wire-Network/wire-sysio/actions/runs/28244239953/job/83678838341

The failing job timed out in interrupt_trx_test.py after the producer was relaunched. During recovery, the restarted node could receive duplicate or already-present blocks at/above the peer's known fork DB root. Because those blocks were not newly applied, the old sync path did not emit the accepted-block transition that normally advances LIB catchup to head catchup. The node stayed in lib_catchup, kept handshaking/waiting, and the test never reached the recovered producer round.

Fix

  • Add a sync_manager helper that completes LIB catchup when the local fork DB root already satisfies the known peer root.
  • Cancel any outstanding sync wait before clearing the sync source so an old timer cannot later close a healthy peer.
  • When this recovery path promotes to head catchup, request from the fork DB root instead of reusing a fork DB head id from the stale/poisoned transition.
  • Assert the post-relaunch producer waits in interrupt_trx_test.py so the recovery failure is reported directly instead of silently hanging until the outer CI timeout.

Branch Base

This branch was rewritten to include only the delta from PR #356 to this fix, then replayed onto current origin/master. The PR now contains one commit, 5a4d34ecf1, and only these files:

  • plugins/net_plugin/src/net_plugin.cpp
  • tests/interrupt_trx_test.py

Review

A subagent review found the missing timer cancellation in the first version of the patch. That was fixed, then re-reviewed with no remaining findings.

Validation

  • git diff --check origin/master...HEAD
  • cmake --build build/codex-interrupt-trx --target net_plugin nodeop clio sys-util kiod sysio_testing_contracts_copy -- -j6
  • timeout 600 python3 tests/interrupt_trx_test.py -v

Note: local validation used -DBUILD_OPP_BUNDLES=OFF because wire-protobuf-bundler is not installed on this machine.

@huangminghuang
huangminghuang requested a review from heifner June 26, 2026 21:06

@heifner heifner left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Targeting the wrong branch or needs to be rebased.

@huangminghuang
huangminghuang force-pushed the fix/interrupt-trx-recovery branch from df8a50d to 5a4d34e Compare June 27, 2026 17:54
@huangminghuang
huangminghuang requested a review from heifner June 27, 2026 17:57
@huangminghuang
huangminghuang merged commit 74b2aaf into master Jun 27, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants