Skip to content

Recover from a lost UDP reply in seconds instead of minutes (#77, #78, #79, #93) - #140

Merged
bertysentry merged 1 commit into
mainfrom
fix/transport-lost-reply-recovery
Oct 8, 2026
Merged

bertysentry merged 1 commit into
mainfrom
fix/transport-lost-reply-recovery

Conversation

@bertysentry

Copy link
Copy Markdown
Contributor

Problem

A single dropped UDP reply from the BMC stalled an IpmiClient call for the 300 s per-message timeout and ended with a bare TimeoutException and nothing collected. On the GIGABYTE test BMC this happens in roughly one run out of three.

Four transport defects conspired:

Issue Defect Fix
#78 MessageListener kept the previous try's Message timed out error, so every retry rethrew it immediately instead of waiting for the resent message The outcome is reset per try; the listener waits on its monitor instead of polling
#93 A timed-out message stayed in the queue for another timeout period, the tag reservation was inverted, a listener exception could kill the queue timer, and the wait for a queue slot was unbounded Timed-out messages leave the queue at once and are reported once; the timer survives listener exceptions; the slot wait is bounded by the message timeout
#79 Connection.waitForResponse counted 1 ms sleeps instead of elapsed time and swallowed InterruptedException; the receiver thread and timers were non-daemon; close() threw NPE when the connector was never created Wall-clock deadline on a monitor; interruption rolls the state machine back and propagates (the connector retry loops no longer retry an interrupted step); daemon threads; null-safe close()
#77 The per-message timeout was 300 s and the client layer never set it Default is now 5 s, and IpmiClient caps it by the overall timeout; IpmiConnector.getTimeout(handle) added

Behaviour changes

  • A lost reply costs the per-message timeout (5 s) plus the retry pause, then the request is resent. A BMC that never answers fails the handshake with ExecutionException wrapping ConnectionException: Command timed out after about 20 s, where it used to throw TimeoutException at the overall timeout.
  • The overall timeout now really cancels the worker, and no library thread keeps the JVM alive: System.exit() is no longer needed at the end of command-line programs.
  • QueueElement.isTimedOut(), makeTimedOut() and refreshTimestamp() are removed (dead with the new queue behaviour). Listed in upgrading.md.

Verification

  • mvn verify on JDK 17: 41 tests, 0 checkstyle / PMD / CPD / SpotBugs findings.
  • New unit tests: MessageListenerTest (stale error, interrupt), MessageQueueTest (removal on first timeout, timer survives a throwing listener), ConnectionTest (wall-clock timeout, interrupt rollback, daemon threads).
  • Live, with a scratch harness outside the repository:
    • GIGABYTE BMC, three sequential sensor walks: two of them lost a Get SDR reply, each recovered after one retry and returned the 62 sensors in 9.3 s and 11.7 s (the clean run took 2.4 s). Before this change these runs timed out at 120 s.
    • Same BMC, three concurrent processes × two walks: all six completed in 2.3 s each.
    • Lenovo IMM, three walks: 49 sensors each.
    • Unreachable host (192.0.2.1), overall timeout 60 s: Command timed out after 20.4 s, JVM exited on its own. Overall timeout 5 s: TimeoutException at 5.05 s, JVM exited at 5.25 s.

Not in this PR (next one, same code area): #109 (state machine stuck after an authentication failure, credentials sent 4 times) and #132 (cleanup after the overall timeout runs alongside the worker, Close Session unconfirmed).

Fixes #77, fixes #78, fixes #79, fixes #93.

🤖 Generated with Claude Code

A dropped BMC reply used to cost the whole 300 s per-message timeout and
an empty result. Four defects in the transport path made every retry
pointless and every wait unbounded:

- MessageListener kept the "Message timed out" error of the previous try,
  so a retry rethrew it at once instead of waiting for the resent message
  (#78). The listener now resets its outcome per try and waits on a
  monitor instead of polling.
- MessageQueue kept a timed-out message for another timeout period and
  its tag reservation was inverted (#93). A timed-out message now leaves
  the queue at once and is reported once; a listener that throws no
  longer kills the timer thread, and the wait for a free queue slot is
  bounded by the message timeout.
- Connection.waitForResponse counted sleeps instead of elapsed time and
  swallowed interruption (#79). It now waits on a wall-clock deadline,
  and an interrupt rolls the state machine back and propagates, so
  Future.cancel(true) from IpmiClient really stops the worker. The retry
  loops of the connectors no longer retry an interrupted step. The UDP
  receiver and the timers are daemon threads, and AbstractIpmiRunner
  .close() tolerates a session that never got a connector.
- The default per-message timeout is 5 s instead of 300 s, and
  IpmiClient caps it by the overall timeout of the call (#77).
  IpmiConnector exposes getTimeout(handle).

Verified on a GIGABYTE and a Lenovo BMC: a dropped Get SDR reply is now
retried after 5 s and the sensor walk completes in about 9 s instead of
timing out at 120 s; an unreachable host fails with "Command timed out"
after 20 s, and the JVM exits without System.exit().

Fixes #77, fixes #78, fixes #79, fixes #93.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-08T18:36:08.499560Z cc485ac PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cc485ac4c8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +428 to 429
} catch (IllegalArgumentException | InterruptedException e) {
throw e;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Unregister the listener when interruption propagates

When a thread using the synchronous low-level IpmiConnector.sendMessage is interrupted inside waitForAnswer, this new branch immediately rethrows, but sendMessage() unregisters its MessageListener only after a normal return. The listener therefore remains in IpmiAsyncConnector; repeated cancellations leak listeners, and every later response must traverse these stale registrations. Move listener removal into a finally block.

Useful? React with 👍 / 👎.

pingPeriod=30000
#Time in ms after which a message times out.
timeout=300000
timeout=5000

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Document the new timeout behavior in README

This changes the library-wide per-message timeout from five minutes to five seconds and accompanies new retry and cancellation semantics, but the README's 1.2.03 upgrade summary remains unchanged and omits this major user-visible behavior change. Update that summary so users relying on the previous default are warned.

AGENTS.md reference: AGENTS.md:L33-L33

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment