Skip to content

opensm/osm_multicast.c: Treat send only full members like full members on partial leave to avoid MLID leak - #43

Open
myurasov-nv wants to merge 1 commit into
linux-rdma:masterfrom
myurasov-nv:fix-sendonly-full-mlid-leak
Open

opensm/osm_multicast.c: Treat send only full members like full members on partial leave to avoid MLID leak#43
myurasov-nv wants to merge 1 commit into
linux-rdma:masterfrom
myurasov-nv:fix-sendonly-full-mlid-leak

Conversation

@myurasov-nv

@myurasov-nv myurasov-nv commented Aug 13, 2026

Copy link
Copy Markdown

Description

osm_mgrp_add_port() counts full_members using IB_JOIN_STATE_FULL | IB_JOIN_STATE_SEND_ONLY_FULL (since 0dc7b6d, "Add support for send only full member multicast joins"), but the keep-vs-remove condition in osm_mgrp_remove_port() still tests IB_JOIN_STATE_FULL only. A port that joins with JoinState 0x9 and partially leaves with 0x1 is removed entirely while the full_members decrement is skipped: the group stays empty with full_members stuck at 1, osm_mgrp_cleanup() never runs, and the MLID is pinned until SM restart. The fix uses the same mask in the keep condition, so such a member remains as send-only-full and its final leave takes the normal remove/decrement/cleanup path.

Before After
join(0x9) + leave(0x1): member removed, full_members stuck at 1, MLID leaked until SM restart; ~1018 cycles exhaust the MLID space and every new group creation fails with ERR 1B19 (SA status NO_RESOURCES) despite no active members member kept as send-only-full; the final leave reclaims the MLID; 1100 join/partial-leave/final-leave cycles complete with no failures, plain full join/leave behavior unchanged

Reproduction

  1. Build OpenSM from master and run it against an ibsim simulated fabric (2 switches, 4 HCAs) with the umad2sim shim: LD_PRELOAD=umad2sim.so opensm -f opensm.log.
  2. From a umad SA client, send an MCMemberRecord SubnAdmSet join with JoinState 0x9 for a fresh MGID, then a SubnAdmDelete leave with JoinState 0x1; repeat with a new MGID per cycle.
  3. Unpatched: after ~1018 cycles every further join fails with SA status 0x0100 (NO_RESOURCES) and the log shows mcmr_rcv_create_new_mgrp: ERR 1B19: get_new_mlid failed request mlid 0x0000, while SA MCMemberRecord queries confirm the pinned groups have no members. Only an SM restart recovers. Patched: the same loop runs 1100 cycles clean and each final leave reclaims its MLID.

…s on partial leave to avoid MLID leak

Signed-off-by: Mikhail Yurasov <[email protected]>
@myurasov-nv
myurasov-nv force-pushed the fix-sendonly-full-mlid-leak branch from d5ea3f4 to 125c2f9 Compare August 13, 2026 04:02
@myurasov-nv myurasov-nv changed the title Treat send only full members like full members on partial leave to avoid MLID leak opensm/osm_multicast.c: Treat send only full members like full members on partial leave to avoid MLID leak Aug 13, 2026
@myurasov-nv
myurasov-nv marked this pull request as ready for review August 13, 2026 05:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants