[vanceeasleaf]

Summary

A failed control teardown can leave an agent record online after its control session is gone. The client then keeps retrying, but every login is rejected with agent_id [...] for user [...] is already online, and dashboard kick reports client not online because it only looks at live controls. Restarting the server was the only recovery. This PR makes login conflict handling reclaim such stale sessions and hardens teardown so the leak is much less likely.

Root cause

1. Control::shutdown() held the tunnel-manager mutex while awaiting every tunnel close, with no timeout. Teardown could stall before agents.release() ran.
2. register_control() rejected every same-agent conflict unconditionally, so a restarted client could never take over its own leaked record.
3. kick_client() only searched controls. When the control was absent it returned client not online and never cleared the leaked agent record.
4. Global tunnel name/port slots stayed claimed by the dead generation, so even after a successful login, tunnel registration could fail.

Changes

• On login conflict, take over only when the old control is missing, closed, heartbeat-stale, or has zero tunnels. A healthy session with active tunnels is still rejected, preserving protection against two clients sharing one agentId.
• Takeover and dashboard kick now force-release the stale agent record and free tunnel name/port slots owned by that generation.
• Control::shutdown() drains tunnels under a bounded lock, closes them outside the lock with a timeout, and bounds writer/background-task shutdown so it always reaches mark_finished().
• Added heartbeat staleness detection based on the last received Ping.
• kick_client() now cleans a leaked agent record when no control exists, and bounds all waits.
• The dashboard snapshot marks closed controls offline and shows leaked online records as offline instead of hiding them.

Tests

The new unit tests reproduce the leak (stale_generation_release_leaves_agent_online) and verify recovery:

• force_release_clears_leaked_record_and_allows_new_session
• force_release_does_not_clobber_newer_generation
• force_release_removes_anonymous_session
• conflict resolution tests: missing / closed / stale / zero-tunnel / healthy / unknown tunnel count
• heartbeat staleness tests
• tunnel registry and port slot cleanup tests

Results:

• cargo test -p orbien-server --locked: 14 passed, 0 failed
• cargo fmt --all -- --check: clean
• cargo clippy -p orbien-server --locked --all-targets: no new warnings
• Manual smoke test: release server starts, dashboard root returns 200 with Basic auth and 401 without, /api/v1/clients returns an empty list

Compatibility and scope

Server-side only. No client or protocol changes; wire compatibility is unchanged.

This fixes the stale-session lockout and takeover path. A client whose tunnels are lost while its control is still healthy with zero tunnels will now be allowed to reconnect, but client-side tunnel re-registration retry remains a separate follow-up.

Related issues

This targets the server-side stale-session and stale-slot leak behind several reports: #28 (restart cannot reconnect), #27 (restart shortly after disconnect: tunnel already registered), #19 (long-running client disconnects), and #29 (dashboard stats inconsistent). It does not add client-side automatic reconnect or tunnel retry.

🤖 Generated with Claude Code