Linux Dedicated Server reaches 100% CPU due to busy polling in Transport (poll/select loop in Unity Update)

[Saber-ccc]

Environment:

OS: Linux (x86_64)
Tested on: Ubuntu 20.04 / 22.04 (kernel 5.x / 6.x)
Unity: 2022.3.34f1c1 (IL2CPP build)
Mirror: 96.8.5
Run 30 Dedicated Servers

Transport:

KcpTransport
ThreadedKcpTransport (reduced but still observable)
Build type: Dedicated Server (headless, -batchmode -nographics)

Load:

Even with 0 or very few clients
CPU already near 100%

Description

On Linux Dedicated Server, Mirror causes near 100% CPU usage even when there are no connected clients or very low traffic.
Profiling shows that CPU time is spent almost entirely in kernel polling syscalls, indicating a busy-poll loop caused by calling non-blocking socket Poll() / select() from Unity’s per-frame Update() loop.
This behavior does not reproduce on Windows, or is far less severe.

Observed Behavior

Server process consumes ~100% of one core
CPU usage remains high even when:
No players connected
No network traffic

CPU usage drops only when:

Artificial delays are introduced
Network polling frequency is manually throttled
A dedicated blocking I/O thread is implemented

Profiling Evidence (perf)

Using perf top / perf record on Linux:

Overhead  Shared Object     Symbol                                                                                                                                                                                            
  14.80%  [kernel]          [k] do_poll.constprop.0
   9.03%  [kernel]          [k] _raw_spin_lock_irqsave
   8.58%  [kernel]          [k] x64_sys_call
   7.20%  [kernel]          [k] copy_user_enhanced_fast_string
   6.72%  [kernel]          [k] sock_poll
   5.29%  [kernel]          [k] tcp_poll
   4.37%  [kernel]          [k] __fdget
   3.18%  [kernel]          [k] __fget_files
   3.09%  [kernel]          [k] do_sys_poll
   3.02%  [kernel]          [k] srso_alias_safe_ret
   2.97%  [kernel]          [k] __lock_text_start
   2.62%  [kernel]          [k] __fget_light
   2.57%  [kernel]          [k] fput
   2.46%  [kernel]          [k] fput_many
   2.13%  [kernel]          [k] do_syscall_64
   1.57%  [kernel]          [k] poll_freewait
   1.36%  [kernel]          [k] __pollwait
   1.25%  [kernel]          [k] remove_wait_queue
   1.18%  [kernel]          [k] entry_SYSCALL_64_after_hwframe
   1.16%  libc.so.6         [.] __libc_calloc
   1.14%  [kernel]          [k] __raw_callee_save___pv_queued_spin_unlock
   0.99%  [kernel]          [k] pipe_poll

I tried to modify the script, but after running for a while, the issue of CPU usage at 100% still persists

ThreadedKcpTransport.cs:

        protected override int GetWaitTimeoutMs()
        {
            if (ServerActive() )
            {
                return server.connections.Count == 0 ? 3000 : 1;
            }
            else
            {
                return 1;
            }
        }

ThreadedTransport.cs:

        protected virtual int GetWaitTimeoutMs()
        {
            return 1;
        }

        bool ThreadTick()
        {
            // was the device put to sleep?
            if (sleepTimer != null &&
                sleepTimer.Elapsed.TotalSeconds >= sleepTimeoutInSeconds)
            {
                Debug.Log("ThreadedTransport: entering sleep mode and stopping/disconnecting.");
                ThreadedServerStop();
                ThreadedClientDisconnect();
                sleepTimer = null;

                // if the device was put to sleep, end the thread gracefully.
                // all threads must end, otherwise putting down the device would
                // slowly drain the battery after a day or more.
                return false;
            }

            // early update the implementation first
            ThreadedClientEarlyUpdate();
            ThreadedServerEarlyUpdate();

            // process queued user requests
            ProcessThreadQueue();

            // late update the implementation at the end
            ThreadedClientLateUpdate();
            ThreadedServerLateUpdate();

            // save some cpu power.
            int waitMs = GetWaitTimeoutMs();
            // Debug.LogError("sleep :" + waitMs);
            Thread.Sleep(waitMs);
            return true;
        }

Extensions.cs

        public static bool ReceiveFromNonBlocking(this Socket socket, int ConnectedCount, byte[] recvBuffer, out ArraySegment<byte> data, ref EndPoint remoteEP)
        {
            data = default;
            int pollTimeout = ConnectedCount == 0 ? 100000 : 0;   // 100ms: 0;   
            try
            {
                int size = socket.ReceiveFrom(recvBuffer, 0, recvBuffer.Length, SocketFlags.None, ref remoteEP);
                data = new ArraySegment<byte>(recvBuffer, 0, size);
                return true;
            }
            catch (SocketException e)
            {
                if (e.SocketErrorCode == SocketError.WouldBlock) return false;
                throw;
            }
        }

using UnityEngine;
using System.Diagnostics;
using Mirror;
using System.Threading;

public class ServerIdleFrameLimiter : Singleton<ServerIdleFrameLimiter>
{
    public override void Init()
    {
        base.Init();
        Application.targetFrameRate = -1;
        QualitySettings.vSyncCount = 0;

        idleFrameTicks = Stopwatch.Frequency / idleFps;
        lastFrameTicks = Stopwatch.GetTimestamp();
    } 

    public int idleFps = 5;

    long lastFrameTicks;
    long idleFrameTicks;

    public void OnUpdate()
    {
        // Headless Server 
        if (!Application.isBatchMode)
            return;

        if (!NetworkServer.active)
            return;

        if (NetworkServer.connections.Count > 0)
            return;

        long now = Stopwatch.GetTimestamp();
        long elapsed = now - lastFrameTicks;

        if (elapsed < idleFrameTicks)
        {
            int sleepMs = (int)((idleFrameTicks - elapsed) * 1000 / Stopwatch.Frequency);
            if (sleepMs > 0)
            {
                //Log.Info($"Sleep ： {sleepMs}");
                Thread.Sleep(sleepMs);
            } 
        }

        lastFrameTicks = Stopwatch.GetTimestamp();
    }
}

[SoftwareGuy]

Hmm, from what you've described it sounds like the 100% CPU usage might be due to the transport getting stuck in a loop when there's nothing to do. When the issue occurs, does the other connected clients seem to freeze or disconnect/timeout? Does the actual game itself freeze up?

If the latter happens, then there is likely a stray loop that's spinning and hogging single/multiple CPU core(s). I'm not sure if that can also be if a peer is choked or went unresponsive and the transport is spinning awaiting a response.

On a side-note: I'm afraid that your markdown formatting is kinda broken - you might want to use a online markdown editor and then paste that into the issue ticket.

[Saber-ccc]

Thank you for your response. I have tested on my end, and the client can indeed connect to these 100% occupied servers without any issues—it doesn't freeze or disconnect, and everything appears to function normally.
The issue with CPU% occurs after running for a period of time. For example, when running 30 servers on my Linux server, 3–4 of them will show this problem after some time, which could be anywhere between 1 and 10 hours.
Haha, I wanted to upload some screenshots, but it seems that option isn't available.

[SoftwareGuy]

Upon second thoughts... I spotted this in your Init() routine:

        Application.targetFrameRate = -1;
        QualitySettings.vSyncCount = 0;

I am not sure what you're trying to achieve here, because this is literally telling Unity to burn through CPU, the same as turning VSync off. The whole "oh but server must run as fast as it can" argument is meh, speaking from personal experience, without a target frame rate, you'll just be burning CPU time because the game will constantly be running it's loops every cycle (instead of say, 60 times a second with some time left over to have a quick smoko break).

Try setting the target to 60 and remove the vSyncCount on Dedicated Server build target since that's redundant (it uses a Null GPU anyway). That might help.

[JesusLuvsYooh]

@Saber-ccc ^ Mirror sets framerate automatically for server builds, it matches the send rate variable on NetworkManager.
Change it as Software guy above has mentioned, or remove your manual set framerate, and retry :)

[Saber-ccc]

I turned off the ServerIdleFrameLimiter code and set the SendRate to 60 as mentioned above, and enabled logs for testing. The logs also show the following content

Server Tick Rate set to 60 Hz.
Mirror.NetworkManager:ConfigureHeadlessFrameRate()
Mirror.NetworkManager:SetupServer()
Mirror.NetworkManager:StartServer()
d__2:MoveNext()
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

but ultimately there is still a CPU 100% usage issue.

Overhead Shared Object Symbol
12.56% [kernel] [k] x64_sys_call
12.24% [kernel] [k] _raw_spin_lock_irqsave
11.67% [kernel] [k] do_poll.constprop.0
9.52% [kernel] [k] copy_user_generic_unrolled
4.27% [kernel] [k] __lock_text_start
3.70% [kernel] [k] sock_poll
2.96% [kernel] [k] do_sys_poll
2.92% [kernel] [k] remove_wait_queue
2.42% [kernel] [k] srso_alias_safe_ret
2.39% [kernel] [k] tcp_poll
2.11% [kernel] [k] __fget_files
1.98% [kernel] [k] fput
1.97% [kernel] [k] __fdget
1.97% libc.so.6 [.] __libc_calloc
1.96% [kernel] [k] __fget_light
1.68% [kernel] [k] fput_many
1.51% [kernel] [k] __raw_callee_save___pv_queued_spin_unlock
1.48% [kernel] [k] entry_SYSCALL_64_after_hwframe
1.24% [kernel] [k] pipe_poll
1.21% [kernel] [k] exit_to_user_mode_prepare
1.20% [kernel] [k] add_wait_queue
1.20% [kernel] [k] __pollwait
1.19% libc.so.6 [.] __poll

[Saber-ccc]

Hey buddy, any progress on this issue? I'm still waiting for your solution. Feel free to @ me if you need any relevant parameters. Thanks. @SoftwareGuy @JesusLuvsYooh @miwarnec

[SoftwareGuy]

What version of Mirror are you using? Have you updated lately?

In addition, are you fluent with strace? You might want to run the server using that, this will flood your SSH session so best to pipe it into a file. Run the server and get it into the loop, let it run for a few minutes inside the 100% CPU loop. Kill it, then inspect the log file and see where exactly it's spinning - if it's Unity Engine or native. I know we have a sys_call entry in that list, but what syscall is it spinning on and what response is it getting from the kernel...?