Skip to content

Normalize Channel timeout deadlines (fix lock spin in Channel:pop/push) - #1025

Closed
stefanmielke wants to merge 1 commit into
bjornbytes:devfrom
stefanmielke:fix/channel-timeout-nsec
Closed

stefanmielke wants to merge 1 commit into
bjornbytes:devfrom
stefanmielke:fix/channel-timeout-nsec

Conversation

@stefanmielke

Copy link
Copy Markdown

What

lovrChannelPush and lovrChannelPop build the cnd_timedwait deadline as start.tv_nsec + fraction * 1e9 without carrying the overflow into tv_sec. When that sum reaches 1e9 the deadline is invalid: cnd_timedwait returns an error immediately without releasing the mutex, and the wait loop retries while holding channel->lock until the wall clock rolls into the next second. Every other thread using that channel blocks for up to ~1 s. Whole-second timeouts are affected too, because the remaining timeout after a wakeup with no message is fractional.

This change carries the overflow into tv_sec in both functions. start.tv_nsec < 1e9 and fraction * 1e9 < 1e9, so one subtraction is enough.

Testing

Linux x86-64 (glibc 2.39), Release builds of dev and of this branch with identical settings. The repro (3 small Lua files, below) has the main thread push N messages, one every 3 ms, while waiter threads pop from the same channel, and times each push.

Build lovr . fraction (1 waiter, pop(0.75), 1000 pushes) lovr . whole 2 100 (2 waiters, pop(1), 100 pushes)
dev (bf04f88) worst push 747 ms, 13 pushes over 10 ms worst push 997 ms, 38 pushes over 10 ms, 38.9 s total
this branch worst push 0.2–4.8 ms, 0 over 10 ms (4 runs) worst push 0.1 ms, 0 over 10 ms, 0.39 s total

An LD_PRELOAD wrapper around cnd_timedwait counted 32,815,094 invalid deadlines (tv_nsec >= 1e9, each returning an error) in a 300-push fraction run on dev, and 0 on this branch.

lovr test --headless passes (65 passes, 0 errors).

Repro (3 files; run as lovr . fraction or lovr . whole 2 100)

conf.lua

function lovr.conf(t)
  t.window = nil
  t.modules.headset = false
  t.modules.audio = false
  t.modules.graphics = false
end

main.lua

-- Usage: lovr . <fraction|whole|forever|poll> [waiters] [pushes]
-- The main thread pushes N messages (default 1000), one every 3 ms, and times each push.
-- Waiter threads pop from the same channel using the selected wait mode.
local mode = arg[1] or 'fraction'
local waiters = tonumber(arg[2] or '1')
local count = tonumber(arg[3] or '1000')

function lovr.load()
  local channel = lovr.thread.getChannel('repro')
  local threads = {}
  for i = 1, waiters do
    threads[i] = lovr.thread.newThread('consumer.lua')
    threads[i]:start(mode)
  end
  lovr.timer.sleep(0.05)

  local worst, slow = 0, 0
  for i = 1, count do
    local start = lovr.timer.getTime()
    channel:push(i)
    local elapsed = lovr.timer.getTime() - start
    worst = math.max(worst, elapsed)
    if elapsed > 0.010 then slow = slow + 1 end
    lovr.timer.sleep(0.003)
  end

  for i = 1, waiters do channel:push('stop') end
  for i = 1, waiters do threads[i]:wait() end
  print(('mode=%s waiters=%d  worst push=%.1f ms  pushes over 10 ms=%d/%d'):format(mode, waiters, worst * 1000, slow, count))
  lovr.event.quit()
end

consumer.lua

-- Waits on a channel the way the selected mode says.
local mode = ...
local thread = require('lovr.thread')
local timer = require('lovr.timer')
local channel = thread.getChannel('repro')
while true do
  local message
  if mode == 'fraction' then message = channel:pop(0.75)
  elseif mode == 'whole' then message = channel:pop(1)
  elseif mode == 'forever' then message = channel:pop(true)
  else message = channel:pop(false); if message == nil then timer.sleep(0.01) end end
  if message == 'stop' then break end
end

lovrChannelPush/lovrChannelPop built the cnd_timedwait deadline without
carrying tv_nsec overflow into tv_sec. When start.tv_nsec + fraction * 1e9
reached 1e9, cnd_timedwait failed immediately (EINVAL) while still holding
the channel lock, and the loop retried until the clock rolled into the next
second, blocking other threads on that channel for up to ~1 second.
@bjornbytes

Copy link
Copy Markdown
Owner

Was this PR description written with AI? It's hard to follow. Can you explain the bug you were running into more clearly?

@stefanmielke

Copy link
Copy Markdown
Author

yes for both.

what I was trying to do was record things that were happening on the game in the background (to not lock the game on any disk writes) using channels.

so the reader sleeps until there's something to write to the disk (using channel:pop), while the writer writes when it's needed.

when setting the alarm, LOVR adds the wait to the current clock time, split into seconds and nanoseconds, but never carries the overflow from nanoseconds into seconds. The wait then fails instantly, but it keeps the mailbox locked and retries in a tight loop until the clock rolls over to the next second.

so while the writer thread spins with the mailbox locked, the game can’t post anything to it, freezing until the next second (when the lock is released).

the tests are there to show it happening.

@bjornbytes

Copy link
Copy Markdown
Owner

Pushed a fix to stable/dev!

@bjornbytes bjornbytes closed this Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants