Skip to content

Wait for the restarted AudioUnit in the audio device-stall test - #317

Closed
v0lumehi wants to merge 1 commit into
seedds:mainfrom
v0lumehi:audio-stall-flake
Closed

v0lumehi wants to merge 1 commit into
seedds:mainfrom
v0lumehi:audio-stall-flake

Conversation

@v0lumehi

@v0lumehi v0lumehi commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

The "device stall" check in dm/host/tests/t_audio.c fails now and then on the macos-26 runners (job "Host unit tests and CPU-only gfx tests") and has hit unrelated PRs, e.g. #123 and #213. It passes on a rerun and locally. From run 40254505220 (#213):

[dm] audio #26 Start (call site host)                                                            01:37:05.301
[dm] [RED] audio #26: the device has not called back for 212 ms while playing: the silent sink ...  01:37:05.708
[dm] audio #26: AudioUnit restarted                                                              01:37:06.828
[dm] [RED] audio #26: the device has not called back for 204 ms while playing: the silent sink ...  01:37:07.086
[dm] audio #26 Close (call site host)                                                            01:37:08.097
device stall: released 20/20 in order=1 in 2630 ms (expected 1900-5000), AudioUnit restarted and calling back again=0 ✗

Run 40245898469 (#123) shows the same sequence: restart at 52.094, a second stall at 52.625 (247 ms without callbacks), Close at 53.349, calling back again=0.

Cause

The test, not the host. The first stall is the one the test causes (it stops the AudioUnit at 150 ms). After the watchdog restarts the unit, the VM's virtual output calls back for a moment and then pauses again for more than 200 ms. Passing CI runs show the same device running slowly: the suspend/resume scenario takes 1047 ms for 600 ms of audio. The watchdog handles this as designed: the silent sink stands in again, and because callbacks were seen after the restart, the next retry is 1 s later. The test, though, read backend == BK_AU && !stalled once, 100 ms after the last release. The next retry fell after that read (in the first run above, Close came 11 ms before it), so the check failed even though the restarted AudioUnit had called back. The counters are read correctly (cb_n is atomic and the backend is read under g_aos_lock). The host's restart path has no race here.

Fix

Scenario 3 now follows the backend under g_aos_lock on every 1 ms poll. It counts each time the silent sink stands in, takes cb_n when it sees a restarted AudioUnit (BK_AU, not stalled, after a stand-in), and passes once cb_n has grown after that point. If the restarted unit has not called back by the last release, the test keeps polling for up to 20 s, which covers the watchdog's 1, 2 and 4 s back-off. A passing run still ends at the last release, as before (about 2.2 s locally). Release order and the 1900-5000 ms playback window are checked as before. The test still fails if no restarted AudioUnit ever calls back. The result line now also prints the number of stand-ins and the time waited:

device stall: released 20/20 in order=1 in 2241 ms (expected 1900-5000), AudioUnit restarted and calling back again=1 (silent sink stood in 1 times, 2241 ms) ✓

Validation (M-series Mac, macOS 27)

  • The flake does not happen naturally on real hardware: 100 loops of the old test with 3 parallel workers and 8 CPU hogs all passed (0/100), and so did 100 loops of the new test (0/100).
  • CI's sequence, reproduced by injecting a second AudioOutputUnitStop 30 ms after the restart is seen (scratch copy, not committed): old test 30/30 failed with calling back again=0, new test 30/30 passed (stood in 2 times).
  • Negative check (scratch copy where AudioOutputUnitStart fails after the cut): the new test fails after the 20 s deadline with calling back again=0 ... ✗.
  • t_audio built with -fsanitize=thread (CI does not run it under TSan): the old and new tests report the same pre-existing host warnings, which are handoffs from the render thread through AudioOutputUnitStop, where TSan cannot see Core Audio's synchronisation, and none of them involve the test's reads. make -C dm/host/tests GAME=010015100B514000 -j3 run and ... tsan memory-tsan pass. git diff --check is clean.

The device-stall scenario of t_audio read the backend once, 100 ms after
the last release. On GitHub's macOS VMs the virtual output can pause for
more than 200 ms again right after the watchdog restarts the AudioUnit.
The watchdog then correctly stands the silent sink in again and retries
1 s later, which can fall after the test has already closed the output
(CI: "AudioUnit restarted", a second "has not called back for 204 ms",
Close 1 s later, "calling back again=0").

Follow the backend under g_aos_lock on every 1 ms poll: count each time
the silent sink stands in, take the callback count when a restarted
AudioUnit is seen, and pass once it has grown. Keep polling after the
last release until then, for at most 20 s, which covers the watchdog's
1, 2 and 4 s back-off. Release order and the 1.9-5 s playback window are
checked as before, and the test still fails if no restarted AudioUnit
ever calls back.

Validation: 100 loops each of the old and new test under 3 workers and
8 CPU hogs pass locally; with an injected second device pause 30 ms
after the restart, the old test fails 30/30 and the new one passes
30/30; with every restart refused, the new test fails after 20 s. Full
host tests and the tsan and memory-tsan targets pass.
seedds added a commit that referenced this pull request Oct 27, 2026
The device-stall scenario of t_audio read the backend once, 100 ms after
the last release. On GitHub's macOS VMs the virtual output can pause for
more than 200 ms again right after the watchdog restarts the AudioUnit.
The watchdog then correctly stands the silent sink in again and retries
1 s later, which can fall after the test has already closed the output
(CI: "AudioUnit restarted", a second "has not called back for 204 ms",
Close 1 s later, "calling back again=0").

Follow the backend under g_aos_lock on every 1 ms poll: count each time
the silent sink stands in, take the callback count when a restarted
AudioUnit is seen, and pass once it has grown. Keep polling after the
last release until then, for at most 20 s, which covers the watchdog's
1, 2 and 4 s back-off. Release order and the 1.9-5 s playback window are
checked as before, and the test still fails if no restarted AudioUnit
ever calls back.

Validation: with a second device stop injected 30 ms after the restart,
the old test fails 10/10 with CI's message and the new one passes 10/10;
with every restarted AudioUnit stopped, the new test fails after 20 s.
Host unit tests pass.

Co-authored-by: v0lumehi <[email protected]>
@seedds

seedds commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Thanks — confirmed and landed on main as b7654d4 (your change unchanged, with you as co-author).

Confirmation: both failing runs you cited (40254505220, 40245898469) show the AudioUnit restarted, then a second stall, then Close before the watchdog's next retry, while six recent passing runs on main never stall a second time. With a second device stop injected 30 ms after the restart, the old test failed 10/10 with CI's calling back again=0 and the new one passed 10/10; with every restarted AudioUnit stopped, the new test fails after 20 s as it should. Host unit tests pass on current main. The failing runs also show no audio device buffer line around the restart, so the host did not reconfigure the device there.

Closing as merged via b7654d4.

@seedds seedds closed this Oct 27, 2026
Know-It-All-Brother pushed a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
…ds#317)

The device-stall scenario of t_audio read the backend once, 100 ms after
the last release. On GitHub's macOS VMs the virtual output can pause for
more than 200 ms again right after the watchdog restarts the AudioUnit.
The watchdog then correctly stands the silent sink in again and retries
1 s later, which can fall after the test has already closed the output
(CI: "AudioUnit restarted", a second "has not called back for 204 ms",
Close 1 s later, "calling back again=0").

Follow the backend under g_aos_lock on every 1 ms poll: count each time
the silent sink stands in, take the callback count when a restarted
AudioUnit is seen, and pass once it has grown. Keep polling after the
last release until then, for at most 20 s, which covers the watchdog's
1, 2 and 4 s back-off. Release order and the 1.9-5 s playback window are
checked as before, and the test still fails if no restarted AudioUnit
ever calls back.

Validation: with a second device stop injected 30 ms after the restart,
the old test fails 10/10 with CI's message and the new one passes 10/10;
with every restarted AudioUnit stopped, the new test fails after 20 s.
Host unit tests pass.

Co-authored-by: v0lumehi <[email protected]>
ltrdpper pushed a commit to ltrdpper/BananaNative that referenced this pull request Nov 16, 2026
…Edition: profile for program 1, built-in fonts, pool moves without doubling RAM' (seedds#317) from Wick3dPhant0m/BananaNative:kirby-s2e into android

Reviewed-on: https://git.eden-emu.dev/moceure/BananaNative/pulls/317
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants