Repository navigation
feat: Real-time PCM streaming, custom DSP filters/sources, and desktop VST3 plugin hosting - #593
Merged
Merged
Conversation
…d custom sound sources
…s and GUI example from flutter_vst3
…user data handling in VST3 callbacks
…X11/GDK editor handling
…hit-testing issues
…eam buffer views and refilling
…T3 plugin example
The Linux editor shares its state with the helper process through a file that both map. The file was /tmp/soloud_vst3_shm_<pid>_<n>.bin, opened with O_CREAT and O_TRUNC but not O_EXCL or O_NOFOLLOW, so any local user could guess the name and create it first. Where the kernel's fs.protected_symlinks and fs.protected_regular are off, a symlink there to one of the user's files got that file truncated and overwritten with the shared memory header, and a file of their own with mode 0666 let them read and change what the host and the helper exchange. Where those settings are on, such a file only made the open fail, and the editor did not open. The file is now created by mkstemp(): a random name, O_EXCL and mode 0600. It goes in $XDG_RUNTIME_DIR, which only the user can enter, and falls back to $TMPDIR and then /tmp when that is unset or unusable. Not tested here: the host tests run on macOS only, and this code is Linux-only. It passes a syntax check against the macOS headers.
disposeSound() stopped each voice while iterating the sound's own handle vector. stop() can dispatch the voice-ended callback synchronously, and that callback reaches removeHandle(). The handle had just been dropped from mSoundByHandle, so removeHandle() fell back to its linear scan and erased the handle from the vector being iterated, invalidating the loop (AddressSanitizer reports a container-overflow). Iterate a copy instead. The engine lifecycle test now plays one sound on several voices and disposes it, checking every voice ends once and the sound is gone.
After a seek out of its buffered range, a pull stream resets an Opus or Vorbis decoder (prepareForSeek()) and feeds it from the seek's file offset, but the reset restarted the Ogg seek index's byte count at 0 (ogg_seek_index.h), so the pages indexed from then on were recorded at offsets relative to the seek. A later seek read them as file offsets. covers() also checked only the last entry, so a target before the first entry interpolated a negative offset into a uint64_t. With a buffer of about one second, seeking to 5 s and back to 2 s requested nothing (UBSan: -46767.9 converted to unsigned long long) and the stream stayed at [2.00, 2.00]. Seeking to 5 s, playing on and seeking to 6.6 s requested byte 7826 instead of 68593 for Opus (12180 instead of 70341 for Vorbis), and played audio from near the start. prepareForSeek() now also takes the file offset the data resumes from, and the Ogg decoders restart their index there. covers() requires the target to lie between the first and last indexed pages; a target outside them falls back to the stream's estimate from the probed duration. FLAC and WAV take the offset and ignore it: they do not index offsets as they decode. Test: test/pull_ogg_seek_test.cpp (run_pull_ogg_seek_test.sh, plain and --asan) drives Opus and Vorbis pull streams through the exports on the null backend, with a feeder answering the requests as Dart does. Before the fix 6 of 16 assertions fail, with the UBSan report; after it all pass: the seek back to 2 s requests the estimate (24307 for Opus) and plays on, and the seek to 6.6 s requests 68593 (Opus) and 70341 (Vorbis).
A pull stream replaced its whole stream decoder on every MP3 seek out of the buffered range (resetDecoderForSeek()). The new MP3 decoder built its seek table from the data that starts at the seek's offset, counting time and bytes from there, and the next seek read that table as a map of the whole file. At a constant bit rate the two nearly agree; with a variable bit rate they do not. After a seek to 6 s in a VBR file, a seek back to 0.5 s requested byte 14823 instead of 2020. MP3 now resets for a seek like Ogg, FLAC and WAV do: prepareForSeek() restarts the codec and the input at the new offset but keeps the seek table built from the start of the file, with that table's sample rate. A table is never built from data that starts at a seek, and cleanup() discards it. Test: test/pull_mp3_seek_test.cpp (run_pull_mp3_seek_test.sh, plain and --asan; the runner encodes a VBR fixture with ffmpeg). Through the exports on the null backend, it seeks a playing stream to 6 s and back to 0.5 s, twice: each seek back must request an offset within four frames of the whole file's seek map, and play on from there. A direct check covers the table surviving prepareForSeek() and cleanup() discarding it. Before the fix both seeks back request 14823 (2 of 17 assertions fail); after it they request 2020 and all pass.
A pull stream builds the MP3 seek table from the first data it gets, so the table can cover only the start of the file. A seek past the table waited for more data, but once all the data had been delivered (mDataEnded) timeToByteOffset() mapped it to the table's last entry, and canSeekToTime() reported any target as covered. On a 12 s constant bit rate file without a Xing header, played to the end of its data, a seek to 6 s requested byte 31346 instead of about 94876, and playback restarted near 2 s. canSeekToTime() now reports a target as covered only between the table's first and last entries, with or without the end of data, and timeToByteOffset() maps only covered targets. The pull stream estimates the others from the probed duration, as it already did before the end of data. Test: test/pull_mp3_seek_test.cpp (run_pull_mp3_seek_test.sh, plain and --asan; the runner now also encodes the CBR fixture) plays the file to the end of its data, seeks to 6 s and compares the requested offset with the whole file's seek map. Before the fix it requests 31346 (1 of 24 assertions fails); after it, 96000, within four frames.
scanBoxes() (m4a_metadata.cpp) recursed into every nested container box, and each level of nesting costs only 8 bytes of input, so 4 MiB of nested 'moov' boxes overflowed the stack, natively and in WebAssembly. It runs on the first chunk of an M4A stream (StreamDecoder::decode(), so buffer and pull streams), on whatever thread delivers it. The scan stops descending at 32 container levels, far deeper than real metadata nests. Test: test/metadata_parsers_test.cpp (new, run_metadata_parsers_test.sh with --asan and --wasm32): `valid` checks that well-formed M4A, ID3 and OpusTags metadata still parse, and `m4a-nested` parses the nested boxes on a thread with a 1 MiB stack, in a child process. Before the fix the child crashes (SIGBUS plain; ASan: stack-overflow in scanBoxes), and the wasm32 build in node fails with "Maximum call stack size exceeded". After it all 10 assertions pass plain and under ASan, and the 7 of the wasm32 build (no Xiph there) pass in node.
The M4A box, ID3 frame and OpusTags parsers bounded a length read from the stream with `offset + length > size`. With a 32-bit size_t (the web build, armeabi-v7a) a large length wraps the sum, the check passes, and the parser reads past the data or builds a string of nearly 4 GiB. A 'moov' box declaring 2^32 - 16 bytes after a 16-byte 'ftyp' wraps to 0; an ID3v2.3 TIT2 frame declaring 0xFFFFFFF0 bytes wraps to 16. A shared check (metadata_bounds.h) compares the length with what is left after the offset, in 64 bits. The M4A, MP3 and AAC ID3, AAC/AC-3 frame header, and OpusTags/OpusHead parsers use it wherever they bound a length. Test: test/metadata_parsers_test.cpp `m4a-wrap`, `mp3-id3-wrap`, `aac-id3-wrap` and `opus-wrap`, plus `m4a-large-wrap` (an extended box size of UINT64_MAX, coverage for the 64-bit size path: it passes before the fix too). On a 64-bit host the sums do not wrap, so the wasm32 build in node shows the bug: before the fix, each scenario run on its own, `m4a-wrap` traps (memory access out of bounds) and `mp3-id3-wrap` and `aac-id3-wrap` abort. After it the wasm32 suite passes (11 assertions), and the host suite passes plain and under ASan (14). Not tested here on a 32-bit target: `opus-wrap`, since the wasm32 build here has no Xiph libraries. Its sum wraps the same way (a size_t offset plus a uint32_t length, against the packet's long length), and it passes on the host.
The FLAC stream decoder caches the header ("fLaC" and every metadata
block) so that a seek can restart the codec from it. A delivery can end
inside the metadata, as the first one does when the file carries a
large picture or padding block. Three related failures followed, fixed
together here because each fix alone leaves one of the others:
- findFlacHeaderLength() (flac_stream_decoder.cpp:26) stepped over each
block by its declared length without checking that the block had
arrived. It returned a header end past the received bytes, and
decode() (:170) copied that range: a 62-byte first delivery of a file
whose metadata ends at 65582 read 65520 bytes past the input vector.
- decode() waited for the rest of the header only while the codec was
not initialized, but StreamDecoder initializes it before the first
delivery. The partial header went to the codec, the input never
began with "fLaC" again, the header was never cached, and every
later seek produced silence.
- Waiting for the header must end when the data does. A fresh decoder
whose data stopped inside the metadata (a truncated file) stayed
pending forever, and with the wait above an initialized one would
too; the native buffer stream's decode loop polls while
hasPendingData() is true.
The scanner now reports the header complete only once every block has
fully arrived, using metadataHasBytes() so the sum cannot wrap.
decode() waits for the whole header whatever the codec state; the codec
cannot produce samples before it anyway, so no audio is delayed. Once
the data has ended with the header incomplete, decode() drops the
input, marks the decoder drained and reports a codec error.
Test: test/metadata_parsers_test.cpp flac-split-header,
flac-seek-split-header and flac-truncated-header. The fixture's final
metadata block is 64 KiB of padding. It is delivered as its first 44
bytes (inside that block's header) or 62 bytes (inside its payload) and
then the rest, to a fresh decoder and to one initialized first as
StreamDecoder does. It must decode the same 512 samples as when
delivered whole, again after prepareForSeek(0, 65582) and the frame
bytes from 65582, and, when the data ends after the first part, stop
reporting pending data within 8 decodes.
On the parent the seek after a 44-byte split decodes 0 samples instead
of 512 and the fresh decoder's truncated data stays pending (2
failures); under ASan each scenario also reports a heap-buffer-overflow
(a 65582-byte read at flac_stream_decoder.cpp:170). With the scanner
fix alone the 62-byte split fails the same two ways. With this commit
the suite passes plain and under ASan (78 assertions), and the wasm32
build, which leaves the FLAC scenarios out, still passes (11).
Metadata strings leave the native side as UTF-8; M4A, Vorbis and Opus tags are UTF-8 already. The ID3v2 text-frame parser (parseId3TextFrame(), mp3_stream_decoder.cpp:50, and its copy in aac_stream_decoder.cpp:420) did not convert to it: - Latin-1 text (encoding 0) passed through unchanged, so a title "Caf\xe9" came out as those bytes, which are not valid UTF-8. - Each UTF-16 code unit was encoded on its own, so a character outside the Basic Multilingual Plane (U+1F600, or any other emoji) became two three-byte surrogate encodings, "\xed\xa0\xbd\xed\xb8\x80", which are not valid UTF-8 either. Unpaired surrogates were encoded the same way. - A byte-order mark under encoding 2 came out as U+FEFF. Both parsers now use one converter, id3TextToUtf8() in the new audiobuffer/id3_text.h. It maps Latin-1 bytes to UTF-8, combines a surrogate pair into one four-byte character, replaces an unpaired surrogate with U+FFFD, honours a byte-order mark under encodings 1 and 2, and passes UTF-8 (encoding 3) through. Trailing NULs are dropped as before. Test: test/metadata_parsers_test.cpp id3-latin1 and id3-utf16, through the AAC parser and through the title the MP3 decoder reports: "Cafe" with an acute e in Latin-1 and in UTF-8; "A", U+1F600, "B" in UTF-16LE and UTF-16BE, with and without a BOM, under encodings 1 and 2; lone high and low surrogates; and BMP text as a control. On the parent id3-latin1 fails 2 checks (both titles stay "Caf\xe9") and id3-utf16 fails 16 (all but the BMP control). With the fix the suite passes plain and under ASan (133 assertions) and in the wasm32 build (66). This fixes the native side only. The Dart readers of these strings (_arrayToString() in native_metadata_ffi.dart, _readString() in native_metadata_web.dart) take one byte per character and do not decode UTF-8 yet.
The Apple AAC stream decoder queues 3 packets before it starts decoding (aac_stream_decoder_apple.mm:92) and did not consider the end of the data. A whole stream of one or two packets (up to 2048 frames, 46 ms at 44.1 kHz, as in a short sound effect) never reached the threshold: every decode returned nothing and hasPendingData() stayed true. The stream produced no audio, and a buffer stream's background decode thread (audiobuffer.cpp:472) polled every 2 ms until the sound was disposed. Once the data has ended, the decoder now decodes and flushes the queued packets whatever their number. Test: test/native_decoder_test.cpp testShortAacStreamDrains (Apple only: the fixture is made with afconvert). The first two ADTS packets of a 0.25 s stereo tone are decoded, the end of the data is signalled, and the decoder is drained at most 20 times. It must stop reporting pending data and produce samples. test/run_native_decoder_test.sh gains an --asan option. Before the fix the scenario fails, plain and under ASan: still pending after 20 drains, no samples. With the fix the native decoder suite passes plain and under ASan, the scenario draining 4096 samples.
When data decoded for a pull stream completed a pending seek, applyPendingSeek() ran checkBuffering() (pull_buffer_stream.cpp:1006), which calls Player::getPause() (:1147) and setPause(). Both take SoLoud's audio mutex, which is not recursive, and two callers already held what that waits for: - The mixer. getAudio() decodes data a feeder has queued but not yet decoded (tryDecodePendingData(), :135). When that data completed a seek out of the buffered range, the audio thread called getPause() with the audio mutex held and blocked on itself: the mixer stopped, and every later engine call that takes the audio mutex hung. A feeder preempted between queuing its bytes and decoding them was enough. - A delivery decoding under mDecodeMutex. SoLoud seeks with the audio mutex held, and PullBufferStreamInstance::seek() then takes mDecodeMutex (:197). A delivery whose decode completed a seek waited in getPause() for a seeking thread that waited for the delivery's mDecodeMutex. applyPendingSeek() now only records that the seek completed. Every delivery runs the buffering check anyway once it has decoded and released the mutex (addAudioData(), setDataIsEnded()), and an empty delivery now runs it too when its decode completed a seek. Nothing is lost when the mixer completes the seek: it only does that for a voice it is mixing, and the delivery that brought the data runs the check. Test: test/pull_stream_race_test.cpp, run by test/run_pull_stream_race_test.sh (plain, --asan, --tsan), with new barriers pullDataQueued, pullSeekCompleted and pullBufferingCheckEntered, on a playing 16-bit stereo PCM stream: - mixer-seek: after a seek to 300 s, a feeder queues the data there and parks before decoding it, and the mixer completes the seek. The audio mutex must be free again within 3 s (getPause() returns), and the stream then holds the data from 300 s. - deliver-seek: a delivery that completed the seek to 300 s parks in its buffering check while another thread seeks to 600 s. Once it is released, both must return within 3 s. With only the barriers added, both scenarios deadlock (the run reports it and exits with failure). With the fix both pass plain, under ASan and under TSan. TSan still reports one race in deliver-seek, the seek discarding the decoded backlog without mDecodeMutex while the delivery reads its size: seeks, rewinds and resets change the decode state without the mutex, a separate bug. The pull Ogg and MP3 seek tests still pass.
A pull stream's seek, rewind and reset replaced the decoder, cleared the buffers and moved the read offsets without the stream's decode mutex, while a delivery (addAudioData on the feeding thread) or the mixer decoded under it. getBufferTimeRange, getLength and getBufferedLength read the same state unlocked. Seeking a voice to 0 rewinds it, and rewind() replaced the decoder of a compressed stream while a delivery on another thread was still decoding a chunk with it. With an MP3 fed in 32 KB chunks while another thread seeks every 7 ms (every fifth seek to 0), resets every 60 ms and reads the buffered range every 1 ms, ASan reports a heap-use-after-free in the decoder within 1.5 s. Under TSan the same run reports races in rewind(), resetPullBufferStream() and requestMoreDataIfNeeded(), then dies with SIGBUS in a memmove. Parked right before it discards the buffer, a seek or a rewind let a delivery decode a whole chunk meanwhile, which the seek then threw away or which landed at the wrong position. A reset completed while the mixer was in the middle of a decode, which then wrote the old stream's samples into the reset buffer. seek(), rewind() and resetPullBufferStream() now hold the decode mutex for their whole update, and the getters take it too. getLength() and the buffered-range getters get *Locked variants for internal callers, and timeToEncodedByteOffset() becomes the private timeToEncodedByteOffsetLocked(). SoLoud applies seeks and rewinds with its audio mutex held, often on the mixer (a looping voice rewinds at its end and on a short read), and a delivery holds the decode mutex for its whole decode, 60 to 120 ms for a 3.4 MB MP3. So neither the mixer nor anything else holding the audio mutex waits for it: - a seek or a rewind only tries the mutex; one that finds it held is deferred, the voice reports the new position at once, and the mixer's first block that gets the mutex applies it, playing silence until then (on the web's AudioWorklet thread a contended lock aborts); - a rewind of a stream that has played nothing yet keeps its buffer, so a looping voice waiting for its first data, which the mixer rewinds every block, does not discard the delivery it waited for; - metadata and duration found while decoding are collected and reported once the mutex is released; - the request for more data is decided under the mutex and made after it, and the mixer only try-locks for it; - a delivery that completes a pending seek hands the new play position to the mixer's next getAudio() instead of writing the voice's position while the mixer advances it. Verified with pull_stream_race_test, which gains seek, rewind, reset, seek-during-decode, loop-during-decode and stress scenarios. At the parent, with the two new test barriers added: seek 6 assertions, 1 failure; rewind 6, 1; reset 7, 1; stress heap-use-after-free under ASan, races then SIGBUS under TSan. Waiting on the decode mutex instead of deferring, seek-during-decode fails 4 of 11 (the seeks and getPosition() blocked while a delivery is parked mid-decode) and loop-during-decode 3 of 6 (the mixer blocked, then the rewind discarded the delivered data). With the fix: 62 assertions, 0 failures plain, with ASan+UBSan and with TSan. pull_ogg_seek_test and pull_mp3_seek_test still pass.
Player::addPullBufferDataStream() looked the stream up under sounds_mutex, released the lock and then called addAudioData() on the raw pointer. A disposeSound() or disposeAllSound() on another thread could destroy the PullBufferStream in between, and the feeder then decoded into freed memory. getPullBufferTimeRange() and resetPullBufferStream() had the same gap. With a feeder parked right after its lookup, disposeSound() and disposeAllSound() both returned within the 300 ms the test waits. Released, the feeder read the destroyed sound: ASan reports a heap-use-after-free in Player::addPullBufferDataStream(), freed by disposeSound(). The three functions now pin the stream under sounds_mutex and use it after releasing the mutex, and a disposal, having removed the sound under the mutex, waits without it until the calls pinned before have unpinned it. Holding sounds_mutex for the whole call instead would make every other user of it, a voice's end on the mixer's dispatch too, wait for a delivery's decode: 107 to 140 ms for the first megabyte of a 3.4 MB MP3. The buffering check reads the sound's handle list and pauses the voice under sounds_mutex, taken before the audio mutex as everywhere, and does nothing once the stream is disposed. This also ends the TSan report in pull_ogg_seek_test of the check reading the sound's handle list while a voice-ended callback erased from it. A disposed stream no longer asks Dart for data. On the web, where every caller runs on the browser's main thread, a disposal does not wait. Verified with pull_stream_race_test, new dispose and dispose-all scenarios (new pullDataSoundLookedUp barrier): the disposal must not finish while the delivery is parked, the delivery then succeeds, the voice is stopped and a later delivery finds no stream. At the parent both fail (6 assertions, 1 failure each), and a variant that releases the parked delivery anyway gives the use-after-free above. The new lock-during-decode scenario parks a delivery mid-decode and needs sounds_mutex free meanwhile, also while a disposal waits for the delivery; holding the mutex for the whole call fails both (6 assertions, 2 failures). With the fix: 84 assertions, 0 failures plain, with ASan+UBSan and with TSan.
rewind() discards the buffered audio and asks for the stream again from offset 0, but left mDataIsEnded set once the whole file had been delivered. hasEnded() then saw an ended, empty stream: seek(0) on a stream that had finished loading ended its voice, a looping voice went silent at its first wrap, and every later delivery was refused with streamEndedAlready. With a 0.5 s PCM file delivered whole, seek(0) after 150 ms of playback made no request for offset 0, the voice ended, and delivering the file again returned error 22 for both halves. Played looping, the stream asked for nothing at its wrap and refused the refill the same way. rewind() now reopens the feed, as an out-of-buffer seek already does. That covers the mixer's rewind of a looping voice at its end too. Verified with the new replay scenario in pull_stream_race_test: after seek(0) the stream asks for offset 0, its voice waits, the file is accepted again and plays through to its end; looping, the stream asks for offset 0 at the wrap, accepts the refill and wraps again. At the parent it fails 6 of its 16 assertions. With the fix the whole test passes, 100 assertions, 0 failures, plain, with ASan+UBSan and with TSan.
resetPullBufferStream() clears the buffers and zeroes the decoded-sample counters under the decode mutex, but the mixer takes samples from the circular buffer and adds them to mDecodedSamplesRead without that mutex (it must not wait for a delivery that is decoding). A reset landing between the two left the empty, reset stream counting the old read: getPullBufferTimeRange() reported a buffered range with no data, and the window and request calculations started from the wrong count. With the mixer parked between its read and its count, a reset finished at once, and the reset stream then reported [0, 0.011610] (one 512-frame block at 44.1 kHz) instead of [0, 0]. The reset now also holds SoLoud's audio mutex, which the mixer holds for the whole read. It takes it after the decode mutex: a reset waits for a delivery's decode, 107 to 120 ms for a 3.4 MB MP3, and holding the audio mutex meanwhile would stop the mixer and every getPosition() for as long. Nothing holding the audio mutex waits for the decode mutex (the mixer, seeks and rewinds only try it), so this adds no cycle. Verified with two new scenarios in pull_stream_race_test. read-reset (new pullSamplesRead barrier): the reset must wait for the parked mixer and leave [0, 0]; at the parent, with the barrier added, both checks fail (8 assertions, 2 failures). reset-during-decode: with a delivery parked mid-decode, the reset must wait for it while the audio mutex stays free; taking the audio mutex first fails that (9 assertions, 1 failure). With the fix the whole test passes, 117 assertions, 0 failures, plain, with ASan+UBSan and with TSan.
getBufferEndTimeLocked() summed the samples read and the samples available in the circular buffer. The mixer takes samples from the buffer before it adds them to the read count, and does so without the decode mutex, so a range read in between came out one block short. With 0.743 s delivered and the mixer parked between its read and its count, getPullBufferTimeRange() reported [0, 0.731] instead of [0, 0.743], 512 frames short. The end is now the samples written minus the decode backlog not yet in the buffer, the same value when the mixer is between blocks, and both change only under the decode mutex that this function holds. Verified with the new range scenario in pull_stream_race_test: the range must be [0, 0.743] between blocks and stay so while the mixer is parked at the pullSamplesRead barrier. At the parent the second check fails (7 assertions, 1 failure). With the fix the whole test passes, 124 assertions, 0 failures, plain, with ASan+UBSan and with TSan.
A BufferStream is used by a delivering thread (under sounds_mutex), by its background decode thread, which decodes what a compressed stream still holds once its data has ended, by play(), which runs its buffering check, and by the mixer. Several things they share had no common lock: - the buffering check reads and updates the sound's handle list, which a play appends to and a voice's end removes from under sounds_mutex, but the decode thread and play() ran it under check_buffer_mutex only; - the decode thread's check pauses and unpauses voices through the Player, which takes sounds_mutex, while a reset or a disposal holds that mutex and waits for the thread to stop; - the mixer, pausing a voice that ran out of data, read the same list to find the voice's handle for the buffering callback, and wrote its user-paused flag; - the sample and byte counters and the end flag were plain fields that a delivery and the decode thread update, the mixer updates for a RELEASED stream, and the length getters and hasEnded() read; - a compressed stream's first decode set the format the mixer reads under the buffer mutex without taking it. With the decode thread parked before a check that will unpause a voice and a reset waiting for it, the released thread blocked on sounds_mutex in Player::setPause() and the reset never returned. Under TSan, feeding streams in 30 ms bursts while other threads play, stop and read them reports 6 races: the mixer's handle list writes against a delivery's check, the RELEASED counters against a delivery and a reader, the received-bytes count against a reader, and the format against the mixer's read. checkBuffering() now runs holding sounds_mutex, taken before check_buffer_mutex as a delivery holds them, through a new private Player::runWithSoundsLock(). The decode thread only tries the mutex, and gives up once it is being stopped; a disposal marks the stream destroyed under the mutex, and the thread checks that again once it holds it. The mixer finds the voice it pauses in SoLoud's voice table, under the audio mutex it already holds, and leaves the handle list alone. The user-paused flag it cleared there can still be set on a playing voice: setPause() on a voice group finds no voice of a sound for the group's handle, so a voice paused by the user and unpaused through its group keeps it, and the check would never resume that voice once it runs dry. So the mixer marks the voice instead, and the next check clears the flag when it takes the mark; a pause or unpause by the user discards the mark, so a user pause made while the voice buffers still holds. The counters and the end flag are atomic, and the format is set under the buffer mutex. Verified with the new test/buffer_stream_race_test.cpp: decode-reset (new bufferStreamBackgroundDecoded barrier) deadlocks at the parent (9 assertions, 1 failure) and passes with the fix; stress reports the 6 races at the parent under TSan and none with the fix. group-unpause plays a voice unpaused through its group until it runs dry: leaving the flag alone, the next delivery never resumes it (12 assertions, 3 failures); without discarding the mark on a user pause, a delivery resumes the voice the user paused while it buffered (1 failure). With the fix: 40 assertions, 0 failures plain, with ASan+UBSan and with TSan.
resetBufferStream() empties an AUTO buffer stream so that a new source can be delivered, but a voice still playing it kept the format it had adopted from the first source: BufferStreamInstance::getAudio() reads the detected rate and channels only while samplerateAlreadySet is false, and nothing cleared it on a reset. When a voice did adopt a format, it set its mixing rate to the bare base rate, dropping its relative play speed. A 44.1 kHz source replaced by a 22.05 kHz one kept the 44100 Hz rate: a 1 kHz tone gave 233 zero crossings in 5120 engine frames instead of about 116, twice the pitch at half the duration. A voice set to 2x before its first chunk arrived played at 1x from the next mix block on: 128 crossings instead of about 233. A reset now advances a format generation on the stream. A voice remembers the generation it adopted its format at and re-reads the format when that changed, so seeks and loop restarts keep it; a voice snapshot keeps the generation, so restoring one taken before a reset still re-reads it. Adopting a format sets the mixing rate to the base rate times the voice's overall relative play speed, as updateVoiceRelativePlaySpeed_internal() does. Every voice now reads the detected format after a reset, so the reset clears it, and advances the generation, under SoLoud's audio mutex and then the buffer mutex, the order the mixer holds them in; nothing takes the audio mutex while it holds the buffer mutex. A new voice reads the generation and the format under the buffer mutex. The background decode thread is still joined before either lock is taken. Verified with the new test/buffer_stream_format_test.cpp, which mixes the engine by hand. At the parent: 39 assertions, 5 failures (reset: rate 44100 and 233 crossings; play-before-data: 128 crossings; reset-mixing, 50 resets while another thread mixes: rate 44100 and 232 crossings). With the fix: 0 failures plain, with ASan+UBSan and with TSan; speed-seek passes in both. Without the reset's locks, reset-mixing gets 2 TSan reports: the mixer's reads of the generation and of the rate against the reset's writes.
A stream turns each AudioMetadata it reports into the struct Dart reads (convertMetadataToFFI() in audiobuffer.cpp and pull_buffer_stream.cpp): it maps the type with a switch and copies the Vorbis, Opus and FLAC fields, whatever the format. AudioMetadata left the type and those fields uninitialized, so a default-constructed one held whatever its storage did. The decoders set the type but fill only their own format's fields, so an MP3, AAC or WAV stream handed Dart indeterminate Ogg fields; and soloudTestInvokeStreamCallbacks(), the test export that dispatches a stream's callbacks, passes one as it is. Built with UBSan, engine_lifecycle_test reports a load of value 1040047929, "not a valid value for type 'DetectedType'", at the switch in PullBufferStream::convertMetadataToFFI(). The type now defaults to BUFFER_UNKNOWN, which the streams convert to UNKNOWN, and the three Ogg info structs are value-initialized. Verified with the new metadata-defaults scenario in test/metadata_parsers_test.cpp, which builds an AudioMetadata over bytes of 0xa5. Without the fix it fails all 4 checks (type 2779096485, and every Ogg field left at 0xa5 bytes), and UBSan reports the invalid enum load. With the fix the suite passes plain and under ASan+UBSan (137 assertions) and in the wasm32 build (70), and engine_lifecycle_test under ASan+UBSan reports nothing (173 assertions, 0 failures; run with ASan's container-overflow check off, for an unrelated report in Player::disposeSound()).
The web binding tests replace Module_soloud with a stand-in whose HEAPU8 was a Dart Uint8List converted with .toJS. On dart2js that is a cast, so the JS side and the Dart side share one buffer. On dart2wasm a Dart typed list lives in Wasm GC memory and .toJS clones it, so the stand-in's JS exports wrote to a copy that the Dart getValue and heap never saw, and Dart's writes never reached the HEAPU8 the bindings read. Handles, packet bytes and string arguments all read back as zeros. The heaps are now JS Uint8Arrays, as on the real module, and the tests read and write them through a ByteData view of that array, which shares its memory on both compilers. The bindings are unchanged: against the real module HEAPU8, getValue and the exports all use the one JS heap.
Merged
2 tasks done
fix: rtAudioMixer hardening, round 2 (mixer core, filters, streams, capture, VST3, platform)
A bus mixes its voices at its own base rate, 44.1 kHz by default, and the engine then resamples the bus's output to the engine's rate. A scheduled or clocked start, or a setDelaySamples delay, is counted in engine samples, but a voice inside a bus counts it down at the bus's rate. On a 48 kHz engine it ran 48000 / 44100, about 9%, long: a start scheduled 4 s ahead was heard about 0.35 s late. Everything played through a bus was also resampled twice. Player::busPlayOnEngine() now plays the bus at the engine's rate. The engine keeps its rate across device changes, so setting it when the bus is played is enough. test/bus_samplerate_test.cpp checks the bus voice's rate on 48 and 96 kHz engines. It also times a start scheduled 1 s ahead inside a bus on a hand-mixed 48 kHz engine: heard at frame 48002 for frame 48000 at the engine's rate, and at 52247 at 44.1 kHz. The bus visualization doc no longer says a bus mixes at 44100 Hz.
fix: mix a bus at the engine's sample rate
alnitak
marked this pull request as ready for review
November 13, 2026 18:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces real-time audio generation and processing capabilities to
flutter_soloud, unlocking low-latency PCM streaming, custom DSP callbacks, and third-party VST3 audio plugin hosting on desktop platforms.1. Real-Time PCM Streaming (
setRealtimeStream) docspushRealtimeClocked) or engine clock time (pushRealtimeScheduled).SoLoudIsolate, or the Web platform.2. Procedural Custom Audio Sources (
createCustomAudioSource) docsplay3d), and routed through mixing buses.3. Custom Multi-Slot DSP Filters (
customFilter) same as previousBus, or individualAudioSources.fadeFilterParameter,oscillateFilterParameter), wet/dry mixing, and zero-cost hard bypass.4. Desktop VST3 Plugin Hosting (
Vst3Host) docsCustomFilterSloton Global or Bus scopes.CustomAudioSources with real-time MIDI input (Note On/Off, CC, Pitch Bend)..vstpresetfiles.enable_vst3: truein Dart Native Assets (hook/build.dart) using precompiled hosting libraries, keeping the core plugin lightweight with zero overhead for mobile and web builds.5. New Example Demonstrations (
example/lib/realtime_audio)Here some links to free VST3 plugins I found for testing:
Steinberg
pluginboutique
Loops
Voxengo
bpb