Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
c399ed3
refactor(workspace): remove functions and fields nothing calls
datagutt Oct 1, 2026
ad0cfef
docs(workspace): drop port-process tags and comments that describe th…
datagutt Oct 1, 2026
d71587b
fix(irl-source): low-latency clock suspend reads the pump's injected …
datagutt Oct 1, 2026
12cf0e4
refactor(workspace): read pts repair gaps and network buffer from consts
datagutt Oct 1, 2026
4f7913e
refactor(workspace): share the byte pcm helpers through irl-core dsp
datagutt Oct 1, 2026
2c6e891
refactor(workspace): derive the shared state constructors from defaults
datagutt Oct 1, 2026
1095d53
refactor(workspace): one rate limit helper for the decoder and lead w…
datagutt Oct 1, 2026
67b443a
refactor(workspace): pass pts and bytes to the pacing queue instead o…
datagutt Oct 1, 2026
3e5f48c
refactor(workspace): inline single-use video helpers and share the ns…
datagutt Oct 1, 2026
14d4ffa
refactor(irl-source): store hot values in one call and expose the pac…
datagutt Oct 1, 2026
120f90d
refactor(workspace): drop obs api the plugin does not call
datagutt Oct 1, 2026
eacb461
refactor(workspace): drop ffmpeg api the plugin does not call
datagutt Oct 2, 2026
92cdf6c
refactor(obs-sys): drop the unused obs_data_get_double declaration
datagutt Oct 2, 2026
5f88ae7
test(irl-source): share fixtures and fold repeated setup into helpers
datagutt Oct 2, 2026
b5a8992
test(irl-core): table-drive repeated cases and drop redundant tests
datagutt Oct 2, 2026
159c5c3
test(irl-core): pin consts to their own values, not the c plugin's
datagutt Oct 2, 2026
e01668f
chore(workspace): drop transfer alignment consts the ffmpeg crate own…
datagutt Oct 2, 2026
58bbae2
refactor(irl-source)!: report only the stats that diagnosed real issues
datagutt Oct 2, 2026
2a0615d
docs(obs-sys): trim comments to what the declarations cannot say
datagutt Oct 2, 2026
1dc79b1
docs(obs): drop c plugin references and history from comments
datagutt Oct 2, 2026
6212bf5
docs(ffmpeg): trim comments to what the code cannot say
datagutt Oct 2, 2026
6528da1
docs(irl-core): trim comments to the reasons the code cannot carry
datagutt Oct 2, 2026
06982e7
docs(irl-provider): drop a comment that restates its name
datagutt Oct 2, 2026
3b6320c
docs(irl-source): trim shell and receiver comments to what the code c…
datagutt Oct 2, 2026
7d0ac24
docs(irl-source): trim audio and video thread comments to what the co…
datagutt Oct 2, 2026
8cb376b
docs(irl-source): drop port history and change narration from test co…
datagutt Oct 2, 2026
4804a97
refactor(irl-core): audio buffer construction cannot fail, so it no l…
datagutt Oct 2, 2026
9540fd1
docs(workspace): cut CLAUDE.md to the rules and move the design narra…
datagutt Oct 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ body:
placeholder: |
[irl-source] Video stream 0: hevc 1920x1080 (d3d11va)
[irl-source] First keyframe received (1920x1080 fmt=171 hardware decode)
[irl-source] Stats: video=1801 audio=2997 buf=100ms target=120ms speed=1.000 ...
[irl-source] Stats: video=1801 audio=1406 buf=118ms peak=164ms target=120ms speed=1.000 ll=off ...

- type: input
id: plugin-version
Expand Down
290 changes: 55 additions & 235 deletions CLAUDE.md

Large diffs are not rendered by default.

39 changes: 13 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ const { responseData } = await obs.call("CallVendorRequest", {
requestData: { source_name: "IRL Source" },
});

console.log(responseData.stream_delay_ms, responseData.buffer_fill_ms);
console.log(responseData.buffer_fill_ms, responseData.current_speed);
```

There is no event stream, so poll `GetStats` at whatever rate your overlay refreshes; once a second is plenty. The request reads the same snapshot the Lua path does, so both transports always report identical numbers.
Expand Down Expand Up @@ -251,14 +251,14 @@ The 70ms and 2000ms thresholds are fixed internally and match the libobs behavio
- Low-delay decode: no B-frame reorder buffering, capped decode threading.
- Zero-copy for supported pixel formats, planes go straight to OBS. Native 10-bit passthrough for YUV420P10LE (I010) and P010. Unsupported formats fall back to swscale.
- Mid-stream resolution changes are detected and handled without recreating the source.
- Damaged H.264 frames are passed through with their timestamps rather than dropped, which preserves cadence instead of freezing on every corrupt frame. HEVC frames predicted from a reference that never arrived are held back instead: HEVC has no error concealment, so FFmpeg synthesizes the missing reference as flat gray and everything predicted from it is gray until the next keyframe. The last good frame stays on screen for that stretch (`video_corrupt_held` counts them).
- Damaged H.264 frames are passed through with their timestamps rather than dropped, which preserves cadence instead of freezing on every corrupt frame. HEVC frames predicted from a reference that never arrived are held back instead: HEVC has no error concealment, so FFmpeg synthesizes the missing reference as flat gray and everything predicted from it is gray until the next keyframe. The last good frame stays on screen for that stretch, and the log says how many frames were held once the picture resumes.
- Video PTS is mapped through the audio playout offset for lip sync.

## Decoder recovery

Repeated audio decode errors trigger a throttled audio decoder flush and a reset of bad timing state. This is what stops SRT bitrate starvation from permanently breaking audio, which is the failure mode the built-in Media Source hits.

The video decoder is deliberately never flushed. Flushing empties the reference picture buffer and clears the decoder's recovery state, and neither the H.264 nor the HEVC decoder produces a real picture again until the next keyframe: H.264 paints frames gray until a recovery point, HEVC synthesizes each missing reference as flat gray. On a lossy stream that turned a few damaged frames into a whole GOP of gray. A decode error on a live stream is a property of the packet, not of the decoder, so the next intact packet decodes fine without a reset. Bursts are still counted and logged (`video_corrupt_frames`, and the `corrupt=`/`held=` fields of the stats line).
The video decoder is deliberately never flushed. Flushing empties the reference picture buffer and clears the decoder's recovery state, and neither the H.264 nor the HEVC decoder produces a real picture again until the next keyframe: H.264 paints frames gray until a recovery point, HEVC synthesizes each missing reference as flat gray. On a lossy stream that turned a few damaged frames into a whole GOP of gray. A decode error on a live stream is a property of the packet, not of the decoder, so the next intact packet decodes fine without a reset. Bursts are still logged.

## Compared with the Media Source

Expand Down Expand Up @@ -312,44 +312,31 @@ Stats are exposed through OBS's `proc_handler` API under the `get_stats` call, a
| --- | --- | --- |
| `buffer_fill_ms` | int | Current audio jitter buffer fill level (ms) |
| `current_speed` | float | Current audio correction factor. Buffered mode keeps this near 1.0. |
| `adaptive_latency_control` | bool | Whether buffered steady-state latency correction is enabled |
| `reconnecting` | bool | Whether the source is currently reconnecting |
| `total_audio_frames` | int | Total audio frames decoded since connection |
| `total_video_frames` | int | Total video frames decoded since connection |
| `pts_repairs` | int | Number of non-normal PTS discontinuities repaired |
| `pts_normalizations` | int | Number of frame-sized PTS cadence offsets normalized without treating them as damage |
| `pts_interpolations` | int | Number of non-frame-sized small PTS gaps smoothed by timestamp interpolation |
| `pts_resets` | int | Number of large PTS gaps that triggered a timing reset |
| `pts_last_gap_ms` | int | Most recent repaired PTS gap size |
| `pts_max_gap_ms` | int | Largest repaired PTS gap size since the current connection/reset |
| `silence_insertions` | int | Number of silence insertions for gap filling |
| `pts_max_gap_ms` | int | Largest repaired audio PTS gap since the current connection |
| `audio_underruns` | int | Number of plugin-side underruns that emitted silence to keep OBS audio timestamps monotonic |
| `audio_resync_skipped_chunks` | int | Number of buffered audio chunks skipped by low-latency backlog capping or startup trims |
| `audio_hidden_trimmed_chunks` | int | Number of buffered chunks trimmed before playback primed (never audible) |
| `audio_quality_events` | int | Aggregate audible-risk counter for underruns, inserted silence, resyncs, PTS resets, and audio decoder flushes |
| `audio_output_restarts` | int | Output clock restarts after the audio thread stalled (should stay 0) |
| `obs_lead_ms` | int | How far ahead of real time audio is queued inside OBS (healthy is roughly 60 to 100ms) |
| `audio_decoder_flushes` | int | Number of audio decoder flushes after repeated decode errors |
| `video_corrupt_frames` | int | Decoded frames the decoder flagged as damaged (concealed slice errors on H.264, missing-reference prediction on HEVC) |
| `video_corrupt_held` | int | HEVC frames held back instead of shown because they were predicted from a missing reference and would have rendered gray; the last good frame stays on screen until the next keyframe |
| `video_lead_ms` | int | How far ahead of real time the last video frame was timestamped. Tracks the audio buffer; a value climbing well past Target Buffer and staying there means concealment has inflated the A/V mapping |
| `video_lead_excess` | int | Frames whose lead exceeded what OBS's async queue can absorb. Harmless while the lead is steady; sustained growth is what makes OBS drop queued video |
| `video_delay_ms` | int | Standing delay added to the video schedule because video reached the plugin too late to be paced against the audio playout (the encoder sends video later than audio by more than Target Buffer covers). It is sized at connection start and then checked against what frames actually need: when ten seconds of frames all needed less, it ramps back down with video playing at the Catch-Up Speed, so a bad first second on a poor link does not leave the picture behind the sound for the whole stream. Lip sync is off by this much while it lasts. When the sender keeps its video this far behind its audio for two seconds, `audio_hold_ms` rises to cover it, and this delay drops back to zero as the audio slows into step. In Low Latency Audio mode, or with Adaptive Latency Control off, the hold is only sized when the stream starts. A delay that stays above zero with the hold in force means video is late for a reason on this machine (decoding, a busy GPU); raising Target Buffer by at least this much takes it back to zero |
| `av_skew_ms` | int | Video PTS minus audio PTS of what last reached the plugin: how the sender stamps its two streams against each other, before any buffering here. Near zero for a healthy sender; a standing negative value is video stamped behind its audio, which is what `audio_hold_ms` then covers. Measured as packets arrive, so the delay itself does not show up in it |
| `audio_hold_ms` | int | How much longer than Target Buffer audio waits before it plays, so that video which reaches the plugin later than its audio is in hand when the two are due. Measured before audio starts, so a sender that always sends its video late (pocketSRT, a phone with video stabilization on) is in sync from the first second. After that it rises when the skew lasts two seconds, which the speed controller builds by playing up to 2% slow, and falls when ten seconds of video needed less. 0 for a sender whose skew Target Buffer already covers |
| `stream_delay_ms` | int | End-to-end stream delay (SRT latency + decode + buffering) |
| `low_latency_audio` | bool | Whether OBS async unbuffered low-latency mode is enabled |
| `reconnect_count` | int | Number of reconnect attempts since the source was created |
| `reconnecting` | bool | Whether the source is currently reconnecting |

### OBS log stats

The plugin also logs stats to the OBS log every 30 seconds:

```
[irl-source] Stats: video=1801 audio=2997 buf=100ms target=120ms speed=1.000 ctrl=on pts_repairs=0 norm=0 interp=0 silence=0 resets=0 last_gap=0ms max_gap=0ms underruns=0 resync_skips=0 hidden_trims=0 quality_events=0 audio_flushes=0 corrupt=0 held=0 obs_lead=99ms chunk=960@48000 stream_chunk=20ms obs_chunk=20ms restarts=0 res=1920x1080
[irl-source] Stats: video=1801 audio=1406 buf=118ms peak=164ms target=120ms speed=1.000 ll=off max_gap=0ms underruns=0 restarts=0 av_drift=0ms av_skew=-14ms hold=0ms vdelay=0ms vfps=60.0 vq_drops=0 pktq=7/38(96KB,116ms) paced=14/16(43MB) eagain=0/0 pktdrop=0/0 res=1920x1080
```

A healthy stream shows `speed=1.000`, `underruns=0`, `restarts=0`, and a constant `chunk` size. `buf` plus `obs_lead` is your plugin-side latency (fill wanders inside a deadband around the target by design).
A healthy stream shows `speed` near 1.000, `underruns=0`, `restarts=0`, `av_drift` near 0, and `vdelay` and `hold` at 0. `buf` wanders inside a deadband around `target` by design, and `peak` is the highest fill since the source was created. `ll` is Low Latency Audio, and `ctrl=off` appears only when Adaptive Latency Control is turned off. The rest describe the video path:

- `vfps`: the frame rate the sender is delivering.
- `vq_drops`: compressed video packets dropped because the decode queue overflowed (should stay 0).
- `pktq`: compressed video waiting to be decoded, as now/peak packets (size, duration). Its duration tracks Target Buffer.
- `paced`: decoded frames waiting for their due time, as now/peak (memory).
- `eagain` and `pktdrop`: packets a decoder refused at first and packets it never took, as video/audio.

## Building from source

Expand Down
58 changes: 14 additions & 44 deletions crates/ffmpeg/src/codec.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
//! Decoders.

use core::ffi::{CStr, c_void};
use core::ffi::c_void;
use core::mem::ManuallyDrop;

use crate::format::StreamRef;
Expand Down Expand Up @@ -57,21 +57,6 @@ impl Codec {
index += 1;
}
}

pub fn name(&self) -> &'static str {
// SAFETY: `self.0` is a live static descriptor with a static `name`.
let ptr = unsafe { (*self.0).name };
if ptr.is_null() {
return "unknown";
}
// SAFETY: `name` is a NUL-terminated string constant inside libavcodec.
unsafe { CStr::from_ptr(ptr) }.to_str().unwrap_or("unknown")
}

#[doc(hidden)]
pub fn as_ptr(&self) -> *const ffmpeg_sys_next::AVCodec {
self.0
}
}

/// The `get_format` callback type for forced-NVDEC negotiation. Receives the
Expand Down Expand Up @@ -206,8 +191,8 @@ impl CodecBuilder {
self
}

/// Whether `hw_device_ctx` is currently attached (the C plugin's
/// "hardware was requested" check before the open attempt).
/// Whether `hw_device_ctx` is currently attached, i.e. hardware was
/// requested.
pub fn has_hw_device(&self) -> bool {
// SAFETY: `self.ptr` is an unopened context we own.
!unsafe { (*self.ptr).hw_device_ctx }.is_null()
Expand Down Expand Up @@ -248,7 +233,10 @@ impl CodecBuilder {
return Err(Error(ret));
}

Ok(CodecContext { ptr, codec, opaque })
Ok(CodecContext {
ptr,
_opaque: opaque,
})
}
}

Expand All @@ -264,32 +252,21 @@ impl Drop for CodecBuilder {
/// An open decoder.
pub struct CodecContext {
ptr: *mut ffmpeg_sys_next::AVCodecContext,
codec: Codec,
// Box<CtxOpaque> behind ctx->opaque, kept so it outlives the context.
opaque: Option<Box<CtxOpaque>>,
_opaque: Option<Box<CtxOpaque>>,
}

// SAFETY: an AVCodecContext has no thread affinity of its own (its internal
// worker threads are owned by libavcodec); the plugin gives each decoder to a
// single thread and never shares one.
unsafe impl Send for CodecContext {}

#[doc(hidden)]
pub struct CtxOpaque {
pub get_format: GetFormatFn,
pub codec: Codec,
struct CtxOpaque {
get_format: GetFormatFn,
codec: Codec,
}

impl CodecContext {
pub fn codec(&self) -> Codec {
self.codec
}

pub fn codec_id(&self) -> AVCodecID {
// SAFETY: `self.ptr` is an open context we own.
unsafe { (*self.ptr).codec_id }
}

/// `hw_device_ctx != NULL` after open (FFmpeg may drop it on failure).
pub fn has_hw_device(&self) -> bool {
// SAFETY: as above.
Expand All @@ -305,8 +282,8 @@ impl CodecContext {

/// `avcodec_receive_frame` into `frame` (unrefs it first).
pub fn receive_frame(&mut self, frame: &mut Frame) -> Result<()> {
// avcodec_receive_frame unrefs internally, but the C plugin's callers
// rely on the frame being blank on every error path too.
// avcodec_receive_frame unrefs internally, but callers rely on the
// frame being blank on every error path too.
frame.unref();
// SAFETY: `self.ptr` is an open decoder and `frame` a live frame we own.
Error::check(unsafe {
Expand All @@ -319,18 +296,12 @@ impl CodecContext {
// SAFETY: `self.ptr` is an open decoder.
unsafe { ffmpeg_sys_next::avcodec_flush_buffers(self.ptr) };
}

#[doc(hidden)]
pub fn as_mut_ptr(&mut self) -> *mut ffmpeg_sys_next::AVCodecContext {
let _ = &self.opaque;
self.ptr
}
}

impl Drop for CodecContext {
fn drop(&mut self) {
// SAFETY: `&mut self.ptr` is our sole owning pointer. Rust runs this
// body before dropping `self.opaque`, so the get_format opaque stays
// body before dropping `self._opaque`, so the get_format opaque stays
// alive until libavcodec can no longer call back into it.
unsafe { ffmpeg_sys_next::avcodec_free_context(&mut self.ptr) };
}
Expand All @@ -343,7 +314,6 @@ mod tests {
#[test]
fn finds_a_built_in_decoder() {
let codec = Codec::find_decoder(AVCodecID::AV_CODEC_ID_H264).expect("h264 decoder");
assert_eq!(codec.name(), "h264");
// The bundled build has hardware configs compiled in; the list must at
// least be readable and terminate.
let configs = codec.hw_configs();
Expand Down
19 changes: 3 additions & 16 deletions crates/ffmpeg/src/dict.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,23 +24,13 @@ impl Dictionary {

/// Consume the dictionary for `avformat_open_input` (which takes it by
/// `&mut` and leaves the unconsumed entries behind).
#[doc(hidden)]
pub fn as_mut_ptr(&mut self) -> *mut *mut ffmpeg_sys_next::AVDictionary {
pub(crate) fn as_mut_ptr(&mut self) -> *mut *mut ffmpeg_sys_next::AVDictionary {
&mut self.0
}

/// Number of entries left in the dictionary.
pub fn len(&self) -> usize {
self.iter_keys().count()
}

pub fn is_empty(&self) -> bool {
self.0.is_null()
}

/// Keys left after `avformat_open_input`, i.e. options the demuxer did not
/// recognise (logged by the C plugin).
pub fn remaining_keys(&self) -> Vec<String> {
/// recognise.
pub(crate) fn remaining_keys(&self) -> Vec<String> {
self.iter_keys().collect()
}

Expand Down Expand Up @@ -85,7 +75,6 @@ mod tests {
#[test]
fn set_then_read_back_keys() {
let mut dict = Dictionary::new();
assert!(dict.is_empty());
assert!(dict.remaining_keys().is_empty());

dict.set(c"probesize", c"1000000").unwrap();
Expand All @@ -95,7 +84,5 @@ mod tests {
let mut keys = dict.remaining_keys();
keys.sort();
assert_eq!(keys, vec!["fflags".to_string(), "probesize".to_string()]);
assert_eq!(dict.len(), 2);
assert!(!dict.is_empty());
}
}
Loading
Loading