Skip to content

fix(streamer): run FEC gap waits side by side, not one after another - #1450

Open
etgbrown wants to merge 1 commit into
OpenCloudGaming:devfrom
etgbrown:fec-parallel-gap-waits
Open

etgbrown wants to merge 1 commit into
OpenCloudGaming:devfrom
etgbrown:fec-parallel-gap-waits

Conversation

@etgbrown

@etgbrown etgbrown commented Nov 15, 2026 •

Copy link
Copy Markdown

Problem

The FEC reorder queue starts a damaged block's 52 ms retransmission wait, and sends its first NACK, only when that block reaches the head of the queue. A burst that damages N blocks therefore holds the queue for N × 52 ms. The 0x301 invalidation request (queued when the next frame assembles) waits behind it, and so does the recovery frame the server sends in reply. The backlog is then released at once, which overflows the decoder hand-off and costs another IDR round trip.

On a link that drops packets around periodic handovers (Starlink, every 15 s), five 2-minute 3840×2160@120 AV1 sessions had 48 loss episodes. In 21 of them the queue gave up on two or more gaps in series, 52 ms apart, before the first invalidation left. The six longest (0.65–1.5 s from first gap to a usable frame) gave up on 15–66 gaps each.

Change

FecReorderBuffer (nvst.rs). Waits now run side by side:

  • A block's wait starts when a packet from a later block arrives, and a hole's when the block behind it arrives.
  • Every gap is requested as soon as it is evident, with up to eight NACK reports per 4 ms tick instead of one.
  • A stalled block asks at once for every missing data run it needs to reconstruct, because its wait is no longer extended for a second round.
  • Giving up on a gap forgets only that gap's NACK requests (forget_nacks_through) instead of all of them.
  • The queue reports one discontinuity per pass, so a complete frame between two abandoned gaps is delivered instead of discarded.

The plain reorder queue (streams without an FEC layout) is unchanged. docs/streamer-comparison/transport-recovery.md describes the new behaviour.

Verification

Linux x86_64, Steam Runtime 4 SDK container:

  • cargo fmt --manifest-path native/opennow-streamer/Cargo.toml --all -- --check: clean
  • cargo clippy --locked --manifest-path native/opennow-streamer/Cargo.toml --workspace --all-targets -- -D warnings: clean
  • cargo test --locked --manifest-path native/opennow-streamer/Cargo.toml --workspace: 1468 passed, 0 failed, 14 ignored

The 6 new tests are in nvst_recovery_tests.rs:

  • consecutive_damaged_frames_share_one_nack_window
  • a_recovery_frame_between_two_expired_gaps_is_delivered
  • giving_up_on_a_gap_keeps_the_requests_for_later_gaps
  • fec_holes_wait_from_the_packet_that_revealed_them
  • a_block_that_fills_a_hole_does_not_restart_the_wait
  • a_stalled_block_requests_every_run_it_needs_at_once

multiple_reorder_gaps_do_not_revalidate_post_gap_references now expects the gaps to be reported together.

Live comparison: two 160 s 3840×2160@120 AV1 10-bit sessions on the same link, back to back, on v1.0.5 with local patches (including #1305 and #1306), one session without this change and one with it. Each session had 12 loss episodes.

Without With
Time spent waiting on earlier gaps before the first invalidation, summed 1872 ms 104 ms
Decoder-queue-overflow keyframe requests 3 0

Not shown, and why:

  • Total freeze time: the two sessions saw different link conditions (111 against 265 network gaps), and in the second the server took longer to answer the invalidation. Total freeze time was not measurably lower in this pair.
  • Other platforms: measured on one Linux machine over one Starlink link. The transport is shared, but Windows and macOS were not run.

A repaired gap can still overflow the core's 8-slot media channel. #1451 fixes that separately.

🤖 Generated with Claude Code

The FEC reorder queue started a block's 52 ms retransmission wait, and sent
its first NACK, only when the block reached the head of the queue. A loss
that damaged N blocks therefore held the queue for N x 52 ms, and with it
the reference-invalidation request (queued when the next frame assembles)
and the recovery frame the server sends in reply. The backlog was then
released at once, which overflowed the decoder hand-off and cost another
IDR round trip.

A block's wait now starts when a packet from a later block arrives, and a
hole's when the block behind it arrives. All waits run in parallel:

- every gap is requested as soon as it is evident, with up to eight NACK
  reports per 4 ms tick instead of one;
- a stalled block asks for every missing data run it needs to reconstruct,
  because its wait is no longer extended for a second round;
- giving up on a gap forgets only that gap's NACK requests instead of all;
- the queue reports one discontinuity per pass, so a complete frame between
  two abandoned gaps is delivered instead of discarded.

Evidence, five 2-minute 3840x2160@120 AV1 sessions on a link that loses
packets every 15 s (48 loss episodes): 21 gave up on two or more gaps in
series, 52 ms apart, before the first invalidation left. The six longest
(0.65-1.5 s) gave up on 15 to 66 gaps each, and in five of them a recovery
keyframe was released 4-52 ms before another keyframe that the server had
sent 0.3-0.5 s after it: both had been waiting in the queue.

The plain reorder queue (streams without an FEC layout) is unchanged.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@coderabbitai

coderabbitai Bot commented Nov 15, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2a9f1749-db68-4b4f-95b3-5bd6667247dc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant