Skip to content

feat: add video support for intercom lines (WHIP/WHEP, H264) - #684

Open
bjowe wants to merge 12 commits into
Eyevinn:mainfrom
bjowe:feat/video-support
Open

bjowe wants to merge 12 commits into
Eyevinn:mainfrom
bjowe:feat/video-support

Conversation

@bjowe

@bjowe bjowe commented Sep 15, 2026 •

Copy link
Copy Markdown
Collaborator

Adds in-call participant video: publish the local camera via WHIP and consume remote video via WHEP (H264/VP8 over SMB), with self-preview, per-tile video grid, source pinning, return-feed and fullscreen controls, plus the video-enabled toggle in production setup.

Closes #692

Companion PR (backend): Eyevinn/intercom-manager#315

Adds in-call participant video: publish the local camera via WHIP and
consume remote video via WHEP (H264 over SMB), with self-preview,
per-tile video grid, source pinning, return-feed and fullscreen
controls, plus the video-enabled toggle in production setup.
@bjowe
bjowe force-pushed the feat/video-support branch from e37fafa to e94e56f Compare September 15, 2026 11:03

@birme birme left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated code-reviewer verdict (daily-backlog-pr Phase 3): NEEDS CHANGES

Reviewed the full +4752/−261 diff across 69 files with the branch checked out. Feature-scoped work that mirrors the existing audio patterns well, with good coverage of the new pure logic. Two concrete media-lifecycle bugs should be fixed before shipping a camera feature:

High

  1. Camera capture is never stopped on call exit (privacy/hardware leak). production-line.tsx:149 destructures only const [inputVideoStream] = useVideoInput({...}), discarding the third tuple element — the reset() that stops the camera tracks. useVideoInput.reset (use-video-input.ts:67-72) is the only place videoInput.getTracks().forEach(t => t.stop()) runs, and it's never called. The effect cleanup (:62-64) only sets aborted = true; REMOVE_CALL (global-state-reducer.ts:123) doesn't stop mediaStreamVideoInput; and rtcPeerConnection.close() doesn't stop local getUserMedia tracks. Net: camera stays live + recording indicator on after leaving a video line, until the tab closes. Capture and call the video reset on exit/unmount.
  2. Frame-monitor setIntervals leak on unmount. attachShowWhenReady (video-element-factory.ts:470-490) starts a 500ms interval per remote video element, cleared only by stopFrameMonitor, which is called only from removeThisElement on track ended/removetrack. useVideoElements cleanup (use-video-elements.ts:11-24) nulls srcObject but doesn't call stopFrameMonitor, nor does the useRtcConnection teardown. On abrupt teardown/navigation the interval + rVFC loop keep running against detached elements. Call stopFrameMonitor for every element in cleanUpVideo.

Medium

  1. checkbox.tsx:76-87 — input made readOnly + tabIndex={-1} with toggle delegated to wrapper onClick, synthesizing a fake ChangeEvent via as unknown as. Removes keyboard focus/space-to-toggle from a shared primitive (a11y regression + scope creep). Prefer a native disabled prop.
  2. PC held in useState (never recreated) but stream refs are in the effect deps — a stream-ref change would close() the PC and the re-run no-ops behind signalingState === "closed" guards, dead-ending the call. Latent today (no device-switch re-acquire); add a guard/comment.
  3. No test asserting camera tracks stop / frame monitors clear on exit — exactly where the leaks live.

Nits

  • Raw console.warn/error in prod paths vs the project logger util; video-element-factory.ts builds ~540 lines of tile chrome via imperative createElement+inline styles vs Emotion styled() used elsewhere.

Notes: no user-input URL construction (all via backend API_URL), no new bundle secrets, reducer ERROR change well-contained, CI green. Both High items are small localized fixes.

…n#692)

Capture the useVideoInput reset() and invoke it on exit and on unmount so
camera tracks (getUserMedia) are stopped, releasing the hardware/indicator.
Call stopFrameMonitor for every element in useVideoElements cleanUpVideo so
the 500ms frame-monitor setInterval no longer leaks on teardown.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@birme

birme commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

This PR looks like part of a multi-repo/multi-PR dependency stack together with Eyevinn/intercom-manager#315 (backend video support). Holding off on automated review/merge here — this needs a human (or the relevant implementation agent) to assess the whole stack together, not a per-PR pass.

@bjowe

bjowe commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator Author

#692 covers both High items — I'd written the same two fixes independently and landed on the same
shape, so nothing further from me there.

Two things still open:

main has a broken lockfile. #650 bumped react to ^19.2.7 but left react-dom at
^19.2.3, which resolves to 19.2.4, and React 19 refuses to run when the two differ — 15 of 24
test files fail on a pristine main. This branch is green because it predates the bump, but
inherits the breakage as soon as the conflict below is resolved. Opened separately as #NNN.

This PR currently conflicts with main, in calls-page.tsx against #693. When resolving:
#693's !isGuest guard belongs on the add-call JoinProduction, but the new VideoLineJoinCard
block should stay unguarded — pendingVideoRefs comes from the same validated-refs path as
pendingCallRefs, so a guest arriving on a share link still needs to join the line they were
invited to. Worth a second opinion if that reading is wrong.

Unrelated, noticed while in there: useAudioInput has the same shape the camera bug had —
resetAudioInput is only wired to the device-switch paths, never to call exit, so the mic track
outlives the call. Pre-existing on main, not this PR.

@Saelmala
Saelmala requested a review from birme September 28, 2026 13:23
# Conflicts:
#	src/components/calls-page/calls-page.tsx
#	src/components/production-line/use-line-polling.ts
#	src/components/production-line/use-rtc-connection.ts

@birme birme left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review — video support (WHIP/WHEP, H264) — Needs Changes

Reviewed by daily-backlog-pr (Phase 3). A large, well-structured video feature that correctly routes all state through the reducer (reusing UPDATE_CALL/extending DEVICES_UPDATED, no ad-hoc action types) and all REST calls through the API object — no blocking architectural or WebRTC violations. But several substantial new modules ship with no tests, there is a camera-track resource leak, and a block of pin-styling code is dead. Per project rules this cluster warrants Needs Changes. CI (build/lint/prettier/tests/e2e) is green — these are correctness/coverage findings, not formatting.

Blocking

  • None.

Warnings

  • No tests for the largest new modules. video-element-factory.ts (543 lines: tile factory, draggable PiP, fullscreen, rVFC stall monitor) has no test file; use-video-source-pin.ts (retry/backoff-on-HTTP-425, latest-wins ref) has no renderHook test; pin-data-channel.ts sendPinnedEndpoint (data-channel message shape — a correctness-sensitive SMB contract) is untested; use-video-tile-grid.ts and use-shadow-rtc-connection.ts (a full second WHIP/WHEP session lifecycle: peer connection, heartbeat, teardown) are untested. These pure/factory/hook units are the highest-value test targets.
  • Camera-track leak on input change — use-video-input.ts:45-65. When videoInputId changes, the effect cleanup only sets aborted = true and never stops the previously acquired stream's tracks; setVideoInput(stream) replaces the reference. Switching cameras mid-call leaves the old camera active (indicator light stays on) and the old track detached but unstopped. Cleanup should stop the prior stream's tracks (only the explicit reset() on exit does today).
  • Dead pin-styling code — video-element-factory.ts:394-403. updateVideoTileContainerPinned queries [data-tile-options-btn], but that attribute is never set on any element created in createVideoTileContainer, so the function always early-returns and container.dataset.pinned is never set. Pin reordering still works via tile.style.order, so the effect is cosmetic — but this code cannot run as written.
  • Duplicated sibling-line polling — production-line.tsx:626-644. Reimplements participant polling as a raw window.setInterval(... API.fetchProductionLine ...) every 2s with .catch(() => {}), duplicating use-line-polling.ts and silently swallowing all errors.

Suggestions

  • Use the project logger (logger.red/logger.yellow) instead of console.warn/console.error (use-rtc-connection.ts:84, use-video-input.ts:52).
  • use-video-input.ts:54-59 — the ERROR dispatch omits callId, so a per-camera failure shows a global banner rather than a call-scoped one.
  • production-line.tsx:620,631,655 — inline import("./types.ts").TParticipant/TLine refs; use the existing named import.
  • ice-gathering.ts waitForIceGathering changed from reject-on-timeout to resolve-on-timeout (5s→8s). This is a shared function, so it also alters the audio-only flow (a timeout now proceeds to PATCH with partial candidates instead of tearing down). Defensible robustness improvement, but validate against SMB in a real environment.

Cross-repo coupling (hard dependency on intercom-manager #315)

Non-functional in isolation: requires #315's new routes (PATCH .../whep-source, PATCH /session/:id/video-source), the new hasVideo (required)/isWhepReceiver/videoEnabled/whepSourceSessionId response fields (the auto-pin filter p.hasVideo && !p.isWhepReceiver silently no-ops if hasVideo is absent), the HTTP 425 not-ready contract (retry only triggers on 425), and SMB forwarding H264 + honoring PinnedEndpointsChanged. Do not merge ahead of #315 — treat as an atomic pair.

Moving the board item back to Ready for these changes.

@Saelmala
Saelmala requested a review from birme October 1, 2026 12:05
# Conflicts:
#	src/components/production-line/use-line-polling.test.ts
# Conflicts:
#	src/components/production-line/production-line.tsx
#	src/components/production-line/use-rtc-connection.test.ts
#	src/components/production-line/user-list.tsx
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add video support for intercom lines (WHIP/WHEP, H264)

3 participants