Add H.264 WebRTC signaling and snapshots, moved from the Home Assistant integration

README.md

@@ -44,6 +44,7 @@ Scripts that do not use asyncio can use the same API as blocking calls through
   certificates, sessions, passwords and user accounts
 - [API reference](docs/api.md): every method and the hardware and firmware it
   needs
+- [Video streaming](docs/streaming.md): snapshots, MJPEG and WebRTC signaling
 - [Blocking client](docs/sync.md): the API without asyncio
 - [Errors](docs/errors.md)
 - [SSH](docs/ssh.md)

docs/api.md

@@ -117,8 +117,8 @@ while loop.time() < end:
 
 | Methods | Requires |
 |---|---|
-| `mjpeg_stream`, `mjpeg_stream_bytes`, `open_ws` | all |
-| `set_screen`, `set_frame_detect`, `pause_frame_detect` | non-Pro |
+| `mjpeg_stream`, `mjpeg_stream_bytes`, `open_ws`, `open_webrtc_signaling` | all |
+| `get_snapshot`, `set_screen`, `set_frame_detect`, `pause_frame_detect` | non-Pro |
 | `get_input_region`, `set_input_region`, `get_input_resolution` | non-Pro 2.5.1 |
 | `reset_hdmi` | PCIe 2.1.5 |
 | `get_hdmi_state`, `enable_hdmi`, `disable_hdmi` | PCIe 2.2.8 |
@@ -130,10 +130,15 @@ while loop.time() < end:
 | `set_rate_control_mode` | Pro 1.2.6 |
 | `set_fps` | Pro 1.2.8 |
 
-`mjpeg_stream()` yields decoded `PIL.Image` frames; `mjpeg_stream_bytes()`
-yields the raw JPEG bytes. `open_ws(path)` opens a new WebSocket on a path
-relative to the API base, such as `stream/h264/webrtc`, for protocols the
-client does not wrap. The caller closes it before the client exits.
+`mjpeg_stream()` yields decoded `PIL.Image` frames, `mjpeg_stream_bytes()` the
+raw JPEG bytes, and `get_snapshot()` the first one. `open_webrtc_signaling()`
+returns the H.264 WebRTC signaling session for the detected hardware.
+[Video streaming](streaming.md) covers both, and why the Pro refuses
+snapshots.
+
+`open_ws(path)` opens a new WebSocket on a path relative to the API base, such
+as `stream/h264/webrtc`, for protocols the client does not wrap. The caller
+closes it before the client exits.
 
 ### Images and scripts
 

docs/errors.md

@@ -11,6 +11,7 @@ All exceptions derive from `NanoKVMError` and are importable from
 | `NanoKVMPermissionError` | The account lacks permission; the session stays valid |
 | `NanoKVMApiError` | The device answered with an error code |
 | `NanoKVMInvalidResponseError` | The response could not be parsed |
+| `NanoKVMVideoModeError` | The Pro stopped a WebRTC stream because another video mode took the encoder |
 
 `NanoKVMPermissionError` carries `status`, `method` and `path`.
 `NanoKVMApiError` carries the numeric `code`; the device's `msg` and `data` are

docs/streaming.md

@@ -0,0 +1,102 @@
+# Video streaming
+
+The NanoKVM serves the HDMI input as an MJPEG stream and as H.264 over
+WebRTC. The client reads MJPEG itself; for WebRTC it handles the signaling and
+leaves the media to a WebRTC peer, such as a browser or `aiortc`.
+
+| Need | Method |
+|---|---|
+| One still image | `get_snapshot()` (non-Pro) |
+| Raw JPEG frames | `mjpeg_stream_bytes()` |
+| Decoded `PIL.Image` frames | `mjpeg_stream()` |
+| Live H.264 video, and audio on the Pro | `open_webrtc_signaling()` |
+
+## The Pro's single encoder
+
+The Pro encodes one stream mode at a time (`StreamMode`, set with
+`set_stream_mode()`). Opening the MJPEG stream switches it to MJPEG, and any
+WebRTC session in progress stops: its iteration raises
+`NanoKVMVideoModeError`. For that reason `get_snapshot()` raises
+`NanoKVMNotSupportedError` on the Pro without opening the stream
+([NanoKVM-Pro#157](https://github.com/sipeed/NanoKVM-Pro/issues/157)).
+`mjpeg_stream()` and `mjpeg_stream_bytes()` still work on the Pro, at that
+cost.
+
+## MJPEG
+
+```python
+from contextlib import aclosing
+from pathlib import Path
+
+Path("screen.jpg").write_bytes(await client.get_snapshot())
+
+async with aclosing(client.mjpeg_stream_bytes()) as frames:
+    async for frame in frames:
+        ...  # one JPEG per iteration
+```
+
+`get_snapshot(timeout=20.0)` returns the first frame, raises `TimeoutError`
+when none arrives in time, and `NanoKVMError` when the stream ends without one.
+The streams run until the caller stops iterating; `aclosing()` closes the HTTP
+response as soon as the loop is left.
+
+## WebRTC signaling
+
+`open_webrtc_signaling()` picks the endpoint for the detected hardware and
+returns a `nanokvm.webrtc.WebRTCSignaling` on its own WebSocket:
+
+```python
+from nanokvm.webrtc import IceCandidate, WebRTCAnswer, WebRTCCandidate
+
+async with await client.open_webrtc_signaling() as signaling:
+    await signaling.send_offer(offer_sdp)
+    async for event in signaling:
+        if isinstance(event, WebRTCAnswer):
+            ...  # hand event.sdp to the peer as the remote description
+        elif isinstance(event, WebRTCCandidate):
+            ...  # hand event.candidate to the peer
+```
+
+1. Send the peer's offer with `send_offer(sdp)`. Ask for video, and audio too
+   on the Pro; the non-Pro has video only.
+2. Send each of the peer's candidates with
+   `send_candidate(IceCandidate(...))`, after the offer. `IceCandidate` takes
+   the browser's `RTCIceCandidateInit` field names (`sdpMid`, `sdpMLineIndex`,
+   `usernameFragment`) or the snake_case ones.
+3. Iterate for exactly one `WebRTCAnswer` and any number of
+   `WebRTCCandidate` events. Iterating and sending can run concurrently, for
+   example the iteration in its own task.
+4. Iteration ends when the device closes the WebSocket. Close the session, or
+   leave its `async with`, before the client exits: the session owns the
+   WebSocket, not the client.
+
+A `ValueError` from `send_offer()` means a Pro offer without audio or video.
+The Pro ignores WebSocket pings, so the session sends its application
+heartbeat every 60 seconds; a failed heartbeat closes the session, which ends
+the iteration.
+
+### What happens on the Pro
+
+The non-Pro negotiates one video peer connection on `stream/h264`, so the
+session passes the offer, answer and candidates through. The Pro negotiates
+video and audio as two peer connections on `stream/h264/webrtc`; the session
+hides that, so the caller sees one offer and one answer on any hardware:
+
+- The offer is split into one single-section offer per media kind.
+- The answers are merged into one answer to the original offer, once all
+  kinds have answered. Candidates that arrive before that are held, then
+  yielded after the answer.
+- Candidates are mapped both ways between the sections of the original offer
+  and the single-section peers.
+
+The split and merge are public in `nanokvm.webrtc` for callers that run their
+own signaling: `split_pro_webrtc_offer()` returns a `ProWebRTCOffer`, whose
+`upstream_candidate_for_kind()` and `client_candidate_for_kind()` map
+candidates, and `merge_pro_webrtc_answers()` builds the answer.
+
+## Blocking client
+
+`NanoKVMSyncClient` offers `get_snapshot()` and the MJPEG streams as blocking
+calls. WebRTC signaling is async-only, because its WebSocket belongs to the
+client's event loop: reach it through `run()`, as in
+[Blocking client](sync.md).

docs/sync.md

@@ -40,8 +40,8 @@ the wrapped `client` on its event loop and keeps the result type:
 gpio = kvm.run(kvm.client.get_gpio())
 ```
 
-`run()` is also the way to the async-only `open_ws()`, whose WebSocket belongs
-to the client's event loop:
+`run()` is also the way to the async-only `open_ws()` and
+`open_webrtc_signaling()`, whose WebSockets belong to the client's event loop:
 
 ```python
 async def read_message(client):

nanokvm/client.py

@@ -27,6 +27,7 @@
     NanoKVMNotAuthenticatedError,
     NanoKVMNotSupportedError,
     NanoKVMPermissionError,
+    NanoKVMVideoModeError,
 )
 from .models import *  # noqa: F403
 from .models import GetAccountRsp, HWVersion, IsPasswordUpdatedRsp
@@ -43,6 +44,7 @@
     "NanoKVMNotAuthenticatedError",
     "NanoKVMNotSupportedError",
     "NanoKVMPermissionError",
+    "NanoKVMVideoModeError",
     "PASTE_CHAR_MAP",
     "SessionController",
     "StreamController",

nanokvm/components/stream.py

@@ -13,10 +13,17 @@
 from aiohttp import BodyPartReader, ClientWebSocketResponse, MultipartReader, hdrs
 from PIL import Image
 
+from ..compatibility import require_hardware
+from ..exceptions import NanoKVMError
+from ..models.common import HWFamily
+from ..webrtc import WebRTCSignaling
 from .session import Controller
 
 _LOGGER = logging.getLogger(__name__)
 
+_NON_PRO_WS_HEARTBEAT_SECONDS = 30.0
+_WS_CLOSE_TIMEOUT_SECONDS = 15.0
+
 
 class StreamController(Controller):
     """Own MJPEG frame transport and JPEG decoding."""
@@ -48,6 +55,22 @@ async def mjpeg_stream_bytes(self) -> AsyncGenerator[bytes, None]:
                 session.check_session_generation(generation)
                 yield data
 
+    @require_hardware(HWFamily.NON_PRO)
+    async def get_snapshot(self, timeout: float = 20.0) -> bytes:
+        """Return one raw JPEG frame from the MJPEG stream.
+
+        Not on the Pro: its MJPEG and H.264 streams share one encoder mode, so
+        opening MJPEG interrupts any WebRTC session
+        (https://github.com/sipeed/NanoKVM-Pro/issues/157).
+        """
+        async with (
+            asyncio.timeout(timeout),
+            contextlib.aclosing(self.mjpeg_stream_bytes()) as frames,
+        ):
+            async for frame in frames:
+                return frame
+        raise NanoKVMError("MJPEG stream ended without a frame")
+
     async def mjpeg_stream(self) -> AsyncGenerator[Image.Image, None]:
         """Stream decoded MJPEG frames."""
         session = self._session
@@ -70,3 +93,20 @@ async def open_ws(self, path: str, **kwargs: Any) -> ClientWebSocketResponse:
         caller closes the WebSocket, before the client exits.
         """
         return await self._session.open_ws(path, **kwargs)
+
+    async def open_webrtc_signaling(self) -> WebRTCSignaling:
+        """Open an H.264 WebRTC signaling session, on either hardware family.
+
+        The caller closes it, before the client exits.
+        """
+        session = self._session
+        if session._hw_version is None:
+            await session.detect_hardware()
+        pro = session.is_hardware_family(HWFamily.PRO)
+        websocket = await session.open_ws(
+            "stream/h264/webrtc" if pro else "stream/h264",
+            # The Pro gets an application-level heartbeat from WebRTCSignaling.
+            heartbeat=None if pro else _NON_PRO_WS_HEARTBEAT_SECONDS,
+            timeout=aiohttp.ClientWSTimeout(ws_close=_WS_CLOSE_TIMEOUT_SECONDS),
+        )
+        return WebRTCSignaling(websocket, pro=pro)

nanokvm/exceptions.py

@@ -50,3 +50,7 @@ class NanoKVMInvalidResponseError(NanoKVMError):
 
 class NanoKVMNotSupportedError(NanoKVMError):
     """Feature not supported on this hardware variant."""
+
+
+class NanoKVMVideoModeError(NanoKVMError):
+    """The NanoKVM Pro stopped a stream because another video mode is active."""

nanokvm/sync_client.py

@@ -18,7 +18,7 @@
 T = TypeVar("T")
 
 # Their results stay bound to the client's event loop; use ``run()`` instead.
-_ASYNC_ONLY = frozenset({"open_ws"})
+_ASYNC_ONLY = frozenset({"open_ws", "open_webrtc_signaling"})
 
 # Wait in short slices: before Python 3.14, an untimed wait on Windows ignores
 # Ctrl-C until the operation finishes.