From 22c302e9f842a7b7448dc57e98b93bcb9f2d1c8d Mon Sep 17 00:00:00 2001 From: styu12 Date: Mon, 13 Apr 2026 14:54:23 -0700 Subject: [PATCH 01/18] docs(plan): OpenCV to PyAV migration plan Co-Authored-By: Claude Opus 4.6 (1M context) --- .../2026-04-13-opencv-to-pyav-migration.md | 1468 +++++++++++++++++ 1 file changed, 1468 insertions(+) create mode 100644 docs/plans/2026-04-13-opencv-to-pyav-migration.md diff --git a/docs/plans/2026-04-13-opencv-to-pyav-migration.md b/docs/plans/2026-04-13-opencv-to-pyav-migration.md new file mode 100644 index 0000000..bee9be2 --- /dev/null +++ b/docs/plans/2026-04-13-opencv-to-pyav-migration.md @@ -0,0 +1,1468 @@ +# OpenCV → PyAV Migration Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace OpenCV (`cv2`) with PyAV (`av`) across syncfield-python's video capture / encoding paths so the SDK gains UVC format negotiation (e.g. 720p request), hardware-accelerated H.264 encoding (VideoToolbox on macOS), and a lighter runtime footprint — while preserving every existing Stream SPI contract (`prepare → connect → start_recording → stop_recording → disconnect`, legacy `start/stop`, `latest_frame`, `FinalizationReport`) and the host-monotonic timestamping policy (`time.monotonic_ns()` immediately after the frame is read). + +**Architecture:** Introduce a shared internal `VideoEncoder` module that wraps PyAV's output container / H.264 encoder (auto-selecting `h264_videotoolbox` on macOS, falling back to `libx264`). `UVCWebcamStream` replaces `cv2.VideoCapture` with a PyAV input container (`avfoundation` / `v4l2` / `dshow`) and uses the shared encoder for MP4 output. `OakCameraStream` keeps its DepthAI capture path and only swaps `cv2.VideoWriter` for the shared encoder. The viewer's MJPEG endpoint swaps `cv2.imencode` for Pillow. Frames remain in numpy BGR so the existing `latest_frame` contract and viewer both stay untouched at the boundary. Timestamps stay `time.monotonic_ns()` — PTS is intentionally **not** adopted (see `docs/` conversation history for the rationale). + +**Tech Stack:** +- `av` (PyAV ≥ 12.0) — video I/O, demux, encode +- `Pillow` (PIL ≥ 10.0) — single-frame JPEG encoding in the viewer +- `numpy` — frame buffers (already a transitive dep) +- `depthai` — unchanged for OAK capture +- `pytest` — unit tests with `sys.modules` mocking for `av` and `PIL` + +--- + +## File Structure + +**New files:** +- `src/syncfield/adapters/_video_encoder.py` — shared PyAV H.264 MP4 writer used by UVC and OAK. One responsibility: "accept BGR numpy frames at a fixed rate, produce a playable MP4." +- `tests/unit/adapters/test_video_encoder.py` — unit tests for the encoder module using a fake `av` module. + +**Modified files:** +- `src/syncfield/adapters/uvc_webcam.py` — swap `cv2.VideoCapture` for PyAV input container; swap writer for `VideoEncoder`; drop the `cv2` import. +- `src/syncfield/adapters/oak_camera.py` — swap `cv2.VideoWriter` for `VideoEncoder`; drop the `cv2` import. DepthAI capture path is unchanged. +- `src/syncfield/viewer/server.py` — swap `cv2.imencode(".jpg", …)` for `PIL.Image.save(…, format="JPEG")`; drop the `cv2` import. +- `src/syncfield/types.py` — add optional `jitter_p95_ns` / `jitter_p99_ns` fields to `FinalizationReport` (both default `None` so unchanged for non-video streams). +- `tests/unit/adapters/test_uvc_webcam.py` — replace `mock_cv2` fixture with `mock_av` fixture. +- `tests/unit/adapters/test_oak_camera.py` — replace the writer-related `cv2` mocks with `mock_av` fixture. DepthAI mock unchanged. +- `tests/unit/viewer/test_cluster_endpoints.py` — update any `cv2.imencode` mock to `PIL.Image` (only if present). +- `pyproject.toml` — remove `opencv-python`, add `av` and `Pillow` to the relevant extras. + +**Rationale for the split:** UVC and OAK both need an "MP4 writer that accepts numpy BGR frames and is hardware-accelerated on macOS." Duplicating that across two adapters would invite divergence. A single `VideoEncoder` with a clear interface (`open(path, width, height, fps) → encoder`; `encoder.write(frame_bgr)`; `encoder.close() → None`) isolates the PyAV detail and makes both adapters one-liners at the write site. The module lives under `adapters/` (not a top-level `video/`) because nothing outside adapter internals should depend on it. + +--- + +## Task 0: Create the worktree and verify baseline + +**Files:** +- Read: `pyproject.toml` (verify baseline test pass) + +- [ ] **Step 1: Create an isolated worktree for the migration** + +```bash +cd /Users/jerry/Documents/syncfield-python +git worktree add ../syncfield-python-pyav -b feat/pyav-migration +cd ../syncfield-python-pyav +``` + +- [ ] **Step 2: Verify the baseline test suite passes on `main`** + +```bash +uv sync --all-extras +uv run pytest tests/unit -x -q +``` + +Expected: all tests pass. Record the pass count (it's the floor we must preserve). + +- [ ] **Step 3: Commit the worktree marker (no file changes)** + +Skip — the worktree exists on the branch with no changes yet. + +--- + +## Task 1: Add `av` and `Pillow` dependencies + +**Files:** +- Modify: `pyproject.toml` (extras `uvc`, `oak`, `viewer`) + +- [ ] **Step 1: Inspect the current extras** + +Run: `grep -n -A2 '^\[project.optional-dependencies\]' pyproject.toml` + +Expected to see (roughly): +``` +uvc = ["opencv-python>=4.5"] +oak = ["depthai>=3.0.0"] +viewer = ["opencv-python>=4.8.0", ...] +``` + +- [ ] **Step 2: Modify extras — replace `opencv-python` with `av` + `Pillow`** + +Exact edits in `pyproject.toml`: + +```toml +[project.optional-dependencies] +uvc = ["av>=12.0.0"] +oak = ["depthai>=3.0.0", "av>=12.0.0"] +viewer = [ + "av>=12.0.0", + "Pillow>=10.0.0", + # ...keep the other viewer deps (fastapi, uvicorn, etc.) exactly as they were +] +``` + +Also update `all = [...]` to union these (no `opencv-python` anywhere). + +- [ ] **Step 3: Resolve and install** + +```bash +uv sync --all-extras +``` + +Expected: `opencv-python` is gone from `uv.lock`; `av` and `Pillow` are added. + +- [ ] **Step 4: Verify `av` imports and an encoder exists** + +```bash +uv run python -c "import av; c = av.codec.Codec('h264', 'w'); print(c.name, c.long_name)" +``` + +Expected: `h264 ...` printed without error. + +- [ ] **Step 5: Verify VideoToolbox is available on macOS (informational)** + +```bash +uv run python -c "import av; print(av.codec.Codec('h264_videotoolbox', 'w').name)" +``` + +On Apple Silicon: prints `h264_videotoolbox`. On other platforms: raises — expected, we fall back to `libx264`. + +- [ ] **Step 6: Commit** + +```bash +git add pyproject.toml uv.lock +git commit -m "build: replace opencv-python with av + Pillow in extras" +``` + +--- + +## Task 2: Introduce `VideoEncoder` — interface + failing test + +**Files:** +- Create: `src/syncfield/adapters/_video_encoder.py` +- Create: `tests/unit/adapters/test_video_encoder.py` + +- [ ] **Step 1: Write the failing contract test** + +Create `tests/unit/adapters/test_video_encoder.py`: + +```python +"""Unit tests for VideoEncoder — the shared PyAV MP4 writer. + +The real ``av`` module is replaced with a fake in ``conftest.py`` style so +these tests do not depend on a working FFmpeg build. We only assert that: + +* The encoder opens an output container with the right path and format. +* It adds one video stream with the requested width, height, fps, pixel + format and a usable codec (h264_videotoolbox on mac, libx264 elsewhere). +* ``write(frame)`` encodes and muxes one packet per call. +* ``close()`` flushes the encoder and closes the container exactly once. +* Double ``close()`` is idempotent (no double-flush, no exception). +""" + +from __future__ import annotations + +import sys +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import MagicMock + +import numpy as np +import pytest + + +@pytest.fixture +def fake_av(monkeypatch: pytest.MonkeyPatch) -> SimpleNamespace: + """Install a fake ``av`` module that records every interaction.""" + container = MagicMock(name="OutputContainer") + stream = MagicMock(name="VideoStream") + stream.width = 0 + stream.height = 0 + stream.pix_fmt = "yuv420p" + container.add_stream.return_value = stream + + # ``encode`` returns a list of packets; we fake one packet per call. + packet = MagicMock(name="Packet") + stream.encode.return_value = [packet] + + av = SimpleNamespace() + av.open = MagicMock(name="av.open", return_value=container) + + # ``av.VideoFrame.from_ndarray`` returns a fake frame carrying the array. + def _from_ndarray(arr: np.ndarray, format: str) -> MagicMock: + frame = MagicMock(name="VideoFrame") + frame.to_ndarray = lambda format="bgr24": arr + frame._source_format = format + return frame + + video_frame = SimpleNamespace(from_ndarray=MagicMock(side_effect=_from_ndarray)) + av.VideoFrame = video_frame + + # ``av.codec.Codec`` returns an object if the codec exists, raises if not. + def _codec(name: str, mode: str) -> SimpleNamespace: + if name in {"h264_videotoolbox", "libx264"}: + return SimpleNamespace(name=name) + raise ValueError(f"unknown codec {name}") + + av.codec = SimpleNamespace(Codec=MagicMock(side_effect=_codec)) + + monkeypatch.setitem(sys.modules, "av", av) + return SimpleNamespace(av=av, container=container, stream=stream, packet=packet) + + +def test_open_creates_output_container(tmp_path: Path, fake_av: SimpleNamespace) -> None: + from syncfield.adapters._video_encoder import VideoEncoder + + out = tmp_path / "clip.mp4" + enc = VideoEncoder.open(out, width=1280, height=720, fps=30.0) + + fake_av.av.open.assert_called_once() + args, kwargs = fake_av.av.open.call_args + assert args[0] == str(out) + assert kwargs.get("mode") == "w" + + fake_av.container.add_stream.assert_called_once() + stream_args, stream_kwargs = fake_av.container.add_stream.call_args + # codec preference: h264_videotoolbox (macOS) or libx264 (fallback) + assert stream_args[0] in {"h264_videotoolbox", "libx264"} + assert stream_kwargs.get("rate") == 30 + assert fake_av.stream.width == 1280 + assert fake_av.stream.height == 720 + + enc.close() + + +def test_write_encodes_and_muxes_one_frame(tmp_path: Path, fake_av: SimpleNamespace) -> None: + from syncfield.adapters._video_encoder import VideoEncoder + + enc = VideoEncoder.open(tmp_path / "clip.mp4", width=64, height=48, fps=30.0) + frame = np.zeros((48, 64, 3), dtype=np.uint8) + + enc.write(frame) + + fake_av.av.VideoFrame.from_ndarray.assert_called_once() + fake_av.stream.encode.assert_called() + fake_av.container.mux.assert_called_with(fake_av.packet) + + enc.close() + + +def test_close_is_idempotent(tmp_path: Path, fake_av: SimpleNamespace) -> None: + from syncfield.adapters._video_encoder import VideoEncoder + + enc = VideoEncoder.open(tmp_path / "clip.mp4", width=64, height=48, fps=30.0) + enc.close() + enc.close() # should not raise, should not double-close + + assert fake_av.container.close.call_count == 1 +``` + +- [ ] **Step 2: Run — expect failure (module missing)** + +```bash +uv run pytest tests/unit/adapters/test_video_encoder.py -v +``` + +Expected: `ModuleNotFoundError: No module named 'syncfield.adapters._video_encoder'`. + +- [ ] **Step 3: Implement `VideoEncoder`** + +Create `src/syncfield/adapters/_video_encoder.py`: + +```python +"""VideoEncoder — shared PyAV-based MP4 writer for video adapters. + +Used by :class:`~syncfield.adapters.uvc_webcam.UVCWebcamStream` and +:class:`~syncfield.adapters.oak_camera.OakCameraStream`. The interface is +deliberately narrow: open with geometry, write BGR numpy frames, close. + +The encoder auto-selects the best available H.264 encoder: +* ``h264_videotoolbox`` on macOS (hardware, near-zero CPU) +* ``libx264`` everywhere else (software, widely available) + +All frames are assumed to be BGR24 (numpy ``uint8``, shape +``(height, width, 3)``) to match the rest of the SDK's convention. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Optional + +import numpy as np + +try: + import av # type: ignore[import-not-found] +except ImportError as exc: # pragma: no cover - exercised via sys.modules patch + raise ImportError( + "syncfield video adapters require PyAV. " + "Install with `pip install syncfield[uvc]` (or [oak], [viewer])." + ) from exc + + +def _pick_h264_encoder() -> str: + """Return the best H.264 encoder name available in this FFmpeg build.""" + for candidate in ("h264_videotoolbox", "libx264"): + try: + av.codec.Codec(candidate, "w") + except Exception: # noqa: BLE001 - PyAV raises generic errors here + continue + return candidate + raise RuntimeError( + "No H.264 encoder found in PyAV. Reinstall `av` with libx264 support." + ) + + +class VideoEncoder: + """Thin wrapper around an ``av`` output container + H.264 stream.""" + + def __init__( + self, + container: "av.container.OutputContainer", + stream: "av.video.stream.VideoStream", + ) -> None: + self._container = container + self._stream = stream + self._closed = False + + @classmethod + def open( + cls, + path: str | Path, + *, + width: int, + height: int, + fps: float, + codec: Optional[str] = None, + pixel_format: str = "yuv420p", + ) -> "VideoEncoder": + """Open ``path`` for writing and configure the H.264 stream.""" + chosen_codec = codec or _pick_h264_encoder() + container = av.open(str(path), mode="w") + stream = container.add_stream(chosen_codec, rate=int(round(fps))) + stream.width = int(width) + stream.height = int(height) + stream.pix_fmt = pixel_format + return cls(container, stream) + + def write(self, frame_bgr: np.ndarray) -> None: + """Encode and mux a single BGR frame. + + Must not be called after :meth:`close`. Callers that interleave + writes with other hot-path work should keep the frame buffer + alive until this call returns. + """ + if self._closed: + raise RuntimeError("VideoEncoder.write called after close") + video_frame = av.VideoFrame.from_ndarray(frame_bgr, format="bgr24") + for packet in self._stream.encode(video_frame): + self._container.mux(packet) + + def close(self) -> None: + """Flush the encoder and close the container. Idempotent.""" + if self._closed: + return + self._closed = True + # Flush: passing None drains any remaining packets in the encoder. + try: + for packet in self._stream.encode(None): + self._container.mux(packet) + finally: + self._container.close() +``` + +- [ ] **Step 4: Run the test — expect pass** + +```bash +uv run pytest tests/unit/adapters/test_video_encoder.py -v +``` + +Expected: 3 passed. + +- [ ] **Step 5: Commit** + +```bash +git add src/syncfield/adapters/_video_encoder.py tests/unit/adapters/test_video_encoder.py +git commit -m "feat(adapters): add VideoEncoder shared PyAV MP4 writer" +``` + +--- + +## Task 3: Build `_open_capture` platform helper — UVC input via PyAV + +**Files:** +- Modify: `src/syncfield/adapters/_video_encoder.py` — add a sibling helper for input-side platform dispatch (keeping video I/O internals in one module). +- Modify: `tests/unit/adapters/test_video_encoder.py` — add tests for the helper. + +Rationale: opening a UVC device with PyAV is platform-dependent (macOS uses `avfoundation`, Linux `v4l2`, Windows `dshow`). That logic is small but non-trivial; keeping it next to the encoder keeps the "video I/O primitives" in one place. + +- [ ] **Step 1: Add failing tests for `open_uvc_input`** + +Append to `tests/unit/adapters/test_video_encoder.py`: + +```python +def test_open_uvc_input_macos(monkeypatch: pytest.MonkeyPatch, fake_av: SimpleNamespace) -> None: + from syncfield.adapters import _video_encoder + + monkeypatch.setattr(_video_encoder.sys, "platform", "darwin") + input_container = MagicMock(name="InputContainer") + fake_av.av.open.return_value = input_container + + result = _video_encoder.open_uvc_input( + device_index=0, width=1280, height=720, fps=30.0 + ) + + args, kwargs = fake_av.av.open.call_args + assert args[0] == "0:none" # avfoundation URL: "