Skip to content

feat: migrate video capture/encoding from OpenCV to PyAV - #6

Merged
styu12 merged 18 commits into
mainfrom
feat/pyav-migration
Apr 14, 2026
Merged

feat: migrate video capture/encoding from OpenCV to PyAV#6
styu12 merged 18 commits into
mainfrom
feat/pyav-migration

Conversation

@styu12

@styu12 styu12 commented Apr 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Migrates syncfield-python's video capture, encoding, and MJPEG serving paths from OpenCV (cv2) to PyAV (av), while preserving the Stream SPI contract exactly.

  • UVC adapter: cv2.VideoCapture → PyAV platform dispatch (avfoundation/v4l2/dshow); cv2.VideoWriter → shared VideoEncoder (H.264 via VideoToolbox on Apple Silicon, libx264 fallback). Defaults to 1280×720@30 — fixes a silent-ignore bug on main where cv2.set(CAP_PROP_FRAME_WIDTH) was never called.
  • OAK adapter: DepthAI capture path unchanged; only the MP4 writer swapped to VideoEncoder. Depth .bin write path untouched.
  • Viewer: cv2.imencode → Pillow for MJPEG endpoint.
  • Metrics: FinalizationReport gains optional jitter_p95_ns / jitter_p99_ns fields, populated from inter-frame intervals during the recording window (≥20 sample threshold).

Timestamp strategy intentionally unchanged: time.monotonic_ns() taken immediately after each decoded frame. PTS NOT adopted — design rationale in the plan doc.

Architecture

  • New src/syncfield/adapters/_video_encoder.py — shared PyAV MP4 writer + open_uvc_input() platform dispatch + compute_jitter_percentiles() helper.
  • Public Stream SPI (constructor, 4-phase lifecycle, legacy start/stop, latest_frame, FinalizationReport shape) preserved verbatim.
  • pyproject.toml: removed opencv-python; added av>=12.0.0 and Pillow>=10.0.0 to extras.
  • Tests: shared mock_av fixture hoisted to tests/unit/adapters/conftest.py; UVC and OAK tests migrated; PytestUnhandledThreadExceptionWarning leaks cleaned.

Also hardened on the way:

  • _capture_loop emits a HealthEvent(kind=ERROR) on decode error (was silent on main).
  • Per-recording counter reset in start_recording so back-to-back recordings in one session produce clean reports.
  • VideoEncoder.close() idempotent even when flush raises.
  • Jitter collection gated on _recording flag — no race with stop_recording, no preview-phase pollution.

Files changed

  • src/syncfield/adapters/_video_encoder.py (new, ~180 lines)
  • src/syncfield/adapters/uvc_webcam.py (rewritten internals)
  • src/syncfield/adapters/oak_camera.py (writer swapped)
  • src/syncfield/viewer/server.py (Pillow JPEG)
  • src/syncfield/types.py (+ jitter fields)
  • src/syncfield/adapters/__init__.py, src/syncfield/viewer/__init__.py, src/syncfield/discovery/types.py (docstring cleanup)
  • pyproject.toml + uv.lock
  • tests/unit/adapters/conftest.py (new)
  • tests/unit/adapters/test_video_encoder.py (new, 10 tests)
  • tests/unit/adapters/test_uvc_webcam.py, test_oak_camera.py, test_discover_* (migrated)
  • tests/unit/viewer/test_cluster_endpoints.py (cv2 stub removed)
  • tests/unit/test_finalization_jitter.py (new, 2 tests)
  • examples/*/README.md (doc updates)
  • docs/plans/2026-04-13-opencv-to-pyav-migration.md (plan)
  • docs/plans/2026-04-13-opencv-to-pyav-migration-hw-smoketest.md (HW smoke-test checklist)

Test plan

  • Full unit suite: 646 passed, 15 pre-existing failures (orchestrator/writer — identical on main)
  • Migration-scope tests: 18 encoder + 9 UVC + 9 OAK + 14 discovery + 58 viewer + 2 jitter dataclass + 1 UVC jitter-lifecycle = 111 passing
  • Zero cv2 / opencv references in src/ or tests/
  • Zero PytestUnhandledThreadExceptionWarning
  • Hardware smoke test per docs/plans/2026-04-13-opencv-to-pyav-migration-hw-smoketest.md — requires real M3/M4 + cameras, to be run by reviewer

Notes for reviewer

  • The 15 pre-existing test failures are all on main as well (verified via git stash); they are in test_orchestrator.py and test_writer.py and appear to be a version-string and manifest-path issue unrelated to this migration.
  • The pace_seconds parameter in the test fixtures is a pragmatic wall-clock workaround noted as TODO(test-harness) — a deterministic event-driven pump would be cleaner but the current pacing is stable. Revisit if CI flakes appear.
  • 17 granular commits preserved for reviewability (one per logical step), not squashed. Squash on merge is fine.

🤖 Generated with Claude Code

styu12 and others added 18 commits April 13, 2026 17:43
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Replace cv2.VideoCapture with open_uvc_input() platform dispatch
- Replace cv2.VideoWriter with shared VideoEncoder
- Default resolution 1280x720@30 (enables request at UVC level)
- Timestamp strategy unchanged (time.monotonic_ns after decode)
- Discover helpers preserved verbatim
- Existing UVC tests now fail; Task 5 migrates them

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Remove cv2 import block
- Use shared VideoEncoder.open() instead of cv2.VideoWriter
- Reset per-recording counters on start_recording (same as UVC)
- .release() -> .close() in _release_writers
- DepthAI capture path and depth .bin write unchanged
- OAK tests now fail on cv2 mock; Task 7 migrates them

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Prevents two issues:
- Preview-phase intervals no longer poison p95/p99 on short recordings
- Capture thread no longer mutates _intervals_ns while stop_recording
  is iterating it via sorted() in compute_jitter_percentiles

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@styu12
styu12 force-pushed the feat/pyav-migration branch from b249ac1 to 94a33a2 Compare April 14, 2026 01:08
@styu12

styu12 commented Apr 14, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto latest origin/main (brought in 10 new commits: multihost resolver fixes, orchestrator __init__ refactor, meta_quest health events).

Conflict resolution:

  • pyproject.toml: auto-merged cleanly (main's version bump + our PyAV extras).
  • uv.lock: took main's base, re-ran uv sync --all-extras to regenerate with PyAV deps. New test deps (pytest-timeout, pytest-xdist) from main came through.

Post-rebase verification: migration-scope tests all pass — 240 tests across tests/unit/adapters/, tests/unit/viewer/, and tests/unit/test_finalization_jitter.py (1 deselected: pre-existing sounddevice env failure).

The 18 migration commits were all rebased successfully with no logic conflicts; only the uv.lock needed regeneration.

@styu12
styu12 merged commit f5ffe36 into main Apr 14, 2026
0 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant