feat(examples): Meta Quest 3 egocentric recording + 3D hand pose viewer - #12
Merged
Conversation
…ample New example `examples/meta_quest/` pairs with the newly-landed MetaQuestCameraStream (PR #10) and the MetaQuestHandStream that's been on main since PR #7. Captures, in one synchronised session: - ``quest_tracking`` (UDP 14043, 72 Hz) — 26-joint hand skeleton × 2 hands + head pose, all timestamped in the ``remote_quest3`` clock domain. - ``quest_cam`` (HTTP 14045) — 720p stereo passthrough MJPEG recorded on the Quest and pulled to output_dir after stop. Viewer extensions to render the hand skeleton in 3-D: 1. State / poller / SSE backend now passes vector-valued channels (e.g. ``hand_joints``: 156 floats) through to the frontend under a new ``pose`` field on each SensorEvent. ``StreamStatsBuffer`` retains only the latest sample per list channel — 3-D pose panels render instantaneous state, buffering 300 × 500-float poses per adapter would blow memory. 2. New ``Quest3PosePanel`` component (React Three Fiber), ported from opengraph-studio/showcase with egocentric framing and the viewer's warm-paper palette. Draws the 26-joint skeleton per hand (cyan / orange) plus an RGB-axis head rig inside the same 0.84-m workspace box used by the showcase. 3. ``SensorPanel`` dispatcher routes streams with ``hand_joints`` payloads to the new 3-D panel; roll/pitch/yaw IMUs still go to the existing cube panel; everything else falls back to the multi-channel line chart. All existing adapters keep their rendering unchanged. 4. Frontend deps: ``three``, ``@react-three/fiber``, ``@react-three/drei``, ``@types/three``. Bundle grows ~1 MB gz which is acceptable for a desktop-launched viewer. Backward compatibility: adapters that emit only scalar channels never populate ``latest_pose``, ``pose`` on the SSE event stays ``null``, and the hook / dispatcher fall through to the existing chart path — zero behaviour change for non-Quest streams. Co-Authored-By: Claude Opus 4.6 <[email protected]>
# Conflicts: # src/syncfield/viewer/frontend/src/components/sensor-panel.tsx # src/syncfield/viewer/server.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
새 예제
examples/meta_quest/— MetaQuestHandStream + MetaQuestCameraStream 조합으로 Quest 3 하나에서 head pose / 양손 skeleton / 스테레오 egocentric 카메라를 동기화된 단일 세션으로 기록. Viewer 에 3-D hand pose 패널 추가해 opengraph-studio/showcase 와 동일한 스타일의 실시간 시각화 제공.What the example does
모든 샘플이 `clock_domain="remote_quest3"` + `uncertainty_ns=10ms` 로 통일 → Mac-host 센서들과 standard sync 파이프라인으로 정렬.
Viewer 확장
`hand_joints` (156 floats) / `head_pose` (7 floats) 같은 vector-valued 채널을 viewer 가 이제 처리:
Backend (Python)
Frontend (React + Three.js)
Dependencies 추가: `three`, `@react-three/fiber`, `@react-three/drei`, `@types/three`. 번들 ~1MB gz 증가 (desktop viewer 라 수용 가능).
호환성
기존 scalar-only 어댑터는 `pose=null` 로 스트림 → hook/dispatcher 모두 기존 chart 경로로 폴백 → 행동 변화 0.
84개 unit 테스트 통과 (`pytest tests/unit/viewer tests/unit/adapters/test_meta_quest.py`).
스크린샷 참고
Panel 레이아웃/색상 tone:
Test Plan
Out of scope
🤖 Generated with Claude Code