Skip to content

feat(examples): Meta Quest 3 egocentric recording + 3D hand pose viewer - #12

Merged
styu12 merged 2 commits into
mainfrom
feat/meta-quest-example
Apr 15, 2026
Merged

feat(examples): Meta Quest 3 egocentric recording + 3D hand pose viewer#12
styu12 merged 2 commits into
mainfrom
feat/meta-quest-example

Conversation

@styu12

@styu12 styu12 commented Apr 15, 2026

Copy link
Copy Markdown
Contributor

Summary

새 예제 examples/meta_quest/ — MetaQuestHandStream + MetaQuestCameraStream 조합으로 Quest 3 하나에서 head pose / 양손 skeleton / 스테레오 egocentric 카메라를 동기화된 단일 세션으로 기록. Viewer 에 3-D hand pose 패널 추가해 opengraph-studio/showcase 와 동일한 스타일의 실시간 시각화 제공.

What the example does

Stream Data Rate Transport
`quest_tracking` 26-joint hand skeleton × 2 + head pose (pos3+quat4) 72 Hz UDP :14043
`quest_cam` (left/right) 720p stereo passthrough MJPEG-AVI 30 fps HTTP :14045

모든 샘플이 `clock_domain="remote_quest3"` + `uncertainty_ns=10ms` 로 통일 → Mac-host 센서들과 standard sync 파이프라인으로 정렬.

Viewer 확장

`hand_joints` (156 floats) / `head_pose` (7 floats) 같은 vector-valued 채널을 viewer 가 이제 처리:

  1. Backend (Python)

    • `StreamStatsBuffer` 에 `_latest_pose: Dict[str, List[float]]` 추가 — 최신 1개만 retain (300-sample rolling 은 메모리 폭발)
    • `StreamSnapshot` 에 `latest_pose` 필드 추가
    • SSE endpoint 가 `{channels, pose, label}` 로 전송 (pose=null 이면 기존 scalar-only 어댑터와 호환)
  2. Frontend (React + Three.js)

    • `quest3-pose-panel.tsx` 신설 — R3F 로 cyan/orange hand skeleton + RGB-axis head rig 렌더링, OrbitControls 로 자유 시점
    • `use-sensor-stream.ts` 에 `pose` 스냅샷 state 추가 (버퍼링 안함 — 순간 pose 렌더링용)
    • `sensor-panel.tsx` dispatcher 가 `hand_joints` 감지 시 새 패널로 라우팅
  3. Dependencies 추가: `three`, `@react-three/fiber`, `@react-three/drei`, `@types/three`. 번들 ~1MB gz 증가 (desktop viewer 라 수용 가능).

호환성

기존 scalar-only 어댑터는 `pose=null` 로 스트림 → hook/dispatcher 모두 기존 chart 경로로 폴백 → 행동 변화 0.

84개 unit 테스트 통과 (`pytest tests/unit/viewer tests/unit/adapters/test_meta_quest.py`).

스크린샷 참고

Panel 레이아웃/색상 tone:

  • Cyan 왼손, 오렌지 오른손 (showcase 동일)
  • 머리 RGB 좌표축 (R=X, G=Y, B=Z)
  • 0.84m workspace 박스 + 그리드 바닥

Test Plan

  • `yarn build` — frontend 타입체크 + 빌드 성공
  • `pytest tests/unit/viewer tests/unit/adapters/test_meta_quest.py -q` — 84 passed
  • 실제 Quest 3 + Unity sender 앱 실행 → example 로 end-to-end 녹화 검증

Out of scope

  • 실제 Quest 3 하드웨어 검증은 별도로 사용자가 실행 필요 (`QUEST_IP` 수동 설정)
  • Camera discovery 자동화 (Phase 2)
  • Phase 2 H.264 인코딩 (현재 MJPEG-AVI)

🤖 Generated with Claude Code

styu12 and others added 2 commits April 14, 2026 17:24
…ample

New example `examples/meta_quest/` pairs with the newly-landed
MetaQuestCameraStream (PR #10) and the MetaQuestHandStream that's
been on main since PR #7. Captures, in one synchronised session:

  - ``quest_tracking``  (UDP 14043, 72 Hz) — 26-joint hand skeleton
    × 2 hands + head pose, all timestamped in the ``remote_quest3``
    clock domain.
  - ``quest_cam``       (HTTP 14045) — 720p stereo passthrough MJPEG
    recorded on the Quest and pulled to output_dir after stop.

Viewer extensions to render the hand skeleton in 3-D:

  1. State / poller / SSE backend now passes vector-valued channels
     (e.g. ``hand_joints``: 156 floats) through to the frontend under
     a new ``pose`` field on each SensorEvent. ``StreamStatsBuffer``
     retains only the latest sample per list channel — 3-D pose
     panels render instantaneous state, buffering 300 × 500-float
     poses per adapter would blow memory.
  2. New ``Quest3PosePanel`` component (React Three Fiber), ported
     from opengraph-studio/showcase with egocentric framing and the
     viewer's warm-paper palette. Draws the 26-joint skeleton per
     hand (cyan / orange) plus an RGB-axis head rig inside the same
     0.84-m workspace box used by the showcase.
  3. ``SensorPanel`` dispatcher routes streams with ``hand_joints``
     payloads to the new 3-D panel; roll/pitch/yaw IMUs still go to
     the existing cube panel; everything else falls back to the
     multi-channel line chart. All existing adapters keep their
     rendering unchanged.
  4. Frontend deps: ``three``, ``@react-three/fiber``,
     ``@react-three/drei``, ``@types/three``. Bundle grows ~1 MB gz
     which is acceptable for a desktop-launched viewer.

Backward compatibility: adapters that emit only scalar channels
never populate ``latest_pose``, ``pose`` on the SSE event stays
``null``, and the hook / dispatcher fall through to the existing
chart path — zero behaviour change for non-Quest streams.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
# Conflicts:
#	src/syncfield/viewer/frontend/src/components/sensor-panel.tsx
#	src/syncfield/viewer/server.py
@styu12
styu12 merged commit ab05f5e into main Apr 15, 2026
0 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant