fix(meta_quest): direct-push tracking target + camera fixes - #17
Merged
Conversation
…view Three on-device fixes after live debugging with a Quest 3 on the same WiFi as the Mac recorder: 1. **Tracking auto-discovery never converged.** UDP :14044 broadcast probes from Quest weren't reaching Mac on this AP — tcpdump caught zero packets across 10s windows. Discovery is fundamentally unreliable on networks with client isolation, IPv6-only segments, etc., so add a deterministic alternative: MetaQuestHandStream now accepts a quest_host kwarg and POSTs the Mac's IP to the Quest's CameraHttpServer /tracker/target endpoint on connect(). The Quest sender unicasts directly back, no broadcast required. 2. **Camera preview was upside-down.** Quest's passthrough frames use OpenGL's Y-up texture origin; the viewer was rendering them flipped. Add a single FLIP_TOP_BOTTOM in _decode_jpeg. 3. **Composite stereo panel: right eye stuttered.** ARFoundation's TryAcquireLatestCpuImage hands one image per camera-frame event — asking for it twice in a row (once per eye) means the second call usually returns false, so right preview only updated on coincident frames. Companion app fix lands separately; on the Python side, make latest_frame fall back to whichever eye is currently fresh instead of going dark when the pair desyncs. Also: collapse MJPEG retry log spam (first failure → WARN, subsequent → INFO) with exponential backoff up to 8s — a Quest app restart no longer floods 60+ identical log lines. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The viewer's quest_tracking panel sat on "Waiting for data..." even after the orchestrator confirmed the stream was Connected and the adapter had pushed the "First Quest 3 packet received" health event. Cause: _process_packet only called _emit_sample inside `if self._recording:`, so samples never flowed into the viewer's state buffer (latest_pose / plot_points) until the user pressed Record. Other adapters (UVC, BLE) emit during preview as well — that's how the viewer's video / pose panels stay live before recording starts. Always emit the SampleEvent and increment frame_count; keep first/ last_at gated by _recording since they describe the recorded segment boundary, not the live preview. start_recording resets frame_count to 0, so the FinalizationReport count is unchanged. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The TypeScript build was failing with TS2339 errors on every R3F intrinsic (mesh, group, sphereGeometry, ambientLight, ...) because @react-three/fiber's JSX namespace augmentation wasn't loaded by default in this tsconfig. Add a triple-slash reference at the top of quest3-pose-panel.tsx to pull in the ThreeElements global. Without this, `yarn build` errors out and the served bundle stays at the pre-Quest3 snapshot — Quest tracking samples flow over SSE but the dispatcher never routes them to Quest3PosePanel, so the panel sits on "Waiting for data..." even with 60Hz packets arriving. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
… is legible The default camera at [0.5, 0.35, 0.5] with FOV 50 made the workspace box dominate the panel and shrank the head + hand skeleton to a postage stamp at the centre. Move the camera in to [0.28, 0.22, 0.32] @ FOV 55, tighten OrbitControls maxDistance from 2 to 1.2, and bump the joint marker / head axis multipliers so the skeleton reads from across the room. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three on-device fixes after live debugging with a Quest 3 on the same WiFi as the Mac recorder:
1. Tracking auto-discovery never converged → direct push
UDP :14044 broadcast probes from Quest weren't reaching Mac on this AP —
tcpdumpcaught zero packets across 10s windows. Discovery is fundamentally unreliable on networks with client isolation, IPv6-only segments, etc.Add a deterministic alternative:
MetaQuestHandStreamaccepts aquest_hostkwarg and POSTs the Mac's IP to the Quest companion app'sCameraHttpServer/tracker/targetendpoint onconnect(). The Quest sender unicasts directly back, no broadcast required.2. Camera preview was upside-down
Quest's passthrough frames use OpenGL's Y-up texture origin; the viewer rendered them flipped. One-line
FLIP_TOP_BOTTOMin_decode_jpeg.3. Composite stereo: graceful single-eye fallback + retry-spam mitigation
latest_framefalls back to whichever eye is fresh when the pair desyncs (rather than going dark).Test Plan
pytest tests/unit/adapters/test_meta_quest.py tests/unit/adapters/meta_quest_camera/)/tracker/targetwith Mac IP → Quest logcat shows[SyncField] Target endpoint pushed → 192.168.4.35:14043🤖 Generated with Claude Code