feat(viewer): web viewer with sync trigger and episode review - #5
Merged
Conversation
Replace DearPyGui desktop GUI with browser-based web viewer using FastAPI + Vite + React + shadcn/ui + Tailwind. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
… viewer Replace the GPU-accelerated DearPyGui desktop GUI with a FastAPI + React web viewer. The Python SDK starts a uvicorn server and opens a browser tab. The public API (viewer.launch / viewer.launch_passive) is preserved with the same signatures plus new optional host/port parameters. Backend: FastAPI server with WebSocket (10Hz snapshot broadcast + control), MJPEG video streaming, SSE sensor data push, and REST discovery endpoints. Frontend: Vite + React + Tailwind CSS, porting the Recorder design system (Inter font, teal primary, warm gray palette). Components include video preview, real-time SVG sensor charts, health table, countdown overlay, and device discovery modal. Dependencies changed: dearpygui removed, fastapi + uvicorn + opencv-python added. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…ication - Stream cards: responsive grid layout (min 320px, fills available width), aspect-ratio video previews instead of fixed 146px height - Health events: moved to compact sidebar (272px) instead of full-width section below streams - Countdown audio: removed browser Web Audio playback entirely — the recording PC handles all audio (ticks + chirps) via sounddevice. Server now delegates to session.start(on_countdown_tick=...) so the session's native countdown drives both audio and visual overlay, eliminating the double 3-2-1 and double chirp Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…ing glow - Header: replace "SyncField" text with OpenGraph Labs SVG logo - Header: red-tinted background during recording (matching Recorder style) - App: inset red border glow during recording state - Footer: show full output path with folder icon, live wall clock - Stream cards: remove "ms ago" latency display (viewer-only metric that misleadingly suggests recording latency) - Discovery: fix broken API — use syncfield.discovery.scan() instead of non-existent discover_devices(). Show device description, in_use, and warnings from the real DiscoveryReport - Remove "Reconnecting..." banner — explicit connect/disconnect buttons make auto-reconnect UI unnecessary Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Replace raw state strings with friendly labels: - idle/connected/stopped → "Ready" - recording → "Recording" - stopping → "Saving…" - countdown/preparing → "Starting…" Remove redundant State display from session info bar. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
SessionOrchestrator now auto-creates a timestamped episode subdirectory
(ep_{YYYYMMDD}_{HHMMSS}_{hex}) inside the provided output_dir. Callers
just pass the data root — no more boilerplate for episode dir creation.
session.output_dir returns the episode-specific path so streams can
reference it directly: session.add(UVCWebcamStream("cam", 0, session.output_dir))
Examples simplified from ~40 lines to ~20 lines each:
- Removed argparse boilerplate for device indices (inline constants)
- Removed manual episode dir creation (SDK handles it)
- Removed datetime/secrets imports
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Add a complete Review system alongside the existing Recording view,
accessible via a header segment control (Record | Review).
Review - Episode List:
- Grid and Table view toggle for browsing recorded episodes
- Each episode shows sync status, stream count, host ID, timestamp
- Server scans output directory for ep_* folders automatically
Review - Episode Detail:
- Multi-camera synchronized video playback with timeline scrubber
- Sync quality sidebar (grade, confidence, per-stream offsets)
- Before/After drift chart from frame_map.jsonl
- Drift badges on secondary video streams
Sync Trigger:
- One-click sync via local Docker container (localhost:8080)
- Progress bar with phase/percentage during processing
- Auto-refreshes sync report on completion
Backend (server.py):
- GET /api/episodes — list episodes from data root
- GET /api/episodes/{id} — manifest + sync_report
- GET /api/episodes/{id}/video/{stream} — serve video (synced preferred)
- POST /api/episodes/{id}/sync — proxy to Docker sync API
- GET /api/episodes/{id}/sync-status/{job_id} — poll proxy
- GET /api/episodes/{id}/frame-map — parsed JSONL
Tests: 9 Python tests for episode scanning, 6 TypeScript tests for
sync grade logic.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- use-episodes: extract .episodes array from JSON response wrapper
- use-drift-data: parse JSON response (server returns {"frames": [...]})
instead of treating it as raw JSONL text
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Episode list defaults to Table view instead of Grid
- Grid cards smaller (h-20 thumbnails, 5-6 columns)
- Header: replace segment control with separate nav links (larger, clearer)
- URL routing: /record and /review paths with history.pushState
- Sync error: proper banner with Docker container instructions
("docker compose up" or configure remote sync_endpoint)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- globals.css: cursor:pointer on all interactive elements (button, select, checkbox, links), cursor:not-allowed on disabled - Header nav: minimal underline indicator style instead of background pill — active link gets a 2px bottom bar, inactive is muted/60 with hover transition. Tighter tracking for professional feel. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…de dir The sync trigger now uses POST /sync/upload (multipart) instead of POST /sync (JSON with local paths). This eliminates the Docker volume mount requirement — files are uploaded directly from the host to the container regardless of filesystem layout. On sync completion, the status poller downloads sync_report.json, frame_map.jsonl, and synced videos back into the episode's synced/ subdirectory so the Review UI can display them immediately. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
… URL - Video: preload="auto" + seek to first frame on load so thumbnails show instead of black - Sync button: larger progress bar (h-2 w-32), spinner icon, phase labels (Analyzing streams, Aligning, Re-encoding...), handles undefined progress (no NaN) - Auto-refresh: episode detail re-fetches when sync transitions from syncing→complete, no manual page reload needed - Terminal: print "SyncField Viewer running at: http://..." on startup Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Playback: - Rewrite usePlayback to use ontimeupdate events instead of rAF (more reliable for <video> elements, fires on actual time change) - Fix stale closure bug: use direct event handler assignment instead of addEventListener with unbound functions - Remove playbackRate dependency from videoRef callback Timeline: - Add pointer capture drag-seeking (smooth scrubbing) - Thicker progress bar (h-1.5, h-2 on hover) with scrubber handle - Bigger play/pause button with hover state Drift Chart: - Fix "before" drift calculation: add back sync_report offset_ms to frame_map delta_ms (was using nonsensical frame index math) - Fetch sync_report in parallel with frame_map for offset data - Use timesSec for x-axis (was using frame indices) - Add axis labels (time in seconds, drift in ms/s) - Downsample to 400 points max for performance - Show mean after-drift in legend Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Add interactive sync comparison modal (ported from opengraph-studio analyzer): - Click any secondary stream video or sidebar stream row to open - Dual-canvas frame capture at corrected vs uncorrected timestamps - Horizontal slider to reveal Before (amber) / After (blue) comparison - Stats bar: timestamps, drift amount, frame offset - Drag hint for discoverability, keyboard Escape to close Supporting changes: - ReviewVideoPlayer: clickable secondary streams with hover ring + "Click to compare" hint, color-coded drift badges (<5ms green, 5-20ms amber, >20ms red) - SyncQualityPanel: primary stream marked with blue "REF" label, secondary streams clickable with chevron indicator - EpisodeDetail: wire up comparison modal state Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Sync Comparison Modal: - Add primary stream reference panel (right side, 35% width) showing the reference frame at the same timestamp for visual comparison - Accept primaryStreamId prop and capture frame from primary video Video Player: - "REF" → blue "Primary" badge, more visible on video overlay - Secondary videos sync play/pause state with primary - Debounce secondary time sync to avoid stutter (300ms threshold, 500ms cooldown) Playback: - Use requestAnimationFrame during playback for smooth ~60Hz timeline updates (was using ontimeupdate at ~4Hz causing 1-second jumps) - rAF starts on play, stops on pause — no wasted cycles when paused Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Drift chart moved from below timeline into right sidebar (below sync quality panel) for a cleaner layout - Sidebar widened to 288px (w-72) to fit drift chart comfortably - Sync comparison modal: cursor-ew-resize on slider area - Playback: throttle React state updates to ~15Hz (every 66ms) instead of 60Hz — primary video still decodes natively at full frame rate, only the timeline display + secondary sync updates are throttled. This eliminates the stutter caused by 60 re-renders/sec cascading to secondary video currentTime assignments. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
styu12
changed the title
Port the Quest 3 hand tracker from opengraph-studio/recorder into the syncfield-python SDK as a proper StreamBase adapter with 4-phase lifecycle. MetaQuestHandStream receives JSON packets over WiFi UDP from a Unity app running on Meta Quest 3. Channels emitted: - hand_joints: 156 floats (26 OpenXR joints x 3 xyz x 2 hands) - joint_rotations: 208 floats (26 joints x 4 quaternion x 2 hands) - head_pose: 7 floats (position + quaternion) Supports two modes: - "hand": full skeleton with 26 OpenXR joints per hand - "controller": controller pose mapped to wrist joint slots No external dependencies — uses stdlib socket + json only. Default port 14043 (Manus uses 14042). 20 unit tests covering schema dimensions, joint/rotation/head extraction, controller mode, channel parsing, and full UDP lifecycle with real sockets. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
HostAudioStream adapter:
- Records microphone audio to WAV (44.1 kHz, 16-bit, mono) via
sounddevice InputStream + stdlib wave module
- Emits real-time RMS/peak metrics as SampleEvent channels (~10 Hz)
for viewer visualization
- 4-phase lifecycle: connect (validate device) → start_recording
(open stream + WAV) → stop_recording (flush + close) → disconnect
- Requires 'audio' extra (sounddevice + numpy), graceful ImportError
Orchestrator auto-inject:
- In connect(), if no stream has provides_audio_track=True, auto-detect
host microphone and inject HostAudioStream("host_audio")
- Tracked via _auto_audio_stream, auto-removed on disconnect()
- Skips gracefully: no audio extra, no mic, or user already added audio
Viewer — Recording mode:
- AudioLevelChart component: real-time VU-meter with waveform history
bars, green/yellow/red gradient based on level
- StreamCard routes kind="audio" to AudioLevelChart
Viewer — Review mode:
- GET /api/episodes/{id}/waveform/{stream_id}: reads WAV, returns
downsampled min/max envelope (1000 points)
- WaveformChart component: SVG waveform with time axis labels
Tests: 22 new (17 adapter + 5 orchestrator auto-inject)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…ility Move audio stream pre-registration from connect() to add() — when the first non-audio stream is added, the orchestrator checks for a mic and registers HostAudioStream immediately. This makes the audio card visible in the viewer before the user presses Connect or Record. The actual device open still happens in connect() along with all other streams. connect() has a fallback _maybe_inject_host_audio() for edge cases where add() was skipped. Update tests to patch before add() since pre-registration now happens at add() time, not connect() time. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Detect when SSE data stops arriving (1.5s timeout) and switch to an idle state showing a microphone icon + "Microphone ready" text instead of frozen waveform bars at the last recorded level. During recording: live waveform bars + VU meter (unchanged). After stop: clean idle state, no stale data. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Restructure HostAudioStream lifecycle to match video adapters: - connect() now opens sounddevice.InputStream for live preview (emits RMS/peak metrics via SampleEvent immediately) - start_recording() adds WAV file writing on top of the live stream - stop_recording() stops WAV writing, but preview continues - disconnect() closes the input stream entirely The audio callback always emits metrics (for viewer waveform), and conditionally writes PCM16 to WAV only when _recording is True. AudioLevelChart: show "Microphone ready" only when no SSE data has arrived (truly disconnected). Once connected, waveform bars show immediately from the preview stream. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Review pages now use URL paths:
- /review → episode list
- /review/{ep_id} → episode detail
Browser back/forward and direct links work. Episode selection
pushes to history via pushState, back button returns to list.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Server: "manifest.json not found" → "No recorded data in this episode. Record a session first." (clearer for empty ep_ dirs) - Sync hook: parse server error body instead of showing generic "Failed to trigger sync (400)" - Error banner: only show Docker instructions for 502 errors (connection refused), not for 400 (data issues) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Empty ep_* directories were left behind when the viewer was opened but no recording was made. Now: - Track _has_recorded flag, set to True when RECORDING state is reached - __del__ calls _cleanup_empty_episode_dir() if no recording happened - Cleanup removes the episode dir only if it has zero files (safe) Also add tests/unit/conftest.py with autouse fixture to disable audio auto-injection in all unit tests — prevents test failures on machines with microphones where the auto-inject would add unexpected streams. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Stop now validates output files on disk and reports per-stream status: Server (server.py): - _stop_and_report() captures SessionReport from session.stop() - Validates each stream: checks file_path exists on disk, checks timestamps JSONL for sensor streams, checks manifest.json and sync_point.json were written - Broadcasts "stop_result" WebSocket event with status per stream Frontend: - StopResultEvent type with per-stream status, file_exists, warnings - use-session hook exposes stopResult + dismissStopResult - StopResultBanner component: - "saving" → spinner animation - "success" → green banner with ✓ per stream + frame counts - "partial" → amber with ⚠ for streams with warnings - "error" → red with ✗ for failed streams + error messages - Dismissable with × button, auto-cleared on next Record Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…n disconnect 1. Sync auto-reload: wrap EpisodeDetail in key-based re-mount so sync completion triggers full data refresh (episode + drift + videos) without manual page reload 2. Audio waveform in review: filter audio streams out of video player area — host_audio was rendering as black video. Audio streams now only appear as WaveformChart in the sidebar. 3. Keep audio stream on disconnect: don't remove auto-injected HostAudioStream from _streams on disconnect — keep it registered so it remains visible in the viewer. It will be reconnected on next connect() call. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…ale dirs Previously, __init__ created the ep_* directory immediately, leaving empty directories when the viewer was opened without recording. Now: - __init__ generates the episode PATH but does NOT mkdir - Data root directory is created (so output_dir property works) - Episode directory is only created when start() is called - Session log writer also deferred to start() (no session_log.jsonl in non-recording sessions) This eliminates stale episode directories entirely — if you never press Record, no episode directory is created on disk. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Task system for organizing recording sessions:
Backend:
- tasks.json in data root — flat list of {name, description}
- REST endpoints: GET/POST/PUT/DELETE /api/tasks, POST /api/task/select
- SessionOrchestrator.task property — written to manifest.json
- Episode scan includes task field from manifest
Frontend:
- TaskSelector component in Recording mode header — dropdown with
inline create (type + Add), delete (× per item), select, and clear
- useTasks hook — CRUD operations + current task selection
- EpisodeTable shows Task column in Review mode
- EpisodeSummary type includes task field
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Record button: - Disabled unless a task is selected (hasTask gate) - "Select a task first" hint shown when connected without a task Cancel button: - Implement SessionOrchestrator.cancel() — stops all streams without chirp, deletes the episode directory entirely (no partial files), generates a fresh episode path for the next recording - Server broadcasts cancel result via WebSocket - UI transitions back to connected state cleanly Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Backend: - Fix task API endpoints: use fastapi.Request instead of Any for request parameter — FastAPI was treating it as a query param Frontend: - TaskSelector: refined design — selected task shown with teal primary tint, unselected shows dashed border placeholder - Remove "Select a task first" text from Record button — the task selector's empty state and Record's disabled opacity naturally guide the user - Dropdown: rounded-xl, proper hover states, delete button appears on row hover Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Move self._output_dir.mkdir() from prepare() to start_recording() in UVCWebcamStream and OakCameraStream. This ensures the episode directory is only created on disk when recording actually begins. Previously, prepare() (called during connect()) created the directory, leaving empty ep_* folders when the user connected devices but never pressed Record. Now the lifecycle is fully deferred: - __init__: generate episode path (no mkdir) - connect/prepare: open devices only (no filesystem) - start_recording: mkdir + open writers (first disk touch) - stop_recording: finalize files - cancel: delete episode dir HostAudioStream already had mkdir in start_recording (correct). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…t record 1. Audio stream card: skip duplicate "audio" tag when kind is already "audio" (was showing "audio audio file") 2. Record button task gate: lift useTasks() to parent RecordView and pass state as props to TaskSelector — both components now share the same hook instance, so selecting a task immediately enables the Record button 3. Episode path: regenerate at start() time so the ep_ timestamp reflects when recording began, not when the script launched. Footer only shows the path during/after recording. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Cancel fixes: - Close log writer BEFORE rmtree (was leaving open file handle) - Close log writer regardless of auto_connected state - Server: wrap cancel in try/catch with error broadcast - Server: send "saving" state before cancel starts UI: - StopResultBanner: new "cancelled" state — muted gray banner with "Recording cancelled — data discarded" message - StopResultEvent type: add cancelled boolean flag - Distinguish cancelled from saved in all banner states Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
start() was regenerating the episode path with _generate_episode_path(), creating a second ep_* directory. Streams already referenced the original path from __init__, so their start_recording() mkdir'd the old path while the orchestrator mkdir'd the new one — two directories. Fix: only mkdir the existing path in start(), never regenerate. The path from __init__ is shared with streams via output_dir and must remain stable across the session lifecycle. cancel() still regenerates for the next recording (correct). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Simplify tasks.json to name-only: [{"name": "pick_cup"}, ...]
Remove description from server CRUD, frontend Task type, and
TaskSelector props.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
FPS measurement:
- snapshot_fps() now computes from time span between first and last
sample in the 1-second window: (count-1) / span_seconds
- Gives fractional precision (29.47 Hz) instead of integer count (29)
- Use min/max for span calculation (order-independent)
Health Events:
- Emit HEARTBEAT health event ("connected") for each stream after
successful connect() — gives visual confirmation in the viewer's
Health Events sidebar that devices are alive
- Non-noisy: one event per stream per connect cycle
Also: log episode dir path on creation for debugging.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Cancel→Record episode duplication: - After cancel generates a new episode path, update all stream adapters' _output_dir, _file_path, _mp4_path to match - Prevents streams from writing to the old (deleted) path when the next recording starts Health Events UI: - Time: monotonic seconds (635899.6s) → relative "just now" / "3s ago" / "2m ago" format - Layout: stream_id as primary text, icon-based kind indicator (● heartbeat, ⚠ warning, ✗ error, ↻ reconnect) - Heartbeat events now show green ● instead of gray - Dividers lighter for less visual noise macOS warning: AVCaptureDeviceTypeExternal deprecation is harmless Apple system log — not actionable from the SDK side. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…_episode Root cause fix for episode directory reuse after stop: _prepare_next_episode() helper — called after both stop() and cancel(): 1. Generates a fresh episode path with new timestamp 2. Resets _episode_dir_created flag 3. Updates all stream adapters' output paths (_output_dir, _file_path, _mp4_path, _wav_path) to the new directory stop() now also closes the log writer before preparing next episode, ensuring clean separation between consecutive recordings. 10 comprehensive tests covering: - 2 and 5 consecutive record→stop cycles - cancel→record, multiple cancels→record - mixed record→stop→record→cancel→record→stop - connect→disconnect (no dirs) - episode isolation (each has its own files) - no stale dirs after full lifecycle conftest: also patch _maybe_inject_host_audio to prevent segfault from sounddevice in connect-heavy test flows. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Stream card "issue" count: - Server: add problem_count field (non-heartbeat events only) - Frontend: use problem_count instead of health_count for the red "issues" badge. Heartbeat "connected" events no longer appear as alarming red issues. Video cache-bust after sync: - Append ?v=<mount_token> to video src URL using useMemo so browser re-fetches after EpisodeDetail re-mounts (post sync completion). - Previously, identical URLs caused browsers to serve stale/empty responses until manual page refresh. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DearPyGui 데스크톱 뷰어를 브라우저 기반 웹 뷰어로 교체하고, Synchronization 트리거 + Episode Review 기능을 추가합니다.
localhost:8080)로 multipart upload 방식 sync 요청. 진행 상태 실시간 표시, 완료 시 결과 자동 다운로드SessionOrchestrator가 에피소드 디렉토리 자동 생성 (ep_{timestamp}_{hex}), 예제 코드 간결화Changes
Python Backend
viewer/server.py: FastAPI 서버 — WebSocket, MJPEG, SSE, REST, Episode API (6개 엔드포인트), Sync proxy (multipart upload + result download)viewer/app.py: FastAPI + uvicorn +webbrowser.open(). 터미널에 URL 출력viewer/__init__.py:launch()/launch_passive()시그니처 유지,host/port추가orchestrator.py:_make_episode_dir()자동 에피소드 디렉토리 생성pyproject.toml:dearpygui→fastapi+uvicorn+opencv-pythonReact Frontend (
viewer/frontend/)/record/reviewURL 라우팅, 헤더 네비게이션 링크Removed
viewer/theme.py,viewer/fonts.py,viewer/widgets/(DearPyGui 전용)Test Plan
python -m syncfield.viewer.demo정상 실행python examples/iphone_mac_webcam/record.py녹화 + 뷰어🤖 Generated with Claude Code