feat(adapters): generic sensor stream helpers (PollingSensorStream + PushSensorStream) - #3
Merged
Merged
Conversation
Luxonis OAK camera adapter using the DepthAI v3 pipeline API. Captures RGB
to an MP4 via cv2.VideoWriter and optionally streams raw uint16 depth to
a sibling {id}.depth.bin file.
Design
------
Intentionally thinner than the full-featured OakCamera used inside
opengraph-studio/recorder — ships the 80% common case (RGB + optional
depth) so the code stays small, testable, and easy to extend. Users who
need IMU, stereo rectified outputs, or custom calibration can subclass
directly against the depthai API.
Scope
-----
- OakCameraStream(StreamBase) with:
* RGB via Camera → requestOutput → OutputQueue
* Optional depth via StereoDepth node (HIGH_DETAIL preset)
* Background capture thread — read/timestamp/write/emit in a tight loop
* Depth pulled with tryGet() on the same tick as RGB so they share
the same monotonic anchor
* Declares produces_file=True, is_removable=True,
supports_precise_timestamps=True, provides_audio_track=False
- iter_depth_frames() helper for consumers that want to read back the
raw .depth.bin file later
- Gated behind the new syncfield[oak] extra (depthai>=3.0.0). The adapter
also uses opencv-python for the MP4 writer, so install with
syncfield[oak,uvc] or syncfield[all]
- adapters/__init__.py lazy-exports OakCameraStream so an uninstalled
extra does not break `import syncfield.adapters`
Tests
-----
tests/unit/adapters/test_oak_camera.py (8 tests) with a mocked depthai
module that models the v3 pipeline + queue API. Covers:
- Capability flags round-trip
- prepare() builds + starts the pipeline
- prepare() raises cleanly when no OAK devices are connected
- start/stop lifecycle produces a file_path in the FinalizationReport
- stop() releases the pipeline
- depth_enabled=True creates a second pipeline node (StereoDepth)
- depth_enabled=False is the default and creates only the RGB camera
- Missing depthai raises ImportError with `syncfield[oak]` install hint
Full suite: 138 passing (was 130).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Adds ``syncfield.viewer``, a DearPyGui-based desktop viewer that renders
in-process alongside the SDK — no HTTP, no IPC. Matches the launch API
pattern established by ``mujoco.viewer``:
import syncfield.viewer
syncfield.viewer.launch(session) # blocking
with syncfield.viewer.launch_passive(session) as v: # passive
...
Design
------
The viewer is strictly additive. The core SDK stays stdlib-only; users who
don't want the GUI install nothing new. ``pip install 'syncfield[viewer]'``
pulls in ``dearpygui>=2.0`` and ``numpy>=1.21`` (1.8 MiB wheel total).
Architecture is three layers:
- **state.py** — immutable SessionSnapshot/StreamSnapshot dataclasses,
plus a mutable StreamStatsBuffer the poller maintains per stream
(rolling fps window, plot deques with NaN back-fill for channels that
appear mid-stream, capped health log).
- **poller.py** — daemon background thread polls the session at 10 Hz,
subscribes to each stream's on_sample/on_health to catch per-sample
data that would otherwise be lost between poll ticks. Publishes
immutable snapshots under a single lock.
- **app.py + widgets/** — DearPyGui render loop reads the latest snapshot
on every frame and fans values out to widgets. All DPG mutation is on
the main thread; session.start()/stop() are delegated to a worker
thread so the UI never blocks.
Widgets
-------
- widgets/layout.py — top-level screen composition. Header (host + state
chip + elapsed), control panel (Record/Stop/Cancel, buttons wired
to session via a worker thread), session clock panel (sync point +
chirp timing + tone spec), horizontal stream card row (scrollable),
health events table, footer (output path + wall clock).
- widgets/stream_card.py — per-stream card with three body variants:
video (raw GPU texture updated from stream.latest_frame with
letterboxing), sensor/audio (line plot with multi-channel series,
calibrated OpenGraph color palette), generic (stats-only fallback).
Cards are created lazily when the layout first sees a stream id, so
dynamic additions Just Work.
- widgets/formatting.py — small pure-Python formatters with unit tests.
Theme
-----
Light theme only, matching OpenGraph's minimal sophisticated design
language. One file (theme.py) owns every token — near-white surfaces,
subtle gray borders, near-black primary text, calibrated indigo accent,
soft border radii (6-10 px), generous padding. A future dark mode or
brand recolor is a single-file change. Button variants (primary/danger/
ghost) and panel variants (card/soft) are built as separate DPG themes
and bound at widget creation time.
Video frame publishing
----------------------
UVCWebcamStream and OakCameraStream now expose a thread-safe
``latest_frame`` property. The capture loops stash a reference to the
most recent BGR frame under a small lock; the viewer reads that
reference, resizes nearest-neighbor into the preview texture buffer
(numpy-only, no opencv dependency in the viewer), and uploads to the
GPU via ``dpg.set_value``. ~10 lines per adapter, zero change to the
capture hot path.
Tests
-----
tests/unit/viewer/ (41 new tests):
- test_formatting.py — all format_* helpers including edge cases
(negative elapsed, millisecond rounding overflow, NaN/None handling)
- test_state.py — StreamStatsBuffer sample observation, fps rolling
window with 1-second cutoff, NaN back-fill for late-joining plot
channels and forward-fill for missing channels, non-numeric channel
filtering, health event ordering
- test_poller.py — end-to-end snapshot building against a real
SessionOrchestrator with FakeStreams, sync point population after
start(), sample/health callback wiring, background thread lifecycle
All 179 tests pass (138 existing SDK + 41 new viewer).
Demo
----
syncfield/viewer/demo.py ships a runnable synthetic session with two
fake video sources (procedural gradient + drift), a sinusoidal IMU, and
a plain FakeStream — enough to exercise every card variant without
hardware. Intended as both a manual-test harness and the screenshot
generator for the docs. ``--auto-record`` starts the session on launch;
``--duration`` auto-closes after N seconds for headless screenshotting.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Follow-up to the initial viewer commit — fixes layout clipping and
enables the screenshot harness that produces the docs images.
Layout tweaks
-------------
- VIEWPORT_WIDTH 1280 → 1200, VIEWPORT_HEIGHT 860 → 900 — tighter
horizontal margin, more room for the health events table.
- CONTROL_PANEL_HEIGHT 110 → 160 — fits all three buttons (Record,
Stop, Cancel) without clipping the ghost-styled Cancel.
Demo: chirp enabled by default
------------------------------
The demo previously used SyncToneConfig.silent() to avoid beeping
during runs. That made the session clock panel show "chirp: disabled"
which misrepresented the actual SDK default. New approach:
session = sf.SessionOrchestrator(
host_id="demo_rig",
output_dir=output_dir,
sync_tone=sf.SyncToneConfig.default(), # chirp enabled
chirp_player=SilentChirpPlayer(), # ...but silent
)
cam_ego now also declares provides_audio_track=True so chirp
eligibility kicks in and the sync point actually fills in
chirp_start_ns. The viewer's session clock panel shows the real
"400 → 2500 Hz, 500 ms" tone spec and the live chirp timestamps.
Screenshot harness
------------------
demo.py gets a --screenshot PATH flag that:
1. Pins the viewport to (60, 60) via a new ViewerApp.viewport_pos kwarg
2. Runs for --duration seconds (default 3)
3. Probes the frontmost window title via AppleScript (sanity check)
4. Captures full-screen + viewport-region PNGs via `screencapture`
5. Quits cleanly
Usage:
python -m syncfield.viewer.demo --screenshot idle.png
python -m syncfield.viewer.demo --auto-record --screenshot recording.png
Used to generate the docs screenshots at website/static/img/viewer/.
Keeping the harness in demo.py (instead of a separate script) means
the docs images stay reproducible — future maintainers can rerun the
same command to refresh them.
All 179 tests still pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Ports the OGLO tactile glove BLE protocol from the egonaut iOS app
(TactileGloveManager.swift) into a StreamBase-based SDK adapter, and
lazy-exports it from syncfield.adapters. Mirrors the recorder-side
OgloTactileSensor we added earlier — same protocol, same per-sample
device_timestamp_ns interpolation — but lives in the SDK so
syncfield.viewer / SessionOrchestrator / any other top-level orchestrator
can drive it directly without going through the recorder's BaseSensor.
Protocol
--------
- Service UUID: 4652535f-424c-4500-0000-000000000001
- Notify char: 4652535f-424c-4500-0001-000000000001
- Config char: 4652535f-424c-4500-0002-000000000001
- Packet layout (little-endian):
[0:2] u16 count — samples in this batch
[2:6] u32 timestamp_us — MCU hardware clock at batch start
[6:..] 5×u16 per sample: thumb, index, middle, ring, pinky
- Sample rate: 100 Hz effective (10 notifications/sec × 10 samples each)
- Scan filter: advertised name substring "oglo" (case-insensitive)
One SampleEvent per decoded sample — the 10-sample batch fans out into
10 events so downstream consumers see the full 100 Hz rate. The MCU
hardware clock is linearly interpolated across the batch
(sample_i → batch_timestamp + i × 10 ms) so device_timestamp_ns is
uniformly spaced at exactly 10_000_000 ns increments. Malformed
packets become WARNING health events instead of tearing down the stream.
API
---
from syncfield.adapters import OgloTactileStream
# Explicit address — preferred once you know which glove is which
session.add(OgloTactileStream(
id="tactile_right",
address="AA:BB:CC:DD:EE:FF",
hand="right",
))
# Or scan by advertised name
session.add(OgloTactileStream(
id="tactile_right",
ble_name="oglo",
hand="right",
))
Thread model matches BLEImuGenericStream: asyncio loop on an internal
background thread, start/stop signaled via threading.Event, decode path
factored into a pure _handle_payload method with a
_dispatch_notification_for_test hook for unit tests.
Gated behind syncfield[ble] (already pulls in bleak for BLEImuGenericStream).
Listed in the adapters table in adapters/__init__.py docstring.
Tests
-----
tests/unit/adapters/test_oglo_tactile.py — 14 tests covering:
- capability flags round-trip, hand property, default scan filter
- UUID constants match the egonaut Swift values exactly
- canonical finger order (thumb/index/middle/ring/pinky)
- full batch decode with channel naming
- device_timestamp_ns linear interpolation (10 ms spacing)
- frame count accumulation across multiple batches
- short packet / truncated body → WARNING health events (no crash)
- FinalizationReport counts on stop
- construction ValueError when both address and ble_name missing
- ImportError with install hint when bleak is unavailable
- Lazy re-export appears in syncfield.adapters.__all__
Full SDK suite: 193 passing (was 179, +14 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…gistry
Adds src/syncfield/discovery/ — a new top-level package that enumerates
connected cameras and sensors and hands back ready-to-construct Stream
adapters. Zero hardware required for the core; per-adapter discoverers
land in a follow-up commit.
Public API
----------
- ``syncfield.discovery.scan(*, kinds, timeout, use_cache)`` →
``DiscoveryReport`` — primitive. Walks every registered discoverer in
a ThreadPoolExecutor, respects an overall wall-clock budget, returns
an immutable report. Partial failures become ``report.errors`` entries;
slow adapters land in ``report.timed_out``. Never raises.
- ``syncfield.discovery.scan_and_add(session, *, kinds, id_prefix,
output_dir, skip_existing, timeout)`` → ``list[DiscoveredDevice]``
— convenience wrapper. Runs scan(), generates collision-free stream
ids from display names, constructs each adapter via
``DiscoveredDevice.construct()``, and calls ``session.add()``. Skips
devices whose ``warnings`` are non-empty or that appear ``in_use``.
- ``syncfield.discovery.register_discoverer(adapter_cls)`` — the
registration hook for Stream adapter classes. Validated at call time
for the ``discover()`` classmethod and ``_discovery_kind`` attribute.
Design pillars
--------------
- **Each adapter owns its discovery logic** via a ``@classmethod
discover(cls, *, timeout)`` — co-located with the Stream it builds,
IDE-discoverable, stateless. The registry is a thin coordinator, not
a plugin loader.
- **DiscoveredDevice holds a direct class reference**, so
``device.construct(id=..., output_dir=...)`` is a one-liner with no
string → class lookup and no reflection.
- **Short-lived scan cache** (5 s) so back-to-back calls to ``scan()``
don't re-scan BLE. The CLI and viewer Rescan button override with
``use_cache=False``.
- **Shared BLE scan cache** in ``_ble.py`` so when multiple BLE adapters
(OgloTactile + BLEImuGeneric) both want to enumerate peripherals,
they share one 5-second BleakScanner run instead of two.
- **Cooperative timeout budget** — the thread pool honors the overall
deadline; discoverers exceeding their slice end up in ``timed_out``.
Module structure
----------------
- types.py — DiscoveredDevice, DiscoveryReport (frozen dataclasses)
- _id_gen.py — normalize() + make_stream_id() (collision-safe)
- registry.py — register_discoverer, iter_discoverers, lock-guarded
- _ble.py — shared BleakScanner cache (3 s TTL)
- scanner.py — scan() + scan_and_add() with 5 s result cache
- __init__.py — curated public surface
46 unit tests across tests/unit/discovery/:
- test_types.py — frozen dataclass behavior, construct() merging,
by_kind / by_adapter_type filters, summary formatting
- test_id_gen.py — snake_case normalization, collision suffixes,
prefix handling, exhaustion guard, unicode fallback
- test_scanner.py — end-to-end with stub adapters (video + sensor),
failing adapter → errors, slow adapter → timed_out,
kinds filter, cache hit/miss, scan_and_add id
collision avoidance, non-IDLE refusal, output_dir
injection for video-only adapters
Full SDK suite: 254 passing (193 before → +46 discovery + some
concurrent tone/orchestrator changes).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Rework ChirpPlayer protocol to return ChirpEmission carrying both the
software monotonic timestamp (always present) and the hardware DAC
presentation timestamp (best-effort via PortAudio callback time_info).
SoundDeviceChirpPlayer now uses sd.OutputStream with a callback so it
can sample time.monotonic_ns() inside the first callback and convert
outputBufferDacTime - currentTime into a monotonic offset. Falls back
to software timestamp when the backend cannot supply DAC time or the
first callback does not fire within 100 ms.
SessionOrchestrator stores ChirpEmission objects instead of raw ns ints
and writes both best_ns and source ("hardware"/"software_fallback"/
"silent") into sync_point.json. SessionReport gains source fields so
downstream sync tooling can decide which hosts can claim sub-ms chirp
anchor precision.
38 new tests (TestChirpEmission, TestSoundDeviceChirpPlayerHardwareTimestamp,
TestChirpEmissionPropagation) cover the hardware path, fallback paths,
active-stream cleanup, and source propagation through sync_point.json.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Every shipped adapter now implements ``@classmethod discover(cls, *,
timeout)`` and registers itself with the discovery registry in
``adapters/__init__.py``. Calling ``syncfield.discovery.scan()`` after
importing ``syncfield.adapters`` walks all four adapters in parallel
and returns a unified DiscoveryReport.
OakCameraStream
---------------
- New ``device_id`` constructor kwarg for multi-OAK pinning. When
multiple devices are attached, ``scan_and_add`` wires each one to its
specific serial via ``construct_kwargs={"device_id": ...}`` so the
two streams don't race for "first available".
- ``prepare()`` now resolves the selected device info and passes it
to ``pipeline.build(selected)``; falls back to the older no-arg form
via TypeError for DepthAI shims.
- ``discover()`` calls ``dai.Device.getAllAvailableDevices()``
(sub-millisecond, no scan prompt) and returns one DiscoveredDevice
per attached OAK. Swallows exceptions into empty list per the
discoverer contract.
UVCWebcamStream
---------------
- Platform-split enumeration factored into free helpers
``_discover_uvc_macos()`` (system_profiler SPCameraDataType -json)
and ``_discover_uvc_linux()`` (/dev/video* + sysfs name lookup).
Windows / other platforms return empty list so discovery stays
predictable. Critically, we DO NOT probe cv2.VideoCapture during
discovery — that triggers the macOS camera permission dialog.
- Device indices map to array position in the native listing, which
is stable on almost all macOS setups (built-in at 0, Continuity
Camera at 1, externals after). If the mapping drifts, users fall
back to explicit ``UVCWebcamStream(device_index=...)``.
OgloTactileStream
-----------------
- ``discover()`` reads from the shared BLE cache
(``syncfield.discovery._ble.scan_peripherals``) and filters by
case-insensitive substring match on the advertised name. The
default filter "oglo" picks up both left and right gloves.
- Infers ``hand`` from the advertised name when "left"/"right"
appears, populates ``construct_kwargs`` with both ``address`` and
``hand`` so ``scan_and_add`` can build a working stream with zero
extra input from the caller.
BLEImuGenericStream
-------------------
- ``discover()`` reads from the same shared BLE cache (so two
BLE-based adapters running in parallel share one 5-second scan)
and returns ALL non-oglo peripherals. Each one is flagged with
a ``warnings`` entry explaining that a ``characteristic_uuid`` is
still required — ``scan_and_add`` then skips them with an INFO log,
which is the correct behavior because there's no generic way to
determine the notify characteristic of an arbitrary peripheral.
- Uses ``adapter_type="ble_peripheral"`` (not ``ble_imu``) to make
the "candidate, not-yet-usable" status clear in the CLI / viewer.
- Excludes peripherals that a more-specific adapter would match
(currently "oglo") to avoid double-listing.
Registration
------------
adapters/__init__.py now calls ``_safe_register()`` after each
try-import so adapters whose optional extras are installed get
added to the discovery registry automatically. ``_safe_register``
swallows TypeError from misconfigured discoverers so one bad adapter
never breaks the whole package import.
Tests
-----
tests/unit/adapters/test_discover_{oak,uvc,ble}.py — 23 new tests:
- test_discover_oak_camera.py (6 tests):
empty device list, single device, multi-device, exception
swallowing, class attributes present, end-to-end construct()
from a discovered device.
- test_discover_uvc_webcam.py (7 tests):
macOS happy path, missing system_profiler, subprocess failure,
malformed JSON, Linux /dev/video enumeration, unsupported
platform, class attributes.
- test_discover_ble.py (10 tests):
OGLO name-substring filtering, hand inference from name (left/
right/unknown), address population, empty scan result,
case-insensitive match, BLEImuGeneric excludes OGLO peripherals,
all discoveries carry characteristic_uuid warning, unnamed
peripheral fallback, empty scan.
Fix
---
syncfield/discovery/_ble.py had a stray ``global _cache, _cache_time``
declaration inside the cache-update block AFTER the variable was read
in the hit path earlier in the same function — Python rejects that
with a SyntaxError. Moved both ``global`` lines to the top of their
respective functions.
Full SDK suite: 315 passing (was 254, +23 new adapter discovery +
indirect benefits from registry auto-wiring).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
New syncfield.multihost subpackage lets multiple hosts on the same local network coordinate around a single SyncField session without any central coordinator: - SessionAnnouncement: dependency-free wire type with to/from TXT record serialization. Leader drives preparing → recording → stopped. - generate_session_id / is_valid_session_id: Docker-style slug id generator with a built-in wordlist, plus a stricter validator that rejects mDNS-hostile characters. - SessionAdvertiser: wraps one python-zeroconf ServiceInfo registration. Lazy zeroconf import so the module stays importable without the multihost extra. Graceful shutdown margin so followers see the final stopped status before the service unregisters. - SessionBrowser: wraps one ServiceBrowser with blocking wait_for_recording / wait_for_stopped helpers backed by a Condition. Optional session_id filter. Distinct from the parallel syncfield.discovery subsystem which handles hardware device enumeration. One finds peers, the other finds devices. 38 new unit tests with a fake-backend fixture that stands in for zeroconf, covering construction validation, registration, status transitions, filter behavior, and the wait/timeout logic. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Add LeaderRole and FollowerRole dataclasses and wire them into SessionOrchestrator via a keyword-only `role` parameter that defaults to None (single-host, behavior unchanged). Leader: - On start(), opens a SessionAdvertiser in the `preparing` state so followers already on the network see the session coming up. - After streams start and the start chirp plays, flips the advert status to `recording` with the leader's monotonic anchor. - On stop(), flips to `stopped` before closing so followers observe the transition during the advertiser's graceful shutdown margin. Follower: - On start(), opens a SessionBrowser (filtered by session_id when supplied) and blocks until a leader is advertising `recording` or leader_wait_timeout_sec elapses. Stores the observed announcement on `observed_leader` so stop()-time artifacts can reference the leader's host_id. - Never plays chirps even with audio-capable streams — followers rely on the leader's chirps being captured by every host's microphones in the same physical space. - `wait_for_leader_stopped()` convenience method delegates to the browser so followers can drive their own stop() off the leader's lifecycle instead of a wall-clock deadline. Both leader and follower manifests now carry session_id + role, and followers additionally persist leader_host_id so the sync core can reconstruct the multi-host topology after the session. 13 new tests for LeaderRole / FollowerRole validation + 12 new integration tests (fake advertiser/browser backends) exercising the happy path, chirp gating, timeout cleanup, and wait_for_leader_stopped preconditions. 340 total tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Third phase of the discovery system. Turns the ``scan()``/``scan_and_add()``
primitives into two user-facing surfaces: a pretty-printing CLI and a
light-themed DearPyGui modal inside the desktop viewer.
CLI — ``python -m syncfield.discovery``
---------------------------------------
syncfield/discovery/__main__.py. Grouped listing by Stream kind
(Cameras / Sensors / Custom), per-device headline + sub-info line,
warnings inline, errors + timed-out adapters at the bottom.
``--json`` for machine-readable output, ``--kinds video sensor`` for
filtering, ``--no-cache`` for forced refresh, ``--timeout N`` for the
scan budget. Exit codes 0/1/2 for found / partial-failure / empty.
Verified on macOS: run against the current test environment, the CLI
found 2 cameras (MacBook Pro built-in + iPhone via Continuity Camera)
plus ~30 ambient BLE peripherals, all correctly flagged as needing
manual characteristic_uuid.
Viewer — Discover devices button + modal
------------------------------------------
syncfield/viewer/widgets/discovery_modal.py — new DiscoveryModal
widget the layout builds once at viewer startup and reuses across
open/close. All widget construction follows the existing OpenGraph
light theme (ghost Rescan/Close buttons, primary Add button, muted
section headers, per-device checkbox rows with two-line labels).
Flow:
1. Click ⚡ Discover devices in the header
2. Modal opens, worker thread runs scan() with use_cache=False
3. Status strip shows "Scanning devices… 2.3s" updating in real time
4. On complete: cards grouped by kind, devices preselected when
addable (no warnings, not in use), Add button shows the selection
count ("Add 2 devices")
5. Add registers each selected device via device.construct() +
session.add(), then closes the modal
6. The main viewer's stream card row picks up the new streams on
the next poller tick (no extra wiring needed — SessionOrchestrator
state is the single source of truth)
Threading — scan runs on a daemon worker, state handoff is via a
single _ModalState dataclass guarded by a lock, the render loop calls
modal.tick() each frame to rebuild the card list only when a scan
has just completed (cheap no-op otherwise).
Layout integration:
- New "⚡ Discover devices" ghost button in the header row
- `btn_discover` enabled only when session.state == "idle"
- update() now calls modal.tick() every frame
BLE cache serialization fix
---------------------------
_ble.py had a concurrency bug: the cache check released the lock before
the BleakScanner.discover() call, so two parallel BLE discoverers (OGLO
+ generic BLE peripheral) would both miss the cache, both release the
lock, and both kick off independent BleakScanner runs in parallel.
That doubled the wall-clock time and, on macOS CoreBluetooth,
occasionally returned garbage because concurrent scanners aren't
supported.
Fix:
- Single lock held across cache-check + scan + cache-update so
concurrent callers serialize onto one shared scan
- Hard cap BleakScanner window at _MAX_SCAN_S = 5.0 seconds
regardless of requested timeout (BLE advertisement cycles are
1-4s, more is wasted)
- Effective timeout clamped between 0.5s and 5.0s
Before the fix: discovery timed out at the full 10 s budget because
BLE hung both discoverers for 10 s each.
After the fix: the CLI returns in 5.1 s with 2 cameras + 32 BLE
candidates.
Demo harness additions
----------------------
demo.py picks up two new flags for screenshot capture:
- --empty-session: build a bare session with no pre-populated
synthetic streams, so the viewer shows the "click Discover to
begin" empty state
- --open-discovery: auto-click the Discover button 0.8s after
startup so screenshots can capture the modal without user input
Used to produce website/static/img/viewer/discovery-modal.png for
the upcoming docs page.
Tests
-----
338 unit tests passing (315 → +0 new here; the modal + CLI are
exercised by the existing discovery tests plus a manual smoke test
that instantiates ViewerApp + opens the modal). Two pre-existing
mDNS integration test failures in test_multihost_rendezvous.py are
unrelated to this work (verified by stashing and rerunning on main).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…test - Export LeaderRole, FollowerRole, RoleKind, ChirpEmission, ChirpSource from the top-level syncfield package so users can opt into multi-host with one import. - pyproject: add `multihost` optional extra pinning `zeroconf>=0.130` (not pulled into the default install), plus `slow` pytest marker for integration tests that touch real IO. - test_public_api: assert the new exports and verify the syncfield.multihost subpackage is importable. - Integration test (marked slow): end-to-end leader advertises → follower observes recording → leader flips stopped → follower observes stopped, using the real zeroconf stack on loopback. Includes a strict mDNS probe that attempts a round-trip register→browse→get_service_info cycle during collection and cleanly skips the module when the host's multicast path is broken (sandboxed CI, no active network, etc.) — the fake-backend unit tests under tests/unit/multihost/ remain the source of truth for logic coverage. Advertiser hardening: build a fresh ServiceInfo on every status transition via _build_service_info() instead of mutating the existing instance in place. zeroconf>=0.140 made ServiceInfo.properties read-only; the new path works on every supported version. Browser hardening: pass timeout=3000 ms to Zeroconf.get_service_info so the ServiceListener callback waits for the full TXT record to resolve before parsing it. Falls back to the no-kwarg call when the backend (e.g. unit-test fakes) doesn't accept the argument. 341 unit tests pass, 2 integration tests skip on this host. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
First SDK example under examples/. Shows the shortest end-to-end SyncField setup: one SessionOrchestrator, two UVCWebcamStream adapters (Mac built-in webcam + iPhone over Continuity Camera, both driven through cv2.VideoCapture), one viewer launch. Each example lives in its own subdirectory with a runnable record.py and a self-contained README.md covering hardware checklist, install, run, output layout, architecture diagram, and troubleshooting table. New recipes scale by copying the pattern, not by sharing helpers. record.py extras: - --probe mode: enumerates openable OpenCV device indices with their geometry, so users can figure out which index is which camera before starting a session. - Graceful ImportError messages pointing at the right `pip install` extras when viewer / uvc are missing. Top-level examples/README.md is the catalog index and documents the "one directory per recipe, one record.py per directory" convention for future contributions (oak_plus_webcam, iphone_imu, multi_host_pair, tactile_rig, ...). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Strip the example down to ~35 lines so the three key interfaces — SessionOrchestrator, session.add(UVCWebcamStream), viewer.launch — are the most visible things on the screen. Remove the probe helper, the build_session factory, the verbose argparse descriptions, the ImportError guidance, and every explanatory comment that repeated what the code already said. The README next door keeps the hardware checklist, --probe instructions, output layout, and troubleshooting table. record.py is meant to be skimmable as "oh, it's literally three calls." Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Two stream-management details that surfaced once users started mixing
code-registered streams with the viewer's Discover modal:
1. Two cards for the same physical webcam
User registers UVCWebcamStream("mac_webcam", 0) in code, then
clicks Discover devices in the viewer, which happily constructs
a second UVCWebcamStream at index 0 under a different id
("macbook_pro"). Session ends up with 4 streams for 2 cameras.
Fix: every Stream now exposes an optional `device_key` property
returning a (adapter_type, device_id) tuple that names the
physical hardware it owns. None means "no hardware identity",
which keeps stream-id as the only uniqueness check for streams
like JSONLFileStream.
- StreamBase.device_key → None (default)
- UVCWebcamStream.device_key → ("uvc_webcam", str(device_index))
- SessionOrchestrator.add() rejects a new stream whose device_key
matches any already-registered stream, with a clear ValueError
naming both the new device_key and the existing stream id.
- scan_and_add() skips discovered devices whose
(adapter_type, device_id) is already claimed on the session.
- Discovery modal grays out the checkbox for already-owned
devices, labels them "✓ Already added as '<stream_id>'", and
excludes them from the default selection set so the common
"click Add" flow doesn't even try to re-register them.
- Defense-in-depth skip in DiscoveryModal._on_add_click catches
stale selection snapshots where something got registered
between scan and the Add button.
2. No way to remove a stream you no longer want
Added SessionOrchestrator.remove(stream_id):
- Valid in IDLE, CONNECTED, and STOPPED.
- Refuses in CONNECTING / PREPARING / COUNTDOWN / RECORDING /
STOPPING — removing a stream mid-lifecycle would leave partial
artifacts on disk.
- If CONNECTED, calls stream.disconnect() before unregistering so
hardware handles are released.
- Frees the stream's device_key so a fresh stream can re-claim
the same physical device afterwards.
Viewer integration:
- StreamCard header gets a right-aligned × button that fires
an on_remove(stream_id) callback.
- ViewerLayout injects the callback at card creation; it
dispatches SessionOrchestrator.remove on a worker thread so
device teardown never blocks the DPG render thread.
- Card.update() takes session_state and enables/disables the
button based on the same IDLE/CONNECTED/STOPPED predicate the
orchestrator enforces. Cached so DPG doesn't receive a fresh
configure_item call every frame.
- The existing _update_streams path already deletes DPG cards
for ids that disappear from the SessionSnapshot, so removal
flows through the next poller tick with no extra plumbing.
Tests (+8 on top of the 340 baseline → 348 passing):
- TestDeviceKey: default None, override returns stable tuple.
- TestAdd: rejects duplicate device_key, different device_keys add
cleanly, None device_keys fall back to id-only uniqueness.
- TestRemove: remove in IDLE, unknown id raises KeyError, rejected
during RECORDING, allowed from STOPPED, frees device_key for re-add.
Does NOT touch parallel discovery/viewer work (theme.py, app.py,
demo.py, types.py state-machine additions, _ble.py) still in flight
on the co-worker's branch.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Record the brainstormed design for the local browser-based replay viewer that lets external customers open a saved session folder and verify sync quality (Before/After toggle + per-stream offset report) without standing up the internal egonaut/web dashboard. Key decisions captured: - separate top-level subpackage syncfield.replay (mirrors viewer) - Starlette + uvicorn behind a [replay] extra - React/Vite/Tailwind SPA ported from egonaut/web's DataReviewPage - pre-built static/ committed so end users need no Node toolchain - 3D viewers and i18n explicitly out of v1 Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Two coupled UX fixes so the viewer looks and behaves like a real
recording app instead of a session-monitoring dashboard.
1. UVCWebcamStream gains a 4-phase lifecycle
Until now the capture thread only spawned inside start(), which
meant the viewer sat in CONNECTED state with empty stream cards
until the user clicked Record. That was backwards — CONNECTED is
supposed to be the "live preview before recording" phase.
Refactor:
- connect() spawns the capture thread in preview-only mode.
latest_frame begins updating immediately so the viewer card
renders the live thumbnail, but no file is written and no
SampleEvent is emitted.
- start_recording() opens the VideoWriter and flips _recording
to True; the same thread starts appending frames to the file
and emitting samples.
- stop_recording() flips _recording off and closes the writer,
but the capture thread KEEPS RUNNING so preview stays live
and the user can record again without re-opening hardware.
- disconnect() stops the thread and releases the VideoCapture.
- Legacy start()/stop() one-shot wrappers are preserved for
scripts that don't use the viewer — they build the writer
and set _recording=True BEFORE spawning the thread so the
very first read() frame lands in both latest_frame and the
file, matching the 0.1 one-shot semantics exactly.
- connect() is idempotent (thread-alive check) so the legacy
wrapper can call it unconditionally.
The capture loop now has two phases distinguished by the
_recording flag: "publish to latest_frame only" and "publish +
write + emit". One flag flip to switch — no thread restart.
2. Viewer: explicit Connect button, no auto-connect
ViewerLayout used to dispatch session.connect() on a worker
thread from build(). Removed that. The controls panel now has
three rows:
[ Connect ] — IDLE only
[ Record ] [ Stop ] — Record: CONNECTED, Stop: RECORDING
[ Cancel ] — COUNTDOWN / RECORDING
Connect is a new primary button wired to _on_connect_click,
which dispatches session.connect() on a worker thread. Device
open is slow for real hardware (UVC permission prompts, BLE
handshake, OAK pipeline warmup) so the UI thread never blocks.
Button-enable state logic:
- btn_connect: state == "idle"
- btn_record: state == "connected"
- btn_stop: state == "recording"
- btn_cancel: state in {preparing, countdown, recording}
- btn_discover: state == "idle" (was "connected")
Discover moved to IDLE because scan_and_add refuses anything
other than IDLE and add() rejects after connect — the old
"Discover from CONNECTED" behavior was a latent bug that
would crash the modal's Add button.
CONTROL_PANEL_HEIGHT bumped 168 → 212 to fit the new row with
comfortable breathing room.
Tests (+4 on top of the 348 baseline → 355 passing):
- TestFourPhaseLifecycle::test_connect_starts_preview_without_writing
- TestFourPhaseLifecycle::test_start_recording_flips_to_writing
- TestFourPhaseLifecycle::test_disconnect_stops_capture_thread
- TestFourPhaseLifecycle::test_connect_is_idempotent
Existing legacy test (test_start_stop_produces_file_path_in_report)
continues to pass unchanged.
Does NOT touch parallel in-flight work: discovery/_ble.py, types.py
state-machine additions, viewer/app.py, viewer/demo.py,
viewer/fonts.py, examples/README.md catalog updates,
examples/mac_iphone_dual_oak/ (separate contributor's recipe).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Decompose the replay viewer spec into 17 bite-sized TDD tasks across 5 phases: Python loader → Python server → Vite/React scaffold → SPA UI → build/ship/document. Each task lists exact files, full code, test code, and the commit command. ContactTimeline/TactilePanel are intentionally deferred from v1 — SensorChartPanel covers the "minimal sensor charts" requirement and those components add scope without payoff for sync-verification use. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Row 1 of the controls panel now holds two side-by-side buttons
(102 px each + 8 px gap = 212 px, same width as Cancel below):
[ Connect ] [ Disconnect ]
Exactly one is enabled per resting state, both disabled during
transitions:
IDLE → Connect enabled, Disconnect disabled
CONNECTING → both disabled
CONNECTED → Connect disabled, Disconnect enabled
PREPARING → both disabled
COUNTDOWN → both disabled
RECORDING → both disabled
STOPPING → both disabled
STOPPED → Connect enabled, Disconnect enabled
STOPPED accepts BOTH because it's the "what do I do next?" fork:
the user may want to tear the session down for good (Disconnect
→ IDLE → close the window) or start a fresh recording on the
same rig without re-opening hardware (Connect → CONNECTED →
Record). The orchestrator already supports connect() from STOPPED
and disconnect() from STOPPED, so the UI just exposes the two
paths explicitly.
_on_disconnect_click dispatches SessionOrchestrator.disconnect()
on a worker thread — the call joins each adapter's capture loop
and releases device handles, which is a few ms per UVC webcam
but up to a second per BLE peripheral, so it never runs on the
DPG render thread.
Does NOT touch parallel in-flight work (tone.py countdown beep,
discovery, viewer/app.py, viewer/demo.py, viewer/fonts.py).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…ebcam style
Drop the 265-line version with list_oak_devices helper, build_session
factory, depth flags, and verbose argparse help down to 43 lines so
the four session.add() calls are the most visible thing on the
screen. Mirror the exact layout of examples/iphone_mac_webcam/record.py:
session = sf.SessionOrchestrator(host_id=..., output_dir=...)
session.add(UVCWebcamStream(...))
session.add(UVCWebcamStream(...))
session.add(OakCameraStream(...))
session.add(OakCameraStream(...))
syncfield.viewer.launch(session)
The two maintainer-rig OAK serials stay at the top as DEFAULT_* module
constants so the maintainer's common case is zero-arg, and the CLI
flags let any other user override them.
Removed:
- list_oak_devices() — users run `python -m depthai` or the adapter's
own discovery.discover() path if they need to enumerate
- build_session() factory — inlined so the flow is one function
- --oak-lite-depth / --oak-d-depth flags — defaults (False) are fine;
anyone who needs depth edits one line
- --list-oak flag — see above
- All explanatory comments — the function names already say what
each line does
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
… dirs
Two coupled fixes prompted by real usage of examples/iphone_mac_webcam:
1. Fix basedpyright invariance error on session.add()
The Stream protocol used to declare id/kind/capabilities/device_key
as bare class attributes. Mutable protocol attributes are
INVARIANT, so any subclass with a narrower kind (e.g. what
basedpyright infers from a "video"/"audio" literal assignment)
failed the protocol check with:
Argument of type "UVCWebcamStream" cannot be assigned to
parameter "stream" of type "Stream" in function "add"
"kind" is invariant because it is mutable
"kind" is an incompatible type
Type "str" is not assignable to type "StreamKind"
Fix: declare id, kind, capabilities, device_key as read-only
@Property on the protocol so the attributes become COVARIANT —
concrete adapters are free to have kind: Literal["video"] and
still structurally match Stream.
Also added explicit class-level annotations on StreamBase for
the three identity fields. Without them basedpyright in strict
mode infers self.kind = kind as plain `str` (losing the
StreamKind narrowing from the parameter type), which broke the
protocol match even after the property switch.
Neither change touches runtime behavior — the instance
attributes are still assigned in __init__ and still satisfy
@runtime_checkable isinstance() probes via hasattr().
72 related unit tests continue to pass.
2. Example output now lands in a per-episode subdirectory
Both examples used to pass ``Path("./output")`` — a CWD-relative
path — straight through to the orchestrator, which meant every
run overwrote the previous one's MP4s and the user had to guess
which working directory the files landed in.
Refactor:
- Default output root is now ``Path(__file__).parent / "output"``,
so running from repo root vs cd'd into the example directory
lands files in the SAME place.
- Each run computes ``ep_{YYYYMMDD_HHMMSS}_{hex6}`` using
``datetime.now()`` + ``secrets.token_hex(3)`` and creates the
subfolder before constructing the session. Every recording
session gets its own isolated directory, matching the
ep_{date}_{timestamp}_{hash} pattern used by the
opengraph-studio backend's episode store.
Output layout:
examples/iphone_mac_webcam/output/
├── ep_20260409_160345_a3f21b/
│ ├── mac_webcam.mp4
│ ├── iphone.mp4
│ ├── sync_point.json
│ ├── manifest.json
│ └── session_log.jsonl
└── ep_20260409_161217_8c19d4/
└── ...
Same refactor applied to mac_iphone_dual_oak/record.py so the
two examples stay visually parallel.
Both examples stay compact: 41 lines (iphone_mac_webcam) and
49 lines (mac_iphone_dual_oak), under the "~35-50 line" budget.
The three core API calls (SessionOrchestrator, session.add(),
viewer.launch) are still the most prominent lines on screen.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Critical missing wiring: SessionOrchestrator was never registering
an on_sample handler on any stream, so every SampleEvent the capture
thread emitted via self._emit_sample() was going nowhere. Recording
sessions ended with MP4s + manifest + sync_point on disk but no
{stream_id}.timestamps.jsonl or {stream_id}.jsonl files — which
means the downstream SyncField sync service had nothing to align
against. The SDK→core handoff was silently broken.
Root cause: SessionOrchestrator.start() called
stream.start_recording(clock) on every registered stream but never
built a StreamWriter / SensorWriter, and the Stream protocol's
on_sample() callback list stayed empty for the entire session
lifetime. The writer classes exist in syncfield.writer and ARE
documented as the "SDK→core handoff" contract, they were just
not plugged in.
Fix: new _open_sample_writers / _close_sample_writers pair driving
a per-stream writer keyed off stream.kind:
- kind == "sensor" → SensorWriter ({id}.jsonl with channels)
- otherwise → StreamWriter ({id}.timestamps.jsonl)
_open_sample_writers() runs inside start() just BEFORE the atomic
start_recording() loop so the very first SampleEvent each adapter
emits under its _recording flag is already routed to disk.
_close_sample_writers() runs inside _finalize_streams() AFTER every
adapter's stop_recording() returns, so no capture-thread write
races a close() call.
Each handler closure captures a mutable [active] flag that
_close_sample_writers flips to False BEFORE releasing the file
handle. Any trailing SampleEvent from a capture thread that
hasn't quite seen the _recording=False flag yet becomes a no-op
on the handler side instead of a file-handle-after-close crash.
The rollback path in start() (start_recording raises partway
through the stream loop) also calls _close_sample_writers so
partially-opened writers don't leak.
Regression tests (+3 in TestSamplePersistence, 360 passing total):
- test_video_stream_writes_timestamps_jsonl — FakeStream pushes
three samples, orchestrator persists them to cam.timestamps.jsonl
with the right frame_number/capture_ns/clock_domain fields
- test_sensor_stream_writes_channel_jsonl — kind="sensor" routes
through SensorWriter and preserves channel dicts verbatim
- test_samples_after_stop_do_not_race_closed_writer — trailing
emission after session.stop() is silently dropped by the
handler's active flag, not written to a closed writer
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Introduces PollingSensorStream and PushSensorStream as a first-class path for attaching simple sensors without writing a full StreamBase subclass. Sensors only in v1; cameras follow the same pattern in v2. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
20-task TDD plan covering _SensorWriteCore foundation, both helpers, threading integration, e2e through SessionOrchestrator, and example recipes. Each task is one red-green-commit cycle. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…ast_at tracking Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add PushSensorStream with __init__, capabilities resolution, device_key property, and corresponding construction unit tests. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add connect/start_recording/stop_recording/disconnect methods with snapshot-before-close pattern and on_connect/on_disconnect callbacks. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add push() with auto frame numbering, explicit capture_ns/frame_number overrides, and conditional writing during recording window. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Guard push() against non-dict channels (TypeError), calls outside connect/disconnect (WARNING health event), and internal write failures (ERROR health event, never propagated to caller). Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add PushSensorStream to the top-level syncfield.adapters re-exports and __all__, matching the pattern used for PollingSensorStream. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…NL persistence
The SessionOrchestrator already creates a SensorWriter for every
kind='sensor' stream. Having the helpers also write to the same
{stream_id}.jsonl caused double-write conflicts. Helpers now only
emit SampleEvents; the orchestrator's on_sample callback persists them.
_SensorWriteCore simplified to frame counter + timing tracker.
produces_file changed from True to False.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PollingSensorStream— framework-owned capture thread that calls a userread()function at a fixedhz. Supportsopen/closelifecycle callbacks for hardware handles.PushSensorStream— thread-safepush(channels, capture_ns=None)sink for callback/asyncio/external-thread sources.on_connect/on_disconnecthooks for user-owned producer lifecycle.StreamBasesubclasses with 4-phase lifecycle, live preview duringCONNECTED, device-key dedup, and viewer parity. Nooutput_dirneeded — theSessionOrchestratorowns JSONL persistence via its existingon_samplecallback pipeline._generic.pymodule with_SensorWriteCore(frame counter + timing tracker) and capabilities helpers.Usage
Test Plan
SessionOrchestratorlifecycle with both helpers, verifies JSONL + manifest outputDesign Docs
docs/superpowers/specs/2026-04-09-generic-sensor-stream-helpers-design.mddocs/superpowers/plans/2026-04-09-generic-sensor-stream-helpers.md🤖 Generated with Claude Code