feat: platform health telemetry (Sentry-style incident tracking) - #20
Merged
Conversation
Sensor-agnostic incident tracking system — detects stream stalls, FPS drops, jitter, startup failures, writer backpressure, and adapter- reported faults via a pluggable detector registry, and surfaces them in the viewer as Sentry-style Incidents (open/close, severity, fingerprint-grouped events, attached artifacts). Motivated by OAK failure modes (X_LINK_ERROR, device crashes, reconnect) that currently vanish into stderr; bridged into the unified telemetry channel via a depthai logger handler. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
27 TDD-style tasks across 8 phases: core types + detector framework, six default detectors (stall, fps-drop, jitter, startup-failure, backpressure, adapter-passthrough), orchestrator + writer integration, OAK depthai logger bridge with crash-dump artifact capture, viewer server+frontend incident panel, per-adapter target_hz rollout, and a manual OAK-hardware verification checklist. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…ment Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…__ and Iterator types
… conditions Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
… tick loop) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Detects when a stream stops producing samples for longer than a configurable threshold. Supports per-stream independent state, deduplication while stalled, and recovery-based incident closure. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Implements p95-based inter-sample interval anomaly detector for monitoring streaming data quality. Detects jitter spikes when 95th percentile interval exceeds target by a configurable ratio, with sustain and recovery phases. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…counter) Implements detector for writer queue saturation and frame drops. Fires when queue fullness sustained above threshold, or on any drop increment. Closes when queue recovers to low levels with no new drops for the recovery window. Backdating via _first_bad_observed_at: on first observation above threshold, record that timestamp; clear when fullness drops below. This ensures the sustain window is measured from when the bad state started, not from when we first check it. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Ties together Registry + Tracker + Worker into a single user-facing facade (HealthSystem). Installs all six default detectors on construction and exposes lifecycle, observer, incident-view, and callback hooks. Populates syncfield.health.__init__ public API using lazy imports to avoid the existing syncfield.types → syncfield.health.severity circular import. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…incidents - StreamCapabilities gains `target_hz: float | None = None` inserted before `live_preview`; `to_dict()` includes the new key. - FinalizationReport gains `incidents: list = field(default_factory=list)` at the end of the field list; typed as bare `list` to avoid a circular import with syncfield.health.types.Incident. - test_types.py: update existing to_dict round-trip assertion to include the new `target_hz` key. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Extends SessionLogWriter to track incidents alongside events. Each incident state is serialized as a JSON line to incidents.jsonl and flushed immediately for crash-resilience. - Add incidents_path property and _incidents_handle file management - Implement log_incident(incident) method that mirrors log_event() - Update open()/close() to manage incidents file lifecycle Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…, health, state, incidents) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Wrap tracker.ingest() calls in _safe_ingest() to prevent malformed events from crashing the worker thread - Log warnings instead of silently exiting on IncidentTracker.ingest() failures - Update FakeStream.push_health() to provide non-empty fingerprint, severity, and source so test events flow through the tracker cleanly - Eliminates PytestUnhandledThreadExceptionWarning in TestHealthRouting Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Translates depthai Python log records into HealthEvent payloads. Parses five fingerprint patterns (xlink-error, device-crash, reconnect-attempt, reconnect-success, connection-closed), with fallback to unparsed-log. Installed on depthai logger; owned by AdapterEventPassthrough detector. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…et_hz threading Fix 1: Convert IncidentTracker/HealthSystem incident callbacks from single-slot Optional[Callable] to list-based multi-listener API (add_on_opened/updated/closed on tracker, on_incident_opened/updated/closed method-call registrars on system). SessionPoller no longer clobbers SessionOrchestrator's persist listener. Fix 2: Wire _emit_writer_stats in orchestrator.add() so BackpressureDetector receives WriterStats on every stream at ~10 Hz (throttled per stream_id). Fix 3: HealthSystem maintains _target_hz_by_stream dict; register_stream(id, hz) is called from orchestrator.add(). FpsDropDetector and JitterDetector now receive a target_getter closure that reads from this dict instead of returning None always. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…rt search) Replace _find_dip_start binary-search with the simple pattern used by JitterDetector: record dip_began_at = now_ns on first low-FPS tick, then check elapsed duration on subsequent ticks. The existing sustain_ns gate already debounces spurious fires, making the retroactive search redundant. Also remove the dead _last_observed_state field and the now-unused _observed_fps_from helper. Update unit tests to tick repeatedly so the sustain_ns threshold can be crossed progressively. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The Tests workflow was installing the package with `pip install -e .`, but pyproject.toml force-includes `src/syncfield/viewer/static` (the built frontend bundle), which is gitignored. CI therefore failed at the install step with "Forced include not found" on every run — pre-dating this PR. Mirror the frontend-build step from publish.yml so the wheel build finds the bundle. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Tests that import MetaQuestCameraStream, OakCameraStream, Insta360Go3S, and the BLE IMU adapter fail at collection time without httpx / depthai / bleak / aiohttp / zeroconf. `[dev]` alone ships only the base install. Add `[all]` to pick up every optional adapter dependency. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…tra) The prior fallback (`.[dev]` → fail → `pip install pytest`) was removed, so CI's pytest command turned up exit 127 (command not found). `dev` lives in PEP 735 [dependency-groups] which `pip install .[extra]` doesn't resolve. List the pytest stack explicitly — kept in sync with pyproject.toml [dependency-groups].dev. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Test uses `float | None` at runtime (FakeStream constructor default), which requires Python 3.10+ without the __future__ import. The project declares `requires-python = ">=3.9"` so we need the shim. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Three pre-existing test files use PEP 604 union syntax (`X | None`) at runtime in type annotations. `from __future__ import annotations` makes all annotations strings and keeps them 3.9-compatible. Drive-by since we're already adjusting CI; pyproject declares `requires-python = ">=3.9"`. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Extends the health-telemetry platform (2026-04-22 spec) with: partial
connect (per-stream state machine, survivors proceed on single-stream
failure), structured startup-failure HealthEvent emission that activates
the previously-dormant StartupFailureDetector, a new NoDataDetector for
"connected but never emitted a sample" case (the observed OAK black-
screen symptom), state-aware StreamCard overlays (connecting / waiting /
failed), and a degraded-state header chip ("Ready (3/5)").
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
18 TDD tasks — Detector.observe_connection_state hook, HealthWorker queue, HealthSystem passthrough, NoDataDetector + default registration, orchestrator per-stream state dicts + partial-connect rewrite + disconnect cleanup, StreamSnapshot fields + poller + server serializer, frontend types + overlay components + StreamCard branch + header chip, integration tests for partial-connect and no-data. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds a no-op default for the per-stream connection-state observer that NoDataDetector will override. Protocol + base mirror the existing observe_sample / observe_health / observe_state shape. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
New _ConnectionStateMsg + push_connection_state() + _drain_once fan-out so NoDataDetector (next commit) can track per-stream CONNECTING → CONNECTED transitions via the standard observer pattern. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Passthrough to the worker's ingress queue, matching the existing observe_sample / observe_health / observe_state shape. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Fires when a stream is in 'connected' state for N seconds without any sample. Complements StreamStallDetector which requires prior samples. Catches the OAK "black square" symptom where connect() succeeds but the pipeline never pumps frames. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Now installed automatically on HealthSystem construction alongside the other six default detectors. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
_stream_states and _stream_errors become the source of truth for per-stream status. _set_stream_state is the single update point — it also forwards to HealthSystem.observe_connection_state so NoDataDetector sees the transitions. add() initializes each new stream to 'idle'. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…ilure One stream raising no longer rolls the whole session back to IDLE. We record the error, emit a structured HealthEvent (phase=connect, outcome=error) that activates the previously-dormant StartupFailureDetector, and continue with the rest. Session only fails if every adapter raises. Success path emits the complementary outcome=success signal so the detector's recovery tracking works. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…antics Tests in TestStartRollback that encoded the old all-or-nothing rollback are now inverted: a single failing stream is isolated, not session-ending. Assertions updated to reflect that the failed stream goes to 'failed' state while the rest proceed to recording. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
After partial-connect, failed streams have no open hardware. Iterating them in start_recording / stop_recording would call lifecycle methods on devices that aren't open — at best a no-op, at worst a cascading error. Switch those loops to iterate self._connected_streams (set by connect() to only the successful streams). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…nected' _connected_streams only ever held successful streams, so the existing rollback helper is already correct — but every stream the session knows about (including previously-failed ones) should have its snapshot state flipped to 'disconnected' and its error entry cleared so the viewer renders a clean slate after teardown. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Defaults ('idle' / None) preserve existing behavior for any construction
that doesn't set them. Poller (next task) wires the orchestrator's per-
stream state into these fields.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Every per-stream StreamSnapshot now carries connection_state and connection_error, sourced from the orchestrator's per-stream dicts populated by _set_stream_state. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Frontend consumes these to select the right StreamCard overlay and to populate the degraded-state header chip. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
ConnectionState enum and two new fields on StreamSnapshot so the overlay branching in StreamCard (next commit) is type-safe. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Three small presentational components used by StreamCard (next commit) to replace the browser broken-image fallback. FailedOverlay is click- to-expand for long error messages. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Connecting / waiting-for-first-frame / failed now render explicit overlays instead of the browser's broken-image placeholder. Healthy streams still render the existing video / sensor / audio body. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
'Ready (3/5)' in yellow when one or more streams are not in 'connected' state. Normal tone when everything is healthy. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…ence Real SessionOrchestrator, mix of passing + failing FakeStreams; asserts the session reaches CONNECTED, the failed stream surfaces as an open startup-failure incident, and the fingerprint lands in incidents.jsonl. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
SilentFakeStream connects but withholds samples until the test opens its gate. Asserts the no-data incident opens within 1.5s of connect and closes within one tick of samples resuming. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Platform-level, sensor-agnostic health telemetry for syncfield — automatically detects recording-time failures (stream stalls, FPS drops, jitter, startup failures, writer backpressure, adapter faults), groups them into Sentry-style Incidents with severity + open/close lifecycle, and surfaces them live in the viewer and in post-session
incidents.jsonl/FinalizationReport.incidents.OAK's native depthai logger is bridged into the unified channel, so
X_LINK_ERROR/ device crash / reconnect events no longer disappear into stderr — they become structured incidents with crash-dump paths attached as artifacts.docs/superpowers/specs/2026-04-22-health-telemetry-design.mddocs/superpowers/plans/2026-04-22-health-telemetry.mdChanges
src/syncfield/health/(new package):Severity,HealthEventenrichment,Incident/IncidentArtifact/IncidentSnapshot/WriterStats,Detectorprotocol +DetectorBase,DetectorRegistry,IncidentTracker,HealthWorker(daemon thread + lock-free ingress),HealthSystemfacadeAdapterEventPassthrough,StreamStallDetector,FpsDropDetector,JitterDetector,StartupFailureDetector,BackpressureDetector— all installed automaticallyDepthAILoggerBridge:logging.Handlerthat translates depthai native logs intoHealthEvents; auto-installed inOakCameraStream.connect()SessionOrchestrator: constructsHealthSystem, wires per-stream sample + health observers, forwards state transitions, runs worker acrossconnect/disconnect, emitsWriterStatsat 10 Hz, attachescrash_dumpartifacts for:device-crashfingerprints, embeds incidents inFinalizationReport.incidentsat stopSessionLogWriter: newlog_incident()+incidents.jsonlsidecarStreamCapabilities.target_hz: declared on UVC, Meta Quest camera, OAK, host audio, polling sensor (others deliberately unset →FpsDropDetectorfalls back to baseline-learning)SessionSnapshot.active_incidents/resolved_incidentsreplacehealth_log/health_count/problem_count; poller subscribes toHealthSystemcallbacks; server serializes incidents to WebSocketIncidentPanelcomponent (replacesHealthTable), per-stream severity badge onStreamCard,Severity/IncidentSnapshot/IncidentArtifactTypeScript typesTest Plan
tests/unit) — 865 passingtests/integration/health) — stall + recovery,incidents.jsonlwritten, poller does not clobber persist listener, writer stats flow toBackpressureDetectortsc --noEmit) and bundle build (npm run build)crash_dump.jsonartifact, (c)IncidentPanelrenders Active → Resolved transitions, (d)incidents.jsonlcontains the fingerprintsBreaking changes
session_log.jsonlformat changed (new fields onHealthEvent:severity,source,fingerprint,data). No migration shim — consumers must update.SessionSnapshotno longer carrieshealth_log,StreamSnapshot.health_count, orStreamSnapshot.problem_count. The WebSocket payload shape changed accordingly. No external consumers at time of change.🤖 Generated with Claude Code