Skip to content

debugability.md

Latest commit

 

History

History
134 lines (116 loc) · 7.5 KB

debugability.md

File metadata and controls

134 lines (116 loc) · 7.5 KB
Raw

Debugability — triggering scenarios on demand

Every lever for REPRODUCING a conversational/pipeline scenario without waiting for it to happen naturally. Companion to observability.md (which covers how to SEE what happened). Station base: http://<station>:8099.

Contents

Inject utterances (no mic)

  • POST /api/brain/:bot/debug/say {text} — tap + addressed final in one shot → a REAL turn (LLM runs, the bot speaks/moves). The basic "make it do a turn".
  • POST /api/brain/:bot/debug/hear {text, [startedAt], [endedAt], [confTier]} — a final with NO tap: exactly what a live mic final looks like. Exercises skip:not-addressed, queue:busy (mid-turn), merge:supersede (while thinking), stop-intent ("stop"/"wait"), wake matching, skip:garbage (confTier:"garbage"). Response echoes the resulting conversation state.
  • POST /api/brain/:bot/debug/event {event, [endedAt], [text]} — inject a raw conversation event (tap etc.) to drive the state machine directly.
  • adb (debug builds): adb shell am broadcast -a dev.orbit.dock.debug.SAY -e text '…' — a turn driven from the phone side (DebugTestReceiver).
  • POST /api/perception/debug/speech {bot, text, [who], [addressed]} — inject a SPEECH SNAPSHOT into the perception ring, exactly as the STT processor emits one. This is a different seam from the two above, and the distinction bites: debug/say and debug/hear inject a turn straight into the BRAIN and never touch the perception feed. So anything reading the feed — the rolling summarizer, the durable memories it notices, grounding, the enricher — sees nothing from them. Three "memory is broken" investigations were chased before that was understood; the summarizer reporting "offered 0" was correct, it had never been shown the sentence. Simulates the STT OUTPUT only: VAD, endpointing, the engine and its confidences are upstream and NOT exercised. Use the real-audio path below for those.
  • All injected paths are TAGGED in the UIs (💉 debug-api / phone-debug, via utteranceId debug:<ts> / via phone:turn-request) so they never read as a real human utterance later.

Real-audio scenarios (the honest path)

  • Laptop speaker near the bot: say "hey orbit, …" (macOS TTS) → real mic → SFU → server VAD → STT → brain. The only path that exercises endpointing, voiced-fraction, echo, and barge for real.
  • Barge-in: start a long reply (debug/say {"text":"tell me a long story"}), then say "stop" mid-playback — watch vad:onset → barge:hold → stop/resume/ yield on the timeline.
  • Skills (this repo, for Claude sessions): stress-dock-pipeline — hammer the pipeline with real audio + adb taps, judge from station ground truth; run scenarios MANY times (real bugs are intermittent). two-dock-converse — full spoken conversation from the laptop.
  • Verification rule (docs/memory): sign-off = the USER's modality. Injected paths prove mechanism only; a fix "works" when a human try works.

Conversation-state levers

  • Tap/palm: tap the bot’s face; or debug/event with a tap. Palm = open-only.
  • POST /api/brain/:bot/debug/tts-hold {hold:bool} — drive the barge pause (tts-hold frame) directly, no audio needed.
  • POST /api/brain/:bot/quiet {on} — 🤐 quiet mode (skip:quiet path).
  • POST /api/brain/:bot/think {text?} — poke a self-thought turn.
  • POST /api/brain/:bot/session/end — close the session lane.
  • GET /api/brain/:bot/conversation — live mode/window probe while driving.
  • GET /api/brain/:bot/listening — the interim listening gate state.

Replays (re-run recorded reality)

  • Turn replay: POST /api/brain/:bot/replay {sessionId, turnId} (or obs ▶) — re-drives a recorded turn's responses through the LIVE pipeline, no LLM: isolates sequencing/TTS/body from model behavior.
  • Perception takes: Studio /#perception takes save/load/replay (/api/perception/takes*) — A/B replay recorded A/V through changed perception code/prompts.
  • Utterance clips: every clip (incl. dropped) at GET /api/perception/utterance-audio/<bot>/<audioStartMs> — re-feed to the STT sidecar (POST :8078/transcribe) to re-judge a transcript offline.
  • Capture harness: /#capture + capture module — record a window of clean ambient perception (bot responses suppressed; brain traces skip:recording).

Face / body / vision probes

  • GET /api/brain/:bot/debug/face — what the phone face is ACTUALLY showing; POST {expression} forces a mood (no LLM).
  • POST /api/brain/:bot/debug/face/film {count, everyMs} — sample the face over time → JPEGs under var/debug/face// (state-vs-render races).
  • Body: POST /api/bodylink/command (raw servo), /api/bodylink/play (sequence), GET /api/bodylink/state, /holder (lease), /health-check.
  • Live camera frame: GET /api/perception/frame/<peerId> (see memory: ota-auto-apply); vision turn input frames land in obs (📷).

Kill-switches & thresholds (config console /#config or env)

  • brainBargeHold=false — disable the barge reflex (v2: duck-on-onset + cut-on-sustained; key name predates the duck).
  • BARGE_SUSTAIN_MS (700) — cumulative loud ms that CONFIRMS a barge (cuts the reply); BARGE_DUCK_RELEASE_MS (1500) — un-duck timer when no confirm comes.
  • STT_MIN_CONFIDENCE (0.90) — finals under this are recorded + shown as dropped · low-confidence in the Studio but never become turns.
  • brainVoiceStop=false — disable reflex voice-stop/dismiss.
  • brainThinkingMerge=false — disable merge-supersede while thinking.
  • STT_ECHO_GATE=1 — re-enable the echo mute (kills barge audio; default off).
  • STT_MIN_VOICED_PCT (35, while the bot speaks) / STT_MIN_VOICED_PCT_IDLE (10, otherwise) — the two-tier voiced-fraction gate; 0 disables. Re-judge a tuning by transcribing the kept dropped clips (2026-07-23 sweep = baseline).
  • STT_SILENCE_RMS, STT_ENDPOINT_MS, STT_MIN_UTTERANCE_MS, STT_ONSET_RMS, STT_ONSET_SUSTAIN_MS — VAD/endpoint/barge-onset floors.
  • CONV_LISTEN_MS, CONV_FOLLOWUP_MS, CONV_GRACE_MS, CONV_FACE_ARRIVAL=1 — window timings / wake-on-look.
  • Perception enricher paths: console toggles (speech / non-speech triggers); PERCEPTION_ENRICH_SAVE=1 dumps enricher WAVs to listen back.

Passive monitoring (no driving)

To READ what already happened rather than trigger something new — the ~daily health routine, thresholds, and the results log — see monitoring/README.md.

End-to-end sanity

  • cd brain && npm run smoke — fake phone + body peers against a live station (no hardware).
  • npm test --workspace server — unit suite (conversation-state, stop-intent, busy-queue, addressed, obs, cost…). cd bot-desk-d1/app && ./gradlew test.
  • After any scenario: read it back on /#timeline or pull GET /api/observability/incident?bot&from&to&format=md and hand the markdown to an LLM.