Homework voice mode: Nova Sonic STT + TTS (#64) - #65
Closed
tpaulshippy wants to merge 3 commits into
Closed
Conversation
Backend: - NovaSonicService drives Amazon Nova Sonic (amazon.nova-sonic-v1:0) over Bedrock InvokeModelWithBidirectionalStream for turn-based speech-to-text and text-to-speech, with WAV/PCM handling and ffmpeg fallback for compressed uploads - POST /api/chats/<id>/voice: gated by Bot.enable_voice AND Profile.voice_enabled (403 otherwise), short-circuits on daily cap before STT/LLM (429), stores transcript as user message with meta.voice_input, runs the normal agent path in voice mode (spoken answers under 80 words), returns base64 WAV of spoken reply - Safety service checks transcripts and blocks unsafe turns - STT/TTS metered into the daily cost cap via Message.voice_cost with per-minute rates configurable in settings Frontend: - Hold-to-talk voice input (expo-audio) replacing the composer when voice is enabled for both selected bot and profile; recording timer, cancel button, TTS autoplay toggle - Parent controls: Enable Voice switches in bot editors, Allow Voice switch in profile editor; mic permission copy in app.json Tests cover the issue's success list: 403 gating, mocked-STT message creation, TTS audio response, safety block, over-limit short-circuit.
The voice tests that exercise the agent path constructed a real ChatBedrock client, which fails AWS credential validation in CI where no AWS region is configured. Patch ChatBedrock in those tests so the agent path is mocked, matching how test_chat.py injects a mock client.
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements issue #64 (Homework Voice Mode) using Amazon Nova Sonic (
amazon.nova-sonic-v1:0) as the speech engine, replacing the Transcribe/Polly path originally sketched in the issue.Backend (Django)
NovaSonicServicedrives turn-based speech-to-text and text-to-speech over BedrockInvokeModelWithBidirectionalStream(asyncaws-sdk-bedrock-runtime). WAV/PCM handled natively; compressed uploads fall back to ffmpeg.POST /api/chats/<id>/voice: gated byBot.enable_voiceandProfile.voice_enabled(403 otherwise), short-circuits on the daily cost cap before STT/LLM (429), stores the transcript as a user message withmeta.voice_input, runs the normal agent path in voice mode (spoken replies kept under 80 words), and returns a base64 WAV of the spoken reply.safety.pychecks transcripts before the agent runs.Message.voice_cost(per-minute rates are configurable in settings).Frontend (Expo)
VoiceInput.tsx, expo-audio) replaces the composer when voice is enabled for both selected bot and profile, with a recording timer, cancel button, and TTS autoplay toggle. Mic permission added toapp.json.Tests
Covers the issue's success list — 403 gating, mocked-STT message creation, TTS audio response, safety block, and over-limit short-circuit. 92 backend tests pass; 55 frontend jest tests pass; ruff and lint clean.
Test plan
pytestinback/(92 passing)npm run lint && npm testinfront/(lint clean, 55 passing)NOVA_SONIC_*env vars + AWS creds on the server, enable voice on a bot and profile, and try hold-to-talk on a deviceCloses #64