Homework voice mode: Nova Sonic STT + TTS (#64) - #66
Open
tpaulshippy wants to merge 2 commits into
Open
Conversation
Backend: - NovaSonicService drives Amazon Nova Sonic (amazon.nova-sonic-v1:0) over Bedrock InvokeModelWithBidirectionalStream for turn-based speech-to-text and text-to-speech, with WAV/PCM handling and ffmpeg fallback for compressed uploads - POST /api/chats/<id>/voice: gated by Bot.enable_voice AND Profile.voice_enabled (403 otherwise), short-circuits on daily cap before STT/LLM (429), stores transcript as user message with meta.voice_input, runs the normal agent path in voice mode (spoken answers under 80 words), returns base64 WAV of spoken reply - Safety service checks transcripts and blocks unsafe turns - STT/TTS metered into the daily cost cap via Message.voice_cost with per-minute rates configurable in settings Frontend: - Hold-to-talk voice input (expo-audio) replacing the composer when voice is enabled for both selected bot and profile; recording timer, cancel button, TTS autoplay toggle - Parent controls: Enable Voice switches in bot editors, Allow Voice switch in profile editor; mic permission copy in app.json Tests cover the issue's success list: 403 gating, mocked-STT message creation, TTS audio response, safety block, over-limit short-circuit.
The voice tests that exercise the agent path constructed a real ChatBedrock client, which fails AWS credential validation in CI where no AWS region is configured. Patch ChatBedrock in those tests so the agent path is mocked, matching how test_chat.py injects a mock client.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements GitHub issue #64 (Homework Voice Mode) using Amazon Nova Sonic for speech-to-text and text-to-speech. Supersedes #65 (whose CI check suite became wedged by rerun attempts); this PR carries the same fix on a fresh commit SHA so checks run cleanly.
Verified locally: 92 backend pytest tests pass, ruff clean, 55 frontend jest tests pass.
Closes #64.