Share the sound of the controlled computer - #2
Open
math65 wants to merge 5 commits into
Open
Conversation
The controlling computer can now hear what the controlled one plays - music, videos, alert sounds, the other side of a call - alongside the shared screen or on its own. NVDA+Control+Shift+J starts and stops it, and the controlled computer is asked before anything is sent. The sound travels as a WebRTC audio track on the peer to peer session screen sharing already establishes, so it never goes through the relay. A sound only session sends no picture at all. The sound is captured with the screen reader of that computer left out, by a small native helper using the process loopback mode Windows exposes from build 20348. No web API reaches that mode, so the helper serves the samples to the signalling page over a loopback WebSocket and the page rebuilds a track from them. It repeats the protections of the existing local bridge - loopback only, a port chosen by the system for each session, a token compared in constant time and a strict origin check - with the token passed on standard input rather than on the command line, which other processes can read. Leaving the screen reader out is what lets the forwarded speech keep being spoken while the sound flows. It reaches the listener faster than the audio stream and with their own synthesiser, voice and rate. On an older Windows, or when the helper cannot start, the browser captures the whole mix as before and the forwarded speech is silenced instead, so that the remote screen reader is not heard twice. Which of the two applies is answered to the peer, and a peer that predates this says nothing, which reads as the browser capture and keeps the previous behaviour. A connection that drops without ending the session now hands the forwarded speech back for as long as the interruption lasts. It used to leave the user with neither the sound nor the speech, and no message, for as long as it took to turn into a failure. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01AV4tMCFa9DeRWUSGbKuCfA
The repository refuses files over 500 kB, through check-added-large-files in the pre-commit configuration, and a statically linked C++ binary sails past that without trying: the helper weighed 749 kB. Optimising for size rather than speed, dropping exceptions and RTTI, giving each function its own section and letting the linker discard the unreachable ones brings it to 256 kB. Nothing it does changes: excluding NVDA from the capture, the loopback WebSocket and its checks, and the offer the page then produces were all measured again against the smaller binary. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01AV4tMCFa9DeRWUSGbKuCfA
The controlled computer only ever asked for them once, when it joined the channel, while stop() drops the ones the finished session used. Every session after the first therefore started with an empty list and gathered nothing but host candidates, so it could only ever link up with a computer already on the same network. Two computers on different connections negotiated on local addresses alone and never linked up, with nothing said anywhere about why. The controlling side already asked again for each session, and says why in a comment: the relay only hands the servers to clients which asked, and the credentials expire. The controlled side simply never followed that rule. Ask again when a request arrives, before putting the question to the user: the time taken to answer a dialog is far more than the relay needs to reply. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MNU1WA7oTJLVQhrZXzsoY3
Opus is negotiated in mono unless the session description says otherwise, and a browser never says so on its own. The sound was captured in stereo, fed to the encoder in stereo, and given a ceiling of 128 kbit/s meant for music, only to be mixed down to one channel before it left. The comments here and the readme both described a stereo stream that never existed. Add "stereo=1", which tells the other end this one can decode two channels and is what makes it encode them, along with "sprop-stereo=1" and the ceiling, to the Opus parameters of every description this end sends. The payload number is read from the rtpmap line rather than assumed, existing parameters are kept, and a description offering Opus without any parameters gets a line of its own. Both computers run this, so both directions carry two channels. One running an older build announces nothing and keeps the mono it understands. Measured on the page itself: the parameters come back parsed and reordered by the browser, and the offer now carries stereo=1 and sprop-stereo=1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MNU1WA7oTJLVQhrZXzsoY3
The add-on ships bin/nvda_audio_capture.exe, but not the code it is built from, so the one binary in the package that is neither vendored nor third-party was also the only one nobody could rebuild or read. The repository already answers this question the other way round: url_handler.exe sits next to url_handler.cpp. Add the source, its CMake build and a README covering why the helper exists, how to rebuild it, and how to confirm on a single machine that the exclusion works. The build options were already commented; the README explains the program itself. Its messages were written in French while the rest of the repository is in English. Translate them, and rebuild the binary so that what ships matches the source that is now next to it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01UonSBbAsT8n34n5aH14puS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds optional peer to peer sharing of the sound the controlled computer plays,
alongside the existing screen sharing or on its own. Toggled with
NVDA+Control+Shift+J.
What it does
The controlling computer hears the whole mix of the controlled one — music,
videos, alert sounds, the other side of a call — with that computer's own screen
reader left out, so what arrives is the sound of the machine rather than a
recording of NVDA talking over it. The speech it reports separately keeps being
spoken through the relay, with the synthesiser, voice and rate of whoever is
listening.
Sound can be used without any picture. When none is asked for, none is captured
or encoded, so a sound only session costs a fraction of a shared screen.
Consent and privacy
The controlled computer is always asked before anything is sent, and the answer
lasts only for the session. Adding or removing the sound changes what is being
asked for, so its user is asked again. A new setting, enabled by default,
lets a computer refuse outright: when it is cleared the request is denied
without anyone being asked, and clearing it does not affect screen sharing. Nothing
is ever recorded at either end, and the stream travels directly between the two
computers.
Leaving the screen reader out
This is done by a small helper shipped with the add-on, which uses the process
loopback capture Windows gained in Windows 10 21H2 — an API a web page cannot
reach, which is why the browser capture alone cannot do it.
On an older Windows, or if the helper cannot start, the add-on falls back to the
browser capture as before, screen reader included. The speech that computer
reports is then not spoken a second time, so it is heard as it really sounds
rather than announced twice. A second setting controls that. Braille is
unaffected throughout.
If the link drops without the session ending, the sound stops but the speech is
handed back for as long as the interruption lasts, announced by Sound
interrupted and then Connection restored.
About the helper binary
bin/nvda_audio_capture.exeis built fromnative-audio-capture/, included inthis branch with its CMake build and a README covering why it exists, how to
rebuild it and how to confirm the exclusion on a single machine — following the
precedent of
url_handler.cppnext tourl_handler.exe. It is linkedstatically, needs nothing installed alongside it and stays around 256 kB, under
the
check-added-large-filesceiling.Testing
No automated tests exist in this repository, so this was exercised by building
the add-on and installing it in NVDA: between two instances on one machine, and
between two machines through a relay. The helper's capture modes were also
checked on their own, with and without the exclusion, using the measure mode
documented in its README.
protocol.mddocuments the added messages and capability.readme.mddocumentsthe feature, and the French catalogue is updated.
🤖 Generated with Claude Code
https://claude.ai/code/session_01UonSBbAsT8n34n5aH14puS