Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,11 @@ npm run build:native-dev # build the dev native core (needs ZOOM_SDK_DI
dotnet build native-shell/CoreVideoPro.WinUI/CoreVideoPro.WinUI.csproj -c Release -p:Platform=x64
```

`ZOOM_SDK_DIR` must point at the staged Zoom SDK x64 dir for the full core. `npm run app`
`ZOOM_SDK_DIR` must point at the staged Zoom SDK x64 dir for the full core: **Meeting
SDK 7.1.5.43953** since Comms (talkback controller), e.g.
`C:\Users\walla\Downloads\zoom-sdk-windows-7.1.5.43953\zoom-sdk-windows-7.1.5.43953\x64`.
`scripts/stage-zoom-sdk.ps1` restages `native-core/zoom-runtime` (a 7.0.5 runtime lacks
the talkback header, so it is replaced; do not copy an old worktree's runtime). `npm run app`
runs `scripts/app.ps1`; the dev launcher is `scripts/run-studio.ps1` (now respects a
pre-set `COREVIDEO_ZOOM_ENGINE_PATH`).

Expand Down Expand Up @@ -196,6 +200,7 @@ phrase they quote.
|---|---|
| [`docs/reference/beta-release-runbook.md`](docs/reference/beta-release-runbook.md) | cutting, tagging or publishing a beta |
| [`docs/reference/browser-sources.md`](docs/reference/browser-sources.md) | browser (URL) sources, WebView2 host |
| [`docs/reference/comms-talkback.md`](docs/reference/comms-talkback.md) | the **Comms** page (Zoom talkback, the ZComms station): engine `EngineComms` kept byte-identical with CoreVideo Edge, core relay `ZoomCommsRelay` (comms-* answered lock-free, key-ups urgent), `zoom-comms-state` facts, `CommsViewModel` / `CommsPage` / `TalkKey`, leases + release-all, the **Private comms** toggle (OFF default; ON = silent virtual mic via MicGuard, OFF leaves the operator muted), on air = `StudioAirBrush`, `scripts/probe-comms.mjs` |
| [`docs/reference/core-on-air-policy.md`](docs/reference/core-on-air-policy.md) | handshake, Engine on, Record or Join gating; a stub adapter left in a real core |
| [`docs/reference/core-restart-drill.md`](docs/reference/core-restart-drill.md) | core crash recovery, respawn, Zoom rejoin |
| [`docs/reference/d3d-device-loss.md`](docs/reference/d3d-device-loss.md) | D3D devices, shared textures, device-removed handling |
Expand Down
6 changes: 3 additions & 3 deletions docs/operator-validation-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,17 @@ Current native proof surface:
| Node.js 22+ | `node --version` |
| Visual Studio 2022+ or 2026+ (C++ workload) | `scripts/test-native.ps1` finds `VsDevCmd.bat` |
| CMake | `cmake --version` |
| Zoom Meeting SDK **7.0.5** x64 | Archive with `x64/h/zoom_sdk.h` and `x64/lib/sdk.lib` |
| Zoom Meeting SDK **7.1.5** x64 | Archive with `x64/h/zoom_sdk.h` and `x64/lib/sdk.lib` |
| FFmpeg runtime for RTMP (optional) | `ffmpeg.exe` on `PATH`, or `COREVIDEO_FFMPEG_BIN_DIR` points to a `bin` folder with `avformat*.dll` |

## 2. Stage Zoom SDK

```powershell
cd C:\path\to\CoreVideoPro
# Repo-local SDK (auto-discovered when present):
# ZoomSDK\zoom-sdk-windows-7.0.5.39292\x64
# ZoomSDK\zoom-sdk-windows-7.1.5.43953\x64
.\scripts\stage-zoom-sdk.ps1
# Or override: $env:ZOOM_SDK_DIR = "C:\path\to\zoom-sdk-windows-7.0.5.39292\x64"
# Or override: $env:ZOOM_SDK_DIR = "C:\path\to\zoom-sdk-windows-7.1.5.43953\x64"
```

Staged target: `native-core/zoom-runtime/windows/x64`. Settings -> Zoom readiness should show SDK runtime checks passing after a dev build.
Expand Down
158 changes: 158 additions & 0 deletions docs/reference/comms-talkback.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
# Comms: Zoom talkback (the ZComms station in Pro)

_Added 2026-11-09 (branch `codex/comms-talkback`). Design shared with CoreVideo
Edge: Edge `docs/superpowers/specs/2026-11-09-comms-design.md`._

The **Comms** page lets the operator cue talent over Zoom talkback from Pro
itself: one grid of person cells where a cell IS a talk key, ALL CALL above it,
LATCH / EDIT TALENT / SETTINGS / LEAVE below, lamps LINK / MTG MIC / CHANNEL /
TX and a one-line ops strip. It is the ZComms talkback station in Pro's look.
Intercom-in-Zoom is a product pillar (marketing positioning); keep it native.

## Where it runs (forced by Zoom)

Talkback is sent by a participant that is **host or co-host**, and the SDK is
one participant per process. So the station is Pro's own Zoom engine
(`native/zoom-engine`, `EngineComms` + `engine/comms/**`), the same participant
that receives video. Consequences:

- Pro's Zoom account must be host or co-host; the page says so otherwise and
video keeps working.
- The engine builds against **Meeting SDK 7.1.5** (talkback controller).
- Guests need the Zoom desktop/mobile client (web cannot receive talkback;
they are listed as NO TALKBACK with the fix) and must be in the station's
room (no cross-breakout delivery; this slice is not breakout-aware).
- **Pro joins are never capture-only**, so the meeting mic follows the
operator's **Private comms** toggle (strip button; owner ruling 2026-11-10;
persisted as `ProductionOutputPreferences.CommsPrivate`, **OFF by default**):
- **OFF**: the operator's own mic is left exactly as they set it. Talkback is
delivered only while the meeting mic is OPEN (ZComms law 1), and the room
hears that mic too. MTG MIC shows it; a muted mic is said in the ops log.
- **ON**: the meeting mic becomes the never-fed silent virtual mic, exactly
Edge's capture-only path (`comms/mic_guard`): the operator is muted at once,
the virtual mic is installed only with `HasRawdataLicense()`, the mic is
unmuted ONLY after `onMicInitialize`, no initialise within 5 s = stay muted
with a "Talkback is silent: …" reason, a drop = release ownership then mute.
The room never hears the operator; they speak only through talkback.
- **ON -> OFF**: `setExternalAudioSource(nullptr)` and the operator is left
**MUTED**, never auto-unmuted; the ops log says so. A meeting ending also
removes the silent mic on a non-capture-only station.
- The shell re-sends the preference whenever a report disagrees (a new engine
starts OFF), at most once per 2 s. The state reports `private` (Edge, always
capture-only, always reports `true`; `comms-private` changes nothing there).
- Comms failures never touch subscriptions, ISO, raw media or outputs.

## Data path

```
CommsPage (WinUI) ── CommsViewModel ── IZoomCommsTransport (MediaCoreBridgeService)
│ press/release, digits, Space │ SendCommsAsync(verb, payload)
▼ ▼
MediaCoreSupervisor.SendCommsAsync ──stdin──▶ core JsonRpcServer (comms-*: LOCK-FREE,
no coreMutex, no assembly slot)
→ MediaCore::zoomCommsCommandUnlocked
→ ZoomEngineRuntime::commsCommand
(ZoomCommsRelay validates; sender queue)
──pipe──▶ engine comms_* line
engine comms_state ──pipe──▶ ZoomEngineRuntime (re-keyed camelCase, held latest)
→ drainFrameEvents "zoom-comms-state" (zoom pump, no coreMutex; repeated 1/s
while held because that lane may drop) ──stdout──▶ supervisor
→ ZoomCommsStateReceived → StudioViewModel (UI thread) → CommsViewModel.ApplyComms
```

`comms-key`, `comms-latch`, `comms-release-all` and `comms-private` are **urgent** in the core
mailbox (`CommandMailbox::urgent`) and on the EngineHost wire, so a key-up is
never refused for capacity.

## Wire contract (shell ↔ core)

Requests are ordinary core requests: `{"id","type":<verb>,"payload":{...}}`.
The verbs and their fields are **CoreVideo Edge's** (`docs/PROTOCOL.md`
"Comms" in Edge), relayed by the same code (`src/modules/ZoomCommsRelay` is
Edge's `CommsRelay` verbatim):

| type | payload |
|---|---|
| `comms-key` | `{"slot":0..15}` or `{"all":true}`, `"on":bool`, optional `"leaseMs":0..60000` |
| `comms-latch` | same as key, no lease |
| `comms-assign` | `{"slot","participantId","on"}` |
| `comms-mic` / `comms-sidetone-out` | `{"device":"<name>"}` (`""` = system default) |
| `comms-sidetone` / `comms-tone` | `{"on":bool}` |
| `comms-gain` | `{"db":-20..20}` |
| `comms-release-all` | `{}`: every held key and ALL CALL up; latches stay |
| `comms-private` | `{"on":bool}`: Private comms (above). OFF is never refused |

Answer: `{"ok":true,"type":<verb>}` or failure code `comms-refused` with the
operator text in `error.message` (shown verbatim in the ops strip). Releasing
(`on:false`, release-all) is never refused, even with no engine.

State: unsolicited `{"type":"zoom-comms-state","comms":{...}}`; `comms` is the
engine's last `comms_state` re-keyed to camelCase (Edge's state object:
`available`, `reason`, `lamps{link,meetingMic,channel,tx}`, `room`,
`channels[16]{slot,label,listeners,ready,keyed,latched,room,reachOk,reachDark}`,
`roster[]{participantId,name,talkback,room,channels}`, `mic{device,devices,
peakDbfs,gainDb,sidetone,sidetoneDevice,sidetoneDevices,tone}`,
`stats{framesSent,channelSends,underruns,sendErrors}`, `log[]`, `private`).
`"comms":null` = the report was dropped (Leave, engine gone, core restart);
the shell then shows no key as live. Parsed by `ZoomCommsProtocol`.

**Deviations from Edge's node protocol**: arguments ride in `payload` (Pro's
request convention) instead of top-level fields; the state arrives as its own
`zoom-comms-state` fact instead of a `comms` member of a state line. Names,
fields and values are identical.

## Leases and release

A held key is **leased** (Edge contract): every press sends `leaseMs:1500`
and the shell re-sends it every 500 ms while held (`StudioViewModel.Comms.cs`
DispatcherQueueTimer → `CommsViewModel.RenewLeasesAsync`). A hung shell
therefore drops its keys in 1.5 s. Every held key goes up when: focus leaves
the page, the window deactivates (`MainWindow.OnWindowActivated`), the
operator navigates to another tab, the meeting ends, or the report is dropped.
Shell shutdown sends `comms-release-all` (`PrepareForShutdown` → `ShutdownComms`).
If the shell dies, the core's stdin closes, the core exits and the engine
process goes with it (talkback ends with the participant).

## UI rules (do not regress)

- `TalkKey` (Controls): press = down, release/capture-loss/cancel = up, with
pointer capture; **Click and Command are never used**; hover/pressed visual
states are suppressed so the colour is always the engine's truth; Space and
Enter bubble to the page.
- Never rebuild a control under the operator's finger: cells, people, device
pickers and the ops history are `ObservableCollectionSync`-synced (no
`Reset`), and only on a structure-signature change; values update in place.
Tests pin it (`CommsViewModelTests`).
- A cell is never "on air" unless the engine reports it keyed **and** someone
confirmed in the channel. Keyed with nobody = armed.
- Digits 1-9 (first nine reachable cells) and Space (ALL CALL) are
**panel-scoped** (`CommsPage` PreviewKeyDown: focus inside the page, not in a
text field, no modifier, auto-repeat ignored). Never global.
- Colours are Pro tokens only (`CommsBrushConverter`): **on air = filled
`StudioAirBrush`** (Pro's "air red - on-air, record") with
`StudioBackgroundBrush` text; armed = `StudioWarningBrush` outline and text;
ready = `StudioTextBrush`; idle = `StudioTextSecondaryBrush`; unreachable =
`StudioDimBrush`; lamps on = `StudioLiveBrush`.
- Explicit states: not in a meeting; waiting for the engine's report; not up
(verbatim reason, host/co-host hint); desk.

## Testing

- Native: `CommsRelay*`, `CommsCore*`, `CommsProtocol*`, `MicGuard*`,
`ZCommsPorted*`, `ZoomEngineRuntime.Comms*` in `corevideo-native-tests`
(the shim takes ONE `--gtest_filter` pattern).
- Shell: `CommsViewModelTests` (WinUI.Tests): the JSON each action sends,
leases, release-all, structure gates, states, colour tokens.
- End to end without Zoom: `scripts/probe-comms.mjs` drives the real core
binary over stdio against `corevideo-zoom-engine-fake` (join, wait for the
bank, key/latch/assign/tone, read the `zoom-comms-state` facts).
- Rendered page: `CoreVideoPro.WinUI.exe --verify-comms-page <report.json>`
(`CommsPageProbe`, offscreen and never activated, no core) renders the real
`CommsPage` through not-in-meeting, not-host, desk and edit+settings states,
checks key count/width and the on-air/armed tokens, and writes
`comms-*.png` beside the report; part of `scripts/test-studio-layout.ps1`.
An `x:DataType` that disagrees with the bound item type fail-fasts this page
as `0xc000027b` (seen once in development with the ops history).
- **Not proven**: real talkback against Zoom (needs a live meeting with this
account as co-host and desktop-client guests), audio at the guest, the 7.1.5
raw video/ISO path in a live take.
12 changes: 6 additions & 6 deletions docs/zoom-windows-sdk-packaging.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@ CoreVideo Pro's Windows helper must package the Zoom Meeting SDK beside the nati

Verified local SDK archive:

- Source: `C:\Users\walla\Downloads\zoom-sdk-windows-7.0.5.39292.zip`
- Version: `7.0.5.39292`
- Source: `C:\Users\walla\Downloads\zoom-sdk-windows-7.1.5.43953.zip`
- Version: `7.1.5.43953`
- Target architecture: `x64`
- Runtime source: `zoom-sdk-windows-7.0.5.39292/x64/bin/*`
- Headers: `zoom-sdk-windows-7.0.5.39292/x64/h/**/*`
- Import library: `zoom-sdk-windows-7.0.5.39292/x64/lib/sdk.lib`
- Runtime source: `zoom-sdk-windows-7.1.5.43953/x64/bin/*`
- Headers: `zoom-sdk-windows-7.1.5.43953/x64/h/**/*`
- Import library: `zoom-sdk-windows-7.1.5.43953/x64/lib/sdk.lib`
- Helper runtime target: `native-core/zoom-runtime/windows/x64`

Required files for the first real media-path spike:
Expand Down Expand Up @@ -52,7 +52,7 @@ Embedded OAuth identity (same broker as [CoreVideo plugin](https://github.com/ia
Stage the SDK beside the helper target:

```powershell
$env:ZOOM_SDK_DIR = "C:\path\to\zoom-sdk-windows-7.0.5.39292\x64"
$env:ZOOM_SDK_DIR = "C:\path\to\zoom-sdk-windows-7.1.5.43953\x64"
.\scripts\stage-zoom-sdk.ps1
```

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
using System.Text.Json;
using CoreVideoPro.MediaCore.Services;
using Xunit;

namespace CoreVideoPro.MediaCore.Tests;

/// <summary>Comms wire format between the shell and the core (docs/reference/comms-talkback.md).</summary>
public sealed class ZoomCommsProtocolTests
{
[Fact]
public void VerbsAreEdgesPlusReleaseAll()
{
Assert.Equal(
["comms-assign", "comms-gain", "comms-key", "comms-latch", "comms-mic", "comms-private", "comms-release-all",
"comms-sidetone", "comms-sidetone-out", "comms-tone"],
ZoomCommsProtocol.Verbs.Order(StringComparer.Ordinal));
}

[Fact]
public void ARefusalCarriesTheCoreMessageVerbatim()
{
using var refused = JsonDocument.Parse(
"""{"id":"7","ok":false,"error":{"code":"comms-refused","message":"Join a meeting first."}}""");
var r = ZoomCommsProtocol.ParseReply(refused);
Assert.False(r.Ok);
Assert.Equal("Join a meeting first.", r.Error);

using var accepted = JsonDocument.Parse("""{"id":"8","ok":true,"type":"comms-key"}""");
Assert.True(ZoomCommsProtocol.ParseReply(accepted).Ok);
}

[Fact]
public void AStateFactWithAnIdIsAResponseNotAFact()
{
Assert.False(ZoomCommsProtocol.TryParseStateLine("""{"id":"1","type":"zoom-comms-state","comms":null}""", out _));
Assert.False(ZoomCommsProtocol.TryParseStateLine("""{"type":"zoom-comms-state","comms":7}""", out _));
Assert.True(ZoomCommsProtocol.TryParseStateLine("""{"type":"zoom-comms-state","comms":{"available":false,"reason":"Not in a meeting."}}""", out var u));
Assert.False(u.State!.Available);
Assert.Equal("Not in a meeting.", u.State.Reason);
Assert.Empty(u.State.Channels);
Assert.Null(u.State.Mic.PeakDbfs); // nothing unmeasured is invented
Assert.False(u.State.Private);
Assert.True(ZoomCommsProtocol.TryParseStateLine("""{"type":"zoom-comms-state","comms":{"available":true,"private":true}}""", out var p));
Assert.True(p.State!.Private);
}

[Theory]
[InlineData("comms-key")]
[InlineData("comms-latch")]
[InlineData("comms-release-all")]
[InlineData("comms-private")]
public void KeyUpsUseTheReservedLane(string type) =>
Assert.True(EngineHostWire.IsUrgent(JsonSerializer.SerializeToElement(new { type })));

[Fact]
public void CommsVerbsAreMutationsNotObservations() =>
Assert.False(EngineHostWire.IsObservation(JsonSerializer.SerializeToElement(new { type = "comms-key" })));
}
44 changes: 44 additions & 0 deletions native-shell/CoreVideoPro.MediaCore/Models/ZoomCommsModels.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
namespace CoreVideoPro.MediaCore.Models;

// Comms (Zoom talkback, docs/reference/comms-talkback.md): the core's
// "zoom-comms-state" fact, i.e. the Zoom engine's last comms_state re-keyed to
// camelCase. Same shape as CoreVideo Edge's `comms` state object (EdgeComms),
// so the two apps read one contract. Every value is what the engine reported;
// nothing here is inferred by the shell.
public sealed record ZoomCommsChannel(int Slot, string? Label, int Listeners, bool Ready, bool Keyed, bool Latched,
string? Room, int ReachOk, int ReachDark);

public sealed record ZoomCommsPerson(uint ParticipantId, string Name, bool Talkback, string? Room, IReadOnlyList<int> Channels);

public sealed record ZoomCommsMic(string? Device, IReadOnlyList<string> Devices, double? PeakDbfs, int GainDb, bool Sidetone,
string? SidetoneDevice, IReadOnlyList<string> SidetoneDevices, bool Tone);

public sealed record ZoomCommsState(
bool Available,
string? Reason,
bool LampLink,
bool LampMeetingMic,
bool LampChannel,
bool LampTx,
string? Room,
IReadOnlyList<ZoomCommsChannel> Channels,
IReadOnlyList<ZoomCommsPerson> Roster,
ZoomCommsMic Mic,
long FramesSent,
long ChannelSends,
long Underruns,
long SendErrors,
IReadOnlyList<string> Log,
// Private comms applied by the engine: the meeting mic is the silent
// virtual mic (Pro's toggle, or a capture-only station, always private).
bool Private = false);

/// <summary>One "zoom-comms-state" fact. <see cref="State"/> null = the core dropped
/// the report (Leave, engine gone): the shell must stop showing any key as live.</summary>
public sealed record ZoomCommsUpdate(ZoomCommsState? State);

/// <summary>The core's answer to one comms-* request. <see cref="Error"/> is operator text.</summary>
public sealed record ZoomCommsCommandResult(bool Ok, string? Error)
{
public static readonly ZoomCommsCommandResult Success = new(true, null);
}
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,9 @@ public static async Task WriteAsync<T>(Stream stream, T message, CancellationTok
}

public static bool IsUrgentType(string? type) => type is "stop-recording-session" or "stop-encoder-session" or
"stop-program-output" or "zoom-leave" or "zoom-cancel" or "zoom-stop-capture";
"stop-program-output" or "zoom-leave" or "zoom-cancel" or "zoom-stop-capture" or
// Comms talk keys: a key-up must never queue behind ordinary work.
"comms-key" or "comms-latch" or "comms-release-all" or "comms-private";

// Never let a mixed batch smuggle ordinary work into reserved capacity.
public static bool IsUrgent(JsonElement payload)
Expand Down
Loading
Loading