Skip to content

ui-server: Gemini Live rejects binary JSON frames from the live endpoint #2358

Closed

Description

@schlessera

Out of scope

  • Changing provider/model selection, weakening frame/queue limits, trusting vendor tool arguments, changing permission authority, adding paid CI, or claiming audio/device qualification.

Context

The separately authorized #961 prerequisite recheck of #2356 candidate ebb4a12def46412216a6d4acbd7cb4db89dd3bea observed a real Gemini Developer API v1beta binary JSON setupComplete frame for gemini-3.8-live on 2026-10-17. Bun 1.4.3 delivered MessageEvent.data as a Uint8Array. A passive diagnostic decoded only the top-level JSON keys; no credential, raw payload, session identifier, audio or tool execution was recorded.

The adapter message handler (socket.addEventListener("message"), packages/ui-server/src/voice/gemini-conversation.ts:132-137 on that candidate) accepts only strings and fails with Gemini Live returned an invalid or uncorrelatable frame. The provider had emitted setupComplete; this is a separate failure from #2356's invalid function parameter schema. The existing loopback fixture sends JSON as text and does not cover the actual binary transport.

Part of #54. Related qualification: #961.

Scope

  • Normalize Gemini text/binary JSON at the adapter's existing message boundary using the installed Bun WebSocket behavior.
  • Enforce the existing frame bound before decoding/parsing, preserve arrival order and reject malformed/unsupported data visibly without leaking provider diagnostics.
  • Exercise real loopback binary setup and representative event frames alongside text compatibility.

Acceptance criteria

  • A real Bun loopback peer sending UTF-8 JSON setupComplete as binary completes the actual adapter handshake; show the regression failing on the current string-only guard before the fix.
  • Representative binary input/audio/interruption/tool frames retain their existing normalized semantics and host-owned authority. Text frames still work.
  • Oversized binary frames, malformed JSON and unsupported payload types fail closed under the existing bounds without leaking credentials; no guard is removed to accept binary data.
  • Decode/normalization cannot reorder concurrent messages or admit callbacks after abort/close/replacement. Test asynchronous normalization if that implementation requires it.
  • Existing Gemini lifecycle, queue, correlation and host tests pass; record the corrected candidate for a separately authorized ui: qualify both live conversation engines on real devices before activation #961 live recheck.

Verification

Keyless real Bun WebSocket tests through bun run test, strict types/lint, failing-first/restored evidence and bun run check:pr --base origin/main before ready. CI remains offline. Live connectivity and device/audio qualification are separate #961 evidence, not inferred from loopback tests.

Contract impact

Expected internal transport normalization only; no shared machine schema or capability promotion. Preserve existing byte bounds and authority semantics; reassess any exported or shared-wire change.

2026-10-22: removed the stale Blocked by #2356 declaration (#2356 is closed as completed). No open blocker remains.

Activity

  1. added
    type: fixSomething is broken and must behave the way it is documented.
    area: ui-serverHono app factory: turn coordinator, auth, session catalog, routes, cron.
    priority: p2Wanted, not scheduled. Pulled in when a milestone has room.
    agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.
    blockedWaiting on another issue or an upstream dependency. Say which in the body.
    on Oct 17, 2026
  2. schlessera commented on Oct 20, 2026

    @schlessera
    OwnerAuthor

    Unblocked: #2356 is closed, so scripts/sync-project.ts is removing the blocked label.

  3. removed
    blockedWaiting on another issue or an upstream dependency. Say which in the body.
    on Oct 20, 2026
  4. added this to the 0.44.0 milestone on Oct 22, 2026
  5. schlessera commented on Oct 22, 2026

    @schlessera
    OwnerAuthor

    Maintainer ruling 2026-10-22: pulled into 0.44.0. It is agent-ready, keyless and small. The string-only message guard is still on main in packages/ui-server/src/voice/gemini-conversation.ts, and this fix unblocks the Gemini half of #114's listening pass.

    Tracker: removed the stale Blocked by #2356 line, since #2356 is closed as completed; no open blocker remains. Blocker priorities check out. This issue blocks #114 and #961, both priority: p2, and is itself priority: p2, so no change is needed.

  6. self-assigned this
    on Oct 22, 2026
  7. schlessera commented on Oct 22, 2026

    @schlessera
    OwnerAuthor

    Claiming #2358 for offline implementation. Claim token: issue-2358-binary-044-20261022; branch: fix/2358-gemini-binary-frames-044. Scope: bounded synchronous binary JSON normalization, real Bun loopback regression and failure/lifecycle coverage. Handoff stops at a draft PR plus local preflight and draft gates; the coordinator owns confirmation and merge.

  8. schlessera commented on Oct 22, 2026

    @schlessera
    OwnerAuthor

    Resuming the preserved implementation for #2358 after the earlier worker ended. Claim token: issue-2358-resume-044-20261022; branch: fix/2358-gemini-binary-frames-044. I will requalify the real loopback regression and mutation evidence, then stop at a draft PR with local preflight; confirmation and merge remain with the coordinator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.area: ui-serverHono app factory: turn coordinator, auth, session catalog, routes, cron.priority: p2Wanted, not scheduled. Pulled in when a milestone has room.type: fixSomething is broken and must behave the way it is documented.

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions