Skip to content

ui-react: announce a pending approval in conversation mode and accept only a user-attributed spoken refusal #114

Closed
Closed
ui-react: announce a pending approval in conversation mode and accept only a user-attributed spoken refusal#114
Labels
area: ui-reactReact chat/files/voice components, stores, WebSocket transport.priority: p2Wanted, not scheduled. Pulled in when a milestone has room.securityAuth, isolation, egress, secrets, or anything in the threat model.type: featA new capability or a change in user-visible behaviour.
Milestone
0.44.0

Description

@schlessera
schlessera
opened on Sep 22, 2026

Context

Part of #54. docs/decisions/voice-permission.md binds the permission-specific client behaviour: voice can refuse, never grant. The approved two-engine architecture (#317/#962), the adapters (#957 / #1008, #958 / #2063, #959 / #2070), the conversation client (#960 / #2142) and this issue's permission implementation (#2145, 9423920fcd696c57156a20424789e871066c21f4, with 39 real pinned offline browser scenarios and four failing/restored permission mutations) are delivered, as are the Gemini setup corrections #2356 and #2358 (#2549, 5b476a74). #110–#113 and #316/#317 supplied the completed policy and design prerequisites. The maintainer's reserved basic fallback listening pass is now complete on both engines and both output routes, as recorded in the completion receipt below. Broader qualification remains under #961; no exact-speech or spoken-attribution capability is promoted.

What ships, and what the listening pass therefore tests:

  • No exact permission speech. GPT-Live reports exactPermissionSpeech: "unsupported" and Gemini "unproven"; the server announces only when an engine reports supported (packages/ui-server/src/ws/conversation.ts:863). The approved R1 fallback ships instead: one local earcon, the visual card and the strip line (quiet, packages/ui-react/src/voice/conversation-permissions.ts:54-68).
  • No attributable spoken refusal yet. Both adapters emit recognized input with origin: "unknown" (packages/ui-server/src/voice/gemini-conversation.ts:163-166; packages/ui-server/src/voice/gpt-live-conversation.ts:161), and the refusal guard ignores any input whose origin is not user (input, packages/ui-react/src/voice/conversation-permissions.ts:109-118). On the shipped engines, a spoken "brain, stop" therefore does not deny; refusal goes through the screen.

Binding rulings. 2026-10-05: interaction design approved (design comment §1–§8): R1 visual-only fallback without exact-copy speech, R2 the spoken ambiguity line, R3 the headphones notice with spoken refusal kept on. 2026-10-16: asynchronous tool calls genuinely continue during voice conversation, and the ordinary chat UI keeps showing progress and interactive elements throughout. 2026-10-17: protected provider access and numerical spend limits for live work are authorized under #961 (US$100 total; per-session US$5 and 10 minutes); count this pass against #961's cumulative reservation.

Design input

Consume the voice conversation proposal recorded by #316. It separates connection from microphone capture and distinguishes already-denied posture actions from genuinely pending approvals. Its interaction choices are proposals, not a speech-architecture ruling. Use its speech ownership, interruption, once-only announcement, input-routing and correlation rules for this permission-specific implementation; defining the wider conversation flow stays outside this issue.

The design requires refusal recognition during announcement playback without feeding the announcement's own words back into recognition. A scoped refusal takes priority over ordinary question/message binding and uses original recognized input before pronunciation overrides. Multiple live approvals in the selected session's current turn are all denied on an ambiguous refusal snapshot; later requests are not silently added. Keep the existing visual grant controls and request kind/rememberability behavior.

Scope

All spoken-copy requirements below are conditional on a qualified deterministic host-owned exact-copy path. Without it, the approved earcon/card/strip branch governs; a generated or retained script is not proof that the user heard it. Current remaining work is the reserved qualification of the delivered branch, not a new speech-path implementation.

  • When a tool_approval_request frame arrives during a voice session, a qualified host-owned exact-copy speech path may speak the announcement described below. Without that capability, the approved local earcon, visual card and strip fallback applies; this issue does not require unproven engine-paraphrased speech. The conditional announcement is built from the host-approved matched-pattern
    effect (currently carried as description for command approvals): what
    class of thing is about to happen, that the command is on screen, and the one
    answer voice can give. Roughly ten seconds, flat in the payload size, and it
    never reads the payload. A free-text description is not automatically a
    vetted speakable effect; use the tool-name fallback without approved effect
    provenance, and let ui-sdk: decide the live voice architecture and speech seam #317 assess the required transport.
  • A two-word refusal phrase, recognised only while a request is pending —
    the voice analogue of D36's focus-scoped a / d
    (docs/decisions/design-kit.md, "D36: single-key shortcuts are focus-scoped"). Outside that window it is ordinary
    transcript text and reaches the composer like any other words.
  • A recognised refusal sends tool_denial with a message naming the phrase,
    and is attributed to the voice channel (ui-server: an approval decision records the channel it was made on #113).
  • Ambiguity resolves as denial and the user hears that it was taken as a
    refusal: a low-confidence transcript, a partial match, overlapping speakers,
    or the phrase heard while more than one request is pending.
  • Under the enforced voice posture with no grant surface, ungrantable actions
    are already denied: announce that refusal without inventing a pending request.
    Ordinary chat with voice active can still have genuine pending approvals.
  • On no answer to a pending request, nothing further is said. When the turn
    lapses on its budget, the server drains it as a denial; say once that the
    action did not happen and must be asked for again when the user can look.
    Reconcile the on-screen card; do not claim an expired request is still waiting.
  • The approval card is unchanged and remains the only surface that can grant.

Out of scope

Scoped live listening evidence — 2026-10-27

The GPT-Live basic permission fallback listening checks passed on both headphone and speaker playback. For each route the maintainer heard exactly one short earcon, saw the genuine ordinary-turn approval card, and used screen Deny to obtain the host-confirmed result “The request was denied. No command ran.” Alert-only subchecks used a muted microphone.

Open-microphone refusal-boundary checks also passed on both routes. Speaking the refusal phrase elicited a conversational response from GPT-Live while the card remained pending; subsequent screen Deny resolved it. The headphone check confirmed microphone continuity. During the speaker reply the maintainer stayed quiet and observed no response to its own playback in that trial. This is the expected current unknown-attribution fallback, not a successful voice-channel denial or a general echo-isolation qualification.

The task backend was scripted, with no backend-model call or command execution. Experimental frontend source: 0f0fd07538c27dbca64d30d5b2a5de0467b0c90f. These results complete the scoped GPT-Live basic fallback listening portion. Duplicate/reconnect/race and broader qualification trials remain separate; Gemini's receive correction #3061 is merged and its bounded post-merge digital audio probe succeeded; the maintainer's Gemini physical listening checks are now next. Keep this issue open and do not promote provider capabilities or release qualification.

Acceptance criteria

  • Pending approval uses the approved capability branch: without a qualified deterministic exact-copy path, one local earcon plus the visual card/strip appears and no model-paraphrased permission speech is emitted. For a qualified exact-copy path, its generated announcement includes only the approved effect/tool label and no command payload. The current implementation uses the fallback; hand qualification must test what actually ships.

  • The refusal phrase resolves a pending request as tool_denial, proven
    against a fake ASR stream.

  • The same phrase spoken while no request is pending reaches the composer
    as text and resolves nothing.

  • A low-confidence or partial recognition of the phrase resolves as deny,
    not as "ask again" — asserted.

  • No spoken input, of any wording, can produce an allow. A test enumerates
    affirmative phrasings and asserts none of them resolves a request.

  • Silence leaves the request pending and emits nothing further until the
    turn’s budget expires. Expiry resolves it as denial; the spoken feedback
    and on-screen state do not claim that the expired request is still waiting.

  • Tests use a fake speech provider and a fake ASR stream; nothing needs a
    key or the network.

  • Heard by hand once by the maintainer, since a spoken interface that was
    never listened to has not been designed. Said so in the PR.

  • Each request is announced at most once across duplicate frames, interrupted playback and reconnect; resolved requests are not announced from a stale queue.

  • Refusal remains recognizable during announcement playback, and the generated announcement itself cannot trigger the detector.

  • An ambiguous refusal with multiple pending requests denies every request in the correlated current-turn snapshot, while other sessions and later arrivals remain untouched.

  • Visual-grant/refusal races and late callbacks report the host's actual result. Sending a denial alone is not treated as confirmation; disconnection shows uncertainty until reconciliation.

  • Host terminal/timeout reconciliation removes expired approval controls even when no tool result is delivered, retaining the resolved receipt and correct spoken outcome.

  • Approved design (2026-10-05, design comment §1–§8): an action already denied by the voice posture produces one outcome receipt and no card, refusal window or invitation to refuse.

  • With each §7 evidence item absent from the fake provider, its fallback ships: no exact-copy speech path means earcon plus visual card and strip line, never an engine paraphrase; missing confidence flags treat every phrase-like candidate as ambiguous; unproven echo isolation shows "Refusal by voice needs headphones with this service." on entry and keeps spoken refusal enabled.

  • An ambiguous, user-attributed refusal records the approved script "I couldn't hear that clearly, so I took it as a refusal. The action was denied." and the card shows heard unclearly · taken as refusal. Spoken delivery remains subject to the exact-copy capability/fallback rule; unknown acoustic origin cannot authorize a denial.

  • The maintainer's hand pass covers both speaker and headphone playback.

  • Existing permission cards and other chat interactions remain responsive during live audio and asynchronous host work; sending a voice refusal does not freeze the chat UI or replace host-confirmed outcomes.

Verification

Keyless proof for the implementation is delivered by #2145 (bun run test, bun run typecheck, bun run lint; nothing needs a key or the network).

Reserved maintainer listening pass, on a real device, with both configured services, on speaker and on headphones, under #961's authorization and spend accounting:

  1. Trigger a genuine pending ordinary-turn approval (a confirm-pattern command) in a voice session.
  2. Hear the delivered fallback: exactly one earcon per request, the card and strip shown, and no spoken grant prompt or model-paraphrased permission copy. Exact spoken copy is tested only on an engine that reports exactPermissionSpeech: "supported"; neither does today.
  3. Check that the chat and card controls stay usable and that asynchronous host work keeps progressing while audio plays.
  4. Say "brain, stop" while the request is pending. With origin: "unknown", confirm it does not deny, and refuse on screen instead; confirm the Actions record shows the host-confirmed outcome. Record each engine's input-attribution capability as observed. Only where an engine is qualified to report origin: "user" is a voice-channel tool_denial expected, and then the ambiguity line and heard unclearly · taken as refusal card are checked too.
  5. Note whether assistant output leaks into recognition, on speaker and on headphones.
  6. Publish only sanitized pass/fail/unknown evidence on this issue and ui: qualify both live conversation engines on real devices before activation #961: no credentials, no recordings with personal content, no deployment details.

A pass of the fallback qualifies the fallback; it does not qualify the unavailable speech or attribution capability, and the record says which one was tested.

Contract impact

The existing voice-channel tool_denial shape is the refusal transport. #317 must assess whether host acknowledgement, announcement receipts or terminal/reconnect reconciliation need additional contract changes; do not assume #113 is the only possible impact. Any resulting machine-surface change requires the integration-contract update and versioning rules.

2026-10-24: replaced the old hand-pass instruction (listen to the announcement, refuse it by voice, confirm a voice-channel denial) with the current listening protocol: test the delivered earcon/card/strip fallback, record attribution capability, and verify that unknown-origin speech cannot deny. Folded the delivered prerequisites and rulings into Context; removed the dated handoff paragraphs (preserved in a comment). Scope, safety criteria and the reserved listening requirement are unchanged.

Gemini interactive finding — 2026-10-28 UTC

After #3061's receive correction, the maintainer heard Gemini replies but ordinary follow-up speech after completed playback closed capture and required Resume. The adapter's input-finalization state was unsatisfied; #3064 owns the offline reproduction and bounded resolution. Do not attribute this to missing provider metadata alone: continued appended PCM can also clear the observed flag. This is separate from the delivered empty-envelope fix. No Gemini permission-listening completion is claimed; avoid repeated manual tests while the continuity finding is investigated.

The maintainer also reports that Gemini feels faster than GPT-Live. This is a subjective listening observation, not a measured latency comparison. The previously recorded scoped GPT-Live fallback results remain valid.

Gemini SPDIF permission-fallback listening result — 2026-10-28

The maintainer heard the alert chime and saw the approval card on SPDIF speakers with a directional microphone. Speaking the refusal phrase produced an audible conversational reply and a generated-speech entry; it did not resolve the approval. Screen Deny then produced the host-confirmed scripted result that the request was denied and no command ran. The listening report does not explicitly establish chime deduplication across retries/reconnect or general echo isolation.

Exact production source was f4fd0de35b97a2e004629fc813953a6163bba7b9. This fresh permission-only harness used a scripted ordinary-turn permission backend, no model backend call and no command execution. Its repeating waiting text and no-command receipt were explicit test output, not prescribed production copy. The live Gemini response was actual generated conversation speech. Voice opened at 20:10:45.428 UTC, recognition arrived at 20:11:06.304, and the connection settled at 20:11:24.566. The scripted status confirms one request, no pending card and denied-no-command.

Important qualification boundary: this permission-only fixture advertises contextual tasks unsupported. It deliberately exercised the retained manual-review path, which saves non-interim recognition into the composer; the maintainer observed the refusal phrase there and found it confusing. The source's automatic contextual path keeps recognition separate and returns before draft saving. Therefore this is not evidence that the newer automatic path inserts recognition into a typed draft, nor a second speech recognizer. #3050 retains the presentation finding. This pass does not qualify automatic model-driven permission admission/policy parity, unavailable exact permission speech or attributable spoken denial.

The scoped Gemini speaker fallback check passed. Gemini headphones remain pending, and broader matrix/race/echo checks remain separate. No capability was promoted. The prior ordinary-follow-up blocker is superseded for this targeted test by the merged fixes and both-provider physical concurrency receipts; it is no longer a reason to pause this listening pass.

Reserved fallback listening pass completed — 2026-10-28

The maintainer reports the fresh Gemini headphone check completed and behaved as instructed: alert/card, open-microphone refusal phrase without a voice permission decision, screen Deny and denied/no-command feedback. The server's bounded metadata confirms one Gemini opening at 20:30:41.565 UTC, ready at 20:30:42.041, recognized input at 20:30:54.984, generated output and normal final epoch teardown/settlement at 20:31:04.474. Physical audibility and the visible Deny result are maintainer observations; this scripted backend did not retain a separate durable denial receipt in the inspected message-block/activity tables, so none is claimed. No model backend or command execution occurred.

Combined scope: GPT-Live's earlier speaker/headphone fallback checks and Gemini's current SPDIF/headphone checks complete this issue's reserved basic listening pass. The shipped branch is one local earcon with the visual permission card/strip and screen authority. Neither engine gains exact permission speech or trustworthy user-origin attribution from these results. Spoken conversational acknowledgement is not a permission decision. This does not qualify general echo isolation, duplicate/reconnect/race matrix cells, automatic model-driven policy parity, or interruption recovery; those retain their own evidence requirements under #961 and the linked work.

The recognized phrase appearing in the composer was traced to this fixture's explicitly non-contextual legacy review profile and recorded under #3050. Repeating waiting text/no-command narration is scripted test output, not required product copy. Neither observation invalidates the scoped alert/card/screen-Deny listening result.

The fresh fixture has already reached its finite owner lifetime and exited. Cleanup confirms no retained owned mount, loop attachment, loop creation or backing file, no cleanup errors, and unchanged user-namespace quota. All voice/backend reservations are zero. No provider capability is promoted. The full live qualification issue #961 remains open.

Activity

  1. schlessera
    added
    type: featA new capability or a change in user-visible behaviour.
    area: ui-reactReact chat/files/voice components, stores, WebSocket transport.
    priority: p2Wanted, not scheduled. Pulled in when a milestone has room.
    blockedWaiting on another issue or an upstream dependency. Say which in the body.
    securityAuth, isolation, egress, secrets, or anything in the threat model.
    on Sep 22, 2026
  2. schlessera
    added a parent issue on Sep 22, 2026

  3. schlessera commented on Sep 25, 2026

    @schlessera
    schlessera
    on Sep 25, 2026
    OwnerAuthor

    Labelled needs: design and needs: decision 2026-09-25: this issue is blocked by #316 (waiting on a design answer) and #317 (waiting on a maintainer ruling). The labels mirror those blockers. Remove them together with blocked when both close.

  4. schlessera
    added
    needs: designBlocked on a design drop or a visual answer.
    needs: decisionBlocked on a maintainer or designer ruling, not on code.
    on Sep 25, 2026

  5. schlessera commented on Sep 28, 2026

    @schlessera
    schlessera
    on Sep 28, 2026
    OwnerAuthor

    Decision dependency review (2026-09-28): The September 25 comment confirms needs: decision was inherited from #317, not an independent choice here. The maintainer has since ruled that architecture must follow #316's live-conversation design. Remove the inherited label; #114 remains blocked by #316/#317 and needs: design. Do not reopen the already binding voice-refusal-only permission rules. It is not agent-ready.

  6. schlessera
    removed
    needs: decisionBlocked on a maintainer or designer ruling, not on code.
    on Sep 28, 2026

  7. schlessera commented on Sep 30, 2026

    @schlessera
    schlessera
    on Sep 30, 2026
    OwnerAuthor

    Needs design — prompt refreshed 2026-09-30 after #657. Permission policy is already decided. #316 has recorded the conversation proposal in docs/plans/voice-conversation.md; #317 has selected both live engines and the common host-owned contract, while delivery remains a dependency. This issue still needs the reviewed permission interaction realized against that ruling. The prompt covers how that interaction reaches the existing approval surface without reopening spoken grants or assuming a separate TTS provider.

    Paste to Claude:

    Design the spoken approval-announcement and scoped-refusal interaction for brain-kit issue #114, downstream of voice design #316 and architecture #317 under epic #54.

Read first:
- #114's current scope/acceptance criteria, parent #54, the proposal in docs/plans/voice-conversation.md (completed design spike #316, merged #657), and #317's current architecture ruling. The maintainer selected both Gemini Live and OpenAI Realtime behind the common host-owned adapter contract in #317. Its complete specification is [recorded on #317](https://github.com/schlessera/brain-kit/issues/317#issuecomment-5974104901); the decision is preserved in unmerged draft #962, so it is an input rather than a current-main interface. #957 owns contracts, #958/#959 the two adapters, #960 the conversation controls and #961 the hearing/proof/adoption gate. #317 remains an implementation dependency while that work is undelivered. Proposal timings and controls remain proposed choices.
- docs/decisions/voice-permission.md: “What the user hears, and how a decision is expressed”, “When the announcement actually fires”, “How a spoken decision is audited” and “What this constrains in #54”. Read the corrections on silence/timeout: the server denies and drains expired approvals.
- Denial wire (`export interface ClientToolDenial`, `packages/ui-sdk/src/protocol.ts:307-315`), request wire (`export interface ServerToolApprovalRequest`, `packages/ui-sdk/src/protocol.ts:582-607`) and enforced posture (`export interface StartTurnRequest`, `packages/ui-sdk/src/server/backend.ts:298-372`). Inspect current permission/approval handlers and Actions records, plus the phase-1 useDictation driver.
- docs/decisions/design-kit.md D36 (scoped shortcuts), D37 (approval focus/navigation) and D38 ruling 1 (ordinary ask_user answers are not permissions); docs/integration-contract.md’s current ask_user_list and composer-binding rules (#594). List judgments are withheld from noGrantSurface turns and a message while a list is pending does not answer the list.

Settled: no spoken grant or always-allow. The visual approval card stays the only grant surface. A two-word “brain, stop” is a denial only while an actual request is pending; otherwise it is ordinary transcript text. Ambiguous recognition denies and tells the user it was taken as a refusal.

Design:
1. Separate already-denied calls under the no-grant voice posture from pending requests in ordinary chat with voice active. The first needs honest denial feedback, not an invitation to refuse something already resolved.
2. For a pending request, specify a single roughly ten-second payload-free announcement using the host-approved matched-pattern effect phrase when its provenance is established, and the tool-name fallback otherwise. An arbitrary description field is not approved spoken copy. It names the class of action, says the command is on screen and offers refusal. Never read arbitrary command/input payloads or assume the default phrases cover every request kind.
3. Apply #316's interruption rules to user speech, model speech and announcements. Describe how scoped refusal stays available during playback without recognizing the announcement's own audio as user input. Route original recognized input before pronunciation overrides; refusal takes priority over ordinary question/message binding. Do not select providers or invent a new interface.
4. Single/multiple pending requests, partial/low-confidence/overlapping recognition, denial acknowledgement, a screen decision racing the voice phrase, cancellation and connection failure. Consume the proposal's deterministic request/turn correlation: ambiguous refusal denies the selected session's current-turn pending snapshot, preserving other sessions and later arrivals. Use host confirmation for spoken outcome; sending denial alone is not confirmation. No ambiguity path can grant or ask an affirmative follow-up.
5. Silence and timeout: no reminder loop; on expiry report that the action did not happen and must be requested again, and reconcile any stale on-screen card. Do not point users to an expired request “still waiting”.
6. Transcript and Actions evidence: what was announced, what phrase/refusal resolved which request, voice-channel attribution, and the state visible to a person who switches from listening to looking. No separate replacement approval card.

Hold: #316 owns conversation endpointing/barge-in and ask_user; #317 owns architecture. Preserve keyboard/touch approval behavior, existing tokens, dark/paper themes, focus, effective targets and 320px legibility. Mic/provider disclosures remain honest. Implementation stays keyless in tests with scripted speech/events and a real-device listening pass.

Deliver: sequence/state diagrams, exact announcement/fallback/refusal/timeout copy, ASCII mocks at 320px and desktop with the existing card, and acceptance criteria for pending/non-pending phrases, ambiguity, silence, races and expiry. Realize the selected #317 contract and state any remaining transport evidence needed for recognition uncertainty, echo-safe playback, acknowledgement, effect provenance and terminal/card reconciliation. Preserve host ownership of permissions, serialized brain work and transcript/context authority; live providers never gain those powers. Do not assume the existing denial wire supplies every requirement. A minimal announcement layer is acceptable if it preserves every binding rule.

    Source pointers verified against remote main e0875269b21b707482545be9e94901acb897893b on 2026-09-30; search the named anchors if lines move.

    2026-10-01: Refreshed denial, approval-request and StartTurnRequest citations in the existing prompt. The phase-1 driver citation still matches. Preserve #317 as the unresolved architecture blocker and the existing design and permission constraints. Changed pointers checked at main 35fb78d40542426b9bac4a1c3a2a32cf43803837; other dated evidence retains its original provenance.

    2026-10-01: Reverified the denial/approval frame and StartTurnRequest anchors after the form-tool merge at b333db6fb6fba30e7a2a92186822e78b782c86d4. The #317 architecture dependency and needs: design remain.

    2026-10-01 triage at main 619ee2b2d48409fbd9d6a74ccc86bd73507362ac: Refreshed StartTurnRequest after the autonomous-turn additions. Denial/request frame anchors still match; #317 and the interaction design remain unresolved, with voice refusal and timeout/drain semantics unchanged.

    2026-10-01: Refreshed the denial and approval-request frame anchors in the existing prompt. #317 and the architecture-dependent design remain unresolved; permission and timeout rules are unchanged. Source inspected at main 3f0870d5338c9139f9bb53050847c23974662a00.

  8. schlessera commented on Sep 30, 2026

    @schlessera
    schlessera
    on Sep 30, 2026
    OwnerAuthor

    #316's design proposal is recorded and merged in #657. This body now links it and incorporates its permission-specific interruption, original-input refusal routing, once-only announcement, current-turn multi-request snapshot, visual-grant race, host-confirmed feedback and stale-card/terminal reconciliation requirements. The effect phrase must have host-approved provenance; a free-text description alone is not sufficient.

    Removed the completed #316 dependency. #317 still blocks implementation and must assess the speech architecture plus any additional contract needs. blocked and needs: design remain for the architecture-dependent implementation design; this handoff does not select a provider/seam or mark implementation agent-ready. The existing keyless runtime and maintainer real-device listening evidence remain required for this implementation issue.

  9. schlessera commented on Oct 1, 2026

    @schlessera
    schlessera
    on Oct 1, 2026
    OwnerAuthor

    2026-10-01 triage — Refreshed denial, approval-request and StartTurnRequest citations in the existing prompt. The phase-1 driver citation still matches. Preserve #317 as the unresolved architecture blocker and the existing design and permission constraints. Source inspected at main 35fb78d40542426b9bac4a1c3a2a32cf43803837.

  10. schlessera commented on Oct 1, 2026

    @schlessera
    schlessera
    on Oct 1, 2026
    OwnerAuthor

    2026-10-01 triage: repaired the existing design prompt's moved denial/approval frame and StartTurnRequest anchors at b333db6fb6fba30e7a2a92186822e78b782c86d4. The recorded conversation proposal remains an input; #317 and the architecture-dependent design are still required. No permission, architecture or readiness ruling is changed.

  11. 16 remaining items

  12. schlessera commented on Oct 16, 2026

    @schlessera
    schlessera
    on Oct 16, 2026
    OwnerAuthor

    Current draft PR #2145 head fc033a77e28487d92a41b9274266219178fdc49b targets UI parent 2ded64053f2ba6d952567d5defde82cf5b9886ff in native stack #2135. It contains the permission implementation and an additional recovery correction. The rebase preserves main-model/title-usage accounting alongside the scoped voice-posture outcome handler. All moved source citations are anchored; milestone remains 0.44.0.

    The earlier confirmation 38906390560 exposed a real regression: applying a voice receipt changed transcript tool status before ordinary result/recovery routing could recognize the pending restored card. This hid terminal closure and cross-page answer evidence. Receipt phases now lock decisions through awaitsDecision, while ordinary host results retain ownership of transcript status. The existing restored-store tests fail on the intended ended assertions before this change and pass after it. The visual-grant race still verifies the confirmed grant receipt, visible wording and disappearance of approval controls.

    Current candidate evidence:

    • Focused host/SDK/restoration tests: 106 passed / 326 assertions, including the posture outcome and restored-card lifecycle.
    • Pinned offline Chromium conversation scenarios: all 39 pass on the rebased source with rebuilt SDK/client browser inputs; this includes continuing chat progress/cards, permission races, scope/epoch/capture guards and phone/desktop layout checks.
    • The original real-browser approval recovery test now passes both scenarios / 16 assertions: reload recovery and a second page answering while the first becomes read-only with the correct receipt. Its setup successfully completed the full current package build; the test took 136.58 seconds including setup.
    • Strict types and lint pass. An earlier simultaneous lint walk raced a temporary host-test fixture's deletion; the subsequent isolated scan is clean.
    • check:pr --base feat/960-voice-ui-42413928 completed all initial gates and 699 focused tests / 4,420 assertions. Its separate import-graph feedback was deliberately interrupted after those passes while waiting for workspace access; the entire preflight is not claimed green. The earlier browser setup timeout and stale build import failure remain setup failures, not behavioral mutation receipts; rebuilt inputs and the direct native test provide the passing evidence above.

    The fixes and tests are pushed. GPT's next dependency-order confirmation 38933150766 is still waiting for a hosted runner; upper native aggregate support remains #2141. Keep this PR draft until lower layers land and its resulting current candidate can receive complete hosted confirmation. The earlier failed hosted run is not a passing receipt, and the local checks do not replace its selected categories.

    This remains Refs #114. The maintainer-reserved listening check remains open under #961, alongside provider access and an explicit numerical spend limit before any paid measurement. No live provider or device qualification was performed.

  13. schlessera commented on Oct 16, 2026

    @schlessera
    schlessera
    on Oct 16, 2026
    OwnerAuthor

    2026-10-16 dependency reconciliation: removed the canonical #958 blocker after verifying its completed adapter output and merged #2063 (cd5e8bb04b0d0e24f1a6c4d4f73e17f1b3fe4908) on inspected main. The #958 closure records source-bound selected confirmation separately from queued actual-main proof. Remaining #959/#960 and permission/qualification obligations stay open; retain blocked and needs: human. Adapter delivery provides no live samples, numerical spend authorization or reserved listening completion.

  14. schlessera
    removed
    blockedWaiting on another issue or an upstream dependency. Say which in the body.
    on Oct 17, 2026

  15. schlessera commented on Oct 17, 2026

    @schlessera
    schlessera
    on Oct 17, 2026
    OwnerAuthor

    Merged #2145 as 9423920fcd696c57156a20424789e871066c21f4. Verified on remote main: parent 85873db77bf09143c51818255d2d4733a8eb1f67, tree 56ba27792b64c808b81df0412b4b49344bb7a96a, exactly equal to the fully passing tested tree. PR and advanced issue #114 milestones read back as 0.44.0; #114 remains open under Refs for the reserved listening requirement. All four voice implementation layers are now delivered. The post-merge automatic run is separate from the successful pre-merge confirmation; no later run result is inferred.

  16. schlessera
    added
    blockedWaiting on another issue or an upstream dependency. Say which in the body.
    on Oct 21, 2026
  17. schlessera
    removed this from the 0.44.0 milestone on Oct 22, 2026

  18. schlessera commented on Oct 22, 2026

    @schlessera
    schlessera
    on Oct 22, 2026
    OwnerAuthor

    Maintainer ruling 2026-10-22: moved out of 0.44.0, so this is now unscheduled. The code was delivered by #2145 (9423920). What remains is the maintainer's speaker and headphone listening pass with both services, under the spend authorization recorded in #961. The Gemini half of that pass waits on #2358, which is now in 0.44.0. This issue stays open with needs: human and Blocked by #2358. Schedule it again when #2358 has landed and the listening pass can be booked.

  19. schlessera
    removed
    blockedWaiting on another issue or an upstream dependency. Say which in the body.
    on Oct 22, 2026

  20. schlessera commented on Oct 22, 2026

    @schlessera
    schlessera
    on Oct 22, 2026
    OwnerAuthor

    2026-10-22 triage at main aa8c8966b59f031f054f4876dbca84048b40ae4f: #2358 is completed through merged PR #2549, squash 5b476a747769e6c8703252a1fb17a6d2ac39fb13, verified in the inspected main ancestry. The actual Gemini message boundary now bounds and synchronously decodes Uint8Array/ArrayBuffer JSON. This retires the binary-frame code-delivery dependency only; it supplies no live-provider handshake, audio, device, acoustic-attribution or listening result. Existing numerical authorization/reservation accounting and the reserved speaker/headphone listening requirements remain unchanged. No live call or capability activation was performed.

  21. schlessera commented on Oct 22, 2026

    @schlessera
    schlessera
    on Oct 22, 2026
    OwnerAuthor

    2026-10-22 acceptance reconciliation at a310317: aligned the opening scope and announcement/ambiguity criteria with the already approved R1 earcon/card/strip fallback and the delivered original-user attribution guard. The issue must not demand unqualified model-paraphrased speech or treat unknown acoustic origin as a refusal. #2145 is merged and ancestral; this changes no runtime or approved policy. The unscheduled needs: human handoff remains the reserved speaker/headphone listening pass with both services under #961’s existing authorization/accounting.

  22. schlessera commented on Oct 24, 2026

    @schlessera
    schlessera
    on Oct 24, 2026
    OwnerAuthor

    2026-10-24: the body was consolidated and its Verification rewritten to the current listening protocol. The text removed from the body is preserved here verbatim:

    Removed body text

    Delivered prerequisite: #2358 / PR #2549 (5b476a747769e6c8703252a1fb17a6d2ac39fb13).

    Delivered prerequisite: #957 (PR #1008, 533c0a7c).

    Context

    docs/decisions/voice-permission.md binds the permission-specific client behavior: voice can refuse, never grant. The approved two-engine architecture, adapters and conversation controls are delivered, and #2145 delivered this permission implementation as 9423920fcd696c57156a20424789e871066c21f4. The issue remains open for the explicitly reserved maintainer listening pass, rather than further implementation or a new interaction/architecture ruling.

    Current source at inspected main b7f6360a86be68f7da8cefa93c316ecc0bf9bf42 is createConversationPermissions (packages/ui-react/src/voice/conversation-permissions.ts:20-22): the exact-copy speech path is absent, so the approved local earcon/card/strip fallback is used (quiet, same file:54-68). The original-input refusal guard (input, same file:109-120) requires user attribution; unknown acoustic origin cannot authorize a denial. The earlier tap-to-dictate description and outstanding adapter/design dependencies below are historical inputs, not current implementation gaps.

    #110–#113 and #316/#317 supplied completed policy/design prerequisites. The scope and criteria below preserve what was approved and delivered; they do not claim live-provider/device qualification. The current remaining handoff and prior proof identities are recorded below.

    Verification

    bun run test
bun run typecheck
bun run lint

    Plus one hand pass on a real device: trigger a confirm-pattern command in a
    voice session, listen to the announcement, refuse it, and confirm the Actions
    record shows a voice-channel denial.

    Body updated 2026-09-25: the blocker was epic #54, which cannot close while this sub-issue is open, so blocked could never clear. It now names the two issues filed from #54's open questions, #316 (conversation model) and #317 (output seam). D36 citation moved to design-kit.md:1799.

    Updated 2026-09-28: removed the inherited needs: decision label. The maintainer deferred speech architecture to #316 in the ruling on #317; this issue has no separate architecture choice. Existing voice-permission rules remain binding. blocked and needs: design remain; restore a decision label only if the resulting design exposes a distinct unresolved choice here.

    Updated 2026-09-30: added the self-contained design prompt; aligned architecture sequencing, already-denied versus pending cases, and timeout wording with the current voice-permission record. #316/#317 remain blockers.

    Updated 2026-09-30 after #316: linked the recorded conversation proposal, incorporated its permission-specific input routing, playback, identity/race and stale-card requirements, and removed the completed design-production blocker. #317 remains the blocker. needs: design remains for the architecture-dependent implementation design; this update does not approve an architecture or mark the implementation agent-ready.

    Current architecture input — 2026-10-04

    #317 selected Gemini Live and OpenAI GPT-Live behind one host-owned adapter contract; its complete specification and merged #962 supply the durable design inputs on main. They describe a future interface; implementation remains in #957–#959. #957–#961 divide contract, adapters, conversation controls and proof/adoption work. The existing prompt is refreshed to consume this selection. Keep the actual contract/adapter dependencies #957–#959 for required runtime delivery and needs: design for the reviewed permission interaction. Refusal-only policy, effect provenance, host confirmation, echo-safe recognition, expiry and real-device listening requirements remain unchanged. No architecture choice is reopened and no speech capability is inferred from the current dictation driver.

    2026-10-04 prerequisite delivery: #317/#962 is merged and verified on remote main. Its architecture record is complete; runtime contract/adapters #957–#959 and this issue’s permission-specific interaction design/listening proof remain required. No dictation capability is promoted to conversation support.

    2026-10-05 maintainer ruling: the interaction design is approved (R1 visual-only fallback without exact-copy speech, R2 spoken ambiguity line, R3 headphones notice with refusal kept on). Acceptance criteria extended from its §8. needs: decision removed; still blocked by #957, #958, #959.

    2026-10-06 triage at main 543231e1: the design and residual choices are approved (2026-10-05 decision comment), so the 2026-09-28 note about needs: design above is historical. #110, #111, #112, #113, #316 and #317 are all closed as completed; the use-dictation.ts:26 citation still matches. Added agent-ready alongside blocked under the maintainer's 2026-10-06 ruling that a fully specified, dependency-blocked issue carries both; it supersedes the 2026-10-05 comment's "not agent-ready until" line. #957, #958 and #959 are still open, so blocked keeps this out of selection. The maintainer's real-device hand pass is an end-stage check and does not block coding.

    2026-10-06 triage at main 825cd044: #957 merged as PR #1008 (533c0a7c), so it is retired from the active blocker lines. #958 and #959 are still open, so blocked and agent-ready both stay.

    2026-10-16 maintainer clarification: asynchronous tool calls must genuinely continue during voice conversation, and the ordinary chat UI must keep showing progress and interactive elements throughout. Added the corresponding runtime acceptance criterion without changing selected engines, approved interaction design or permission authority.

    Implementation candidate and reserved verification — 2026-10-16

    PR #2145 implements the approved permission slice above client PR #2142 and adapter PRs #2063/#2070. It uses Refs #114: the issue stays open for the reserved listening evidence. Independent proof passed: 39 real pinned offline browser scenarios, four failing/restored permission mutations, 87 host/SDK tests, full build, strict types and 638-test parent-local preflight. This includes continued async work/chat interactions, attributed original input, once-only native earcons, snapshot isolation, host confirmation, visual-grant races, expiry, reconnect receipts and 320px layout.

    The selected providers have no proven deterministic exact permission-speech path, so the approved earcon/card/strip fallback ships. Both adapters still report unknown acoustic origin; those fragments cannot authorize a refusal. No model-paraphrased permission copy, spoken grant, real echo isolation or device qualification is claimed. Native upper-layer CI identity is tracked in #2141; no gate is bypassed.

    Required maintainer step: the implementation is integrated and #961’s access/spend scope is already authorized within the recorded caps. Hand-hear the permission flow on speaker and headphones with both configured services. Trigger a genuine pending ordinary-turn approval, listen to the earcon/visual fallback, verify that chat/card controls remain usable, and inspect host-confirmed outcomes. Record whether original-user attribution permits scoped refusal and whether assistant output leaks into recognition. Unknown attribution remains closed even if the phrase sounds clear. Publish only sanitized pass/fail/unknown evidence under #114/#961, without credentials, recordings containing personal content or deployment details. Broader corpus/race measurements belong to #961 and are automated; this hand-hearing step remains explicitly reserved.

    The implementation is reviewable; this handoff authorizes no paid/live calls. Removing agent-ready and adding needs: human preserves the open acceptance/verification boundary.

    2026-10-16 triage at main 4ba792473152c89eb74c286fc914e54ca1dae8fc: removed the fulfilled #958 Gemini adapter blocker after reading its closure and merged #2063 / cd5e8bb04b0d0e24f1a6c4d4f73e17f1b3fe4908. #959/#960 and the permission candidate remain unfinished; retain blocked and needs: human for the reserved qualification/listening handoff. Keyless adapter proof is not live provider/device qualification.

    Maintainer ruling — 2026-10-17: The known ActionCard intermediate-opacity and final-disappearance timing findings in #1946 remain under evidence collection and are non-blocking for other issues. Matching failures must retain their actual failed identities and explicit attribution; this is accepted animation risk, not a repair or a green CI claim. Other proof and merge requirements remain applicable.

    The retained ActionCard 713.9 ms final-disappearance failure does not block this voice-permission work. Adapter/client integration, current candidate qualification, native CI identity support and the explicitly reserved real-device/listening requirements retain their independent scope. No live provider qualification or completion is inferred.

    2026-10-17 prerequisite delivery: #959 merged as #2070 (18a65206d4ed5f6fc4790728293ac2ce72858026) and #960 merged as #2142 (85873db77bf09143c51818255d2d4733a8eb1f67). Their active blocker lines are retired. Permission implementation #2145 is now ready on the merged UI base; selected hosted proof is running. Keep this issue open with needs: human for the explicitly reserved listening/qualification evidence; no live provider or device result is inferred.

    2026-10-17 implementation delivered: permission PR #2145 merged as 9423920fcd696c57156a20424789e871066c21f4, on UI squash 85873db77bf09143c51818255d2d4733a8eb1f67. Its complete selected confirmation run 39038370902 passed all 14 product jobs and the aggregate, including 14,478 unit tests and all 39 real pinned conversation/permission browser cases. The actual squash tree 56ba27792b64c808b81df0412b4b49344bb7a96a exactly matches the tested tree and is verified on remote main. Both adapter PRs #2063/#2070 and client PR #2142 are also merged; there are no remaining code-delivery dependencies in this stack.

    Keep this issue OPEN with needs: human. The maintainer must hand-hear the announcement/refusal flow on speaker and headphones with both selected services, under separately authorized access and numerical spend limits coordinated in #961. Use the approved earcon/card/strip fallback, verify continued chat interaction and actual host outcomes, and record sanitized pass/fail/unknown evidence for attribution and output leakage. Unknown acoustic origin still cannot authorize a denial. The merge does not satisfy this reserved listening criterion.

    2026-10-17 qualification prerequisite evidence: the maintainer authorized protected provider access and the numerical qualification limits recorded in #961 (US$100 total; US$40 Gemini voice, US$40 OpenAI voice, US$20 backend; US$5 and 10 minutes maximum per session). Authorization no longer blocks this listening step. Unmodified current-main live probes reached GPT-Live ready once, but Gemini setup failed twice before audio because the function parameter schema is rejected; #2356 tracks that correction. No audio, backend work or permission listening pass was performed. No headphone route is currently available. Keep this issue open with needs: human; no safe attribution, exact speech or real-device result is inferred. The reserved maintainer listening pass and #961 automated qualification remain outstanding.

    2026-10-21 triage: reconciled the opening context and listening handoff with the delivered permission implementation, current earcon/original-input guards and the existing October 17 spending authorization. #2356 is delivered; #2358 remains an open Gemini binary-frame qualification obstacle under #961 and blocks completion of the required both-service listening pass; independent speaker/headphone preparation and the other service’s authorized checks can proceed. Added the canonical dependency and blocked label. Keep needs: human for the reserved maintainer speaker/headphone listening evidence; no paid call, acoustic attribution proof or listening pass was performed.

    2026-10-22 triage at main aa8c8966b59f031f054f4876dbca84048b40ae4f: #2358 is completed through merged PR #2549, squash 5b476a747769e6c8703252a1fb17a6d2ac39fb13, verified in the inspected main ancestry. The actual Gemini message boundary now bounds and synchronously decodes Uint8Array/ArrayBuffer JSON. This retires the binary-frame code-delivery dependency only; it supplies no live-provider handshake, audio, device, acoustic-attribution or listening result. Existing numerical authorization/reservation accounting and the reserved speaker/headphone listening requirements remain unchanged. No live call or capability activation was performed.

    2026-10-22 acceptance reconciliation at a310317: aligned the opening scope and announcement/ambiguity criteria with the already approved R1 earcon/card/strip fallback and the delivered original-user attribution guard. The issue must not demand unqualified model-paraphrased speech or treat unknown acoustic origin as a refusal. #2145 is merged and ancestral; this changes no runtime or approved policy. The unscheduled needs: human handoff remains the reserved speaker/headphone listening pass with both services under #961’s existing authorization/accounting.

  23. schlessera
    changed the title [-]ui-react: speak the approval announcement and accept a scoped spoken refusal[/-] [+]ui-react: announce a pending approval in conversation mode and accept only a user-attributed spoken refusal[/+] on Oct 24, 2026
  24. schlessera
    added this to the 0.44.0 milestone on Oct 28, 2026
  25. schlessera
    closed this as completedon Oct 28, 2026
  26. schlessera
    removed
    needs: humanWaiting on an action only a human can perform. State the required action in the issue.
    on Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: ui-reactReact chat/files/voice components, stores, WebSocket transport.priority: p2Wanted, not scheduled. Pulled in when a milestone has room.securityAuth, isolation, egress, secrets, or anything in the threat model.type: featA new capability or a change in user-visible behaviour.

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions