Conversation
7 of 11 tasks
usnavy13
marked this pull request as ready for review
September 20, 2026 23:26
usnavy13
force-pushed
the
feat/native-media-port
branch
from
September 24, 2026 14:23
b8071e8 to
a09e642
Compare
Validate native output and replay boundaries, preserve model usage through failures and subagent execution, integrate tenant tracing, and ship the shared CJS/ESM contract fixture.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
…y conventions Remove config/native-media-contract.test.mjs with its npm script and CI step; the consuming application keeps its own contract against the installed package. Drop the unused root initializeLangfuseTracing export, restructure ADR 0010 and 0011 to the existing ADR layout, remove host-specific wording from the native media docs, and list the doc under README Documentation. Co-Authored-By: Claude Fable 5.1 <[email protected]>
usnavy13
force-pushed
the
feat/native-media-port
branch
from
October 3, 2026 03:25
a09e642 to
c39c66a
Compare
usnavy13
added a commit
to usnavy13/LibreChat
that referenced
this pull request
Oct 4, 2026
Moves the native media bridge to @librechat/agents 4.0.1, the release dev targets: both consumers pin it exactly and patches/@librechat+agents+4.0.1.patch is regenerated from LibreChat-AI/agents#553 rebased onto 4.0.1, replacing the 3.9.0 patch. Conflict resolutions carry Studio onto dev's reshaped code: - "Create media" moves from the removed AttachFileMenu into the Attach and tools palette (useAttachItems), with the same chat/create gating. - Native media stream handling follows dev's move into hooks/SSE/steps. - Media file claims and native signatures wrap dev's single writeMessage path. - Cookie authentication for share, image and media files stays one module (images/cookies.ts) and gains dev's two-factor enrollment and token retirement checks; dev's parallel auth/share.ts is dropped. - Ban checks use dev's getBanIp without mutating req.ip. Studio components now satisfy dev's design lint: spacing and color move off Collapsible primitives onto plain wrappers, section actions use the section-action Button variant, image frames size through a --media-ratio custom property, and two shared variants cover needs other screens share (Input leading-icon for search fields, Button composer for composer-row controls; pressed header-action buttons fill through aria-pressed). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NativeMediaPort, an injected host boundary thatCustomChatGoogleGenerativeAIcalls to authorize an invocation, persist each text or image part before it is emitted, record completion or failure, and restore signed parts for continuationimage_filecontent alongside text through_generate,_streamResponseChunksandstreamEvents, and keep that order through graph dispatch and content aggregationresponseModalitiesandthoughtSignatureon text and inline-data parts, including empty signed text, so image continuations replay exactlyUsageBearingError/NativeMediaErrorso provider consumption survives a failed persistence, a blocked response or a cancellation, and record that usage on the Langfuse generation spantraceModelInvocation(params, work, project?)so a host can trace model calls made outside a graph run with the SDK owning handler creation, attribute scoping and disposalAbortSignaltogenerateContentStreamand observe its aggregate-response promise so cancellation closes the provider connection without a later unhandled rejectionsystemInstructionper request instead of mutating the shared clientmodelRunIdto subagent usage events so native failure usage can be matched to its callSupersedes #41. Related: LibreChat-AI/LibreChat#6065 (native inline image output). Proposal and host integration: LibreChat-AI/LibreChat#16140.
Why
Gemini image models return
inlineDataparts in an ordinary completion, no tool call involved. A host cannot let raw image bytes enter graph state, in-run replay, Langfuse, SSE or subagent results, and it cannot emit a file reference before the file exists. The adapter therefore hands each part to the host and waits for a durable reference before yielding, which gives bounded backpressure without a queue of unpersisted bytes. Authorization, storage, retention, accounting and recovery stay on the host side of the port; the adapter owns provider parsing and ordered emission.Hosts also run model calls outside a
Run(titles, bounded provider transports). Exporting the raw Langfuse handler factory and disposal helpers would make every host responsible for the SDK's internal lifecycle, so the package exposes one owned lifecycle instead and keeps those internals private. Seedocs/adr/0010-persist-native-media-before-model-output.mdanddocs/adr/0011-trace-external-model-invocations.md.Design
startresponseModalities. Rejection prevents the provider request.partthoughtSignature; return visible text or animage_filereference only after persistence succeeded.completefailaborted,providerorstoragereason, failure-only usage and an optional provider outcome.restore/restoreBatchIMAGEadmission enables rejection of empty, blocked or invalid image responses; a port attached to a text model does not change ordinary response handling or add a modality selection.native_media.continuationRefin history is rejected rather than silently dropped.isStructuredGoogleContentPartreplaces the server-side-tool predicate inGraph.tsandstream.tssoimage_fileandnative_mediaparts follow the same ordered-dispatch path Google tool payloads already use; the aggregator copiesimage_fileparts andnative_mediamarkers through.traceModelInvocationinitializes the resolved destination, builds a scoped handler, gives LangChain callers callbacks and external clients a result projection, records a generic failure and rethrows the original error, and disposes the handler infinally. Initialization, export and projection failures never repeat inference or replace a result.NativeMediaPort,NativeMediaPart,NativeMediaContent,NativeMediaReference,NativeMediaRestoreInput,NativeMediaProviderOutcome,UsageBearingError,NativeMediaError,traceModelInvocationandModelInvocationTrace. The Langfuse initializer, handler factory, attribute wrapper and disposal function stay internal.Compatibility
Existing Google server-tool signatures, stream smoothing, usage conversion and the no-port code paths are preserved;
responseModalitiesonGoogleClientOptionsis applied only when a port is configured. Vertex is out of scope for this port: it needs raw-part interception at the Vertex connection boundary and a host binding scoped to project and location, anddocs/native-media.mdrecords it as a separate extension.Tests
src/llm/google/native.test.ts: admission, ordered output, serialization, signed replay, storage failure, cancellation and text compatibility against the real wrapper and graph with a controlled provider boundarysrc/llm/google/native.http.test.ts: real HTTP server exercising stream and non-stream requests, blocked and empty responses, per-requestsystemInstruction, and continuation replay bytessrc/llm/google/native.abort.test.ts: socket-level cancellation and provider disconnect without unhandled rejectionssrc/specs/langfuse-callbacks.test.ts: failure usage on the generation span,traceModelInvocationlifecycle for callback and projected clients, redaction and disposalsrc/llm/google/inherited-stream-events.spec.ts,streamSmoothing.test.ts,graph-subagent.test.ts,subagent.test.ts: updated for the shared predicate andmodelRunIdNo built-package test is included; the consuming application runs its own contract against the installed package.
Run locally on this head:
npx tsc --noEmitsrc/llm/google,src/graphs,langfuse-callbacks,graph-subagent,subagent: 358 passed across 19 suitesnpm run build:dev,npm run test:circular-depsnpm run sort-imports:checkreports no changed fileNot run locally: full Jest suite and live Gemini requests.
src/llm/google/llm.spec.tsneedsGOOGLE_API_KEYand fails to start onmainas well.