Skip to content

feat: add native media port for Google image output and host model-invocation tracing - #553

Open
usnavy13 wants to merge 7 commits into
LibreChat-AI:mainfrom
usnavy13:feat/native-media-port
Open

usnavy13 wants to merge 7 commits into
LibreChat-AI:mainfrom
usnavy13:feat/native-media-port

Conversation

@usnavy13

Copy link
Copy Markdown
Contributor

Summary

  • add NativeMediaPort, an injected host boundary that CustomChatGoogleGenerativeAI calls to authorize an invocation, persist each text or image part before it is emitted, record completion or failure, and restore signed parts for continuation
  • return Gemini image parts as ordered image_file content alongside text through _generate, _streamResponseChunks and streamEvents, and keep that order through graph dispatch and content aggregation
  • carry per-invocation responseModalities and thoughtSignature on text and inline-data parts, including empty signed text, so image continuations replay exactly
  • add UsageBearingError / NativeMediaError so provider consumption survives a failed persistence, a blocked response or a cancellation, and record that usage on the Langfuse generation span
  • add traceModelInvocation(params, work, project?) so a host can trace model calls made outside a graph run with the SDK owning handler creation, attribute scoping and disposal
  • forward the request AbortSignal to generateContentStream and observe its aggregate-response promise so cancellation closes the provider connection without a later unhandled rejection
  • send systemInstruction per request instead of mutating the shared client
  • attach modelRunId to subagent usage events so native failure usage can be matched to its call

Supersedes #41. Related: LibreChat-AI/LibreChat#6065 (native inline image output). Proposal and host integration: LibreChat-AI/LibreChat#16140.

Why

Gemini image models return inlineData parts in an ordinary completion, no tool call involved. A host cannot let raw image bytes enter graph state, in-run replay, Langfuse, SSE or subagent results, and it cannot emit a file reference before the file exists. The adapter therefore hands each part to the host and waits for a durable reference before yielding, which gives bounded backpressure without a queue of unpersisted bytes. Authorization, storage, retention, accounting and recovery stay on the host side of the port; the adapter owns provider parsing and ordered emission.

Hosts also run model calls outside a Run (titles, bounded provider transports). Exporting the raw Langfuse handler factory and disposal helpers would make every host responsible for the SDK's internal lifecycle, so the package exposes one owned lifecycle instead and keeps those internals private. See docs/adr/0010-persist-native-media-before-model-output.md and docs/adr/0011-trace-external-model-invocations.md.

Design

Port callback Host responsibility
start Authorize the invocation; return the permitted responseModalities. Rejection prevents the provider request.
part Persist a text or image part with its optional thoughtSignature; return visible text or an image_file reference only after persistence succeeded.
complete Record that the response finished and its visible parts were persisted.
fail Record an incomplete invocation with an aborted, provider or storage reason, failure-only usage and an optional provider outcome.
restore / restoreBatch Authorize a continuation reference and return the exact signed part.
  • Only an explicit IMAGE admission enables rejection of empty, blocked or invalid image responses; a port attached to a text model does not change ordinary response handling or add a modality selection.
  • Without a port, ordinary text invocation is unchanged, inline image output is rejected before it can be emitted, and a native_media.continuationRef in history is rejected rather than silently dropped.
  • isStructuredGoogleContentPart replaces the server-side-tool predicate in Graph.ts and stream.ts so image_file and native_media parts follow the same ordered-dispatch path Google tool payloads already use; the aggregator copies image_file parts and native_media markers through.
  • traceModelInvocation initializes the resolved destination, builds a scoped handler, gives LangChain callers callbacks and external clients a result projection, records a generic failure and rethrows the original error, and disposes the handler in finally. Initialization, export and projection failures never repeat inference or replace a result.
  • Public exports: NativeMediaPort, NativeMediaPart, NativeMediaContent, NativeMediaReference, NativeMediaRestoreInput, NativeMediaProviderOutcome, UsageBearingError, NativeMediaError, traceModelInvocation and ModelInvocationTrace. The Langfuse initializer, handler factory, attribute wrapper and disposal function stay internal.

Compatibility

Existing Google server-tool signatures, stream smoothing, usage conversion and the no-port code paths are preserved; responseModalities on GoogleClientOptions is applied only when a port is configured. Vertex is out of scope for this port: it needs raw-part interception at the Vertex connection boundary and a host binding scoped to project and location, and docs/native-media.md records it as a separate extension.

Tests

  • src/llm/google/native.test.ts: admission, ordered output, serialization, signed replay, storage failure, cancellation and text compatibility against the real wrapper and graph with a controlled provider boundary
  • src/llm/google/native.http.test.ts: real HTTP server exercising stream and non-stream requests, blocked and empty responses, per-request systemInstruction, and continuation replay bytes
  • src/llm/google/native.abort.test.ts: socket-level cancellation and provider disconnect without unhandled rejections
  • src/specs/langfuse-callbacks.test.ts: failure usage on the generation span, traceModelInvocation lifecycle for callback and projected clients, redaction and disposal
  • src/llm/google/inherited-stream-events.spec.ts, streamSmoothing.test.ts, graph-subagent.test.ts, subagent.test.ts: updated for the shared predicate and modelRunId

No built-package test is included; the consuming application runs its own contract against the installed package.

Run locally on this head:

  • npx tsc --noEmit
  • src/llm/google, src/graphs, langfuse-callbacks, graph-subagent, subagent: 358 passed across 19 suites
  • npm run build:dev, npm run test:circular-deps
  • ESLint on changed files; npm run sort-imports:check reports no changed file

Not run locally: full Jest suite and live Gemini requests. src/llm/google/llm.spec.ts needs GOOGLE_API_KEY and fails to start on main as well.

@usnavy13
usnavy13 marked this pull request as ready for review September 20, 2026 23:26
@usnavy13
usnavy13 force-pushed the feat/native-media-port branch from b8071e8 to a09e642 Compare September 24, 2026 14:23
usnavy13 and others added 7 commits October 2, 2026 23:17
Validate native output and replay boundaries, preserve model usage through failures and subagent execution, integrate tenant tracing, and ship the shared CJS/ESM contract fixture.
…y conventions

Remove config/native-media-contract.test.mjs with its npm script and CI step; the consuming application keeps its own contract against the installed package. Drop the unused root initializeLangfuseTracing export, restructure ADR 0010 and 0011 to the existing ADR layout, remove host-specific wording from the native media docs, and list the doc under README Documentation.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
@usnavy13
usnavy13 force-pushed the feat/native-media-port branch from a09e642 to c39c66a Compare October 3, 2026 03:25
usnavy13 added a commit to usnavy13/LibreChat that referenced this pull request Oct 4, 2026
Moves the native media bridge to @librechat/agents 4.0.1, the release dev
targets: both consumers pin it exactly and patches/@librechat+agents+4.0.1.patch
is regenerated from LibreChat-AI/agents#553 rebased onto 4.0.1, replacing the
3.9.0 patch.

Conflict resolutions carry Studio onto dev's reshaped code:
- "Create media" moves from the removed AttachFileMenu into the Attach and
  tools palette (useAttachItems), with the same chat/create gating.
- Native media stream handling follows dev's move into hooks/SSE/steps.
- Media file claims and native signatures wrap dev's single writeMessage path.
- Cookie authentication for share, image and media files stays one module
  (images/cookies.ts) and gains dev's two-factor enrollment and token
  retirement checks; dev's parallel auth/share.ts is dropped.
- Ban checks use dev's getBanIp without mutating req.ip.

Studio components now satisfy dev's design lint: spacing and color move off
Collapsible primitives onto plain wrappers, section actions use the
section-action Button variant, image frames size through a --media-ratio
custom property, and two shared variants cover needs other screens share
(Input leading-icon for search fields, Button composer for composer-row
controls; pressed header-action buttons fill through aria-pressed).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant