You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Promote TwelveLabsVideoParser and MarengoEmbedder from the Video RAG template into pathway.xpacks.llm #255
pathwaycom/serviette#129 added a Video RAG with TwelveLabs template containing two reusable building blocks:
TwelveLabsVideoParser (pw.UDF) — uploads video bytes as a TwelveLabs asset and turns them into a rich text description via the Pegasus video-understanding model. Output is the standard list[(text, metadata)], so it chunks/embeds/indexes exactly like the built-in PDF parsers.
MarengoEmbedder (BaseEmbedder) — a retriever embedder backed by the Marengo multimodal model, returning 512-dim vectors in a shared text/image/audio/video embedding space.
Both currently live in a per-template pathway_twelvelabs package. This issue proposes promoting them into pathway.xpacks.llm as first-class, supported components.
Motivation
Fills a real gap in modality coverage.xpacks/llm/parsers.py already ships ImageParser and AudioParser but has no video parser. MarengoEmbedder would be the first xpacks embedder in a shared multimodal space. New modality, zero new infrastructure.
Already conforms to xpacks conventions.MarengoEmbedder subclasses BaseEmbedder like OpenAIEmbedder/GeminiEmbedder/BedrockEmbedder and reuses udfs.async_executor, ExponentialBackoffRetryStrategy, CacheStrategy, max_batch_size. TwelveLabsVideoParser mirrors the existing pw.UDF parsers. Minimal adaptation required.
Generic, not template-specific. Prompt, model, API key, retries, and caching are all parameterized; no template-specific assumptions in the classes themselves.
Proposed scope
Add the two components under pathway/xpacks/llm/ fold into embedders.py / parsers.py with exports in the respective __init__.
Keep twelvelabs an optional dependency via the existing lazy-import + ImportError guidance pattern, so plain import pathway is unaffected. Create a twelvelabs section in pyproject.toml so that the dependency can be installed via pip install pathway[twelvelabs].
Port the unit tests (stubbed SDK, no network) into python/pathway/xpacks/llm/tests/; keep the live smoke test gated behind TWELVELABS_API_KEY.
Ensure all introduced classes are properly documented.
Add a CHANGELOG entry (user-visible Python API).
Shortly after the components are ported into the Pathway framework and the changes are released, it makes sense to refactor the llm-app template so that it imports components directly from Pathway.
Hi @zxqfd555 ! I'd love to take this one on if it's available. I'll get started on porting the TwelveLabsVideoParser and MarengoEmbedder over to xpacks.llm along with the optional twelvelabs dependency.
Activity
thinkapoorv commentedon Jun 30, 2026
Hi @zxqfd555 ! I'd love to take this one on if it's available. I'll get started on porting the
TwelveLabsVideoParserandMarengoEmbedderover toxpacks.llmalong with the optionaltwelvelabsdependency.Promote TwelveLabs Video RAG components into core xpacks/llm
Promote TwelveLabs Video RAG components into core xpacks/llm