Add webrtc.pipecat, a Pipecat transport on the native API

.github/workflows/ci.yml

@@ -63,6 +63,9 @@ jobs:
         # Intel macOS with the full matrix only
         exclude:
           - os: ${{ !(startsWith(github.ref, 'refs/tags/') || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && 'macos-15-intel' || '' }}
+        # the pipecat job's wheel on PRs, untested there
+        include:
+          - { os: ubuntu-latest, python: cp313 }
     env:
       WRTC_CACHE_DIR: ${{ github.workspace }}/.wrtc-cache
       WRTC_ARCHIVE_DIR: ${{ matrix.os == 'windows-latest' && format('{0}/.wrtc-cache/archives', github.workspace) || '' }}
@@ -71,6 +74,8 @@ jobs:
       CIBW_ENVIRONMENT_PASS_LINUX: UV_HTTP_RETRIES WRTC_TEST_TIME_SCALE
       # --gc-on-emit not on tags
       CIBW_TEST_COMMAND: ${{ github.event_name != 'push' && 'pytest -n auto {project}/tests && pytest -n auto {project}/tests --gc-on-emit' || 'pytest -n auto {project}/tests' }}
+      # built only for the pipecat job outside the full matrix
+      CIBW_TEST_SKIP: ${{ matrix.python == 'cp313' && !(startsWith(github.ref, 'refs/tags/') || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && '*' || '' }}
     steps:
       - uses: actions/checkout@v7
         with:
@@ -160,10 +165,39 @@ jobs:
       - uses: astral-sh/setup-uv@v7
       - run: .github/scripts/sanitizers-macos.sh -n auto ${{ matrix.args }}
 
+  # it pulls onnxruntime and numba
+  pipecat:
+    name: pipecat (${{ matrix.os }})
+    needs: wheels
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 20
+    strategy:
+      fail-fast: false
+      matrix:
+        # PRs: Linux only; every OS with the full matrix
+        os: ${{ fromJSON((startsWith(github.ref, 'refs/tags/') || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && '["ubuntu-latest","macos-15","windows-latest"]' || '["ubuntu-latest"]') }}
+    env:
+      UV_HTTP_RETRIES: 10
+    defaults:
+      run:
+        shell: bash
+    steps:
+      - uses: actions/checkout@v7
+      - uses: astral-sh/setup-uv@v7
+      - uses: actions/download-artifact@v8
+        with:
+          name: wheels-${{ matrix.os }}-cp313
+          path: wheelhouse
+      # webrtc from the wheel, the tests from the checkout
+      - run: |
+          uv venv --python 3.13
+          uv pip install wheelhouse/*.whl --group test --group pipecat-test
+      - run: uv run --no-sync pytest -n auto tests/pipecat
+
   # branch protection check
   ci:
     if: always()
-    needs: [lint, sdist, wheels, sanitizers, sanitizers-macos]
+    needs: [lint, sdist, wheels, sanitizers, sanitizers-macos, pipecat]
     runs-on: ubuntu-latest
     steps:
       # a skipped job passes: the sanitizers on tags

CHANGES.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+* Add `webrtc.pipecat`, Pipecat's SmallWebRTC transport on the native API, installed with `wrtc[pipecat]`, with `install_alias()` for Pipecat's development runner. See [Use with Pipecat](https://wrtc.marshal.dev/guides/pipecat/) by [@MarshalX](https://github.com/MarshalX)
+* Add a Pipecat voice bot example by [@MarshalX](https://github.com/MarshalX)
 * Add `webrtc.aiortc.install_alias()`, which makes `import aiortc` give the compat layer, for libraries that import aiortc themselves by [@MarshalX](https://github.com/MarshalX)
 * Add aiortc's `rtp`, `jitterbuffer`, `utils` and `clock` modules and the ICE server helpers of `rtcicetransport` to the compat layer by [@MarshalX](https://github.com/MarshalX)
 * Drop a compat layer track's buffered frames when its receiver's `_enabled` is set to `False`, and add the track's `_queue`, for libraries that skip stale frames by [@MarshalX](https://github.com/MarshalX)

README.md

@@ -150,6 +150,13 @@ Complete programs, ready to copy. The ones with dependencies declare them inline
 uv run https://raw.githubusercontent.com/MarshalX/python-webrtc/main/examples/janus_streaming.py
 ```
 
+## Pipecat
+
+Install `wrtc[pipecat]` and import the SmallWebRTC transport from `webrtc.pipecat`, and your
+[Pipecat](https://www.pipecat.ai) bot runs on Chrome's engine, with its clients unchanged. For Pipecat's development
+runner, call `webrtc.pipecat.install_alias()`. See [Use with Pipecat](https://wrtc.marshal.dev/guides/pipecat/) and the
+[voice bot example](https://github.com/MarshalX/python-webrtc/blob/main/examples/pipecat_bot.py).
+
 ## Status
 
 Pre-1.0: the API follows the specifications, but compatibility between versions isn't guaranteed until 1.0.0. See

docs/source/examples/index.md

@@ -48,6 +48,13 @@ Talks to the model through your microphone and speakers.
 Talks to Gemini from a browser page, through a Python server that keeps the API key.
 :::
 
+:::{grid-item-card} {octicon}`hubot;1em;sd-mr-1` Pipecat voice bot
+:link: pipecat_bot
+:link-type: doc
+
+A Pipecat bot on the development runner: a voice pipeline, or an echo with no keys.
+:::
+
 :::{grid-item-card} {octicon}`broadcast;1em;sd-mr-1` Live stream in the terminal
 :link: janus_streaming
 :link-type: doc
@@ -72,6 +79,7 @@ recorder
 browser_server
 openai_live
 gemini_live
+pipecat_bot
 janus_streaming
 whip_whep
 ```

docs/source/examples/pipecat_bot.md

@@ -0,0 +1,21 @@
+---
+description: "A Pipecat voice bot on Python WebRTC, served by Pipecat's development runner: speech to text, a model and text to speech, or an echo with no keys."
+---
+
+# Pipecat voice bot
+
+A [Pipecat](https://www.pipecat.ai) bot on [`webrtc.pipecat`](../guides/pipecat.md), served by Pipecat's development
+runner. With keys of [Deepgram](https://deepgram.com), [OpenAI](https://platform.openai.com) and
+[Cartesia](https://cartesia.ai), it's an interruptible voice assistant:
+
+```bash
+export DEEPGRAM_API_KEY=... OPENAI_API_KEY=... CARTESIA_API_KEY=...
+uv run https://raw.githubusercontent.com/MarshalX/python-webrtc/main/examples/pipecat_bot.py
+```
+
+With `--echo`, or without the keys, it sends your voice back. Open <http://localhost:7860> and press Connect.
+
+```{literalinclude} ../../../examples/pipecat_bot.py
+:language: python
+:caption: examples/pipecat_bot.py
+```

docs/source/guides/aiortc.md

@@ -49,6 +49,8 @@ import some_library  # its aiortc imports now get webrtc.aiortc
 `aiortc.x` then gives `webrtc.aiortc.x`, and a module the layer lacks raises `ModuleNotFoundError`. Call it first,
 in each process: it raises `RuntimeError` if `aiortc` is imported already.
 
+For Pipecat, use [`webrtc.pipecat`](pipecat.md) instead.
+
 The library's aiortc dependency stays installed but unused. Keep it if the library imports aiortc's dependencies, such
 as `aioice` or `cryptography`, directly.
 

docs/source/guides/index.md

@@ -1,5 +1,5 @@
 ---
-description: "Guides to Python WebRTC: connections, data channels, media, events, H.264, field trials, choosing a library and migrating from aiortc."
+description: "Guides to Python WebRTC: connections, data channels, media, events, H.264, field trials, choosing a library, migrating from aiortc and using Pipecat."
 ---
 
 # Guides
@@ -66,4 +66,11 @@ Python WebRTC, aiortc, the LiveKit SDK and GStreamer webrtcbin compared, and whe
 Run aiortc code by changing its imports, and what changes on the way.
 :::
 
+:::{grid-item-card} {octicon}`hubot;1em;sd-mr-1` Use with Pipecat
+:link: pipecat
+:link-type: doc
+
+Run Pipecat bots on the engine, with their runner and clients unchanged.
+:::
+
 ::::

docs/source/guides/pipecat.md

@@ -0,0 +1,150 @@
+---
+description: "Run Pipecat bots on Chrome's WebRTC engine: install wrtc[pipecat], import the transport from webrtc.pipecat, or alias it for the development runner."
+---
+
+# Use with Pipecat
+
+`webrtc.pipecat` replaces [Pipecat](https://www.pipecat.ai)'s SmallWebRTC transport: same classes, same HTTP
+signalling, same data channel messages, without aiortc, PyAV or OpenCV.
+
+```bash
+pip install "wrtc[pipecat]"
+```
+
+Tested with Pipecat 1.15.0 on Python 3.11+, and with Pipecat's JavaScript client (`@pipecat-ai/client-js` 1.16.0,
+`@pipecat-ai/small-webrtc-transport` 1.11.2) unchanged. Python WebRTC isn't affiliated with Pipecat or Daily.
+
+## Why a transport of its own
+
+Pipecat's own transport also runs on the [aiortc layer](aiortc.md), with `webrtc.aiortc.install_alias()`. But there
+every frame is converted to aiortc's and PyAV's objects and back, and Pipecat's workarounds for aiortc's limits run
+although the engine doesn't need them. `webrtc.pipecat` talks to the engine directly and keeps Pipecat's classes and
+protocol, so bots and clients don't change. Use the aiortc layer only for code that reaches into aiortc's objects.
+
+## Changing the imports
+
+```python
+# before
+from pipecat.transports.smallwebrtc.connection import IceServer, SmallWebRTCConnection
+from pipecat.transports.smallwebrtc.request_handler import SmallWebRTCRequest, SmallWebRTCRequestHandler
+from pipecat.transports.smallwebrtc.transport import SmallWebRTCTransport
+
+# after
+from webrtc.pipecat.connection import IceServer, SmallWebRTCConnection
+from webrtc.pipecat.request_handler import SmallWebRTCRequest, SmallWebRTCRequestHandler
+from webrtc.pipecat.transport import SmallWebRTCTransport
+```
+
+The rest of the bot stays as it is, except what [What differs](#what-differs) lists. `request_handler` is the only
+module that needs FastAPI.
+
+## The development runner and unchanged bots
+
+Pipecat's runner, its WhatsApp support and bots you don't change import Pipecat's transport themselves. Call
+`install_alias()` first:
+
+```python
+import webrtc.pipecat
+
+webrtc.pipecat.install_alias()
+
+from pipecat.runner.run import main  # its WebRTC routes now run on webrtc.pipecat
+```
+
+It raises `RuntimeError` if Pipecat's transport is imported already. Call it in every process: one started with spawn
+imports everything again.
+
+The runner shows its WebRTC routes only when aiortc is installed, though nothing imports it, so add Pipecat's
+`webrtc` extra:
+
+```bash
+pip install "wrtc[pipecat]" "pipecat-ai[runner,webrtc]"
+```
+
+## A bot
+
+```python
+from fastapi import BackgroundTasks, FastAPI
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.worker import PipelineWorker
+from pipecat.transports.base_transport import TransportParams
+from pipecat.workers.runner import WorkerRunner
+
+from webrtc.pipecat import IceServer, SmallWebRTCConnection, SmallWebRTCTransport
+from webrtc.pipecat.request_handler import SmallWebRTCPatchRequest, SmallWebRTCRequest, SmallWebRTCRequestHandler
+
+app = FastAPI()
+handler = SmallWebRTCRequestHandler(ice_servers=[IceServer(urls='stun:stun.l.google.com:19302')])
+
+
+async def run_bot(connection: SmallWebRTCConnection) -> None:
+    transport = SmallWebRTCTransport(
+        webrtc_connection=connection,
+        params=TransportParams(audio_in_enabled=True, audio_out_enabled=True),
+    )
+    # stt, llm, tts and the context aggregators, as in any Pipecat bot
+    pipeline = Pipeline([transport.input(), stt, user, llm, tts, transport.output(), assistant])
+    runner = WorkerRunner(handle_sigint=False)
+    await runner.add_workers(PipelineWorker(pipeline))
+
+    @transport.event_handler('on_client_disconnected')
+    async def on_client_disconnected(transport, connection):
+        await runner.cancel()
+
+    await runner.run()
+
+
+@app.post('/api/offer')
+async def offer(request: SmallWebRTCRequest, background_tasks: BackgroundTasks) -> dict[str, str]:
+    async def start(connection: SmallWebRTCConnection) -> None:
+        background_tasks.add_task(run_bot, connection)
+
+    return await handler.handle_web_request(request, start)
+
+
+@app.patch('/api/offer')
+async def add_candidates(request: SmallWebRTCPatchRequest) -> dict[str, str]:
+    await handler.handle_patch_request(request)
+    return {'status': 'success'}
+```
+
+`await handler.close()` on shutdown disconnects every client. The [Pipecat bot example](../examples/pipecat_bot.md)
+is a complete bot on the runner.
+
+## Configuration
+
+- **ICE servers.** As in Pipecat: URLs, or `IceServer(urls=..., username=..., credential=...)`.
+- **When the answer goes out.** The protocol sends the bot's candidates only in the answer, so it waits for the host
+  candidates of every interface, plus a server-reflexive one with STUN and a relay one with TURN, at most
+  `ice_gathering_timeout_secs` (5 s). A STUN server on the bot's own private network can't give a server-reflexive
+  candidate, so there the answer waits the whole timeout.
+- **Port range.** `port_range=(low, high)` keeps the bot's candidates in a range of ports. In Docker, publish the same
+  range, like `-p 40000-40019:40000-40019/udp`. The runner makes its own handler, so this needs a server of your own.
+- **Codecs.** The answer follows the client's offer. For H.264, see [H.264](h264.md).
+
+## What differs
+
+- **Tracks.** The `*_input_track()` methods return a {obj}`webrtc.MediaStreamTrack`, with no `SmallWebRTCTrack`
+  around it; a source turned off comes with the `track-status` event. The `replace_*_track()` methods are coroutines.
+- **Leaving.** A client that closes its connection or data channel is closed at once. A client that pings, as
+  Pipecat's JavaScript client does, and stays `disconnected` for 8 s without a ping, offer or candidate is closed
+  before its connection would fail. `connection_timeout_secs` counts from the answer.
+- **Requests.** A `restart_pc` offer drops the old `pc_id` from the handler.
+- **Not there.** `force_transceivers_to_send_recv()`, `PIPECAT_SCTP_MAX_CHUNK_SIZE`,
+  `SCREEN_VIDEO_JITTER_BUFFER_CAPACITY` and the key frame renegotiation on connect: the engine doesn't need them.
+  `capture_participant_video('camera')` without a participant id, deprecated in Pipecat 1.13, isn't supported.
+
+## What it fixes
+
+Each checked against Pipecat's transport on aiortc, with Pipecat's workaround removed where it has one:
+
+- Large screen-share key frames, like 1080p ones, arrive whole.
+- A video track the pipeline doesn't read stays flat in memory ([#3066](https://github.com/pipecat-ai/pipecat/issues/3066)).
+- The data channel works on paths with a small MTU, like VPNs.
+- Lost audio packets are concealed, so audio stays whole on a lossy network.
+- Video adapts to a capped link instead of stalling.
+- The answer goes out as soon as the candidates are enough, not after seconds of gathering ([#2426](https://github.com/pipecat-ai/pipecat/issues/2426)).
+- `port_range` lets Docker publish only a range of UDP ports ([#1808](https://github.com/pipecat-ai/pipecat/issues/1808), [#1570](https://github.com/pipecat-ai/pipecat/issues/1570)).
+- A bot uses less CPU per session: media stays native, with no PyAV or OpenCV conversions.
+- A client that vanishes is noticed sooner ([#3140](https://github.com/pipecat-ai/pipecat/issues/3140)).
+- A dead first TURN server falls back to the next.

docs/source/index.md

@@ -203,6 +203,7 @@ guides/h264
 guides/field-trials
 guides/comparison
 guides/aiortc
+guides/pipecat
 ```
 
 ```{toctree}

docs/source/llms-txt.md

@@ -17,6 +17,7 @@ of date.
 - Handlers are registered with `on`, as a call or a decorator, and can be coroutines.
 - Dictionaries are typed dataclasses, and enums compare equal to the strings of the specifications.
 - Code written for aiortc runs after installing `wrtc[aiortc]` and changing `from aiortc` to `from webrtc.aiortc`.
+- Pipecat bots run on it with `wrtc[pipecat]`, importing their transport from `webrtc.pipecat`.
 - Prebuilt wheels for CPython 3.9 to 3.15 and free-threaded 3.14t and 3.15t, on Linux, macOS and Windows.
 
 ## Start
@@ -25,6 +26,8 @@ of date.
 - {doc}`Choosing a Python WebRTC library <guides/comparison>`: Python WebRTC, aiortc, the LiveKit SDK and GStreamer
   webrtcbin compared, and when each is the better choice.
 - {doc}`Migrating from aiortc <guides/aiortc>`: run aiortc code by changing its imports, and what changes on the way.
+- {doc}`Use with Pipecat <guides/pipecat>`: run Pipecat bots on the engine with `wrtc[pipecat]`, their runner and
+  clients unchanged.
 
 ## Guides
 
@@ -62,6 +65,8 @@ of date.
   speakers.
 - {doc}`Voice agent with Gemini Live <examples/gemini_live>`: a browser talks to Gemini over WebRTC through a Python
   server, which streams the audio to the Live API over a WebSocket.
+- {doc}`Pipecat voice bot <examples/pipecat_bot>`: a Pipecat bot on the development runner, a voice pipeline or an
+  echo with no keys.
 - {doc}`Live stream in the terminal <examples/janus_streaming>`: plays a stream of the public Janus demo server.
 - {doc}`WHIP and WHEP <examples/whip_whep>`: plays a WHEP stream, a public demo by default, or publishes the synthetic
   camera and microphone over WHIP.

examples/README.md

@@ -36,6 +36,15 @@ OPENAI_API_KEY=sk-... uv run https://raw.githubusercontent.com/MarshalX/python-w
 GEMINI_API_KEY=... uv run https://raw.githubusercontent.com/MarshalX/python-webrtc/main/examples/gemini_live.py
 ```
 
+### [pipecat_bot.py](pipecat_bot.py)
+
+**A Pipecat voice bot on Pipecat's development runner.** An interruptible voice assistant with Deepgram, OpenAI and Cartesia keys; with `--echo`, or without the keys, it echoes your voice. Needs Python 3.11+. Open http://localhost:7860 and press Connect:
+
+```bash
+export DEEPGRAM_API_KEY=... OPENAI_API_KEY=... CARTESIA_API_KEY=...
+uv run https://raw.githubusercontent.com/MarshalX/python-webrtc/main/examples/pipecat_bot.py
+```
+
 ### [janus_streaming.py](janus_streaming.py)
 
 **Watching a live stream in the terminal.** Plays a stream of the public [Janus](https://janus.conf.meetecho.com/) demo server: the video drawn with colored characters, the audio on your speakers, if there are any, so it runs on a server too. No account or key: