Streaming responses

[niutech]

This is a follow-up on issue #46. Is it possible to stream responses from LLM in Q&A? For example, LiteLLM allows streaming responses by adding stream=True. Thanks!

[Saksham65]

Hi there,
Thanks for your question. As discussed internally:

1. Current Support: Our LiteLLMChat wrapper does not support using stream=True to split a single LLM output into multiple messages over time. The current wrapper returns the full response once generated.

2. Technical Considerations: Implementing streaming would require a significant redesign—specifically around how UDFs return results (for instance, allowing them to yield partial outputs). This isn’t supported by the current engine.

3. Kafka and Output Connectors: While you can stream question and answer pairs to systems like Kafka by adjusting how the webserver handles responses, splitting one answer into multiple parts is not currently implemented.

[niutech]

Thank you for your reply. Is it possible to bypass UDFs altogether and just stream raw responses from LiteLLM?

[aniruddhaadak80]

The limitation makes sense technically, but it does leave an obvious product gap because streaming changes how responsive these systems feel in practice. Routing through output connectors can work, though it pushes the solution into architecture workarounds instead of something the framework supports directly.

If native streaming is not near-term, even a recommended pattern for incremental response delivery would help. Right now the missing piece is less whether it is theoretically possible and more what the cleanest supported compromise looks like.