Repository navigation
Can Pathway llm-app be used for open source LLM without OpenAI embedding API? #57
Description
I have installed several quantized open source LLM (e.g. LLaMA.cpp, ChatGLM2-6B-int4 ..etc) for private Chatbot apps and I want to use some locally installed embedding model (e.g. SentenceTransformers) instead of OpenAI embedding API.
My question is:
Can Pathway llm-app be used for open source LLM without OpenAI embedding API?
Any example or tutorials on how to do that?
Thanks
Benny
Activity
janchorowski commentedon Aug 23, 2023
Yes, it can be used without OpenAI models, both the embedder and the LLM are customizable. For start chek out this example which uses local models: https://github.com/pathwaycom/llm-app/blob/main/examples/pipelines/local/app.py
dxtrous commentedon Aug 23, 2023
@bennylam Just to make sure if the embedder model interface, linked by @janchorowski above, is what you were looking to use?
If you prefer to rely on a different embedding library than sentence-transformers, happy to see what can be done.