Probably not, if you know what you're doing already or if you have a lot of time. I'm developing a a LLM application on my own designed to be run offline in businesses and am looking into Haystack. I don't have a lot of infrastructure by any means, and even I can manage to do stuff like this at a decent scale (even when i rolled by own RAG without Haystack). The key things you need: - some vector store (could be something as simple as SQLite in combo with FAISS or PyTorch/Pandas) - an "primary" LLM to be used for answering with a decent context size (15k+) - a model that will be used to create vectors from documents - a place to store your documents - a way to get questions and reply with answers (i.e., "serve the [primary] model") Stuff like LangChain and Haystack just make the things above easier to use together. Its actually all very simple at its core, and the most difficult part really becomes the complicated string manipulation (if you're not using LangChain/Haystack etc for your full stack). If you have any questions I'd be happy to assist as much as I'm able.
Is it hard for someone to set this up in a small firm ?
Probably not, if you know what you're doing already or if you have a lot of time. I'm developing a a LLM application on my own designed to be run offline in businesses and am looking into Haystack. I don't have a lot of infrastructure by any means, and even I can manage to do stuff like this at a decent scale (even when i rolled by own RAG without Haystack).
The key things you need:
- some vector store (could be something as simple as SQLite in combo with FAISS or PyTorch/Pandas)
- an "primary" LLM to be used for answering with a decent context size (15k+)
- a model that will be used to create vectors from documents
- a place to store your documents
- a way to get questions and reply with answers (i.e., "serve the [primary] model")
Stuff like LangChain and Haystack just make the things above easier to use together. Its actually all very simple at its core, and the most difficult part really becomes the complicated string manipulation (if you're not using LangChain/Haystack etc for your full stack).
If you have any questions I'd be happy to assist as much as I'm able.