Private AI chatbots: Brief introduction to running your own open "ChatGPT"

A brief introduction to large language models and software that you can use to run your own private AI chatbot. I first talk about why you may not want to rely on ChatGPT as your chatbot, the reasons being the censorship by their model and its guardrails, data security issues related to working with client data or personal data, and the API costs of large-scale applications, such as mining entire PubMed. I then give a quick comparison of OpenAI's GPT-4 and GPT-3.5 models to open alternatives available from Hugging Face, such as Falcon 180B, Mixtral 8x7B, Zephyr 7, and Microsoft's Phi-2. I cover two software tools that can be used for running LLMs locally, namely LM Studio and Oobabooga text generation web UI, and talk about their advantages and disadvantages. Finally, I give an overview of what it requires in terms of hardware to run these.
0:00 Introduction: ChatGPT, why you may not want to use it, and the open alternatives
0:59 Hugging Face: open LLM repository, LLM size comparison, and the power of small models
2:48 LM Studio: the easy way to run a chat bot, Hugging Face integration, and user interface
3:50 Oobabooga: open source, harder to install, more features, and ability to use a GPU server
5:00 Hardware: CPU mode is slow, GPU acceleration on a PC, GPU servers required for larger models
6:09 Conclusions: data scientists should try this and companies should have an in-house AI chatbot
Resource links:
Hugging Face - huggingface.co/
LM Studio - lmstudio.ai/
Oobabooga - github.com/oobabooga/text-gen...

[@RCQazi]

Indeed an inspiration for many young bioinformaticians out there, including me.

[@larsjuhljensen]

Thank you!

[@maasha2001]

Thanks Lars, Great overview - keep it up!

[@larsjuhljensen]

Thanks Martin, glad you liked it!

[@sigridopps3049]

Very well structured. Very informative. Wish you to succeed.

[@larsjuhljensen]

Thank you!

[@deweihu1003]

On macbook pro 13, m2, if you use GPU acceleration, it's super fast with zephyr 7b! RAM usage is less than 50 MB!

[@larsjuhljensen]

I’m surprised how it can use only 50 MB. The model file is a few GB. But I have indeed heard from several people that Apple silicon is very efficient for running LLMs.