Unleash the power of Local LLM's with Ollama x AnythingLLM
HTML-код
- Опубликовано: 28 май 2024
- Running local LLMS for inferencing, character building, private chats, or just custom documents has been all the rage, but it isn't easy for the layperson.
Today, with only a single laptop, no GPU, and two free applications you can get a fully private Local LLM RAG chatbot running in less than 5 minutes!
This is no joke - the teams at Ollama and AnythingLLM are now fully compatible, meaning that the sky is the limit. Run models like LLama-2, Mistral, CodeLLama, and more to make your dreams a reality with only a CPU.
Ollama: ollama.com/
AnythingLLM: useanything.com/download
And you are off to the races. Have fun!
Chapters
0:00 Introduction to Ollama x AnythingLLM on a laptop
0:36 Introduction to Ollama
1:11 Technical limitations
1:48 Ollama Windows is coming soon!
2:11 Let’s get started already!
2:17 Install Ollama
2:25 Ollama model selection
2:41 Running your first model
3:33 Running the Llama-2 Model by Meta
3:57 Sending our first Local LLM chat!
4:53 Giving Ollama superpowers with AnythingLLM
5:31 Connecting Ollama to AnythingLLM
6:45 AnythingLLM express setup details
7:28 Create your AnythingLLM workspace
7:45 Embedding custom documents for RAG for Ollama
8:22 Advanced settings for AnythingLLM
8:53 Sending a chat to Ollama with full RAG capabilities
9:30 Closing thoughts and considerations
#opensource #llm #privategpt #localagent #chatbot #ragchatbot #rag #openai #gpt #customgpt #localai #ollama #freechatbot #aitools #aitoolsyouneed #aitoolsforbusiness #freeaitool #freeaitools #llama2 #mistral #langchain #tutorial #aitutorial #aitools2024 #aiforbeginners #aiforproductivity
Ollama is now officially on windowssss!!!!
.. Docker... It always was
Docker image is a WSL2...Linux port.
Its not Windows, you're just using a third party app to run it in windows at the cost of resources to run the docker server. (which ive been doing for a long time, duh...)
I mean its out on windows which makes development SIGNIFICANTLY easier. ;)@@geekazoid47
lol yeah I just saw the release right after this video posted. Good call. I currently have my GPU passed through to my windows VM for different compute things. Glad to see ollama on windows. I’ll definitely try this out.
@@geekazoid47 pain in the ass to setup for someone with no knowledge on setting up docker
@@3ace846 wdym docker is the easiest thing much better than regular commandline installing of things
Thank you for your hard work!
This is really a game changer,now people can build they personal chatbots,with massive databases and knowledge about they favorite topics,without using or paying online services.
This is my new favorite piece of software together with LM studio.
Huge respect and keep up with good work.
Amazing insight! I was already using Ollama, and adding AnythingLLM is the icing on the cake 👍. Thanks for the video!
What do you think about this model , what’s your experience?
The only video I found that gave me exactly what I was looking for...THANKS!
This is mind blowing! Thank you, much appreciated!
Great video, Thanks very much
This is great, and nice to see Linux support is there now as well.
Perfect video! It has changed a bit though, so you don't need to pick lance etc, it just shows it as chosen.
Great video thanks for developing this
thank you for your work
This is exactly what I was looking for! :)
This is very cool Tim!
Excellent! Thank you!
Nicely done. I like the flexibility of the tool.
I love AI, and I'm dabbling in all kinds of things, but I could never get LLMs to really work on my local machine. THIS is amazing. I got it working on my gaming laptop.
Looks cool! Gonna try it right now
what about agents? do you plan to integrate agents? with crewai, or autpgpt, also why only ollama, you can integrate llm studio as well! sounds like a promising project.
Hi, have to thank you for the tutorial. It is very useful to quick test different LLM models, vector databases and other points with different data content. And just after that think about what stack is suitable to index your own data and try to deploy it on a server-side. Thanks!
Great stuff, will be putting it on my windows machine...can't wait till a linux/Ubutu Appimage is released.
Working on it - just having trouble getting a stable Ubuntu (with UI) running so i can fully test the end-to-end flow
@@TimCarambatTake your time, better to get it right then having to patch-it.
@@GregRutkowski I got an AppImage live on the main site now. Tested and built on Ubuntu 22, not sure how compatible that build system is with other distros, but I dont have inifinite linux machines (or time!)
useanything.com/download
@@TimCarambat Time to test it out...
Really Thank you for your help. I'm trying.
well done. Looks solid.
Nice tool indeed, thanks
I have a spare iMac, nice to see it works on intel.
I get to use it for something other than youtube videos on the corner of my desk.
You are amazing, great video
This is awesome! I'm so glad I found this video cause being able to talk to my pdfs was something i've been wanting to do for a while!
I do have some questions though. I have pdfs of tabletop RPG books and when I ingest a certain book and begin asking it rules questions it seems to 'reach out' to other sources and rules that aren't in the book and the responses are incorrect. Is there a way to narrow the chat to only answer rules and such about that particular game or book instead of adding knowledge on other games it has?
Ah, this is an error class i just refer to as "general knowledge creep". The only semi-good remedy is just a really strong system prompt explicitly detailing that only knowledge in the context should be used and to not use it's existing understanding of a game on how rules for this game should work - or something along those lines.
System prompt can def fix this and depending on the model will vary in performance.
Great explanations in your its eas very useful and helpful information thanks Sir😊😊😊😊😊😊
Many thanks for introduction the AnythingLLM. Is there any date mentioned when this app will be release for Ubuntu. I have to admit it blocks me 😀but there are so cool features to automate our work. Or maybe is there any similar app to integrate Ollama with vector DB, embeddings, etc in similar way ?
Kudos on Anything LLM.❤
Hello, anything in the pipeline for using the new ollama embeddings? they are super super fast :-)
Already live in main repo. Will be live in next weekly release
You are a legend❤️
THIS IS AMAZING
Thanks Tim for this great stuff !
I am interested in generating a document based on data within multiple local files. Do you think it's easily feasible with ollama/AnythingLLM?
thank you for this awesome explanation
great work tim
Awesome thank you ! The possibility to learn by giving it docs/URL is so cool. I have one question though: Is it possible to use AnythingLLM with an API ? So we can use a model (Llama 3), add data thanks to AnythingLLM, then use an API to talk to AnythingLLM that returns response ? Thank you in advance for the answer, I guess many people wonder the same ^^.
Ollama works very slow on Windows. Tested using LM studio and it's working.
This was nice. What other integration options are available, particularly on window with a good GPU
If on windows you can run LMStudio (just like ollama but with a UI) and LocalAI (requires docker to run).
Outside of that you can also turn to hosted LLMs on OpenAI, Azure OpenAI, Anthropic, Google Gemini
Together AI, and Mistral
Just for the record Ollama does support embedding models. They work very well (very fast) for text, not sure about other modalities.
We support Ollama embedding now! When i made this video we had not yet (or maybe it didnt even exist in Ollama yet?)
That was really fantastic. Tm once at work i’d go full hog at it and they let you know! By the way, do you plan to implement DSPy somehow?
Unlikely because it cant fit into our stack (since its Python). The idea of the framework however is not impossible to support
When you embedd website, does it only embedd that one webpage or goes recursively through the entire domain (like web spyders) do?
Can I specify the context size to use myself (for models with very large contexts like command-r ?)
Awesome!
Bro this is amazing!
Currently training models seems like a big hastle most training is done via google collab and such any plans on adding an easy UI interface to train models locally inside AnythingLLM?
Being able to fine tune off the chat and such is on the roadmap, but is still conceptual since 99% of people cannot train a model on their device.
So we may integrate or connected to some provider which can enable this to happen more easily. We allow you to export the chats in a fine-tune format already to encourage this behavior and id love to close the loop on it
@@TimCarambat Thanks Tim appreciate you.
This looks great thanks!
@Tim This is GREAT STUFF. Your putting in some serious effort and it shows. Can I use this to setup on a client site? I am a software developer and would like to create embeddings specific to ai python projects and documentation... what are a couple options i should consider. Thanks
You absolutely could but youll want to run our Docker offering because it is meant for that. The desktop app is really a local "single-player" solution.
The docker app has a full API, multi-user support, etc etc
would you consider adding more plaintext files, such as .py files? Just installed the latest version in docker and its great, love everything about this!
You know it's funny, we allow pulling in and embedding of whole github repos, but not the independent code files. I will work on resolving that for you - thanks for pointing it out 😄
@@TimCarambat you are absolutely welcome, if i helped even 0.01% it made my day!
Reply just to say we support this now (and have for a few weeks now!)
I love the idea of AnythingLLM but for some reason I cant get it to relate to the documents I provide.. and Ideas what might be causing that?
Thanks a lot. I have 2 questions. Does it have access to the internet to fetch recent updated information? What's the best model to use for software engineers?
You can scrape websites with the built in scraper but its not "internet connected". Not like an agent-LLM.
CodeLlama for local LLM. Claude-3 Opus for closed source
Thank you Tim very helpful. May I ask the similarities/diff between anythingllm and privategpt ?
We are a one-click single Desktop app you can install, no code or python or docker config
We support many LLM, embedders, and vectordatabases where the defaults are all local
We come with built-in data connectors like Github, website scraping, bulk website scraping, Confluence, etc etc
We also have agent capabilities for _all_ models, even if that model does not support tool calling.
We also support multi-user on our Dockerized version of the app (if you need that)
- that is just off the things i do know about PrivateGPT.
Nothing wrong with PrivateGPT at all, i think our tool is much more "accessible" and more all-in-one. Different strokes for different folks
very impressive :-o
sweet tool bro!
I get "-- loading available models --" in the Chat Model Selection combobox after selecting Ollama Base URL (that is correct btw) because I can open ollama one ui. What am I missing here?
Same. Whats the issue? Have you found any solutions?
@@Musaddique_MMALI I reinstalled today because I formatted my computer. Now is working fine.
can you make few more videos specifically for the agents and details of embedding and possible troubleshootings please.
Can you put an example using Hugging Face? I don't know what happens but Anythingllm is asking ChatGPT and not my documents.
Good stuff. Are there any ways to connect to APIs (similar to how chatGPT works) where we could use large data sets for free? Adding individual websites and files could take a lifetime...any workarounds?
Next we just need to serve a webpage with authentication. Complete package.
The docker version has multi-user with auth :)
I have a M2 Max Macbook Pro. Ollama with 7B Llama works super fast when used in Terminal. Why is it slow when I am using AnythingLLM along with it. There is a drastic time difference! Pls advise.
That's cool!
btw i think you've rendered a 16x10 ratio video to a 16:9
It looks a little weird 😅
Is there a timer or countdown clock for the embedding and processing time?
Not currently, it would be nice to add though for obvious reasons
Looks great! Do you have an idea of when Linux will be available?
Been working on it this week! Looks like the app image should be good by next release, likely next week
It seems to be having an issue reading the entirety of my Docx, only 103KB. It's the CBA for my workplace. Once embedded it states that there is no context for the question I've asked, though there most certainly is. (IE, shift schedule times, etc...), it will then proceed to read OTHER parts of the document.
I wonder if there are certain document types that work better than Docx? or if anyone has any suggestions?
Hi, I noticed you are still in chat mode when you asked questions, isn't it better to switch to query mode ?
Chat is general knowledge + documents
Query is just documents. In this case, since each query had used context - the results would have been the same
When you scape a URL does it extend the scrape to every page on the site? Or would I have to add all the pages of a site manually? How deep is the URL scrape?
The one in the uploader is currently just the url. Understandably that is not great for large url ingestion and there is a piece of work being done to add deep scraping (or at least multi link, possibly agent based) to the tool.
Thank
Hello, Tim, how are you? Thanks for the video! I have a question, I have Ollama on a ex mining rig with 5 GPUs, the response time is very good, but when I ask 2 questions in parallel it does not answer in parallel, he answers the first question and the second question remains in “queue” until I finish answering the first question, could you help me? Thank you
Can I use AnythingLLM locally and privately only as an API like the OpenAI API? In addition, do you offer anything like the Assistants API, where you can have models that have pre-defined instructions on how to respond?
Thanks for the video, but I'm not sure why, but my windows version of anythingllm is slightly different and I can only follow so far before what's on my screen is different than yours. Is there a video for the windows version? Please make sure that you explain the differences.
Where is the disconnect for you? This video is quite old now and we have improved the product a lot since
2 months have passed since the original video, 2 questions. I am running a 12 GB vram RTX3060 GPU. Will LM Studio be a better solution than Ollama? 2. Now Llama 3 has come out, will thos work with what is presumably a much more capable LLM?
Thanks for the tutorial, and the combination with Ollama and AnythingLLM are exactly what I need. But when I try to embed with several PDFs, always got failed error. Do you know what I did wrong? I'm running Ollama from AnythingLLM with Mistral as LLM. Normal chat is ok. I'm on mac.
Actually I reported a bug, maybe it's not a bug but something I set wrong. Sorry for false reporting then. github.com/Mintplex-Labs/anything-llm/issues/821, thanks again for making this great app.
@@Gabriecielowhat kind of use case you have tested with the embed files (what format?)?
I am interested in generating a document based on data within multiple local files. So you think it's feasible with ollama/AnythingLLM?
@@m.bouanane4455 I tried PDF, more text books. Mitral 7B and Gemma tested, Gemma lost a lot to Mitral. And it works with multiple docs too. One thing I felt is AI needs my questions to drive him read and think deeper into the book.
Tested with PDF mainly, works fine. And we could easily switch all open-source LLMs. For using AnythingLLM to generate doc, you should ask Tim, I only tried use interface, didn't know if API is available or other tools for you to drive. @@m.bouanane4455
I love the ease of set up and the local private models.
First attempts at use are not good in terms of models ability to understand and formulate answers to questions based on provided context docs even with only 8 documents provided. I did try it with just one document provided, and answers seems less confused.
Ideally I'd like to be able to point to a whole drive and chat with my own currated knowledge base. Is that an inappropriate use of this tool?
Can "recursive similarity" be set as the RAG method?
Would it allow local server which gives REST api? not in tcp ip, but normal android app backend api format @GET, etc..?
The docker image comes with a full REST API. It is accessed via TCP/IP though. You can wrap it though on the Android side with Retrofit or something like that
For ollama on Macbook pro 2016 x86 and 32gb or ram I recommend Solar, Mixtral is good but slow.
How can I get a list 📝 of models that work on OLLAMA?
@@ajarivas72 Lllama3 now is not bad, but still solar and Mixtral are good too.
Got this working. How do i get it to summarise a pdf please?
I used LLama 2 LLM model with anything LLC just shown in this video and use my website to train the LLM model but I am still getting generic answers that are unrelated to my website. how do I fix this?
Ollama now supports embedding models.
And we support those models for embedding now as well!
I installed it on windows and when I try to upload a document the my documents windows comes up but it says document processor unavailable. how can I fix it
AnythingLLM only recognizes the first document I uploaded, but not the 2nd doc and not even see the website link after the 1st upload. I reset, deleted workspace, created a new workspace, restarted my computer, but AnythingLLM still not seeing the 2nd upload. Is it a bug? Please fix. Thanks, Tim!
Are you just downloading the HTML web page for a site and then compiling that as a datasource is is there an agent actually crawling the site looking for the data based on a given user input?
Just downloading the page. No agent support currently
Can you do a review on AirLLM it’s lets you run a 70b model on 4gb of vram, or is there a way to run it on your software?👍👍👍
Can we use Dalle 3?
I tried Anything LLM (using the default suggested tools for embedding, vectors and so on), with LM Studio, tested with multiple LLMs, it works fine, I can chat without trouble, until I decide to upload a document file (tried with a PDF paper from arxiv) or a web url and move it to the workspace, it gives me no error, everything seems fine, but now whatever prompt I use, whether in chat or query mode I get an error in the chat window as a response:
⚠ Could not respond to message.
An error occurred while streaming response. Failed to fetch.
I do not get any error in the LM studio log, so I suppose no data get sent to the endpoint server.
Everything freeze down cause if trying to go in the options for the workspace it doesn't do anything, if I go in the general options I do get an infinite loading circle loop. To fix it I need to reload and remove the document from the workspace.
My System: i7 3770k, 32GB + RTX 3060 12GB. Windows 10.
Note: Since my AnythingLLM doesn't have that thread option in the GUI I suppose I've to update AnythingLLM and check if the error doesn't show up in the new build.
The thread feature if available in the latest version of the desktop app - so you can redownload and just overwrite the application and youll still keep your data.
It looks like you have plenty of power and memory. "Failed to fetch" with LMStudio almost always means that the endpoint being used or the API key (if applicable) is incorrect.
The LMStudio endpoint should be somthing like `localhost:1234/v1` in AnythingLLM. Make sure you started the LMStudio inference server in that application as well since it does not start on boot.
medium.com/@ingridwickstevens/running-a-local-openai-compatible-mixtral-server-with-lm-studio-cd602efbf808
@@TimCarambatI don't think it's LM Studio related at all, cause I used it with multiple tools without trouble, as I said i used localhost:1234/v1 (you can't set the API key in AnythingLLM), but chat work fine in AnythingLLM, but as soon I try to chat or query after embedding anything I get that error (Could not respond to message. An error occurred while streaming response. Failed to fetch.)
Update:
I installed the latest. First tried without deleting anythingllm preference folder and got a fail in trying to save and embed the document suggest trying a different schema than LanceDB. Since the error was different I thought maybe the preferences have compatibility issues, so I cleaned up the preference folder and started from scratch.
And got the original message on the arxiv pdf (but no failure on embedding and saving this time), this partially froze the process as I explained above, I also got multiple wmic.exe and conhost.exe processes, a few times multiple powershell.exe spawning on my system and high CPU usage from process: WmiPrvSE.exe.
So I restarted and tried uploading a new story from the web from a basic webpage (longstoryshort.squarespace_com/great-example) and got an error when attempting to move the doc to the workspace(Save and embed): It fails again and suggest trying a different schema than LanceDB but at least it doesn't freeze (it still spawns multiple processes). So I think it's a problem with the LanceDB wheter it does fail to save and embed or not. I will try installing Chroma and see what happens, will try Ollama as well, I would still check.
Doesn't seem to work with RUclips URLs - I add the URL and ask what the key messages are and I just get a load of nosnsense about something totally different!
Hi, how to see if this LLM is using my GPU as well? I have a nvidia 3060 laptop gpu. Don't have the cuda installed.
The easiest way is have open your control panel and see if the gpu spikes during a chat, that is the easiest way to know
Are there any good programs that give ollama memory of the previous conversations it's had? like a memory recall function and storage
That is built into anythingllm automatically. Outside of that you have to manage it directly in ollama
@@TimCarambat "outside of that" what do you mean by that?
@@ogr3dblade6 I mean outside of just using anythingllm, sorry
Literally 5 minutes later, it stops calling me by my name and it forgets who i am.
@@TimCarambat is the session RAG feature flawed or do you think i messed something up?
Hi there, Thank you for this video.... I just can't get it to work i followed your steps.... I get an error message "pinecone evn settings" or something like that... I don't know how to fix it or where to find the EVN settings.... I would greatly appreciate the help
Go to "Vector database" and select LanceDB. Not sure how you got to that state since lancedb is the default, but that would fix it.
Thank you very much. I really appreciate it. I will try it out tonight. Have a great day.
Best regards
Please release the Linux version!
We have a docker image if that works for you??
Thank you Tim I'll look into that!
Did you have to give ollama 2 admin perms aswell?
already using ollama... and must intergrate it with Anything LLM.
I will say ollama works a bit slow, may be because of the 7B Parameters. It take generally 70-80 seconds to deliver an answer.
Are you on an intel-based processor with no GPU? If so this is par for the course and normal.Bigger model can lead to even longer time-to-first-token or crashes
So my question is is it possible to set up LLM anything to be accessible over a local network?
Yes, absolutely
Is the GUI app for AnythingLLM open-source?
Yes, it's the same repo as the main app, it's just the desktop branch.
github.com/Mintplex-Labs/anything-llm
@@TimCarambat Thank you.
no linux???
Does docker not satisfy that need? Working on an AppImage but the docker image is so much more featured than the desktop client
Awesome,
Army Swiss Knife for LLM
Can I spin up a Streamlit app with this ?
Are you planning on adding support for Nvidia Chat with RTX?
Do they have an api for that tool?
Can’t connect to base url on app my Ollama shows url
I thought I was too stupid to do this but I guess I'm smarter than I thought or this is easier than i thought
I have ollama in docker [OllamaBAseURL: host.docker.internal:11434 ...cannot set the ChatModelSelection...guess it should take the default and only model: llama2] &anythingllm in docker. Get this error when trying to chat...Could not respond to message.
Ollama:streaming - could not stream chat. Error: unknown scheme.
Does anythingLLM support ollama in docker?
What could be be problem?
Ollama in docker does for sure work, some do it so its def a configuration issue between the containers and your host relaying the connection
How cai i use local Ollama in Visual Code please for agents Crewai?
Linux coming soon :)
Linux not there yet :(
AppImage is live for AnythingLLM useanything.com/download
And Ollama supports a linux download now as well. I think its just an install script
😍
@@TimCarambat:D
Can we buy a domain and hosting it as a website from my home computer ?
Sure, the docker version you can host anywhere like render, aws, railway, gcp. Doesn't matter. You can use custom domains there so no reason why not