Unlimited AI Agents running locally with Ollama & AnythingLLM
HTML-код
- Опубликовано: 8 июн 2024
- Hey everyone,
Recently in AnythingLLM Desktop, we merged in AI Agents. AI Agents are basically LLMs that do something instead of just replying. We support both tool-call-enabled models like OpenAI but have even now have a no-code way to bring AI agents to every open-source LLMs like with Ollama or LMStudio.
Now, with no code required, you can take any LLM and get automatic web scraping, web-browsing, chart generation, RAG memory, and summarization all autonomously and running locally.
If the future of AI is agents, AnythingLLM is where it is going to happen!.
Download AnythingLLM: useanything.com/download
Star on Github: github.com/Mintplex-Labs/anyt...
Chapters:
0:00 Introduction to adding agents to Ollama
0:45 What is Ollama?
1:08 What is LLM Quantization?
1:28 What is an AI Agent?
2:54 How to pick the right LLM on Ollama
5:11 Pulling Ollama models and running the server
5:45 Downloading AnythingLLM Desktop
6:17 AnythingLLM - Initial setup
7:21 Sending our first chat - no RAG
8:22 Uploading a document privately
8:43 Sending a chat again but with RAG
9:10 How to add agent capabilities to Ollama
10:45 Add live web-searching to Ollama LLMs (Free)
11:41 Using agents in AnythingLLM demonstration
13:24 Agent document summarization and long-term memory
14:35 Why you should use AnythingLLM x Ollama
15:00 Star on Github, please!
15:06 Thank you 
Наука
@TimCarambat I had to pause the video just to leave a comment! I'm deeply impressed by the excellence and simplicity of the content presented here. It's truly remarkable to have access to such tools, created by a team that clearly demonstrates passion and a keen ear for what we all think and wish would be great to have, and at every update, distilling all p of these wishes into a few simple clicks within this amazing piece of technology! I'm immensely grateful for the opportunity to experienceh the brilliance of software engineering and development of Anything LLM, especially within the context of open-source communities. Participating in the advancement of genuine and incredible open tools is a privilege. Thank you Tim! I will be promoting this project to the moon and back, because this deserves to be known.
This is so incredibly kind. Sharing with team!
haha I was just about to leave a comment when I read yours. I feel the same. What a champion Tim is. I do not know if I will ever install AnythingLLM but I think I will donate to Tim regardless.
Aye, I was interested in anythingLLM a while back but chose another project for my inference server. I've found getting half decent agent capabilities to be a huge time sink for someone with my skill set (I'm a physical security guy, not a programmer) and the results just weren't worth the time invested.
Even basic agent capabilities with RAG, memory and so on in a package that I can just plug into ollama sounds awesome.
Prepping the server now. Here's hoping.
Been using for the past few months and is my go-to app for local RAG. Adding agents huge plus. Looking forward to being able to add my own AutoGen agents to the list with their own special tools. Thanks for the great work Tim.
You built an amazing piece of software. Thank god that I stumbled across this video.
Yo honestly it feels great when guys like you make your software completely free and i also think you should keep a option of donation. after seeing guys like you i will make something great and i will make it completely free to use and open source. again thanks dude!❤.
Thank you for explaining quantisation in details for niebiews.
What I love about your tutorials is that you succinctly explain all the things that come across during the tutorial. Thanks!
Im a chronic video skipper but watched this back to back. Great explanations and can't wait to try this out! Would love to see more videos, tutorials or even lectures from you. You really have a knack for explaining things!😊
PS I've starred on Github!
Tim, thank you for making the world a better place with this awesome tool! :)
This is by far the easiest and most powerful way to use LLMs locally, full support, like and sub. And many thanks for the amazing work, especially being open source.
🫡
Great this will make LLM more understandable for many ppl.
Big thanks man. This video helps alot for me as an beginner to understand how good a local llm is and which Usecases we have. Thumbs up for this great video.
Thank you! Will test it for sure. I think you guys are on the exact right path 😎👍
Amazing Tim. Keep up the good work.
This is the easiest all-in-one platform. Thanks. More videos please ❤
I had no idea you had a channel talking about your software. Im a big fan of your work!
This is so dope. Great no-code solution and it's awesome that it's open source.
Incredible.. gonna make building tools so much easier. Cant wait to see more agent abilities added!
Awesome vid! Really impressed with how you presented the information. 🙏 thank you
@TimCarambat
I'm excited to see the features you talked about work with the ollama like in the video for the agent, as of now, its same as before I updated, but it's exciting to think of the future.
Fantastic Tim! Mine doesnt have agent config, guess i need to delete and udate, ill try that, looks great! keep up good work, i love anythingllm i really do!
Damm,... finally found the tools that i been looking for..MAN you save my day, i have been crazy stuck finding webui for my ollama remote server..your a gift from heaven keep it up your helping alot of people like us..thank you so much..❤❤❤😂😅😊😊
Good stuff, will try it out. Subscribed. Looking forwards to seeing how this develops.
Anythingllm is awesome. Glad to hear custom agents are on the roadmap. It's the big hole in capability. Also need config to change agent promt. I scan a lot of code and the @ is used often to define decorators.
This is awesome work. I looked at the other simple to install Windows front ends and stumbled on this. Pretty cool stuff and I love how you can add documents and external websites to feed it information. An offline LLM is soooooo much more preferred. The only item I don't understand is why you could just ask a regular question once you provided the document, but used @agent when asking to summarize a document.
IMO, i find having a local LLM that even is **only** like 75% as good as on online alternative is just much more rewarding.
Like i can be on an airplane, open my laptop, and start brainstorming with an AI. Pretty neat.
Next evolution would be a local AI on your phone but i dont think we have that tech _yet_
Can't wait to use this. Thank you!
You are amazing. Thank you 🎉
Wow this is amazing... I'm gonna go star you!
Great work. Thanks a lot 🙏
🎯 Key Takeaways for quick navigation:
00:00 *🤖 Introduction to Ollama & AnythingLLM and AMA*
- Introduction to Ollama and AnythingLLM
- Explanation of AMA application for running LLMs on local devices
- Overview of quantization process and agent capabilities in LLMs
02:30 *🧠 Understanding Model Quantization and Selection*
- Importance of selecting the right quantization level for LLMs
- Differences between various quantization levels like Q1 and Q8
- How quantization impacts model performance and reliability
06:07 *🛠 Setting up AnythingLM with Q8 Model and AMA*
- Instructions for setting up AMA with Q8 LLW model
- Steps to download and run AnythingLM on local devices
- Connecting to AMA server and configuring privacy settings
08:27 *💬 Enhancing Model Knowledge Using RAG and Workspace*
- Uploading documents for model referencing in workspace
- Improving model responses by utilizing documents in the workspace
- Configuring workspace settings for better model performance
11:41 *🌐 Using Agents for Advanced Functionality in AnythingLLM*
- Utilizing agents to enhance LLMs capabilities beyond basic text responses
- Enabling web scraping, file generation, summarization, and memory functions
- Integrating external services like Google for web browsing functionalities
Made with HARPA AI
Awesome! thank you for this. looking forward to more information/details/examples on using agents w/AnythingLLM!
Amazing work 👏
Can it also use dalle but unrestricted?
Funny ! I heard yesterday for the first time about AnythingLLM during an AI-info event.... and discarded the idea of giving it more attention because it was presented as "just another local RAG support". And now I stumble across this video by chance - and the additional agent functionality changes everything ! BTW, very well presented , this feature !
My immediate idea & feedback: if there was ANY chance to model custom agents in Flowise and re-import the JSON exports of this Flowise flow as input for an AnythingLLM custom agent, you'd save yourself the trouble of designing your own agent editor AND would start with a comparably large installed base. OK, maybe that's just wishful thinking..... but maybe I'm also not the only one with this wish to facilitate local agent building ;-)
looking forward to utilizing AnythingLLM, it looks really awesome! congrats on creating such an impressive application! thank you!
Great job! This is wonderful! I will be responding after using to let you know my thoughts if you care to see them :)
very cool!!! subbed!
This explained the rag and agents parts I couldn't set up. Great educational content for those who are not programmers. Appreciate your explanations being without that much of "pre-supposed" know-how, that coders have - which is most tutorials on youtube...
I still didn't get why there's a difference between @agent commands and just regular chat
In a perfect world, they are the same. AnythingLLM originally was only rag. In the near future @agent won't be needed and agent commands will work seamlessly in the chat.
So @agent is temporary for now so you know for sure you want to possibly use some kind of tool for your prompt. Otherwise, it's just simple rag
A fantastic beginning! When do you think we willbe able to create our own agents?
Debug mode would be ideal. Agent to scrape the web just exits without any error even though I do have search engine api defined
So training/finetuning is coming up as well? Loving the progress and process updates, keep up the great work Tim!
how'd you know!?
We will likely make some kind of external supplemental process for fine-tuning, but at least make the tuning process easy to integrate with AnythingLLM.
RAG + Fine-tune + agents = very powerful without question
@@TimCarambat That's awesome to hear!! I created an agent to get insider info, that's how I know of course!
@@FlynnTheRedhead !!!!! I thought i was hearing clicks during my phone calls!!!
Hi Tim, thank you very much for the great video showcasing open source llms, and tools like anythingllm to create agents. I followed your video and successfully was able to do everything in your video. Are there other agentic videos for other usecases you made, look forward to see them. Cheers
Great Work :)
There were some concepts I didn't quite understand; for example, tunneling from the Windows PC to the Mac (if it's on your local network, why work with VPN protocols rather than client/server - due to needing a stateful connection vs. 200 response code or something?). But the interface itself is brilliant! And I think that when it becomes agent-swarm-capable it's going to be a much better option for me than Crew AI, as it feels more intuitive, I am just going to need multiple agents working together. I have never installed a local LLM, but you have inspired me to give it a try. Thanks!
Would love to see this run stable diffusion and comfy ui workflows
Great work Tim! If using 'AnythingLLM' in the 'LLM Provider' section, can I load other LLMs that are not listed? Like the '8b-instruct-q8_0' you mention? So I don't have to rum Ollama separately to load a model?
Great video! I also want to know how well the RAG function of AnythingLLM performs. It's important that text, images, and papers are handled properly and meaningful chunking are achieved
Wow, this looks *_amazing!_*
I’m just starting to experiment with local LLMs and wanting to play with agents; this looks SO easy! I’m going to download and set it up right away.
I’m also interested in Open Interpreter for having an AI assistant do things on my local machine. Can this interface with that, or is it really meant as a substitute/enhancement to it?
(Also, how can I support your project? I gather your biz model is selling the cloud service, but my usage will be purely local. Anywhere I could send a token few bucks?)
Really awesome demonstration. I am excited about agents. Would be nice to be able to build custom tools in python for agents to use.
Awesome info……thx
This is incredible..
really nice!
I've been using it for months. Love it! Will you enable voice to voice soon?
We just did in our most recent update. TTS is live for all, STT is only live for the docker version. There are some restrictions and limitations we need to work around to get STT to fully function cross-platform. It will be solved soon
this is seriously very neat tool👏👏👏 Pls add some feature to custom develop agents with function calls. It will be helpful for our local automations.
This is shown in the UI that we will be supporting custom agents soon!
🔥 🔥 🔥 🔥 🔥 🔥 🔥 🔥 🔥 🔥 🔥 🔥 🔥 🔥
Phenomenal
Been struggling to get custom agents to integrate reliably with external tooling, using frameworks like crewui with local LLMs. Would love a video guide explaining best practices for this
Awesome tutorial Tim! Can this extract specific data from PDF files and save it to an Excel file?
Literally sitting here wondering this when you dropped the video
Amazing ⭐⭐⭐⭐⭐
@sergiofigueiredo1987, @TimCarambat, I agree with Sergio. Wow! I have Ollama installed locally on a Windows machine in WSL. (I was leery of the Windows preview, but I may switch because NATing the Docker container is a pain.) I also pondered how to build a vector DB on my machine and integrate agents. You guys have already done it!
Your the best
Great work - thanks for sharing - Question - how can we send data to the agent via webhooks - is this a possibility?
Hi Tim, thank you for your dedication and effort in teaching us about local LLMs. I have a medium-spec computer with 4GB VRAM and 16GB RAM. The last time I installed ALLM, the inference speed was a bit slower compared to other alternatives. How does it perform with the new version? Thanks again.
Unfortunately, i doubt much would change on the inference side. When you say alternatives, what were you using? You might get slower responses in AnythingLLM vs just chatting via CLI in ollama, but that is because we are adding that valuable context to the prompt. More tokens = more work on the LLM to respond!
What agents do you have planned for future?
Agent Smith.
thanks a lot for your video, this the first video that i see and it's really simple to understand, i have a question, if for example the model that i use know that the capital of france is paris, how can i change that information and make the answer different from paris? best regards
Amazing software!! Congratulations and thank you! Very similar to MemGPT server but seems easier to set up and use. I wonder whether you can save a whole company database (i.e. ERP data: products, materials etc.) in it and being g able to ask questions about it? Also can you instigate more than one agent simultaneously?
In theory, this would be better delegated by some purpose-built agent that can traverse the data. Currently, we only have one-agent conversations but the code _does_ support multi-agent. We just find it to be really messy and cumbersome when many agents are once are trying to do something and your Ollama instance is already at max use generating tokens!
Great video! however, i followed every step you described in every detail, but i could not make the agents communicate with outside world. in any ''search'' or 'webscrape'' request, the model is hallusinating, and presents data that are already to its knowledge insted of real time data (i.e. current gold price ). i used llama3 Q8, i inserted google api and id code, i also tried the other search engine.. nothing. the logs show that it really creates json commands, but nothing comes in from the internet.... any help ?
Does Internet Search Work for You ?
I configured the agent to use Google Custom Search Engine but search does not return any results.
With some models you _might_ have to word a prompt more directly. Like even explicitly asking it to call `web-browsing` and run this search. Which i know breaks the "fluidity" of conversation, but this is just a facet of the non-determinisic non-steerable nature of LLMs and trying to get them to listen.
Mostly, its the model that needs to be better so it can follow prompts more closely, but its also not always that simple!
Hey, thank you for putting out anytingllm. I have two questions: 1. When I ask the llm to remember something, is that long term memory stored on my pc on a server? 2. is the summary part of the long term memory necessary? And I have a feature request for an automatic long term memory. Meaning that I don't have to say specifically what to remember but that the llm will be able to recall the entire chat history - eveything i have ever said in that thread. Is that in the picture?
Looks really good and simple. I tried PrivateGPT using conda/Poetry and could never get it to work, so jumped into WSL for Windows connecting to Ubuntu running ollama, via WEBUI. Works great, but this just looks so much easier. Will have to give it a try. What I do like with the WEBUI I have is I can select different model, and even use multiple models at the same time.
Yeah, we didnt want to "rebuild" what is already built and amazing like text-web-gen. No reason why we cant wrap around your existing efforts on those tools and just elevate that experience with additional tools like RAG, agents, etc
I'm so happy I came across your video. Thank you. I am having trouble on where you to get the base link that you pasted in @6:36 mark to install the ollama3
I was able to follow along from your other video to install it. Thank you I'm now a new sub.
This is perfect it just needs more tools and agent customization like crew ai and it is going to be an absolute killer for the ai industry.
Will be coming soon! Just carving out how agents should work within the context of AnythingLLM and should be good.
Also, it would be nice to be able to just import your current CrewAI and use it in AnythingLLM - save you the work you have done so far
Great things shown.
Tx for all the work and commitment.
🎉 Here is a kind of dedicated use case I am interested to get acces:
I am a mind mapping addict. I use Mind Manager, that stores the mm in .mmap format.
I would like to ask ANYTHINGLLM to help me scan all folders for mind maps on different subjects and Rag & summarize on them, without having to export all mmap files in another format. Is this doable at this stage? What else should have or have created?
I think it's a very good application, easy to use, and after testing it for a day or so, I have some wishes.
1. direct commands Bypass Agent LLM in Agent mode. It takes time for the agent to understand the sentence and convert it into internal command, and url parsing sometimes fails depending on the agent. For example, a command that scrapes the specified URL and shows the result, or a command that lists the currently registered documents with numbering. And a command that summarizes the document by this number instead of its full name.
2. I wish there was a way to pre-test the settings in the options window to make sure they are correct, such as specifying LLM or search engine.
I hope this application is widely known and loved by many people.
@TimCarambat
Hey Tim it wont let me select anything under Workspace Agent LLM Provider even though everything is setup and working, obviously ollama is running and everything else in anything is using ollama fine in the app, but this selection option doesn't show like yours does.
Hi,
How does quantization type affects the system resources needed to properly run that model?
Great video by the way!
It mostly impacts the RAM and overall storage side of the GGUF modelfile. It's tricky to determine the exact requirement decrease because it has to do with the specific model parameters and other factors. Im not aware of a simple equation or expression that is a direct calculation for all models.
In general, lower quant -> Smaller file size and memory footprint when loaded, but much worse output performance
Brilliant! Could it use my own computer as a data source for the agents? Can I scrape my mac?
Many thanks for nice introduction !
Is there a way to configure this LanceDB ? Is there a doc how it's integrated with the AnythingLLM ?
There is nothing to configure, it is preinstalled and saves to the same location as the application's main storage folder!
Hi Tim.. I am absolutely impressed with the capabilities of AnythingLLM. Just a small query..how can I deploy it on a cloud machine and serve it as a chat agent on my website?
I actually want to add few learning resources as pdf for the rag document of this llm so that my users can chat with the content of those pdfs on my website.
I also want to understand how many such parallel instances of similar scenario but with different set of pdf is possible? For instance, if I am selling ebooks as digital product to my users, can I have unique instances autogenerated for each user based on their purchase?
We offer a standalone docker image that is a multi-user version of the desktop app. It has a public chat embed that is basically a publicly accessible workspace chat window. You can deploy a lot of places depending on what you want to accomplish: github.com/Mintplex-Labs/anything-llm?tab=readme-ov-file#-self-hosting
For this, you could do one AnythingLLM instance, multiple workspace where each has its own set of documents, and then a chat widget for each. This would give you the end result you are looking for
Hey, just found you on a random youtube video suggestion. Love this concept.. A few questions, how deep into a website can this scrape? Can it read a sitemap or robots.txt and download all the data, summarize, etc? Can I hook it into different LLMs? For instance, assign agents to different LLMs? Most importantly, if we're using a vector database, can I feed it rows and rows of data to remember forever?
The one in the document uploader is a single site, but we have a deep website scraper as you mentioned.
You can use a different LLM per workspace and also per workspace-agent. So yes.
The vector database we use runs locally and is built in. It works like any other and yes does persist information - so yes to the last point as well
Great!!!...
wOOHOO I GOT IT NOW! ID LOVE A UPDATE BUTTON LOL!
It probably just was not refreshed yet. I think we have it on a 1 hour expiration to check so it may have been in between checks
Hi, Tim. Great work. I'm trying to use Google. But... it is free for 100 queries per day. How I can control it or limit it? Other options are payable :)
@Tim: I am a professional translator (English to French), and I've just discovered AnythingLLM. Sometimes I have to translate confidential documents that cannot be shared on the cloud. They need to remain locally on my own computer. Once the translation is done, they have to be encrypted to be sent to clients.
Could I use AnythingLLM to help me with the translation process?
Could I use it with my actual Lexicum, glossaries and personal dictionaries? Most are PDF or DOCX files.
How would I do that? What are the first steps?
Many thanks if you can give me some hints on how to proceed.
I'm now a new subscriber! 😊
Thank you so much the only problem i have is i cant seem to find anything llm github pdf file. Where do i download it from?
@Tim do you know anything about home assistant, home automation application. Reason i ask is they already have some intergration with LLM but not with agents and not specialized for home assistant auto automations. When you have time check it out and see if its possible to integrate this with home assistant that would be great. Great job with the video!
Would it be possible to see a vide of setting up your Ollama models on Anything LLM, I followed these instructions but my ollama models never load.
Will not load models in the linux version when I select local Ollama
What is the chunking method you use to create the vectors?
Is there a way that the user can control the method of chunking?
Say : Short, Token Size, Semantic, Long etc...
We currently use a static recursive chunk splitter. So basically just character counts. You can modify those chunking settings in the settings when you go to "embedder preference". So you can define max length and overlap
Hello. I was just playing with RAG. It seams that the acuracy and results are very poor. I tried with laama 3, wizardlm etc. LLM is unclear of my questions. Is the context windows too short? LLM is giving answeres in a hindsight
Just went through this whole setup and for some odd reason It keeps telling me it can't search the internet. I've tried local LLMs and OpenAI API with GPT-4o. Also have both Google Search API and Serper API. Neither seems to be able to 'reach' the internet.
What in the heck am I missing? I understand this stuff pretty well and I just can't get it to search the web.
i tried anything LLM and RAG sort of works but I can never pull anything factually from my uploaded text files which are configuration files.
Is this a model issue? I was using Llama3 Q8 via ollama and llm studio.
Can I edit my question and regenerate the result in AnythingLLM? I use OpenAI GPT-4o api, but I don't find the edit button in AnythingLLM UI.
guessing things have changed but I cannot seem to find where my programmatic access api key would be
Hello! Thanks for uploading this...very helpful. I'm stuck on something though. When I try to setup agents for Ollama, it says that agents only work with OpenAI currently. When I try to scrape sites like you do in the video using Ollama, the AI tells me that it can't. Am I missing something?
I have the same issue, but i am using lmstrudio as backend
You are able to use Ollama as you agent correct? If that is the case, are you using a small quantized model? Sometimes models have issues calling tools when they were built for that. Our system we implement works well, but we dont "force" the model to call a tool, it still has to generate a valid response to call it.
@TimCarambat sir kindly have a for loop so that multiple agents can talk to each other in a chatroom style conversation
Today when I tried to start up AnythingLLM I'm just getting a spinning circle on the main startup screen. I've left it sit for 30 minutes and it was still doing it. Tired rebooting but no luck, even uninstalled and reinstalled but still does it. Any ideas?
Hey Tim,what is actual folder that Anything LLM use to store models?
I have all models downloaded using it on other apps so i would rather just put the model in the right folder then download it again.
Thanks in Advance!
on Mac: /Library/Application Support/anythingllm-desktop/storage/models
On window: /Users/user/AppData/Roaming/anythingllm-desktop/storage/models
@@TimCarambat thanks!
can you upload a video showing how to embed a chat widget onto a website using the llm ran locally on ollama?
Also, will there be a text to speech and speech to text option?
It is a pending issue at this time, yes