Thank you for your hard work! This is really a game changer,now people can build they personal chatbots,with massive databases and knowledge about they favorite topics,without using or paying online services. This is my new favorite piece of software together with LM studio. Huge respect and keep up with good work.
Docker image is a WSL2...Linux port. Its not Windows, you're just using a third party app to run it in windows at the cost of resources to run the docker server. (which ive been doing for a long time, duh...) I mean its out on windows which makes development SIGNIFICANTLY easier. ;)@@geekazoid47
lol yeah I just saw the release right after this video posted. Good call. I currently have my GPU passed through to my windows VM for different compute things. Glad to see ollama on windows. I’ll definitely try this out.
@@GregRutkowski I got an AppImage live on the main site now. Tested and built on Ubuntu 22, not sure how compatible that build system is with other distros, but I dont have inifinite linux machines (or time!) useanything.com/download
Very cool. Thanks You! This open-source, ethically crafted and versatile application makes ollama RAG and more easy. Clear and transparent Linux install and update instructions too. Well done.
I love AI, and I'm dabbling in all kinds of things, but I could never get LLMs to really work on my local machine. THIS is amazing. I got it working on my gaming laptop.
what about agents? do you plan to integrate agents? with crewai, or autpgpt, also why only ollama, you can integrate llm studio as well! sounds like a promising project.
Excellent work🎉 , Offline- RAG-Opensource!!! Very useful, working well in my pc. at 5:46 when I type & select ollama, base url is detected automatically 👍 thank you for this master piece Anything LLM👏👏
Hi, have to thank you for the tutorial. It is very useful to quick test different LLM models, vector databases and other points with different data content. And just after that think about what stack is suitable to index your own data and try to deploy it on a server-side. Thanks!
I get "-- loading available models --" in the Chat Model Selection combobox after selecting Ollama Base URL (that is correct btw) because I can open ollama one ui. What am I missing here?
would you consider adding more plaintext files, such as .py files? Just installed the latest version in docker and its great, love everything about this!
You know it's funny, we allow pulling in and embedding of whole github repos, but not the independent code files. I will work on resolving that for you - thanks for pointing it out 😄
Good stuff. Are there any ways to connect to APIs (similar to how chatGPT works) where we could use large data sets for free? Adding individual websites and files could take a lifetime...any workarounds?
@@TimCarambat that sound amazing. I will try that in the next days. Today I‘ve installed privateGPT and that is quite cool, but very limited in inputs. Yours looks much more flexible with all the possible inputs like links to websites and so on. I have thousands of pdf documents (scanned my file cabinet. That took months). Is it possible to load them all in and then kind of talk to my entire file cabinet?
Awesome thank you ! The possibility to learn by giving it docs/URL is so cool. I have one question though: Is it possible to use AnythingLLM with an API ? So we can use a model (Llama 3), add data thanks to AnythingLLM, then use an API to talk to AnythingLLM that returns response ? Thank you in advance for the answer, I guess many people wonder the same ^^.
Thanks a lot. I have 2 questions. Does it have access to the internet to fetch recent updated information? What's the best model to use for software engineers?
You can scrape websites with the built in scraper but its not "internet connected". Not like an agent-LLM. CodeLlama for local LLM. Claude-3 Opus for closed source
If on windows you can run LMStudio (just like ollama but with a UI) and LocalAI (requires docker to run). Outside of that you can also turn to hosted LLMs on OpenAI, Azure OpenAI, Anthropic, Google Gemini Together AI, and Mistral
We are a one-click single Desktop app you can install, no code or python or docker config We support many LLM, embedders, and vectordatabases where the defaults are all local We come with built-in data connectors like Github, website scraping, bulk website scraping, Confluence, etc etc We also have agent capabilities for _all_ models, even if that model does not support tool calling. We also support multi-user on our Dockerized version of the app (if you need that) - that is just off the things i do know about PrivateGPT. Nothing wrong with PrivateGPT at all, i think our tool is much more "accessible" and more all-in-one. Different strokes for different folks
I'm sorry, I'm aware that it may be an incredibly stupid question, but If my LLM runs on the virtual Linux under Windows, then do i need the Linux or the Windows version of AnythingLLM?:) I'm sorry again about it.
already using ollama... and must intergrate it with Anything LLM. I will say ollama works a bit slow, may be because of the 7B Parameters. It take generally 70-80 seconds to deliver an answer.
Are you on an intel-based processor with no GPU? If so this is par for the course and normal.Bigger model can lead to even longer time-to-first-token or crashes
Ok, the Claude API works very well. I have a question: how can I fetch an entire website, not just a single page link? If I want to load all pages of a website, how can I do that?
2 months have passed since the original video, 2 questions. I am running a 12 GB vram RTX3060 GPU. Will LM Studio be a better solution than Ollama? 2. Now Llama 3 has come out, will thos work with what is presumably a much more capable LLM?
Bro this is amazing! Currently training models seems like a big hastle most training is done via google collab and such any plans on adding an easy UI interface to train models locally inside AnythingLLM?
Being able to fine tune off the chat and such is on the roadmap, but is still conceptual since 99% of people cannot train a model on their device. So we may integrate or connected to some provider which can enable this to happen more easily. We allow you to export the chats in a fine-tune format already to encourage this behavior and id love to close the loop on it
Many thanks for introduction the AnythingLLM. Is there any date mentioned when this app will be release for Ubuntu. I have to admit it blocks me 😀but there are so cool features to automate our work. Or maybe is there any similar app to integrate Ollama with vector DB, embeddings, etc in similar way ?
It seems to be having an issue reading the entirety of my Docx, only 103KB. It's the CBA for my workplace. Once embedded it states that there is no context for the question I've asked, though there most certainly is. (IE, shift schedule times, etc...), it will then proceed to read OTHER parts of the document. I wonder if there are certain document types that work better than Docx? or if anyone has any suggestions?
When you scape a URL does it extend the scrape to every page on the site? Or would I have to add all the pages of a site manually? How deep is the URL scrape?
The one in the uploader is currently just the url. Understandably that is not great for large url ingestion and there is a piece of work being done to add deep scraping (or at least multi link, possibly agent based) to the tool.
Can I use AnythingLLM locally and privately only as an API like the OpenAI API? In addition, do you offer anything like the Assistants API, where you can have models that have pre-defined instructions on how to respond?
This is awesome! I'm so glad I found this video cause being able to talk to my pdfs was something i've been wanting to do for a while! I do have some questions though. I have pdfs of tabletop RPG books and when I ingest a certain book and begin asking it rules questions it seems to 'reach out' to other sources and rules that aren't in the book and the responses are incorrect. Is there a way to narrow the chat to only answer rules and such about that particular game or book instead of adding knowledge on other games it has?
Ah, this is an error class i just refer to as "general knowledge creep". The only semi-good remedy is just a really strong system prompt explicitly detailing that only knowledge in the context should be used and to not use it's existing understanding of a game on how rules for this game should work - or something along those lines. System prompt can def fix this and depending on the model will vary in performance.
I have a M2 Max Macbook Pro. Ollama with 7B Llama works super fast when used in Terminal. Why is it slow when I am using AnythingLLM along with it. There is a drastic time difference! Pls advise.
I love the ease of set up and the local private models. First attempts at use are not good in terms of models ability to understand and formulate answers to questions based on provided context docs even with only 8 documents provided. I did try it with just one document provided, and answers seems less confused. Ideally I'd like to be able to point to a whole drive and chat with my own currated knowledge base. Is that an inappropriate use of this tool?
Thanks Tim for this great stuff ! I am interested in generating a document based on data within multiple local files. Do you think it's easily feasible with ollama/AnythingLLM?
This is great in, do you offer any support? I just uploaded a simple document and tried even a web site but when I ask any question I get the following error "couldn't respond to message , fetch failed!"
I can tell you alreayd that the issue is you probably put in the wrong Ollama connection info. Search in the Github issues and youll see we have helped dozens of people solve this issue. Its a common gotcha github.com/Mintplex-Labs/anything-llm/issues?q=is%3Aissue+fetch+failed+is%3Aclosed
I have tried this out a bit now and I really like it. However, I would like to import lots of dokuments and not via a GUI. Is it possible somehow to use python to connect to anythingLLM to post text files via an API or something like that? It would be fun to pump in huge amounts of text files and ask stuff.
The docker image comes with a full REST API. It is accessed via TCP/IP though. You can wrap it though on the Android side with Retrofit or something like that
Will any LLM be able to run its own server? I find that I get the best performance using it with LM Studio, but I’d like to connect it to OpenWebUI since it looks nicer.
@Tim This is GREAT STUFF. Your putting in some serious effort and it shows. Can I use this to setup on a client site? I am a software developer and would like to create embeddings specific to ai python projects and documentation... what are a couple options i should consider. Thanks
You absolutely could but youll want to run our Docker offering because it is meant for that. The desktop app is really a local "single-player" solution. The docker app has a full API, multi-user support, etc etc
Hello, Tim, how are you? Thanks for the video! I have a question, I have Ollama on a ex mining rig with 5 GPUs, the response time is very good, but when I ask 2 questions in parallel it does not answer in parallel, he answers the first question and the second question remains in “queue” until I finish answering the first question, could you help me? Thank you
Are you just downloading the HTML web page for a site and then compiling that as a datasource is is there an agent actually crawling the site looking for the data based on a given user input?
i am confused on the step where u copy the url from ollama to anything Llm. I am at the same machine and can't figure out what is the url. I tried the direction that appear when using ollama serve but doesn't work.
AnythingLLM only recognizes the first document I uploaded, but not the 2nd doc and not even see the website link after the 1st upload. I reset, deleted workspace, created a new workspace, restarted my computer, but AnythingLLM still not seeing the 2nd upload. Is it a bug? Please fix. Thanks, Tim!
I tried Anything LLM (using the default suggested tools for embedding, vectors and so on), with LM Studio, tested with multiple LLMs, it works fine, I can chat without trouble, until I decide to upload a document file (tried with a PDF paper from arxiv) or a web url and move it to the workspace, it gives me no error, everything seems fine, but now whatever prompt I use, whether in chat or query mode I get an error in the chat window as a response: ⚠ Could not respond to message. An error occurred while streaming response. Failed to fetch. I do not get any error in the LM studio log, so I suppose no data get sent to the endpoint server. Everything freeze down cause if trying to go in the options for the workspace it doesn't do anything, if I go in the general options I do get an infinite loading circle loop. To fix it I need to reload and remove the document from the workspace. My System: i7 3770k, 32GB + RTX 3060 12GB. Windows 10. Note: Since my AnythingLLM doesn't have that thread option in the GUI I suppose I've to update AnythingLLM and check if the error doesn't show up in the new build.
The thread feature if available in the latest version of the desktop app - so you can redownload and just overwrite the application and youll still keep your data. It looks like you have plenty of power and memory. "Failed to fetch" with LMStudio almost always means that the endpoint being used or the API key (if applicable) is incorrect. The LMStudio endpoint should be somthing like `localhost:1234/v1` in AnythingLLM. Make sure you started the LMStudio inference server in that application as well since it does not start on boot. medium.com/@ingridwickstevens/running-a-local-openai-compatible-mixtral-server-with-lm-studio-cd602efbf808
@@TimCarambatI don't think it's LM Studio related at all, cause I used it with multiple tools without trouble, as I said i used localhost:1234/v1 (you can't set the API key in AnythingLLM), but chat work fine in AnythingLLM, but as soon I try to chat or query after embedding anything I get that error (Could not respond to message. An error occurred while streaming response. Failed to fetch.) Update: I installed the latest. First tried without deleting anythingllm preference folder and got a fail in trying to save and embed the document suggest trying a different schema than LanceDB. Since the error was different I thought maybe the preferences have compatibility issues, so I cleaned up the preference folder and started from scratch. And got the original message on the arxiv pdf (but no failure on embedding and saving this time), this partially froze the process as I explained above, I also got multiple wmic.exe and conhost.exe processes, a few times multiple powershell.exe spawning on my system and high CPU usage from process: WmiPrvSE.exe. So I restarted and tried uploading a new story from the web from a basic webpage (longstoryshort.squarespace_com/great-example) and got an error when attempting to move the doc to the workspace(Save and embed): It fails again and suggest trying a different schema than LanceDB but at least it doesn't freeze (it still spawns multiple processes). So I think it's a problem with the LanceDB wheter it does fail to save and embed or not. I will try installing Chroma and see what happens, will try Ollama as well, I would still check.
Thanks for the video, but I'm not sure why, but my windows version of anythingllm is slightly different and I can only follow so far before what's on my screen is different than yours. Is there a video for the windows version? Please make sure that you explain the differences.
@@TimCarambat You have a pop-up window that has a spot for a URL. There is no pop-up window. I'm trying to figure out how to replicate that. When I try to web-scrape with the agent, it says it can't access the web.
So I downloaded Ollama on my Mac and all, but when I got everything LLM, it promoted me to download Ollama too so it runs on it rather than having to run both at the same time. What's the difference?
@@TimCarambat Hi, I checked the task manager, and it seems that the LLM is mostly using my regular CPU memory and not my GPU. I tried it to use my GPU, but I am not able to work it out. Any idea how i can force it use my GPU? Is there some setting in windows which can force the LLM to use GPU, or do I need to do this manually? Info: I am not very good at manual stuff, but I can follow the basic instructions. Thanks for the help, and have a good day!
I used LLama 2 LLM model with anything LLC just shown in this video and use my website to train the LLM model but I am still getting generic answers that are unrelated to my website. how do I fix this?
Yes, We have an openAI compatible endpoint so you can just use that. You can find the exact endpoint by going to the Developer API page in AnythingLLM and opening the documentation. OpenAI Compatible endpoints are at the bottom
Hey Tim, thanks. please share more light, how to serve llmanything chatbot on a website. secondly, does llmanything has api ?? can it be used with devika?
Hi, I was excited to have this new powerful tool, and a convenient way to add a wide range of inputs and contexts into a LLM. I am running llama2:70b with Ollama on a maxed out MacBook. While not really an AnyLLM issue, I am disappointed that when I add a bunch of documents to the workspace and thread, Llama2 is terrible about using the info uploaded. I can upload a doc and immediately ask a question about the uploaded doc and it gets it wrong. I give it hints and correct it, and it still gets it wrong again and again. So one of the most important uses cases for me is a flop, as I had hoped to upload a series of papers and then ask relevant questions, combining the global knowledge with the local uploaded knowledge.
Yes, it hardly appears to use any added material at all and instead pulls about 50% from prior knowledge as well as halucinating the rest. Tested it on a couple of articles. One article about a historical figure who died (peacefully) in 1958 and it has him tragically dying of scurvy as a very young man in 1913. His actual death date is given in the heading of the article. Another one randomly picked from today's paper was so fresh that it didn't have any prior knowledge besides the given news item. Not being able to use the inputted text it just invented a story. Can't imagine the scandals that would follow using this for anything involving actual paperwork at my workplace.
If you see this error, add your local ollama again and this time save it. The save button was positioned on the far right and I never noticed it. This is an irritating af trend caused by the pressure to make all apps "mobile friendly."
Thank you for your hard work!
This is really a game changer,now people can build they personal chatbots,with massive databases and knowledge about they favorite topics,without using or paying online services.
This is my new favorite piece of software together with LM studio.
Huge respect and keep up with good work.
Thank you for the info.
Is it possible to fine-tune an LLM through Ollama/AnythingLLM?
As well, is it possible to build a multiuser server locally?
Ollama is now officially on windowssss!!!!
.. Docker... It always was
Docker image is a WSL2...Linux port.
Its not Windows, you're just using a third party app to run it in windows at the cost of resources to run the docker server. (which ive been doing for a long time, duh...)
I mean its out on windows which makes development SIGNIFICANTLY easier. ;)@@geekazoid47
lol yeah I just saw the release right after this video posted. Good call. I currently have my GPU passed through to my windows VM for different compute things. Glad to see ollama on windows. I’ll definitely try this out.
@@geekazoid47 pain in the ass to setup for someone with no knowledge on setting up docker
@@3ace846 wdym docker is the easiest thing much better than regular commandline installing of things
Amazing insight! I was already using Ollama, and adding AnythingLLM is the icing on the cake 👍. Thanks for the video!
What do you think about this model , what’s your experience?
Great stuff, will be putting it on my windows machine...can't wait till a linux/Ubutu Appimage is released.
Working on it - just having trouble getting a stable Ubuntu (with UI) running so i can fully test the end-to-end flow
@@TimCarambatTake your time, better to get it right then having to patch-it.
@@GregRutkowski I got an AppImage live on the main site now. Tested and built on Ubuntu 22, not sure how compatible that build system is with other distros, but I dont have inifinite linux machines (or time!)
useanything.com/download
@@TimCarambat Time to test it out...
This is great, and nice to see Linux support is there now as well.
Very cool. Thanks You! This open-source, ethically crafted and versatile application makes ollama RAG and more easy. Clear and transparent Linux install and update instructions too. Well done.
The only video I found that gave me exactly what I was looking for...THANKS!
Perfect video! It has changed a bit though, so you don't need to pick lance etc, it just shows it as chosen.
I love AI, and I'm dabbling in all kinds of things, but I could never get LLMs to really work on my local machine. THIS is amazing. I got it working on my gaming laptop.
what about agents? do you plan to integrate agents? with crewai, or autpgpt, also why only ollama, you can integrate llm studio as well! sounds like a promising project.
Excellent work🎉 , Offline- RAG-Opensource!!! Very useful, working well in my pc. at 5:46 when I type & select ollama, base url is detected automatically 👍 thank you for this master piece Anything LLM👏👏
Hi, have to thank you for the tutorial. It is very useful to quick test different LLM models, vector databases and other points with different data content. And just after that think about what stack is suitable to index your own data and try to deploy it on a server-side. Thanks!
This is mind blowing! Thank you, much appreciated!
Ollama works very slow on Windows. Tested using LM studio and it's working.
Hello, anything in the pipeline for using the new ollama embeddings? they are super super fast :-)
Already live in main repo. Will be live in next weekly release
Just for the record Ollama does support embedding models. They work very well (very fast) for text, not sure about other modalities.
We support Ollama embedding now! When i made this video we had not yet (or maybe it didnt even exist in Ollama yet?)
Nicely done. I like the flexibility of the tool.
This is exactly what I was looking for! :)
I have a spare iMac, nice to see it works on intel.
I get to use it for something other than youtube videos on the corner of my desk.
This is very cool Tim!
For ollama on Macbook pro 2016 x86 and 32gb or ram I recommend Solar, Mixtral is good but slow.
How can I get a list 📝 of models that work on OLLAMA?
@@ajarivas72 Lllama3 now is not bad, but still solar and Mixtral are good too.
Great video, Thanks very much
I get "-- loading available models --" in the Chat Model Selection combobox after selecting Ollama Base URL (that is correct btw) because I can open ollama one ui. What am I missing here?
Same. Whats the issue? Have you found any solutions?
@@Musaddique_MMALI I reinstalled today because I formatted my computer. Now is working fine.
would you consider adding more plaintext files, such as .py files? Just installed the latest version in docker and its great, love everything about this!
You know it's funny, we allow pulling in and embedding of whole github repos, but not the independent code files. I will work on resolving that for you - thanks for pointing it out 😄
@@TimCarambat you are absolutely welcome, if i helped even 0.01% it made my day!
Reply just to say we support this now (and have for a few weeks now!)
Next we just need to serve a webpage with authentication. Complete package.
The docker version has multi-user with auth :)
can you make few more videos specifically for the agents and details of embedding and possible troubleshootings please.
Good stuff. Are there any ways to connect to APIs (similar to how chatGPT works) where we could use large data sets for free? Adding individual websites and files could take a lifetime...any workarounds?
Can you put an example using Hugging Face? I don't know what happens but Anythingllm is asking ChatGPT and not my documents.
I love the idea of AnythingLLM but for some reason I cant get it to relate to the documents I provide.. and Ideas what might be causing that?
thank you for your work
This Looks so Great!
Is it possible to run it on a Server and Access the um via a Web Browser?
Yes, in this demo I was just using the desktop app but we have a docker server based/web UI as well
@@TimCarambat that sound amazing. I will try that in the next days.
Today I‘ve installed privateGPT and that is quite cool, but very limited in inputs. Yours looks much more flexible with all the possible inputs like links to websites and so on.
I have thousands of pdf documents (scanned my file cabinet. That took months). Is it possible to load them all in and then kind of talk to my entire file cabinet?
That's cool!
btw i think you've rendered a 16x10 ratio video to a 16:9
It looks a little weird 😅
Awesome thank you ! The possibility to learn by giving it docs/URL is so cool. I have one question though: Is it possible to use AnythingLLM with an API ? So we can use a model (Llama 3), add data thanks to AnythingLLM, then use an API to talk to AnythingLLM that returns response ? Thank you in advance for the answer, I guess many people wonder the same ^^.
Thanks a lot. I have 2 questions. Does it have access to the internet to fetch recent updated information? What's the best model to use for software engineers?
You can scrape websites with the built in scraper but its not "internet connected". Not like an agent-LLM.
CodeLlama for local LLM. Claude-3 Opus for closed source
Beautiful work, thanks a lot.
Can I specify the context size to use myself (for models with very large contexts like command-r ?)
This was nice. What other integration options are available, particularly on window with a good GPU
If on windows you can run LMStudio (just like ollama but with a UI) and LocalAI (requires docker to run).
Outside of that you can also turn to hosted LLMs on OpenAI, Azure OpenAI, Anthropic, Google Gemini
Together AI, and Mistral
Hi, I noticed you are still in chat mode when you asked questions, isn't it better to switch to query mode ?
Chat is general knowledge + documents
Query is just documents. In this case, since each query had used context - the results would have been the same
Awesome,
Army Swiss Knife for LLM
Can I spin up a Streamlit app with this ?
Thank you Tim very helpful. May I ask the similarities/diff between anythingllm and privategpt ?
We are a one-click single Desktop app you can install, no code or python or docker config
We support many LLM, embedders, and vectordatabases where the defaults are all local
We come with built-in data connectors like Github, website scraping, bulk website scraping, Confluence, etc etc
We also have agent capabilities for _all_ models, even if that model does not support tool calling.
We also support multi-user on our Dockerized version of the app (if you need that)
- that is just off the things i do know about PrivateGPT.
Nothing wrong with PrivateGPT at all, i think our tool is much more "accessible" and more all-in-one. Different strokes for different folks
I'm sorry, I'm aware that it may be an incredibly stupid question, but
If my LLM runs on the virtual Linux under Windows, then do i need the Linux or the Windows version of AnythingLLM?:)
I'm sorry again about it.
No need to use WSL or linux, just use the Windows version!
Pretty freaking sweet, thanks!
already using ollama... and must intergrate it with Anything LLM.
I will say ollama works a bit slow, may be because of the 7B Parameters. It take generally 70-80 seconds to deliver an answer.
Are you on an intel-based processor with no GPU? If so this is par for the course and normal.Bigger model can lead to even longer time-to-first-token or crashes
Ok, the Claude API works very well. I have a question: how can I fetch an entire website, not just a single page link? If I want to load all pages of a website, how can I do that?
Open the file manager/upload modal -> Data connectors -> Bulk link scraper
Is there a timer or countdown clock for the embedding and processing time?
Not currently, it would be nice to add though for obvious reasons
2 months have passed since the original video, 2 questions. I am running a 12 GB vram RTX3060 GPU. Will LM Studio be a better solution than Ollama? 2. Now Llama 3 has come out, will thos work with what is presumably a much more capable LLM?
Bro this is amazing!
Currently training models seems like a big hastle most training is done via google collab and such any plans on adding an easy UI interface to train models locally inside AnythingLLM?
Being able to fine tune off the chat and such is on the roadmap, but is still conceptual since 99% of people cannot train a model on their device.
So we may integrate or connected to some provider which can enable this to happen more easily. We allow you to export the chats in a fine-tune format already to encourage this behavior and id love to close the loop on it
@@TimCarambat Thanks Tim appreciate you.
Nice tool indeed, thanks
Great video thanks for developing this
Many thanks for introduction the AnythingLLM. Is there any date mentioned when this app will be release for Ubuntu. I have to admit it blocks me 😀but there are so cool features to automate our work. Or maybe is there any similar app to integrate Ollama with vector DB, embeddings, etc in similar way ?
It seems to be having an issue reading the entirety of my Docx, only 103KB. It's the CBA for my workplace. Once embedded it states that there is no context for the question I've asked, though there most certainly is. (IE, shift schedule times, etc...), it will then proceed to read OTHER parts of the document.
I wonder if there are certain document types that work better than Docx? or if anyone has any suggestions?
When you scape a URL does it extend the scrape to every page on the site? Or would I have to add all the pages of a site manually? How deep is the URL scrape?
The one in the uploader is currently just the url. Understandably that is not great for large url ingestion and there is a piece of work being done to add deep scraping (or at least multi link, possibly agent based) to the tool.
Can I use AnythingLLM locally and privately only as an API like the OpenAI API? In addition, do you offer anything like the Assistants API, where you can have models that have pre-defined instructions on how to respond?
Please release the Linux version!
We have a docker image if that works for you??
Thank you Tim I'll look into that!
Speaking of Privacy, why you skipped telling people to turn off "Anonymous Telemetry Enabled" ??
Really Thank you for your help. I'm trying.
That was really fantastic. Tm once at work i’d go full hog at it and they let you know! By the way, do you plan to implement DSPy somehow?
Unlikely because it cant fit into our stack (since its Python). The idea of the framework however is not impossible to support
This is awesome! I'm so glad I found this video cause being able to talk to my pdfs was something i've been wanting to do for a while!
I do have some questions though. I have pdfs of tabletop RPG books and when I ingest a certain book and begin asking it rules questions it seems to 'reach out' to other sources and rules that aren't in the book and the responses are incorrect. Is there a way to narrow the chat to only answer rules and such about that particular game or book instead of adding knowledge on other games it has?
Ah, this is an error class i just refer to as "general knowledge creep". The only semi-good remedy is just a really strong system prompt explicitly detailing that only knowledge in the context should be used and to not use it's existing understanding of a game on how rules for this game should work - or something along those lines.
System prompt can def fix this and depending on the model will vary in performance.
I have a M2 Max Macbook Pro. Ollama with 7B Llama works super fast when used in Terminal. Why is it slow when I am using AnythingLLM along with it. There is a drastic time difference! Pls advise.
I love the ease of set up and the local private models.
First attempts at use are not good in terms of models ability to understand and formulate answers to questions based on provided context docs even with only 8 documents provided. I did try it with just one document provided, and answers seems less confused.
Ideally I'd like to be able to point to a whole drive and chat with my own currated knowledge base. Is that an inappropriate use of this tool?
Looks great! Do you have an idea of when Linux will be available?
Been working on it this week! Looks like the app image should be good by next release, likely next week
Kudos on Anything LLM.❤
Linux coming soon :)
Linux not there yet :(
AppImage is live for AnythingLLM useanything.com/download
And Ollama supports a linux download now as well. I think its just an install script
😍
@@TimCarambat:D
Thanks Tim for this great stuff !
I am interested in generating a document based on data within multiple local files. Do you think it's easily feasible with ollama/AnythingLLM?
This is great in, do you offer any support? I just uploaded a simple document and tried even a web site but when I ask any question I get the following error "couldn't respond to message , fetch failed!"
I can tell you alreayd that the issue is you probably put in the wrong Ollama connection info. Search in the Github issues and youll see we have helped dozens of people solve this issue. Its a common gotcha
github.com/Mintplex-Labs/anything-llm/issues?q=is%3Aissue+fetch+failed+is%3Aclosed
I have tried this out a bit now and I really like it. However, I would like to import lots of dokuments and not via a GUI. Is it possible somehow to use python to connect to anythingLLM to post text files via an API or something like that? It would be fun to pump in huge amounts of text files and ask stuff.
The desktop and docker app both ship with a full API that would enable this!
Would it allow local server which gives REST api? not in tcp ip, but normal android app backend api format @GET, etc..?
The docker image comes with a full REST API. It is accessed via TCP/IP though. You can wrap it though on the Android side with Retrofit or something like that
Looks cool! Gonna try it right now
Will any LLM be able to run its own server? I find that I get the best performance using it with LM Studio, but I’d like to connect it to OpenWebUI since it looks nicer.
@Tim This is GREAT STUFF. Your putting in some serious effort and it shows. Can I use this to setup on a client site? I am a software developer and would like to create embeddings specific to ai python projects and documentation... what are a couple options i should consider. Thanks
You absolutely could but youll want to run our Docker offering because it is meant for that. The desktop app is really a local "single-player" solution.
The docker app has a full API, multi-user support, etc etc
Got this working. How do i get it to summarise a pdf please?
Hello, Tim, how are you? Thanks for the video! I have a question, I have Ollama on a ex mining rig with 5 GPUs, the response time is very good, but when I ask 2 questions in parallel it does not answer in parallel, he answers the first question and the second question remains in “queue” until I finish answering the first question, could you help me? Thank you
Looking forward to release for Linux
How cai i use local Ollama in Visual Code please for agents Crewai?
Are you just downloading the HTML web page for a site and then compiling that as a datasource is is there an agent actually crawling the site looking for the data based on a given user input?
Just downloading the page. No agent support currently
i am confused on the step where u copy the url from ollama to anything Llm. I am at the same machine and can't figure out what is the url. I tried the direction that appear when using ollama serve but doesn't work.
well done. Looks solid.
AnythingLLM only recognizes the first document I uploaded, but not the 2nd doc and not even see the website link after the 1st upload. I reset, deleted workspace, created a new workspace, restarted my computer, but AnythingLLM still not seeing the 2nd upload. Is it a bug? Please fix. Thanks, Tim!
I tried Anything LLM (using the default suggested tools for embedding, vectors and so on), with LM Studio, tested with multiple LLMs, it works fine, I can chat without trouble, until I decide to upload a document file (tried with a PDF paper from arxiv) or a web url and move it to the workspace, it gives me no error, everything seems fine, but now whatever prompt I use, whether in chat or query mode I get an error in the chat window as a response:
⚠ Could not respond to message.
An error occurred while streaming response. Failed to fetch.
I do not get any error in the LM studio log, so I suppose no data get sent to the endpoint server.
Everything freeze down cause if trying to go in the options for the workspace it doesn't do anything, if I go in the general options I do get an infinite loading circle loop. To fix it I need to reload and remove the document from the workspace.
My System: i7 3770k, 32GB + RTX 3060 12GB. Windows 10.
Note: Since my AnythingLLM doesn't have that thread option in the GUI I suppose I've to update AnythingLLM and check if the error doesn't show up in the new build.
The thread feature if available in the latest version of the desktop app - so you can redownload and just overwrite the application and youll still keep your data.
It looks like you have plenty of power and memory. "Failed to fetch" with LMStudio almost always means that the endpoint being used or the API key (if applicable) is incorrect.
The LMStudio endpoint should be somthing like `localhost:1234/v1` in AnythingLLM. Make sure you started the LMStudio inference server in that application as well since it does not start on boot.
medium.com/@ingridwickstevens/running-a-local-openai-compatible-mixtral-server-with-lm-studio-cd602efbf808
@@TimCarambatI don't think it's LM Studio related at all, cause I used it with multiple tools without trouble, as I said i used localhost:1234/v1 (you can't set the API key in AnythingLLM), but chat work fine in AnythingLLM, but as soon I try to chat or query after embedding anything I get that error (Could not respond to message. An error occurred while streaming response. Failed to fetch.)
Update:
I installed the latest. First tried without deleting anythingllm preference folder and got a fail in trying to save and embed the document suggest trying a different schema than LanceDB. Since the error was different I thought maybe the preferences have compatibility issues, so I cleaned up the preference folder and started from scratch.
And got the original message on the arxiv pdf (but no failure on embedding and saving this time), this partially froze the process as I explained above, I also got multiple wmic.exe and conhost.exe processes, a few times multiple powershell.exe spawning on my system and high CPU usage from process: WmiPrvSE.exe.
So I restarted and tried uploading a new story from the web from a basic webpage (longstoryshort.squarespace_com/great-example) and got an error when attempting to move the doc to the workspace(Save and embed): It fails again and suggest trying a different schema than LanceDB but at least it doesn't freeze (it still spawns multiple processes). So I think it's a problem with the LanceDB wheter it does fail to save and embed or not. I will try installing Chroma and see what happens, will try Ollama as well, I would still check.
is there an option to utilize GPU? is this an advantage of something like openwebui? Still learning and barely getting into understanding local llms
Ollama/LMStudio/etc providers all opt to use GPU by default and should be able to do so without downloading CUDA drivers
Thanks for the video, but I'm not sure why, but my windows version of anythingllm is slightly different and I can only follow so far before what's on my screen is different than yours. Is there a video for the windows version? Please make sure that you explain the differences.
Where is the disconnect for you? This video is quite old now and we have improved the product a lot since
@@TimCarambat You have a pop-up window that has a spot for a URL. There is no pop-up window. I'm trying to figure out how to replicate that. When I try to web-scrape with the agent, it says it can't access the web.
Did you have to give ollama 2 admin perms aswell?
So I downloaded Ollama on my Mac and all, but when I got everything LLM, it promoted me to download Ollama too so it runs on it rather than having to run both at the same time. What's the difference?
Hi, how to see if this LLM is using my GPU as well? I have a nvidia 3060 laptop gpu. Don't have the cuda installed.
The easiest way is have open your control panel and see if the gpu spikes during a chat, that is the easiest way to know
@@TimCarambat Hi, I checked the task manager, and it seems that the LLM is mostly using my regular CPU memory and not my GPU.
I tried it to use my GPU, but I am not able to work it out. Any idea how i can force it use my GPU?
Is there some setting in windows which can force the LLM to use GPU, or do I need to do this manually?
Info: I am not very good at manual stuff, but I can follow the basic instructions.
Thanks for the help, and have a good day!
I used LLama 2 LLM model with anything LLC just shown in this video and use my website to train the LLM model but I am still getting generic answers that are unrelated to my website. how do I fix this?
Is it possible to stay in OpenWEBUI and use AnythingLLM by using API on port 3001 like RAG ?
I don't find solution in Internet :(
Yes, We have an openAI compatible endpoint so you can just use that. You can find the exact endpoint by going to the Developer API page in AnythingLLM and opening the documentation. OpenAI Compatible endpoints are at the bottom
great work tim
Great explanations in your its eas very useful and helpful information thanks Sir😊😊😊😊😊😊
Hey Tim, thanks. please share more light, how to serve llmanything chatbot on a website.
secondly, does llmanything has api ?? can it be used with devika?
Are you planning on adding support for Nvidia Chat with RTX?
Do they have an api for that tool?
Can you do a review on AirLLM it’s lets you run a 70b model on 4gb of vram, or is there a way to run it on your software?👍👍👍
Ollama now supports embedding models.
And we support those models for embedding now as well!
So my question is is it possible to set up LLM anything to be accessible over a local network?
Yes, absolutely
Hi, I was excited to have this new powerful tool, and a convenient way to add a wide range of inputs and contexts into a LLM. I am running llama2:70b with Ollama on a maxed out MacBook. While not really an AnyLLM issue, I am disappointed that when I add a bunch of documents to the workspace and thread, Llama2 is terrible about using the info uploaded. I can upload a doc and immediately ask a question about the uploaded doc and it gets it wrong. I give it hints and correct it, and it still gets it wrong again and again. So one of the most important uses cases for me is a flop, as I had hoped to upload a series of papers and then ask relevant questions, combining the global knowledge with the local uploaded knowledge.
Yes, it hardly appears to use any added material at all and instead pulls about 50% from prior knowledge as well as halucinating the rest. Tested it on a couple of articles. One article about a historical figure who died (peacefully) in 1958 and it has him tragically dying of scurvy as a very young man in 1913. His actual death date is given in the heading of the article.
Another one randomly picked from today's paper was so fresh that it didn't have any prior knowledge besides the given news item. Not being able to use the inputted text it just invented a story. Can't imagine the scandals that would follow using this for anything involving actual paperwork at my workplace.
Hey how do you configure it to use your you GPU / cuda and not your CPU
if i even try to download it my pc will just burst up and catch fire
Is the GUI app for AnythingLLM open-source?
Yes, it's the same repo as the main app, it's just the desktop branch.
github.com/Mintplex-Labs/anything-llm
@@TimCarambat Thank you.
This looks great thanks!
Could not respond to message.
Ollama call failed with status code 404: model "llama2" not found, try pulling it first
If you see this error, add your local ollama again and this time save it. The save button was positioned on the far right and I never noticed it. This is an irritating af trend caused by the pressure to make all apps "mobile friendly."
This is actually not caused by that because the app doesnt even have a mobile form. Its place there for a totally unrelated reason
Excellent! Thank you!