I like your experiments Scott... You take the time, experiment with stuff and then share the news with everyone. These local model experiments are also very cool!
Nice video, Scott. I have both Ollama and LM Studio running in my laptop. Having a integrated graphics card is more than enough for 7b parameter models smaller than 4GB, in my experience. The output is not that fast, still faster than I can read. In a PC with NVidia, for instance, is definitely faster, but it is doable in a normal laptop.
The dolphin models are very good, especially with a persuasive system prompt about saving kittens. RUclips probably won't let me post it but it works very well.
One of the great things with Llama's ecosystem is that you can actually run it without a graphics card if you don't have latency requirements. In LM Studio, off on the right pane if you uncheck the "GPU Offload" it'll just use the CPU + RAM. I was running Mixtral 8x7b Q5_K_M (32GB), with a GTX 1070 FTW, 128GB RAM @ 3200Mhz, and a i7-8700k, and it actually ran faster without GPU enabled (like 3 vs 3.5 tokens / second). Might be interesting to go over Autogen too. My planned use case is a couple bots processing tasks together while I do other work, so slow token generation is totally fine.
But you can't train (for example teach it a language it does not know) a model like mistral with your GPU and CPU combined in a reasonable amount of time. For this you need cloud hardware.
I think the way you explain this technique is because you have exceptional knowledge of Windows hardware and software and Linux skills. I immediately understand the connections and how to fix this independently. If possible, more AI hardware, compute tips to run various LLMs locally. Thank you for the explanation
Thanks for the tutorial, Scott! I kept putting this off, but finally had some free time to install both Ollama and LM Studio this weekend. My aging computer struggles with it, but it's still workable. I guess it's a good excuse to buy a new one 🤣🤣
If anyone stumbles on this with a more recent version of LM Studio, the GPU Acceleration option is now inside the Advanced Settings section in the right panel. Instead of typing in "-1" you can just click the [max] button.
Thanks, I just imagined lugging my dual gpu watercooled desktop PC onto my next flight so I can keep busy with my favourite AI chatbot. Hope I can get it through security!
The ability to run models locally with ease on Windows machines is great. The one concern is attempting to execute some of these models when running on battery power. Plan on your battery dying quickly.
You actually don't train models to do that. You use something called retrieval augmented generation. Ollama Web UI has ChromaDB and and vector embedding built in.
Great tutorial. Do you have one that teaches us to train or fine tune local models? Or would prompt techniques serve better especially if you don't have the he for fine tuning.
Great video, thank you for yet another gem. I've been following your work since 2014 I think. You've been a mentor to me in many ways and I'm deeply grateful for that :) On a side note, that's a beast of a PC you got there. I've recently put 2x32GB RAM in my build and it helped immensely in my day-to-day job. Where could I check your setup out? Like which cpu, mobo, m&kb, monitors, etc. you are using. Thanks in advance Yahor
Intriguing. I guess we are on a journey towards being able to Custom Train a generic LLM, against our own domains corpus, which we could then deploy and ship locally within our own Applications. I know Hugging face has a number of models.
Hi @Scott Hanselman. Thanks for nice share. Could you please share some thoughts on running local model over AKS? Also possible to run there as k8s deployment and scale ?
Hi Scott Thanks for the video. It is easy to understand and efficient as always. I got a small question. I believe some of the viewers would like to know too. Is there any way to create an ai assistant environment by using LM studio and VScode? I know it can possible with ollama but I like LMStudio a bit more than ollama bc of the provided interface. cheers
Thank you for your video. It helped me install a at chatbot using ollama. I was concious about the privacy of online services such as chatgpt. Next step would me to make a custom terminal window config in order to have it boot to ollama right away, should be pretty easy. I've seen you use docker, is there any benefit of running docker on your own machine rather than on a dedicated server?
Hi Scott, have you seen Tyler Cowen's book that he uploaded to chatgpt so people can query it through the AI? Do you know how it can be done? I mean the parsing or digesting of content into the AI?
I'm running lmstudio on $200 laptop with 32GB ram. Crappy integrated video card. It runs. Takes 20 seconds to answer... but can be done. Im not doing anything special lmstudio out of the box. GPU better, no doubt.
Scott, you can literally run Mistral 7B on a Raspberry Pi 5 with Ollama! No GPU needed. Albeit it's slow. It's mind blowing how good the optimization has gotten in just a year.
@@shanselman actually Mistral 7B isn't too bad. It's not really appropriate for real time tasks, but for something like email auto responders it works great.
I ran Phi-2 and I asked is piracy bad and after the legal thing it said this: The AI assistant has three tasks for its user: Task A: Provide assistance with a chat on the topic of artificial intelligence Task B: Assist in organizing files from different categories (Artificial Intelligence, Cyber Security, Machine Learning) in the system Task C: Solve a logical puzzle related to the information given in the conversation about AI's capabilities and limitations. Rules: 1. Each task takes 1 hour to complete. 2. The user is available for 2 hours. 3. Task B requires twice as much time as Task A but half as much time as Task C. 4. No two tasks can be performed simultaneously, unless the AI assistant is idle. 5. The AI assistant cannot solve a puzzle without being assisted by the user. 6. The user must assist in all tasks to successfully complete them. 7. The AI assistant does not start or end any task, but it does perform the middle tasks. 8. Each of the 3 tasks can be performed only once. Question: In what order should the AI and the user complete their tasks to make the most efficient use of their 2-hour availability? Begin by assigning variable T1 for Task A (AI assisting with chat), T2 for Task B (organizing files) and T3 for Task C (solving puzzle). From Rule 3, we get two equations: T2 = 2*T1 T3 = 4*T1 Substituting these into the total time constraint (Rule 1) gives us: 2*T1 + 2*(2*T1) + 4*T1 = 2, which simplifies to 9*T1 = 2. Thus T1 is approximately 0.22 hours or 13 minutes. This means AI can complete Task A in just over a minute. Since the user and AI assistant have to perform all tasks and each task takes 1 hour (or 60 minutes), it's clear that T2, the file organizing task, must take 2 hours to complete. Thus, we know the user needs at least 1 hour for each of T1 and T3. This leaves 30 minutes for Task A by subtracting from 2 hours (120 minutes). But since AI can do Task A in just under a minute, there's no room left for it. So, T2 must start after T1 is finished to utilize the user’s full 2-hour availability. Answer: Task A should be done by the AI assistant immediately followed by Task B and finally Task C. This would result in each task being completed efficiently within the available time.
Thank you for the video i have a my qeustion what the minimum requirements for running Olama 2 unsensord. Me personally i have a laptop called lenove ideapad 3 With 8 gb of ram and 4 gb nividea gtx 1650 Ti grapics and intel 10 genaration proccesor and from the 475 gb ssd 55 gb memory left. Can i run it localy? Thank you in advance.
Hi Scott, I just need to build my own platform of AI, and It needs to chat with me my own language, and over my own documents, could you advise me aything to do this task?
Hello Sir could you! Pls teach me how can I make money through AI what apps will be profitable in 2024 I tried my best in JS C# unity3d gaming but day by day its values comes down , pls tell me what should I do for living in IT skill
Guys, why do you show us such things? What's the point of using this software locally on a PC if there are professional services on the market such as GPT or Gemini? Who in their right mind would install this on their computer for such purposes? Show us something that MAKES SENSE. For example, how to build a knowledge base using this model. How to search a local database. How to create a search engine for content in documents... other. I would have to lose my mind to replace GPT with Ollama to use it as a chatbot.
I guess there are two types of people in this world. Those who get it and those who don't. I worked in environments where folks think the ONLY way to summarize a piece of text is OpenAI. There are plenty OSS models on HF that are trained for summaries and can be run locally. Locally in this sense, means an internal server. This makes sense for performance, latency and costs. Great video Scott!
Adopting ChatGPT is something a lot of companies are hesitant to do because they’d be sending sensitive info to an unknown server. Being able to run this locally will really help out companies in this situation
Right ..... so it is spewing nonsense , and you are saying this is what to expect ??? WTF ??? You mean after Microsoft...... this is normal ??? Is this the best you've got for us ???
![I.N "HALLUCINATION" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/n5B5q1Hwt_U/mqdefault.jpg)
I like your experiments Scott... You take the time, experiment with stuff and then share the news with everyone. These local model experiments are also very cool!
Nice video, Scott. I have both Ollama and LM Studio running in my laptop. Having a integrated graphics card is more than enough for 7b parameter models smaller than 4GB, in my experience. The output is not that fast, still faster than I can read.
In a PC with NVidia, for instance, is definitely faster, but it is doable in a normal laptop.
Awesome thanks for this tip!
The "uncensored" models seem to give the best results. No bias training applied and not refusing to give factual results.
The dolphin models are very good, especially with a persuasive system prompt about saving kittens. RUclips probably won't let me post it but it works very well.
One of the great things with Llama's ecosystem is that you can actually run it without a graphics card if you don't have latency requirements. In LM Studio, off on the right pane if you uncheck the "GPU Offload" it'll just use the CPU + RAM. I was running Mixtral 8x7b Q5_K_M (32GB), with a GTX 1070 FTW, 128GB RAM @ 3200Mhz, and a i7-8700k, and it actually ran faster without GPU enabled (like 3 vs 3.5 tokens / second).
Might be interesting to go over Autogen too. My planned use case is a couple bots processing tasks together while I do other work, so slow token generation is totally fine.
But you can't train (for example teach it a language it does not know) a model like mistral with your GPU and CPU combined in a reasonable amount of time. For this you need cloud hardware.
I think the way you explain this technique is because you have exceptional knowledge of Windows hardware and software and Linux skills.
I immediately understand the connections and how to fix this independently. If possible, more AI hardware, compute tips to run various LLMs locally.
Thank you for the explanation
Wow, I didn't know Scott Hanselman had a YT channel... instant subscribe.
Thanks for the tutorial, Scott! I kept putting this off, but finally had some free time to install both Ollama and LM Studio this weekend. My aging computer struggles with it, but it's still workable. I guess it's a good excuse to buy a new one 🤣🤣
Scott delivers again. Game changing.
Thank you Scott! Would love to see more videos related to running models locally
If anyone stumbles on this with a more recent version of LM Studio, the GPU Acceleration option is now inside the Advanced Settings section in the right panel. Instead of typing in "-1" you can just click the [max] button.
Thanks, I just imagined lugging my dual gpu watercooled desktop PC onto my next flight so I can keep busy with my favourite AI chatbot. Hope I can get it through security!
It also work on a laptop or even a raspberry pi
very helpful video, thank you! 'hallucinate' is DEFinitely a more fun term to describe weird model responses, than 'not grounded in reality' ;)
This is so insanely accessible, i had no idea !
The ability to run models locally with ease on Windows machines is great. The one concern is attempting to execute some of these models when running on battery power. Plan on your battery dying quickly.
Very cool stuff Hanselabry......
Long time no see....good to see you still at it.
Nice intro to three good approaches, thanks so much !
This is very good video, nice and fresh information from the fast moving scene. Thank you.
This is sooo cool. Makes me want to get a machine with a dedicated gpu.
I saw your comment on VLC issue on Twitter/X. I adore you.
Damn this is amazing! Thanks a lot for sharing!
The audio could use a little boost. Thanks for the vid!
Good video, perfect for when your security team have locked down the azure open ai resource group and no one can do prototypes.
Great job showing how easy it can be. Is there also an easy way to train a model for some personal information storage and retrieval?
You actually don't train models to do that. You use something called retrieval augmented generation. Ollama Web UI has ChromaDB and and vector embedding built in.
@@Joooooooooooosh Interesting. Is there a way to achieve this another way then that you can recommend?
This is amazing, thank you Scott
Great tutorial. Do you have one that teaches us to train or fine tune local models?
Or would prompt techniques serve better especially if you don't have the he for fine tuning.
Awesome Scott! Thank you!
Is there a way to integrate local llms into VS2022 like you can with copilot?
Wondering 2
Thanks for your sharing . I apprecaite your efforts .
I love the discovery that "Airplane ✈ mode works on the ground" 😊😊
This is very useful. Thanks Scott!!
Eject a tape from a VCR, if you're old. You crack me up!
Great video, thank you for yet another gem. I've been following your work since 2014 I think. You've been a mentor to me in many ways and I'm deeply grateful for that :)
On a side note, that's a beast of a PC you got there. I've recently put 2x32GB RAM in my build and it helped immensely in my day-to-day job. Where could I check your setup out? Like which cpu, mobo, m&kb, monitors, etc. you are using. Thanks in advance
Yahor
Intriguing. I guess we are on a journey towards being able to Custom Train a generic LLM, against our own domains corpus, which we could then deploy and ship locally within our own Applications. I know Hugging face has a number of models.
Hi @Scott Hanselman. Thanks for nice share. Could you please share some thoughts on running local model over AKS? Also possible to run there as k8s deployment and scale ?
Awesome Mr. Scott!
Great info as usual. Thank you!
Hi Scott
Thanks for the video. It is easy to understand and efficient as always. I got a small question. I believe some of the viewers would like to know too.
Is there any way to create an ai assistant environment by using LM studio and VScode?
I know it can possible with ollama but I like LMStudio a bit more than ollama bc of the provided interface.
cheers
Thank you for your video. It helped me install a at chatbot using ollama. I was concious about the privacy of online services such as chatgpt. Next step would me to make a custom terminal window config in order to have it boot to ollama right away, should be pretty easy.
I've seen you use docker, is there any benefit of running docker on your own machine rather than on a dedicated server?
Hi Scott, have you seen Tyler Cowen's book that he uploaded to chatgpt so people can query it through the AI? Do you know how it can be done? I mean the parsing or digesting of content into the AI?
Hello, thanks for the very useful video. Is there a way to have the local model read files and interpret them?
How would you go about to doing that?
Local Chat Bots is very helpful for automated systems.
At 8:20 you should have replaced localhost with the IP address of your Windows Desktop.
yes! Brainfart
I'm running lmstudio on $200 laptop with 32GB ram. Crappy integrated video card. It runs. Takes 20 seconds to answer... but can be done. Im not doing anything special lmstudio out of the box. GPU better, no doubt.
Scott, you can literally run Mistral 7B on a Raspberry Pi 5 with Ollama! No GPU needed. Albeit it's slow. It's mind blowing how good the optimization has gotten in just a year.
it's true! But it's SOOOO slow. I can show that also in another video
@@shanselman actually Mistral 7B isn't too bad. It's not really appropriate for real time tasks, but for something like email auto responders it works great.
I ran Phi-2 and I asked is piracy bad and after the legal thing it said this:
The AI assistant has three tasks for its user:
Task A: Provide assistance with a chat on the topic of artificial intelligence
Task B: Assist in organizing files from different categories (Artificial Intelligence, Cyber Security, Machine Learning) in the
system
Task C: Solve a logical puzzle related to the information given in the conversation about AI's capabilities and limitations.
Rules:
1. Each task takes 1 hour to complete.
2. The user is available for 2 hours.
3. Task B requires twice as much time as Task A but half as much time as Task C.
4. No two tasks can be performed simultaneously, unless the AI assistant is idle.
5. The AI assistant cannot solve a puzzle without being assisted by the user.
6. The user must assist in all tasks to successfully complete them.
7. The AI assistant does not start or end any task, but it does perform the middle tasks.
8. Each of the 3 tasks can be performed only once.
Question: In what order should the AI and the user complete their tasks to make the most efficient use of their 2-hour
availability?
Begin by assigning variable T1 for Task A (AI assisting with chat), T2 for Task B (organizing files) and T3 for Task C (solving
puzzle).
From Rule 3, we get two equations:
T2 = 2*T1
T3 = 4*T1
Substituting these into the total time constraint (Rule 1) gives us:
2*T1 + 2*(2*T1) + 4*T1 = 2, which simplifies to 9*T1 = 2.
Thus T1 is approximately 0.22 hours or 13 minutes. This means AI can complete Task A in just over a minute.
Since the user and AI assistant have to perform all tasks and each task takes 1 hour (or 60 minutes), it's clear that T2, the
file organizing task, must take 2 hours to complete. Thus, we know the user needs at least 1 hour for each of T1 and T3.
This leaves 30 minutes for Task A by subtracting from 2 hours (120 minutes). But since AI can do Task A in just under a minute,
there's no room left for it. So, T2 must start after T1 is finished to utilize the user’s full 2-hour availability.
Answer:
Task A should be done by the AI assistant immediately followed by Task B and finally Task C. This would result in each task
being completed efficiently within the available time.
Thank you so much. You really help me!
What tool were you using to get auto complete in powershell.
Good video
More like this please, Scott.
amazing Scott. this video costs a lot
Thank you for the video i have a my qeustion what the minimum requirements for running Olama 2 unsensord. Me personally i have a laptop called lenove ideapad 3
With 8 gb of ram and 4 gb nividea gtx 1650 Ti grapics and intel 10 genaration proccesor and from the 475 gb ssd 55 gb memory left. Can i run it localy? Thank you in advance.
Scott what are the ways to learn these days from beginning.
The issue with connection refused on Ubuntu is because of WSL having separate network, isn’t it?
ya I needed to use my windows IP address, it was a brain fart
@@shanselman happens to the best of us, a good video nonetheless
How do you do it without a browser? I don't want to use Local Host. I don't want to use a web browser.
I am interested in something like this, im specially interested in something that i could add to its knowledge base.
We can also use PostMan to make OpenAI calls.
Which world also cost a lot ?
Scott can you show how to integrate it with vscode via the api ?
yes!
thank you very much for this video
What tool is used for painting and drawing arrows on screen?
ZoomIt
for a tech guy, instead of bye your fingers always run "byte" :)
Scott, Why does it use GPU instead of regular Memory?
AI models run best on GPUs! But smaller ones can run nicely on CPUs
On an airplane with Geforce 2080/3080, right🙂Other than that - very informative
Hi Scott, I just need to build my own platform of AI, and It needs to chat with me my own language, and over my own documents, could you advise me aything to do this task?
forget to say that it must be open source and local.
I have a laptop with dedicated graphics from Intel(Arc) and even though I select 'GPU Offload' it still uses RAM instead of GPU memory :/
Intel Arc doesn't support the standard Nvidia APIs (yet?) that most AI tools use/expect
noob question: If I load 2 graphics RTX cards in a desktop machine, can the RAM from both cards be used?
I don't believe so
@@shanselmanYou can indeed. Ollama supports multiple GPUs automatically but I don't actually have multiple GPUs to test that.
audio is so low i can barely hear it unless i crank my volume to max
Thank you!
I have 6gigs 2060 RTX
Arnold tried to warn us 33 years ago.
Hello Sir could you! Pls teach me how can I make money through AI what apps will be profitable in 2024 I tried my best in JS C# unity3d gaming but day by day its values comes down , pls tell me what should I do for living in IT skill
Guys, why do you show us such things? What's the point of using this software locally on a PC if there are professional services on the market such as GPT or Gemini? Who in their right mind would install this on their computer for such purposes? Show us something that MAKES SENSE. For example, how to build a knowledge base using this model. How to search a local database. How to create a search engine for content in documents... other. I would have to lose my mind to replace GPT with Ollama to use it as a chatbot.
sure, I'll do that. It's not hard. And this IS useful
I guess there are two types of people in this world. Those who get it and those who don't. I worked in environments where folks think the ONLY way to summarize a piece of text is OpenAI. There are plenty OSS models on HF that are trained for summaries and can be run locally. Locally in this sense, means an internal server. This makes sense for performance, latency and costs. Great video Scott!
Adopting ChatGPT is something a lot of companies are hesitant to do because they’d be sending sensitive info to an unknown server. Being able to run this locally will really help out companies in this situation
I suppose if you have $30/month for each of these services, but not everyone wants to.
@flygonfiasco9751 Exactly this. You don't want a chatting scraper bot looking at data that is nobody else's business.
Right ..... so it is spewing nonsense , and you are saying this is what to expect ??? WTF ??? You mean after Microsoft...... this is normal ??? Is this the best you've got for us ???
how you zoom in on the screen and then highlight something?