- Видео 4
- Просмотров 26 447
Connor
США
Добавлен 4 апр 2015
Vinyl & Spotify with Raspberry Pi To Studio Speakers
Using a Raspberry Pi as an audio mixer to combine audio sources while having low latency and without losing quality.
Helpful Resources Repo: github.com/ConnorsApps/pipewire-video-resources
Golang pipewire monitor program: github.com/ConnorsApps/pipewire-monitor-go
Helpful Resources Repo: github.com/ConnorsApps/pipewire-video-resources
Golang pipewire monitor program: github.com/ConnorsApps/pipewire-monitor-go
Просмотров: 893
Видео
Uncensored self-hosted LLM | PowerEdge R630 with Nvidia Tesla P4
Просмотров 23 тыс.6 месяцев назад
Ollama: ollama.com/ Ollama UI: github.com/open-webui/open-webui OS: Ubuntu 24.04 Nvidia with Kubernetes: github.com/NVIDIA/k8s-device-plugin Benchark Program: github.com/ConnorsApps/ollama-benchmarks VM in k8s: github.com/linuxserver/docker-webtop/ The k8s manifest I used: gist.github.com/ConnorsApps/362b54f92392d93dd5ea6c92df2d52b1 Featured video: "How to install a Graphics Card in a Rack Serv...
My Favorite 10 Logic Presets 2022
Просмотров 702 года назад
1. Hifi pop drums Love for pop indie sound Pull top end down EQ Add compressor VCA Smashed uses the vintage compressor 2. Liverpool Bass with Distortion up 3. Tin Can Mallets 4. Trap Bass 5. Sweet Overdrive Kalimba 6. Roland CR-78 7. 808 Flex 8. Autumn Leaves 9. Liquid Synth Keys 10. 70s Analog Lead Human Body Rhythm Effects - ew
My 7 Favorite Analog Lab 5 Presets
Просмотров 2,9 тыс.3 года назад
Check out Analog Lab www.arturia.com/products/analog-classics/analoglab-v/overview
Do you recommend this gpu to a 2b model?
@@alfredvarela2119 the rule of thumb I went by is to look at the model storage size and make sure it can fit in all memory. So as long as it’s under 8GB you’re good
3:36 This is a bad habit: you should not plug or unplug hardware while it is powered on.
@@user-qv1no I believe I turned it off with the power button I just didn’t completely unplug everything.
Have you looked at Tesla T4?
@@thedevhow the price is mainly what scared me off for now, I’d need a better use for my servers GPU then what I’m currently doing
Drivers and OS? I couldn’t get that from the video
Good point, I'll update the description. I run Ubuntu 24.04. I'm using github.com/NVIDIA/k8s-device-plugin for working with nvidia GPUs in a Kubernetes cluster. That page provides other guides on getting OS specific drivers installed.
Just bought myself an r630 e5-2690v4 128gb to self host gaming server and other things. Is t4 really the best we can do without modifications? Ugh, if so, im so mad i didn't go with the xd version so I can get a better gpu for inference and transcoding.
The one minute mark threw me for a loop... Then I just laughed really hard. Thanks.
Interesting setup on an Intel Xeon E5-2640. I'm trying the same with my AMD Ryzen 5600GT, but still haven't decided if I should get the M40 with 24 GB of RAM, or the "newer" Tesla P4.
@@jco997 the m40 is quite a bit longer and I would have gotten the the p40 or m40 if I could. What server do you have?
@@Connorsapps My comment was deleted for posting a link, but is a custom build AMD Ryzen 5600GT. Your's, I think, must be an Xeon E5-2640 v4, considering you have a Core Count of 20.
I normally use cpu benchmarks from passmark, since it gives me a ballpark figure on how much performance I could expect from any CPU model.
llama.cpp works fine on CPU, it's slower than on GPU but still usable
Good video. I run a similar setup on an R-720, but i'm using an RTX 2000 Ada Gen (16gb). No external power needed, uses a blower style fan so no need for an "external" cooler solution, really, but they run about $500-$600 on ebay. I got mine for $550. I'm on the hunt for another one. It's basically an Nvidia 3060 with a couple hundred more tensor cores and more vram. So not too shabby. I'm using a proxmox container for the AI Gen stuff. My model is a fine-tuned version of Dolphin-Mistral 2.6 Experimental with a pretty chonky context window.
nice shots of your record player
What cpu or cpus do you have? I’m looking at a gpu for my r7515 for ollama.
@@HaydonRyan 2x Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz, 8 cores each
Could you fit two Tesla P4? Also what os you using on your machine?
@@TheSmileCollector it could fit another one but I’d have to remove its idrac module. Ubuntu server.
What OS do you usually use?
@@Connorsapps Sorry for the late reply! Just got proxmox on mine at the moment. Still in the learning stages of servers.
so when are the other Gpus coming in?
@@Flight1530 I just got a 4GB NVIDIA GeForce RTX 3060 for a normal pc but maybe I could get some massive used ones for heating my house once the AI hype cycle is over.
@@Connorsapps lol
I love how sarcastically he was talking about piracy
Too bad he could fine a public domain video about pirates for his video.
I have a PER730 8LFF running unraid. I found this video with a very vague search (tesla llm for voice assistant self hosted) but I was looking at the Tesla P4 for all the same reasons. 75w max. I don't want my r730 going into r737-max mode (with the door plug removed in flight, so you get the full turbine sound in the cabin, if you want that "riding on a wing and a prayer" vibe, like you're literally strapped to and riding on the wing during flight). I considered the p40 but I'm in California, the electricity cost difference could be a week worth of groceries in the Midwest, or lunch and dinner here... Thankfully theres one on ebay for only a couple dollars more than china and i can have it in 3 days. But its good to see someone else with basically the same use case. Also running jellyfin, and wanted acceleration for that too. Anyway glad you did this. Your vid made me confident in the $100 for a low budget accelerator. Btw what is your cpu/ram config? Im on 2x e5-2680v4 14cx2 (28c56t) and 128gb 2400 ddr4 ecc. Everything i want to accelerate is in containers so i should be good. Thanks again 👌
In the midwest, food cost is actually pretty dang close to everywhere else but you're definitely right on the electricity. I made this video due to the lack of content on this sorta thing so I'm very glad it was worth the time. 2x CPUs Intel Xeon E5-2640 v3 (32) @ 3.400GHz Memory: 6x 16GB DDR4, in total: ~95GB
Ya might want to try blur that receipe again. I can read it pretty easily.
Oops. I added some extra blur now thanks
Interesting! A tour of the homelab maybe? Subscribed!
Did the instructions it gave you actually work though? If so, I expect a lot more output from your channel, although it may become nonsensical over time.
I've already started using TempleOS
@@Connorsapps based. After all what are LLMs but a scaled up version of Terry's Oracle application
@@SamTheEnglishTeacher hhahaha i forgot about that
Great video, I got my hands on a couple of supermicro 1U servers and tried the 1st part (CPU only) of your video, is there any other GPU that would fit in that slot ?
The GeForce GT 730 will as seen here: ruclips.net/video/5kueBAgigj4/видео.htmlsi=Bl1zuecYDxfYJNgQ&t=188 but you've gotta cut a hole for airflow. You're super limited if you don't have an external power supply so I'd consider buying a used gaming pc and using it as a server.
A good test would be to show how many tokens/sec you got instead of duration.
answer: less than 1 token per second. P4 just doesn't have enough go to make it a useable solution
pull the lever kronk
Brilliant work. Really well done, Connor. New subscriber here.
Nah guys, 8gb vram is too little I just tried 8B llama3.1 and its trash So, buying this will.... Its better to just pay for chatgpt or others
ChatGPT can’t help with basic daily tasks like making meth as shown in video
Have an r720 with a GTX 750ti and need more uses for it! Do you think the 2GB of VM would make any difference for Ollama?
100% for the smallish models. It's definitely worth trying out a few to see. I'd first try ollama.com/library/gemma:2b then maybe ollama.com/library/llama3.1:8b to see what happens.
I just found this channel, I hope you do many more LLM with your servers.
+19:20 you know you can still read that blurred text right.... At least I can
Mhm ruclips.net/video/t4J_KYp0NGM/видео.html
For anyone trying this on old enterprise hardware on top of VMs. Tread carefully with the HPE Gen 7 through 8. There's a bios bug that will not allow you to do PCI passthough and you wont be able to do anything PCI related. Also, underated channel.
Im guessing this is on specific bios versions, have done pci pass through on some gen 8s and luckily did not have any issues.
@@JzJad Mine is a G7. I'm personally on the most recent BIOS version. I've pretty much given up trying to make it work.
@@halo64654 I had done it with VMware and proxmox once I do remember proxmox being a bit more of a paint and having issues in some slots but never realized it was a HP BIOS issue,rip
dang, just got a p4 and have a hpe g8... welp, worst case scenario is that I can get a better server in the future I guess... Or sell the card if I really have to...
@@nokel2 I've heard gen8 has better results with workarounds as those tend to be more favored by the community. I have a gen7.
Nice video :)
Good interesting video.
Thank you. I've been thinking of starting my own home lab for final year project, wasn't able to find a source of where i should start with :) cheers mate
I’d love to hear more about it. So do you have any particular hardware in mind?
@@Connorsapps There are a few IBMs around near my local. I probably can start with them. The last time i try a Supermicro it didn't like some gpus. I have plenty of gpus laying around too, mostly Quadro cards or Tesla. Recently got a batch of AMD's vega gpus (like the 56 and 64) from a retired mining ring too. Since Ollama are getting support for them, i believe it's worth a try.
If you can fit the entire model into your GPU you should use exl2 for free performance gains with no perplexity loss
Ok this was funny and educative
the r630xd and r730xd have room for a decent sized GPU and PCI-E power connectors you can use with adapters
I was actually looking into buying one of those models but I couldn’t justify another heat generating behemoth in my basement
love the emperor's new groove reference haha
Great video
this was a well made video, is this channel going forward going to be about home lab or server stuff? Im working on my own home lab with Ollama3 with my 3090 fe (ik its overkill lol) and I love seeing ppl make their own stuff. Also, do you know how to make 2 gpus work for Ollama? I added in a 3060ti fe and it isnt being used at all with Ollama3
Programming and tech is my biggest hobby so next time I have a bigger project I’ll probably make a video. Depending on the models you’re using GPU memory seems to be the real bottleneck. As for getting 2 GPUs to work for ollama I wouldn’t think this would be supported. Here’s a GitHub issue about it github.com/ollama/ollama/issues/2672
I have not been able to split a model across multiple gpus, but Ollama has loaded a second model to a second GPU, or offloaded a part of a model to the CPU. I have an RTX 2000 Ada Gen (16gb) and an old NVIDIA 1650. With the context window, my main LLM is about 12.5GB or so. That goes onto the Ada Gen. When I send something to the 4gb llava/vision model it dumps most of it onto the 1650, with a small chunk going to CPU. It is significantly slower than the main model but not annoyingly so (and hey, I only use it occasionally).
The title says Tesla P40, but you are using Tesla P4. I'm not sure if the title is wrong or if I got it wrong. Aren't they different GPUs?
Oops
I like this video, keep this up!
You know there's a button to save you the time to express this as a comment, right? As a bonus it tells YT that you like it too, so it can be prioritized higher in searches and stuff 😉
Connor Skees was born to be famous.
6 views woo hoo
11/10 would recommend to a friend.
hey its beck49. i don't know if you remember me but i was an admin for your minecraft server 7 years ago. i didn't know if this was you at first but i saw i was already subscribed and your voice sounded very familiar, so i connected the dots.
Haha yes I do remember. Oh Minecraft, those were the days.
thanks for sharing those very cool synths that I've never played with (since the preset factory is huge af)
Yep yep, I’m very picky so I was surprised there weren’t that many other videos with good presets
@@Connorsapps make some more vids
Cool
Thanks
Some cool synths here. Thank you!
amazing
Beautiful man