Видео

At least 64GB of detotated WAM

P40 is better at fp32 and int8 but it sucks at fp16. P100 is the opposite.

Thanks so much for the kind words!

Ah this is an excellent tip! Will definitely come in handy in the future. Thanks so much for the comment here!

Hi there. Thanks for the comment! I am not super familiar with koboldccp or sillytavern so please take what I say with a several grains of salt, but from my brief research they need an AI model integrated in some way. I am assuming you want to host this locally. For this I would go with the GPU with the largest VRAM, so the 4060 16GB TI is the clear choice in my book.

@@TheDataDaddi yeah. Basically locally hosting an AI model to my pc. I'm not really into machine learning as of yet. Currently I have a 2060 on my pc and 6 GB isn't really enough. Another question is AMD not a good alternative?

@@rukitorin1998 It can be, but AMD is generally not as easy to use for machine learning. However, it may be applicable for your use case. AMD definitely seems to offer better price to performance, but there are still a lot of bugs from what I understand. I also cannot recommend it strongly AMD because I have not personally dabbled there. What I am telling you now is just based on feedback I have received from viewers.

@@TheDataDaddi thanks for responding, sorry for the late reply. :D i will be buying the 4060 ti 16gb soon when i find a cheaper price point. Living in the Philippines prices are somewhat higher. 30 kph peso(519.08$) to 32k (553.68$) is the prices im looking at right now.

So glad you enjoyed it!

Hey there! Thanks for the comment Yeah, I have gotten a couple of comment on this, and I think I misspoke in the video. I think I meant 20 logical cores not physical. Sorry for the confusion here.

Hey there. Thanks for the comment! The prices vary almost daily so it hard to say why the price is so different. However, I got the values for this video from EBAY with respect to my region. I found the lowest price from a reputable seller (ie good ratings and greater than 500 reviews).

Awesome! That is gonna be a great setup for you. Yes, I think I misspoke in the video. I probably meant 20 logical cores not physical. Sorry for that.

Hi there. Thanks so much for the question! Yep, that is a fairly accurate view. The first ml rig that was a custom build is more for smaller/single projects. It have also work for me well as a nice test bed before scaling up projects. I think this was about $1000 at the time I made the video. Here is what it would cost today: pcpartpicker.com/user/sking115422/saved/#view=zrXn23 The server in this video was designed to work with large scale DL application like working with some of the smaller open source LLMs, diffusions workloads, running many projects at once (this one has been a god send), large scale GNNS. I would say that this server is pretty much equipped to do anything, but work with very large LLMs (think Llama 70B as this pretty much requires a cluster to work with). I believe this rig cost me about $8300 to build. However, a large portion of that was RAM. I also bought RTX 3090s instead of p40s or p100s. So you could scale things down a bit and save a lot money here. I choose these specs for particular use cases that may not be applicable to everyone. I believe I address that in part of the video series if you are interested in more details there.

Hi there. Thanks so much for the comment! This is great to hear! What AI/ML/DL applications have you used the ARC A770 for? I would be super curious to know if you have tried it out for anything yet!

and do I need the 3090 then?

and do i need the 3090s then? :) Thank you

Hi there. Thanks so much for your comment and kind words! So if you goal is to exclusively work with LLMs. I would go with the RTX 3090 or the RTX Titan both are well suited for LLM workloads.

Hi there. Thanks so much for the comment! I have successfully run CUDA 12.2 with no issues. I have not tried any newer versions though as of yet.

Hi there. Thanks so much for your comment! I have a video related to this topic. Link is below: ruclips.net/video/oQqcXyS2WQ4/видео.htmlsi=iisyvWNAJlWJwQQD If it is just error light on the drive, you should be fine. I have been using TeamGroup drives for a while now and been very happy with them. If there are issues booting after installation, there is likely a problem with the drive and you might want to have it replaced. I believe TeamGroup offers a 3 year warranty at minimum on most of their products. Might be worth reaching out to have it replaced if there are issues.

​I ended up getting it working by flashing the perc h710 mini D1 with Lsi IT mode firmware and installing the boot images using the guide on fohdeesha. It's working great and I now have 4 p40's mounted on a gpu rack above the case. I still haven't got the fans not to run at 100% yet but when I get some extra time I'll continue digging into the ipmi tool. I've gotten it to return the fan info but it doesn't seem to respond to turning off automatic fan control or manual controlling the fan speed ​@@TheDataDaddi

I resolved my fan problem by using RACAdm. I cleared the idrac logs and disabled the PCI fan response. Fans are running at 10% at idle now. I watched your video on installing the idrac service module and I'd like to note that version 10 point something has a Ubuntu specific version so you don't have to do any converting. Although I am running Ubuntu version 20 so all I did was run the shell script and voila all done

Hi there. Thanks so much for the kind words! Really appreciate you watching!

Hey there. Thanks so much for the comment! Unfortunately, I cannot seem to find the deb package from this solution. However, here is an alternative way that you might be able to covert the RPM to DEB. stackoverflow.com/questions/61932118/convert-rpm-to-deb-without-alien Let me know if it works!

Hi there. Thank so much for the comment! 1) So I think this is related to a similar question by another view so I will post the response here: This has to do with the way the average is calculated in this case. The CPU-scaled throughput for each scenario is divided by the price of the GPU(s) for each different scenario (GPU, Number of GPUs, Model, Precision, Task, etc). For more specific details, please take a look at the raw data in the Google Sheet link in the video description. It should make things clearer if there is confusion here. After some review, in a fair number of cases, the LSTM did not see much performance benefit over the CPU. This drove the GPU-scaled throughput per dollar way up nominally. These values artificially dragged up the overall average. I tested this by switching the aggregation method to median rather than average, and the values are more in line with what you would expect based on your observation here. You can also see this if you just look at the BERT or RESNET50 model scenarios. These are much more in line with what you would expect. In summary, the numbers do appear to be correct even if they are higher than expected globally. The data when training the LSTM was significantly different than the other models. This leads me to believe there may be some issue with the model setup, dataset, or hyperparameters (or some other reason I am missing). My gut tells me I just did not use large enough batch sizes or a large enough dataset. In any case, more digging will be required to understand why this occurred. 2) This does look weird. This is because the values list are for each part of the bar graph so you would need to add both together so that the top of the bar alight with the Y-axis. I have since changed it so that it is 2 separate bars. I you visit the report now you should find that it is much more readable. Sorry for the confusion here. Really appreciate the kind words! Thanks so much for watching.

Hi there. Thanks so much for the question! There is definitely a lot of confusion around which GPUs support NVLINK. I myself was confused until recently. Basically, for the Pascal series GPUs NVLINK only exist for the SMX form factor. If you go with PCIE, like most people there is not option for NVLINK (or at least not that I have ever seen). So to clarify both the P100 and P40 GPUs test in this video are not NVLINKed.

Hi there. Thanks so much for the comment! Man! That is an excellent price. Great pick up! Hope it works well for you!

Hi there. Thanks for the comment! Yeah the RTX 3060 is a good choice for sure, and it has 2nd gen tensor cores as an added bonus. Definitely agrees that having noisy fans on a small PC is not the best way to go.

Hi there! So glad you found this video helpful. Really appreciate the kind words!

Hi there. So glad you enjoy the content! Really appreciate the kind words!

Hi there! Thanks so much for the comment. Happy to do it. So glad this was able to help you!

Thanks again for commenting! Can't tell you how much I appreciate the support!

@@TheDataDaddi I just bought a P100 based on your testing and my budget. Already have a P40 for inference.

@@callmebigpapa Awesome man! Hope it works out well for you. Let me know if you have any questions on setup or anything.

Hi there. Thank you so much for the like, sub, and the kind words. Really appreciate the support!

Hi there. Thanks so much for the question! This is tough. I think you really can't go wrong either way. Both are solid choices for ML/DL GPUs. However, for about the same price (I checked just now and they seem to be about the same price on EBAY in my area) I think I would go with the RTX 3080. You get much better performance and the difference in VRAM from 12 to 16 GB is not significant enough to justify the performance difference.

Hi there. Thanks for the question! I am really happy with them. I still think that they are the best GPUs for the price at this current moment.

Absolutely. Please feel free to contact me any way you like. All of my contact information can be found in my RUclips bio. I will also paste it below for convenience. 🐦 X (Formerly Twitter): @TheDataDaddi 📧 Email: skingutube22@gmail.com 💬 Discord: discord.gg/RyRHEn3yMx

Hi there. Thanks for the comment! I would definitely agree here. If you are not really sure if AI/ML/DL is a career or a passion for you, taking advantage of resources like Google Colab, Kaggle, and others like this would be a great option to figure things out before purchasing any hardware. Once you decide though that you do want to want to work a lot in this area, I think buying your own hardware makes to most sense right now for the majority of people.

Hi there! Thank you so much. This is amazing. Really really appreciate your generosity. Can't tell you how much this helps the channel.

Hi there. Thanks so much for the comment! A lot to unpack here. 1) I want to start by saying from my experience my DL models tend to use FP32 by default. Maybe with some of the LLMs this is not the case. I am honestly not sure here. Your point though still stands the P40 is generally bad at FP16 or Mixed Precision (MP) operations, or put another way, you do not get any speed up from using FP16 or MP with this GPU. Check out my video here for the actual numbers: ruclips.net/video/OCx2xr5Xaj8/видео.html This point I am trying to make here is with the P40 while you don't get any speed up for FP16 or MP you do not lose any either. So it is not nearly as bad as the theoretical specs indicate. Also, for many programs you would not need to search out specific version that are FP32 compatible it would actually be the opposite. You would need to find version that use FP16 or MP by default. Additionally, even if the program was set to FP16 or MP by default the P40 (based on my testing) would run it just as as well as if it was FP32. 2) If you are going to go the AMD route, the MI series is probably the best way to go. AMD GPUs in general offer a ton of compute for the money and look really attractive on paper. However, based on the feed back I have gotten from viewer that have gone this route it is a pain in the ass to get things set up correctly. I will paste below the exact conversation I had with a viewer recently. """ > What are your thoughts on AMD GPUs in general? There is so much to say here I don't even know where to start. AMD is not worth it if you value your time, but once it's working it is fairly decent and a good alternative to Nvidia. I built my PC around a year ago and back then I would not have recommended it due to the lack of online resources. But since then, backends and troubleshooting support have gotten significantly better. The hardware is great! But it's the software, optimization and lack of general AMD support that make it incredibly time consuming to deal with. The documentation for Instincts is so poor that I had to use Nvidia's P100 flowrate recommendations to get even an idea on how to build my system haha. MLC LLM is a promising project that is optimizing LLM inference though, and it is looking to be an alternative to current CUDA wrappers. > How buggy is ROCm? Is it still bad as of now? ROCm has improved a lot since then. In fact ROCm itself is generally working very well! (Non-driver portion) The problem comes from their closed source proprietary drivers, most infamously amdgpu-dkms. You need very specific kernels and very specific distributions to successfully build the kernel module. Not only that, AMD refuses to fix the infamous GPU reset bug which makes stopping and starting VMs incredibly frustrating. For modern AMD cards, you *should run into fewer problems, but for Instinct cards you will need to do a lot of troubleshooting. Motherboard support is also very poor for Instinct cards, you will run into problems with the GPU reset bug, trouble initiating the GPU, detecting the GPU, and so much more. It took me weeks of non-stop work to get x2 MI100s even detected last year. It took countless hours of headache and trial and error without any help from AMD. I tried messaging their custom service and only got this response: "Your motherboard is unsupported. I can't help you." Unfortunately, the bugs keep coming. To get amdgpu-dkms working on Debian 12 I need to troubleshoot a driver build problem. AMD shipped with corrupted build packages. Pain lol. I'm currently dealing with a problem where the amdgpu driver keeps failing to get user pages, and it is extremely inconsistent to get llama.cpp running. I could keep going but I think you get the point. haha """ Personally, I have not gone down the AMD rabbit hole at this point for many of the reasons mentioned above. However, when I get more time I would like to. I think if you could ever get things running AMD might be a much more cost effective way to go. It just a question of how much time is it worth to work through all the bugs. 3) Damn that is awesome. Love your setup. Are you running a cluster or do you have them a two separate machines? Yeah I bet you paid quite a bit more a couple years back. Interestingly, I feel like a lot of servers and adjacent hardware has come down in price significantly in recently years, but has started to get more expensive again recently with the AI craze.

Hi there. Thanks for letting me know! Here is the link to the cable that has worked best for me: a.co/d/0317ir6Y

Hi there. Thanks for letting me know! Here is the link to the cable that has worked best for me: a.co/d/0317ir6Y

Hi there. Thanks so much for the feedback! Lol. Not sure if being compared to Elon is good thing here, but I do know I am a bit slow when presenting. This is feedback I have received in the past. I am working on making my videos faster, more concise, and trying to talk faster. I hope you found the video useful despite my delivery. I appreciate the feedback again and your view!

I am not super familiar with ComfyUI. For my research though, it does look like it support a dual GPUs setup. Looks like you can run separate instances on each GPUs and also run a distributed setup by using the ComfyUI_NetDist extension. Here are some links that might be helpful to you: github.com/comfyanonymous/ComfyUI/issues/155 github.com/city96/ComfyUI_NetDist github.com/comfyanonymous/ComfyUI/discussions/836

@@TheDataDaddi i was not aware of the NetDist node. that solves a problem for me. thank you for your responds.