Your New 8 GPU AI Daily Driver Rig: Supermicro SuperServer 4028GR-TRT
HTML-код
- Опубликовано: 14 окт 2024
- Welcome to Part 1 of our exciting series, where we embark on a deep dive into the transformative world of artificial intelligence, machine learning, and deep learning with the Supermicro SuperServer SYS-4028GR-TRT. In this episode, we're set to explore the myriad benefits of making this powerhouse server your go-to daily driver for all your AI/ML/DL endeavors.
Uncover the incredible capabilities of the SuperServer SYS-4028GR-TRT, a server designed to accommodate up to 8 dual-slot GPUs, revolutionizing your workflow with unmatched computational power and speed. But we won't stop there. We'll delve into the crucial questions: Why should you invest in and manage your own hardware? What makes this the premier GPU-dense server on the market? Why did I select these specific components? And, perhaps most intriguingly, how much did this entire setup cost?
Join me as we navigate these topics and more, providing you with the insights and knowledge needed to elevate your AI projects to new heights. Stay tuned for an enlightening journey into the heart of AI technology.
#SupermicroSuperServer #AIrig #MLrig #DLrig #GPUserver #ArtificialIntelligence #MachineLearning #DeepLearning #TechReview #AIhardware #GPUDenseServer
🎥 Other Videos in the Series:
Part 2 | Setup | Your New 8 GPU AI Daily Driver Rig: Supermicro SuperServer 4028GR-TRT | • 8 GPU Server Setup for...
Part 3 | External GPU Setup | Setting Up External Server GPUs for AI/ML/DL - RTX 3090 | • Setting Up External Se...
📚 Additional Resources:
Link to Cost Breakdown Spreadsheet
docs.google.co...
Supermicro SuperServer 4028GR-TRT Specs
www.supermicro...
AI/ML/DL GPU Buying Guide 2023: Get the Most AI Power for Your Budget
• AI/ML/DL GPU Buying Gu...
HOW TO GET IN CONTACT WITH ME
🐦 X (Formerly Twitter): @TheDataDaddi
📧 Email: skingutube22@gmail.com
💬 Discord: / discord
Feel free to connect with me on X (Formerly Twitter) or shoot me an email for any inquiries, questions, collaborations, or just to say hello! 👋
HOW TO SUPPORT MY CHANNEL
If you found this content useful, please consider buying me a coffee at the link below. This goes a long way in helping me through grad school and allows me to continue making the best content possible.
Buy Me a Coffee
www.buymeacoff...
Thanks for your support!
For the 3090 you should just look for models that has heatsink fins vertically aligned towards the back bracket vent slots and remove the fans.
This is very informative, even if we aren't really in the same market so the price aren't the same (and even if, the prices are really volatiles these days) ! Thank you!
Of course. I am so glad you found this video useful!
Around 39:00 you talk about an application that requires/can use obscene amounts of RAM.. care to share which one? Just curious.
I really appreciate the in depth analysis. I'd be very curious to see a comparison of, say, the DL580 g8, g9 and g10 vs the various incarnations of the 4027, 4028 and 4029.. and if we're going down the rabbit hole, how many of which gpus will work with each of them.
I'm seeing k40s at dirt cheap prices but figuring out how to implement them en masse seems kind of futile because the systems that are contemporaneous with those gpus are so out of date that finding compatibility info is tough.
Sure! In on of my research project I have been working with the entire BTC ledger. It is absolutely massive with billions of nodes. Some computations surrounding that have taken well over a 1TB of ram to efficiently complete. Of course, I could have done it with less, but the extra RAM is always makes things easier.
I certainly would be as well. Unfortunately, until Supermicro and HP decide they want to sponsor my video with some free hardware that is a little out of budget 🤣
Yeah. The K40s are tough to work with in present day. Sometimes when things are super cheap there is a reason. lol
I would only want to run large language models, are the p40s good enough or do I need the p100s?
PS: Great video, very informative!
and do I need the 3090 then?
and do i need the 3090s then? :) Thank you
Hi there. Thanks so much for your comment and kind words!
So if you goal is to exclusively work with LLMs. I would go with the RTX 3090 or the RTX Titan both are well suited for LLM workloads.
But will it run Crysis?
Seriously though, I would love to have something like this. I'm just a hobbyist that is just now getting into AI for fun. I've always been into tech and gaming, but over the last year started playing with stable diffusion in most of my free time. Few days ago I got a subscription to AI Dungeon to do some choose your own adventure RP, and now I am obsessed. I don't want to think about what your power bill is going to look like. I imagine with a rig like this, One could do a whole RP novel with complete context, have it do voice recognition and character voice generation, periodically generate images of the story, and as video gen becomes open source, be able to generate clips of the key points in the story. I don't think I could do 9 or 10 grand though, ouchies!
Haha. It probably would 🤣I would imagine so. That would be a very interesting application of this rig. I would also be curious to see what it could do there.
Yeah the 8K was hard to stomach, but for me it has been worth it since I use it everyday and probably will for the foreseeable future. I will be measuring power consumption over the next month so I should have some real data on that soon. Sad part is this is still on the cheap end as far as rigs go.
@@TheDataDaddi I really hate that mainstream AI ML development has been only aimed at massive resource consumption, subscription business models with invasive telemetry practices, and restrictive licensing. It's basically all of the worst parts of the internet and software sectors from the last 15 years ramped up to 11. I'm hopeful that we will see optimization and efficiency breakthroughs, as well as more interesting use applications as more people get into it for non-enterprise purposes. It's not going to go well for us consumers if we rely on the folks that produce, sell, and rent the hardware to be the ones that lead in the development of the software. I appreciate your channel. I just started watching some of yours and others' videos over the last few days learning about the required hardware for the various ML applications. I am realizing that I may well be in way over my head.
Yes!!!
You will both see from the sad Newbie level of my questions that I BARELY understand enough about all of this to even select a cloud hosted ‘system’…. So what am I doing watching these videos??? Well even an old Newbie Boomer can see EXACTLY what TPTB (The Powers That Be) have decreed for the rest of us! Want to utilize real cutting edge AI? You will be forced to ‘rent’ the hardware because the only readily available ‘consumer’ level (sub $10k) will NOT even be systems the average person can understand….
Thanks for the ‘pushback’ on this!
Hey how about making a discord server for your fans to gather and share info, also interact with you?
Hey there. Thanks so much for commenting!
This is a fantastic idea. This had not even occurred to me! I will do this as soon as I get a chance!
@@TheDataDaddi Thank you for consideration
Here is the discord server in case you are interested.
💬 Discord: discord.gg/RyRHEn3yMx
@@PokerFaceMAN
Hi~ I have a question!!
2 Xeon e5 cpu have maximal 80 pcie lane, is it possible more than 4 gpu on this system each gpu have 16 lanes?
Hi there. Excellent question!
Great observation that there are only 80 PCIE lanes. This confused my greatly a first as well. To deal with the limited number of PCIE lanes, the server makes use of PCIE Switches. PCIe switches allow multiple devices, like GPUs, to share a limited number of PCIe lanes by dynamically allocating bandwidth. This enables systems to support more GPUs or other devices than would be possible with direct lane connections to the CPU. PCIe switches optimize resource use, ensuring efficient data flow and efficiently allocate lanes, so the actual performance impact is minimized for most workloads.
@@TheDataDaddi pcie switches can also (at least in some cases) allow communication directly between pcie devices, removing the need to travel through the CPU.
Hi, I have recently bought this system. I see there are two EPS 8-pin connectors (JPW3 and 5) on the mother board, in addition to the other 8-pin PIC power connectors. My question is that I have bought Tesla M10 32 GB GPU which requires an 8-pin EPS connector. Can I connect that GPU to one of these EPS connectors. The card needs an 8-pin 12V EPS connector.
What would you suggest.
Many thanks for your amazing support.
Hi there. Thanks so much for the question!
You should be able to connect if they are the traditional PCIe 8 Pin (6+2) GPU connectors. The 4028GR-TRT has 8 12 EPS power cables that support most Tesla GPUs naturally. From what I can tell, the M10 uses the more traditional 8 pin connector. You could buy an adapter from 12 V EPS to PCIe 8 Pin (6+2) power. I think the following should work for you.
a.co/d/hMUEDvs
Hope this help you! Please let me know how it goes.
@@TheDataDaddi thanks for your prompt reply. I am wonder if instead of 6+2 pin, I can use 4+4 pin since I have it at home.
@@wasifmasood969 Unfortunately, I do not think the 4+4 pin will work in your case. You can certainly try, but it will likely not fit.
Hi, did you run any inference of an LLM with the P40 and P100 cards? Could you share your experience with that in a video? Do you have plans to try and build a small cluster and spread training or inference across nodes?
Hi there! Thanks so much for the comment.
As of this moment, I have actually not used any of my GPUs for working with LLMs. I definitely plan on doing this as soon as I get a chance. My plan is to make a video series devoted to exactly what you need to work with LLMs as a home lab or small business.
Eventually, I would love to try this! I just do not really have the resources at the moment to turn all of my current servers into a cluster. So, for the moment I will be using my SM 4028GR-TRT as my main DL machine. I do plan on doing that eventually, and I will be sure to share the journey.
Get yourself a Perc H750 Raid card for that 8x slot in the back and you have 8 more drive bays.
Hey man. Thanks so much for letting me know. I actually just found the equivalent Supermicro part just a little while ago: AOC-S2208L-H8IR
@@TheDataDaddi Ive got the same budget for a workstation ATM. my next ML rig will be using a ASUS Pro WS WRX80E-SAGE SE WiFi II AMD WRX80. Not sure if I want to go with A5000s or 5000 Adas. Im hoping for a decent drop in prices on the secondary market this time next year. I was able to price out from FalconNorthwest a Tallon with a 4090 FE and that motherboard for just a hair over $8k. Considering it but not sure if I'm going that route yet.
Gotcha. Yeah that is a tough decision. If you have the money probably the 5000 Ada, but from last I looked those were significantly more expensive than the A5000 so I believe the A5000 would likely present a better value. I agree it will certainly be interesting to see what happens to the overall secondary market once the 5000 series GPUs are release. It could possible make older models cheaper, but honestly at this point I am not sure. Hardware prices continue to skyrocket so I am not sure that will slow any time soon. I guess only time will tell. @@ToddWBucy-lf8yz
How much noise is this rig making?
Hi there. Thanks for the comment.
When running at idle its not too bad. But on boot and with all GPUs maxed out its can get over 90dB. On boot especially it sounds like a jet airplane.
@@TheDataDaddi thanks for that info! I loved the video and subbed. I'm building my home lab out, but I don't have a great sound proof area for these old supermicro servers (yet)... I'm jealous of the price/performance though OMG
I am so glad you enjoyed it! Really appreciate the feedback!
Yeah, it is definitely the loudest server I have ever worked with so I would be sure to have a plan to manage the sound before you get one.
I have been super pleased with the setup so far. Having all the GPUs accessible in the same place has totally change my workflow. I am loving it so far. Highly recommend going this route! @@tbranch227
28:17, using 8xH100 is a better idea in my opinion. They offer significantly more compute power and VRAM, and would easily fit in this server.
I wish lol. That would literally cost more than my house at this point.
www.ebay.com/itm/334736271213
@@TheDataDaddi yep but your house can't run inference. Those GPU prices are insane... I'm going to cry 😭
Yeah. Lol. Its tough even for my research lab to have access to enough compute resources to be competitive. @@gileneusz
@@TheDataDaddiI feel it's more about your prompt than it is about compute. I'm sure they both matter. But asking better questions gets better answers. That's the greatest difference between talented human engineers and scientists compared to crappy ones.
for the $ you'd be better off with 2 of the 6 card boards/servers, without nvlink between the gpus memory bandwidth will still be the bottleneck assuming the software is cuda optimized.
8x24gb gpu, will tie the cpu up a lot dealing with large resizable bar across the link between the 2 cpu.
higher tbw endurance ssd would also be a more rugged solution.
Hi there. Thanks so much for the suggestions!
Do you have a link or more specific name for the 6 card boards/servers? I'd love to take a look more closely at these. Also, wouldn't the memory bandwidth be a bottleneck for the 6 card boards/servers as well?
Interesting. May need to make sure you get CPUs that are beefy enough to support that effectively.
Do you have a recommendation of higher tbw ssds?
36:05 the cat is missing 😢
He was at the vet today unfortunately