Mistral 7B LLM AI Leaderboard: GPU Contender Nvidia RTX 4060Ti 16GB
HTML-код
- Опубликовано: 21 окт 2024
- Mistral 7B LLM AI Leaderboard: GPU Contender Nvidia RTX 4060Ti 16GB
This week in the RoboTF lab:
The standard card on the channel an Nvidia RTX 4060Ti 16GB gets put through the Mistral 7B leaderboard gauntlet.
Final results at 12 Min Mark.
GPU Link: 4060ti 16GB amzn.to/3NeSEGT
Leaderboard is live: robotf.ai/Mist...
Leaderboard reports (from these videos if you want a hands on look): robotf.ai/Mist...
Model in testing: huggingface.co...
Just a fun evening in the lab, grab your favorite relaxation method and join in.
Our website: robotf.ai
Machine specs here: robotf.ai/Mach...
GPU Bench Node
Open Air Case amzn.to/3U08Y27
30cm Gen 4 PCIe Extender amzn.to/3Unhclh
20cm Gen 4 PCIe Extender amzn.to/4eEiosA
1 TB NVME amzn.to/4gWFcFb
Corsair RM850x amzn.to/3NkITa4
128GB Lexar SSD amzn.to/3TZYYGh
G.SKILL Ripjaws V Series DDR 64GB Kit amzn.to/4dAZrWm
Core I9 9820x amzn.to/47UuIST
Nocuta NH-U12DX i4 CPU Cooler: amzn.to/3TZ7O6R
Supermicro CX299-PGF Logic Board amzn.to/3BxbWVr
Remote Power Switch amzn.to/3BubQOg
Recorded and best viewed in 4K
Your results may vary due to hardware, software, model used, context size, weather, wallet, and more!
I'm interested in more tests on how two or four 4060ti comparing against a single more expensive consumer card like 3090 or 4090. The benefits of larger total VRAM 2x16GB or 4x16GB against 1x24GB. Also, the graphs not showing CPU usage, is there any bottleneck from CPU? Are there benefits from more cores, or single threaded performance is crucial?
thx
Thanks for watching!
I think 4060 ti 16 gb is an excellent gpu for LLM price, performance, efficiency & vram and all your 4060ti videos confirms it.
For the c urrent price of 1, 4090 you can get upto 3 to 5, 4060ti, i.e, a lot more VRAM, which means much more capabilities, load bigger models, even if evaluation is a bit slow.
Obviously you will need workstation CPU like threadripper or xeon to get so many pcie lanes.
The only downside of 4060 is its bandwidth, which gets cut down to x4 pcie, if used with PCIiE gen 3 system as it is a pcie gen 4 x8 card.
I originally had 7 4060's in the lab, have since moved to other cards but still keep a few around. Very good cards for price, power consumption, and being small for certain rigs. At least obtainable by most people's budgets that are just playing around.
@@RoboTFAI Is it a good idea to pair a 3060 ti 8gb with 4060ti 16gb?
Replied to your other comment more in depth - short answer is sure! or what is your expectations/needs?
mates I have a question. currently I have a 3060 ti 8gb and I want to pair it with a 4060 ti 16gb for deep learning. do you thinks it is a good chois to go with a 3060ti 8gb and a 4060 ti 12gb for deep learning?
Sure can absolutely use mixed multiple gpus. If that's what you have available it's a great choice, else it's a matter of your budget and needs (or wants). I hope these type leaderboard tests help put some information out there for people to decide those things for themselves. 4060's are great for price/power usage/form factor - but the more expensive cards will also smack them down in TPS. But how fast do need responses? or again how fast do you want responses?
@@RoboTFAI Honestly I do not want it to be super fast and based on your leader board I am ok with the 4060 ti 16gb result.
The key note is i am from IRAN and due to heavily sanctions I cannot buy a new 3090 or else I would buy it and the used market are filled with overly mined 3090 that ruined the card and that is why I asked about rhe 4060.
I just wanted to know if I can keep my 3060 ti 8gb and add a 4060 ti 16gb since my motherbord(TUF gaming b660m-plus) supports two gpus. another note that I have a 650w psu and I think I can sqeez the 4060 on to the system without changing the psu but I was thinking not to take any risk and buy a 750w psu. what is you recommend on the psu wattage?
and the other thing is with these dual gpu if I use parallelism, can I run 30b models or I need to upgrade?
@@RoboTFAI Sorry for the late response , man I do not want it be super fast and based on what I saw on your leaderboard I think the 4060ti alongside my current 3060ti 8gb would be a good starting combo.
Im Persian and due to the heavily sanctions by the US, now I am unable to buy 3090 and the used market is very untrustworthy due to the vast mining and I though the new 4060 ti alongside my 3060 ti would be better than the used 3090 which have bing heavily mined.
The other thing is do you think with parallelism, Can I run 30b models?what about some fine-tuning?
The last thing that I want to ask you is that a my 650w psu would be enough to support the 3060ti and 4060ti or i need to get 750w psu?based on my research the two gpu and the rest consume about 600w.
Men in Black: Noisy cricket. I've got this card and I think it was a good first buy. Should I get a 4090 or wait for the 5090 for inference work?
Wait for the 5090 release, based on recent leaks, it shows that 5080 is as performant as 4090 with 50 watt lower in TDP.
5090 is much more performant but its 550w to 600w TDP.
Depends on your needs and budget of course. the upcoming 50** going to be expensive, 4090's still really expensive - do you need the speed? There is a chance we will have a 4090 in the leaderboard series here 🤔
@@RoboTFAI I'd be very interested in seeing the 4090 especially compared to the 4060 Ti done in the video above.