How Fast Will Your New Mac Run LLMs?
HTML-код
- Опубликовано: 28 авг 2024
- How fast can the new Apple Silicon Mac you so desperately want run LLM's and is it worth the price?
llama.cpp benchmarks: github.com/gge...
Ollama: ollama.ai
00;00 Intro
00:47 Benchmarks
05:06 Unbox
05:47 Results
Support My Work:
Check out my website: www.ianwootten...
Follow me on twitter: / iwootten
Subscribe to my newsletter: newsletter.ian...
Buy me a cuppa: ko-fi.com/iwoo...
Learn how devs make money from Side Projects: niftydigits.gu...
Wow nice video! Very helpful, and well explained. Thanks
TG largely depends on memory-bandwidth (the SoC has to pump all of the parameters and the KV-caches from RAM into the SoC's caches for each token generated). PP (and ML) is dependent on compute (GPU-horsepower) because token-processing can be batched.
The M4 has 20% faster memory-bandwidth in addition to the faster GPUs. Let's see when Apple will do MacBooks with these chips, maybe I will upgrade my M2. For me, the M3 is not interesting enough for an M2 upgrade.
Very helpful thank you so much
Pretty impressive, for the same tests, I was getting around 73 tokens/s on a Windows 11 WSL Ubuntu setup with a RTX 4070 Super GPU (AMD CPU)
Oh nice. Thanks for sharing - Definitely think investing in a better GPU for my PC could work out more financially viable if I ever need it.
Hi Ian , um.. this year i will be going to college for my btech degree for 4 years and its on artificial intelligence and data science . In this course their will be core subjects of computer science as well as on ai and ds. I will be dealing with machine learning, computer vision, deep learning, etc and I plan on doing competitive programming in my second year. As well as I would love to have additional functionalities like using davinci resolve and blender, etc but that's just well.. additional. Nevertheless can you suggest what laptop I should buy? My budget is around 1.2 to 1.6 or 1.7 lakh INR laptop which roughly about 1,437.14 us dollars to 2,035.95 United States Dollars. As a professional laptop which one should I buy. windows or MacBook? and which models would it suffice? I'm even ready to wait for the new ai powered chip sets by both m4 MacBooks and xelite windows. Right now I have and radeon hp laptop with 8 gb ram and 1 tb HDD storage with and ryzen processor. What should I do ? Can you help? What and how to manage hpc ? What to buy? My work load generally require 32 gb of ram and graphics of 4060- 4090 nvidia. And I don't know much difference between by how much the processor differ from intel and and but i prefer and. Can you suggest...
yeah minimum requirement say 16 gb or 32 gb ram and vram like 8 gb would suffice. In this course i woul be required to do ml inference and not training. The vram is the limiting factor. And no laptop in this world can do that. Thats why you need data centre and cloud compute.
I need to do basic machine learning inference upto 1-2 billion parameters atleast. And also my course has computer vision, nlp, iot etc.. which has its own requirements....
But the thing here is that in my college they have both data centers as well as cloud set up. And its good.
Why i am asking is that i need the laptop for myself and not to complete 4 years. What should i buy for the best bang to the buck but as well it wouldn't need heavy repairs and it would be reliable through out for atleast 5-6 years.
I understand macbooks soc chips have different efficiency as per what ram to be used and it is same as system. And it is efficient but while talking of windows a lot to be considered as each part means something crucial to the high compute processing and result. Even the overall health.
What I would say is that since buying my Macs after uni, they've generally held up for about 5 or so years without issue if they're looked after. However, I've mainly used them for web dev, rather than pushing them very hard. Currently my M1 Max is working out really well as my only work machine and was a reasonable price since it's older hardware when I got it at the start of the year. That said, if you do have hardware issues with them, they'll likely be expensive to fix, since they're basically completely unrepairable at this point and would need to be handled by apple or a specialist.
@@IanWootten thank you
Can you try stablediffusion?
Yessss, and also something like SAM or YOLO too
great deal!
How did you get a brand new M1 Max in 2024. I’d love one but can only find refurb 😢
costco clearing out remaining M1 stock here in the UK. They're probably one of the best Apple discounters I've seen when it comes to clearance lines.
@@IanWoottenthank you I’m in UK too and found them in the Costco site. No brainer got me as the M3 is eye watering for what I’d like where as your M1 looks perfect.
Can you run MistralAI ?
Sure can. I get around 55 t/s. But I could really well on my M1 Pro. I can also run mixtral - I think that's the more interesting one since it's a huge 26GB model and will run at 33 t/s.
@@IanWootten a video about it would be more than worth it, just saying
I want to start experimenting with llms and I have a budget for laptop or pc or a compromise of both , I was going for a great Mac or an ok one and a pc , what’s your advise ?
A lot of it will come down to personal preference. I'm familiar with Macs, really like that they are silent and have great battery. Most of my choice is based on that, the fact they're very good for llms too works in my favour. I'm sure there's some pretty good PCs out there too, and now Ollama works there too.
@@IanWootten yes I like Mac OS much more than windows but my concern was the speed and size of the model , I am concerned with that 16gb of unified memory wouldn’t be enough
If I was really concerned about performance I would not buy a laptop. An M2 Ultra MacStudio in the refurb store can be had for $3500.
Sure, I still need portability in this case. You also have limited options - only studio available in my region is £5k
@@IanWootten yea
Nice
Definitely over value-engineered the M3 Pro, probably using LPDRR5 and PCIe 5.0 clock efficiencies on fewer chips to make-up the differences while increasing profit margins. Also curiously the M3 8gb chips for the Macbook Pro and iPad are the same thing. So also possibly many 8gb Macbook Pros are also possibly iPad chips. Some weird Apple logic going on here. I still think Apple's iPad Pro forecasts for this year were off and they made a product their customers want, a Macbook Pro that does iPad Pro duties.