H100 is not inferencing, it is for training (but widely used for inferencing too, super wasteful). There is a massive market hole for inference hardware.
For your 4090 machine give it a try with ExLlamaV2 or Tensor-RT as an inference engine. They will give you +50% performance boost compared to ollama / gguf models. Also this initial 2 seconds performance hit is only ollama specific thing with default settings - and it happens because by default ollama unloads model from memory after some idle timeout. You can turn this behavior off
Running the GPU at a reduced power level can also be quite viable. It's common that you can sacrifice a bit of speed for significantly better efficiency. Especially for tasks that bottleneck on memory bandwidth. I rarely run my 3080ti past 80% power. It varies a lot from card to card and from task to task, so it takes a bit of testing to find the right balance. Though, the 4090 especially is known to be tuned deep into diminishing returns from the factory, so you can cut a lot of power without losing much performance.
Energy efficiency and heat are the reason I go for mac at the moment, upgraded a few years ago from a 1080 ti to a 3080 ti. Basically sold my pc a month later and bought a PS5 and MacBook. So I really appreciate the inclusion of efficiency in your testing.
I highly appreciate your focus to run LLMs locally, especially on affordable mini PCs. Very helpful. This kind of edge computing will grow massively soon. So please continue to create/run such tests.
You don't have to load the model into the vram every time. It is actually crazy to compare like that. You load once and then send requests, then RTX latency will be the lowest.
If you wanna make your tests deterministic with ollama, you can use the /set command to set the parameters for top k of 0 and top p, temperature of 0, and set the seed parameter to a same value. Also, if instead of running a chat if you put the prompt in the run command and use -verbose you’ll get the tps
Currently NPUs can't run the AI models of the tests well. llama.cpp (the inference code behind ollama) does not run on the NPUs (yet). It's all marketing only. Qualcomm/QNN, Intel, AMD, Apple all have different NPU architectures and frameworks, which make it very hard for llama.cpp to support them. Apple does not even support their own NPU (called ANE) with their own MLX-framework. llama.cpp does not even support the Qualcomm Snapdragon's GPU (Adreno), but ARM did some very clever CPU-tricks, so that the Napdragon's X Elite's CPU is approx. as fast as a M2 10-GPU for llama.cpp via their Q4_0_4_8 (re-)quantization. You can also use this speed-up with ollama (but need specially re-worked models).
@@flyingcheesecake3725 you mean Apple's CoreML and not MLX. CoreML supports the GPU and NPU (Apple calls it ANE), but with limited control. Apple's great open-source machine-learning framework MLX does NOT support the ANE (at least until now), only the CPU+GPU.
I’m with you! Apple wins by a mile. I have compared my 64G Mac Studio M2 Ultra to my Windows WS that has dual Nvidia A4500 using NVlink (20G for each card) and at half the price the Mac Studio easily competes with the Nvidia cards. I can’t wait to get an M4 Ultra Mac Studio with 192G Ram -maybe more RAM 😂. I use AI for LOCAL RAG and research and local code assistant for Vs code
The timing of this video could not have been more perfect. Literally working on figuring out how to get an LLM running locally because I want the freedom of choosing the LLM I want and for privacy reasons. I only have an RTX 4060 Ti w/ 16GB but it should be more than sufficient for my purposes. Love this style of format! It makes sense to consider electric cost when running these sorts of setups. Awesome quality as always!
You can make LLMs nearly deterministic if not entirely with the “Temperature” setting. I haven’t yet had time to experiment with this myself, but I’ve been following this with great interest. I’d be joyous to see a similar comparison woth the (hopefully just days away) M4 Pro and M4 Max alongside your other unique, insightful, and well designed testing.
Not to mention setting the seed value. I wonder if there would be any per-machine difference among them with the same seed/temp, or if it would generate the same on both. But without them pinned to the same output, a 'speed test' that isn't about tokens/sec is pointless.
@@TheHardcard “Hallucinations” (i.e. incorrect answers) and temperature are two different things. Although increasing the temperature (variability in the probability of the next token) is likely to produce more incorrect responses, it also can create more creative responses (that may be ‘better’) in many situations. Incorrect responses are simply the result of both the training data and the architecture of a next word prediction model.
I'll be totally honest... I don't have the slightest clue what's happening in this video but the little bit that I could understand seems really cool lol.
This was fantastic, thanks. I have done a video or two about Ollama, but haven't been able to do something like this because I haven't bought any of the nvidia cards. It is crazy to think that we have finally hit a use case where the cheapest way to go is with a brand new Mac.
Really interesting to see the Mac mini and how well it performs. As someone who owns a Mac mini M1, it’s simple design, relatively small footprint, exceptionally quiet, operation, and low power consumption are all pluses. I think with my experience on the original Apple Silicon, I would definitely go with the M2, especially since good for chunks of the day, this system would be idle.
Very cool content thanks Alex, I've always wanted to try one of these models locally but I couldn't find the time, at least I can follow your steps and do it a bit faster :) See you in your next video!
I got the K9 and 96gb after your first video about it, and am happy with the output speed and power consumption for the price. I did get a silent fan for it though cause it is loud. I only really do coding work so I think the Mac would be great, but the RTX is just overkill. I'm very curious the next gen of Intel processors so please do that test. I don't care about the RTX honestly cause I needed something that good I can put that money towards claude or chat gippity online and not have to deal with the heat, power consumption, maintenance, and run a huge model. I personally don't see the benefits of running a 7B parameter model at lightning speed. I'd rather run a bigger model slower at home. Great video! So much to think about.
Great test! This is exactly what I'm interested in, especially the idle power test is important, as you won't be running inference 24/7 usually. Amazing to see that "mini PC" with the eGPU takes more power in idle than my full blown desktop PC with a 4090 😅
thanks for a great and interesting video! I have a Linux machine with a nvidia card and a macbook pro m3, and personally I really care about the noise and heat from the linux/nvidia machine - after a while it's bothering. the mac makes zero noise, and no heat and is always super responsive. in my opinion, it's incredible what Apple has built. thanks again for your fun and cool technical videos!
It is not only the hardware, but the CUDA (Compute Unified Device Architecture) framework that allows developers to harness the massive parallel processing. The question is whether Apple will develop an MLX framework more suitable for AI development.
I'd love to see the same comparison with the Mac Studio with the Ultra chip since it has similar cost to a high spec PC with a RTX4090. This would be quite useful to know for workstation situations
Great work Alex! Really timely information about running LLMs locally. If you are wanting to protect your IP and security this is the way to go. It's a pioneering time. I am sure there will be more dedicated setups to do the inferenence efficiently as right now it seems still very clunky at the hardware level. It would also be great to compare a EC2 Instance spun up on demand to do the inference too. This comparison would give an option to protect your IP/Security but get perhaps a cost effective solution that doesn't act as a room heater.
That gigantic rig with the rubber fingers is comedy gold. Like you couldn't trigger them remotely, or just test them not simultaneously. Clearly it's for laughs. Testing weirdly tiny computers though, for local LLM performance and using a cable attached GPU. Who would seriously use machines like that for this purpose?
Thanks for this content and comparison. I think as this becomes more mainstream, there will be less nerdy setup’s and software to use this on a pc, Mac or tablet in the near future. 👍🤖
0.53 EUR/kWh in Germany? Where did you get these numbers from? About 0.28 EUR/kWh would be an average price. In fact prices dropped considerably during the last year or so.
Great video! Thanks for doing not only the comparison, but also the analysis for the second half of the video. I can’t wait when you test the new Intel Core CPU that just came out. Since you use the iGPU on the NUC, please mention the speed of the DRAM in your next comparison video, as it can have an impact on results (just like DRAM can have an impact on the performance of a video game). I hope a desktop machine with a 4090 makes it in your next comparison video. Even if it is a machine with a mini-ITX MB with a 16x pie slot would be much better
Great video! Thanks In the verdict you forgot to mention that the only way to run bigger LLMs is with Mac’s if you have enough unified memory. yes it’s gonna be slow but still possible and faster than cpu You mentioned it in the beginning though :)
I was actually interested in building a dedicated LLM server, but after a lot of looking around for language models, I realized most open source "coding" focused LLMs are either extremely outdated or planely not good. LLama is working with data from 2021, DeepSeek Coder 2022 etc. Unfortunately the best models for coding purposes are still Claude and ChatGPT and those are closed.
I think the 4060 ti 16GB should have been included in the comparison. It seems like a most valuable solution. It combines a small price, good performance, low consumption and fairly compact size. I think for local llm it should be the best solution from nvidia for the average user.
You can actually measure the heat VERY EASILY because every watt consumed is converted to heat with EFFECTIVELY 100% efficiency! It's just resistive heating with a few extra steps, but all those extra steps only produce heat as a loss anyway. Technically a very small amount is lost to things like vibrations and maybe UV radiation leaving through an open window or whatever, but you can basically ignore that for all practical purposes. 1W/h of power consumption is around 3.5 BTU/h and 1000J is around 1 BTU. So in your case, a single run consumes (rounding for easiness) let's say 5 BTU for the intel, 2.5 for the M2 and 4 for the RTX.
Great comparison Alex just shows if a model is tuned to run on a specific hardware it will outperform in terms of effficiency. However I saw an article today that shows Microsoft open-sourced bitnet.cpp a blazing-fast 1-bit LLM inference that runs directly on CPUs. It says that now you can run 100B parameter models on local devices. Will be waiting for your video on who this changes everything.
LLM inference is largely determined by RAM bandwidth. The newest SoCs (Qualcomm/AMD/Intel/Apple) almost all have 100-130 GB/s. While the M2/M3 Max has 400GB/s, the Ultra 800 GB/s, the 4090 has >1TB/s. And all the new CPUs have very fast matrix-instructions, rivalling the GPUs for performance. 1.5-Bit quantization might be some future thing (but not in any useable model), but currently 4-Bit is the sweet-spot. Snapdragon X Elite CPU Q4_0_4_8 quantized inference is already similar in speed to M2 10-GPU Q4_0 inference with the same accuracy.
Living in a cold climate the 4090 is perfect, in a machine with fast pcie it is super fast when coding, it sits on my home server and it really rocks. I did test the llama 3.1 8b and codegemma on a 4060 Ti 16 gig card which worked really nice, not blistering fast as the 4090 but fast enough. a 4060 Ti 16 gig is not expensive, so if cost is important that is a great option. It also is way less power hungry than a 4090. The test rig i made was running as a server with several services running and i let others play with the llm as well, nobody ever complained about speed. But the issue was if more than one was using the llm at the same time it started to slow down, or when you had a long sessions with it. That is when i upgraded the server to a 4090 card and all those issues went away. It would be cool if you tested multiple users on tests as well. From what I understand many people who run local llm share it with the family and even friends.
I wonder if the benchmark uses the latest MLX apple library, when I switch to it (on LM Studio), it was an incredible difference. I can't wait until Ollama add it!
Ollama uses the llama.cpp code-base for inference. And llama.cpp doesn't use MLX. I admit, that I did not benchmark MLX via llama.cpp recently, I have to look into it. MLX in my understanding can use GGUF-files, but only limited quantization variants. Q4_0 seems a good compromise, but llama.cpp can do better.
I like this style of video and 100% want to see more, especially with continuously updated test as hardware is updated like an m4 device or if you get a mac studio or ultra, intel, etc.
Interesting analysis! Two things: 1) I believe heat is pretty much just the energy consumed. There's no chemistry involved, just physics, so aside from negligible light and sound and a small amount of work from spinning fans, a computer using 100 watts is essentially a 100W heater. 2) The Intel mini pc loses in speed and efficiency, but its real strength is the amount of RAM. You could run a 70b model with minimal quantization and get better output than either of the other machines is capable of (though quantifying that and putting it on a chart would be difficult).
Hey Alex. Thank your for this kind of comparison. Often I only see speed matrixes, this with average energy consumption, speed, initial cost and heat generated was exactly what I wanted to see. A full array with multiple tests. Thank you, I will definitely see more if you make more power/speeds/cost/Wcost/.... Comparison. ! Thank you.
Great assessment- so glad you're including power usage. Power and CO2 cost of AI is critical for people to consider going forward. (for the Earth and our grandchildren)
@@brulsmurfI use 123B models on my M1 ultra 128GB. It can be as slow as 7 tokens per second, but I find that still usable for interactive chat. I’m more into quality than speed.
There is quite a lot of talk about this in r/localllama. For LLM work 4090 is faster than an M2 Ultra. You can get more memory on a M2 Ultra, but you can also get multiple 4090 (or perhaps even better, 3090) and put in a computer for typically less money (and better performance). If you get over 90 GB requirements (4 xx90 cards) then neither model is really fast enough to be useful. (When you get to about 1 token per second then you're probably better off just renting a machine online or buying tokens from eg OpenAI.)
Would be really interesting to see a more comprehensive benchmark in price-tiers. Like @500-750 USD (NUC, maybe a M1 MBAir 8GB (often on sale) or Mac Mini M2 8GB) § @1000-1500 USD (Mac Mini M2Max, ARM-box, ???) § @2000-3000 USD (NUC+RTX4090, Mac Mini M2Max, Base Mac Studio) § @>3000 USD (Built PC with dual 3090 24GB, H100, Mac Studio Ultra)
@@ALCE-h7b No they are still doing the what they were doing before but I suppose they took a big leap now with the release of their 'Asahi game playing toolkit' which in short makes playing AAA games on Mac very probable Maybe if you wanna lean more, why not read their blog.
Well, yes, of course we want to know how the 200H core ultra series will perform. Have you heard any news about when Asus is releasing the Nuc 14 AI? I haven't seen any public information yet.
How about on the Snapdragon CPUs? Do they have Ollama running natively on those yet? I'm guessing not, but it would be interesting to see how they match up against the Mac hardware.
There's plenty of frontends on Android(Layla, Pocket Pal, chatterAI, MLCChat, etc) or you can run ollama through Termux if you want. Snapdragons do pretty well on phones, considering they're also a phone. About 2.5-3 tokens/sec on Snapdragon 695 on Llama 3.2 8B parameter model (~$200USD phone, ala Motorola g84. Slow memory, slow CPU, 12GB RAM. About 5 watts usage). About 12-22 tokens/sec+ on Snapdragon 8 gen 3 (~$600-700USD phone, ala OnePlus Ace 3 Pro. Good memory, good CPU, 24GB RAM, so can do bigger models. About 17.5 Watts usage). So, they're not bad, but they're nowhere near a decent desktop CPU/ GPU/ or Mac integrated RAM thingy. But they can run them, to vaguely "usable" speeds (smaller models go quite a bit quicker, and are getting pretty good too. There's Llama 4B Magnum and Qwen 2.5 3B that will double and a bit those speeds, especially using ARM optimized versions. They're not "super smart/ knowledgeable", but they're good enough for entertainment purposes).
I'm surprised the the Oculink doesn't create more of a performance hit than it did. My 4090 plugged directly into the Mobo gets about 145 tokens/sec on llama3.1:8b verses the ~130 tokens/sec that you got. Kind of make sense since the model is first loaded into memory on the GPU.
Token-generation during inference is largely memory-bandwidth bound (OK, with some minimal impact for quantization-calculations) - Intel 1TB/s. And the LLM runs entirely on the 4090. The 4090 really shines during (batched) prompt-processing, blowing the Intel/Apple machine away - probably >20x faster than the M2 Pro, and way, way more so than the Intel CPU/GPU.
I've got an 4090 for my gaming PC, and a good old trusty lenovo thinkpad for work stuff. Just recently jumping on this LLM bandwagon, installed the LMStudio on my Windows desktop and exposed the LLM as Rest API, I think I have been badly spoiled by this having been my first experience running LLMs. Also now I can say: "It's for scientific purposes" when purchasing expensive GPUs lol
at 15:10 you show a document that says both that M2 16GB costs $0.11 per day and $.03 per day, and yearly costs are $40 and $12 per day. I looked closer because 3 cents per day seems suspisouly low.
Excellent comparison! For me, an interesting future comparison would be between the RTX setup and an equally priced Mac Studio. And then re-compare after the new M4 Pro/Max is available (hopefully next week?).
I don't think speed is all that important when you're running it locally. The most important thing is the memory and the bandwidth between the memory and the processing.
For an AI server, once the model is loaded into vram, you probably want it to remain there for repeated inferences. So it depends on how you set things up. Also if you want apples to apples determinism, reduce the temperature at inference time.
Speed and accuracy is what matters. Screw the extra power usage because you aren't running an LLM 24/7 you are querying it for a combined 5 minutes per day at most even if you are using it for hours during your work day. Your power usage is only spiking during actual generations which is seconds long on the 4090. The only real exception is running super big slow models for long periods of time or using AI agents which currently give crap results.
The test is also flawed by the fact that an LLM can't really "count" its own tokens, so when you ask for a 1000 essay, it's just going to to a best effort, but results might be wildly different. Setting a token cutoff and having all machines reach the cutoff would be much more accurate.
I ended up with a used studio M1 Ultra with 128GB RAM. I can run any model up to 96GB RAM, and often do. It could be faster, but the important thing is I have few limitations for large-ish models and context. What really competes with this?
12:00 Are you sure? Normally ollama will keep the model in memory for a few minutes. So if you do something like coding, the model should already be in VRAM for most queries.
If the model can fit in 16gb, the nVidia 4060ti/16gb is a solid performer, you should give it a try. That card can even run Codestral/Mistral Small at 4bit quantization at decent speeds (My old 3060 12gb was just a couple gigs short on vram)
Can you compare some AMD GPUs in these tests? They're significantly cheaper especially for the extra RAM. So I'm curious if the 'less performant' AMD GPUs can do the same as a 4090 or a lesser Nvidia GPU or a linear result of RAM= performance.. like a 20-24g AMD vs 16g Nvidia? What about using a real motherboard with full PCI-E GEN 5 and DDR5 RAM? Where is the bottleneck? Is there a limit or benefit to doing tons of RAM to either not bother with a GPU or to supplement a GPU? How can you use both the RAM & GPU? Lastly, VERY curious about the latest AMD and Intel CPUs coming that are supposed to be more power efficient and 'built for AI' 🍻
Have you looked into Bitnet at all? It's Microsoft's new inference framework for 1 bit LLMs. People running like 70 billion parameters at home. Fun stuff.
If you run ollama command with `--verbose` flag, it will give you the tokens/sec for each prompt. So you don't have to time each machine separately.
Man this question was literally on my mind. Not everyone can afford a H100
H100 is not inferencing, it is for training (but widely used for inferencing too, super wasteful).
There is a massive market hole for inference hardware.
@@adamrak7560 jup! And somehow the m4 max with 128gb is a cheap option for it now..
By orin kit 64 G or THOR
@@malloottwaiting for it but my M1 Max just need to be replace then I will test the M2 MAX 64G/2T
For your 4090 machine give it a try with ExLlamaV2 or Tensor-RT as an inference engine. They will give you +50% performance boost compared to ollama / gguf models. Also this initial 2 seconds performance hit is only ollama specific thing with default settings - and it happens because by default ollama unloads model from memory after some idle timeout. You can turn this behavior off
Running the GPU at a reduced power level can also be quite viable. It's common that you can sacrifice a bit of speed for significantly better efficiency. Especially for tasks that bottleneck on memory bandwidth. I rarely run my 3080ti past 80% power. It varies a lot from card to card and from task to task, so it takes a bit of testing to find the right balance. Though, the 4090 especially is known to be tuned deep into diminishing returns from the factory, so you can cut a lot of power without losing much performance.
Energy efficiency and heat are the reason I go for mac at the moment, upgraded a few years ago from a 1080 ti to a 3080 ti. Basically sold my pc a month later and bought a PS5 and MacBook.
So I really appreciate the inclusion of efficiency in your testing.
Electricity prices are really expensive in some parts of the world. I'd be happy running a MacBook, Mac Mini or some small NUC for a private LLM.
@@fallinginthed33p AND it good for environment!
@@vadym8713 sooolaaaar
7:42 If you set the temperature of the model to zero, the output should be deterministic, and give the same results across all machines.
I highly appreciate your focus to run LLMs locally, especially on affordable mini PCs. Very helpful. This kind of edge computing will grow massively soon. So please continue to create/run such tests.
Very instructive, I have never thought about power costs. It makes paying $20 a month for a LLM service seem reasonable.
Thanks! My small participation in the electricity cost :)
Appreciate you!
You don't have to load the model into the vram every time. It is actually crazy to compare like that. You load once and then send requests, then RTX latency will be the lowest.
Yeah definitely please keep making this style of video!
If you wanna make your tests deterministic with ollama, you can use the /set command to set the parameters for top k of 0 and top p, temperature of 0, and set the seed parameter to a same value. Also, if instead of running a chat if you put the prompt in the run command and use -verbose you’ll get the tps
Would be interested to test iGPU vs (new) NPU perf.
Yeah tops per watt
Currently NPUs can't run the AI models of the tests well. llama.cpp (the inference code behind ollama) does not run on the NPUs (yet). It's all marketing only. Qualcomm/QNN, Intel, AMD, Apple all have different NPU architectures and frameworks, which make it very hard for llama.cpp to support them. Apple does not even support their own NPU (called ANE) with their own MLX-framework.
llama.cpp does not even support the Qualcomm Snapdragon's GPU (Adreno), but ARM did some very clever CPU-tricks, so that the Napdragon's X Elite's CPU is approx. as fast as a M2 10-GPU for llama.cpp via their Q4_0_4_8 (re-)quantization. You can also use this speed-up with ollama (but need specially re-worked models).
@@andikunar7183 good info, thanks
@@andikunar7183i heard apple mlx do use npu but we don't have control to manually target it. correct me if i am wrong
@@flyingcheesecake3725 you mean Apple's CoreML and not MLX. CoreML supports the GPU and NPU (Apple calls it ANE), but with limited control. Apple's great open-source machine-learning framework MLX does NOT support the ANE (at least until now), only the CPU+GPU.
I’m with you! Apple wins by a mile. I have compared my 64G Mac Studio M2 Ultra to my Windows WS that has dual Nvidia A4500 using NVlink (20G for each card) and at half the price the Mac Studio easily competes with the Nvidia cards. I can’t wait to get an M4 Ultra Mac Studio with 192G Ram -maybe more RAM 😂. I use AI for LOCAL RAG and research and local code assistant for Vs code
BTW, the Mac Studio was half the price of the Windows workstation
The timing of this video could not have been more perfect. Literally working on figuring out how to get an LLM running locally because I want the freedom of choosing the LLM I want and for privacy reasons. I only have an RTX 4060 Ti w/ 16GB but it should be more than sufficient for my purposes.
Love this style of format! It makes sense to consider electric cost when running these sorts of setups. Awesome quality as always!
LM studio should meet your needs if you're looking for a one click install system
This video is exactly the answer for me. Thank you!
You can make LLMs nearly deterministic if not entirely with the “Temperature” setting. I haven’t yet had time to experiment with this myself, but I’ve been following this with great interest.
I’d be joyous to see a similar comparison woth the (hopefully just days away) M4 Pro and M4 Max alongside your other unique, insightful, and well designed testing.
Not to mention setting the seed value. I wonder if there would be any per-machine difference among them with the same seed/temp, or if it would generate the same on both.
But without them pinned to the same output, a 'speed test' that isn't about tokens/sec is pointless.
@@TheHardcard “Hallucinations” (i.e. incorrect answers) and temperature are two different things. Although increasing the temperature (variability in the probability of the next token) is likely to produce more incorrect responses, it also can create more creative responses (that may be ‘better’) in many situations. Incorrect responses are simply the result of both the training data and the architecture of a next word prediction model.
I'm not sure how I feel about you reading all our minds but I'm glad you made this
I'll be totally honest... I don't have the slightest clue what's happening in this video but the little bit that I could understand seems really cool lol.
Best comment 😂
This was fantastic, thanks. I have done a video or two about Ollama, but haven't been able to do something like this because I haven't bought any of the nvidia cards. It is crazy to think that we have finally hit a use case where the cheapest way to go is with a brand new Mac.
As long as you don't upgrade anything that is. 2TB costs $800, while a Gen4 2TB Samsung 990 costs $160
Really interesting to see the Mac mini and how well it performs. As someone who owns a Mac mini M1, it’s simple design, relatively small footprint, exceptionally quiet, operation, and low power consumption are all pluses. I think with my experience on the original Apple Silicon, I would definitely go with the M2, especially since good for chunks of the day, this system would be idle.
Very cool content thanks Alex, I've always wanted to try one of these models locally but I couldn't find the time, at least I can follow your steps and do it a bit faster :) See you in your next video!
I got the K9 and 96gb after your first video about it, and am happy with the output speed and power consumption for the price. I did get a silent fan for it though cause it is loud. I only really do coding work so I think the Mac would be great, but the RTX is just overkill. I'm very curious the next gen of Intel processors so please do that test. I don't care about the RTX honestly cause I needed something that good I can put that money towards claude or chat gippity online and not have to deal with the heat, power consumption, maintenance, and run a huge model. I personally don't see the benefits of running a 7B parameter model at lightning speed. I'd rather run a bigger model slower at home. Great video! So much to think about.
Great test! This is exactly what I'm interested in, especially the idle power test is important, as you won't be running inference 24/7 usually. Amazing to see that "mini PC" with the eGPU takes more power in idle than my full blown desktop PC with a 4090 😅
thanks for a great and interesting video! I have a Linux machine with a nvidia card and a macbook pro m3, and personally I really care about the noise and heat from the linux/nvidia machine - after a while it's bothering. the mac makes zero noise, and no heat and is always super responsive. in my opinion, it's incredible what Apple has built. thanks again for your fun and cool technical videos!
It is not only the hardware, but the CUDA (Compute Unified Device Architecture) framework that allows developers to harness the massive parallel processing. The question is whether Apple will develop an MLX framework more suitable for AI development.
I'd love to see the same comparison with the Mac Studio with the Ultra chip since it has similar cost to a high spec PC with a RTX4090. This would be quite useful to know for workstation situations
I was thinking the same think. And more model size headroom due to unified ram up to 128gb whereas 4090 is locked at 24gb
Great work Alex! Really timely information about running LLMs locally. If you are wanting to protect your IP and security this is the way to go. It's a pioneering time. I am sure there will be more dedicated setups to do the inferenence efficiently as right now it seems still very clunky at the hardware level. It would also be great to compare a EC2 Instance spun up on demand to do the inference too. This comparison would give an option to protect your IP/Security but get perhaps a cost effective solution that doesn't act as a room heater.
I was waited for that video, thanks for comparison
Totally agree M2 Pro would do it for now. I gave thumbs up up up
Thanks for making this video! I needed to watch something like this!
That gigantic rig with the rubber fingers is comedy gold. Like you couldn't trigger them remotely, or just test them not simultaneously. Clearly it's for laughs.
Testing weirdly tiny computers though, for local LLM performance and using a cable attached GPU. Who would seriously use machines like that for this purpose?
love the comparison. can't wait for a next, maybe try the 4090 in a pc so it can actually stretch its legs and copy the data much quicker to vram
Thanks for this content and comparison. I think as this becomes more mainstream, there will be less nerdy setup’s and software to use this on a pc, Mac or tablet in the near future. 👍🤖
0.53 EUR/kWh in Germany? Where did you get these numbers from? About 0.28 EUR/kWh would be an average price. In fact prices dropped considerably during the last year or so.
Great video! Thanks for doing not only the comparison, but also the analysis for the second half of the video.
I can’t wait when you test the new Intel Core CPU that just came out. Since you use the iGPU on the NUC, please mention the speed of the DRAM in your next comparison video, as it can have an impact on results (just like DRAM can have an impact on the performance of a video game). I hope a desktop machine with a 4090 makes it in your next comparison video. Even if it is a machine with a mini-ITX MB with a 16x pie slot would be much better
thanks yet again !
This is what I want to watch about tech. Amazing man! ❤
Glad you enjoyed it.
@@AZisk Great work sir. No one else does these things except you. 👌🏻
Great video! Thanks
In the verdict you forgot to mention
that the only way to run bigger LLMs is with Mac’s if you have enough unified memory.
yes it’s gonna be slow but still possible and faster than cpu
You mentioned it in the beginning though :)
I was actually interested in building a dedicated LLM server, but after a lot of looking around for language models, I realized most open source "coding" focused LLMs are either extremely outdated or planely not good. LLama is working with data from 2021, DeepSeek Coder 2022 etc. Unfortunately the best models for coding purposes are still Claude and ChatGPT and those are closed.
I think the 4060 ti 16GB should have been included in the comparison.
It seems like a most valuable solution.
It combines a small price, good performance, low consumption and fairly compact size. I think for local llm it should be the best solution from nvidia for the average user.
I love the test; I would love to see the speed of a beefed up M4 laptop and then maybe using 1.58bit interference
People always saying Apple is overpriced but if you actually use them as a work tool they are by far the best bang for the buck.
Storage pricing is not best bang for the buck for sure
You can actually measure the heat VERY EASILY because every watt consumed is converted to heat with EFFECTIVELY 100% efficiency! It's just resistive heating with a few extra steps, but all those extra steps only produce heat as a loss anyway. Technically a very small amount is lost to things like vibrations and maybe UV radiation leaving through an open window or whatever, but you can basically ignore that for all practical purposes.
1W/h of power consumption is around 3.5 BTU/h and 1000J is around 1 BTU. So in your case, a single run consumes (rounding for easiness) let's say 5 BTU for the intel, 2.5 for the M2 and 4 for the RTX.
Great comparison Alex just shows if a model is tuned to run on a specific hardware it will outperform in terms of effficiency. However I saw an article today that shows Microsoft open-sourced bitnet.cpp a blazing-fast 1-bit LLM inference that runs directly on CPUs. It says that now you can run 100B parameter models on local devices.
Will be waiting for your video on who this changes everything.
LLM inference is largely determined by RAM bandwidth. The newest SoCs (Qualcomm/AMD/Intel/Apple) almost all have 100-130 GB/s. While the M2/M3 Max has 400GB/s, the Ultra 800 GB/s, the 4090 has >1TB/s. And all the new CPUs have very fast matrix-instructions, rivalling the GPUs for performance. 1.5-Bit quantization might be some future thing (but not in any useable model), but currently 4-Bit is the sweet-spot. Snapdragon X Elite CPU Q4_0_4_8 quantized inference is already similar in speed to M2 10-GPU Q4_0 inference with the same accuracy.
Just what I needed, have been carrying the idea of a multi mac mini setup for a while now, especially with the use of exo labs.
Living in a cold climate the 4090 is perfect, in a machine with fast pcie it is super fast when coding, it sits on my home server and it really rocks. I did test the llama 3.1 8b and codegemma on a 4060 Ti 16 gig card which worked really nice, not blistering fast as the 4090 but fast enough. a 4060 Ti 16 gig is not expensive, so if cost is important that is a great option. It also is way less power hungry than a 4090. The test rig i made was running as a server with several services running and i let others play with the llm as well, nobody ever complained about speed. But the issue was if more than one was using the llm at the same time it started to slow down, or when you had a long sessions with it. That is when i upgraded the server to a 4090 card and all those issues went away.
It would be cool if you tested multiple users on tests as well. From what I understand many people who run local llm share it with the family and even friends.
I wonder if the benchmark uses the latest MLX apple library, when I switch to it (on LM Studio), it was an incredible difference. I can't wait until Ollama add it!
Ollama uses the llama.cpp code-base for inference. And llama.cpp doesn't use MLX.
I admit, that I did not benchmark MLX via llama.cpp recently, I have to look into it. MLX in my understanding can use GGUF-files, but only limited quantization variants. Q4_0 seems a good compromise, but llama.cpp can do better.
This is such cool content!
I like this style of video and 100% want to see more, especially with continuously updated test as hardware is updated like an m4 device or if you get a mac studio or ultra, intel, etc.
Interesting analysis! Two things:
1) I believe heat is pretty much just the energy consumed. There's no chemistry involved, just physics, so aside from negligible light and sound and a small amount of work from spinning fans, a computer using 100 watts is essentially a 100W heater.
2) The Intel mini pc loses in speed and efficiency, but its real strength is the amount of RAM. You could run a 70b model with minimal quantization and get better output than either of the other machines is capable of (though quantifying that and putting it on a chart would be difficult).
Hey Alex. Thank your for this kind of comparison. Often I only see speed matrixes, this with average energy consumption, speed, initial cost and heat generated was exactly what I wanted to see. A full array with multiple tests. Thank you, I will definitely see more if you make more power/speeds/cost/Wcost/.... Comparison. ! Thank you.
Great assessment- so glad you're including power usage. Power and CO2 cost of AI is critical for people to consider going forward. (for the Earth and our grandchildren)
I like it , interesting to watch mate✌
This is top quality video I was looking for. Surprised that Apple silicon runs LLMs quite well…
this really helps me alot thaks you
Yep, we are waiting for 20A process node CPUs to be released! Please share the performance review as soon as you get your hands on it! Thank you Alex!
I love this channel.
I'm curious how the Mac Studio with M2 Ultra would compare with the RT 4090
Depending on the task, I think it would do pretty well. It won't be as fast on smaller models, but it will destroy the RTX on larger models.
@@AZisk It can run pretty large models, but the low speed makes it close to unusable for real time interaction.
@@brulsmurfI use 123B models on my M1 ultra 128GB. It can be as slow as 7 tokens per second, but I find that still usable for interactive chat. I’m more into quality than speed.
There is quite a lot of talk about this in r/localllama. For LLM work 4090 is faster than an M2 Ultra. You can get more memory on a M2 Ultra, but you can also get multiple 4090 (or perhaps even better, 3090) and put in a computer for typically less money (and better performance).
If you get over 90 GB requirements (4 xx90 cards) then neither model is really fast enough to be useful. (When you get to about 1 token per second then you're probably better off just renting a machine online or buying tokens from eg OpenAI.)
Would be really interesting to see a more comprehensive benchmark in price-tiers. Like @500-750 USD (NUC, maybe a M1 MBAir 8GB (often on sale) or Mac Mini M2 8GB) § @1000-1500 USD (Mac Mini M2Max, ARM-box, ???) § @2000-3000 USD (NUC+RTX4090, Mac Mini M2Max, Base Mac Studio) § @>3000 USD (Built PC with dual 3090 24GB, H100, Mac Studio Ultra)
I really enjoyed this kind of video. More stats and statistics would also be nice but I also know that it's time consuming
I am pretty sure you heard about asahi linux so will you make a video on it ?
it seems like he already did 2 years ago.. Did something changed?
@@ALCE-h7b
No they are still doing the what they were doing before but I suppose they took a big leap now with the release of their 'Asahi game playing toolkit' which in short makes playing AAA games on Mac very probable
Maybe if you wanna lean more, why not read their blog.
Maybe I need to revisit it. Cheers
Really nice video tx
What about performing the test on the Qualcomm dev kit machine? Interesting to see how the snapdragon performs
Well, yes, of course we want to know how the 200H core ultra series will perform. Have you heard any news about when Asus is releasing the Nuc 14 AI? I haven't seen any public information yet.
How about on the Snapdragon CPUs? Do they have Ollama running natively on those yet? I'm guessing not, but it would be interesting to see how they match up against the Mac hardware.
Asking the real question!
There's plenty of frontends on Android(Layla, Pocket Pal, chatterAI, MLCChat, etc) or you can run ollama through Termux if you want. Snapdragons do pretty well on phones, considering they're also a phone.
About 2.5-3 tokens/sec on Snapdragon 695 on Llama 3.2 8B parameter model (~$200USD phone, ala Motorola g84. Slow memory, slow CPU, 12GB RAM. About 5 watts usage).
About 12-22 tokens/sec+ on Snapdragon 8 gen 3 (~$600-700USD phone, ala OnePlus Ace 3 Pro. Good memory, good CPU, 24GB RAM, so can do bigger models. About 17.5 Watts usage).
So, they're not bad, but they're nowhere near a decent desktop CPU/ GPU/ or Mac integrated RAM thingy. But they can run them, to vaguely "usable" speeds (smaller models go quite a bit quicker, and are getting pretty good too. There's Llama 4B Magnum and Qwen 2.5 3B that will double and a bit those speeds, especially using ARM optimized versions. They're not "super smart/ knowledgeable", but they're good enough for entertainment purposes).
I'm surprised the the Oculink doesn't create more of a performance hit than it did. My 4090 plugged directly into the Mobo gets about 145 tokens/sec on llama3.1:8b verses the ~130 tokens/sec that you got. Kind of make sense since the model is first loaded into memory on the GPU.
Token-generation during inference is largely memory-bandwidth bound (OK, with some minimal impact for quantization-calculations) - Intel 1TB/s. And the LLM runs entirely on the 4090. The 4090 really shines during (batched) prompt-processing, blowing the Intel/Apple machine away - probably >20x faster than the M2 Pro, and way, way more so than the Intel CPU/GPU.
I've got an 4090 for my gaming PC, and a good old trusty lenovo thinkpad for work stuff. Just recently jumping on this LLM bandwagon, installed the LMStudio on my Windows desktop and exposed the LLM as Rest API, I think I have been badly spoiled by this having been my first experience running LLMs. Also now I can say: "It's for scientific purposes" when purchasing expensive GPUs lol
@Alex for me it is even more interesting how much longer the Mac Mini will take on a finetuning session than a 4090 - only inference is a bit boring
at 15:10 you show a document that says both that M2 16GB costs $0.11 per day and $.03 per day, and yearly costs are $40 and $12 per day. I looked closer because 3 cents per day seems suspisouly low.
You assumed that the wattage was constant, which it likely isn't. Your outlet meter can measure the total energy usage.
Excellent comparison! For me, an interesting future comparison would be between the RTX setup and an equally priced Mac Studio. And then re-compare after the new M4 Pro/Max is available (hopefully next week?).
Great test!
I don't think speed is all that important when you're running it locally. The most important thing is the memory and the bandwidth between the memory and the processing.
a Mac with MLX backend can go even faster.
For an AI server, once the model is loaded into vram, you probably want it to remain there for repeated inferences. So it depends on how you set things up.
Also if you want apples to apples determinism, reduce the temperature at inference time.
cant wait to see how the new M4 variety fares in a few weeks!
Cool video!
Thanks!
5:38 If you want to compare energy consumption, watt is not a good measure, watt hour is.
Speed and accuracy is what matters. Screw the extra power usage because you aren't running an LLM 24/7 you are querying it for a combined 5 minutes per day at most even if you are using it for hours during your work day. Your power usage is only spiking during actual generations which is seconds long on the 4090. The only real exception is running super big slow models for long periods of time or using AI agents which currently give crap results.
Thanks for the riview 🥰🥰🥰
The test is also flawed by the fact that an LLM can't really "count" its own tokens, so when you ask for a 1000 essay, it's just going to to a best effort, but results might be wildly different. Setting a token cutoff and having all machines reach the cutoff would be much more accurate.
Cool video :)
What about the results? Did you like those ai generated stories?
Now getting the m4 mini and m4 mini pros in this review is what I need 😂
Waiting impatiently to see if a mini pro with 64gb is worth it for llms !
I’m curious to see how it would run on Mac Studio. At $2000, it’s closer to the higher end one you shared.
I ended up with a used studio M1 Ultra with 128GB RAM. I can run any model up to 96GB RAM, and often do. It could be faster, but the important thing is I have few limitations for large-ish models and context. What really competes with this?
what u think about the new mini pro could be pretty good no?
12:00 Are you sure? Normally ollama will keep the model in memory for a few minutes. So if you do something like coding, the model should already be in VRAM for most queries.
The default is just 5 seconds, but if you increase that it might affect your cost substantially. Trade off.
Great! I myself test a lot. Could you make a comparison of cloud based energy consumption and the effect on our planet and local LLM?
Would like to see how lunar lake performs
nice video thank you, now that you have some M4 mini did you run the benchmark on it to compared with the M2 from that video ?
-verbose
please use this option, it’s easier than Schwarzenegger 2.0
but way less fun
Man things look good for future
If the model can fit in 16gb, the nVidia 4060ti/16gb is a solid performer, you should give it a try. That card can even run Codestral/Mistral Small at 4bit quantization at decent speeds (My old 3060 12gb was just a couple gigs short on vram)
Can you compare some AMD GPUs in these tests? They're significantly cheaper especially for the extra RAM. So I'm curious if the 'less performant' AMD GPUs can do the same as a 4090 or a lesser Nvidia GPU or a linear result of RAM= performance.. like a 20-24g AMD vs 16g Nvidia?
What about using a real motherboard with full PCI-E GEN 5 and DDR5 RAM? Where is the bottleneck?
Is there a limit or benefit to doing tons of RAM to either not bother with a GPU or to supplement a GPU? How can you use both the RAM & GPU?
Lastly, VERY curious about the latest AMD and Intel CPUs coming that are supposed to be more power efficient and 'built for AI' 🍻
thank you. i do have some amds i was planning to check out for this. not sure it will do much better than intel igpus
Have you thought about running such tests on the new AMD 8700 on tiny PCs? Heard that they have pretty good iGPU
cool, but where is AMD Ryzen 7 ?
now that would be interesting with the entry level m4 mac mini
Have you looked into Bitnet at all? It's Microsoft's new inference framework for 1 bit LLMs. People running like 70 billion parameters at home. Fun stuff.
Would be great to see AMD in the benchmarks both with CPU and external GPU.
With the 3.2 models, it should be quite fun
Didn’t you add a lot of memory to the Intel Box? Was that in your cost of ownership?
I did, but in this case, the model I was showing wasn't using all that memory and could easily run on the 32GB that the machine came with.
@@AZisk But if you had that box, you probably wouldn't be able to resist adding 64GBs of memory 🤣