@@AZisk Go for the Executive tier at $130. If you don't earn enough from the 2% cash back to make up for the extra cost over the $65 Gold tier, Costco will refund the difference at the end of the year. Also factor in the extra year of warranty Costco gives...
Thank you Alex, I was thinking of doing this exact project! One note: by using a hub you are creating a star topology with a 40Gbps bottleneck shared between all machines. If you used a partially meshed ring topology, you could connect each mini to 3 other minis with a connectivity set of 1:{5,2,3} 2:{1,3,4} 3:{2,4,1} 4:{3,5,2} 5:{4,1,2}. I'd be interesting in seeing if this improved performance. Another potential advantage of the mini cluster vs a single M4 Max is that all M4-series chips have the same 16 ANE cores; you might be able to run distributed inference on the neural engine to benefit from that scaling.
thanks a lot! definitely something to consider. however, the 40Gbps is per thunderbolt controller, and I thought that each port had a separate controller - they do not. Very good observation!!
@@AZisk Ahh I had been wondering why it wasn't just daisy chained, bc that was one of the things Thunderbolt was sold on originally. Would only use 1 port each, but yeah if they. would all share the same bandwidth, that might not work out like we wanted.
The EXO cluster can only expand available memory, not improve inference performance, because each token needs to be processed sequentially on every Mac mini. For a single request, parallel processing is not possible. For example, if you have one Mac mini with 64GB of RAM, its processing performance would be better than two Mac minis with 32GB each.
@dsfsgsgxx unrealistic? Some companies are already doing it, hence why some people had issue with the power button being switched, as some companies have server racks with the Mac minis.
Not a bad idea. I think Mac OS Server failed because of timing. Apple was not as popular with the Dev community as today. The Mac Mini is probably the most popular device for DevOps. Even AWS bought 1000s and rack them. (Yep you can run Mac OS in AWS). It should not be too complicated for them to use the MacPro and make it rackable. 1U unit, shallow depth, with necessary ports to create clusters. I think Alex is onto something. Clusters make sense, they provide redundancy. I hope Apple is watching and paying attention. Alex what would be interesting is the application side. What could these rackable Mac Mini be used for apart from running LLM test? The footprint and low power consumption is great. Need to find the market for it. Any ideas?
@@franckleveneur676Funny thing is that Mac Pro already has a 4U Rack Version... Just some Software and Ports stuff really I think... But Smaller Rackmacs would also be awesome
For the bottom mac being hotter, try giving it space under it like you have the others above, might be heat soaking because it cant dissipate the heat like the others can
Exolabs actually processes the parts of the model segmented onto each Mac sequentially, not in parallel which means it's slower than it would be running it on one machine with a lot of ram due to connection delay. However, if Exolabs supported Mixture of Experts models and allowed the experts to be split between devices, that would give insane performance when using all experts compared to doing it on one device because all the experts could be run in parallel.
@blisphul8084 Yes, but even if you only have 8 computers, you can still use the model with two experts per computer (if you have enough ram). You could also just elect to use fewer experts, and the result wouldn't be that much lower quality. You could also have all the experts loaded onto separate computers and still only actively use a couple of experts per token if you want to use less electricity. To clarify, "using fewer experts" only refers to how many experts are activated at a time for each individual token, MoE models still generally use all the experts across multiple tokens, so you need them all loaded somewhere.
@@blisphul8084 Not as much, MoE are already fast because they are sparse models and do not activate all their weights in every run, so they run pretty fast in a single computer already. The problem is fitting them into memory because all the weights have to be on memory (you do not know in advance which will be used). For example a 8x7b mixtral has the speed of 14b model but needs as much RAM as a 56b model. Maybe you could bring it down to 7b speed if all this worked like that, but it would be a 2x increase in speed not 8x. But such a set up would allow for running bigger models that would not fit into RAM normally.
To add onto this: Evaluating the performance of distributed systems is hard. Pipeline Parallel inference: This involves processing one layer on one device, and the next layer on the next, and so on. It’s probably the most intuitive form of parallelism, and means that your system’s RAM is essentially the sum of all constituent components. Note, that your total token/s will be equal to either the slowest machine, or the network bandwidth (whichever comes first). Asynchronous (Batched) Parallel Inference (Exo goes here): This is like PPI, but you are able to execute asynchronously, which lets you add together the bandwidth of your systems, potentially using batches, which lets you increase your token/s by essentially adding together your device’s compute, but it does incur a latency penalty, and each additional device makes the latency worse. Pretty good if you have a huge batch of prompts to go through or something. Lower bandwidth between devices doesn’t decrease token/s but does worsen latency, to my knowledge. Tensor Parallel: Matrix multiplies aren’t interdependent. What this means is that if I calculate the product of the first, third, fifth, and seventh rows on one device, I can calculate the second, fourth, sixth, and eighth on another device, and then just synchronize the product at the end. There’s a bit more to it (you can optimize communications by not synchronizing and actually just sending the products directly to the next layer), but that’s the basics. This requires less bandwidth than pipeline parallelism for LLMs at least (once the weights are loaded, at least), and does let you add together the bandwidth, capacity, and compute of the constituent devices, but it does have two issues. 1) You need a power of 2 number of devices for most tensor parallel implementations, and you also run into issues with certain calculations, like soft max, which require synchronization, which limits how wide a tensor parallel setup can go. It does improve latency, and improves throughput, though, so it’s pretty based. If you have two GPUs, you could run a model literally just twice as fast as not doing tensor parallel, and the same goes for CPUs. I think beyond 4 devices you have to think pretty carefully about how fast the interconnect is, though. Linear Transformers scale way better, so hypothetically if you had like, 64 CPUs for some reason you could probably get pretty close to linear performance improvements for each CPU in the network. Asymmetrical tensor parallel: I’ve never seen this, but in theory it’s possible. You don’t necessarily have to split the tensors evenly between devices. If you have a slower device (ie: CPU) you should still be able to offload, say, 10-20% of the model to that device, while doing the rest on GPU, which doesn’t sound great, but keep in mind it gives you a 10-20% speed up compared to just GPU, and it also reduces the VRAM required by that GPU. This means that you could probably run a 28GB model on a 24GB GPU, for instance, and run it a little bit faster than if you had the same GPU with 28GB of VRAM. Neat. Hybrid: If you have a tensor parallel setup configured as one device and then a second step in the pipeline going to another device, this would be a hybrid setup. To my knowledge no frameworks support this out of the box and would require some careful setup that would probably be very custom to your specific topology. You would expect the latency characteristics to be roughly equal to the ratio of tensor parallel and pipeline parallel components in the network. I guess it would make the most sense if you’re trying to pair several small devices (ie: 4 Raspberry Pi’s) with a more powerful device like a GPU. I’m guessing there’s probably other types of parallelism out there, and there’s also probably some learned parallelism types that might substitute elements of the communication with learned neural networks, or such, but to my knowledge those are the main types for Transformers. CNNs are where parallelism gets **really** fun, though.
A Mac Mini cluster with MLX and ExoLabs makes sense if you like extending context for the models. If you just need the model for a "Hello" or "Write a story" query, a single machine would be sufficient for the task.
I think the argument is that mini cluster with MLX and ExoLabs would not increase inference speed because inference are still done sequentially? But its definitely make running super big models at least possible with the scalability of memory resources
This kind of investigation is great and truly valuable. It’s a significant contribution to the community. This could become a rabbit hole once you start running tests. Thanks for sharing!
I’ve always imagined trying LLM clustering using a Mac, and you’ve turned that imagination into reality-I’m so thrilled! Additionally, if there are any Thunderbolt network issues, it would be great to create a video showcasing how to use a 10G network instead. This video is incredibly valuable and practical. Thank you so much for your hard work in making it. I truly appreciate it! ❤❤❤
CUDA clusters have 'unified' memory between GPUs thanks BlueField DPUs, so one GPU could be connected to 1TB of memory easily. It's important to have all GPUs working ON THE SAME MEMORY during training. Not all models are compatible with clusters & separate memory spaces.
Correct me if I am wrong. According to your video: 1.If the LLM model can fit in just one Mac mini pro, we will get maximum tokens per second. Adding more Mac minis actually decreases TPS, rather than increasing it. 2.If the LLM model is too big for a single Mac mini, the only way to run the model is by using an exo cluster. However, in this case, the TPS will be very low. So I am wondering wouldn't it be better to buy a MacBook Pro M4 Max with 128GB RAM instead of 5 Mac minis? It might be cheaper, and the performance would be much better than an exo cluster with Mac minis (maybe 2 or 3 times on TPS?) no offense, just want to understand
Can you run more tests and show a chart with the results per $ spent? We are trying to figure out if it makes more sense to buy 4 base models in a cluster vs spending the same amount on a maxed out m4 pro or max.
If you’d like a quick overview that will be roughly correct but without a nice looking chart: Apple products are pretty well balanced between Compute, and Memory Bandwidth for the purposes of LLMs. In other words, you can basically look at the memory bandwidth per second, divide that by the cost, and that gets you your price to performance. You can test it, but it will match up quite closely with tokens per second. If your goal is to get the most tokens per second, that’s fine, but there’s other considerations. If you need to run larger models and don’t really care as much about the speed (ie: You need to ask an oracle a really important question every now and then), your best price to performance will be consumer PCs with maxed out RAM configs, running on CPUs. If you need to run at the most tokens per second, then usually GPUs or specialized accelerators (Tenstorrent) will give you orders of magnitude better performance. I will note that a hybrid approach is possible where the CPU runs the oracle that does the overall planning and analysis while the smaller fast models essentially take the “monkeys and Shakespeare” approach and just generate quickly until they get it. This works surprisingly well, don’t knock it until you’ve tried it. The final case is where you need reasonably fast tokens per second in the largest possible model. This is where Apple’s Silicon makes sense, but do keep in mind it won’t be cheap no matter how you slice it. The exact best deal will depend heavily on how much RAM you need for your model specifically, and there’s multiple points of overlap. Again, your best shot is to take the memory capacity you need for your model, and to measure that against every possible config that can run it, and then divide that against the memory bandwidth to cost ratio of each valid configuration. Rather awkwardly, you’ll run into a slightly different answer for every model size.
Should totally be possible, the config might require some clever hackishness though, but then you might even get link aggregation to work and include WiFI and BT just for the halibut.
Brilliant Alex!!! Thank you for doing this! To be expected that the thunderbolt hub slows things down.... too much contention. I think we will have to wait for M4 ULTRA with 128/192Gb. Maybe worthwhile to run 2x M4 Ultra using EXO.
Thank you for sharing the cluster idea Alex. I was excited just watching you try the different configurations! Definitely interested in any further networking experiments. Thanks again!
Awesome bit of bench work Alex. You always amaze me how you make things can be hard to unpack and make them wholly engaging and great opportunities for learning. Keep. It. Up.
It's interesting that Exo stunts performance over MLX so much but is more convenient to use. If you were going to get a mini to supplement your 128GB M4 Max MacBook, would you just go for the base model mini at 24GB? At $800 its 60% cheaper than the 4090.
Hi Alex. I running a 4 xMacBookPro 2019 Corei9 cluster in my lab (hosting each a VMware Fusion for running multiple Cloudian HyperStore VM´s). When you create a virtual thunderbolt bridge on each Mac mini, then you don´t need any external thunderbolt hub to communicate with each Mac mini. just daisy-chain all Macs with a thunderbolt cable, but don't close the ring. This will lead to a very high CPU load when each machine will each other over two different pathes.. So just Mac1 using TB-cable to connect to Mac2 using another TB-cable to Mac3 and so on. That´s working fine to my use case and should work to your as well.
i think this will be slower, lets say machine A wants to communicate with machine E then the information has to go through B, C and D. Correct me if I'm wrong
Very cool Alex! It would be very interesting to see the same topology with a Mac Studio or Pro, M2 Ultra be the hub of the cluster, with it's 800 GB/S bandwidth and eight Thunderbolt 4 ports. Great content!
i tried exo with 2 macs and found out that it splits the model over the two devices and only downloads half on each. I noticed this after closing down one of my machines and the other one needed to download 50% of the model before it could continue. Great videos!
Hi Alex, thanks for the video! I think the idea of machine learning on parallelised apple silicon is really interesting. I’d like to see you redo the tests but this time repeating the learning/relearning phase. I think that you’ll get much better parallel performance due to the way that batch updates work. Of course that’s just a guess. Anyway, thanks again for the video.
The Jumbo packets are more about using less bandwidth for protocol overhead. The TCP/IP header (20 - 40 Bytes), then the payload. So it's saving about 84 bytes if you are filling every jumbo packet. Great for sending a lot of data quickly. At 40Gb/s and assuming we get all of the pipe, you are saving like ~40MB/s in TCP packet overhead. Nothing to sneeze at.
@@Makronauta The MTU is the max transmission size. If the frame isn't full, it doesn't actually transmit a bunch of white space nulls. It only sends what it needs to. Having a constant MTU size is more about not requiring the edge router / switch from having to convert from 9000 MTU size, and splitting them up into 1500 MTU size packets costing CPU time. If you send a 9000 sized MTU packet and it's less than a 1500 sized packet in the payload, it will not need to spit the packet.
Very interesting video and set up. Maybe the bottom Mac runs hotter because if doesn't have the same vertical space to push out air as the suspended macs above have.
I saw this and will try it this week on my cluster, thank you for showing this I started a business based on building these small clusters for businesses I asked my AI how to connect via Thunderbolt and this is what it said for hubs Install a third-party software like Sonnet's eGFX Breakaway Puck or Razer's Core X Chroma (if using TB3 docks).
I am a PhD student conducting research in computer vision. I would like to ask about the performance of using a Mac Mini cluster for computer vision tasks such as diffusion, object detection, and image segmentation. The video mainly demonstrates inference. If it were used for training, how would its performance compare to NVIDIA GPUs? Additionally, I noticed that MLX also has code applications for computer vision.
The good thing though about exo is that you can plug in just whatever of the new ecosystem. So if you have the m4 128 that needs more you can provide that through some n-tuple of mac minis. Or you have multiple workstations for an office and your need for intensive local inference is rare. Or in future when we all unnecessarily buy M5 devices 😢
Try setting up the cluster in a hierarchical scheme, I've had experience with distributed computing and while this adds CPU overhead, model workloads are primarily GPU, so it doesn't affect running the models. One master node, 3 Sub-Master Nodes, followed by upto 3 worker nodes each per sub-master node. I want to see what kind of workloads it can accomplish
A few years ago I myself personally believed in this idea, I dreamed of the concept of a mini PC with very powerful APUs that (ideally) will be in the form of cells that can be connected to each other and like honeycomb and thus automatically create clusters and having collected a whole cabinet of such you can organize a home computing lab. I think the M4 is the first step towards such a future because of its balance of performance and very small size. Thank you for the video (I got a question, if you compare this computing to the analog in the form of GPU, how much would it be in price?)
Fun fact, WiFi ethernet packets are up to 64kB, by default, with my 2020 wifi 6 router. When streaming videos with DLNA protocol it uses maximum length WiFi ethernet packets.
Hot air go on top,my guess is that the bottom machine is the hottest more for an inefficiency over the distributions of the load from the software plus certain tasks require to be running on single core. So probably an M4 put on the bottom to replace the M4 Pro position would result in the hottest too,maybe just a little less than the M4 Pro.
Your bottom Mac mini is hot because it is supplying power through thunderbolt connections to other Mac minies. Test temperature of each Mac mini and you may see difference. If each Mac mini is powered separately, that may. Not be the case with each Mac. Also, Mac mini with pro chip (and more RAM and SSD storage may have slightly more temperature due to more hardware that remain active.
Thanks Alex. You're asking - and answering - the big questions that are at the edge of everyone's worry & interest. These are the things that SHOULD be obvious, or obvious questions. The elephant in the room! The current and future Macs might be incredible game changers or *just* incredible consumer desktops, but either answer feels super important to answer.
It would be really interesting to see the comparison between Exo and flat MLX with remote hosts to see how the overhead that we observed in a single machine translates!
The main issue I have right now is that most libraries, papers, or any production models, like SAM, generative 3d geometry, etc, they are all built for CUDA, at work we try to explore every model, but when we try to use mac the answer most of the time is to "fall back to CPU" which makes it unusable and slow, LLM stuff is like hello world at this point lol, really cool to see your experiments :)
Jumbo frames reduce networking overhead. Every packet has a bunch of header information associated with it. If you enable jumbo frames, you get more data payload per packet. So, you can send a bunch more information per packet. Fewer packets = less header overhead = better overall transfer speeds for large quantities of information.
I think another part of the reason the lowest unit is the hottest is that it is closest to the table. It has the least airflow underneath and I also suspect the metal chassis is absorbing more heat here.
I'd love to see how Blender rendering works on such a stack! I was considering getting a couple (or more) base model Minis to offload Blender rendering.
There is another trick you can use to increase network bandwidth, Thuntherboldt 4 has huge packet size, not too usefull if the system just need to send an ack or nak. You may see a network improvements if you also allow for ehternet communications between the machines. Small packets can be sent over ethernet, where largeones can be sent over thunderbolt.
I have a Data Modeling and Internet Programming course this coming spring semester. I think this would be a great project for the students. Once a cluster is setup, can one machine be setup to be a web server so a private network of users can ask questions. Would love to train a cluster model for my CS department.
For reference 1B performance on exllama2 engine is like 700 t/s; and if you fit the 70B on a single 3090 it's 33 t/s (for a 70B at 2.5bpw). That GPU costs slightly more than a base M4 Mac Mini tho.
Have you ever tried running LINPAK, or whatever TOP500 is running for benchmarking supercomputers? I've been away from this for a decade but I see LLNL again has the top supercomputer with the next two also being DOE facilities. It would be interesting to see what a base M4, M4 Pro and M4 Max Mac would do. I saw one of your screens showing >17Tflops and I'm wondering how fast a single M4 can run along with how fast a cluster of M4s can run.
The rtx a6000 Ada has 48GB. It’s pricey ($5-6k) but way less than an a100 or h100. Uses the fully enabled die (18k “cuda cores” vs 16k on 4090) which would be another option. Plus I think power limit is 250 rather than 450w on the 4090.
It's interesting to me that despite the relative big leaps in max specifications each generation, theres already demand for more power. I hope Apple would consider some sort of monster server setup which multiple 128/192 m4 ultras in. although they'd need to work out how to up the bandwidth considerably. But of course, it's possible.
I run Qwen 2.5 Coder 32B on my macbook pro without a cluster and its extremely fast and doesnt seem to be an issue at all. I just use run it with MLX 4bit. On a m3 Max 48gb, 16-Core CPU, 40-Core GPU.
The M4 Mac Mini is discounted NOW: ⚡ amzn.to/3CJnjds
Costco has them for $500
@@vinfasi don’t have a Costco membership. How much is that?
@@AZisk $65 in the US. BTW, Costco has the M4/24/512 model for $889.
@@AZisk Go for the Executive tier at $130. If you don't earn enough from the 2% cash back to make up for the extra cost over the $65 Gold tier, Costco will refund the difference at the end of the year.
Also factor in the extra year of warranty Costco gives...
Not true, backordered is way different than discontinued. They’re just backordered now due to popularity.
2000$ in computers, 500$ in Thunderbolt cables.
Thank you Alex, I was thinking of doing this exact project! One note: by using a hub you are creating a star topology with a 40Gbps bottleneck shared between all machines. If you used a partially meshed ring topology, you could connect each mini to 3 other minis with a connectivity set of 1:{5,2,3} 2:{1,3,4} 3:{2,4,1} 4:{3,5,2} 5:{4,1,2}. I'd be interesting in seeing if this improved performance. Another potential advantage of the mini cluster vs a single M4 Max is that all M4-series chips have the same 16 ANE cores; you might be able to run distributed inference on the neural engine to benefit from that scaling.
thanks a lot! definitely something to consider. however, the 40Gbps is per thunderbolt controller, and I thought that each port had a separate controller - they do not. Very good observation!!
@@AZisk Ahh I had been wondering why it wasn't just daisy chained, bc that was one of the things Thunderbolt was sold on originally. Would only use 1 port each, but yeah if they. would all share the same bandwidth, that might not work out like we wanted.
I'd be super interested in seeing this
Hehehehe this brought me back to the old times where I come from... old 1 mbit network setups... damn at that time it was amazing...
The EXO cluster can only expand available memory, not improve inference performance, because each token needs to be processed sequentially on every Mac mini. For a single request, parallel processing is not possible. For example, if you have one Mac mini with 64GB of RAM, its processing performance would be better than two Mac minis with 32GB each.
MacOS servers should be a thing again with M model chips honestly
for sure
just give me a good linux integration
@dsfsgsgxx unrealistic? Some companies are already doing it, hence why some people had issue with the power button being switched, as some companies have server racks with the Mac minis.
Not a bad idea. I think Mac OS Server failed because of timing. Apple was not as popular with the Dev community as today. The Mac Mini is probably the most popular device for DevOps. Even AWS bought 1000s and rack them. (Yep you can run Mac OS in AWS).
It should not be too complicated for them to use the MacPro and make it rackable. 1U unit, shallow depth, with necessary ports to create clusters.
I think Alex is onto something. Clusters make sense, they provide redundancy.
I hope Apple is watching and paying attention. Alex what would be interesting is the application side.
What could these rackable Mac Mini be used for apart from running LLM test?
The footprint and low power consumption is great. Need to find the market for it. Any ideas?
@@franckleveneur676Funny thing is that Mac Pro already has a 4U Rack Version... Just some Software and Ports stuff really I think... But Smaller Rackmacs would also be awesome
For the bottom mac being hotter, try giving it space under it like you have the others above, might be heat soaking because it cant dissipate the heat like the others can
According to the Apple video, the air flows in and out of the bottom, so this makes sense.
Or just lay the rack on its side?
@@rp_npc This is the way. Also what MacStadium does as well.
Exolabs actually processes the parts of the model segmented onto each Mac sequentially, not in parallel which means it's slower than it would be running it on one machine with a lot of ram due to connection delay. However, if Exolabs supported Mixture of Experts models and allowed the experts to be split between devices, that would give insane performance when using all experts compared to doing it on one device because all the experts could be run in parallel.
So essentially, if you have a 16 expert model, you can get speed ups until you hit 16 computers?
@blisphul8084 Yes, but even if you only have 8 computers, you can still use the model with two experts per computer (if you have enough ram). You could also just elect to use fewer experts, and the result wouldn't be that much lower quality. You could also have all the experts loaded onto separate computers and still only actively use a couple of experts per token if you want to use less electricity. To clarify, "using fewer experts" only refers to how many experts are activated at a time for each individual token, MoE models still generally use all the experts across multiple tokens, so you need them all loaded somewhere.
Except for training, aren’t most tasks sequence dependent?
@@blisphul8084 Not as much, MoE are already fast because they are sparse models and do not activate all their weights in every run, so they run pretty fast in a single computer already. The problem is fitting them into memory because all the weights have to be on memory (you do not know in advance which will be used). For example a 8x7b mixtral has the speed of 14b model but needs as much RAM as a 56b model. Maybe you could bring it down to 7b speed if all this worked like that, but it would be a 2x increase in speed not 8x. But such a set up would allow for running bigger models that would not fit into RAM normally.
To add onto this: Evaluating the performance of distributed systems is hard.
Pipeline Parallel inference: This involves processing one layer on one device, and the next layer on the next, and so on. It’s probably the most intuitive form of parallelism, and means that your system’s RAM is essentially the sum of all constituent components. Note, that your total token/s will be equal to either the slowest machine, or the network bandwidth (whichever comes first).
Asynchronous (Batched) Parallel Inference (Exo goes here): This is like PPI, but you are able to execute asynchronously, which lets you add together the bandwidth of your systems, potentially using batches, which lets you increase your token/s by essentially adding together your device’s compute, but it does incur a latency penalty, and each additional device makes the latency worse. Pretty good if you have a huge batch of prompts to go through or something. Lower bandwidth between devices doesn’t decrease token/s but does worsen latency, to my knowledge.
Tensor Parallel: Matrix multiplies aren’t interdependent. What this means is that if I calculate the product of the first, third, fifth, and seventh rows on one device, I can calculate the second, fourth, sixth, and eighth on another device, and then just synchronize the product at the end. There’s a bit more to it (you can optimize communications by not synchronizing and actually just sending the products directly to the next layer), but that’s the basics. This requires less bandwidth than pipeline parallelism for LLMs at least (once the weights are loaded, at least), and does let you add together the bandwidth, capacity, and compute of the constituent devices, but it does have two issues. 1) You need a power of 2 number of devices for most tensor parallel implementations, and you also run into issues with certain calculations, like soft max, which require synchronization, which limits how wide a tensor parallel setup can go. It does improve latency, and improves throughput, though, so it’s pretty based. If you have two GPUs, you could run a model literally just twice as fast as not doing tensor parallel, and the same goes for CPUs. I think beyond 4 devices you have to think pretty carefully about how fast the interconnect is, though. Linear Transformers scale way better, so hypothetically if you had like, 64 CPUs for some reason you could probably get pretty close to linear performance improvements for each CPU in the network.
Asymmetrical tensor parallel: I’ve never seen this, but in theory it’s possible. You don’t necessarily have to split the tensors evenly between devices. If you have a slower device (ie: CPU) you should still be able to offload, say, 10-20% of the model to that device, while doing the rest on GPU, which doesn’t sound great, but keep in mind it gives you a 10-20% speed up compared to just GPU, and it also reduces the VRAM required by that GPU. This means that you could probably run a 28GB model on a 24GB GPU, for instance, and run it a little bit faster than if you had the same GPU with 28GB of VRAM. Neat.
Hybrid: If you have a tensor parallel setup configured as one device and then a second step in the pipeline going to another device, this would be a hybrid setup. To my knowledge no frameworks support this out of the box and would require some careful setup that would probably be very custom to your specific topology. You would expect the latency characteristics to be roughly equal to the ratio of tensor parallel and pipeline parallel components in the network. I guess it would make the most sense if you’re trying to pair several small devices (ie: 4 Raspberry Pi’s) with a more powerful device like a GPU.
I’m guessing there’s probably other types of parallelism out there, and there’s also probably some learned parallelism types that might substitute elements of the communication with learned neural networks, or such, but to my knowledge those are the main types for Transformers.
CNNs are where parallelism gets **really** fun, though.
I like the narrative style without getting too deep into the technical weeds and letting the screen do that talking.
Thanks
hope you enjoyed. thanks so much
A Mac Mini cluster with MLX and ExoLabs makes sense if you like extending context for the models.
If you just need the model for a "Hello" or "Write a story" query, a single machine would be sufficient for the task.
@@MrBlogbar but…. Speeeeeeeeeeeeeed
I think the argument is that mini cluster with MLX and ExoLabs would not increase inference speed because inference are still done sequentially? But its definitely make running super big models at least possible with the scalability of memory resources
Can’t wait for the MServe(Xserve)
Finally we get to see the results! You've hinted and showed the racked minis in many prior videos, I was going mad!
You could really see the joy on you face in this video. Like a kid in a candy store. :)
it was a fun one for sure
@@AZisk this video is a porn for geek!
Thank you for exploring this so expertly.
This kind of investigation is great and truly valuable. It’s a significant contribution to the community. This could become a rabbit hole once you start running tests. Thanks for sharing!
Thanks! Very interested in how this all plays out.
Glad you enjoyed it! and thanks a lot!
I’ve always imagined trying LLM clustering using a Mac, and you’ve turned that imagination into reality-I’m so thrilled! Additionally, if there are any Thunderbolt network issues, it would be great to create a video showcasing how to use a 10G network instead. This video is incredibly valuable and practical. Thank you so much for your hard work in making it. I truly appreciate it! ❤❤❤
CUDA clusters have 'unified' memory between GPUs thanks BlueField DPUs, so one GPU could be connected to 1TB of memory easily.
It's important to have all GPUs working ON THE SAME MEMORY during training.
Not all models are compatible with clusters & separate memory spaces.
Yeah pretty sure multiple NVidia cards connected with nvlink will be a lot better then this solution
Obviously but not all of us are millionaires
@ it doesn’t change the fact that most downloaded model can’t work in clusters
@@FuZZbaLLbee Of course, and as he said, an Nvidia H100 would be better, but this was a bang for the money kind of test.
@@BiP00 I was thinking multiple second hand 3090s connected via NVlink
Correct me if I am wrong. According to your video:
1.If the LLM model can fit in just one Mac mini pro, we will get maximum tokens per second. Adding more Mac minis actually decreases TPS, rather than increasing it.
2.If the LLM model is too big for a single Mac mini, the only way to run the model is by using an exo cluster. However, in this case, the TPS will be very low.
So I am wondering wouldn't it be better to buy a MacBook Pro M4 Max with 128GB RAM instead of 5 Mac minis? It might be cheaper, and the performance would be much better than an exo cluster with Mac minis (maybe 2 or 3 times on TPS?)
no offense, just want to understand
Isn't that what he says at the end of the video?
Its exactly what he says at the end of the video.
For batch size 1 (= one request at a time), yes. However, adding more Mac minis will improve TPS if you are dealing with higher batch size.
@@philippe.139 I see, parallel requests will be processed simultaneously, thanks
@philippe.139 a MacBook has better performance than a desktop MacPro? Or is it about performance per dollar?
Can you run more tests and show a chart with the results per $ spent? We are trying to figure out if it makes more sense to buy 4 base models in a cluster vs spending the same amount on a maxed out m4 pro or max.
If you’d like a quick overview that will be roughly correct but without a nice looking chart:
Apple products are pretty well balanced between Compute, and Memory Bandwidth for the purposes of LLMs. In other words, you can basically look at the memory bandwidth per second, divide that by the cost, and that gets you your price to performance. You can test it, but it will match up quite closely with tokens per second.
If your goal is to get the most tokens per second, that’s fine, but there’s other considerations.
If you need to run larger models and don’t really care as much about the speed (ie: You need to ask an oracle a really important question every now and then), your best price to performance will be consumer PCs with maxed out RAM configs, running on CPUs.
If you need to run at the most tokens per second, then usually GPUs or specialized accelerators (Tenstorrent) will give you orders of magnitude better performance. I will note that a hybrid approach is possible where the CPU runs the oracle that does the overall planning and analysis while the smaller fast models essentially take the “monkeys and Shakespeare” approach and just generate quickly until they get it. This works surprisingly well, don’t knock it until you’ve tried it.
The final case is where you need reasonably fast tokens per second in the largest possible model. This is where Apple’s Silicon makes sense, but do keep in mind it won’t be cheap no matter how you slice it. The exact best deal will depend heavily on how much RAM you need for your model specifically, and there’s multiple points of overlap. Again, your best shot is to take the memory capacity you need for your model, and to measure that against every possible config that can run it, and then divide that against the memory bandwidth to cost ratio of each valid configuration.
Rather awkwardly, you’ll run into a slightly different answer for every model size.
Booyy ooo boyyy ooo boyyyyy!! This is one heck of a video about M4 mac mini! Such a creative video man! Loved it!
thx!
I loved it too!
I was waiting for this video for so long, thanks Alex. ❤
me tooo
Me 3 😂
Can you daisy chain the thunderbolt connections? This would eliminate the hub.
I thought that way to love to know the answer
Should totally be possible, the config might require some clever hackishness though, but then you might even get link aggregation to work and include WiFI and BT just for the halibut.
@@mpsii surprised this wasn’t his solution. There’s precedent for this.
@@abb0tt good to know, I assumed you could daisy chain via Thunderbolt but did not know for sure
@@JeremyAndersonBoise google "High-speed 10Gbps full-mesh network"
Brilliant Alex!!! Thank you for doing this! To be expected that the thunderbolt hub slows things down.... too much contention. I think we will have to wait for M4 ULTRA with 128/192Gb. Maybe worthwhile to run 2x M4 Ultra using EXO.
M4 Ultra will have 256GB. There's also supposed to be a chip coming that'll supersede the M4 Ultra with 512GB of memory.
omg, that power draw is actually amazing!
Thank you for sharing the cluster idea Alex. I was excited just watching you try the different configurations! Definitely interested in any further networking experiments. Thanks again!
Awesome bit of bench work Alex. You always amaze me how you make things can be hard to unpack and make them wholly engaging and great opportunities for learning. Keep. It. Up.
Macs became affordable option . what a time to be alive
Best video in regards of M chips and machine learning/LLM. Keep it up. I love those kind of videos!!!
It's interesting that Exo stunts performance over MLX so much but is more convenient to use. If you were going to get a mini to supplement your 128GB M4 Max MacBook, would you just go for the base model mini at 24GB? At $800 its 60% cheaper than the 4090.
Hi Alex. I running a 4 xMacBookPro 2019 Corei9 cluster in my lab (hosting each a VMware Fusion for running multiple Cloudian HyperStore VM´s). When you create a virtual thunderbolt bridge on each Mac mini, then you don´t need any external thunderbolt hub to communicate with each Mac mini. just daisy-chain all Macs with a thunderbolt cable, but don't close the ring. This will lead to a very high CPU load when each machine will each other over two different pathes.. So just Mac1 using TB-cable to connect to Mac2 using another TB-cable to Mac3 and so on. That´s working fine to my use case and should work to your as well.
If it's Thunderbolt Bridge, can you try daisy-chaining Thunderbolt connections to get rid of the bridge?
@@Alexthesurfer this 👍🏻
i think this will be slower, lets say machine A wants to communicate with machine E then the information has to go through B, C and D. Correct me if I'm wrong
@@zeeshanrabbani8125 you're not wrong
But star connect is possible with 4 macs. I think he mentioned that and I think that would get rid of the bridge.
I was thinking of the sane thing
Very cool Alex! It would be very interesting to see the same topology with a Mac Studio or Pro, M2 Ultra be the hub of the cluster, with it's 800 GB/S bandwidth and eight Thunderbolt 4 ports. Great content!
M4pro has a Thunderbolt 5
I just bought mine today! I connect it to my 100 inch Samsung smart TV mounted to my bedroom wall! It's awesome! I love it so much! 🥰
Another fantastic detailed video, knowing wattage of use, bandwidth and seeing results with IP addresses.
i tried exo with 2 macs and found out that it splits the model over the two devices and only downloads half on each. I noticed this after closing down one of my machines and the other one needed to download 50% of the model before it could continue. Great videos!
I was waiting for this one for weeks!!
Took me long enough, right?
I’m just happy it’s here 🎉
Great beginning! I don‘t think this setup is useless, but yes, a beginning.
16:44 lol - Elmo doesn’t pay his bills 😂
Facts!
This is fantastic! Really thorough overview - thanks!
Great demonstration of what stack of Mac Mini's is capable of.
Font rendering on Mac OS is so smooth!!!
Hi Alex, thanks for the video! I think the idea of machine learning on parallelised apple silicon is really interesting. I’d like to see you redo the tests but this time repeating the learning/relearning phase. I think that you’ll get much better parallel performance due to the way that batch updates work. Of course that’s just a guess.
Anyway, thanks again for the video.
The Jumbo packets are more about using less bandwidth for protocol overhead. The TCP/IP header (20 - 40 Bytes), then the payload. So it's saving about 84 bytes if you are filling every jumbo packet. Great for sending a lot of data quickly. At 40Gb/s and assuming we get all of the pipe, you are saving like ~40MB/s in TCP packet overhead. Nothing to sneeze at.
@@Makronauta The MTU is the max transmission size. If the frame isn't full, it doesn't actually transmit a bunch of white space nulls. It only sends what it needs to. Having a constant MTU size is more about not requiring the edge router / switch from having to convert from 9000 MTU size, and splitting them up into 1500 MTU size packets costing CPU time. If you send a 9000 sized MTU packet and it's less than a 1500 sized packet in the payload, it will not need to spit the packet.
Very interesting video and set up. Maybe the bottom Mac runs hotter because if doesn't have the same vertical space to push out air as the suspended macs above have.
the best review ever ! Thanks Alex.
is it possible to make a chart showing what is the best solution to run LLM with different budgets?
Impressive - wasn’t sure how performance would scale when clustered. Continue to be impressed with Apple Silicon.
your content is concise and well presented, but way over my technical head but still enjoyed the content. excellent job
thanks! I appreciate it! 😊
I saw this and will try it this week on my cluster, thank you for showing this I started a business based on building these small clusters for businesses
I asked my AI how to connect via Thunderbolt and this is what it said for hubs
Install a third-party software like Sonnet's eGFX Breakaway Puck or Razer's Core X Chroma (if using TB3 docks).
I am a PhD student conducting research in computer vision. I would like to ask about the performance of using a Mac Mini cluster for computer vision tasks such as diffusion, object detection, and image segmentation. The video mainly demonstrates inference. If it were used for training, how would its performance compare to NVIDIA GPUs? Additionally, I noticed that MLX also has code applications for computer vision.
I love your creativity and humour, I love your channel man!
The good thing though about exo is that you can plug in just whatever of the new ecosystem. So if you have the m4 128 that needs more you can provide that through some n-tuple of mac minis. Or you have multiple workstations for an office and your need for intensive local inference is rare. Or in future when we all unnecessarily buy M5 devices 😢
Try setting up the cluster in a hierarchical scheme, I've had experience with distributed computing and while this adds CPU overhead, model workloads are primarily GPU, so it doesn't affect running the models. One master node, 3 Sub-Master Nodes, followed by upto 3 worker nodes each per sub-master node. I want to see what kind of workloads it can accomplish
A few years ago I myself personally believed in this idea, I dreamed of the concept of a mini PC with very powerful APUs that (ideally) will be in the form of cells that can be connected to each other and like honeycomb and thus automatically create clusters and having collected a whole cabinet of such you can organize a home computing lab. I think the M4 is the first step towards such a future because of its balance of performance and very small size.
Thank you for the video
(I got a question, if you compare this computing to the analog in the form of GPU, how much would it be in price?)
Fun fact, WiFi ethernet packets are up to 64kB, by default, with my 2020 wifi 6 router. When streaming videos with DLNA protocol it uses maximum length WiFi ethernet packets.
thanks for this Alex. finnaly someone test this thank you
Hope you like it!
The bottom computer in the rack has less ventilation. Increase the space and see if you get cooler results.
This video, the explanation and the tests were so well done. First time on your channel, now I have subscribed. Thank you.
The size and power consumption for that compute is amazing.
Wonderful video, great job and thank you.
Glad you enjoyed it!
Hot air go on top,my guess is that the bottom machine is the hottest more for an inefficiency over the distributions of the load from the software plus certain tasks require to be running on single core.
So probably an M4 put on the bottom to replace the M4 Pro position would result in the hottest too,maybe just a little less than the M4 Pro.
Your bottom Mac mini is hot because it is supplying power through thunderbolt connections to other Mac minies. Test temperature of each Mac mini and you may see difference. If each Mac mini is powered separately, that may. Not be the case with each Mac. Also, Mac mini with pro chip (and more RAM and SSD storage may have slightly more temperature due to more hardware that remain active.
Thanks Alex. You're asking - and answering - the big questions that are at the edge of everyone's worry & interest. These are the things that SHOULD be obvious, or obvious questions. The elephant in the room! The current and future Macs might be incredible game changers or *just* incredible consumer desktops, but either answer feels super important to answer.
I feel like you have all the hardware here you would need for a killer agentic workflow setup. I'll be waiting on that video.
Not all the windows opened on the screen. Great 👌 video sir Alex
its not particularly about parallel processing but vector operations
It would be really interesting to see the comparison between Exo and flat MLX with remote hosts to see how the overhead that we observed in a single machine translates!
The main issue I have right now is that most libraries, papers, or any production models, like SAM, generative 3d geometry, etc, they are all built for CUDA, at work we try to explore every model, but when we try to use mac the answer most of the time is to "fall back to CPU" which makes it unusable and slow, LLM stuff is like hello world at this point lol, really cool to see your experiments :)
Lm studio on M1-M4 should work great
Jumbo frames reduce networking overhead. Every packet has a bunch of header information associated with it. If you enable jumbo frames, you get more data payload per packet. So, you can send a bunch more information per packet. Fewer packets = less header overhead = better overall transfer speeds for large quantities of information.
I loved this video! The fun experiments that tech nerds love!
I think another part of the reason the lowest unit is the hottest is that it is closest to the table. It has the least airflow underneath and I also suspect the metal chassis is absorbing more heat here.
Your videos are getting better and better Alex. This was very interesting and of high quality! Thx!
I'd love to see how Blender rendering works on such a stack! I was considering getting a couple (or more) base model Minis to offload Blender rendering.
I commented this on an earlier video, but please compare two 32GB minis running on exo vs. one 64gb mini. Because both setups cost the same amount.
There is another trick you can use to increase network bandwidth, Thuntherboldt 4 has huge packet size, not too usefull if the system just need to send an ack or nak. You may see a network improvements if you also allow for ehternet communications between the machines. Small packets can be sent over ethernet, where largeones can be sent over thunderbolt.
I do believe the heat from top devices and it being the hub is causing it to have a high temperature
Compared to the others, bottom M4 lacks an open floor, which likely affects air ventilation.
It would be interesting to see how it would work if each machine had internal 10Gb ethernet, connected through a 10Gb switch.
I have a Data Modeling and Internet Programming course this coming spring semester. I think this would be a great project for the students. Once a cluster is setup, can one machine be setup to be a web server so a private network of users can ask questions. Would love to train a cluster model for my CS department.
The thumbnail got me instantly, I used to run a grip of Intel Mac Minis at home.
For reference 1B performance on exllama2 engine is like 700 t/s; and if you fit the 70B on a single 3090 it's 33 t/s (for a 70B at 2.5bpw). That GPU costs slightly more than a base M4 Mac Mini tho.
You could try putting the cluster on it's side and/or place a fan on the back..
the vents on the bottom one is almost toughing the tables. The other ones have more space to blow hot air out.
Great video! Not sure I’ll buy multiple, but good to see the M4 Pro matches the dual M4.
This was really cool to see, I always wanted to see the M series chips pushed like this!
Have you ever tried running LINPAK, or whatever TOP500 is running for benchmarking supercomputers? I've been away from this for a decade but I see LLNL again has the top supercomputer with the next two also being DOE facilities. It would be interesting to see what a base M4, M4 Pro and M4 Max Mac would do. I saw one of your screens showing >17Tflops and I'm wondering how fast a single M4 can run along with how fast a cluster of M4s can run.
I would say as a test for heat. Take the bottom mini out of the rack and elevate it so air circulating around it and see what happens
The rtx a6000 Ada has 48GB. It’s pricey ($5-6k) but way less than an a100 or h100. Uses the fully enabled die (18k “cuda cores” vs 16k on 4090) which would be another option. Plus I think power limit is 250 rather than 450w on the 4090.
the thunderbolt dive would actually be really interesting
i hope you make the video at some point
Accidental Tech Podcast has some great deep dives on Thunderbolt and other Mac stuff starting from many years back
It's interesting to me that despite the relative big leaps in max specifications each generation, theres already demand for more power. I hope Apple would consider some sort of monster server setup which multiple 128/192 m4 ultras in. although they'd need to work out how to up the bandwidth considerably. But of course, it's possible.
I run Qwen 2.5 Coder 32B on my macbook pro without a cluster and its extremely fast and doesnt seem to be an issue at all. I just use run it with MLX 4bit. On a m3 Max 48gb, 16-Core CPU, 40-Core GPU.
If you're using jumbo packets, make sure to enable them everywhere on the network: all hosts and switches/routers.
Alex you are doing a great job. The best Mac-related channel for devs.
Great proof of principle! I anticipate many more RUclips videos on this topic. Would be interesting to see Mac Studio clusters also.
Very interesting project. I would be interested in seeing something where the minis are connected via the LAN port.
I would like to see the comparison of the system over 10gb lan and thunderbolt
Very good, thanks for your demonstration! This points towards a M4 Max Studio with 124 GB of RAM, for me! Thanks again Alex! :)
Great content. Congratulations!
i liked the video, but really could have used a summary table of the configs vs performance metrics. Thanks!
Bottom one is probably hotter because the vents are on the bottom and the desk restricts airflow
M4 Mac studio might be a true beast
Nice video alex! I feel that quantization is a more real use case, you could do 3 x Mini with a big model like Qwen 32B in 4Bits