Building a 4-node Raspberry Pi Cluster
HTML-код
- Опубликовано: 6 окт 2024
- How to build a Raspberry Pi cluster and manage it using Python.
=== Hardware ===
PiRacks kit: www.amazon.com...
=== Software ===
Raspbian: www.raspberryp...
Etcher: etcher.io/
Angry IP Scanner: angryip.org/dow...
Fabric: www.fabfile.org/
Music: www.bensound.com
Davy Wybiral
wybiral.github...
/ davywtf
"You don't have to use color coated cables like this but be aware that it won't look nearly as cool." classic
Dont be shy let your rainbow run,
@@marcsmithsonian9773 My flow of electrons run
@@WaffleTM420 and here you are very wrong my dear... Acirding to RUclips since youtubers set the laws of phisics there are no electrons flowing inside the cables but electricity is a magical radio wave.. propagating instantly.
@@marcsmithsonian9773 Woah, well hey thanks for letting me know, do you have discord by any chance?
@@WaffleTM420 yes there ate things called electron tubes, and it conducts based on the principle.of electron flow and.not a radio vawe.. and I have a Disco- Ball in my living room. The theories about electricity flow was a coordinated research of umiversities to validate degree of stupidity that can be induce by youtube to general population... results are catastrophic.
One Russian schoolboy also made a cluster of only 3 raspberries, and you know what? It was played on TV as the "Supercomputer Inventor." That’s why I haven’t been watching TV for 7 years
lol 3 is not that much for a super computer you need like a few thousand for that level and a few thousand would be much harder to pull off then just three in a cluster
I'm pausing 9 min in to say, this is a really great video. I just got my first pi, and am a little bit afraid of it, so I really appreciate the time you've put into making this and explaining things clearly.
Great video, thank you. I see a lot of comments about it not being a cluster, asking what it can be used for, etc. Let me see if I can help. First and foremost, it is a cluster; it is a cluster of Raspberry Pi's. Each will be and act as an independent Pi. What Davy is showing here is how to use fabric is to easily send commands to all at the same time. You don't need the case to do this; its just a nice way to house them.
Now take one of those Pi's and make it the master and the others slaves. The master receives commands (keyboard, switch, sensor, another computer, whatever) and sends those commands to slave Pi's to do the actual processing. Each Pi can be configured to do specific things. In a robot, you may have one Pi dedicated to differential steering and object avoidance, another for video input processing, etc.
Best comment here. It's the bottom bit I came here for, the video doesn't cover it
3:25 it sounds like he just ate the micro SD cards.
rooooooooooooofl!
CRUNCH
Oof lmao
Lmao
????
I'm glad I can just log into my current router and check the allocation of IP addresses on its LAN devices list quickly rather than having to scan my network. Most of my most recent routers have had a feature to list their DHCP assigned IP addresses in some form or other. I tend to also add devices like this onto a DHCP reservation list as well to make sure the hardware is bound to that IP address in the future by its MAC address. I have had problems in the past where I have shelved projects for a few weeks only to come back to them later and the IP addresses are different as their lease expired and the old adress has been given away to something else in the weeks they have been offline.
This is t really a “cluster”. It a collection of 4 individual Pis. A cluster is a system that allows for parallel computing.
+1
Could have made a legit beowulf cluster with linux easily enough.
The word "cluster" is not defined like that, so you just mad that it is not what you wanted it to be
@Samuel Self yes, but he did not do that either.. This is merely a small rack of Pi's. Disappointing....
but what does it actually do?
I like how you do videos: it just so much more genuine than what you find in many others.
nogosnoqt i heavily agree; my dude seamlessly described in depth a quite complex rpi project, made it funny and threw in some optional CLI powertools usage like its nada. ill press ‘f’ repeatedly!
It's a Bushel of RaspberryPi's not a cluster. Well not yet. Not until you set them up to be a cluster. Right now they are just a lonely bushel of RaspberryPi's in a nice stack.
"close your eyes" XD
Exactly. This video did not - sadly - explain what I came here for.
Yeah, cluster was the wrong word to use here.
@@patrik5123 It still had a couple of cool tricks that would come in handy when it comes time to actually set up a workflow.
@@PaulJohnsonM 100% irrelevant. And not even true.
You solved something that I was looking how to fix it for many hours, really thank you for giving a rich explanation about it. You are the man.
These four PIs still appears to be working independently. It will be useful if they can be combined and used as a single system. Then, we can have multiples cores to do parallel processing. Anyway even in the current state one can assign the job to different pi and share the workload. Thank you for the basic introduction of combining PIs.
it would require a new kernel extension or something like it. I dont think Pi will do real parallel processing out of the box.
I need to try this, as soon as I understand what kind of purpose it is for,
I'm going to use it to setup a training lab for clustering which is where several computers act as a single computer - opensource.com/article/20/6/kubernetes-raspberry-pi. I'm also going to use it to build out Ansible which basically will use one Pi as a master to run and schedule automations on the other 2 or 3 Pi's.
It is 8:17 in the morning and I haven’t slept yet and I’m watching this only because RUclips recommended it to me my life is meaningless
Also this video is pretty cool you got a new sub
Hey, I feel you man. Sometimes I dont get a wink of sleep, just watching youtube videos-we're all just specks of dust floating in an infinite void. If you wanna talk just leave a comment.
No, it isn't a cluster. But it is a good description of how to control multiple RPi computers. Turning his stack of Pi's into a parallel computer using OpenMPI or MPICH or MPI4PY is definitely a topic for another video, but this video gives a good description of how to control a stack of Pi's once you do have a genuine cluster. Give credit where credit is due!
It could be used as web servers, pen testing lab, install ubuntu and different OS meant for it together and try individually the OS, by either considering individual processing with different OS or parallel processing, try out Docker, Kubernetes, Ubuntu Server, Fedora server it's Security. Try to view it as different servers with different methodology, DHCP server on on, Web page on another, Security Server, etc and mimic a complete operational infrastructure that's seen in big huge companies. We could mimic all those properties within a cluster or individually, just install its dependencies. It's like a total Lab we have in front of us with which we could study, play, organize and learn different programming languages and such. Just consider it either as a stack/cluster which we use to learn or individual pi's in such a cradle.
Something like that little stacker sized up to fit Pi 8s with NVME bases / NVME drives would be pretty awesome...
Your title seems misleading. All you're showing is how to use a tool to configure 4 Pis at one time. It would have been cooler if you were configuring them to operate in some sort of actual cluster.
Nice. Great idea on using color coded cables. I have also seen people use usb cables with LED light, that is also looking very impressive.
Very cool. It's very cool that we have the technology now to build a small Linux cluster at a low cost. I really need to advance my skills for work in a pure lab environment and this will give me some tools to learn Kubernetes and docker.
I exptected him to say "Hey Vsauce, Davy here" in the beginning
What a fantastically well done video. Looking to build a video editing cluster from workstations I've got sitting around and know nothing about it. first video out of about 15 I could follow.
Fantastic video. The attention to detail is appreciated for a Raspberry Pi novice such as myself. Keep up the great work!
@8:39 "You don't have to use colour-coded cables like this, but, be aware that it won't look nearly as cool". :D LMAO. That was funny man. Thanks for posting.
Great video. Exactly what I have been looking for. Wish you record the screen so I can understand you commands. Thank you so much!
That Wack-A-Mole entrance was funny. Loved it.
Great video man! Very informative! And I like how you've given attention to certain small details, like the thing about using a USB hub with an external power source. Lol and the small bits of comedy are definitely a plus point!
Keep it up!
Can I ask a simple question?
- what cluster did you built here?
Ok got my point. So misleading title, please correct it. You just showed us how to update our systems but not done any clustering at all.
Making this today!! I'm only using 3 Rpi3 Model B's, but I'm super excited. Thanks for posting this video!!
Sooo to any future watchers of this video, the Fabric code referenced is now deprecated. It's honestly much easier to use Ansible to accomplish the same concept, but amazing video nonetheless! Definitely inspired me to tackle this
Nice video, you should have posted where you bought the cables, switch and usb power hub. And it would be nice for us newbies if you posted the fabric file but I'll keep pausing the video to see if I can figure it out. Thanks again Dave
8:46 - "You don't have to use color coded cables like this, but be aware that it won't look nearly as cool!" :) Thumbs up. This was maybe 10 seconds after I thought - "wow, these colored cables look cool!"
Not only does it look cool, but it helps you figure out which end goes to which CPU when you are troubleshooting your stack!
Hehe, he would have gotten it anyway, but that comment of his triggered my up vote.
Having worked as a network technician for a few years,,,yeah, anything you can do to simplify cabling pays off. Even if it's just 4 devices.
You'll never actually see the savings in things like time, frustration and hardware damage, but it's there. It's like not falling off a cliff that you don't know is there.
They are not for decoration they are to organize. Them
"...be aware that it won't look nearly as cool!" - excellent!
Thank you for teaching me how to do it. I really liked the video. You teach in a very clear way.
I've always wondered what use a Raspberry Pi cluster has. Does it actually increase performance or is it just easier to run a server or something, or do something like calculate a large number easier? (let's say a number like Pi.) Anyways these little things always seem to amaze me and are very cool and unique...
Yes, such a cluster would be very useful in scientific computing. A classic problem is generating all the prime numbers up to N (a sufficiently large number, like a million). Each node will work on a portion of the problem, and then the results can be merged later. This is extremely faster compared to running the same program on a single Pi.
Actually didn't know about Fabric! Nice one :)
This is very cool, and you made it seem super easy! However, I don't have any idea what one does with a cluster of Raspberry Pi, nor do i know what Python is for. Just grabbed my first Pi for audio streaming.
To send a command to multiple devices I suggest to use terminator and select "broadcast all" option
Thanks for explaining things super clearly, but yet in a concise way. For example "pip install" you explained that it's a command for installing Python programs. :) Also, I've thought about building a Pi cluster - thanks for the idea of using a USB charger hub!
Change your default "pi" usernames. Having a default username makes the pi less secure because an attacker is gonna try the username 'pi' because they know its the default username.
Very useful info. Thanks for taking the time to make this video!
Thanks for taking the time to put this video together. I learnt some new tricks :).
you should show the links to the Anker hub. All of the items you are using including the switch. You can then eventually go to these manufacturers and present your base and how you recommend their products, etc.
Thank you for this awesome tutorial, it helped me a lot. You're the Man! Quick question: what should I do in order to use this type of cluster as a home cloud for hosting my website(s) and mysql databases?
Great video, fruiend. I just received 5 of these little creatures to do exactly what you did. I failed to buy colored cables, so you with the "cool " factor. Keep the vids coming - I'm an IT grad student and learn more here than in class/books. Cheers, mate!
I really like the color of those wires, would anyone happen to know the brand?
Great video Davy Wybiral!! I never heard of the fabric program before.
is Fabric actually clustering them though ? Or is it more or less just a way to execute the same SSH command on different devices at the same time. The cluster would distribute data evenly across the pi's, and not the execute the same command but separate parts of the command simultaneously utilizing the full power of the "cluster". I get what you did here with Fabric and it was cool , but how can I use it to combine the hardware resources ?
It's only to execute the same SSH command on all devices.
Still a cool concept. I wound up going with a machine file so the pi's could actually see && speak to eachother. What are some practical applications for a cluster ? The first thing I tried was a bruteforce lol
Yes, Fabric is definitely cool. It's a fairly extensive tool and worth checking out.
I'm currently using my RPi cluster to prototype a small microservice-based infrastructure with multiple Node.js instances communicating with each other over IPC. Next would be using a message broker like RabbitMQ and making the RPis run on multiple different languages.
@Umbrelladroid Actually I am familiar with what a Beowulf cluster or grid computing is. Although I am familiar with the term, I have never built one. I know you can make a cluster that serves the function with mpi/mpich openMPI for arm devices. Have you tried either of these methods beowulf or mpi ? What I was looking to do was find an office that was getting all new computers and getting rid of the old ones in order to attempt making the beowulf cluster. Usually they're recycling or throwing machines away. I have a bunch of machines at home but they're not the same, in order to make this type of cluster, do the machines have to be identical ? I only have two rpi's, a3b+ and pi0w. I was thinking of trying this ruclips.net/video/RqcHj1DgAMM/видео.html , I know either way I am not going to be breaking any benchmarks with the Pi but the concept is cool. Let me know what you think, and what you think about the video I linked. Thanks.
@Umbrelladroid Okay I will see what its about, is it exclusive to Debian or can I make this type of cluster with other distros ? Personally I like Debian , but I have a few systems running Arch, and centOS, thanks for your suggestion.
This was so interesting. Please do more videos like these. Thanks again
Thanks for demonstrating Fabric! I personally don't have any idea why you would cluster or have multiple pi machines but I guess someone will tell me! Cause I don't know! Great video!
I still prefer Amiga!
By the time you purchase 4 boards cant you just buy something like a low-end Intel i3 or get a barebones Asrock Desk Mini that comes with the wireless kit and a power adapter? I have one with an intel i3 8100 4 core CPU. Dont really get what you do with a cluster? Too many wires to connect for a cluster. With 4 Pi's you need 4 network cables, 4 power cables, a swithch and a network cable for the switch. All of that costs money. Color coded cables are just fine and make sense. My son works at an ISP and they use a rasberry PI to monitor messages from the network that need addressing and similar things. Probably most of that is text based on a monitor.
Fans on them would look pretty cool aswell and if they were all matching colours
really clean and cool build
As I start wondering if i should build a pi cluster for plex and octoprint your video came at the right moment. Also, totally unrelated but I have the same shirt! LOL
Thanks I enjoyed that and your right, I like the color coded cables. Love your Work
With the newer Pis having PoE the new setup won't require all those power cables. Provided you buy a PoE switch.
you could also just connect power to the pi's 5v and ground pin
You should do a tutorial on how you program multi pi at once step by step and what programs to get and how to set them up. I tried and it's not working.
Awesome video. Just a friendly helper. Look into OBS (Open Broadcast Software). To record your screen.
To me it looks like you made a typo when you specified the ip range (192.168.60 instead of 192.168.1.60) but it still worked - why?
It was a typo during the video and I'm surprised that it worked that way too. The scan probably would have been faster if I had entered the range correctly.
Meh, I came here to see a cluster and found just 4 pis stacked together. I still learned a couple of things so I guess it wasn't a complete waste of time.
Thanks for the demo. Great job! Not sure when you did it, but `pip install fabric` on the 3-13-2018 version of Stretch fails with a missing dependency (at least this morning). An apt-get install fabric seems to work OK (again, as of this morning) for those that encounter it.
Definition = (cluster )
Posted by: Margaret Rouse
WhatIs.com
1) In a computer system, a cluster is a group of servers and other resources that act like a single system
Clusters are not only used for sharing computational load. A cluster can also be used for redundancy, providing fault tolerances in the event of node failure. Look up "failover cluster". You're definition is very basic and only describes one type of cluster, and I would argue its incorrect, a dumbed down definition for layman.
You forgot to lock the ip address by the mac addresses or you'll have to change the ips each time you restart the PIs or the router.
Perfect for a kubernetes cluster!
It's true this isn't a cluster, but this is still amazingly helpful!!!
you should do an update on how the cluster is performing now
I'd love to build a small mini supercomputer cluster out of raspberry Pi to be honest.
I found it comical how you had used your fingers to “close your eyes”. That’s was funny.
This helped me get off and running. Thanks.
Sooo... This is a cool build indeed. But I missed the part on what exactly you built it for. And what kind of uses there are for it. What are the benefits compared to just a single board?
Thanks!!
you can have more computing power with the 4 raspberry pi's if you connect it to your pc. another thing you can also use raspberry pi's for is using it is a sort of motherboard for other things you might build
@@ghostexblades2137 "you can have more computing power with the 4 raspberry pi's if you connect it to your pc" that is not true at all xD
Instead of Etcher, isn't it easier to just dd if=/path/to/raspbian.img of=/dev/sdb (replace sdb with whatever device the SD card is)? Works for ISOs, and dd is a core Linux program.
Be careful though with it, dd has been nicknamed "Disk Destroyer" by the community, because it writes bytes directly on file/device, so if you write faulty input, I don't know what will happen. Hopefully you can use dd again to repair it, or any partition manager to re-initialize the broken disk, but I never tried.
Get a PoE+ bridge and PoE÷ hats. Connect external m2 ssd and usb boot it. Then PXE boot the other pi boards off that first pi... Personally run arch arm v8 for os.
Great instruction, Thanks for the info, I was wondering what applications or projects this 4 node setup can be used for?
im wondering that too
great video on setup. slowly bringing a 5-node cluster online, which I got from a college student whom besides not wanting to pack it to go home (from Ct. to Ca.), I'm guessing lost interest.
I'm starting with the latest 15-Mar-2024 lite 64-bit image. my question, is once everything is up and running, how do I go about running an actual app, that will take advantage of all the nodes,
i.e. something like rarcrack, which is a pre-compiled app. thanks
The start was creepy bro never do that again
Y are so fucking 😂😂😂
Kudos to color coordinating
WOW! I didn't know about fab command! VERY useful! Will definitely be using it now!
Just subscribed :D
Great video....didnt ever thought of this. Thanks for the share mate.
Just a quick question from a "non-developer"; Is it possible to build a Pi cluster such that it behaves like an ordinary Pi (Raspbian GUI interface etc.), but makes use of the other units in the cluster to increase raw processing power & available RAM, giving you what is in effect some sort of "super Pi"?
As far as I know, there is no easy way to "add" like this. This is a non trivial question and goes down with how easy or hard is to split your problem. You could use some work schedulers like kubernetes or nomad, to deploy jobs to the PIs. Some works can not be done without some algorithms changed. The trivial situation is calculate Pi itself. To calculate a new decimal you need the previous, so it is not easy to split this job on many nodes, although statistical approaches can help you there.
In real world, most f the guys out there are using some combination of job schedulers or network load balancing to use the full power of the cluster.
Aye, something like Docker managed by Kubernetes or RancherOS would be a good way to get that sort of functionality. You get the same practical results, although only in parallel fasion (IE you can scale out horizontally - running multiple apache services - but not vertically - you can't make apache *faster*).
Actually making the services combine into one homogenous cluster is something that is more of the thing that serious high-performance computing types would do, but the lack of grunt on the Pi makes it a bit redundant for that.
If the idea of horizontal scaling sounds good, have a look at Rancher. I'm an idiot when it comes to Docker, but even I managed to get a loadbalanced, scalable Wordpress install out of it in the space of about two hours, having previously been (technically) niave on it (I'm familiar with the concepts, but never tried it before).
I'd say if I can get it working - and I'm an idiot - then others probably can to.
rancher.com/docs/os/v1.2/en/running-rancheros/server/raspberry-pi/
The proper answer to your simple question is simply - no.
Short answer: not really. You could run a graphical program on a secondary node but tell it that the display is vis this ip address which is the node that has your monitor plugged into it.
Thanks. Very informative.
Great stuff, thanks for sharing!
Thanks for this video. Literally everything I was looking for :)
this guy is epic, love the Raspberry Pi
I love the colour coding
Nice, I learned about some new tools. Good show
You are a good explainer ❤️
To instructions on assembly some mention should be made of properly grounding yourself and your work area.
I'm sorry if you've already answered this. What software did you use to edit this video?
I think I was still using OpenShot at that point but I've since started using Blender (and ffmpeg for some quick conversions and track mixing).
This is just four Pi's networked, there are much better solutions to manage the packages across them and please don't run four apt-get updates at once lol you can actually use NFS for that.
Also explain a topic like MPI if you talk about clusters.
I would have liked to know after building this how much of each Pi is combined if it is at all and what you can accomplish with it if at all. I don’t expect everything is literally combined but if it’s a ‘super’ computer I suspect it has more computing power and more memory maybe?
What the benefits of having a cluster? Can they do a single task together? Only with more power and speed?
great video; but fabric is giving me a hard time; can you elaborate more?
best idea for labtop build better than MSI or Acer and in the same time low cost
I love black coffee and Programming and Linux and Windows!!!..... :DDD
so i read all the comments and you mentioned that this devices use is for web applications and web services. im unfamiliar with this term, are you able to clear this up for me?
Thanks for the useful assembly video. Was there a reason to choose python fab over something like ansible (or other CM tool) for machine configuration?
Look forward to what you do with these next, keep up the good work!!
Hi Davy. Great video. Just wondering how reliable was the Anker power supply? Have you tried it with more than four RPi? Thanks. Jonathan.
What is the cluster going to be used for?????
hmmm, how is this a cluster?
Cool. Do you have something more current? Maybe Kubernetes?
Thanks a bunch.
Excellent explanations.
That frame would have been so much cooler if it lit up when the Pi's were active....
It's very very useful and good quality video, thx