As someone interested in homelab stuff but with zero network engineer knowledge, thank you for actually giving real world "what this can run" examples. At my level of layman, it's hard to grasp why some configurations would be run, and real examples are huge in nurturing understanding. Thanks for the vid!
Built a cluster with 3x Minisforum EliteMini HM90 with 32GB RAM (max. is 64GB) + 512GB NVMe + 1TB SSD each - plenty of cluster power with (3x9=) 27 Watts idle. The mini PCs also have two NICs so I'm running a separate Cluster Network for running the synchronization of the nodes as a best practice tip from Proxmox says: "Storage communication should never be on the same network as corosync!" Actually didn't want to run out of CPU power even when operating the one or another Windows OS on the cluster. Anyway: Proxmox rocks! Hope to see more content on your channel - which is one of my favorites. 👍🏻
Yes! That's was exactly why I bought recently the HM80 is exactly like HM90 but with low powered amd ryzen 4800U processor 15wats. They are maybe 20/25% slower but hey! It's only 15 Wats chip! The only thing I didn't like was the fact that probably they don't support ecc ram. But still when you have a cluster I do believe you are covered even when there would be RAM corruption. So maybe that's not the worst idea :-)
The way to get your workloads to prefer a node is to create HA groups for each node. Set the node priority for each group to be the same for all the non preferred nodes, and increase that value by one or two for the preferred node. Workloads will migrate to the node that is up with the highest priority first, and then prefer nodes of equal priority with the most amount of free memory and lowest CPU usage
This is such a cool project. Hadn't taken this board seriously until you posted this but now I keep thinking about it as I'm trying to adjust my new southern, basement-less lifestyle where I'm baking my wife in her office with my homelab servers.
I've seen many of these HA videos. It's highly available but only locally. One fun video idea would be to setup Proper Ha cluster where a secondary node lives in a different network. You should be able to then pull the plug on one of those networks and voila, services are going and the dns provider can failover. I've tried this with portainer, it works but for some reason I've struggled to get portainer to show me the containers running on the remote node.
As others have commented, Ceph needs more memory but it should also have it's own isolated (preferably) network, the faster the better. It will constantly be copying changed blocks on one node to the other two nodes to stay in sync. You should use that second NIC on each node for this.
@@RaidOwl You also have those pcie ports available on each of the zima boards. Maybe use that for additional nvme or 10/25gb nic. I run a similar proxmox cluster on intel NUCs with 6w pentium n3700 cpus but I use ZFS replication instead of ceph. I wish I had those pcie slots though, then I would upgrade the networking and try ceph. The other thing you could try with these boards is a Pfsense/OPNsense CARP HA router setup. They're x86 cpus and have 2x nics so they should be good for that. In a different proxmox cluster I run Pfsense VMs in a combination of Pfsense CARP HA and proxmox ZFS replication HA. It works great, haven't had network downtime in forever.
Actually ceph prefers a single network, "best practices" they want 2, cluster and public. But it's designed against best practises. C.E.R.N. Runs a flat network for ceph, I run 3, networks for Ceph. Management, VM, OSD replication. If you dive deep into the logs you will see ceph complains when running with two networks. But thay error doesn't do anything.
My system not being HA is exactly my problem. I have set up services along with pihole on it, but I don't want to put e.g. pfsense into it and have it work as a router, because if this single device dies, I don't even have access to the internet. Thanks for this video as it shows how to make the server part HA. I wonder if there is an easy way to have a router highly available. My current router worked for years without any trouble, but I am using the pihole as a DNS, because I wanted an intranet domain. Still: if it breaks, I have to manually adjust things to at least have internet back on.
Awesome - gave me some ideas! Some quick questions.. 1) Do all the SSD / HDDs used in the Ceph nodes have to be the same size? 2) If I started off with (say) 500Gb drives and wanted to upgrade to (say) 2Tb drives, could I do this one disk at a time and avoid data loss, or would I have to back everything up, upgrade all the disks, and restore? 3) In your example, you had three 1Tb disks. How much storage do you end up with on the cluster? 4) Can you add more Ceph nodes once it's up and running? 5) If you can add more nodes, will it start with fewer than 3 nodes? (I need to spread the cost of disk purchases out!) TIA
1) practically yes; your usable size will be the size of the smallest disk. 2) yes, it’s a piece of cake. 3) configuration shown was R3 (replication across three nodes), so total usable size is 1TB. You can use EC2+1 (erasure coding of 2 data and 1 parity) that will give you 2TB of space while allowing for one disk/node to fail. 4) yes, it’s a piece of cake. 5) yes, but some deficiencies. Proxmox cluster: 2 nodes depend heavily on each other. You won’t be able to start vms/lxcs, change the configuration and so on while you have only one node online (official/supported flow of things). CEPH cluster: chances are you’ll end up with osd-level redundancy, so your operation will fail if any of your disks/nodes go down.
Ayee, that map screenshot shows my house. What a small world. I need to set something like this up, but w/ more power than the ZimaBoards. I know all the options, but it's hard balancing performance vs cost vs efficiency. Thanks for the vid!
Such a great video. I’m doing the exact same thing with intel nuc. I use ceph at work and I despise it, but seeing how simple it is inside of proxmox has convinced me to try it at home.
I have a similar setup, but use the Lenovo mini-pcs with I5's and 16gb RAM, SSD + NVME, a great setup. Expandable to 32gb on each node, things work great. Just need some fast shared storage now.
I've done the same exact thing on 4 different small form factor hardware. I'm running Open Media Vault which file sharing and docker, portainer support running all kinds of cool stuff. The must have was a UNIFI controller in docker. Having that in a HA cluster on battery backup is AMAZING!. It's very cool. I'll have to give the CEPH a try though. This looks like it save me some power. I'm using an NSF cluster for HA storage on 3 other mini PCs.
I am currently using truenas scale for my vms and containers but this is tempting. I have two questions: 1. What is the performance penalty compared to single node? Is it noticeable for Web services? 2. How does upgrading proxmox to newer version works with cluster like this? You just shut down node A, do upgrade, reconnect it to cluster with older versions? Then next node and so on?
Now that is a super handy cluster. You can feed it with power from a small array of solar panels and boom. You could even host your own satellite with such a setup 😂 And since proxmox is just, plain debian, anything could be done with it.
I run a similar setup (but on NUCs instead of Zima boards) - one thing I've noticed is that LXC containers migrate between nodes in a few seconds, whereas my VMs take about a minute to migrate.
I experimented with a qdevice approach: main server, small x86 where critical services start if main server is down and and old pi to keep quorum. It worked fine, but the way I did it, storage wasn't redundant (it was all on my NAS) and when I wanted to stop using HA I had to mess around with proxmox files on disk directly.
Running Proxmox and a VM just to run Kubernetes in HA seems like an over kill. You are running HA container management on top of HA hypervisor. Why not just install an OS that supports cephfs and install Kubernetes on top. Just need to be sure that your etcd cluster spans all three nodes and each node is schedulable for workloads. One node goes down and Kubernetes can move the workload without having to first move Kubernetes and then the workload. Not as much fun but less hassle in the long run.
Funny your video popped up on my phone this morning. I spent my entire weekend trying to figure out how to auto migrate a VM back to its original node. Apparently, it can be done with hook scripts.
Presumably this would be a good way to run a HA HomeAssistant home automation system? to my knowledge, although folk use Proxmox to host HomeAssistant (I do), no one has posted a video on how to run one in a HA way.
Very interesting project. I see that they are selling the new ZimaBlade in a 3 unit cluster configuration. Hopefully, we can get slightly better performance out of the new ZimaBlades
I'm thinking about doing the same with 3-5 R86S. The nice part being the dual 10gbe ports. 1 port for client traffic, and 1 port for the ceph network. also there are 2.5gbe ports left over for management or something else. Or I could use the two 10gbe ports for a ring network for ceph, and the 2.5gbe ports for client. The ring network is interesting because I could avoid buying a 10gbe switch just for ceph
Proxmox seems pretty heavy for something like this. I'd like to see something like this with updated software. Possibly Fedora CoreOS to get auto OS updates with rollback, with k3s and Rancher dashboard, or just go full first-party with Docker Swarm with ___ dashboard (I wish there was a first-party one), replicated masters in case the master node goes down, but also sharing storage for HA. Double drives (NVMe or SATA) on the Zimaboards for redundancy to handle a single drive outage without losing full HA support for services (can drive mirroring be done with clustered storage within the same host in place of RAID?). I was thinking something like this with Cockpit to manage the system hardware and OS and Rancher to handle the apps. It's too bad that Cockpit doesn't have a Kubernetes extension. They used to, but it can't be found anymore. Probably because they're supporting Podman already. There's nothing in Cockpit (AFAIK) that creates or manages clusters unless you go over to the Red Hat corporate side of things and start looking at OpenShift, and even the community version OKD is too much for a Zimaboard. Minishift is also dead.
that was interesting i probably dont need that but the next time i see an minipc like the one i already have, i'm gonna get just to have fun with cluster lol
@Raid Owl please please make in-depth proxmox tutorial I am struggling to understand the LVM stuff I use EX4, and for instance, if something happens to the system and I plugged that hard disk into a working system, how do I access the files that were inside that proxmox that are like LVM? I don't know. There's just no really good in-depth proximox tutorial. There's a few but a really good one I think a lot of people would really like that
If the middle one died then you have nothing connecting the other two. If you were then to respond with "then loop then back" well then you have two problems. Broadcast storms and no way to network them to anything else (because all ports are taken). The single point of failure still exists here, and yes it's the switch (and probably also the powerboard your using to plug in all those cords). It's a surmountable problem though, just needs more thought on the network side. RSTP on managed switches. Does proxmox support cluster over two networks?
My suggestion would be using low power cluster as backup.. Maybe consider using a setup like this: Cluster A (High Power) - Node A1, Node A2 Cluster B (Lower Power) - Node B1, Node B2 HA - Cluster A and Cluster B (maybe cross wired) for efficient and more practical usecase.. This will also make sense in regards to performance per watt.
"...it sits about 10Watts total.." What hardware/software do you use to monitor these SBC's? Fantastic info. Time to give Proxmox a try (been using UNRAID for years now).
If you plan to have k8s on zima in vm then why to use proxmox in the first? Only reason I can think of is to use other OS then Linux but resources will still limit you. Ceph can be installed inside k8s or next to it. Besides ceph is not only block. You can have s3/Swift object storage or cephfs POSIX compatible server.
I have a question... how those zima boards works in 24[7 schedule? and what doing on if your zima1 (for example) with ssd has been down? how you connect ssd as ha nas?
Cool setup! I want to make my own mini cluster and this inspired me! Maybe Beelink EQ12(I dont know whether it can be sold in your untry, but it is availble in China and Singapore) is an alternative to ZimaBoard if you want more compute power hahahaha. it has the new Intel N100(which may be 2~3X faster than N3450, 4 cores) , and the no memory no ssd version cost about $150 in China.
Very interesting, that is exactly what I am planning to set up for home automation (NodeRed, Grafana, ...). Right now I am hanging with one topic. (How) is it possible to share data between VMs or Container? I think, Proxmox is running on its own (the internal) disk. The VMs, and containers are on the external SSD. When I am setting up a container, that container gets his own virtual hdd assigned that is placed on the external SSD / the Ceph disk. Is it possible / how to have a folder / diskarea, partition, ... lets call it "shared folder" where different container can read and write data. Later on there is maybe a container with a simple NAS software solution or just a SMB share, that gives me acces to that "shared folder" via LAN so I can backup that data from time to time. I would be very happy, if someone can help me out how to do that. Thanks a lot
Can you please let me know as a beginner, is it a good idea to run out of the box casaos on Zimaboard or do I need to set up Proxmox and boot it from a USB Please help me understand, as this is my first mini home server. I would like to learn and build on top of it.
If using a single NAS isn't HA because its failure would down the whole system, then how can using a single switch be HA? Both appear to be single points of failure.
I started to set up an HA ProxMox cluster, and got stuck at the "Fencing" part. Is fencing optional, or does it just seem more intimidating than it is?
You are using a single network interface on each Zima Boards for regular networking as well as cluster networking? That never worked for me, time and again, the cluster would incorrectly detect a network issue and the cluster would go bizzare forcing me to reboot the entire thing!
I'm lost on how you are able to setup HA mode when the storage is local on each device. I have a similar setup and I want to do that. Can anyone point me in the right direction with clear instructions? Or explain in detail for me?
@@RaidOwl Thank you for your reply. I did a little more homework and I think I have it under control. When I first configured CEPH I did it wrong. Someone posted a great tutorial 12 days ago and that's where I noticed my mistakes. I love the channel and I love your work. 1000 thanks....
I can't stop but feeling like ceph is a bit overkill for a cluster in this scale and size and with the given hardware. Instead I would look towards zfs and the use replication between the nodes for the vm's that needs HA. This would also minimize the complexity of the setup a lot and get rid of things like split brain issues which can happen with ceph 😊
@@RaidOwl agree that it's absolutely what should be done in a lab environment to get to know how it works! But I'd still prefer zfs for lower ish end non production environment 😉 Keep up the nice videos though!
You could buy for that money a much better system. For an example, for 120-150 Dollar, u could get a refurbed lenovo thinkcentre m700 with core i5 like 6400t or 6500t, 8 GB RAM and 256GB SSDs, and it takes only 15-25 watt with a much better perfomance rate. You''ll get also a clean case and so on.
I did exactly this. Zimaboard 216 and Fujitsu Q956 with i5-6400t (which I upgraded basically doubling it's price). Zimaboard consumes 2.5w idle, 7.5w at load. Fujitsu is more compilcated: 5.5w idle without anything (but network) attached, 7-9w at very little load (added usb wifi, running podman-compose with nextcloud, mariadb and cloudflared which greatly increases wakeups per seconds, still like 2-5% cpu usage) and 37w at full load. But at full load it's roughly five times faster. Zimaboard costs roughly 130eur including shipping and additional costs were minidp-hdmi cable and 250gb sata harddrive. Fujitsu costs roughly 130eur including 128 sata harddrive and 8gb ram. Running nextcloud on zimaboard feels like it uses half of memory a stuggles a bit when resizing jpegs on fly while handling https connection. Fujitsu feels like overkill for this task. It's almost as good as i5-4590 desktop i used from 2015 to late 2021 and it's faster than my 5 years old i7-7500u notebook. Power efficiency is insane.
"Somewhat available" should be a configurable option! :) Instead of Ceph or some other highly available option, it could make nightly or weekly snapshots or something like that, and still auto migrate.
As someone interested in homelab stuff but with zero network engineer knowledge, thank you for actually giving real world "what this can run" examples. At my level of layman, it's hard to grasp why some configurations would be run, and real examples are huge in nurturing understanding. Thanks for the vid!
It’s ok that you had the original idea to steal my idea, I stole the idea too 😂
One day I’ll have an original idea…one day
With tech, it is less about the tech you are showing and more the perspective of the tech
Good ideas are contagious
Built a cluster with 3x Minisforum EliteMini HM90 with 32GB RAM (max. is 64GB) + 512GB NVMe + 1TB SSD each - plenty of cluster power with (3x9=) 27 Watts idle.
The mini PCs also have two NICs so I'm running a separate Cluster Network for running the synchronization of the nodes as a best practice tip from Proxmox says: "Storage communication should never be on the same network as corosync!"
Actually didn't want to run out of CPU power even when operating the one or another Windows OS on the cluster.
Anyway: Proxmox rocks! Hope to see more content on your channel - which is one of my favorites. 👍🏻
Hell yeah that’s awesome
Yes! That's was exactly why I bought recently the HM80 is exactly like HM90 but with low powered amd ryzen 4800U processor 15wats. They are maybe 20/25% slower but hey! It's only 15 Wats chip! The only thing I didn't like was the fact that probably they don't support ecc ram. But still when you have a cluster I do believe you are covered even when there would be RAM corruption. So maybe that's not the worst idea :-)
😊
The way to get your workloads to prefer a node is to create HA groups for each node. Set the node priority for each group to be the same for all the non preferred nodes, and increase that value by one or two for the preferred node.
Workloads will migrate to the node that is up with the highest priority first, and then prefer nodes of equal priority with the most amount of free memory and lowest CPU usage
Interesting, thanks for the tip!
Works perfectly. Thanks for this tip!
"Cause I can, and I am a nerd." Words to live by.
Should be on a merch item
This is such a cool project. Hadn't taken this board seriously until you posted this but now I keep thinking about it as I'm trying to adjust my new southern, basement-less lifestyle where I'm baking my wife in her office with my homelab servers.
I've seen many of these HA videos. It's highly available but only locally. One fun video idea would be to setup Proper Ha cluster where a secondary node lives in a different network. You should be able to then pull the plug on one of those networks and voila, services are going and the dns provider can failover. I've tried this with portainer, it works but for some reason I've struggled to get portainer to show me the containers running on the remote node.
As others have commented, Ceph needs more memory but it should also have it's own isolated (preferably) network, the faster the better. It will constantly be copying changed blocks on one node to the other two nodes to stay in sync. You should use that second NIC on each node for this.
Noted, thanks for the tip! Not much is happening on the main nic anyway lol
@@RaidOwl You also have those pcie ports available on each of the zima boards. Maybe use that for additional nvme or 10/25gb nic. I run a similar proxmox cluster on intel NUCs with 6w pentium n3700 cpus but I use ZFS replication instead of ceph. I wish I had those pcie slots though, then I would upgrade the networking and try ceph.
The other thing you could try with these boards is a Pfsense/OPNsense CARP HA router setup. They're x86 cpus and have 2x nics so they should be good for that. In a different proxmox cluster I run Pfsense VMs in a combination of Pfsense CARP HA and proxmox ZFS replication HA. It works great, haven't had network downtime in forever.
this is why the R86S are so interesting to me
10gbe ceph network + 10gbe client network
OR
2x10gbe for a ceph ring network and 2.5gbe for client
Actually ceph prefers a single network, "best practices" they want 2, cluster and public. But it's designed against best practises. C.E.R.N. Runs a flat network for ceph, I run 3, networks for Ceph. Management, VM, OSD replication.
If you dive deep into the logs you will see ceph complains when running with two networks. But thay error doesn't do anything.
Very informative and clear. I have now a better understanding about how a Cluster and Ceph work in Proxmox. Thanks
"Because I can, and I'm a nerd." - RaidOwl
That's exactly my reasoning for most of my Home Lab projects.
Thank you for making this video! Proxmox cluster was the very next thing on my lab to do list!
"Or if you want an actual good video ..." lol, that took me by surprise - good to have a sense of humour!
My system not being HA is exactly my problem. I have set up services along with pihole on it, but I don't want to put e.g. pfsense into it and have it work as a router, because if this single device dies, I don't even have access to the internet.
Thanks for this video as it shows how to make the server part HA. I wonder if there is an easy way to have a router highly available. My current router worked for years without any trouble, but I am using the pihole as a DNS, because I wanted an intranet domain. Still: if it breaks, I have to manually adjust things to at least have internet back on.
Thank you, another rabbit hole for me to go in.
Great video! I ultimately want to get to this point as well and possibly include plex in a supermicro 1u cluster with a p400 in each.
A little bit more horsepower than mine lol
Only 10 watts and so much functionality is definitely a winner.
Awesome - gave me some ideas!
Some quick questions..
1) Do all the SSD / HDDs used in the Ceph nodes have to be the same size?
2) If I started off with (say) 500Gb drives and wanted to upgrade to (say) 2Tb drives, could I do this one disk at a time and avoid data loss, or would I have to back everything up, upgrade all the disks, and restore?
3) In your example, you had three 1Tb disks. How much storage do you end up with on the cluster?
4) Can you add more Ceph nodes once it's up and running?
5) If you can add more nodes, will it start with fewer than 3 nodes? (I need to spread the cost of disk purchases out!)
TIA
1) practically yes; your usable size will be the size of the smallest disk.
2) yes, it’s a piece of cake.
3) configuration shown was R3 (replication across three nodes), so total usable size is 1TB. You can use EC2+1 (erasure coding of 2 data and 1 parity) that will give you 2TB of space while allowing for one disk/node to fail.
4) yes, it’s a piece of cake.
5) yes, but some deficiencies. Proxmox cluster: 2 nodes depend heavily on each other. You won’t be able to start vms/lxcs, change the configuration and so on while you have only one node online (official/supported flow of things). CEPH cluster: chances are you’ll end up with osd-level redundancy, so your operation will fail if any of your disks/nodes go down.
I'm going to build my proxmox server with my old 3rd gen quad core laptop. Thanks for inspiring me
Ayee, that map screenshot shows my house. What a small world.
I need to set something like this up, but w/ more power than the ZimaBoards. I know all the options, but it's hard balancing performance vs cost vs efficiency. Thanks for the vid!
I'm surprised back-plane pcie interconnects aren't a thing with SBCs like this, would make for far better clusters.
Such a great video. I’m doing the exact same thing with intel nuc. I use ceph at work and I despise it, but seeing how simple it is inside of proxmox has convinced me to try it at home.
another good option are Dell Wyse thinclients. they are low power, x86, more powerful, and about 3X cheaper than the zima board
Shoutout to the artisanal mountain monks making beautiful server racks
I absolutely love your easy way of explaining. Subscribed ;)
I have a similar setup, but use the Lenovo mini-pcs with I5's and 16gb RAM, SSD + NVME, a great setup. Expandable to 32gb on each node, things work great.
Just need some fast shared storage now.
I've done the same exact thing on 4 different small form factor hardware. I'm running Open Media Vault which file sharing and docker, portainer support running all kinds of cool stuff. The must have was a UNIFI controller in docker. Having that in a HA cluster on battery backup is AMAZING!. It's very cool. I'll have to give the CEPH a try though. This looks like it save me some power. I'm using an NSF cluster for HA storage on 3 other mini PCs.
Fascinating. Would definitely like for more videos to be posted on this SBC.
I am currently using truenas scale for my vms and containers but this is tempting. I have two questions:
1. What is the performance penalty compared to single node? Is it noticeable for Web services?
2. How does upgrading proxmox to newer version works with cluster like this? You just shut down node A, do upgrade, reconnect it to cluster with older versions? Then next node and so on?
Now that is a super handy cluster. You can feed it with power from a small array of solar panels and boom. You could even host your own satellite with such a setup 😂
And since proxmox is just, plain debian, anything could be done with it.
I run a similar setup (but on NUCs instead of Zima boards) - one thing I've noticed is that LXC containers migrate between nodes in a few seconds, whereas my VMs take about a minute to migrate.
Migration of LXC's have to shutdown first, no? They are quicker since no OS per-se
Crazy good. Exactly what i was looking for
I experimented with a qdevice approach: main server, small x86 where critical services start if main server is down and and old pi to keep quorum. It worked fine, but the way I did it, storage wasn't redundant (it was all on my NAS) and when I wanted to stop using HA I had to mess around with proxmox files on disk directly.
Best practices to disable PG Autoscale. It has killed a lot go ceph clusters.
Running Proxmox and a VM just to run Kubernetes in HA seems like an over kill. You are running HA container management on top of HA hypervisor. Why not just install an OS that supports cephfs and install Kubernetes on top. Just need to be sure that your etcd cluster spans all three nodes and each node is schedulable for workloads. One node goes down and Kubernetes can move the workload without having to first move Kubernetes and then the workload. Not as much fun but less hassle in the long run.
This is amazing, thanks for sharing this architecture
Funny your video popped up on my phone this morning. I spent my entire weekend trying to figure out how to auto migrate a VM back to its original node. Apparently, it can be done with hook scripts.
Someone in the comments mentioned you can do it by setting up HA Groups
Presumably this would be a good way to run a HA HomeAssistant home automation system? to my knowledge, although folk use Proxmox to host HomeAssistant (I do), no one has posted a video on how to run one in a HA way.
The memory is a bottleneck here. CEPH needs a good chunk of memory. I found 16GB was not enough for a 500gb CEPH pool. With a vm or two.
Yeah I think for this use case it’ll be fine. For my production system it’ll be more robust
Man proxmox is so good!!
Love it!
❤️❤️❤️
@@RaidOwl Someday I'm gonna have to try to convince you to give VMware a try!
@@2GuysTek goodluck ;) I'll give you exactly 5 min on our next stream to make your case lol
Challenge accepted!
I plan to go K8s native, no proxmox. Using something cheap as Control plane, and this zima boards as workers. Let's see how it goes.,
Goodluck soldier
Building a HA Cluster with single Point of Failure (Switch) is exactly my kind of Humor👍
It’s only running in a single house on a single planet too, how crazy is that???
Very interesting project. I see that they are selling the new ZimaBlade in a 3 unit cluster configuration. Hopefully, we can get slightly better performance out of the new ZimaBlades
I'm thinking about doing the same with 3-5 R86S. The nice part being the dual 10gbe ports. 1 port for client traffic, and 1 port for the ceph network. also there are 2.5gbe ports left over for management or something else.
Or I could use the two 10gbe ports for a ring network for ceph, and the 2.5gbe ports for client. The ring network is interesting because I could avoid buying a 10gbe switch just for ceph
I really want to like the Zima board, but the pcie slot sticking out bugs me. Especially with such a nice looking case/heatsink design
Proxmox seems pretty heavy for something like this. I'd like to see something like this with updated software. Possibly Fedora CoreOS to get auto OS updates with rollback, with k3s and Rancher dashboard, or just go full first-party with Docker Swarm with ___ dashboard (I wish there was a first-party one), replicated masters in case the master node goes down, but also sharing storage for HA. Double drives (NVMe or SATA) on the Zimaboards for redundancy to handle a single drive outage without losing full HA support for services (can drive mirroring be done with clustered storage within the same host in place of RAID?). I was thinking something like this with Cockpit to manage the system hardware and OS and Rancher to handle the apps. It's too bad that Cockpit doesn't have a Kubernetes extension. They used to, but it can't be found anymore. Probably because they're supporting Podman already. There's nothing in Cockpit (AFAIK) that creates or manages clusters unless you go over to the Red Hat corporate side of things and start looking at OpenShift, and even the community version OKD is too much for a Zimaboard. Minishift is also dead.
This looks awsome. For testing like you said its fine. Now i consider one or two of these for pfSense
They’re awesome for pfSense
@@RaidOwl cool. I looked for stuff like this but hadnt ZimbaBoards on my radar here in Germany. And for 110-190€ its kinda a steal for pfSense
I want a setup like this, due to draw power.
I am currently looking at a cluster build with intel nuc, but this set up would be cool to tip my toes in clusters and such.
Intel NUCs are solid too. I snagged some $50 ones off eBay and upgraded the ram and storage to run my main k3s cluster on.
Bitwarden would be nice on a Cluster.
NICE buddy !! this is awesome !!
that was interesting
i probably dont need that but the next time i see an minipc like the one i already have, i'm gonna get just to have fun with cluster lol
Great informative video!
Gives me something to think about with my intel NUC's
@Raid Owl please please make in-depth proxmox tutorial I am struggling to understand the LVM stuff I use EX4, and for instance, if something happens to the system and I plugged that hard disk into a working system, how do I access the files that were inside that proxmox that are like LVM? I don't know. There's just no really good in-depth proximox tutorial. There's a few but a really good one I think a lot of people would really like that
Hi there,
is it possible to daisy chain zimaboards since all of them have 2 ethernet ports or is it neccessary to connect all of them to a switch?
If the middle one died then you have nothing connecting the other two.
If you were then to respond with "then loop then back" well then you have two problems. Broadcast storms and no way to network them to anything else (because all ports are taken).
The single point of failure still exists here, and yes it's the switch (and probably also the powerboard your using to plug in all those cords).
It's a surmountable problem though, just needs more thought on the network side. RSTP on managed switches. Does proxmox support cluster over two networks?
great video, thanks for making it. have my engagement!
Can you run pfSense on a Proxmox cluster?
Having a highly available router/firewall seems like a neat solution
Oh wow! That's pretty much my "wet dream" right there. Nicely done. I'm really jealous.
Would this be a good design for a micro super computer in a small pelican case? Something better than a standard laptop.
My suggestion would be using low power cluster as backup.. Maybe consider using a setup like this:
Cluster A (High Power) - Node A1, Node A2
Cluster B (Lower Power) - Node B1, Node B2
HA - Cluster A and Cluster B (maybe cross wired) for efficient and more practical usecase.. This will also make sense in regards to performance per watt.
"...it sits about 10Watts total.." What hardware/software do you use to monitor these SBC's? Fantastic info. Time to give Proxmox a try (been using UNRAID for years now).
I had them plugged into my Vanspower portable battery bank. I have a video on it if you wanna check it out
@@RaidOwl That would be great if you can post the link. Thanks mate.
@@mochalatte3547 ruclips.net/video/SiZRnyz8R1U/видео.html
You can tell it's monks from the holiness in their design.
…omg
If you plan to have k8s on zima in vm then why to use proxmox in the first? Only reason I can think of is to use other OS then Linux but resources will still limit you. Ceph can be installed inside k8s or next to it. Besides ceph is not only block. You can have s3/Swift object storage or cephfs POSIX compatible server.
Does sound like a fun thing to do!!
Cool. I dig it. 🔥
Hey bro. I heard you want to build an HA proxmox setup to monitor your other HA proxmox setup.
😳😳😳
The best reason for everything is "because i can"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! LOVE IT!
is this pwerful enough to run and train open source LLMs?
What about a K3S cluster
I have a question... how those zima boards works in 24[7 schedule?
and what doing on if your zima1 (for example) with ssd has been down? how you connect ssd as ha nas?
I was recently actually looking for a tutorial to do this lol
Great video!!!!!! Love it
Only thing about this is installing it on emmc... not idea.
I'm suffering from cluster envy
Zima boar support mikrotik router os or not sir 😅😅
Thanks !!
Cool setup! I want to make my own mini cluster and this inspired me!
Maybe Beelink EQ12(I dont know whether it can be sold in your untry, but it is availble in China and Singapore) is an alternative to ZimaBoard if you want more compute power hahahaha.
it has the new Intel N100(which may be 2~3X faster than N3450, 4 cores) , and the no memory no ssd version cost about $150 in China.
Yeah I went that route. But there are a few cons. With the biggest no 1: You don't have support for the ECC Ram memory 😢
@@marcin6386 oh ECC memory is critical for some situations, i forget that...🤣
Very interesting, that is exactly what I am planning to set up for home automation (NodeRed, Grafana, ...).
Right now I am hanging with one topic. (How) is it possible to share data between VMs or Container?
I think, Proxmox is running on its own (the internal) disk. The VMs, and containers are on the external SSD.
When I am setting up a container, that container gets his own virtual hdd assigned that is placed on the external SSD / the Ceph disk.
Is it possible / how to have a folder / diskarea, partition, ... lets call it "shared folder" where different container can read and write data.
Later on there is maybe a container with a simple NAS software solution or just a SMB share, that gives me acces to that "shared folder" via LAN so I can backup that data from time to time.
I would be very happy, if someone can help me out how to do that.
Thanks a lot
What if that single switch fails? It's not high avail then is it
What if my house explodes? Then it’s not highly available either
@@RaidOwl You should definitely get another house to play good. LoL 😅
Can you please let me know as a beginner, is it a good idea to run out of the box casaos on Zimaboard or do I need to set up Proxmox and boot it from a USB
Please help me understand, as this is my first mini home server. I would like to learn and build on top of it.
CasaOS would be perfectly fine as your main OS. It's installed on top of Debian Server anyway so it'll be fine.
The ZimaBoard could have been straight up God tier if it had built-in NVME support. Installing a PCI-E add-in card doesn't count.
I installed proxmox on my Zimmaboard on emmc too. Question is how long will it last until it dies. There’s maybe a reason emmc not being supported
Time will tell
I'm not a fan of running a system with a PF of around or below 0.5 as there's a lot of wasted power being burned...
With the pci express port, can you not attach a video card for plex?
Good idea❤❤❤❤❤❤
If using a single NAS isn't HA because its failure would down the whole system, then how can using a single switch be HA? Both appear to be single points of failure.
I started to set up an HA ProxMox cluster, and got stuck at the "Fencing" part. Is fencing optional, or does it just seem more intimidating than it is?
You are using a single network interface on each Zima Boards for regular networking as well as cluster networking? That never worked for me, time and again, the cluster would incorrectly detect a network issue and the cluster would go bizzare forcing me to reboot the entire thing!
I’ll keep my eye on it. If it acts wonky I’ll use that other one.
Would be nice with POE for [more] convenience.
I agree 100%
Why not using harvester instead of proxmox ?
I like Proxmox
I'm lost on how you are able to setup HA mode when the storage is local on each device. I have a similar setup and I want to do that.
Can anyone point me in the right direction with clear instructions? Or explain in detail for me?
Ceph replicates everything stored locally to the other 2 nodes as well. So if something needs to be migrated the data is already there locally.
@@RaidOwl Thank you for your reply. I did a little more homework and I think I have it under control. When I first configured CEPH I did it wrong. Someone posted a great tutorial 12 days ago and that's where I noticed my mistakes. I love the channel and I love your work. 1000 thanks....
Can you give us the link to the italian monks in the swiss alps to order that board?
My garage lol
I can't stop but feeling like ceph is a bit overkill for a cluster in this scale and size and with the given hardware. Instead I would look towards zfs and the use replication between the nodes for the vm's that needs HA. This would also minimize the complexity of the setup a lot and get rid of things like split brain issues which can happen with ceph 😊
Lol probably but I’ve never tried Ceph so this was my excuse to give it a go since Proxmox makes it easy AF
@@RaidOwl agree that it's absolutely what should be done in a lab environment to get to know how it works! But I'd still prefer zfs for lower ish end non production environment 😉
Keep up the nice videos though!
You could buy for that money a much better system. For an example, for 120-150 Dollar, u could get a refurbed lenovo thinkcentre m700 with core i5 like 6400t or 6500t, 8 GB RAM and 256GB SSDs, and it takes only 15-25 watt with a much better perfomance rate. You''ll get also a clean case and so on.
I did exactly this. Zimaboard 216 and Fujitsu Q956 with i5-6400t (which I upgraded basically doubling it's price). Zimaboard consumes 2.5w idle, 7.5w at load. Fujitsu is more compilcated: 5.5w idle without anything (but network) attached, 7-9w at very little load (added usb wifi, running podman-compose with nextcloud, mariadb and cloudflared which greatly increases wakeups per seconds, still like 2-5% cpu usage) and 37w at full load. But at full load it's roughly five times faster. Zimaboard costs roughly 130eur including shipping and additional costs were minidp-hdmi cable and 250gb sata harddrive. Fujitsu costs roughly 130eur including 128 sata harddrive and 8gb ram.
Running nextcloud on zimaboard feels like it uses half of memory a stuggles a bit when resizing jpegs on fly while handling https connection. Fujitsu feels like overkill for this task. It's almost as good as i5-4590 desktop i used from 2015 to late 2021 and it's faster than my 5 years old i7-7500u notebook. Power efficiency is insane.
Does the shared storage have any redundancy built in? or just creates 1 pool essentially?
The size in the Ceph cluster dictates the redundancy so in my case it was 3 copies.
Very very impressive,could you do it with R86S-U4 which are Intel N6005 CPU with 32GB ram+3*2.5G+2*10.0G SFP+ port? That was so expected!!
Yeah for sure!
Yes, but why run a hypervisor to begin with?
Why not?
You heard the man. Just because he can and he’s a nerd. 🤪
Don’t use rancher for small resources, just use openlens which runs on your desktop machine and connects to your k3s
With my budget I'll have to settle for "Somewhat available"
😂😂😂
"Somewhat available" should be a configurable option! :) Instead of Ceph or some other highly available option, it could make nightly or weekly snapshots or something like that, and still auto migrate.
How do I run one VM across all three nodes in my cluster to take advantage of all cores?
You can only run a single VM on one machine at a time.