One thing I should have mentioned is that it's NOT required to set up a group for HA to work. You should if you want to set priorities and/or use nofailback/restricted. Thanks for watching!
Ceph does not need 10gbe in a scenario like this. So low amount of data needs to be synced. You don't even need separate network interface. The only thing that ceph needs additional memory to work well depending on the disk size.
If you have low IOPS Ceph works totally fine with 1Gbe. If you dont have 3 aditional identical SDD you can even set up in VM's using lvm. For services that don't change frequently or dont have big changes like DNS, Reverse proxies, authenticators and VPN gateways is more than enough.
Hi, could you please make a video as part of this series that shows how to connect a external DAS with USB 3.2 type C to the mini PC using Proxmox and maybe Truenas VM? As an alternative to Sinology
3 месяца назад
I am very curious if you could show us how to mount a Synology drive via the network to the cluster. I'd like to have a high availability Scrypted NVR setup for continuous recording in case on node goes down. I know there'll be a gap in between but each camera would have a built in SD card I can use to fill in the gaps.
How was Proxmox running on a Mini PC compared to Raspberry Pi? Please share your thoughts on how you have been using the Mini PC for a few weeks. Are there any issues? Thank you!
Your videos are great but I get soooo distracted by the fact I can see you are just reading text beside the camera. If you could work out a way to look at the camera it would make these so much more engaging.
This is good feedback, thank you. I bought a teleprompter because I wanted my videos to be more focused (felt like they were all over the place), but I can try and adjust the settings to see if I can make it less obvious. I can't do it for the demos because that's just me using the computer but for the talking sections, I'll make adjustments. I really appreciate it, thank you.
Firstly, thanks for the videos. I have 2 G3 Nucboxes - one currently for low power admin use (linux debian) and the other running Proxmox with just my docker VMs and a couple of LXCs (and stays on all the time, not sure where I got that idea from (thanks!)). I'm about to get another one and was going to cluster them, but I was going to use Synology NFS for HA (with qdevice running on my synology, or mikrotik router if it works) - with a 2nd 2.5Gbe nic via Plugable 2.5 USB 3.2 stick (I have had one attached to my synology 920+ for a year or two now and seems to never disconnect). Is there a reason you didn't want to use NFS for your cluster? (my main proxmox hosts (not clustered, as I don't have them on that often) use the NFS with no issues. I do like the fact you went with replication for pretty much stateless vms as it's a great alternative if someone doesn't have a reliable NAS.
Overall, 2.5Gb should provide decent performance but it will be significantly slower than running off the NVMe drive. That's the main reason I don't use it that way - you can set it up that way and I have, but you're going to notice it when using the VM (especially on things like booting up). The NVMe will just provide better performance.
@@WunderTechTutorials I've got 2 x 1TB WD Red nvmes in my synology, so most of the time my most used VMs sit happily in the cache. But I get it, as I said it was good to that other option of replication (I work in IT, we use replication occasionally, but normally as a last resort if we don't have a SAN or HCI), as it's a great choice for stateless VMs. Might be a cool video idea to go with the HA on NFS - In the UK I can't get Nucboxes from Amazon, so have to go direct , so it can take a while to arrive. Happy to share my experience, if you didn't want to interrupt your setup (others might be able to offer quicker results).
@@WunderTechTutorials Apologies, long shipping and vacation. The two G3s are now clustered (with qdevice running in a container on Synology NAS). Had a nightmare with setup due to a bug in qdevice (importing certs, running manually without batchmode etc). But got it configured and working, add the USB 2.5GbE adapters and now have 2.5GbE connection (with jumbo frames) to NAS via a dedicated storage switch (fancy way of saying dumb 2.5GbE switch that supports jumbo frames :D )... Configured one of the VMs to be HA (after moving storage to NFS share on NAS)... Pings dropped during migration - zero. I'll see what a full shutdown migration of one of the nodes does for ping times, but due to the RAM copying over 2.5GbE I doubt they'll be noticeable. I may lose the USB 2.5GbE adapters (as I can route the storage traffic through my Mikrotik CRS310 2.5GbE switch, seems a bit redundant keeping a switch on solely for the HA (and I've got 2TB NVMEs in both G3's, so maybe replication might be the better option - for keeping power usage at a minimum). But thought I'd let you know how I got on.
I am confused about how you are using replication. If you have a VM running on node 1 and the data is being replicated to node 2 and 3 then if node 1 goes down, you will migrate to another node with up to 2 hours of missing data. You say you are fine with that. Your automation states may be off but that may not be a issue. To me the real issue is that once node 1 is back up replication will resume and new data on the other node will be overwritten by old data on node 1. Is replication smart enough to know it now needs to replicate from other node to node 1 now? I am also unclear on how you are using replication. Does each node have three data sets with each data set considered “primary” on one node and a replica on the other two nodes? I am guessing that a VM started on node 1 would then use the “primary” data set for that node.
Good question on the data in terms of node 1 going offline and being down for X amount of time before a replication occurs with "fresh" data. To be fair, the standard is 15 min so I think it would be a much lower "loss", but I am not exactly sure how it would work in that scenario. I'll have to test it out and see. In terms of why I am okay with it being 2 hours, it's mainly because my data doesn't change. I've been using Home Assistant for well over a year without a single change, so outside of updates, there's no real reason for me to replicate it more often. Honestly, probably not even a reason to replicate it every 2 hours but it's fast enough that it doesn't hurt.
That would be harder. If the devices are connected directly to the zigbee dongle (which I assume they are and not HA), I don't think buying a second would help. You'd lose functionality until the node with it comes back online.
It doesn't work if you have ZigBee, zwave or Bluetooth devices in home assistant. Those are device specific. Home assistant ha only works if you only have WiFi devices.
I have a similar setup, pretty happy for almost couple years, I have a LAN Zigbee adapter, so as it has it's own IP, Home Assistant VM can move from one node to another and connect to the same coordinator via that specific IP :)
@@bluesquadron593 in the past I was using the Tuya/LIDL gateway converted (there are manuals online) but as this is an older chip (well, it cost 5€ second-hand and like tinkering so nothing to say here) and today I have the SLZB-06 which works flawlessly, love it.
One thing I should have mentioned is that it's NOT required to set up a group for HA to work. You should if you want to set priorities and/or use nofailback/restricted. Thanks for watching!
Ceph does not need 10gbe in a scenario like this. So low amount of data needs to be synced. You don't even need separate network interface. The only thing that ceph needs additional memory to work well depending on the disk size.
Good point. I was going on the "official" Proxmox recommendation. Thanks for watching!
@@WunderTechTutorialsGreat content!
If you have low IOPS Ceph works totally fine with 1Gbe. If you dont have 3 aditional identical SDD you can even set up in VM's using lvm. For services that don't change frequently or dont have big changes like DNS, Reverse proxies, authenticators and VPN gateways is more than enough.
I loved this video Frank! you explained everything so clearly! amazing job!
Thanks for watching, Avi!
Great video. I would love to see you do a segment on importing VMware exported ova format into proxmox.
Hi, could you please make a video as part of this series that shows how to connect a external DAS with USB 3.2 type C to the mini PC using Proxmox and maybe Truenas VM? As an alternative to Sinology
I am very curious if you could show us how to mount a Synology drive via the network to the cluster. I'd like to have a high availability Scrypted NVR setup for continuous recording in case on node goes down. I know there'll be a gap in between but each camera would have a built in SD card I can use to fill in the gaps.
Most of these mini PC manufacturers do not provide BIOS update at all. Is it really an issue? Thank you.
I haven't run into any problems but I don't have a long enough track record with them to say if it is or isn't a problem.
How was Proxmox running on a Mini PC compared to Raspberry Pi? Please share your thoughts on how you have been using the Mini PC for a few weeks. Are there any issues? Thank you!
Running great on the Mini PC!
Thanks!
dope vid going to try this with some of my hardware now just stuck on how do i setup a vpn traffic for all my lxc containers for my arr stack
Sorry man, not sure how you'd set that up.
Brill stuff thanks
Great video..how would you handle the zigbee. Zwave hubs in proxmox, do you need multiple dongles? Ty
Thanks! Sadly, there isn't a great way (that I know of) to use a dongle like that with HA.
i like it.
Your videos are great but I get soooo distracted by the fact I can see you are just reading text beside the camera.
If you could work out a way to look at the camera it would make these so much more engaging.
This is good feedback, thank you. I bought a teleprompter because I wanted my videos to be more focused (felt like they were all over the place), but I can try and adjust the settings to see if I can make it less obvious. I can't do it for the demos because that's just me using the computer but for the talking sections, I'll make adjustments.
I really appreciate it, thank you.
Check glusterfs to use with shared storage
Gluster was abandoned in 2022 and will be EOL at the end of 2024.
Firstly, thanks for the videos. I have 2 G3 Nucboxes - one currently for low power admin use (linux debian) and the other running Proxmox with just my docker VMs and a couple of LXCs (and stays on all the time, not sure where I got that idea from (thanks!)). I'm about to get another one and was going to cluster them, but I was going to use Synology NFS for HA (with qdevice running on my synology, or mikrotik router if it works) - with a 2nd 2.5Gbe nic via Plugable 2.5 USB 3.2 stick (I have had one attached to my synology 920+ for a year or two now and seems to never disconnect). Is there a reason you didn't want to use NFS for your cluster? (my main proxmox hosts (not clustered, as I don't have them on that often) use the NFS with no issues. I do like the fact you went with replication for pretty much stateless vms as it's a great alternative if someone doesn't have a reliable NAS.
Overall, 2.5Gb should provide decent performance but it will be significantly slower than running off the NVMe drive. That's the main reason I don't use it that way - you can set it up that way and I have, but you're going to notice it when using the VM (especially on things like booting up). The NVMe will just provide better performance.
@@WunderTechTutorials I've got 2 x 1TB WD Red nvmes in my synology, so most of the time my most used VMs sit happily in the cache. But I get it, as I said it was good to that other option of replication (I work in IT, we use replication occasionally, but normally as a last resort if we don't have a SAN or HCI), as it's a great choice for stateless VMs. Might be a cool video idea to go with the HA on NFS - In the UK I can't get Nucboxes from Amazon, so have to go direct , so it can take a while to arrive. Happy to share my experience, if you didn't want to interrupt your setup (others might be able to offer quicker results).
@@mikepackman1516 Absolutely! Would love to hear your experience with it and how you like it!
@@WunderTechTutorials Apologies, long shipping and vacation. The two G3s are now clustered (with qdevice running in a container on Synology NAS). Had a nightmare with setup due to a bug in qdevice (importing certs, running manually without batchmode etc). But got it configured and working, add the USB 2.5GbE adapters and now have 2.5GbE connection (with jumbo frames) to NAS via a dedicated storage switch (fancy way of saying dumb 2.5GbE switch that supports jumbo frames :D )... Configured one of the VMs to be HA (after moving storage to NFS share on NAS)... Pings dropped during migration - zero. I'll see what a full shutdown migration of one of the nodes does for ping times, but due to the RAM copying over 2.5GbE I doubt they'll be noticeable. I may lose the USB 2.5GbE adapters (as I can route the storage traffic through my Mikrotik CRS310 2.5GbE switch, seems a bit redundant keeping a switch on solely for the HA (and I've got 2TB NVMEs in both G3's, so maybe replication might be the better option - for keeping power usage at a minimum). But thought I'd let you know how I got on.
I am confused about how you are using replication. If you have a VM running on node 1 and the data is being replicated to node 2 and 3 then if node 1 goes down, you will migrate to another node with up to 2 hours of missing data. You say you are fine with that. Your automation states may be off but that may not be a issue. To me the real issue is that once node 1 is back up replication will resume and new data on the other node will be overwritten by old data on node 1. Is replication smart enough to know it now needs to replicate from other node to node 1 now?
I am also unclear on how you are using replication. Does each node have three data sets with each data set considered “primary” on one node and a replica on the other two nodes? I am guessing that a VM started on node 1 would then use the “primary” data set for that node.
Good question on the data in terms of node 1 going offline and being down for X amount of time before a replication occurs with "fresh" data. To be fair, the standard is 15 min so I think it would be a much lower "loss", but I am not exactly sure how it would work in that scenario. I'll have to test it out and see.
In terms of why I am okay with it being 2 hours, it's mainly because my data doesn't change. I've been using Home Assistant for well over a year without a single change, so outside of updates, there's no real reason for me to replicate it more often. Honestly, probably not even a reason to replicate it every 2 hours but it's fast enough that it doesn't hurt.
2 are HA as well..
You need a third for voting; otherwise the first node will be unavailable when the other one goes down. But you can set up a qdevice for that.
👍🏻👍🏻👍🏻👍🏻👍🏻
How does this work if you have a zigbee hardware dongle? Do i have to purchase a dongle for each node in cluster?
That would be harder. If the devices are connected directly to the zigbee dongle (which I assume they are and not HA), I don't think buying a second would help. You'd lose functionality until the node with it comes back online.
It doesn't work if you have ZigBee, zwave or Bluetooth devices in home assistant. Those are device specific. Home assistant ha only works if you only have WiFi devices.
I have a similar setup, pretty happy for almost couple years, I have a LAN Zigbee adapter, so as it has it's own IP, Home Assistant VM can move from one node to another and connect to the same coordinator via that specific IP :)
@@JoaquinVacas which adapter do you use?
@@bluesquadron593 in the past I was using the Tuya/LIDL gateway converted (there are manuals online) but as this is an older chip (well, it cost 5€ second-hand and like tinkering so nothing to say here) and today I have the SLZB-06 which works flawlessly, love it.
So instead of just HA, you're running HAHA?
Sorry, I'll see myself out. 😎
😂