Yeah if I ever setup something like this, I'd need to get solar first to help power it I don't even know why I'd need half of this, since I can do everything I need right now with about 2 server devices and a NAS. But I know these things tend to grow.
Thanks for going into this. Many videos go over hardware and many more state the software they use, but very few actually explain the entire network and how it works. Thank you!
Really interesting, thanks for the video and always being a large inspiration to expand my homelab! I'm in the process of moving services the other way: from 50+ docker-compose containers on a Synology 920+ to a new Kubernetes cluster running Flux, very similar to you. I'm doing so because the NAS is completely overworked and I thought spreading out the load across multiple nodes would help with this - plus GitOps is awesome. Would love to see a future video where you talk about your experience moving these apps to TrueNAS. Keep up the great work!
Glad I had a minute to watch! Now do one on your favorite/top tools/plugins/automations (things that make your life exponentially easier that many of us may not know of)
13:04 In future videos, I request that you refer to these nodes as "nuc nuc nuc" and maybe follow it with a "whoob whoob whoob" while you shuffle. Thanks for the awesome tour!
44:01 you could try out in portainer selecting creating a stack, selecting repository. fill in the relevant info under git repository and turn on gitops updates. any future changes/commits to your compose file in the future will update according to the duration you set in the fetch interval.
What an amazing homelab :) mine is just a single desktop and an OpenWRT router yet it has allowed me to learn a lot about IT, the CI/CD part and how you're handling updates is so amazing :) I will definitely try it
Great video. I’m absolutely with you whether to use mirrors or RAID. My take is that mirrors just makes things less complex; it allows for easy expansion, which also includes increased resilience and performance.
Always love to see your setups. I’ve even borrowed a few ideas in the past. The only thing I do different is I have my wife’s work from home office in its own vlan that only has access to the internet. Hahah. You literally talk about removing her laptop after posted.
FYI . I didn't see nut server in your home assistant services. I use it for the 7 UPS I have running in my home to monitor all of them. Awesome video and homelab!
Great video! Love watching your tours when you put them out. I always end up finding something new to try out in my home lab. 1 video request, I haven't seen anybody put out an update VLAN guide with Unifi with there latest release and the new zone based way of doing it. I really want to make the step of VLANs but want to make sure I get the rules right specifically for things like HomePods and Apple Tvs that would be on Main but can talk to devices on IoT.
I had the same storage problem with sqlite and uptime kuma. I ended up running sqlite commands in a init container to configure my uptime kuma container. So I have a stateless version and still get notifications over my home assistant instance if something is down. There was a problem... getting the right clean DB for the latest version of uptime kuma. So I have a init container starting the normal kuma image, waiting for the logs to tell me that is finished initializing the DB, then killing that container, then running my init container with the sqlite commands, then finally running the main container. But it works :D
I eventually scaled down to my main media server being my container, a dedicated NAS server, and moved away from vms with a 1-2-3 back solution for data to consolidate hardware, heat, power, and equipment. As for additional nodes I went with a few pi’s and mini pcs.
I have many implementations of GitOps with Docker Compose. Currently using Actions to trigger an Ansible playbook with some variables like what docker compose file and mapping that to a host, and it SSH's into the target machine and git pulls and docker compose up -d on the correct compose file and returns all the logs in the actions workflow.
I can't wait for the next major update to TrueNAS. In Fangtooth they looking to over haul the VM section to bring TrueNAS closer to other HyperVisors to go with their Very good NAS and now Docker services.
Great video! I like the Gravity Sync idea, but their site shows they retired it. Gravity Sync: Effective July 26, 2024, this project has been retired should continue to work with Pi-hole 5.x, but not with Pi-hole 6+ due to architecture changes.
Jesus, super video. I have a thinkcentre running proxmox and home assistant and paperless-ngx running on it beside my other Apple hardware and I thought I am advanced ...... 😅😅😅😅
Love the idea of keeping IOT devices contained like that. I never purchase any home automation hardware because I don't want it pinging servers with all my data
Portainer does have a GitOps option for Docker (Compose). Creating is free, but syncing is limited to the Business license since a couple years ago. Not sure if the last version before the licensing stuff has GitOps tho.
I thought my DNS was a little janky but it is similar to yours minus the colo. My physical third node is an RPi 3b+, and I felt like I needed something better, but it just works and has so for years. If it ain't broke..
I like the VIP idea. Does your router failover using...... ARP or.... DHCP extensions??!! Is ping involved?! Who needs to know about the VIPs for DNS? Doesn't your DHCP get confused or you proxy it? Multi-homed IP's seem to fascinate me no idea why.
My NASes use the most power and requires more battery backup so the NAS shuts down first in a power outage...so core home services are run on a low power device that can stay on for days during a power outage.
Great video, personally interesting in the backup section. How do you protect workloads outside of Proxmox? If there is state in PVCs within the Kubernetes cluster how do you protect these? How do you provide application consistent backups of your databases be it VM, Docker or Kubernetes based DBs?
Amazing Tour! I picked up so many good ideas that i need to implement into my PVE Cluster. One question (sorry, you maybe already mentioned it), do you use Proxmox HA or you just split all services into 3 instances across 3 nodes?
49:30 that the storage got corrupted has more to do with longhorn than anything else. I have stopped using it some time ago as I was unsatisfied with it (speed, reliability). If you'd use something like Piraeus (drbd with a zfs backing store) or ceph (much slower than drbd, but its really good), I bet you'd still happily run it in k8s.
I am at where you are saying you you will commit moving some applications to Truenas. One thing I would say, they don't have a way to put your applications to a separate network as far as I know. For example if you want certain Arr apps running via VPN this is no go. Again, as far as I know.
This is amazing!! One question about DNS though, why did you make primary DNS the one in a VM and not the rpi zero!? Is there a reason or just because you wanted it that way!?
@ Yup, and the VM was first and not until way later and many "Is the internet down" moments in my house did I decide to add this 3rd DNS so I can reboot my rack whenever I want.
@technotim can you make a video on Hugo and Jekyll for people who are not really coders but want to migrate away from Wordpress to maintain websites with just markdown, I really like your website.
Does nut server have a home assistant integration? Could be a stop gap until a more dedicated solution? Especially since that’s where you’ve already moved a lot of the monitoring there. HA might be able to handle the graphing instead * I see someone already mentioned this
about those databases: Isn't there some kind of a "meta" database service out there that can speak multiple dialects? I could swear I heard about that at some point but I can't remember for sure. If such thing exists, you could just deploy 3 instances of those and point all your applications to the respective load balancer.
I have a question. When you were just finishing high school, did you know any of the things you know right now because I want to enter the cyber security field and I don’t know if I need to have all the knowledge right now or if I can work my way through university😅
16:30 had the same Problem 😅 K3S wars crashing after some time, because etcd gets bigger and bigger and dose not respond in time witch kills the Custer.
For the camera vlan (protect) don't you have to allow access via a firewall rule to access your protect device? In my case, I had to create a firewall rules to allow the camera vlan to access my UCK-G2-Plus ip explicitly which is on the main (different) vlan.
A ZFS pool "5 x mirror - 2 wide" is only as strong as its weakest mirror group. If both drives in any one mirror group fail, the entire pool can be compromised...
34:12 @technotim How are you running protect locally? I didn’t think it was possible. Thought it had to go through the cloud. Is it because you’re using a UNVR?
@@TechnoTim I did that awhile back, but have not been able to get motion alerts working on iOS. Any ideas on that? Is it possible? I have thought about using homeassistant for motion alerts instead. (edited for grammer/missing word).
@Thanks for your response, I missed a word in my original question. I was trying to ask if you have a way to get motion alerts working on iOS when not using Unifi Cloud.
Thats alot of datastore expenses (cost of drives) for not even using 2%.. You could probs cut some costs of drive replacements by shrinking that down to like 2 - 3 TB lol. Crazy to watch dudes that run 50 different services, especially in colo. What all public services do you host? Anything cool that us followers could benefit from?
Flux in kubernetes will make sure your cluster doesn't drift from the state recorded in git. CI/CD will not do that. Still trying to dig in to whether portainer will stop drift.
I shudder to think of your power bill lol
You can see my power usage in the HomeLab Hardware Tour
Yeah if I ever setup something like this, I'd need to get solar first to help power it
I don't even know why I'd need half of this, since I can do everything I need right now with about 2 server devices and a NAS. But I know these things tend to grow.
The definition of scope creep 😂
An hour of interesting homelab information is exactly what I need this evening
Incredible. It's amazing what you can do when you don't support 100 people that can't remember their password. hahah - Wonderful setup, Tim!
And no printers
@ May your days be filled with people who remeber their passwords and printers that never quit.
@@TechnoTim Printers? Who needs printers nowadays? 😊
"A tale as old as time..." LOL I feel this.
Thanks for going into this. Many videos go over hardware and many more state the software they use, but very few actually explain the entire network and how it works. Thank you!
You’re welcome! Glad this was helpful.
Really interesting, thanks for the video and always being a large inspiration to expand my homelab!
I'm in the process of moving services the other way: from 50+ docker-compose containers on a Synology 920+ to a new Kubernetes cluster running Flux, very similar to you. I'm doing so because the NAS is completely overworked and I thought spreading out the load across multiple nodes would help with this - plus GitOps is awesome. Would love to see a future video where you talk about your experience moving these apps to TrueNAS. Keep up the great work!
Glad I had a minute to watch! Now do one on your favorite/top tools/plugins/automations (things that make your life exponentially easier that many of us may not know of)
You are amazing dude. I am just an engineer that wants to get more out of his homelab and you help me do this. Thank you.
Thanks for producing so many years of nice content!
Greetz from Germany!
13:04 In future videos, I request that you refer to these nodes as "nuc nuc nuc" and maybe follow it with a "whoob whoob whoob" while you shuffle. Thanks for the awesome tour!
Why I Oughta... 😂
@ I was gonna feel so (extra) old if you didn’t get that 😜
Thanks for the updates Tim, always love seeing other people’s setups. Legendary
Thank you. It's always great listening to your perspective and experience.
Hi Tim, thanks a bunch for the tour! cheers!
this was a movie it didn't even feel like the video was over an hour long. i can watch you talk about homelab for a long time 🤓
@@romayojr thanks man!
44:01 you could try out in portainer selecting creating a stack, selecting repository. fill in the relevant info under git repository and turn on gitops updates. any future changes/commits to your compose file in the future will update according to the duration you set in the fetch interval.
What an amazing homelab :) mine is just a single desktop and an OpenWRT router yet it has allowed me to learn a lot about IT, the CI/CD part and how you're handling updates is so amazing :) I will definitely try it
Amazing video mate, thanks for all the good work! You inspired me to start my own Home Lab. Also your tutorials are helpful. Keep it up!
Another year, another great overview! Got so much inspiration from your setup and videos :) keep it up!
Great video and inspiration Tim. All the best to you in the new year. Looking forward to new content. Cheers! 😃
Great video.
I’m absolutely with you whether to use mirrors or RAID.
My take is that mirrors just makes things less complex; it allows for easy expansion, which also includes increased resilience and performance.
Always love to see your setups. I’ve even borrowed a few ideas in the past. The only thing I do different is I have my wife’s work from home office in its own vlan that only has access to the internet.
Hahah. You literally talk about removing her laptop after posted.
Thanks for sharing - what are you using there for the network diagrams?
FYI . I didn't see nut server in your home assistant services. I use it for the 7 UPS I have running in my home to monitor all of them. Awesome video and homelab!
Great video! Love watching your tours when you put them out. I always end up finding something new to try out in my home lab. 1 video request, I haven't seen anybody put out an update VLAN guide with Unifi with there latest release and the new zone based way of doing it. I really want to make the step of VLANs but want to make sure I get the rules right specifically for things like HomePods and Apple Tvs that would be on Main but can talk to devices on IoT.
I always look forward to these videos
Portainer can do GitOps for docker compose
I had the same storage problem with sqlite and uptime kuma. I ended up running sqlite commands in a init container to configure my uptime kuma container. So I have a stateless version and still get notifications over my home assistant instance if something is down.
There was a problem... getting the right clean DB for the latest version of uptime kuma. So I have a init container starting the normal kuma image, waiting for the logs to tell me that is finished initializing the DB, then killing that container, then running my init container with the sqlite commands, then finally running the main container.
But it works :D
v2 will have MariaDB support!
thank you for uploading in 30fps, your videos are so much more watchable now xD
Good stuff man, I am also glad that you're staying humble. Def gave us all something to aspire to.
I eventually scaled down to my main media server being my container, a dedicated NAS server, and moved away from vms with a 1-2-3 back solution for data to consolidate hardware, heat, power, and equipment.
As for additional nodes I went with a few pi’s and mini pcs.
I have many implementations of GitOps with Docker Compose. Currently using Actions to trigger an Ansible playbook with some variables like what docker compose file and mapping that to a host, and it SSH's into the target machine and git pulls and docker compose up -d on the correct compose file and returns all the logs in the actions workflow.
Finally! the video I've been waiting for has been released.
I can't wait for the next major update to TrueNAS. In Fangtooth they looking to over haul the VM section to bring TrueNAS closer to other HyperVisors to go with their Very good NAS and now Docker services.
Great video! I like the Gravity Sync idea, but their site shows they retired it.
Gravity Sync:
Effective July 26, 2024, this project has been retired
should continue to work with Pi-hole 5.x, but not with Pi-hole 6+ due to architecture changes.
Thanks! There is now orbital sync!
Jesus, super video. I have a thinkcentre running proxmox and home assistant and paperless-ngx running on it beside my other Apple hardware and I thought I am advanced ...... 😅😅😅😅
you can add the nutserver to homeassistant and persist the data there
Love the idea of keeping IOT devices contained like that. I never purchase any home automation hardware because I don't want it pinging servers with all my data
"I think thats gonna change... I think thats gonna change" - me coming up with 101 ideas in my lab .
good luck man what a nice home server ❤
Portainer does have a GitOps option for Docker (Compose). Creating is free, but syncing is limited to the Business license since a couple years ago. Not sure if the last version before the licensing stuff has GitOps tho.
I thought my DNS was a little janky but it is similar to yours minus the colo. My physical third node is an RPi 3b+, and I felt like I needed something better, but it just works and has so for years. If it ain't broke..
I just clicked on it thinking cool a video before dinner and wooow, an hour no I first need to eat :D
I like the VIP idea. Does your router failover using...... ARP or.... DHCP extensions??!! Is ping involved?! Who needs to know about the VIPs for DNS? Doesn't your DHCP get confused or you proxy it? Multi-homed IP's seem to fascinate me no idea why.
Great video and the other one I look forward to each year. Gives me new ideas to think about. I'll also be buying more lottery tickets this year 😂
My NASes use the most power and requires more battery backup so the NAS shuts down first in a power outage...so core home services are run on a low power device that can stay on for days during a power outage.
Do you have video on did you setup your 3 pihole?
What network diagram software are you using?
ah, I love this homelab videos from you, buch of new ideas for my homelab, time to screw something :P
Awesome video thanks Tim!
Have you seen Beszel? Lightweight server monitoring hub with historical data, docker stats, and alerts.
Great video, personally interesting in the backup section.
How do you protect workloads outside of Proxmox?
If there is state in PVCs within the Kubernetes cluster how do you protect these?
How do you provide application consistent backups of your databases be it VM, Docker or Kubernetes based DBs?
Thanks for tour !
Amazing Tour!
I picked up so many good ideas that i need to implement into my PVE Cluster.
One question (sorry, you maybe already mentioned it), do you use Proxmox HA or you just split all services into 3 instances across 3 nodes?
I just split them across 3 nodes. I don't make my VMs HA, I make the services running on those VMs HA.
@@TechnoTim Interesting approach. Never thought to do that.
Thank you for answer!
I love your thumbnail man! and your video!
Thank you, I appreciate it!
Did you know you can use portainer business edition for free up to the 3 node for personal use
49:30 that the storage got corrupted has more to do with longhorn than anything else. I have stopped using it some time ago as I was unsatisfied with it (speed, reliability). If you'd use something like Piraeus (drbd with a zfs backing store) or ceph (much slower than drbd, but its really good), I bet you'd still happily run it in k8s.
Thank you! I am going to give it a shot again in the future! Maybe try another storage provider!
I am at where you are saying you you will commit moving some applications to Truenas. One thing I would say, they don't have a way to put your applications to a separate network as far as I know. For example if you want certain Arr apps running via VPN this is no go. Again, as far as I know.
Thank you for the heads up! I will keep this in mind when testing before I move.
I may have to watch this again to.fully understand what is happening.
Can you please make a Video about Pihole, the config sync and the virtual IPs?
I have one and it is linked in the description. There is one for syncing and one for HA (keepalived)
This is amazing!! One question about DNS though, why did you make primary DNS the one in a VM and not the rpi zero!? Is there a reason or just because you wanted it that way!?
more power I guess
@ Yup, and the VM was first and not until way later and many "Is the internet down" moments in my house did I decide to add this 3rd DNS so I can reboot my rack whenever I want.
I'll check it out. Thanks
@technotim can you make a video on Hugo and Jekyll for people who are not really coders but want to migrate away from Wordpress to maintain websites with just markdown, I really like your website.
I have a video on Jekyll and how I build my website, it's in the description!
I wonder if you tried Dawarich for your location history tracking/visualization? 🧐
Does nut server have a home assistant integration? Could be a stop gap until a more dedicated solution? Especially since that’s where you’ve already moved a lot of the monitoring there. HA might be able to handle the graphing instead
* I see someone already mentioned this
what kind of network diagram tools/software u use?. sorry if missed it. first time follow your video
about those databases: Isn't there some kind of a "meta" database service out there that can speak multiple dialects? I could swear I heard about that at some point but I can't remember for sure. If such thing exists, you could just deploy 3 instances of those and point all your applications to the respective load balancer.
You know portainer can do docker compose from a git repo?
I'm a software engineer who just bought a home and have a basic home lab setup and I just found my new man crush
@@superdupe8 software engineer here too! Infrastructure as a Hobby!
Portainer has a gitops feature that works with docker compose.
I have a question. When you were just finishing high school, did you know any of the things you know right now because I want to enter the cyber security field and I don’t know if I need to have all the knowledge right now or if I can work my way through university😅
Nice hoodie! 😎
I love it, thank you!
Does that garage rack stay reasonably clean and cool? Yes it’s enclosed but those racks aren’t air tight.
It does, yes, but everything in there is fanless. (mini pcs were only there for testing)
16:30 had the same Problem 😅
K3S wars crashing after some time, because etcd gets bigger and bigger and dose not respond in time witch kills the Custer.
The other most important video of the year!
For the camera vlan (protect) don't you have to allow access via a firewall rule to access your protect device? In my case, I had to create a firewall rules to allow the camera vlan to access my UCK-G2-Plus ip explicitly which is on the main (different) vlan.
there's a small typo in the yaml file in the rancher href variable shown at 10:37
12:56 I have been running unraid and trunas as a proxmox VM for over 4 years now
A ZFS pool "5 x mirror - 2 wide" is only as strong as its weakest mirror group. If both drives in any one mirror group fail, the entire pool can be compromised...
@@21Lettere for sure but highly unlikely. Let’s hope it doesn’t happen! 😅
What are the other better options for pihole?
I'm a noob on this world and I want to start using my raspberry pi for host adguard home and use traefik. Can you suggest any tutorial to do it?
34:12 @technotim How are you running protect locally? I didn’t think it was possible. Thought it had to go through the cloud. Is it because you’re using a UNVR?
All Protect can be local only, you just have to access your NVR/UDM via local IP and turn on local access only!
@@TechnoTim I did that awhile back, but have not been able to get motion alerts working on iOS. Any ideas on that? Is it possible? I have thought about using homeassistant for motion alerts instead. (edited for grammer/missing word).
@ not sure, you could disable them to be sure. Maybe it was because they were configured at one point?
@Thanks for your response, I missed a word in my original question. I was trying to ask if you have a way to get motion alerts working on iOS when not using Unifi Cloud.
@@john-seaunfortunately you have to enable remote access to get push notifications.
Hey Tim, what software are you using for the diagrams?
Excalidraw! I will have more info on this soon!
@ I'll be waiting for it! Thanks for replying!
insane. what amount maintenance does this require a week?
@@Lenny-zn8hn actually 0 unless I need to patch everything, then maybe 15 minutes with reboot dances
It`s very cool!!
Have you thought about hosting something in your garage, like your pi zero?
Good idea, something low power and fanless wound be great!
I put NUT server into home assistant for long term stats
Which software did you use to illustrate the VLANs?
Thats alot of datastore expenses (cost of drives) for not even using 2%.. You could probs cut some costs of drive replacements by shrinking that down to like 2 - 3 TB lol. Crazy to watch dudes that run 50 different services, especially in colo. What all public services do you host? Anything cool that us followers could benefit from?
This is giving me a bunch of new ideas!
Hey Tim, love your setup! What are you using for the diagramming tools?
Excalidraw!
33:19 wasnt gravity sync archive last year?
a doorbell using wifi? isn't easy to isolate it and get your iot devices wifi through it?
For my VMs that have gpu pass through I’ll name them with suffix like i915 or A310. Also labels in proxmox are 🔥
But your red labels are throwing me off.
Oh shoot. I have gitops for docker compose (gitlab ci/cd via portainer api) in my homelab... 😬
Flux in kubernetes will make sure your cluster doesn't drift from the state recorded in git. CI/CD will not do that. Still trying to dig in to whether portainer will stop drift.
@@minifig404 Portainer website says "Ensures consistent, repeatable deployments by reducing drift", but I'm not sure whether it entirely stops drift.
I think I know the answer to this question but will ask anyway. Are you running the container version of Home Assistant?
I am :) Yes!
8:39 what site to site vpn did you use?
What Diagram tool are you using?
Can we please have something about that Rx fault? I have the same issue and can't find any solutions...
It was so weird. I ended up rebooting the machine and that fixed it. I haven't had it again since.
I got excited when you mentioned gravity sync but then disappointed when I saw that it’s orphaned and will def break at some point
Addendum: Orbital Sync may be the solution.
It is!
Hey how do you access services on different VLAN? via internet and not LAN?
firewall rules i guess...
Freaking awesome 👏