Thanks! I have more real world examples coming up soon! In fact, most of my guides are based on real world examples (I like to base them on real examples that I have done for my own home network either currently or in the past and sometimes I create examples in a lab environment to try new things and to verify the process works properly).
This is great, easy to follow. I'm a complete noob and got the parameters for opnsense set up on my proxmox. Step by step i go slow, but things are looking good! 🎉🎉
Thank you for showing the physical real world implementation. As a beginner, I've always struggled with this and this is the only video that shows from "zero to one to 100"
Yeah you can if you don’t plan to use the untagged parent interface. Since I use a different interface for the LAN for untagged traffic, I don’t need a second untagged interface and just only need to use VLANs on that second interface for just tagged traffic.
I will note one potential gotcha that I encountered when testing out some things. If you want the VLAN interfaces to use a MTU that is higher than the default 1500 used by all interfaces (to enable jumbo frames with MTU of 9000, for instance), you will need to have the parent interface assigned and enabled so that you can set the MTU value on the parent interface. This is likely a rare scenario since typically jumbo frames are used on isolated networks with higher speed interfaces (10Gbps+) rather that for routing traffic across 2 networks with larger frame/packet sizes.
Great detailed overview for anyone starting out. Just some comments on the options when setting up VMs in Proxmox: If you're on using thin provisioned storage, you always want "Discard" checked. It's what makes the guest OS emit TRIM commands necessary to actually free space on the host that was free'd in the VM. SSD emulation just tells the guest that it's flash storage, it doesn't enable TRIM. I have no idea why this isn't a default setting. For anyone setting up a single host, so not a Proxmox cluster: Just always use "host" as the CPU type for a measurable performance gain. All features and abilities are passed through accordingly, and there is no need to enable or disable instruction sets like AES, and there won't happen any translation either. This setting does imply that a VM can't be live-migrated using HA on a cluster, which means the VM is switched from one Proxmox host to another WHILE RUNNING. This is an incredibly rare requirement for a home lab. Even when "host" is selected, it's perfectly fine to shut down a VM, transfer it to another host, then start it again: the "host" CPU will just change meaning during the transfer to represent the other hosts CPU, no problem. Finally, when virtualizing a firewall, it is highly recommended to pass through the actual PCIe hardware directly so it has direct access to the hardware. Yes, this does make a difference. No, it's probably not critical in a home lab, but if you're using 10g you probably want to use that. If it matters, or how much, when using 1g networking depends on the hardware (both system/platform and network). The only exception for me would be when using RealTek network cards. Anything based on BSD (pfSense, OPNsense) has bad enough compatibility that having Proxmox (Linux) handle it and use a bridge like you have shown.
Thanks for the info! Someone already mentioned several of those points and I also made an addendum follow up video pointing out a few settings you could do differently. I’m now running a Proxmox cluster and I like using the live migration feature for my OPNsense VM so I can reboot the Proxmox system for updates without taking the network down (I demonstrated this in my cluster video). It works amazingly well. I was surprised not doesn’t even drop the existing connections. It only adds a slight delay when pinging when the cutover occurs. I don’t think live migrations has to be a rare use case for home labs, haha. Because I have multi-homed my NAS and use dedicated 10G backend networks, I don’t have much of a need to transfer large amounts of data across VLANs so leaving everything as bridges is fine for my use case even though there is a performance hit. It’s not bottlenecking my network (most of my devices are 1G so even bridges can handle that no problem). My internet is about 1.2Gbps down and about 25Mbps up and I can still use Zenarmor with a bridge with no hit to throughput. I’ve measured I can get up to 2.3Gbps on a bridge with my hardware with Zenarmor so I’d only have issues above 2.5G interfaces (which isn’t a problem as I’ve mentioned since my high throughput devices are on the same networks or connected to a dedicated, isolated storage network). With that said it’s good for others to be aware of all of those things you mentioned! It’s good to know the caveats and/or optimal settings depending on the use cases.
This is the video that gave me the reassurance to switch my own home network over from firewalla to a virtualized OPNsense instance this past weekend. It genuinely surprised me that it was a clean cutover with all of my vlans/APs, Thank you! OPNsense has 4 Performance cores of a 14700T, 32GBs of ram and a bridged Intel X550 T2 dedicated.
Great to hear! Glad it gave you reassurance! Make sure you have a good backup plan if you only have one Proxmox server (but even if you have a bare metal installation, it’s good to have a backup plan). If all is configured properly the virtualized instance should function essentially the same as bare metal as you have discovered!
Fantastic video. I learned a ton watching and following along. Thank you so much. I appreciated you walking through each option and briefly discussing why or why not you had chosen said option. Cheers!
Thanks! Glad you liked it! I think it’s helpful to explain the options instead of just picking them. I tend to go in more detail in written guides on my website. I have to be a little more terse in videos to try to stay on topic and keep the length shorter.
Thanks! I hope I covered enough to help people along. It’s a lot of info to cover (and there could be even more but I tried to keep the length somewhat reasonable). Takes a lot of time to produce content in general, let alone during your limited free time. Haha.
Fantastic that you release this video literally the day i get everything together to do exactly this myself, you also helped me with the PCI pass through that nobody else talks about. Thankyou!
That's great! I'm glad the timing worked out. Sometimes I'm just in time for some users and too late for others. haha. I thought I would mention PCI passthrough in the video even though I didn't do it in the video to keep things a bit simple but I also tried to ensure that the instructions should still work if you plan to use a Proxmox cluster. Things get more complicated when doing PCI passthrough with a cluster. I have yet to try all that out as well. Bridges are safer and you will only notice performance issues with 10G interfaces or faster. You can still get 5-6Gbps with the VP6650 I used in the video so it's still faster than the 2.5G interfaces (and really you should try to not route 10G NAS and other traffic when possible to reduce the load on the firewall by having a separate 10G network).
Maybe use the managed switch and create a WAN subnet using a VLAN 🤔 connect the WAN cable to the switch and then any Proxmox node can access the Internet VLAN for a virtual bridge ?!? Just a thought. Might be more complex using the newest SDN feature on Proxmox . . . Guess it's time to experiment around a bit . . . Great work 👍🏻👍🏻
@@homenetworkguy , do you have any thoughts on IPFire ? For example, can I use it to achieve something similar to your "opnsense for beginner" video/post ?
I have thought about learning more about other firewalls (OpenWRT, IPFire, etc) once I have exhausted the main topics I want to cover in OPNsense but after writing on my website for nearly 6 years (and more recently, RUclips videos), I still haven't exhausted everything I'd like to learn about. haha. I think IPFire could be a good Linux based alternative. There are a lot of similar features but also some things it doesn't offer via plugins. I would like to test out the performance of it because it's possible Linux could perform better than FreeBSD depending on driver support, etc.
thanks for all the opnsense and proxmox content. As a opnsense / Truenas scale home user and a vmware enterprise user @ work i enjoy all this content. Proxmox and ncp-ng are in our work test labs for possible move to from vmware. Thank you again!
Thanks! I'm glad you appreciate it! I hope to dig more into Proxmox clustering with OPNsense and how I think I'm going to go about it on my home network so that I can do live migrations (it will be very awesome to have the ability to move my main router/firewall over to a different physical machine with only a split second blip in downtime for my network!). I don't care about high availability/failover as much as being able to live migrate the VMs (because with VMs it's easy to restore from a backup from my PBS system, which is another nice piece of software). The configuration and requirements for live migrations is less intense which I think will suite my needs perfectly.
Thanks! I try to explain how and why you need to do certain things without getting too deep into the weeds. I like to think it’s like teaching you how to operate all the controls in a vehicle rather than how everything works under the hood. Of course the more you know under the hood, the more things you can do.
You’re welcome! I plan to into configuring a Proxmox cluster soon and show how you can live migrate OPNsense to different nodes with very minimal downtime.
I lust got my PVE/ OPNSense machine running and in my rack a couple days ago, and I just found this today! I also used your Pi Hole PVE guide and set that as my DSN server. I used an 8th gen Dell OptiPlex with a dual 2.5gb card, and am thinking of setting up a second machine for a HA cluster.
Nice! If you set up a cluster with 2 nodes, you need to make sure you have a 3rd device as a “Q” device (a 3rd voting member) so you can have quorum. You need an odd number of devices so you can reliably know which nodes are available.
Thanks! I intended on creating a diagram in this video (and some others as well), but I had a lot on my plate and wanted to get it out there. I’d like to spend more time doing polishing the videos further, but it would take me 2-3 months instead of 2-3 weeks per video. Haha. (I only do this in my ‘spare’ time). If I can get caught up on some things I’ll try to do better about including more diagrams in the future even if they are not super fancy.
You should enable "Discard" (for trim) for thin-provisioning to work properly. If you disable "Pre-Enroll keys" then Secure Boot won't be enabled so there's no need to disable it later. OPNsense (and pfSense) recommend to disable all off-loading settings. At least for virtual NICs.
Thanks for those tips! I should’ve looked up Discard to better understand if it was necessary or not. Funny thing is that the pfSense documentation shows to do it that way for disabling Secure Boot (docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html#booting-uefi). I figured their docs would also work fine for OPNsense for recommendations for VMs. Since I always use OPNsense in a VM for demo/testing purposes I didn’t care about optimal settings as much but if I use it as my main router/firewall, it becomes more important! Hardware offloading is disabled by default in OPNsense which is why I never typically mention doing it. I think for pfSense it may be enabled by default.
@@homenetworkguy Yeah that's funny but one can't know everything. The PVE docs (I apparently can't link things without the comment being deleted) say this > pre-enroll-keys specifies if the efidisk should come pre-loaded with distribution-specific and Microsoft Standard Secure Boot keys. It also enables Secure Boot by default (though it can still be disabled in the OVMF menu within the VM). To elaborate on the discard as far as I understand it. On most linux OSs there's a weekly "fstrim" timer which calls "fstrim" which gives unused chunks back to the underlying storage. Assuming the virtual disk is on thin-allocated storage and "Discard" is enabled, of course. I believe windows also needs the "SSD emulation" option. I'm not sure how pfSense/OPNsense/FreeBSD handle trimming. I'm very far from an expert with BSD. Trim seems to be disabled in my OPNsense VM according to "tunefs -p" but I'd recommend to enable "Discard" for every disk on thin-allocated storage.
Yes, I appreciate when others let me know about details such as this so I can continue to learn as well. To clarify from what I looked into this morning-- without discard enabled, the initial VM storage doesn't take up the full 64 GB when I looked at the disk usage. It's sitting at 3GB and I have a few CTs set up as well. However, I'm assuming discard will help free up space on the host when data is deleted from within the VM. It's good to know that it doesn't fully allocate the 64 GB even if discard is left disabled. I'm not sure how trim is handled in OPNsense either.. I think I've seen others talk about it at some point but not sure if it is something that needs to be enabled to make it function properly.
Yep you still need to have virtualization features enabled in the BIOS but if you don’t also enable IOMMU on Proxmox, only raw device pasthrough is available.
Thanks! I’m planning to expand upon this and show clustering in Proxmox. I will demonstrate how to manually live migrate the VM to another Proxmox node as well.
If you want to do this with VM's, I strongly suggest using Triton on SmartOS instead: you get IPFilter, Solaris zones technology, mdb, DTrace, ZFS, Fault Management Architecture (FMA), Service Management Facility (SMF, like systemd but way better) and the Crossbow network virtualization, which allows one to create virtual switches and routers, and yet everything runs on bare metal because Solaris zones virtualize the kernel and not the hardware (and yet still enable one to virtualize other operating systems like Windows, GNU/Linux and FreeBSD with both KVM and Bhyve).
Yeah there are lots of solutions out there you can run but the video is not focused on the option you described. Thanks for the suggestion though. I can look into to it further at some point.
good video. One thing you touched on but did not get into is if your PVE (with OPNSense) goes down, you lose your router. It would be good to understand how you would migrate this over to a second PVE without losing routing. I suspect you would need a machine with the same number of LAN ports which have the same virtual bridge names in order for it to migrate properly. (I want to use OPNSense but I want to be able to migrate it between PVEs in a cluster).
I mentioned that you need a backup plan if you only run a single node since it will take your network down which I mentioned near the beginning about how I prefer bare metal because I’m considering using a Proxmox cluster so I will feel more comfortable about virtualizing OPNsense for my primary router/firewall. I plan to show my cluster configuration in the future. It will be pretty awesome to be able to live migrate my primary router/firewall with less than 1 second downtime!
@@homenetworkguy I always double router... I keep the ISP provided router in front with family wifi. Then have a proxmox/opnsense router behind, so I have my own network I can freely break without affecting the family. Which is good because sometimes I break it a lot 😅 I've heard double router can cause problems but so far I've never faced a single issue caused by double router so not sure what that's about
@@CowCow-o5m I also play around with OPNsense VMs on a separate lab network for the same reasons. I try to keep the main network stable for my family and also because I work from home (and my wife does some work from home too). Having a separate lab network is nice because I can play around with stuff so I can make guides/videos and I don’t get tech support tickets if something breaks. Haha. But I will move to a virtualized OPNsense once I set up a Proxmox cluster because it will provide me with more redundancy so I will feel more comfortable virtualizing the main router. It will allow me to migrate to different hardware much more easily since I tinker with different mini-PCs and other hardware on a regular basis.
@@CowCow-o5m yep when you have a family, you are basicly on -call 18 hours a day 7 days a weeek, the more complicated, you have as part of the family networking, the more likely it wil break, and of course it always breaks when you are the busiest. If do this i'm going to use opensense on a virtual network that doesn't leave the system then i can have more bandwith between VMs and I can play around with rate limiting, and other firewall features.
Even if you only start with one proxmox host it is advisable to create a cluster before creating the 1st VM. Not used v8 but this was the case with v6 and v7, A host with a VM cannot join a cluster.
Good point I hadn’t considered yet. I haven’t created my cluster yet but plan too soon. I can easily back everything up to my PBS system and restore it back on the cluster.
I looked into this further. The primary mode where you create the cluster can have VMs/CTs running but any new nodes that you are adding to the cluster must be empty to avoid naming conflicts between nodes. Makes sense. I was worried I would have to start over with a clean slate to create a cluster. I have backups on PBS so it’s easy enough to start over if need be.
Thanks for this. I'm definitely wanting to setup Opnsense and Proxmox, I just don't know what on. I like the chassis design and ports on these Protectli units, but god they're expensive. The Minisforum MS-01 gives you a mobile i9, the same two SFP+ ports (it's even the same model of Intel NIC), two 2.5G RJ45 ports (also same model of Intel NIC), two USB 4.0 ports that can do 40 Gbps, three NVMe slots (albeit only one of them is PCIe 4.0 x4) instead of an NVMe and 2 SATA slots, for like $220 less than this. If you get the i5 version (which still has a better CPU than this one) it's $460 less. It's pretty ridiculous how expensive this thing is to only have an i5 in it. I'm not sure the extra 2 RJ45 ports, better chassis, better firmware support is worth paying so much more to lose out on hardware. It's quite a dilemma. I only have 1 Gbps for now, so realistically I'd be fine with one of the cheap Protectli boxes if I was going with barebones Opnsense on it (aside from running ZenArmor and such), but I want to upgrade to 10G LAN at some point so I'd like to have the support for it to make routing between VLANs faster among other things.
Yeah you have to weigh the pros and cons. I definitely wouldn’t use the MS-01 as a dedicated OPNsense box. Systems like these are too powerful not to use virtualization to make full use of the hardware. Not all of the services in OPNsense take full advantage of all the cores. In fact some of them may fight for the same couple of CPU cores (the Zenarmor team has noted as much to me).
In your network setup i see there is a cable which connects port 1 of protectli to the port 2 on the switch. Why do you need that when you 10gb SFP+ connected between them?
I like dedicating one interface to manage my Proxmox server so I can plug directly into it, if need be. I’m using the 10G interface for all of the VLANs on my network. You don’t have to do it this way. You can use a single interface for everything. I have the network interfaces to spare so it’s easy enough to use it that way. This also allows you to separate tagged and untagged network traffic as well which is recommended by OPNsense because there is the possibility of allowing traffic intended for the parent interface to the associated VLANs on the same interface (if you’re not careful with how you write the firewall rules and it may also require a network switch which have a specific flaw).
The DHCP protocol RFC specifies that the DHCP client is supposed to solicit address offers, which implies that multiple DHCP servers should be on the network, and indeed, if one is managing one's network with the dynamic host configuration protocol, one should have as many DHCP servers as possible to increase redundancy and availability, but no less than three (so there can always be quorum).
When I say you shouldn’t have multiple DHCP servers on the same network, I’m not referring to high availability but rather 2 different DHCP servers running on 2 different systems not in a HA configuration. There will be conflicts of IP address assignments if you have a rogue DHCP server running on the network if they’re using the same IP ranges. I imagine there are ways to mitigate this but I’m always speaking from a home network context (hence the name of the channel). You don’t really need have a minimum of 3 DHCP servers on a home network (I’m counting DHCP listening on multiple interfaces in OPNsense as one DHCP server). I’ve survived 25+ years of home networking using a single DHCP server. Typically I have other problems that are not DHCP related.
@@homenetworkguy that's exactly what I wrote about: one should have multiple DHCP servers for the same network, even if that network is a home network. The challenge, of course, is keeping them in sync. I keep my configuration packaged in OS packaging format in Git, so when I modify it, I bump the package revision and upgrade the configuration package on all the DHCP servers. Problem solved!
Yeah, that's a great solution for high availability, but I typically throw out the general warning that if you spin up another OPNsense or plug another device on the network which has its own DHCP server running on the network (that is NOT synchronized) that it could cause problems. What you are describing is syncing up the DHCP servers which is an intentional action to have redundancy on the network, which is not a bad idea for improving the reliability of the network when services or hardware fails, and shouldn't cause problems on the network when implemented properly. I've spun up VMs and or connected another OPNsense box to the network when testing and it would temporarily disrupt my network until I disconnected it because I accidentally connected the wrong interfaces or had some configuration incorrect.
I have 8g symmetrical at home, as a non network guy, if I want to use IDS/IPS and pihole / unbound dns + wireguard. is that something Protectli VP6650 can handle. I don't know how much power you really need. I most likely won't vlan too much more of a simple router -> switch to nas and computers and then router-> 2.5 directly link to nas port for DMZ sharing
Without IDS/IPS, it shouldn’t be a problem but it can’t do IDS/IPS on OPNsense at 8 Gbps because not all of those services are fully optimized to take advantage of all the cores on the CPU. You may potentially have better luck with other operating systems. I haven’t tried other firewalls such as IPFire yet. It’s Linux based so it may perform better. I should try it before I start using the VP6650 in my future Proxmox cluster.
@@homenetworkguy would love to see it. i wish there was a chart that just said you need x for y feature somewhere, the information is always vague or refers to buying some enterprise level hardware, pretty sure my wife would not be happy to that purchase. vs something smaller one could build out ^_^
I’ve thought about creating a chart/table for the hardware I have personally tested to help others determine how much hardware they need for certain services in OPNsense. I wasn’t able to test all of the older boxes I have quite as thoroughly but it’s getting easier for me to set up test cases since I have more sponsored hardware and other hardware that I purchased available for testing.
Haha, I take it that you really liked the video. I have one that goes one step further with creating a basic Proxmox cluster that’s dropping soon. It’s very cool to be able to live migrate your router/firewall without the network connection noticeably dropping.
@@homenetworkguy Yes, I watched it too!!! you are awesome man, I hope you know you are doing an amazing service to the community!!! God bless you and prosper you!
Excellent video 👍🏻 I needed this 6 months ago (figured it out the hard way!) 🙄 Have a smoothly running virtual opnsense on an R86s for some time now 👍🏻 quick question, I have a cluster of nodes and want a fallback scenario in case main node with opnsense dies - how would you propose moving the virtual instance to a different node and still keep network settings?!? 🤔 Might make for a great follow up video ?!? 👍🏻 keep up the great work . . .
Thanks! I’m planning to show I will do this in a cluster. With the limited research I’ve done, you would want to ensure the bridge names are the same on both nodes so the 2 machines would need to be configured similarly in that regard. Also if you’re not using shared storage, you would need to restore from a backup (and there might be a step to “manually migrate” the VM to a different node by messing with the config files since the VM wasn’t migrated while the node was still alive- not sure about that one yet until I try it out and/or do more research).
From experience, even running on a multi node cluster with full DRS running, virtualising your firewall is not a good idea in an home lab. It sounds like a good idea, its a good project to get your head around, but just don’t do it. Save yourself a world of pain. Thats said, this is probably the best Proxmox setup video for new users I have seen.
What kind of pain? In planning to mostly keep OPNsense on one on the nodes so I can live migrate it. I’m not going to do any of the high availability features nor mess with shared storage or Ceph to keep it simple as possible. I just want to be able to move VMs between nodes if I take a node down for maintenance or if it fails. I’m not concerned with automation failover scenarios which is another reason (among other reasons) I haven’t implemented high availability with OPNsense itself.
@@homenetworkguy If anything at all goes wrong with your host infrastructure, either physically or you make with a mistake your config then you lose your connectivity. In an enterprise environment which is strictly change controlled then I am happy with virtual firewalls, but in a home environment unless you have similar controls, built and proofed in an dev environment and then rolled out to production, invariably you will make a mistake, mess up a VLAN assignment, trunk, host or the OPsense VM and then you are dead in the water as you will have no connectivity across your VLANs and no internet connectivity. That was my experience and attempting to get my environment back up and running at 4am in the morning and was not fun. It looks like those who have this working as a solid solution have a much better at home based change control than me. Love your video's BTW and thank you for this video in particular.
Yeah, I understand the need for tight control for configuration management in the enterprise, but home networks typically aren't nearly as complex so it should be easier to manage. I don't make major architecture changes very often but I plan for some down time when I do. Also Proxmox clusters can be relatively simple and not be configured with all of the high availability features. At the bare minimum, you can simply group systems together so you can manage them all from a single UI and you can migrate VMs between them. That's mostly what I would be interested in because it's quicker than backing up VM, shutting it down, and restoring the VM on a different independent Proxmox node (if not using clustering). There is a less than 1 second cutover from what I have seen from others which is pretty sweet. Since you mentioned DRS, you might be more familiar with the VMware world which perhaps may be more complex to configure/manage clusters (I don't have personal experience in that area). I'm going to give a Proxmox cluster a shot soon, but I could always keep an extra box with a bare metal installation to swap out if need be. Wouldn't hurt to have a hardware backup!
DETAILS, details! “It’s hard, complicated and error prone!” (Only for “some”). I did run my main pfsense, plus 2 more for HA, under ESXi, for a few years and there was NO SUCH PAIN! The main reason that I run pfsense on a dedicated machine, is because I found cheap used quad core mini PCs that work perfect. The “people” that utter vague claims like this, usually don’t know the stuff well!
what would you see as some other VMs running on that box? Zabbix, Plex, maybe a web/mail server in DMZ? nakivo backup solution? how many more can you have on 4 cores? did you test it?
I completely replaced my old Proxmox server with this Protectli and it runs everything even better than my old server which was a Ryzen 7 1700. I only use 4 cores for the OPNsense VM since I noticed it doesn't tend to use much more than that. I have most of my hosted network services running on the box such as Plex, Nextcloud, Caddy reverse proxy, Vaultwarden, UniFi Controller, Grandstream GWN Manager, RustDesk, Uptime Kuma, Homepage dashboards, etc. I haven't even reached full capacity yet. It runs in a Proxmox cluster so I have stuff running on different nodes for various purposes (one is mostly dedicated to Home Assistant while the 3rd node hosts all of my apps/VMs I use on my LAB network).
@@homenetworkguy man this was perfect and honestly I appreciate the content it’s helpful for poeple who want to try this and the examples and explanations is perfect for beginners . Will be showing my friend as well who’s trying this to
Thanks! Sorry about that. Sometimes it’s easier to remote into another machine but sometimes I also use SPICE for my Proxmox VMs which doesn’t have the second mouse cursor/delay.
should i use this linux bridging or passthrough the nics? i want a mini pc with 2 nics. 1 for wan and 1 going to my switch were i use vlans. still need to figure out how i can use the vlans etc.
If you need maximum performance, you use can use passthrough. Otherwise bridging will be fine if it doesn’t hinder throughput (depends on the speed of the CPU on your system). If you are using a cluster, you would need to be careful when using passthrough especially if using different hardware. There is a resource mapping option for the cluster but I haven’t tested it to see how well that works when live migrating VMs (and haven’t tested full high availability either). I mainly keep my cluster simple and manually do live migrations when I need to reboot one of the nodes that has my OPNsense VM.
Is it possible to have 2 VMs of OPNsense working as Active/Passive mode? For high availability and reliability on home network? If so, how it would be look like? Would you pls consider making a video for that? You can mention me as “Abu Rayyan from Baghdad” next time 😅 never been called out on RUclips algorithm 😂
Yes but it’s almost has value to do so especially if it’s running on the same Proxmox server. VMs are very easy to backup and restore and you can take advantage of deduplicated snapshots with Proxmox Backup Server as well to get you back up and running quickly if something goes wrong. Could make a HA video because it’s interesting to learn even though I wouldn’t personally use it especially since I only have 1 public IPv4 address.
@@homenetworkguy I did some research and it should be possible and it's more ways to do it. And 1 public IP is enough, HA is good for HW failure too. It's my future plan, after I end with this vlans etc. sht I would like to learn and understand properly.
i feel like it's not really your "primary router" if proxmox is still in front of the OPNsense router and using the wan for management, i did it today with proxmox behind OPNsense and it's much safer, just not sure how to set up the pve>system>network,DNS,certificates thing as i am absolutely new to proxmox
It is your primary router but just virtualized. You can plug your modem/ONT directly into the interface used as WAN on Proxmox just like you would on a bare metal installation plugging into the WAN interface. Proxmox is not doing any of the routing or firewalling for your network-- OPNsense in the VM is doing that task. This is the nature of virtualization. Proxmox is not "in front" of the OPNsense router. Rather, Proxmox is simply hosting the router/firewall software in a virtual machine (all routed network traffic flows through that VM just like a bare metal installation). I am currently using a bare metal installation of OPNsense, but I will probably move to a virtualized installation (in a Proxmox cluster) so that I can have more flexiblity to "move" my router to different hardware without doing a separate bare metal installation. I can just migrate it over to a different machine. Since I test out various hardware, that flexibility will be great to have. As far as security is concerned, the main security risk with virtualization vs bare metal is escaping the VM sandbox. If an attacker can break out of the VM, they can get on the host system. Those sorts of attacks are very rare. Other than that, the security is generally pretty much the same. I understand virtualization is not for everyone. I have guides that show both bare metal and virtualized instances of OPNsense.
@@homenetworkguy did you notice that after installing opnsense and setting it up as the main proxmox router that pve>system>network,DNS,certificates etc have to be changed to match the new network?
I could do that even though I'm not an expert on PCI passthrough. However, passthrough of NICs is pretty straightforward, but what I would like to learn is using SR-IOV which basically allows you to passthrough NICs to multiple VMs at the same time (kinda like how you can partition up certain GPUs to multiple VMs). I believe newer versions of Proxmox already have IOMMU enabled so that saves some steps when passing through hardware.
@@homenetworkguy would love to see the whole SR-IOV side. I can get PCIe passthrough to work...but once done how do I use the VFs to assign to other VMs is where I get hung up.
Hi once again thank you for everything. Small question wrt VLANs; let’s assume as in the above video, a Nic / bridge is made VLAN aware, and then connected to a switch that has say 5 physical ports and each port is being used for 5 separate VLANs (VLAN id 2, 3, 4, 5, 6) Let’s say we create a Proxmox CT or VM but want it to have a new VLAN id 7; do we need to do anything on the switch itself ? Or should we just input 7 as vlan id in the CT/Vm. Are the number of VLANs restricted to the number of physical ports on the switch. Would really appreciate your views on this .
First of all, 1 of the 5 ports will need to be connected to Proxmox and that port will have ALL VLANs assigned as a trunk port so the traffic can pass through to Proxmox. A trunk port can have as many VLANs as you want to pass through to other switches, routers, wireless access points, and servers (all of which need to be VLAN aware devices). Each non-trunk port can only be assigned to a single VLAN. In Proxmox you can create a virtual bridge (with or without VLAN tags) that you can use as a virtual network within the Proxmox server or Proxmox cluster- if you wish to have CTs/VMs on their own virtualized network (this can be helpful for lab networks, etc).
@@homenetworkguy the last sentence is what I want to do; create virtualized CTs in 3 Vlans, 1 each for Caddy, Apps, and Arr stack. I created a Vlan aware Bridge that wasn’t connected to any NIC, and all was working fine, but the setup is causing OpnSense to crash / restart. I’m now thinking of using my last remaining Nic, making a VLAN aware bridge, creating 7-8 VLANs and trunking them to my Mikrotik Hex which will only have 4 available ports, to be used to separate physical devices such as office, Iot, guest etc. The only 3 VLANs to be used for Caddy, Apps, and the Stack. Am totally lost here.
Yeah, basically I had allow all rules on every network. I had to decide if I wanted the video to be 40 minutes or 1.5 hours, etc to show a full build (which I have done twice already.. I may do a 3rd in the future as I slowly work to improve overall production quality, etc).
thank you for the video one question, after configuring opnsense, and use it as main router connected to our network, opnsense is the new dhcp server? is something we have to configure it? before was de ISP router...how do we manage the ip given to proxmox for management if we change network with new opnsense router? thank you
You're welcome! Yes, OPNsense will be the new DHCP server. By default the LAN interface of OPNsense has DHCP configured but you need to configure DHCP for each new interface that you add (physical interface or virtual interface aka VLANs). In the example, I made the Proxmox management IP an IP address in the LAN network of OPNsense so it should still work fine once you switch over to OPNsense as your primary router.
Thanks so much for your videos; I have a problem however that after executing a Proxmox apt update/upgrade; I can't access the OPNSense Gateway 192.168.1.1; I am on the LAN network and can ping other devices on the Lan, access the internet, but just can't seem to ping the Gateway or access the Web Interface. Should I have not restricted the Allowed Interfaces to Lan (only). Any advise would be appreciated;
I have not encountered that issue after updating or rebooting Proxmox or the OPNsense VM. It’s hard to say what happened with knowing more details because if everything is on the same network using the same bridge for the LAN/Proxmox management interface, you should be golden.
Yeah you could dedicate one interface for the WAN but the other interface will have to be the Proxmox management interface, the LAN interface and if you want any additional VLANs. You could experience some bottlenecks using a single interface. The configuration will be a bit different than what I demonstrate but the concepts should be the same (you would just use the same bridge interface for all internal networks instead of separate interface(s)).
What did you mean when you say "we are not plugged in" at 32:49? I have one built in NIC and I use 2 USB to ethernet adapters to create 2 more network devices. All 3 are connected to the same switch, so I am plugged in. I see LAN "192.168.1.1/24" and for WAN "192.168.0.204/24". And I can't visit the OpnSense UI on either IP. I know I am doing something wrong. Tried the same steps 3 times.
Depending how you are configuring your devices, it’s sometimes best to not have everything plugged into your existing network because there could be IP address conflicts or you could end up with 2 DHCP servers running on the same network, etc. I believe I have the system I’m configuring OPNsense with plugged directly into the management interface of Proxmox but I manually set a static IP address on that system. You should be able to access the Proxmox web interface as well as the OPNsense web interface if you are using the same bridge for the OPNsense VM.
Is there a way to give Proxmox itself internet access after committing Opnsense as a main router? I'm having issues with updates and LXC installations, as Proxmox keeps returning no internet errors
Thank you so much for your videos. I have gone through your first and this video for configuring OPNsense on Proxmox. Everything works great as you have explained. But I am trying to configure LAGG (LACP) and didn't get success in this approach. Is it possible to do LACP with this approach?
You’re welcome, thanks! Yeah I’m sure you could, but it’s possible to create the LAGG on either Proxmox or OPNsense so I’m not quite sure which would be the best approach (you need to it on on side or the other but not both).
Yeah it’s possible you would need to passthrough the physical interface for it to work in OPNsense so it can have direct access to the network interfaces. But if you do it in Proxmox, I’m thinking you could use the LAGG interface of Proxmox in the OPNsense VM and treat it like a normal single physical interface in OPNsense. I haven’t tried that out so I’m not sure how all that would work. Hah
@@homenetworkguy Yes, I think it will work if I pass through the NIC to OPNsense VM. But I am using a Realtek card, so I am trying it through Proxmox. And also, I am setting up VLAN and want to use this VLAN in other VMs and CTs. Let me try the LAGG in Proxmox and see how it goes.
You are plugging the Zima board into the LAN port of your virtual firewall, so you can get to the GUI correct? I have PCI passthrough on my OPNsense VM but I should be able to just plug my laptop into the port that has my LAN and get going correct? PCI passthrough should not change anything on me?
In that case it would be better to connect a network switch to that interface and have your client plugged into a port on the switch configured with the LAN’s VLAN ID. Unless you can set a VLAN tag from the client if you wish to plug directly into that interface.
@@homenetworkguy figured it out. Needed to set the VLAN on the PC and it came up! I have been fighting this for 4 weeks. I really hate networking sometimes.
@@stevefxp Yeah it can get frustrating. I prefer to use the untagged LAN network for management purposes instead of a dedicated VLAN because everything defaults to it and I can just plug in to get set up. Then I push all other devices to VLANs and restrict access via firewall rules. You of course have to be diligent to ensure everything is on the proper networks (and I like to default unused wall jacks to the GUEST VLAN in case someone plugs in so they’re not on my management network).
When I get to the part where you're in the Web GUI, clicking on damn near anything causes a popup that says DANGER- unexpected error, check log for details. Only, there's no log. And the Dashboard screen is completely empty. If I close out the popups (plural), clicking on just about anything them to start again. Any suggestions?
The web UI of Proxmox or OPNsense in the VM? That is weird to see such an error because I've never seen a "Danger: unexpected error" message before! Almost makes me wonder if there is some file system or RAM corruption-- some sort of hardware related issues rather than a misconfiguration, but if it works for other software, then it seems like it would be a miconfiguration somewhere.
@@homenetworkguy Thanks for responding. After some more testing, it turned out to be Ublock Origin causing the issue... but ONLY on the Brave browser. That specific combination of Browser->Extension was the headache.
Ahh ok. I was wondering in the back of my mind if it was browser specific. That makes sense. I've had uBlock Origin mess up certain things because it's blocking things too aggressively or by keyword.
I notice it uses about 20-22W but I had a couple network interfaces plugged in and I have a second disk (SSD) which would add to the base wattage. However I think that’s a good basic use case for real world wattage. It has faster single threaded performance than my Ryzen 7 1700 Proxmox server but at 1/4th the idle power consumption. It uses about twice as much power as their 4 port models but it’s also much more powerful too. I have the VP2410 and VP2420 and the two systems combined uses nearly the same power at idle as the VP6650.
You could do everything with a single interface if you like! I like using more of the interfaces available because it means less bandwidth is shared. Also for certain things such as clustering Proxmox, it’s recommended to dedicate an interface just for the cluster traffic (on its own network). If you have fewer ports and you have multiple networks/VLANs set up, you can just assign the CTs or VMs to whatever network you like. You have to make sure the network switch is set up for VLANs. The key thing to consider is that you need to use bridges instead of using pass through so multiple CTs and VMs can share the same physical network interface.
Probably not intentional but I don’t use the Proxmox firewall since I use OPNsense to firewall the network (and I use local ufw firewalls on all my Linux containers/VMs). If you have box checked to enable the firewall, there are no rules defined by default so it doesn’t offer any protection. You have to add rules in Proxmox if you want to further restrict access. I suppose it is yet another layer of protection if you want to use it in addition to everything else.
You would have to use a bridge for the LAN interface similar to how I demonstrated in the video. It would be the same interface you use to manage your Proxmox server. You can’t use PCI passthrough on that LAN interface and also use it as the management interface for Proxmox because that interface will be dedicated to the OPNsense VM if using passthrough.
Thanks! I've had a few requests for Kea DHCP. I'll get around to it eventually since it will be the new way forward but currently I do not believe it is considered feature complete so I do not see myself personally migrating any time soon (but I will likely do a video on it before I make the transition on my own home network).
@@homenetworkguyI switched right after update with Kea support and it's not that hard to set up. And it's working without problem. Only con I see is no hostnames, only theirs IP adresses (in DNS server, monitored communications etc.).
hey at first, thx for yout tutorial. i got one question. at the network config you give 4 to queues. why? can you explain int to me pls. i m new in the game and dont find a easy answer in the inet. thx
It allows the guest virtual machine to have virtual CPUs process the network traffic which can help improve throughput. According to the following link, it is recommended to set the multiqueue value only when anticipating a lot of network traffic since it increases the CPU load of the host/guest as network traffic increases: forum.proxmox.com/threads/multiqueue-inside-of-vm.66321/
I have been able to follow the instruction, install Proxmox / Opnsense and everything is working fine. I however am unable to update proxmox, and keep getting a message “download failed unable to resolve host” when trying to download Lxc for pi-hole. I feel proxmox isn’t able to access to web. What can I do to solve this. Please help.
You’re using the same bridge in Proxmox as the LAN interface in OPNsense? As long as you don’t pass through that interface which is used for Proxmox management and you have the LAN configured properly in the OPNsense VM, it should have access to the Internet just like any other device on the LAN network.
Is it possible to do this with only two physical eth ports on my proxmox box? I tried following along best I could with this. My opnsense is running good and I'm getting internet through it, but I cannot reach my proxmox gui. I can ping the box, but can't ssh into it. Both operations time out. From the proxmox box, though, I can ssh to my machine just fine.
You should be able to dedicate one interface to the WAN and the second to the management interface of Proxmox and the LAN network of OPNsense. You can even add VLANs on the second interface as well (but you’d need to add the VLANs to the network switch as well). Technically you could it all from a single interface using VLANs but the config is a little bit more involved. It’s easier to configure separate interfaces and also reduces the potential for bottlenecks in throughput. You need to make sure that Proxmox has an IP address in the same network as the LAN on OPNsense (which defaults to 192.168.1.0/24).
@@homenetworkguy Thanks so much for the reply. I ended up figuring out my issue... my proxmox box needed to have its gateway set to the router. I had set it to opnsense, and from what I've learned that was causing asymmetric routing.
Nice! The gateway is the interface IP of each network- IP address which is used to route the data to other networks essentially so for the default LAN that is 192.168.1.1, as you likely are now aware. Glad you got it working!
I would have to look into this more. One thing that I find annoying to deal with in Docker is its networking. Deployments are easy but then you have to mess with the networking aspect. Simple things aren’t too bad but what what if you want containers to be on different VLANs as you mentioned? I’ve always just put them on the same network in the past but that was before I started segmenting my network. I avoid Docker since I use LXCs (don’t need an extra container layer) so I haven’t tried setting up apps on different VLANs. Also a VM might be more desirable than an LXC for Docker (at least when I tried a while ago, restoring backups of LXCs which use Docker was problematic for me).
@@homenetworkguy I find setting up a Docker container much easier than using LXC, but maybe that's because I know more about it. I think having one LXC with Docker and multiple Docker containers is less overhead than having multiple LXCs. I'd love to read about your findings!
Setting up containers on Docker is easy but the networking aspect is something you have to work through. I’m not sure without researching it further on how to run containers of different VLANs (I don’t know if MACVlan or IPVlan modes are what they sound like they should be used for.. almost seems like it’s for containers to internally communicate on different virtual networks). The nice thing about LXCs (without Docker) is that I can allocate exactly the right amount of host resources that I want as well as install whatever I want inside the LXC (without needing to create custom Docker images for example). It’s very simple to put LXCs on different networks. I also like to utilize the ufw firewall in the LXCs so I can easily block all unused ports on each LXC (I know Docker only exposes certain ports for containers but it also interferes with the firewall on the host machine so you have to do workarounds to be able to use ufw firewall or iptables without Docker interference). I typically setup SSH access for all of the LXCs so I can get into them if I need to do anything. I think you can do that with Docker as well but not sure if it’s as straightforward depending on the networking mode used (I believe I recall logging in more easily to a terminal window using Portainer long ago when I was using it). I also like being able to back up individual services that are in LXCs rather than the whole Docker instance because I only have a few critical LXCs that I backup offsite. With individual LXCs, I can move them around to different Proxmox nodes easily. Ultimately, it’s a matter of preference. There are pros and cons to either approach but I’ve come to like using LXCs better. A lot of people like Docker and I understand its appeal especially in deploying web apps that have a lot of dependencies including setting up databases, web servers, etc.
I will try to follow this at some point later, but I have already done this and I have one issue I do not understand. I have the router software installed and some VM's. I start the router, start the VM's, and the VM's have no connection. I reset all services on OPNsense, and magically I have connection. I do not want to just have to reset the router every time - what can I do to fix this?
I'm not sure I've seen that happen but you can set up Proxmox to start OPNsense first and then make all of your other VMs start only after the OPNsense VM has started. You can even add a short delay to ensure OPNsense is up and running before anything else on Proxmox starts. This could potentially help with your problem, but I'm not sure why it doesn't detect the network is up and running.
@@homenetworkguy unfortunately I have tried this and it does not make a difference which one starts first. No matter whether OPNsense starts first or the VMs, services have to be reloaded. I'm going to have to watch your video and make sure I follow it step by step because if you aren't familiar with this, there is something I have to be doing wrong
Hi, is it possible to install+test this out without having a NIC available for WAN during install process, and add it later (for ex. PCIe addon card)? and is it possible to change NICs later (remap physical interface or vmbr) to "upgrade" from 1Gbe to faster NIC, without reinstalling opnsense?
Sure, I think that would be possible! What you can do is create a bridge in Proxmox that is not assigned to any physical interface. Make sure you select that interface as the WAN when you install OPNsense (hint: if you make it the first network interface for the VM, it will be called vtnet0 inside the VM if you’re not doing PCIe passthrough). Then later you can update that bridge in Proxmox to use a physical interface. You can easily remap network interfaces later. That’s the beauty of virtualization. You can even do this while the VM is running (but caution is advised)!
You can also assign VMs/CTs to that same bridge and everything would be on that same virtual network. This is pretty neat if you want a fully virtualized lab network within Proxmox.
@@homenetworkguy yup I can confirm works exactly as expected, for fully virtualized opnsense install: fake WAN in prox, fake LAN in prox, then make a random VM with Windows or whatever, assign real LAN, fake LAN (static IP in Windows at opnsense LAN range), that way it's easily possible to RDP into this machine and from its web interface tinker with opensense web UI, fun stuff :D
It depends on what you are trying to accomplish. Performance is best with passthrough but you can’t use the interfaces for anything else. With bridges you can have other VMs and CTs be on the same network by sharing the same bridged interface. It’s very flexible but there is a performance penalty. Since I’m planning to cluster it makes it easier to migrate VMs between nodes.
No. I can't recall if I mentioned it in the video but once you have OPNsense set up and you're using the same bridge for the LAN in OPNsense as the bridge used for the management interface of Proxmox, you can simply connect the interface you are using to manage Proxmox into your network switch. As long as you have that port on the switch left at the default of VLAN 1 (untagged), the Proxmox management interface will reside on the LAN network (which I use as the management network in this example and on my home network as well-- I just try to be careful to keep everything else assigned to the proper VLANs so I can keep the LAN network protected and isolated from the rest of my network).
@ thanks! Suppose for example, my original network is on 192.168.0.0/24 then the opnsense vm on proxmox i set up to be 192.168.1.0/24 do i have to change the ip of proxmox to be on 192.168.1.0/24? I hope im making sense. Thank you again for this video!
Nice! Glad you got it figured out. The management IP for Proxmox needs to be in the same range as the OPNsense LAN-- if you leave everything at the default, it should already be set that way, I believe.
@@homenetworkguy i finally have a functioning network. The isp router is on bridge, minipc with proxmox and opnsense on it. I do not have a managed switch so no VLAN. But still its crazy to me. Thank you man!!!!
Ohh yeah. You can use any PC you want to administrate your devices. I was just using a ZimaBoard because it's much smaller than setting a full tower PC on top of my desk to show all of the connections, haha.
@@homenetworkguy Ah, thanks - I was quite confused and honestly didn't know if this board also did something else required. But yes, I tried it with my PC and it seems to be working (can't tell until I did a proper configuration).
Haha yeah I just wanted a small PC to use to set everything up and to show where a PC could be connected on the network. Glad you got your network set up!
I saw you selected ZFS, is it still better than EX4 even if you only use RAID0? Is there any advantage for proxmos such as compression and deduplication?
It supports snapshots so you don’t have to pause or restart your CTs/VMs when you do backups (you can do snapshots with ext4 but you have to select LVM thin and not LVM for this to work). You can also take advantage of built in LZ4 compression which could not only save space but speed up read operations (I believe). Caching certain operations in RAM may help improve performance but I haven’t compared that directly. ZFS can still detect bitrot with a single drive because of the checksums but it wouldn’t be able to correct it without having redundancy. I don’t use deduplication with ZFS even on my TrueNAS system since it requires too much system resources.
@homenetworkguy thank you for answering. One more question if you don't mind. I'm planning to get VP6650 I was just thinking, how can I design the storage wisely give that nvme is for VMs/CTs 2.5 SSD1 - Host 2.5 SSD2 - RAID1? What if I upgrade in the future and it fails, the upgrade will also replicated to it so it will not work? Please advise what is the best storage design for it. Thank you in advance!
Yeah you could mirror the SATA drives (RAID1) for the host OS and use the NVMe for CTs/VMs. That's a good way to set it up and it's similar to how I used to have my 4U rackmount Proxmox server before I migrated it to the VP6650. There's not really a good way to recover from a failed Proxmox update but those sorts of failures are pretty rare. I had an issue long ago but it was when migrating from v6 to v7. Not sure if it was self-inflicted because I was new to using Proxmox back then. If you keep some of your configuration files under the /etc/pve folder, it will help you with a new installation because you can recreate your configuration more quickly. In theory the Proxmox host is supposed to be minimally modified so that it's easy to reinstall or move to a new system (the idea being that your CTs/VMs contain most of the configured apps/services). However, in practice, you still need to backup some of your config files to save time if something fails. I recently set up a Proxmox cluster so if I have a hardware failure, I can remove that node, and add a new one more easily because much of the configuration is at the cluster level (still a good idea to backup the network config because each node needs to have the interfaces configured appropriately).
Hey I'm new to networking and I just build my first home server. However after setting Proxmox up. I can't seem to access the webGUI using the PC to configure the creation of the OPNsense VM. I have assigned a static ip to my laptop. Any idea of what I'm missing? Thank you!
Are you plugged directly into the Proxmox management network interface? Or connected to a network switch? You will need a static IP on your laptop only if you’re plugged directly into the Proxmox management interface. Otherwise you can use DHCP if you’re on the same network as the Proxmox management interface.
Did you configure the subnet of the static IP to be 255.255.255.0? Also make sure it’s not accidentally the same as the Proxmox IP address as well. You could try different interfaces on your Proxmox box in case you have a different one configured than the one you’re plugged into.
@homenetworkguy could I contact you on a discord or something alike to grt a bit more help. I'm really stuck and can't seem to figure out what is going wrong
I do have a Discord account. I don’t always hop on it but you could use that. Keep in mind that it’s becoming a bit more difficult to keep up with everyone’s messages. I still have a couple week backlog left in my email (I caught up on a couple weeks worth of email last night).
Any idea how to show the connected devices on my network? I just switched from a off the shelf router to OPNSense, but I can't seem to figure out how to see all my devices and their IP addresses.
Under the Services > ISC DHCPv4 > Leases page, you will see a list of all devices and IP addresses of the clients using DHCP. You won’t be able to see any devices that are using static IP addresses but you should be able to see everything else.
My goal is to replace my bare metal OPNsense firewall with two virtualized OPNsense firewalls that will be clustered. This means Proxmox clustering and then OPNsense clustering. In my trials I cannot seem to turn off the bare metal firewall and the virtualized firewall takes up the call. Am I missing something? I am watching your tutorial carefully but I am not seeing anything that you did, that I did not do.
This video doesn't demonstrate a Proxmox cluster nor using OPNsense in high availability mode. If you're going to use a Proxmox node with OPNsense in a VM, it might be best to enable the Proxmox high availability features and simply have the VM failover to another Proxmox node. I think it may potentially be easier than setting up CARP on OPNsense (depending on your familiarity with Proxmox) since you only need to maintain 1 OPNsense VM instead of 2 separate OPNsense VMs on 2 different Proxmox nodes (also you can have other VMs failover to other Proxmox nodes and not just OPNsense). Plus I think failover with CARP might introduce more of a network disconnect than with using Proxmox until it detects the node went down (based on what I've seen others say). I simulated Proxmox HA by doing a manual live migration of an OPNsense VM between 2 Proxmox nodes in another video. I don't have full HA setup on my Proxmox cluster (there are tradeoffs with having the VM on shared storage especially if it's NFS storage instead of Ceph, and it also increases the complexity of the Proxmox configuration a bit). I try to manage the complexity the best that I can on my home network so I don't have to spend a lot of time fixing things that break. For the most part, things run pretty solid and only tend to break when I'm tweaking something (if it's not broke, don't fix it-- but you can't really learn as much unless you tinker and potentially break things). haha.
I have 4 Protectli boxes (since I have some sponsored hardware) and nothing has died yet with 24/7 operation. The oldest Protectli is about 3 years old. I run the hardware in my server closet which runs a few degrees hotter than room temperature so the operating environment isn’t very hot. Something to consider when running fanless mini-PCs because the hardware might not last as long if it’s in a hotter room (do not run it in an attic for example in a hot summer, for example). I will say that I always have my systems connected to a UPS and rarely have any hardware die unless is getting very old (which is to be expected). Most of the time my hardware in general becomes essentially ‘obsolete’ before I replace it.
First, thanks for the video. I think this has me most of the way there but am unsure on something. In my case, I will have Proxmox on a server colocated in a datacenter. I passed thru my NIC to OpnSense and it's booting and working. However, how do I allow Proxmox and other VMs to use OpnSense? I'll have a VPN running so I can remote in, and hopefully use the LAN IP address to still access Proxmox from afar. - thanks
You’re welcome! As for your question, I’m not sure of how many interfaces you have in the colocated server. Of it’s only one, then you cannot use passthrough because that means only the OPNsense VM can use that interface. You will have to use the default bridge interface in Proxmox. With only one interface it’s going to be tricker to set up a WAN/LAN interface but it’s possible using VLANs. If you follow the basic principles in the video, you will be able to use the default bridge for both the Proxmox management and the OPNsense LAN interface. You simply just assign the same bridge to other VMs so they can be on the same network.
Could you make deep dive OPNsense firewall video next I'm having trouble understanding the firewall. I have OPNsense running on top of Proxmox with two NICs passed trough (WAN/LAN) and VLAN interfaces (10,20,30,40,50). I'm trying to allow Proxmox hosts in ManagementVLAN10 (10.10.10.0/24) to temporarily (or permanently) access my Unraid NAS VM web GUI in ServerVLAN30 (10.10.30.0/24) but I'm having no luck with it. In the future I also need to allow Proxmox hosts in VLAN10 network to reach Unraid (in VLAN30) for NFS purposes. I'm using Mikrotik SWos switch. The firewall just doesn't click with me. I've watched some of your OPNsense and firewall videos but I'm still struggling. It's feels like OPNsense doesn't know the routes between VLANs since the firewall rules I create seem to do nothing.
It's hard to say where the config is going wrong without seeing any of it. Perhaps you could take a look at my website which the videos are based off of for more details since there may more explanations that will help you understand it better. It does take some time to wrap your mind around firewall rules when you are new to them (at least it did for me): homenetworkguy.com/how-to/set-up-a-fully-functioning-home-network-using-opnsense/
Forgive my elementary level question, but how (and on which device) do you manually generate an ip address to disconnect from your LAN and continue management? Thanks!
I just used another PC from the one I did the recording (I have a couple of mini-PCs I use for demo purposes). If you only have 1 PC/laptop and 1 Proxmox server, you’ll have to temporarily connect your PC/laptop to the Proxmox server to configure it. Once you’re done and have OPNsense installed, you can connect it back to your network. I’m assuming in the video you’re using the default LAN network for both the Proxmox management and the OPNsense LAN interface. That network interface is 192.168.1.1/24 (which means usable IP addresses between 192.168.1.2-192.168.254).
@homenetworkguy: Are you giving "pve-test" the new address of 192.168.1.50? My current network uses a 10.27.27.x scheme, but I don't know how to locate my new proxmox node when I remove it from the network. 😕
You can assign the Proxmox static IP to be whatever you want so you can make it 10.27.27.100/24 if you like (make sure it’s outside your DHCP range to avoid potential IP address conflicts). If you plan to put OPNsense and make it your primary virtualized router and you still want to use that network address, you’ll have to change the default LAN IP addresses or create another interface with the appropriate IP address ranges. I tend to keep the default LAN network of 192.168.1.1/24 since it’s keeps things simple (but I add other VLANs too, of course, with different IP ranges).
Just figured out that it's the ip address for the "stand alone" pc used to configure the proxmox device that needs the new static ip in the proxmox device's network scheme. 🤯 Thanks for all the info (and patience)!
the first ip you will get on the first setpup of proxmox and opnsense need to be change to fit network wish he had shown the part more clearly changing the IP of proxmox and opnsense to fit the network IP address in his network feels that the hardest or complicated stuff
Yeah since I’m using the default 192.168.1.1/24 for both the Proxmox interface and the LAN network of OPNsense, I didn’t have to make any adjustments later to make them be in the same network.
I currently have a topton N5105 with 4 ports 2.5gb i226v. Would I be able to do this? Been reading around reddit that people were having random crashes? Is this still the case? I currently run opnsense as bare metal. But want have snapshots/ backups for quick restore
I believe this was addressed in newer versions of Proxmox. I know many had issues with the N5105 and the N6005 but I’ve used Proxmox with the N6005 without issue several months ago.
@@homenetworkguy that’s great thank you for replying. I plan to change over to proxmox. Can I use a Ethernet adapter (2.5gb) to use for proxmox/setup and setup opnsense. So I can set up my 4 built in ports as follows: WAN, LAN 1 , LAN 2 and LAN 3
You could I suppose but keep in mind if you use bridges, you can share the same port with your Proxmox host/VMs/CTs as demonstrated in the video. You don’t necessarily have to dedicate all the ports to OPNsense (you may need to use passthrough on the N5100 to achieve 2.5Gbps but faster hardware can handle 2.5Gbps even with bridges just fine).
@@homenetworkguy I think I understand it now. I have 3 Ethernet cables from my opnsense, LAN 1= server (unraid) LAN 2= WiFi access point upstairs and LAN 3 for lounge. And last port is my WAN. So if I understand correctly I can e.g use LAN 1 to install/setup proxmox and opnsense and then have my ports work in the same way?
Yes if you use the default vmbr0 bridge that Proxmox sets up during the installation. That’s the great thing about bridges but there is a performance impact depending on your CPU and the speed of the network interface. I’ve discovered that bridging performance in Proxmox is greatly impacted by single threaded performance of the CPU.
You have an amazing voice and know how to make it simple. I have a question. I followed all the steps .. however, i am stuck when accessing the opnsense via web gui.. Initially the proxmox got ip4 address from a different router with 192.168.1.1 then how can i access opnsense with same address. please help i am stuck.noob here
Thanks! Are you trying to set up OPNsense to try it out (or use it as a secondary network) or to eventually use it as your primary router? If using it as a secondary network, you could put the OPNsense LAN on a different physical interface and connect a network switch to that interface so you can have a secondary network as a lab/playground. If you want to make it your primary network router/firewall, if you followed the instructions in the video, you would swap out your existing router. However you could temporarily assign the OPNsense LAN interface to a different physical interface where you could plug a laptop/PC into. Then you would be able to access the LAN of OPNsense even though it has the same 192.168.1.1 address (since you’re not plugging that interface into your existing network which could cause problems with IP address conflicts, multiple DHCP servers, etc). If you take a look at my new video of the Intel C3000 series video, I show setting up OPNsense in a VM on an existing network and plugging devices directly into the mini-PC to test out OPNsense. I don’t recommend setting it up like that in the long run but it shows how you can test it.
@@homenetworkguy i never expected a reply so fast. Highly appreciate your detailed response and time. I was able to connect to OPNsense. Appreciate it.
It’s either that or choose “other”. I think it affects the options that are available for the VM configuration since some options aren’t available for certain OS’s. Not sure if it makes a difference for FreeBSD based VMs or not.
That’s interesting. I haven’t seen that before. What disk type are you using for the VM? Not sure if those settings would cause a problem with the reporting aspect in a virtualized OPNsense.
@@homenetworkguy Hi and thanks so much for responding, I recreated the VM(from scratch), and found the disk space reporting okay - no clue what I did to cause that, but it looks like its no longer an issue - strange issue nonetheless - hope it's not a sign of something more serious(nvme issue etc).
@@homenetworkguy Hey I just wanted to update this, and say that the issue in question was due to 'not installing Opnsense' - hahah In other words, and as we can basically run Opnsense without installing it, web UI and configuration alike, this makes it possible to actually forget the installer portion at the beginning of the process - which is all quite embarrassing to say the least BTW, I discovered this, after installing(not really), Opnsense on bare metal, and finding the exact same phenomenon :/
Ohhh, that would be why the disk is at 100% because it's just running in live mode. Many non-Windows Operating Systems can boot in live mode so you can try it out without installing anything on your system which is pretty cool. With OPNsense, it all depends on how you sign in when you boot the installer. That is an easy detail to overlook if you are new to installing OPNsense! Glad you got it figured out! I wish I would have thought about the 100% disk usage as a clue that it was booting in live mode.
Just watch the video, but you did not showed how to configure firewall rules as by default opnsense block all the traffic i am also having the same setup as you showed in the video but i cant access my internet on lan network Can you please give me some inputs here
ZFS has great features even with a single drive. For the primary OS disk you can take advantage of boot environments and roll back the filesystem to a known good state if an upgrade fails or some other issue. It’s just a nice robust filesystem. It takes advantage of extra RAM for caching, etc.
@@homenetworkguy it appears I have been using the incorrect ISO. I would download the image from Open sense straight into my Proxmox and just realized that this was some kind of a zip file. I unzipped and now uploading manually but this method might take an entire day to complete. Stay Tuned!
@@homenetworkguy Thank you again, Second Question if I set this up as virtualized just for learning can I keep it strictly isolated to my proxmox vms and not have it manage my main network/wifi. And what would be the best setup for that scenario?
Yeah for sure. I have a couple of OPNsense VMs I use for demos/testing, etc. The main thing you need to be careful of is not putting the WAN interface of the OPNsense VM on one of your primary networks while also having LAN interfaces on the OPNsense VM using the same IP addresses because the LAN interfaces will take priority over the WAN interface. It’s hard to explain but I’ll give an example. If you put the WAN interface of your OPNsense VM on the 182.168.1.1 network, the WAN of the OPNsense VM will be assigned something like 192.168.1.100. But if you also have a LAN interface in the OPNsense VM with 192.168.1.1/24, your WAN interface in the OPNsense VM will not be able to communicate with your primary network because the gateway address of the WAN interface will be 192.168.1.1 which happens to be the LAN interface IP address. One other gotcha is you will likely want to enable query forwarding under Unbound DNS if you are running into DNS issues. I’ve found that running a recursive DNS resolver behind my primary OPNsense box doesn’t work (probably since I am using DNS over TLS on my primary OPNsense so it can’t recursively resolve to the root DNS servers).
Firewall on a VM is not a good idea. The "bad packets" must be forwarded through the physical server to the VM. This means that the physical server for the VM is always unprotected. (As an example a bad IP packet triggers a buffer overflow on the kernel) Greetings Marco
I usually run bare metal but I know a lot of people like to virtualize for various reasons. Do you have any documented examples of what you are referring about compromising the hypervisor on a virtualized firewall? I’d be interested in reading up on it.
@@homenetworkguy The IP packet arrives at an interface on the server and is analysed by the server (OSI Layer 2 & 3 analysis) and forwarded to the VM. These steps take place on the server before the packet arrives at the VM. Only the IP tables of the server forward the packet to the VM. This means that the IPTables including the kernel are before the firewall. Draw the path for each OSI layer once on a piece of paper and write who is responsible at each point.
I understand what you are saying. I am just curious how many documented cases of compromise due to virtualizing the firewall. So many people do it that I’m surprised more people say “don’t do it!”
@@homenetworkguy Security is not a question of the frequency of events! The host server is not protected and is therefore directly connected to the "bad" Internet. Why use a firewall then?
@@marcodoehler4089 Because the OPNsense VM uses interfaces that are connected to bridges on the physical Proxmox VE interfaces, Proxmox VE doesn't analyze anything. It will only receive Ethernet frames (layer 2 only), the bridge will look up the destination MAC address (of the OPNsense virtual interface) and simply forward it. Iptables (or soon nftables) on Proxmox VE will not be used for this at all, unless you want to block traffic to and from the OPNsense VM from the host. If you do not set an IP address on any of the bridge interfaces to which the OPNsense VM virtual interfaces are attached, there is no way to communicate with the host.
Yoooo let me just swoop one of those $1300 mini computers 😂 May as well go buy an sonicwall TZ570w with a year of professional support for the same price.
Can you install a hypervisor on the Sonicwall? New prices seem like $3500? I’m assuming you’re referring to used hardware prices. You could also do this guide with a $200-300 mini PC which has 4 network interfaces. It depends on what you need. The VP6650 is faster (single threaded performance) than my old Ryzen 7 1700 Proxmox server at 1/4th the power consumption. I could easily replace my huge 4U server with the Protectli if I wanted but I’ll probably just cluster a few of my systems at some point.
Thanks! It seems like the video is being well received by those interested in the topic. Also, I was genuinely curious in my previous comment if you can run a hypervisor like Proxmox on it and get the device plus a year support for $1300? I wasn’t implying the Protectli box is superior to the Sonicwall but rather it’s an apples to oranges comparison (one is a general purpose computer while the other is a firewall appliance). For a home network, having a general purpose low power mini PC is great for virtualization servers, etc.
I fell off when you started messing with the hardrive. I dont understand, i didnt have to do that when i followed a guide to install homeassistant. I only have 1 drive, a 2tb nvme.
I’m demonstrating how to set up OPNsense to replace an existing router on the network so it’s a more complicated setup than just setting up Home Assistant. There are other ways to set up OPNsense that are a little less complex if you’re using a different network interface configuration.
Yeah, I tried following your guide but I guess its over my head. I have a 4 port N100 mini pc I want to use for proxmox, OPNsense and HA. But I hit roadblocks left and right so I can't get it to work. If only my isp modem wasn't a router also, then maybe. When I put it in bridge modus OPNsense lost internet connection. And with the router active HA and wifi doesn't work together. And I can't change anything on the router, it sucks. I'll go find some other guide or something.
Yes! No, just kidding. I think it was distracting because sometimes with my edits I have to jump around out of chronological order. Also it sometimes takes a few days for me to get time to get all the recordings complete. But I realize that blurring it out makes it too distracting for some users.
Yeah I wasn’t impressed with the UI when I first started using it but over time it has grown on me. I enjoy using the product. I’ve only had a few mishaps over the years and most of it was probably user error on my part. Haha
I really tried to follow your vague and convoluted presentation. You spend too much time hedging vs definitive information, like 7:1. Hypothetical cases ought to be footnotes not part of the main presentation.
the ugly logo of opnsense alone shouts this is for highschool lab testing. I mean they cant pay a logo designer to create a nice logo??? branding is a great part of everything
They have hardware you can purchase and they offer business license support as well if you need professional support. It’s goes far beyond high school lab testing. I’d much rather have a solid product with a simple logo than a terrible product with the most beautiful branding. 😄
@@homenetworkguy what product does not offer professional support??? and good luck to them with that ugly logo, hope they get enough to pay for it. NO company with a brain will use this in DEV environment talk less production. Sure it can work but logos cost like $50 and up
![Stromae, Pomme - “Ma Meilleure Ennemie” (from Arcane Season 2) [Official Visualizer]](http://i.ytimg.com/vi/1F3OGIFnW1k/mqdefault.jpg)
Finally, a video that shows what the REAL WORLD looks like, and takes it from step ZERO. Well done
Thanks! I have more real world examples coming up soon! In fact, most of my guides are based on real world examples (I like to base them on real examples that I have done for my own home network either currently or in the past and sometimes I create examples in a lab environment to try new things and to verify the process works properly).
My thoughts exactly 👍
This is great, easy to follow. I'm a complete noob and got the parameters for opnsense set up on my proxmox. Step by step i go slow, but things are looking good! 🎉🎉
Thanks!
Thank you for showing the physical real world implementation. As a beginner, I've always struggled with this and this is the only video that shows from "zero to one to 100"
I’m glad that was helpful to show all those steps! I’m going to doing the same when showing how to set up a basic 3 node Proxmox cluster soon.
Using OPNsense for years, I never knew you could delete the interface which holds the vlans. Nice video. 👍🏼
Yeah you can if you don’t plan to use the untagged parent interface. Since I use a different interface for the LAN for untagged traffic, I don’t need a second untagged interface and just only need to use VLANs on that second interface for just tagged traffic.
I will note one potential gotcha that I encountered when testing out some things. If you want the VLAN interfaces to use a MTU that is higher than the default 1500 used by all interfaces (to enable jumbo frames with MTU of 9000, for instance), you will need to have the parent interface assigned and enabled so that you can set the MTU value on the parent interface. This is likely a rare scenario since typically jumbo frames are used on isolated networks with higher speed interfaces (10Gbps+) rather that for routing traffic across 2 networks with larger frame/packet sizes.
Great detailed overview for anyone starting out.
Just some comments on the options when setting up VMs in Proxmox:
If you're on using thin provisioned storage, you always want "Discard" checked. It's what makes the guest OS emit TRIM commands necessary to actually free space on the host that was free'd in the VM. SSD emulation just tells the guest that it's flash storage, it doesn't enable TRIM. I have no idea why this isn't a default setting.
For anyone setting up a single host, so not a Proxmox cluster: Just always use "host" as the CPU type for a measurable performance gain. All features and abilities are passed through accordingly, and there is no need to enable or disable instruction sets like AES, and there won't happen any translation either. This setting does imply that a VM can't be live-migrated using HA on a cluster, which means the VM is switched from one Proxmox host to another WHILE RUNNING. This is an incredibly rare requirement for a home lab. Even when "host" is selected, it's perfectly fine to shut down a VM, transfer it to another host, then start it again: the "host" CPU will just change meaning during the transfer to represent the other hosts CPU, no problem.
Finally, when virtualizing a firewall, it is highly recommended to pass through the actual PCIe hardware directly so it has direct access to the hardware. Yes, this does make a difference. No, it's probably not critical in a home lab, but if you're using 10g you probably want to use that. If it matters, or how much, when using 1g networking depends on the hardware (both system/platform and network). The only exception for me would be when using RealTek network cards. Anything based on BSD (pfSense, OPNsense) has bad enough compatibility that having Proxmox (Linux) handle it and use a bridge like you have shown.
Thanks for the info!
Someone already mentioned several of those points and I also made an addendum follow up video pointing out a few settings you could do differently.
I’m now running a Proxmox cluster and I like using the live migration feature for my OPNsense VM so I can reboot the Proxmox system for updates without taking the network down (I demonstrated this in my cluster video). It works amazingly well. I was surprised not doesn’t even drop the existing connections. It only adds a slight delay when pinging when the cutover occurs. I don’t think live migrations has to be a rare use case for home labs, haha.
Because I have multi-homed my NAS and use dedicated 10G backend networks, I don’t have much of a need to transfer large amounts of data across VLANs so leaving everything as bridges is fine for my use case even though there is a performance hit. It’s not bottlenecking my network (most of my devices are 1G so even bridges can handle that no problem). My internet is about 1.2Gbps down and about 25Mbps up and I can still use Zenarmor with a bridge with no hit to throughput. I’ve measured I can get up to 2.3Gbps on a bridge with my hardware with Zenarmor so I’d only have issues above 2.5G interfaces (which isn’t a problem as I’ve mentioned since my high throughput devices are on the same networks or connected to a dedicated, isolated storage network).
With that said it’s good for others to be aware of all of those things you mentioned! It’s good to know the caveats and/or optimal settings depending on the use cases.
How? How do i do pci passthrough???
This is the video that gave me the reassurance to switch my own home network over from firewalla to a virtualized OPNsense instance this past weekend. It genuinely surprised me that it was a clean cutover with all of my vlans/APs, Thank you!
OPNsense has 4 Performance cores of a 14700T, 32GBs of ram and a bridged Intel X550 T2 dedicated.
Great to hear! Glad it gave you reassurance! Make sure you have a good backup plan if you only have one Proxmox server (but even if you have a bare metal installation, it’s good to have a backup plan). If all is configured properly the virtualized instance should function essentially the same as bare metal as you have discovered!
Fantastic video. I learned a ton watching and following along. Thank you so much. I appreciated you walking through each option and briefly discussing why or why not you had chosen said option. Cheers!
Thanks! Glad you liked it! I think it’s helpful to explain the options instead of just picking them. I tend to go in more detail in written guides on my website. I have to be a little more terse in videos to try to stay on topic and keep the length shorter.
Excited to watch in full, now, for learning and entertainment.... Already saved to watch again as a guide
Thanks! I hope I covered enough to help people along. It’s a lot of info to cover (and there could be even more but I tried to keep the length somewhat reasonable). Takes a lot of time to produce content in general, let alone during your limited free time. Haha.
I encountered few errors which were to do with my system, but my god this video and your guide webpage were so helpful. Thank you!
Awesome! Glad you found it helpful!
Perfect timing on this. This is exactly how I plan to setup the mini PC that is out for delivery right now. :)
Sweet! I love it when it's perfect timing for my subscribers (and others). Someone else said it was also perfect timing earlier today.
Fantastic that you release this video literally the day i get everything together to do exactly this myself, you also helped me with the PCI pass through that nobody else talks about. Thankyou!
That's great! I'm glad the timing worked out. Sometimes I'm just in time for some users and too late for others. haha. I thought I would mention PCI passthrough in the video even though I didn't do it in the video to keep things a bit simple but I also tried to ensure that the instructions should still work if you plan to use a Proxmox cluster. Things get more complicated when doing PCI passthrough with a cluster. I have yet to try all that out as well. Bridges are safer and you will only notice performance issues with 10G interfaces or faster. You can still get 5-6Gbps with the VP6650 I used in the video so it's still faster than the 2.5G interfaces (and really you should try to not route 10G NAS and other traffic when possible to reduce the load on the firewall by having a separate 10G network).
Maybe use the managed switch and create a WAN subnet using a VLAN 🤔 connect the WAN cable to the switch and then any Proxmox node can access the Internet VLAN for a virtual bridge ?!? Just a thought. Might be more complex using the newest SDN feature on Proxmox . . . Guess it's time to experiment around a bit . . . Great work 👍🏻👍🏻
Just when I needed the video, no excellent info available on RUclips IMHO, this _is_ _great_
Thanks! I hope it has enough info to get started because there is a lot of information to cover. I tried to keep it focused on the topic at hand.
Solid video. Followed steps but had to use a Mac to do the OPNsense config. Worked like a charm.
Thanks! Glad it worked well for you!
Brilliant work. I'm building my own home network and your guides are excellent.
Thanks! I hope they help you along the way! I have been evolving my network for many years (more so in the last 6 years).
@@homenetworkguy , do you have any thoughts on IPFire ? For example, can I use it to achieve something similar to your "opnsense for beginner" video/post ?
I have thought about learning more about other firewalls (OpenWRT, IPFire, etc) once I have exhausted the main topics I want to cover in OPNsense but after writing on my website for nearly 6 years (and more recently, RUclips videos), I still haven't exhausted everything I'd like to learn about. haha.
I think IPFire could be a good Linux based alternative. There are a lot of similar features but also some things it doesn't offer via plugins. I would like to test out the performance of it because it's possible Linux could perform better than FreeBSD depending on driver support, etc.
thanks for all the opnsense and proxmox content. As a opnsense / Truenas scale home user and a vmware enterprise user @ work i enjoy all this content. Proxmox and ncp-ng are in our work test labs for possible move to from vmware. Thank you again!
Thanks! I'm glad you appreciate it! I hope to dig more into Proxmox clustering with OPNsense and how I think I'm going to go about it on my home network so that I can do live migrations (it will be very awesome to have the ability to move my main router/firewall over to a different physical machine with only a split second blip in downtime for my network!). I don't care about high availability/failover as much as being able to live migrate the VMs (because with VMs it's easy to restore from a backup from my PBS system, which is another nice piece of software). The configuration and requirements for live migrations is less intense which I think will suite my needs perfectly.
Hello just found your channel and really enjoying your videos. You teach advanced networking with simplified understandings. Much appreciated!
Thanks! I try to explain how and why you need to do certain things without getting too deep into the weeds.
I like to think it’s like teaching you how to operate all the controls in a vehicle rather than how everything works under the hood. Of course the more you know under the hood, the more things you can do.
This video shows step by step installation Proxmox to OPnsense.
Thanks
You’re welcome! I plan to into configuring a Proxmox cluster soon and show how you can live migrate OPNsense to different nodes with very minimal downtime.
I lust got my PVE/ OPNSense machine running and in my rack a couple days ago, and I just found this today! I also used your Pi Hole PVE guide and set that as my DSN server. I used an 8th gen Dell OptiPlex with a dual 2.5gb card, and am thinking of setting up a second machine for a HA cluster.
Nice! If you set up a cluster with 2 nodes, you need to make sure you have a 3rd device as a “Q” device (a 3rd voting member) so you can have quorum. You need an odd number of devices so you can reliably know which nodes are available.
@@homenetworkguy Good to know!
Great video. Great explanation.
I would've liked to see a draw io diagram. This helps visualise the intended design.
Thanks! I intended on creating a diagram in this video (and some others as well), but I had a lot on my plate and wanted to get it out there. I’d like to spend more time doing polishing the videos further, but it would take me 2-3 months instead of 2-3 weeks per video. Haha. (I only do this in my ‘spare’ time). If I can get caught up on some things I’ll try to do better about including more diagrams in the future even if they are not super fancy.
I use this on my server in the datacenter. Works perfect!
Following my instructions or you already have an OPNsense VM in your datacenter? Either way, that’s awesome!
@@homenetworkguy In production for +-1,5 years. I can also acces ipmi with a vpn that is not running on the server ;-).
Thx. Very clearly explained. And exactly the process I'm about to do. Thx for the virtual hand-holding!
No problem! Glad it was helpful!
You should enable "Discard" (for trim) for thin-provisioning to work properly. If you disable "Pre-Enroll keys" then Secure Boot won't be enabled so there's no need to disable it later.
OPNsense (and pfSense) recommend to disable all off-loading settings. At least for virtual NICs.
Thanks for those tips! I should’ve looked up Discard to better understand if it was necessary or not.
Funny thing is that the pfSense documentation shows to do it that way for disabling Secure Boot (docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html#booting-uefi). I figured their docs would also work fine for OPNsense for recommendations for VMs. Since I always use OPNsense in a VM for demo/testing purposes I didn’t care about optimal settings as much but if I use it as my main router/firewall, it becomes more important!
Hardware offloading is disabled by default in OPNsense which is why I never typically mention doing it. I think for pfSense it may be enabled by default.
@@homenetworkguy Yeah that's funny but one can't know everything. The PVE docs (I apparently can't link things without the comment being deleted) say this
> pre-enroll-keys specifies if the efidisk should come pre-loaded with distribution-specific and Microsoft Standard Secure Boot keys. It also enables Secure Boot by default (though it can still be disabled in the OVMF menu within the VM).
To elaborate on the discard as far as I understand it. On most linux OSs there's a weekly "fstrim" timer which calls "fstrim" which gives unused chunks back to the underlying storage. Assuming the virtual disk is on thin-allocated storage and "Discard" is enabled, of course. I believe windows also needs the "SSD emulation" option. I'm not sure how pfSense/OPNsense/FreeBSD handle trimming. I'm very far from an expert with BSD. Trim seems to be disabled in my OPNsense VM according to "tunefs -p" but I'd recommend to enable "Discard" for every disk on thin-allocated storage.
Yes, I appreciate when others let me know about details such as this so I can continue to learn as well.
To clarify from what I looked into this morning-- without discard enabled, the initial VM storage doesn't take up the full 64 GB when I looked at the disk usage. It's sitting at 3GB and I have a few CTs set up as well.
However, I'm assuming discard will help free up space on the host when data is deleted from within the VM. It's good to know that it doesn't fully allocate the 64 GB even if discard is left disabled. I'm not sure how trim is handled in OPNsense either.. I think I've seen others talk about it at some point but not sure if it is something that needs to be enabled to make it function properly.
I didn't know you could do raw passthrough on PCI devices without Iommu. That's cool. 👍
Yep you still need to have virtualization features enabled in the BIOS but if you don’t also enable IOMMU on Proxmox, only raw device pasthrough is available.
This is an awesome video. I am trying to learn about this stuff so I can do it in a few months after a move.
Thanks! I’m planning to expand upon this and show clustering in Proxmox. I will demonstrate how to manually live migrate the VM to another Proxmox node as well.
well done, many thanks for this great video. Will help me a lot on the way to my first home-lab :)
Great that you found it helpful in creating your first homelab!
This is SUCH a good video.
Thanks! I appreciate that you liked the video!
Thanks for the content. Playing with some similar setup on mini pc's right now.
You’re welcome! Have fun!
If you want to do this with VM's, I strongly suggest using Triton on SmartOS instead: you get IPFilter, Solaris zones technology, mdb, DTrace, ZFS, Fault Management Architecture (FMA), Service Management Facility (SMF, like systemd but way better) and the Crossbow network virtualization, which allows one to create virtual switches and routers, and yet everything runs on bare metal because Solaris zones virtualize the kernel and not the hardware (and yet still enable one to virtualize other operating systems like Windows, GNU/Linux and FreeBSD with both KVM and Bhyve).
Yeah there are lots of solutions out there you can run but the video is not focused on the option you described.
Thanks for the suggestion though. I can look into to it further at some point.
@@homenetworkguy that'd be cool, another step-by-step video.
good video. One thing you touched on but did not get into is if your PVE (with OPNSense) goes down, you lose your router. It would be good to understand how you would migrate this over to a second PVE without losing routing. I suspect you would need a machine with the same number of LAN ports which have the same virtual bridge names in order for it to migrate properly. (I want to use OPNSense but I want to be able to migrate it between PVEs in a cluster).
I mentioned that you need a backup plan if you only run a single node since it will take your network down which I mentioned near the beginning about how I prefer bare metal because I’m considering using a Proxmox cluster so I will feel more comfortable about virtualizing OPNsense for my primary router/firewall. I plan to show my cluster configuration in the future. It will be pretty awesome to be able to live migrate my primary router/firewall with less than 1 second downtime!
@@homenetworkguy that would be nice, would love to see this. Especially how to do this when your provider allows only one device with the public ip
@@homenetworkguy I always double router... I keep the ISP provided router in front with family wifi. Then have a proxmox/opnsense router behind, so I have my own network I can freely break without affecting the family. Which is good because sometimes I break it a lot 😅 I've heard double router can cause problems but so far I've never faced a single issue caused by double router so not sure what that's about
@@CowCow-o5m I also play around with OPNsense VMs on a separate lab network for the same reasons. I try to keep the main network stable for my family and also because I work from home (and my wife does some work from home too). Having a separate lab network is nice because I can play around with stuff so I can make guides/videos and I don’t get tech support tickets if something breaks. Haha. But I will move to a virtualized OPNsense once I set up a Proxmox cluster because it will provide me with more redundancy so I will feel more comfortable virtualizing the main router. It will allow me to migrate to different hardware much more easily since I tinker with different mini-PCs and other hardware on a regular basis.
@@CowCow-o5m yep when you have a family, you are basicly on -call 18 hours a day 7 days a weeek, the more complicated, you have as part of the family networking, the more likely it wil break, and of course it always breaks when you are the busiest. If do this i'm going to use opensense on a virtual network that doesn't leave the system then i can have more bandwith between VMs and I can play around with rate limiting, and other firewall features.
Thank you for this! You’ve earned a sub
You're welcome! Appreciate it!
Even if you only start with one proxmox host it is advisable to create a cluster before creating the 1st VM. Not used v8 but this was the case with v6 and v7, A host with a VM cannot join a cluster.
Good point I hadn’t considered yet. I haven’t created my cluster yet but plan too soon. I can easily back everything up to my PBS system and restore it back on the cluster.
I looked into this further. The primary mode where you create the cluster can have VMs/CTs running but any new nodes that you are adding to the cluster must be empty to avoid naming conflicts between nodes. Makes sense. I was worried I would have to start over with a clean slate to create a cluster. I have backups on PBS so it’s easy enough to start over if need be.
Thanks for this. I'm definitely wanting to setup Opnsense and Proxmox, I just don't know what on. I like the chassis design and ports on these Protectli units, but god they're expensive. The Minisforum MS-01 gives you a mobile i9, the same two SFP+ ports (it's even the same model of Intel NIC), two 2.5G RJ45 ports (also same model of Intel NIC), two USB 4.0 ports that can do 40 Gbps, three NVMe slots (albeit only one of them is PCIe 4.0 x4) instead of an NVMe and 2 SATA slots, for like $220 less than this. If you get the i5 version (which still has a better CPU than this one) it's $460 less. It's pretty ridiculous how expensive this thing is to only have an i5 in it. I'm not sure the extra 2 RJ45 ports, better chassis, better firmware support is worth paying so much more to lose out on hardware. It's quite a dilemma.
I only have 1 Gbps for now, so realistically I'd be fine with one of the cheap Protectli boxes if I was going with barebones Opnsense on it (aside from running ZenArmor and such), but I want to upgrade to 10G LAN at some point so I'd like to have the support for it to make routing between VLANs faster among other things.
Yeah you have to weigh the pros and cons. I definitely wouldn’t use the MS-01 as a dedicated OPNsense box. Systems like these are too powerful not to use virtualization to make full use of the hardware. Not all of the services in OPNsense take full advantage of all the cores. In fact some of them may fight for the same couple of CPU cores (the Zenarmor team has noted as much to me).
In your network setup i see there is a cable which connects port 1 of protectli to the port 2 on the switch. Why do you need that when you 10gb SFP+ connected between them?
I like dedicating one interface to manage my Proxmox server so I can plug directly into it, if need be. I’m using the 10G interface for all of the VLANs on my network. You don’t have to do it this way. You can use a single interface for everything. I have the network interfaces to spare so it’s easy enough to use it that way.
This also allows you to separate tagged and untagged network traffic as well which is recommended by OPNsense because there is the possibility of allowing traffic intended for the parent interface to the associated VLANs on the same interface (if you’re not careful with how you write the firewall rules and it may also require a network switch which have a specific flaw).
OMG right in time ... Thank you a lot :)
Love it when the content is release just in time!
very detailed video, thanks for share it
You’re welcome!
The DHCP protocol RFC specifies that the DHCP client is supposed to solicit address offers, which implies that multiple DHCP servers should be on the network, and indeed, if one is managing one's network with the dynamic host configuration protocol, one should have as many DHCP servers as possible to increase redundancy and availability, but no less than three (so there can always be quorum).
When I say you shouldn’t have multiple DHCP servers on the same network, I’m not referring to high availability but rather 2 different DHCP servers running on 2 different systems not in a HA configuration. There will be conflicts of IP address assignments if you have a rogue DHCP server running on the network if they’re using the same IP ranges. I imagine there are ways to mitigate this but I’m always speaking from a home network context (hence the name of the channel). You don’t really need have a minimum of 3 DHCP servers on a home network (I’m counting DHCP listening on multiple interfaces in OPNsense as one DHCP server). I’ve survived 25+ years of home networking using a single DHCP server. Typically I have other problems that are not DHCP related.
@@homenetworkguy that's exactly what I wrote about: one should have multiple DHCP servers for the same network, even if that network is a home network. The challenge, of course, is keeping them in sync. I keep my configuration packaged in OS packaging format in Git, so when I modify it, I bump the package revision and upgrade the configuration package on all the DHCP servers. Problem solved!
Yeah, that's a great solution for high availability, but I typically throw out the general warning that if you spin up another OPNsense or plug another device on the network which has its own DHCP server running on the network (that is NOT synchronized) that it could cause problems.
What you are describing is syncing up the DHCP servers which is an intentional action to have redundancy on the network, which is not a bad idea for improving the reliability of the network when services or hardware fails, and shouldn't cause problems on the network when implemented properly.
I've spun up VMs and or connected another OPNsense box to the network when testing and it would temporarily disrupt my network until I disconnected it because I accidentally connected the wrong interfaces or had some configuration incorrect.
I have 8g symmetrical at home, as a non network guy, if I want to use IDS/IPS and pihole / unbound dns + wireguard. is that something Protectli VP6650 can handle. I don't know how much power you really need. I most likely won't vlan too much more of a simple router -> switch to nas and computers and then router-> 2.5 directly link to nas port for DMZ sharing
Without IDS/IPS, it shouldn’t be a problem but it can’t do IDS/IPS on OPNsense at 8 Gbps because not all of those services are fully optimized to take advantage of all the cores on the CPU. You may potentially have better luck with other operating systems. I haven’t tried other firewalls such as IPFire yet. It’s Linux based so it may perform better. I should try it before I start using the VP6650 in my future Proxmox cluster.
@@homenetworkguy would love to see it. i wish there was a chart that just said you need x for y feature somewhere, the information is always vague or refers to buying some enterprise level hardware, pretty sure my wife would not be happy to that purchase. vs something smaller one could build out ^_^
I’ve thought about creating a chart/table for the hardware I have personally tested to help others determine how much hardware they need for certain services in OPNsense. I wasn’t able to test all of the older boxes I have quite as thoroughly but it’s getting easier for me to set up test cases since I have more sponsored hardware and other hardware that I purchased available for testing.
The very first sentence of the video is how IT sounds to people who have zero clue about IT.
Yeah… it’s a mouthful of IT jargon.
dude, I would hug you and kiss you for this amazing video brother, you got a sub and a like
Haha, I take it that you really liked the video. I have one that goes one step further with creating a basic Proxmox cluster that’s dropping soon. It’s very cool to be able to live migrate your router/firewall without the network connection noticeably dropping.
@@homenetworkguy Yes, I watched it too!!! you are awesome man, I hope you know you are doing an amazing service to the community!!! God bless you and prosper you!
Tnx
You’re welcome!
Excellent video 👍🏻 I needed this 6 months ago (figured it out the hard way!) 🙄 Have a smoothly running virtual opnsense on an R86s for some time now 👍🏻 quick question, I have a cluster of nodes and want a fallback scenario in case main node with opnsense dies - how would you propose moving the virtual instance to a different node and still keep network settings?!? 🤔 Might make for a great follow up video ?!? 👍🏻 keep up the great work . . .
Thanks! I’m planning to show I will do this in a cluster. With the limited research I’ve done, you would want to ensure the bridge names are the same on both nodes so the 2 machines would need to be configured similarly in that regard. Also if you’re not using shared storage, you would need to restore from a backup (and there might be a step to “manually migrate” the VM to a different node by messing with the config files since the VM wasn’t migrated while the node was still alive- not sure about that one yet until I try it out and/or do more research).
From experience, even running on a multi node cluster with full DRS running, virtualising your firewall is not a good idea in an home lab. It sounds like a good idea, its a good project to get your head around, but just don’t do it. Save yourself a world of pain. Thats said, this is probably the best Proxmox setup video for new users I have seen.
What kind of pain? In planning to mostly keep OPNsense on one on the nodes so I can live migrate it. I’m not going to do any of the high availability features nor mess with shared storage or Ceph to keep it simple as possible. I just want to be able to move VMs between nodes if I take a node down for maintenance or if it fails. I’m not concerned with automation failover scenarios which is another reason (among other reasons) I haven’t implemented high availability with OPNsense itself.
@@homenetworkguy If anything at all goes wrong with your host infrastructure, either physically or you make with a mistake your config then you lose your connectivity. In an enterprise environment which is strictly change controlled then I am happy with virtual firewalls, but in a home environment unless you have similar controls, built and proofed in an dev environment and then rolled out to production, invariably you will make a mistake, mess up a VLAN assignment, trunk, host or the OPsense VM and then you are dead in the water as you will have no connectivity across your VLANs and no internet connectivity. That was my experience and attempting to get my environment back up and running at 4am in the morning and was not fun. It looks like those who have this working as a solid solution have a much better at home based change control than me. Love your video's BTW and thank you for this video in particular.
Yeah, I understand the need for tight control for configuration management in the enterprise, but home networks typically aren't nearly as complex so it should be easier to manage. I don't make major architecture changes very often but I plan for some down time when I do.
Also Proxmox clusters can be relatively simple and not be configured with all of the high availability features. At the bare minimum, you can simply group systems together so you can manage them all from a single UI and you can migrate VMs between them. That's mostly what I would be interested in because it's quicker than backing up VM, shutting it down, and restoring the VM on a different independent Proxmox node (if not using clustering). There is a less than 1 second cutover from what I have seen from others which is pretty sweet.
Since you mentioned DRS, you might be more familiar with the VMware world which perhaps may be more complex to configure/manage clusters (I don't have personal experience in that area).
I'm going to give a Proxmox cluster a shot soon, but I could always keep an extra box with a bare metal installation to swap out if need be. Wouldn't hurt to have a hardware backup!
DETAILS, details!
“It’s hard, complicated and error prone!” (Only for “some”).
I did run my main pfsense, plus 2 more for HA, under ESXi, for a few years and there was NO SUCH PAIN!
The main reason that I run pfsense on a dedicated machine, is because I found cheap used quad core mini PCs that work perfect.
The “people” that utter vague claims like this, usually don’t know the stuff well!
what would you see as some other VMs running on that box? Zabbix, Plex, maybe a web/mail server in DMZ? nakivo backup solution? how many more can you have on 4 cores? did you test it?
I completely replaced my old Proxmox server with this Protectli and it runs everything even better than my old server which was a Ryzen 7 1700. I only use 4 cores for the OPNsense VM since I noticed it doesn't tend to use much more than that. I have most of my hosted network services running on the box such as Plex, Nextcloud, Caddy reverse proxy, Vaultwarden, UniFi Controller, Grandstream GWN Manager, RustDesk, Uptime Kuma, Homepage dashboards, etc. I haven't even reached full capacity yet. It runs in a Proxmox cluster so I have stuff running on different nodes for various purposes (one is mostly dedicated to Home Assistant while the 3rd node hosts all of my apps/VMs I use on my LAB network).
Thanks!
You’re welcome! Thanks for the support! I appreciate it!
Fire 🔥!
Haha, thanks! Took a bit of effort some to get it made but my favorite videos are real world examples pulling multiple concepts together.
@@homenetworkguy man this was perfect and honestly I appreciate the content it’s helpful for poeple who want to try this and the examples and explanations is perfect for beginners . Will be showing my friend as well who’s trying this to
Thank you!
You’re welcome!
Good video. I wish you didn’t record the slow vnc session though, the delay is like a tick haha
Thanks! Sorry about that. Sometimes it’s easier to remote into another machine but sometimes I also use SPICE for my Proxmox VMs which doesn’t have the second mouse cursor/delay.
should i use this linux bridging or passthrough the nics? i want a mini pc with 2 nics. 1 for wan and 1 going to my switch were i use vlans. still need to figure out how i can use the vlans etc.
If you need maximum performance, you use can use passthrough. Otherwise bridging will be fine if it doesn’t hinder throughput (depends on the speed of the CPU on your system).
If you are using a cluster, you would need to be careful when using passthrough especially if using different hardware. There is a resource mapping option for the cluster but I haven’t tested it to see how well that works when live migrating VMs (and haven’t tested full high availability either). I mainly keep my cluster simple and manually do live migrations when I need to reboot one of the nodes that has my OPNsense VM.
Is it possible to have 2 VMs of OPNsense working as Active/Passive mode? For high availability and reliability on home network? If so, how it would be look like? Would you pls consider making a video for that? You can mention me as “Abu Rayyan from Baghdad” next time 😅 never been called out on RUclips algorithm 😂
Yes but it’s almost has value to do so especially if it’s running on the same Proxmox server. VMs are very easy to backup and restore and you can take advantage of deduplicated snapshots with Proxmox Backup Server as well to get you back up and running quickly if something goes wrong. Could make a HA video because it’s interesting to learn even though I wouldn’t personally use it especially since I only have 1 public IPv4 address.
@@homenetworkguy I did some research and it should be possible and it's more ways to do it. And 1 public IP is enough, HA is good for HW failure too. It's my future plan, after I end with this vlans etc. sht I would like to learn and understand properly.
i feel like it's not really your "primary router" if proxmox is still in front of the OPNsense router and using the wan for management, i did it today with proxmox behind OPNsense and it's much safer, just not sure how to set up the pve>system>network,DNS,certificates thing as i am absolutely new to proxmox
It is your primary router but just virtualized. You can plug your modem/ONT directly into the interface used as WAN on Proxmox just like you would on a bare metal installation plugging into the WAN interface. Proxmox is not doing any of the routing or firewalling for your network-- OPNsense in the VM is doing that task. This is the nature of virtualization. Proxmox is not "in front" of the OPNsense router. Rather, Proxmox is simply hosting the router/firewall software in a virtual machine (all routed network traffic flows through that VM just like a bare metal installation). I am currently using a bare metal installation of OPNsense, but I will probably move to a virtualized installation (in a Proxmox cluster) so that I can have more flexiblity to "move" my router to different hardware without doing a separate bare metal installation. I can just migrate it over to a different machine. Since I test out various hardware, that flexibility will be great to have.
As far as security is concerned, the main security risk with virtualization vs bare metal is escaping the VM sandbox. If an attacker can break out of the VM, they can get on the host system. Those sorts of attacks are very rare. Other than that, the security is generally pretty much the same. I understand virtualization is not for everyone. I have guides that show both bare metal and virtualized instances of OPNsense.
@@homenetworkguy did you notice that after installing opnsense and setting it up as the main proxmox router that pve>system>network,DNS,certificates etc have to be changed to match the new network?
Could you do a video with network PCI passthrough and VF functions?
I could do that even though I'm not an expert on PCI passthrough. However, passthrough of NICs is pretty straightforward, but what I would like to learn is using SR-IOV which basically allows you to passthrough NICs to multiple VMs at the same time (kinda like how you can partition up certain GPUs to multiple VMs).
I believe newer versions of Proxmox already have IOMMU enabled so that saves some steps when passing through hardware.
@@homenetworkguy would love to see the whole SR-IOV side. I can get PCIe passthrough to work...but once done how do I use the VFs to assign to other VMs is where I get hung up.
Hi once again thank you for everything. Small question wrt VLANs; let’s assume as in the above video, a Nic / bridge is made VLAN aware, and then connected to a switch that has say 5 physical ports and each port is being used for 5 separate VLANs (VLAN id 2, 3, 4, 5, 6)
Let’s say we create a Proxmox CT or VM but want it to have a new VLAN id 7; do we need to do anything on the switch itself ? Or should we just input 7 as vlan id in the CT/Vm. Are the number of VLANs restricted to the number of physical ports on the switch.
Would really appreciate your views on this .
First of all, 1 of the 5 ports will need to be connected to Proxmox and that port will have ALL VLANs assigned as a trunk port so the traffic can pass through to Proxmox. A trunk port can have as many VLANs as you want to pass through to other switches, routers, wireless access points, and servers (all of which need to be VLAN aware devices). Each non-trunk port can only be assigned to a single VLAN. In Proxmox you can create a virtual bridge (with or without VLAN tags) that you can use as a virtual network within the Proxmox server or Proxmox cluster- if you wish to have CTs/VMs on their own virtualized network (this can be helpful for lab networks, etc).
@@homenetworkguy the last sentence is what I want to do; create virtualized CTs in 3 Vlans, 1 each for Caddy, Apps, and Arr stack. I created a Vlan aware Bridge that wasn’t connected to any NIC, and all was working fine, but the setup is causing OpnSense to crash / restart.
I’m now thinking of using my last remaining Nic, making a VLAN aware bridge, creating 7-8 VLANs and trunking them to my Mikrotik Hex which will only have 4 available ports, to be used to separate physical devices such as office, Iot, guest etc. The only 3 VLANs to be used for Caddy, Apps, and the Stack.
Am totally lost here.
Excellent but you missed out on showing the temporary firewall rule to allow all vlan networks to see each other.
Yeah, basically I had allow all rules on every network. I had to decide if I wanted the video to be 40 minutes or 1.5 hours, etc to show a full build (which I have done twice already.. I may do a 3rd in the future as I slowly work to improve overall production quality, etc).
thank you for the video
one question, after configuring opnsense, and use it as main router connected to our network, opnsense is the new dhcp server? is something we have to configure it? before was de ISP router...how do we manage the ip given to proxmox for management if we change network with new opnsense router? thank you
You're welcome! Yes, OPNsense will be the new DHCP server. By default the LAN interface of OPNsense has DHCP configured but you need to configure DHCP for each new interface that you add (physical interface or virtual interface aka VLANs).
In the example, I made the Proxmox management IP an IP address in the LAN network of OPNsense so it should still work fine once you switch over to OPNsense as your primary router.
@@homenetworkguy thank you for the info and for answering
Nice video.
Thanks!
Thanks so much for your videos; I have a problem however that after executing a Proxmox apt update/upgrade; I can't access the OPNSense Gateway 192.168.1.1; I am on the LAN network and can ping other devices on the Lan, access the internet, but just can't seem to ping the Gateway or access the Web Interface. Should I have not restricted the Allowed Interfaces to Lan (only). Any advise would be appreciated;
I have not encountered that issue after updating or rebooting Proxmox or the OPNsense VM. It’s hard to say what happened with knowing more details because if everything is on the same network using the same bridge for the LAN/Proxmox management interface, you should be golden.
Super movie! I only have 2 network cards in my computer. Can this be done?
Yeah you could dedicate one interface for the WAN but the other interface will have to be the Proxmox management interface, the LAN interface and if you want any additional VLANs. You could experience some bottlenecks using a single interface. The configuration will be a bit different than what I demonstrate but the concepts should be the same (you would just use the same bridge interface for all internal networks instead of separate interface(s)).
What did you mean when you say "we are not plugged in" at 32:49? I have one built in NIC and I use 2 USB to ethernet adapters to create 2 more network devices. All 3 are connected to the same switch, so I am plugged in. I see LAN "192.168.1.1/24" and for WAN "192.168.0.204/24". And I can't visit the OpnSense UI on either IP. I know I am doing something wrong. Tried the same steps 3 times.
Depending how you are configuring your devices, it’s sometimes best to not have everything plugged into your existing network because there could be IP address conflicts or you could end up with 2 DHCP servers running on the same network, etc. I believe I have the system I’m configuring OPNsense with plugged directly into the management interface of Proxmox but I manually set a static IP address on that system. You should be able to access the Proxmox web interface as well as the OPNsense web interface if you are using the same bridge for the OPNsense VM.
I've thought about doing this.
Cool! I hope it goes well if you do!
Is there a way to give Proxmox itself internet access after committing Opnsense as a main router?
I'm having issues with updates and LXC installations, as Proxmox keeps returning no internet errors
As long as the Proxmox management interface is connected to one of the networks managed by OPNsense, you should have Internet access on Proxmox.
Thank you so much for your videos. I have gone through your first and this video for configuring OPNsense on Proxmox. Everything works great as you have explained. But I am trying to configure LAGG (LACP) and didn't get success in this approach. Is it possible to do LACP with this approach?
You’re welcome, thanks! Yeah I’m sure you could, but it’s possible to create the LAGG on either Proxmox or OPNsense so I’m not quite sure which would be the best approach (you need to it on on side or the other but not both).
@@homenetworkguy Thank you for the response. I tried it from OPNsense, it doesn't work for me. Let me try it from Proxmox, and let you know.
Yeah it’s possible you would need to passthrough the physical interface for it to work in OPNsense so it can have direct access to the network interfaces. But if you do it in Proxmox, I’m thinking you could use the LAGG interface of Proxmox in the OPNsense VM and treat it like a normal single physical interface in OPNsense. I haven’t tried that out so I’m not sure how all that would work. Hah
@@homenetworkguy Yes, I think it will work if I pass through the NIC to OPNsense VM. But I am using a Realtek card, so I am trying it through Proxmox. And also, I am setting up VLAN and want to use this VLAN in other VMs and CTs. Let me try the LAGG in Proxmox and see how it goes.
You are plugging the Zima board into the LAN port of your virtual firewall, so you can get to the GUI correct? I have PCI passthrough on my OPNsense VM but I should be able to just plug my laptop into the port that has my LAN and get going correct? PCI passthrough should not change anything on me?
Yes. It should work with passthrough or bridges. You can connect either a PC or a network switch to that interface.
@homenetworkguy i have the lan side setup as a vlan...still does not matter?
In that case it would be better to connect a network switch to that interface and have your client plugged into a port on the switch configured with the LAN’s VLAN ID. Unless you can set a VLAN tag from the client if you wish to plug directly into that interface.
@@homenetworkguy figured it out. Needed to set the VLAN on the PC and it came up! I have been fighting this for 4 weeks. I really hate networking sometimes.
@@stevefxp Yeah it can get frustrating. I prefer to use the untagged LAN network for management purposes instead of a dedicated VLAN because everything defaults to it and I can just plug in to get set up. Then I push all other devices to VLANs and restrict access via firewall rules. You of course have to be diligent to ensure everything is on the proper networks (and I like to default unused wall jacks to the GUEST VLAN in case someone plugs in so they’re not on my management network).
When I get to the part where you're in the Web GUI, clicking on damn near anything causes a popup that says DANGER- unexpected error, check log for details. Only, there's no log. And the Dashboard screen is completely empty. If I close out the popups (plural), clicking on just about anything them to start again. Any suggestions?
The web UI of Proxmox or OPNsense in the VM? That is weird to see such an error because I've never seen a "Danger: unexpected error" message before! Almost makes me wonder if there is some file system or RAM corruption-- some sort of hardware related issues rather than a misconfiguration, but if it works for other software, then it seems like it would be a miconfiguration somewhere.
@@homenetworkguy Thanks for responding. After some more testing, it turned out to be Ublock Origin causing the issue... but ONLY on the Brave browser. That specific combination of Browser->Extension was the headache.
Ahh ok. I was wondering in the back of my mind if it was browser specific. That makes sense. I've had uBlock Origin mess up certain things because it's blocking things too aggressively or by keyword.
How does the Protectli Vault Pro VP6650-6 Port do on power at idle?
I notice it uses about 20-22W but I had a couple network interfaces plugged in and I have a second disk (SSD) which would add to the base wattage. However I think that’s a good basic use case for real world wattage. It has faster single threaded performance than my Ryzen 7 1700 Proxmox server but at 1/4th the idle power consumption. It uses about twice as much power as their 4 port models but it’s also much more powerful too. I have the VP2410 and VP2420 and the two systems combined uses nearly the same power at idle as the VP6650.
Thanks.
You’re welcome!
do you need 3 ports to have more vm's or can i use the out port for vm aswell i have a mini pc with 2 ports
You could do everything with a single interface if you like! I like using more of the interfaces available because it means less bandwidth is shared. Also for certain things such as clustering Proxmox, it’s recommended to dedicate an interface just for the cluster traffic (on its own network). If you have fewer ports and you have multiple networks/VLANs set up, you can just assign the CTs or VMs to whatever network you like. You have to make sure the network switch is set up for VLANs. The key thing to consider is that you need to use bridges instead of using pass through so multiple CTs and VMs can share the same physical network interface.
why did you disable proxmox firewall for wan but not for lan?
Probably not intentional but I don’t use the Proxmox firewall since I use OPNsense to firewall the network (and I use local ufw firewalls on all my Linux containers/VMs). If you have box checked to enable the firewall, there are no rules defined by default so it doesn’t offer any protection. You have to add rules in Proxmox if you want to further restrict access. I suppose it is yet another layer of protection if you want to use it in addition to everything else.
How can I hide Proxmox behind an OPNsense firewall if I only have 2 Ethernet interfaces (WAN and LAN)?
Thank you!
You would have to use a bridge for the LAN interface similar to how I demonstrated in the video. It would be the same interface you use to manage your Proxmox server. You can’t use PCI passthrough on that LAN interface and also use it as the management interface for Proxmox because that interface will be dedicated to the OPNsense VM if using passthrough.
Hello, very nice video, but could you make another video about OPNsense KEA DHCP New
Thanks! I've had a few requests for Kea DHCP. I'll get around to it eventually since it will be the new way forward but currently I do not believe it is considered feature complete so I do not see myself personally migrating any time soon (but I will likely do a video on it before I make the transition on my own home network).
@@homenetworkguyI switched right after update with Kea support and it's not that hard to set up. And it's working without problem. Only con I see is no hostnames, only theirs IP adresses (in DNS server, monitored communications etc.).
hey at first, thx for yout tutorial. i got one question. at the network config you give 4 to queues. why? can you explain int to me pls. i m new in the game and dont find a easy answer in the inet. thx
It allows the guest virtual machine to have virtual CPUs process the network traffic which can help improve throughput. According to the following link, it is recommended to set the multiqueue value only when anticipating a lot of network traffic since it increases the CPU load of the host/guest as network traffic increases: forum.proxmox.com/threads/multiqueue-inside-of-vm.66321/
@@homenetworkguy thx for your fast help 🙏
I have been able to follow the instruction, install Proxmox / Opnsense and everything is working fine. I however am unable to update proxmox, and keep getting a message “download failed unable to resolve host” when trying to download Lxc for pi-hole. I feel proxmox isn’t able to access to web. What can I do to solve this. Please help.
You’re using the same bridge in Proxmox as the LAN interface in OPNsense? As long as you don’t pass through that interface which is used for Proxmox management and you have the LAN configured properly in the OPNsense VM, it should have access to the Internet just like any other device on the LAN network.
Realized my error; reinstalled proxmox / opnsense and all is well now. I guess this all is the learning process.
@@cyrilpinto418 Nice! Glad you got it resolved. Sometimes missing a minor detail can cause a problem.
Is it possible to do this with only two physical eth ports on my proxmox box? I tried following along best I could with this. My opnsense is running good and I'm getting internet through it, but I cannot reach my proxmox gui. I can ping the box, but can't ssh into it. Both operations time out. From the proxmox box, though, I can ssh to my machine just fine.
You should be able to dedicate one interface to the WAN and the second to the management interface of Proxmox and the LAN network of OPNsense. You can even add VLANs on the second interface as well (but you’d need to add the VLANs to the network switch as well). Technically you could it all from a single interface using VLANs but the config is a little bit more involved. It’s easier to configure separate interfaces and also reduces the potential for bottlenecks in throughput.
You need to make sure that Proxmox has an IP address in the same network as the LAN on OPNsense (which defaults to 192.168.1.0/24).
@@homenetworkguy Thanks so much for the reply. I ended up figuring out my issue... my proxmox box needed to have its gateway set to the router. I had set it to opnsense, and from what I've learned that was causing asymmetric routing.
Nice! The gateway is the interface IP of each network- IP address which is used to route the data to other networks essentially so for the default LAN that is 192.168.1.1, as you likely are now aware. Glad you got it working!
Can I set up an LXC container on a VNET bridge and run Docker with multiple containers on different VLANs? (using MACVlans network?)
I would have to look into this more. One thing that I find annoying to deal with in Docker is its networking. Deployments are easy but then you have to mess with the networking aspect. Simple things aren’t too bad but what what if you want containers to be on different VLANs as you mentioned? I’ve always just put them on the same network in the past but that was before I started segmenting my network. I avoid Docker since I use LXCs (don’t need an extra container layer) so I haven’t tried setting up apps on different VLANs. Also a VM might be more desirable than an LXC for Docker (at least when I tried a while ago, restoring backups of LXCs which use Docker was problematic for me).
@@homenetworkguy I find setting up a Docker container much easier than using LXC, but maybe that's because I know more about it. I think having one LXC with Docker and multiple Docker containers is less overhead than having multiple LXCs.
I'd love to read about your findings!
Setting up containers on Docker is easy but the networking aspect is something you have to work through. I’m not sure without researching it further on how to run containers of different VLANs (I don’t know if MACVlan or IPVlan modes are what they sound like they should be used for.. almost seems like it’s for containers to internally communicate on different virtual networks).
The nice thing about LXCs (without Docker) is that I can allocate exactly the right amount of host resources that I want as well as install whatever I want inside the LXC (without needing to create custom Docker images for example). It’s very simple to put LXCs on different networks. I also like to utilize the ufw firewall in the LXCs so I can easily block all unused ports on each LXC (I know Docker only exposes certain ports for containers but it also interferes with the firewall on the host machine so you have to do workarounds to be able to use ufw firewall or iptables without Docker interference).
I typically setup SSH access for all of the LXCs so I can get into them if I need to do anything. I think you can do that with Docker as well but not sure if it’s as straightforward depending on the networking mode used (I believe I recall logging in more easily to a terminal window using Portainer long ago when I was using it).
I also like being able to back up individual services that are in LXCs rather than the whole Docker instance because I only have a few critical LXCs that I backup offsite. With individual LXCs, I can move them around to different Proxmox nodes easily.
Ultimately, it’s a matter of preference. There are pros and cons to either approach but I’ve come to like using LXCs better. A lot of people like Docker and I understand its appeal especially in deploying web apps that have a lot of dependencies including setting up databases, web servers, etc.
I will try to follow this at some point later, but I have already done this and I have one issue I do not understand.
I have the router software installed and some VM's. I start the router, start the VM's, and the VM's have no connection. I reset all services on OPNsense, and magically I have connection. I do not want to just have to reset the router every time - what can I do to fix this?
I'm not sure I've seen that happen but you can set up Proxmox to start OPNsense first and then make all of your other VMs start only after the OPNsense VM has started. You can even add a short delay to ensure OPNsense is up and running before anything else on Proxmox starts. This could potentially help with your problem, but I'm not sure why it doesn't detect the network is up and running.
@@homenetworkguy unfortunately I have tried this and it does not make a difference which one starts first. No matter whether OPNsense starts first or the VMs, services have to be reloaded.
I'm going to have to watch your video and make sure I follow it step by step because if you aren't familiar with this, there is something I have to be doing wrong
Hi, is it possible to install+test this out without having a NIC available for WAN during install process, and add it later (for ex. PCIe addon card)? and is it possible to change NICs later (remap physical interface or vmbr) to "upgrade" from 1Gbe to faster NIC, without reinstalling opnsense?
Sure, I think that would be possible! What you can do is create a bridge in Proxmox that is not assigned to any physical interface. Make sure you select that interface as the WAN when you install OPNsense (hint: if you make it the first network interface for the VM, it will be called vtnet0 inside the VM if you’re not doing PCIe passthrough). Then later you can update that bridge in Proxmox to use a physical interface. You can easily remap network interfaces later. That’s the beauty of virtualization. You can even do this while the VM is running (but caution is advised)!
@@homenetworkguy perfect, wasn't sure vmbr doesn't need any physical NIC assigned :O learned something new today :D
You can also assign VMs/CTs to that same bridge and everything would be on that same virtual network. This is pretty neat if you want a fully virtualized lab network within Proxmox.
@@homenetworkguy ah fully virtual SDN is something I'm yet to get into :D
@@homenetworkguy yup I can confirm works exactly as expected,
for fully virtualized opnsense install: fake WAN in prox, fake LAN in prox,
then make a random VM with Windows or whatever, assign real LAN, fake LAN (static IP in Windows at opnsense LAN range), that way it's easily possible to RDP into this machine and from its web interface tinker with opensense web UI, fun stuff :D
why do you create bridges instead of passing the device through to the vm?
It depends on what you are trying to accomplish. Performance is best with passthrough but you can’t use the interfaces for anything else. With bridges you can have other VMs and CTs be on the same network by sharing the same bridged interface. It’s very flexible but there is a performance penalty. Since I’m planning to cluster it makes it easier to migrate VMs between nodes.
Hi! I have to connect to the proxmox host everytime I want to manage it?
No. I can't recall if I mentioned it in the video but once you have OPNsense set up and you're using the same bridge for the LAN in OPNsense as the bridge used for the management interface of Proxmox, you can simply connect the interface you are using to manage Proxmox into your network switch. As long as you have that port on the switch left at the default of VLAN 1 (untagged), the Proxmox management interface will reside on the LAN network (which I use as the management network in this example and on my home network as well-- I just try to be careful to keep everything else assigned to the proper VLANs so I can keep the LAN network protected and isolated from the rest of my network).
@ thanks! Suppose for example, my original network is on 192.168.0.0/24 then the opnsense vm on proxmox i set up to be 192.168.1.0/24 do i have to change the ip of proxmox to be on 192.168.1.0/24? I hope im making sense. Thank you again for this video!
I got it figured out now. I misconfig the port, but after rewatching the video i got a better idea. Thanks again for this tutorial!
Nice! Glad you got it figured out. The management IP for Proxmox needs to be in the same range as the OPNsense LAN-- if you leave everything at the default, it should already be set that way, I believe.
@@homenetworkguy i finally have a functioning network. The isp router is on bridge, minipc with proxmox and opnsense on it. I do not have a managed switch so no VLAN. But still its crazy to me. Thank you man!!!!
Any alternative to do this without a ZimaBoard 832 or something similar pricey?
Ohh yeah. You can use any PC you want to administrate your devices. I was just using a ZimaBoard because it's much smaller than setting a full tower PC on top of my desk to show all of the connections, haha.
@@homenetworkguy Ah, thanks - I was quite confused and honestly didn't know if this board also did something else required.
But yes, I tried it with my PC and it seems to be working (can't tell until I did a proper configuration).
Haha yeah I just wanted a small PC to use to set everything up and to show where a PC could be connected on the network. Glad you got your network set up!
I saw you selected ZFS, is it still better than EX4 even if you only use RAID0? Is there any advantage for proxmos such as compression and deduplication?
It supports snapshots so you don’t have to pause or restart your CTs/VMs when you do backups (you can do snapshots with ext4 but you have to select LVM thin and not LVM for this to work). You can also take advantage of built in LZ4 compression which could not only save space but speed up read operations (I believe). Caching certain operations in RAM may help improve performance but I haven’t compared that directly. ZFS can still detect bitrot with a single drive because of the checksums but it wouldn’t be able to correct it without having redundancy. I don’t use deduplication with ZFS even on my TrueNAS system since it requires too much system resources.
@homenetworkguy thank you for answering.
One more question if you don't mind. I'm planning to get VP6650
I was just thinking, how can I design the storage wisely give that nvme is for VMs/CTs
2.5 SSD1 - Host
2.5 SSD2 - RAID1? What if I upgrade in the future and it fails, the upgrade will also replicated to it so it will not work?
Please advise what is the best storage design for it. Thank you in advance!
Yeah you could mirror the SATA drives (RAID1) for the host OS and use the NVMe for CTs/VMs. That's a good way to set it up and it's similar to how I used to have my 4U rackmount Proxmox server before I migrated it to the VP6650.
There's not really a good way to recover from a failed Proxmox update but those sorts of failures are pretty rare. I had an issue long ago but it was when migrating from v6 to v7. Not sure if it was self-inflicted because I was new to using Proxmox back then.
If you keep some of your configuration files under the /etc/pve folder, it will help you with a new installation because you can recreate your configuration more quickly. In theory the Proxmox host is supposed to be minimally modified so that it's easy to reinstall or move to a new system (the idea being that your CTs/VMs contain most of the configured apps/services). However, in practice, you still need to backup some of your config files to save time if something fails.
I recently set up a Proxmox cluster so if I have a hardware failure, I can remove that node, and add a new one more easily because much of the configuration is at the cluster level (still a good idea to backup the network config because each node needs to have the interfaces configured appropriately).
Hey I'm new to networking and I just build my first home server. However after setting Proxmox up. I can't seem to access the webGUI using the PC to configure the creation of the OPNsense VM. I have assigned a static ip to my laptop. Any idea of what I'm missing?
Thank you!
Are you plugged directly into the Proxmox management network interface? Or connected to a network switch? You will need a static IP on your laptop only if you’re plugged directly into the Proxmox management interface. Otherwise you can use DHCP if you’re on the same network as the Proxmox management interface.
@@homenetworkguy im plugged directly into the interface. Followed your guide.
Did you configure the subnet of the static IP to be 255.255.255.0? Also make sure it’s not accidentally the same as the Proxmox IP address as well. You could try different interfaces on your Proxmox box in case you have a different one configured than the one you’re plugged into.
@homenetworkguy could I contact you on a discord or something alike to grt a bit more help. I'm really stuck and can't seem to figure out what is going wrong
I do have a Discord account. I don’t always hop on it but you could use that. Keep in mind that it’s becoming a bit more difficult to keep up with everyone’s messages. I still have a couple week backlog left in my email (I caught up on a couple weeks worth of email last night).
Any idea how to show the connected devices on my network? I just switched from a off the shelf router to OPNSense, but I can't seem to figure out how to see all my devices and their IP addresses.
Under the Services > ISC DHCPv4 > Leases page, you will see a list of all devices and IP addresses of the clients using DHCP. You won’t be able to see any devices that are using static IP addresses but you should be able to see everything else.
you can also use nmap to scan your networks
My goal is to replace my bare metal OPNsense firewall with two virtualized OPNsense firewalls that will be clustered. This means Proxmox clustering and then OPNsense clustering. In my trials I cannot seem to turn off the bare metal firewall and the virtualized firewall takes up the call. Am I missing something? I am watching your tutorial carefully but I am not seeing anything that you did, that I did not do.
This video doesn't demonstrate a Proxmox cluster nor using OPNsense in high availability mode. If you're going to use a Proxmox node with OPNsense in a VM, it might be best to enable the Proxmox high availability features and simply have the VM failover to another Proxmox node.
I think it may potentially be easier than setting up CARP on OPNsense (depending on your familiarity with Proxmox) since you only need to maintain 1 OPNsense VM instead of 2 separate OPNsense VMs on 2 different Proxmox nodes (also you can have other VMs failover to other Proxmox nodes and not just OPNsense). Plus I think failover with CARP might introduce more of a network disconnect than with using Proxmox until it detects the node went down (based on what I've seen others say).
I simulated Proxmox HA by doing a manual live migration of an OPNsense VM between 2 Proxmox nodes in another video. I don't have full HA setup on my Proxmox cluster (there are tradeoffs with having the VM on shared storage especially if it's NFS storage instead of Ceph, and it also increases the complexity of the Proxmox configuration a bit).
I try to manage the complexity the best that I can on my home network so I don't have to spend a lot of time fixing things that break. For the most part, things run pretty solid and only tend to break when I'm tweaking something (if it's not broke, don't fix it-- but you can't really learn as much unless you tinker and potentially break things). haha.
Does anyone have any experience with Protectli devices? I have heard from several people that they had issues with them dying
I have 4 Protectli boxes (since I have some sponsored hardware) and nothing has died yet with 24/7 operation. The oldest Protectli is about 3 years old.
I run the hardware in my server closet which runs a few degrees hotter than room temperature so the operating environment isn’t very hot. Something to consider when running fanless mini-PCs because the hardware might not last as long if it’s in a hotter room (do not run it in an attic for example in a hot summer, for example).
I will say that I always have my systems connected to a UPS and rarely have any hardware die unless is getting very old (which is to be expected). Most of the time my hardware in general becomes essentially ‘obsolete’ before I replace it.
First, thanks for the video. I think this has me most of the way there but am unsure on something. In my case, I will have Proxmox on a server colocated in a datacenter. I passed thru my NIC to OpnSense and it's booting and working. However, how do I allow Proxmox and other VMs to use OpnSense? I'll have a VPN running so I can remote in, and hopefully use the LAN IP address to still access Proxmox from afar. - thanks
You’re welcome! As for your question, I’m not sure of how many interfaces you have in the colocated server. Of it’s only one, then you cannot use passthrough because that means only the OPNsense VM can use that interface. You will have to use the default bridge interface in Proxmox. With only one interface it’s going to be tricker to set up a WAN/LAN interface but it’s possible using VLANs.
If you follow the basic principles in the video, you will be able to use the default bridge for both the Proxmox management and the OPNsense LAN interface. You simply just assign the same bridge to other VMs so they can be on the same network.
Could you make deep dive OPNsense firewall video next I'm having trouble understanding the firewall. I have OPNsense running on top of Proxmox with two NICs passed trough (WAN/LAN) and VLAN interfaces (10,20,30,40,50). I'm trying to allow Proxmox hosts in ManagementVLAN10 (10.10.10.0/24) to temporarily (or permanently) access my Unraid NAS VM web GUI in ServerVLAN30 (10.10.30.0/24) but I'm having no luck with it. In the future I also need to allow Proxmox hosts in VLAN10 network to reach Unraid (in VLAN30) for NFS purposes. I'm using Mikrotik SWos switch.
The firewall just doesn't click with me. I've watched some of your OPNsense and firewall videos but I'm still struggling. It's feels like OPNsense doesn't know the routes between VLANs since the firewall rules I create seem to do nothing.
It's hard to say where the config is going wrong without seeing any of it. Perhaps you could take a look at my website which the videos are based off of for more details since there may more explanations that will help you understand it better. It does take some time to wrap your mind around firewall rules when you are new to them (at least it did for me): homenetworkguy.com/how-to/set-up-a-fully-functioning-home-network-using-opnsense/
Forgive my elementary level question, but how (and on which device) do you manually generate an ip address to disconnect from your LAN and continue management? Thanks!
I just used another PC from the one I did the recording (I have a couple of mini-PCs I use for demo purposes). If you only have 1 PC/laptop and 1 Proxmox server, you’ll have to temporarily connect your PC/laptop to the Proxmox server to configure it. Once you’re done and have OPNsense installed, you can connect it back to your network. I’m assuming in the video you’re using the default LAN network for both the Proxmox management and the OPNsense LAN interface. That network interface is 192.168.1.1/24 (which means usable IP addresses between 192.168.1.2-192.168.254).
@homenetworkguy: Are you giving "pve-test" the new address of 192.168.1.50?
My current network uses a 10.27.27.x scheme, but I don't know how to locate my new proxmox node when I remove it from the network. 😕
You can assign the Proxmox static IP to be whatever you want so you can make it 10.27.27.100/24 if you like (make sure it’s outside your DHCP range to avoid potential IP address conflicts). If you plan to put OPNsense and make it your primary virtualized router and you still want to use that network address, you’ll have to change the default LAN IP addresses or create another interface with the appropriate IP address ranges. I tend to keep the default LAN network of 192.168.1.1/24 since it’s keeps things simple (but I add other VLANs too, of course, with different IP ranges).
Just figured out that it's the ip address for the "stand alone" pc used to configure the proxmox device that needs the new static ip in the proxmox device's network scheme. 🤯
Thanks for all the info (and patience)!
the first ip you will get on the first setpup of proxmox and opnsense need to be change to fit network
wish he had shown the part more clearly changing the IP of proxmox and opnsense to fit the network IP address in his network
feels that the hardest or complicated stuff
Yeah since I’m using the default 192.168.1.1/24 for both the Proxmox interface and the LAN network of OPNsense, I didn’t have to make any adjustments later to make them be in the same network.
I currently have a topton N5105 with 4 ports 2.5gb i226v. Would I be able to do this? Been reading around reddit that people were having random crashes? Is this still the case? I currently run opnsense as bare metal. But want have snapshots/ backups for quick restore
I believe this was addressed in newer versions of Proxmox. I know many had issues with the N5105 and the N6005 but I’ve used Proxmox with the N6005 without issue several months ago.
@@homenetworkguy that’s great thank you for replying. I plan to change over to proxmox. Can I use a Ethernet adapter (2.5gb) to use for proxmox/setup and setup opnsense. So I can set up my 4 built in ports as follows: WAN, LAN 1 , LAN 2 and LAN 3
You could I suppose but keep in mind if you use bridges, you can share the same port with your Proxmox host/VMs/CTs as demonstrated in the video. You don’t necessarily have to dedicate all the ports to OPNsense (you may need to use passthrough on the N5100 to achieve 2.5Gbps but faster hardware can handle 2.5Gbps even with bridges just fine).
@@homenetworkguy I think I understand it now. I have 3 Ethernet cables from my opnsense, LAN 1= server (unraid) LAN 2= WiFi access point upstairs and LAN 3 for lounge. And last port is my WAN. So if I understand correctly I can e.g use LAN 1 to install/setup proxmox and opnsense and then have my ports work in the same way?
Yes if you use the default vmbr0 bridge that Proxmox sets up during the installation. That’s the great thing about bridges but there is a performance impact depending on your CPU and the speed of the network interface. I’ve discovered that bridging performance in Proxmox is greatly impacted by single threaded performance of the CPU.
You have an amazing voice and know how to make it simple. I have a question. I followed all the steps .. however, i am stuck when accessing the opnsense via web gui.. Initially the proxmox got ip4 address from a different router with 192.168.1.1 then how can i access opnsense with same address. please help i am stuck.noob here
Thanks! Are you trying to set up OPNsense to try it out (or use it as a secondary network) or to eventually use it as your primary router?
If using it as a secondary network, you could put the OPNsense LAN on a different physical interface and connect a network switch to that interface so you can have a secondary network as a lab/playground.
If you want to make it your primary network router/firewall, if you followed the instructions in the video, you would swap out your existing router. However you could temporarily assign the OPNsense LAN interface to a different physical interface where you could plug a laptop/PC into. Then you would be able to access the LAN of OPNsense even though it has the same 192.168.1.1 address (since you’re not plugging that interface into your existing network which could cause problems with IP address conflicts, multiple DHCP servers, etc).
If you take a look at my new video of the Intel C3000 series video, I show setting up OPNsense in a VM on an existing network and plugging devices directly into the mini-PC to test out OPNsense. I don’t recommend setting it up like that in the long run but it shows how you can test it.
@@homenetworkguy i never expected a reply so fast. Highly appreciate your detailed response and time. I was able to connect to OPNsense. Appreciate it.
Why does almost everyone choose “Linux” as OS type when creating an OPNsense vm, when in fact OPNsense is FreeBSD 🤔
It’s either that or choose “other”. I think it affects the options that are available for the VM configuration since some options aren’t available for certain OS’s. Not sure if it makes a difference for FreeBSD based VMs or not.
Don't know why, but my Opnsense dashboard shows disk at 100% no matter what it is, memory and CPU look fine, but disk reporting seems to be a problem
That’s interesting. I haven’t seen that before. What disk type are you using for the VM? Not sure if those settings would cause a problem with the reporting aspect in a virtualized OPNsense.
@@homenetworkguy
Hi and thanks so much for responding, I recreated the VM(from scratch), and found the disk space reporting okay - no clue what I did to cause that, but it looks like its no longer an issue - strange issue nonetheless - hope it's not a sign of something more serious(nvme issue etc).
@@homenetworkguy
Hey I just wanted to update this, and say that the issue in question was due to 'not installing Opnsense' - hahah
In other words, and as we can basically run Opnsense without installing it, web UI and configuration alike, this makes it possible to actually forget the installer portion at the beginning of the process - which is all quite embarrassing to say the least
BTW, I discovered this, after installing(not really), Opnsense on bare metal, and finding the exact same phenomenon :/
Ohhh, that would be why the disk is at 100% because it's just running in live mode. Many non-Windows Operating Systems can boot in live mode so you can try it out without installing anything on your system which is pretty cool. With OPNsense, it all depends on how you sign in when you boot the installer. That is an easy detail to overlook if you are new to installing OPNsense! Glad you got it figured out! I wish I would have thought about the 100% disk usage as a clue that it was booting in live mode.
Just watch the video, but you did not showed how to configure firewall rules as by default opnsense block all the traffic i am also having the same setup as you showed in the video but i cant access my internet on lan network
Can you please give me some inputs here
You can create a rule on each interface to allow all access (protocol any, source any, destination any) for testing purposes.
@@homenetworkguy okay let me try it
why use ZFS in the first place if you are only using one drive?
ZFS has great features even with a single drive. For the primary OS disk you can take advantage of boot environments and roll back the filesystem to a known good state if an upgrade fails or some other issue. It’s just a nice robust filesystem. It takes advantage of extra RAM for caching, etc.
I see the same steps on my dell mini pc but for some reason I get a no boot device found no matter what.
No boot device before or after installation? Sounds like a boot order issue possibly?
@@homenetworkguy it appears I have been using the incorrect ISO. I would download the image from Open sense straight into my Proxmox and just realized that this was some kind of a zip file. I unzipped and now uploading manually but this method might take an entire day to complete. Stay Tuned!
Ohh yeah, you will need the DVD ISO image and have it unzipped before importing into Proxmox.
@@homenetworkguy Thank you again, Second Question if I set this up as virtualized just for learning can I keep it strictly isolated to my proxmox vms and not have it manage my main network/wifi. And what would be the best setup for that scenario?
Yeah for sure. I have a couple of OPNsense VMs I use for demos/testing, etc. The main thing you need to be careful of is not putting the WAN interface of the OPNsense VM on one of your primary networks while also having LAN interfaces on the OPNsense VM using the same IP addresses because the LAN interfaces will take priority over the WAN interface. It’s hard to explain but I’ll give an example. If you put the WAN interface of your OPNsense VM on the 182.168.1.1 network, the WAN of the OPNsense VM will be assigned something like 192.168.1.100. But if you also have a LAN interface in the OPNsense VM with 192.168.1.1/24, your WAN interface in the OPNsense VM will not be able to communicate with your primary network because the gateway address of the WAN interface will be 192.168.1.1 which happens to be the LAN interface IP address. One other gotcha is you will likely want to enable query forwarding under Unbound DNS if you are running into DNS issues. I’ve found that running a recursive DNS resolver behind my primary OPNsense box doesn’t work (probably since I am using DNS over TLS on my primary OPNsense so it can’t recursively resolve to the root DNS servers).
Firewall on a VM is not a good idea. The "bad packets" must be forwarded through the physical server to the VM. This means that the physical server for the VM is always unprotected. (As an example a bad IP packet triggers a buffer overflow on the kernel)
Greetings Marco
I usually run bare metal but I know a lot of people like to virtualize for various reasons.
Do you have any documented examples of what you are referring about compromising the hypervisor on a virtualized firewall? I’d be interested in reading up on it.
@@homenetworkguy The IP packet arrives at an interface on the server and is analysed by the server (OSI Layer 2 & 3 analysis) and forwarded to the VM. These steps take place on the server before the packet arrives at the VM.
Only the IP tables of the server forward the packet to the VM. This means that the IPTables including the kernel are before the firewall.
Draw the path for each OSI layer once on a piece of paper and write who is responsible at each point.
I understand what you are saying. I am just curious how many documented cases of compromise due to virtualizing the firewall. So many people do it that I’m surprised more people say “don’t do it!”
@@homenetworkguy Security is not a question of the frequency of events! The host server is not protected and is therefore directly connected to the "bad" Internet.
Why use a firewall then?
@@marcodoehler4089 Because the OPNsense VM uses interfaces that are connected to bridges on the physical Proxmox VE interfaces, Proxmox VE doesn't analyze anything.
It will only receive Ethernet frames (layer 2 only), the bridge will look up the destination MAC address (of the OPNsense virtual interface) and simply forward it.
Iptables (or soon nftables) on Proxmox VE will not be used for this at all, unless you want to block traffic to and from the OPNsense VM from the host.
If you do not set an IP address on any of the bridge interfaces to which the OPNsense VM virtual interfaces are attached, there is no way to communicate with the host.
Yoooo let me just swoop one of those $1300 mini computers 😂
May as well go buy an sonicwall TZ570w with a year of professional support for the same price.
Can you install a hypervisor on the Sonicwall? New prices seem like $3500? I’m assuming you’re referring to used hardware prices.
You could also do this guide with a $200-300 mini PC which has 4 network interfaces. It depends on what you need.
The VP6650 is faster (single threaded performance) than my old Ryzen 7 1700 Proxmox server at 1/4th the power consumption. I could easily replace my huge 4U server with the Protectli if I wanted but I’ll probably just cluster a few of my systems at some point.
@@homenetworkguy each to his own my guy. Great video and I'm sure it'll be very informative and helpful to a lot of people.
Thanks! It seems like the video is being well received by those interested in the topic.
Also, I was genuinely curious in my previous comment if you can run a hypervisor like Proxmox on it and get the device plus a year support for $1300?
I wasn’t implying the Protectli box is superior to the Sonicwall but rather it’s an apples to oranges comparison (one is a general purpose computer while the other is a firewall appliance). For a home network, having a general purpose low power mini PC is great for virtualization servers, etc.
I fell off when you started messing with the hardrive. I dont understand, i didnt have to do that when i followed a guide to install homeassistant. I only have 1 drive, a 2tb nvme.
I’m demonstrating how to set up OPNsense to replace an existing router on the network so it’s a more complicated setup than just setting up Home Assistant. There are other ways to set up OPNsense that are a little less complex if you’re using a different network interface configuration.
Yeah, I tried following your guide but I guess its over my head.
I have a 4 port N100 mini pc I want to use for proxmox, OPNsense and HA. But I hit roadblocks left and right so I can't get it to work.
If only my isp modem wasn't a router also, then maybe. When I put it in bridge modus OPNsense lost internet connection. And with the router active HA and wifi doesn't work together. And I can't change anything on the router, it sucks.
I'll go find some other guide or something.
date and time smeared out? Really? Is that offensive to some people or something? Or does it violate a copyright?
Yes! No, just kidding. I think it was distracting because sometimes with my edits I have to jump around out of chronological order. Also it sometimes takes a few days for me to get time to get all the recordings complete. But I realize that blurring it out makes it too distracting for some users.
This product has the look for typical linux gurus, the UI lacks by a mile comparing to vmware. UI ain't important for you linux ninjas. 😀
Yeah I wasn’t impressed with the UI when I first started using it but over time it has grown on me. I enjoy using the product. I’ve only had a few mishaps over the years and most of it was probably user error on my part. Haha
I really tried to follow your vague and convoluted presentation. You spend too much time hedging vs definitive information, like 7:1. Hypothetical cases ought to be footnotes not part of the main presentation.
Thanks for the feedback!
the ugly logo of opnsense alone shouts this is for highschool lab testing. I mean they cant pay a logo designer to create a nice logo??? branding is a great part of everything
They have hardware you can purchase and they offer business license support as well if you need professional support. It’s goes far beyond high school lab testing. I’d much rather have a solid product with a simple logo than a terrible product with the most beautiful branding. 😄
@@homenetworkguy what product does not offer professional support??? and good luck to them with that ugly logo, hope they get enough to pay for it. NO company with a brain will use this in DEV environment talk less production. Sure it can work but logos cost like $50 and up