The imagery of a Linux bridge being "a network switch" and plugging the network interface into it virtually was really helpful! Thanks for that description.
Dude, so much better and clear sense than other Proxmox videos\guides, no forking aroung like any other guide with no clear narration and usefull knowledge.
Thanks for the videos. I know some other guys are "more popular" to watch for proxmox but there's nobody that does the depth you do and we really appreciate it.
I appreciate your level of thoroughness on the topic at hand. I also appreciate how you will make mention to other related things to bring about awareness without veering off down a rabbit hole or just omitting them altogether. It's a fine balance and I think you've found it!
Excellent video. As someone that started my own homelab and IT journey with 486s in the late 90s and pushed myself ever since, I appreciate you taking the initiative to share this with the community! Gotta keep this stuff in the hands of everyone to learn and build upon it, the "cloud" mentality these days will only destroy what so many have built. Your Ceph on Proxmox video was far more in-depth than one I watched from a large professional outfit (not mentioning any names because they do have a lot of good videos).
Glad you enjoyed it! I definitely like keeping things locally hosted, even if it's just for 'fun'. Hope you enjoy some of the upcoming projects I have!
I’ve watched, what feels like at least, every video on RUclips trying to understand how these concepts work in Proxmox, and this was by far the best and explained every question and issue I had in a single video. Thank you so so so much!
Superb. I'm going to have to watch this on slow about 5 times just to get my head around this whole area of Proxmox I knew so little about 🤯 As the song says : "The more I find out, the less I know" 😁 Thankyou.
4:06 FINALLY someone on fricking RUclips explains that! I was on several "network videos" about Proxmox before and they didn't explained me that concept of vSwitches like you did THANKS MAN ❤
You did a great way of explaining the networking in the most concise way in proxmox, navgiating through the what ifs as well. Ive been kicking myself for days trying to configure vlanning woth ceph and the mgmt ip and so forth. Homelab security is a must.
20:45 saved me!!! My main proxmox is on VLAN 20 192.168.20.17 and one of my VM is on VLAN 60, but tagging the VM as 60 didn't work. And after watching your video I removed the vlan tag setting and wow it works!!! Thank you so much!!
Nice tutorial. The only thing I would say is I would definitely go with active active lacp rather than active passive if you can. I see a lot of people online asking about why they only have double the bandwidth One Direction but not the other
Active-Passive in the context of 802.3ad LACP is not the same as having an active-passive link. It just means which of the two hosts is actively sending unsolicited LACPDUs. When either end receives LACPDUs they will start exchanging LACPDUs and configure the link, regardless of if they are active or passive.
Greatly appreciate this video. I've referred back to it several times now when making networking changes to Proxmox. Your examples are very practical, and I'm often hesitant to make networking changes in Proxmox that I'm not completely certain about because I don't want to lose access to the machine. I'm especially thankful that you mentioned the particular use case that Linux VLAN is used for because I needed exactly that feature for my setup. Thanks!
Your 19:22 just saved my ass and I love the fact that you start by saying "linux vlan i very rarely used", turned out that was my missing factor in my infrastructure environment... With this I got full redundancy from my 2 firewalls to my 2 stacked layer 3 switches to my stacked layer 2 switches, which are connected to my 4 host proxmox cluster... I wasn't able to reach the default gateway without the linux vlan tag on the virtuel switch... Thank you so much!
Amazing video. Very clear explanations. I started my homelab projects with proxmox, pfsense and etc two years ago but never came across your channel before. However, hats off the way you have made everything clear with examples. I will def. be recommending you to the communities I know.
Great video. Finally sat down to re-do my 10G networking and figured it was time to setup active-backup and vlan awareness. When I did the initial setup, my VMs were fine on another 10G bridge I created but my NFS and iSCSI shares were capped at the 1G speeds - Not anymore! Covered exactly what I needed.
Best video I've seen on Proxmox network configurations so far. You cover the details that are lacking in many of other videos that led me to this one. 👍
A really excellent detail oriented tutorial on networking in the larger sense with Proxmox as the implementation - as well as where to actually *find* all the configuration bits. I've been doing Linux a long time (started with Unix) and this video refreshed and educated me about some networking details that had grown hazy over the years (I let the younger guys deal with it at work 😃). Much appreciated, and subscribed. 👍
Yo dawg. Nice video 😉 Proxmox GUI has come a long way since v4. Was great to see you showing off the possibilities and no config editing. I have the task of creating a bond with 10GbE and 1GbE backup, so your video was perfect to help me dry run and visualise how to achieve this without config editing 👍 no doubt this will save me a bunch of time. You've done a great job of making more advanced network topics accessible to a lot of folks. Bravo.
This was quite helpful in configuring proxmox for a pfSense VM that has 3 vlans on a trunk. I missed it in your video, but there was a hint to what I needed to complete the configuration. My host PC has 4x 2.5gb ports, and I wanted to have pfSense serve both the trunk vlans and the local ports with their respective DHCP pools. The bridge was the answer! I was able to bridge the vlan to the local port, with the bridge having the IP address and DHCP server, and the vlan and local port having no IPs.
thanks a lot for that. now i definitely will dive into proxmox again - turned away from it about 2 years ago because of not looking more into the bridge setup also +10 for the mikrotik switch. love their stuff
That was super cool, now you've got me shopping for managed switches so I can get goofy my home network. I've got 4-port cards in all of my infrastructure boxes already...
Thanks for the video! but for your next videos could you please use diagram software to illustrate complex concepts, it definitely helps the community as all other youtubers use it and it's a must in the networking world :)
This is absolutely outstanding. I read a book with similar content, and it was truly outstanding. "The Art of Meaningful Relationships in the 21st Century" by Leo Flint
Wow, [mention specific thing you liked about the video]! I especially found [mention specific part you enjoyed] interesting. [Ask a question related to the video]. Keep up the great work! # [relevant hashtag]
awesome, thanks for the indepth explanations. Very good to understand and follow! Even though i was looking for a tutorial on the GUI VLAN and subnet configuration options that apparently came with proxmox 8 (?). edit: nvm, you have another video on that, awesome!
Very good video. Thanks! If you using bonding bonds check that you are not using VLANs on bond0. I have bond0 (LAGG 10G) and bond1 (backup to 1G). And bond1 not working until I remove all VLANs on bond0.
Hi when you talk about clustering tips, what if my server only has one nic, were you referring to setting up a virtual bridge so three node can talk to each other in that bridge? For example, I have three nodes, they are all in 30 subnet. My initial thought is to just let managed switch pass all vlan traffic to my three nodes nic, then make then VLAN aware. And let the three nodes communicate in 30 subnet, for the VM I can put vlan tag and put them on 10 subnet or 20 subnet. Is this the dedicated interface you were referring to? THank you!
I'm actually not sure if I'll do OVS 'manually' in the Network menu, or go straight to the software-defined networking system. It's a *really nice* gui for cluster-wide networking for VMs. But one of those two is coming up.
I enjoyed much this video and it was so clear that now I understand well how to take full advantage of these features!!. I also use Mikrotik switches and in my case I had to disable VLAN aware on the vmbr0 as it didn't let the pass traffic or talk with Mikrotik switch. I got IP assignment from DHCP Server but traffic didn't passed through. Disabling the VLAN aware solved my problem!
I enjoyed your video. Regarding proxmox networking in general, what is the best approach to reduce latency? For example, if you're working with video or audio where timing is important.
@@apalrdsadventures Thanks. I was thinking that might be the case, since it would use the asic on the nic. I just started using a Connectx-4 card and it can break out multiple devices for use in SR-IOV. I just need to figure out how to best utilize that functionality across multiple VMs/Containers.
Thanks for this great Proxmox HowTo. There is one question I still have: Is it possible to receive a Trunk with Proxmox and split the different VLAN of the Trunk to separate Bridges which act like Access Ports? Meaning, that I can simply add a VM to different Bridges to have it connected to the different VLANs?
You can use Linux VLANs off the interface, and set those as the bridge ports on each bridge. Then the bridges are not vlan-aware and only carry traffic for the VLAN of their bridge port.
Absolutely stunning video. Thank you! QQ. If I want to have a VM that is a router using vLANs, is it more efficient to have multiple virtual NICs on the VM with different vLANs tagged in the Proxmox config, or pass it through to a single virtual NIC and then do the tagging on the router? (I hope that makes sense!)
Excellent video! Learned how I can have VM's on the same host communicate with each other. Is it possible to have them communicate over a cluster or is that more complicated?
Well, maybe a general overview of PCIe devices where IOMMU would be useful. Like maybe (I'm guessing here) IOMMU can be necessary with GPUs used for transcoding or compute; might be useful with some HBA scenarios; and wondering if it could be useful with NICs at all, like when using pfSense OS in a VM.
In general it's needed for PCIe passthrough, be that a GPU, NIC, or HBA. There are other types of passthrough though (bridged NIC, block device, USB) which don't require IOMMU. I'm working on a video on this topic, not pfSense but passthrough methods in general
this is a great tutorial. I have often struggled with this fumbling till it works. The only thing that would have been more helpful is if you went in a little more on the trunk for the vm... I didn't quite follow that.
I am new at proxmox and I will most likely need to look at this video multiple times. So first of all: Thank you. What I have difficulties to grasp at this point is: why would you set an ip address to a bridge? If I should see the bridge as a swithc: a switch does not have an IP address. Is the VM not supposed to set it's own IP address internally, or get one through DHCP? Or is this the fixed IP address, the address for the Proxmox server itself, on this bridge?
I'm a little confused here.. when making a vlan on proxmox, do you need a physical switch for the Vlan nic to work, or is this a virtual switch? Thanks in advance.. and much thanks for the tutorial.
Can you point me to what you used to choose the LACP Bond0 hashing algorithm? I've seen people choose 2+3 and others (like you) choose 3+4 in their guides. I can speculate about it, but in this journey I've embarked on, this is a learning opportunity as much as it is a chance to play with some cool self-hosted homelab stuffs.
Realistically hash based bonds are about the probability of two packets being assigned to different interfaces. So, do you expect to see the most variability in the L2 (MAC), L3 (IP), or L4 (TCP/UDP) headers? In practice it doesn't really matter and the choice is often driven by what hardware offload supports, especially on switches.
Hi ! Good informative video. Wanted to know how much reliable is the usb to ethernet. I came to know from the PFSENSE forums, that these power down automatically and cause the link to be down. Let me know your findings on this? It would be helpful
BSD has worse drivers in general than Linux, so you might have a better time with a Linux-based solution than BSD-based if you aren't using very common PCIe NICs.
@apalrdsadventures How do you diagnose issues when a seemingly simple change breaks this? I have trunked VLANs on 1G (pfSense) and a (GS748Tv5) smart switch. I also have a working bonded LACP link between the switch and a NAS, so I am pretty confident the switch is ok. On proxmox, as soon as I convert the trunk from a NIC port to a bonded NIC (even one), nothing goes through. :( I DID notice that you had to tear it down and build it again to get it to work. I've done that but no joy. Ideas? THANKS!
Thank you! But I don't understand how to make a access from Inet to my virtual machine, and make my VM isolated from all other my network. Yes, maybe VLANs.... But, o my Gos, am I have to block traffic by all to all (subnets) for every new one subnet (group of virtual machines)?
If I remember correctly on a regular switch if the two devices talking to each other is physically connected to it the traffic is never pushed further to for instance a router and is just forwarded directly by the switch from port A to port B. I assume the virtual bridges behave the same way and traffic that never needs to leave the bridge interface to reach it's destination is never sent out to the connected switch although the packets themselves will off course be visible to anyone on the network. So unless it's important to hide traffic between VM's there is no need to actually set up dedicated bridges right?
Traffic from a new source will initially flood the network until the switch 'learns' the MAC addreses on each port. This should happen really quickly before the node even has an IP address due to the DHCP / RA process. But it's still part of the same layer 2 broadcast domain, shares the same DHCP server and RAs, layer 3 subnets, ... So creating multiple vmbr's isn't really to 'hide' traffic between VMs, but to create a unique layer 3 subnet for a special purpose. You might do this to simulate a physical topology where two VMs are directly connected instead of via the main netework, or if you have a virtual router and want the downstream network(s) to be isolated from each other and the upstream network(s).
I never got an IP on my bond, that I made for my internal network, and I set a subnet just like you did. Sometimes quirky stuff like this about proxmox pisses me off.
Hey Bud, great video once again! Question 1: How come you could not get 2Gb/s on the LACP with the 2 x 1GBb ports? Question 2: Can you do a demonstration on OpenVSwitch (i'm also gonna be experimenting with this soon)
With LACP, it determines which link to send a packet on based on the hash of the packet header. So, packets with the same destination will always go through the same link. It's deterministic, fast, and can be massively parallel, but it means a single socket will never reach 2G on 2x 1G links.
@@zparihar Aggregated, yes. But it will still only allow 1Gb between two specific endpoints This means you can have 4 machines concurrently accessing the server at 1Gb each, but each will still only get 1Gb at any given time. the server side will be able to serve up 4 1Gb connections, just not to the same client. The hash component is what allows this (L2 works from mac address, L3 would be the IP, and L3/4 is IP and Application protocol). Using the L4 hash would allow you to use two different services (FTP download and something like streaming a video simultaneously), each tapping into a different connection. I personally never used that high level of a hash. If you're looking for a point to point high speed connection and don't want to spring for multiGig or 10Gb+ switching hardware, you could do a direct 10Gb link, 10Gb network adapter prices (Used) have fallen off a cliff in recent years. could do both NICS and an SFP+ DAC cable for under $100 easily (that would be for dual port cards and two cables under 10m - always thinking redundancy). I'm actually planning on using bridging and cascading cabling in this way for clients (smaller businesses) starting next year to save them the cost (and power requirement - my Cisco 5548s are about 450 watts a pop!) of 10Gb switches.
Think of lacp as a reception desk with 2 receptionists. one receptionist talks at 1gbps (normal conversation speed between 2 people). a guest converses with one receptionist at 1 gbps. if another guest shows up, they will talk to the 2nd receptionist, therefore the reception desk is now talking at 2gbps in aggregate to two different guests. but one guest cannot converse at 2x the normal conversation speed.
Hello, could you create a tutorial on cluster removal of a node and adding another node to that cluster? also, it would be nice to hace a tutorial on how to improve rdp capabilities of the3 vm so full HD video be played on windows via RDP on proxmox?
I have a 5 node cluster setup with HA. Each has one, 1GB NIC that is shared for cluster/VM/PVE GUI. When I attempt to live migrate a VM or Restore from my SAN. I run into the issue where the cluster tires to fence that node. My guess has come down to what you said here. It seems I may need to have a dedicated NIC for corosync. I am assuming the cluster isn't getting the heartbeat do to high latency and is fencing the node. Have you seen this as a problem before?
The VLAN are working wehn I don't have VLAN aware checked. As soon as I check it, it quits working. Also if I migrate the VM to the same node in the cluster as teh pfsense, it quits working. I actually have to shut it down, reboot the node that the pfsense vm is running in for it to start working properly again. I've spent many hours today trying to figure out what's wrong lol. I have 3 vlans and native and pfsense is routing it all properly. But as soon as I migrate a vlan tagged VM to the same node as pfsense or set VLAN aware = yes then the doesn't route the traffic.
I already knew most of the things in the video, but don't ask me how many times I broke my network and locked myself out of proxmox ^^ I never used active-backup and trunks though, very interesting indeed
You mean, that I can have multiple ISP merged together just like a Loadbalancer? And I could use this proxmox as my point of access to the home network with a LB of ISP? thats sounds great. I thought I had to buy complex and expensive routers and configure them somehow. Something like virtual router... Even tho, I'm not sure if VRRP works as LB(that merge multiple ISP speed connextion).
Great video, really appreciate you sharing your knowledge on this subject. I was wondering if I could aggregate two ports that are connected straight through to reach other, I have two proxmox machines with a dual 10gig on each with no switch in between (aside from the gig interface I use for proxmox to get out to the rest of the network) I got the servers to communicate to each other over the dual aggregated pair of nics, but I'm not sure I'm using double the bandwidth. You've definitely honored your neck beard.
LACP should have no issues in an active/active scenario (Proxmox defaults to LACP Active, so it will initiate the lacp bond, but it's fine if both sides do this). With LACP you still have the limit of one nic for a single connection. LACP is usually the best bet since it's standardized and you shouldn't have to deal with any vendor oddities across all of your equipment. Without a switch, you have the option of using balance-rr where it will accept packets on any port and send them out alternating ports, meaning you get truly double bandwidth for a single connection. You can really only use this mode if the other end also supports manually configured link aggregation groups, which is extremely switch dependent and not usually recommended since LACP is guaranteed to work without odd behavior, but since you're going direct Linux-Linux it will work. If you're using a cluster with more than just the two nodes, they will need to do cluster communication (including VM migration) on a network they all have access to. If it's 2 + a QDevice, you can move migration traffic to the bond but still need to keep corosync on the public network - see pve.proxmox.com/pve-docs/pve-admin-guide.html#_guest_migration for more info on this. You would then need to assign a private /30 IPv4 on the link with static IPs on the two ends so the nodes can communicate across the bond (presumably on the bridge you build on top of the bond). Or, you can just let VMs communicate across the bond and Proxmox doesn't need an IP on that link at all.
Hi There, I have 2 dell r720 running pve 8.2.4, connected by 2 1gig links to a cisco 3750 switch. I can get one pve to connect to the switch using lacp for the bond0. the other pve won't connect. same configuration for both bond0s. If you want more info let me know thx Hayward
Great video, thank you very much! I encoutered a proxmox server where vmbr0 bridges the slaveport "eno1.100". Is this the same config as "vmbr0.100" bridging "eno1"?
Not exactly. It changes how untagged traffic on the bridge behaves (and also breaks vlan-awareness on the bridge) If the slave port is eno1.100, then the untagged traffic on vmbr0 becomes vlan 100 on the wire. Tagging a vlan on the bridge would then nest the tags on the wire (although I don't believe this config would be vlan-enabled at all on the bridge). The other way around creates a tagged interface on vlan 100 off of the vmbr *for the proxmox system*, but VMs using vmbr0 directly aren't vlan tagged, but they could be tagged if you set the vlan id on the net interface.
@@apalrdsadventures I see, thank you for your reply! If I understand it correctly, eno1.100 is like a "linux thing" while vmbr0.100 is a "proxmox-thing"?
It changes where in the layering the VLAN tags are added/removed, which changes if vmbr0 can use vlan tags at all. Taking eno1 and bridging to vmbr0 means that the bridge itself is now vlan-aware (can handle any vlan), and vmbr0.100 creates an interface for the system to use vlan 100, but VMs can still use any vlan as the bridge is not confined to only vlan 100. Taking eno1 and making a vlan-tagged interface eno1.100 and bridging *that* to vmbr0 means vmbr0 is now a member of vlan 100 via its parent interface, even if you select no vlan / default vlan on the bridge.
Im trying to follow along but cant seem to get my 2 windows machines to ping each other. I tried using the tag and it didnt work. I have them connected with a bridge and they're still not talking. Not sure what im doing wrong.
The imagery of a Linux bridge being "a network switch" and plugging the network interface into it virtually was really helpful! Thanks for that description.
Glad it helped!
Dude, so much better and clear sense than other Proxmox videos\guides, no forking aroung like any other guide with no clear narration and usefull knowledge.
Much appreciated!
I was able to set up a LAGG in my Proxmox lab using your tutorial first-try (not typical for me). This says a lot about your teaching style. Thanks!
Glad it worked for you!
my guy had me at "yo, dwog"
Thanks for the videos. I know some other guys are "more popular" to watch for proxmox but there's nobody that does the depth you do and we really appreciate it.
Yes! And I also appreciate the side notes that you give, just to make sure everybody understands what the terminology is.
I appreciate your level of thoroughness on the topic at hand. I also appreciate how you will make mention to other related things to bring about awareness without veering off down a rabbit hole or just omitting them altogether. It's a fine balance and I think you've found it!
Excellent video. As someone that started my own homelab and IT journey with 486s in the late 90s and pushed myself ever since, I appreciate you taking the initiative to share this with the community! Gotta keep this stuff in the hands of everyone to learn and build upon it, the "cloud" mentality these days will only destroy what so many have built. Your Ceph on Proxmox video was far more in-depth than one I watched from a large professional outfit (not mentioning any names because they do have a lot of good videos).
Glad you enjoyed it! I definitely like keeping things locally hosted, even if it's just for 'fun'. Hope you enjoy some of the upcoming projects I have!
Agreed, his videos are great! He's doing the community a great service!
I’ve watched, what feels like at least, every video on RUclips trying to understand how these concepts work in Proxmox, and this was by far the best and explained every question and issue I had in a single video. Thank you so so so much!
Brilliant! Now in my 60's, "homelab-ing" is my new passion, and you made a potentially complex subject look (relatively) easy - thanks! 👍
Glad you enjoyed it!
Superb. I'm going to have to watch this on slow about 5 times just to get my head around this whole area of Proxmox I knew so little about 🤯 As the song says : "The more I find out, the less I know" 😁 Thankyou.
Glad you like it! This is just the start, there's also the whole Proxmox SDN solution too :)
4:06 FINALLY someone on fricking RUclips explains that! I was on several "network videos" about Proxmox before and they didn't explained me that concept of vSwitches like you did THANKS MAN ❤
You did a great way of explaining the networking in the most concise way in proxmox, navgiating through the what ifs as well. Ive been kicking myself for days trying to configure vlanning woth ceph and the mgmt ip and so forth. Homelab security is a must.
Finally - after million other videos - this one made it so clear!
If possible, please provide more such videos.
20:45 saved me!!! My main proxmox is on VLAN 20 192.168.20.17 and one of my VM is on VLAN 60, but tagging the VM as 60 didn't work. And after watching your video I removed the vlan tag setting and wow it works!!! Thank you so much!!
Nice tutorial. The only thing I would say is I would definitely go with active active lacp rather than active passive if you can. I see a lot of people online asking about why they only have double the bandwidth One Direction but not the other
Active-Passive in the context of 802.3ad LACP is not the same as having an active-passive link. It just means which of the two hosts is actively sending unsolicited LACPDUs. When either end receives LACPDUs they will start exchanging LACPDUs and configure the link, regardless of if they are active or passive.
@@apalrdsadventures hmm. So why is there so much debate about which setup. I do see what you are saying.
Greatly appreciate this video. I've referred back to it several times now when making networking changes to Proxmox. Your examples are very practical, and I'm often hesitant to make networking changes in Proxmox that I'm not completely certain about because I don't want to lose access to the machine. I'm especially thankful that you mentioned the particular use case that Linux VLAN is used for because I needed exactly that feature for my setup.
Thanks!
The most concise proxmox networking, and linux in general. Thanks
Your 19:22 just saved my ass and I love the fact that you start by saying "linux vlan i very rarely used", turned out that was my missing factor in my infrastructure environment...
With this I got full redundancy from my 2 firewalls to my 2 stacked layer 3 switches to my stacked layer 2 switches, which are connected to my 4 host proxmox cluster...
I wasn't able to reach the default gateway without the linux vlan tag on the virtuel switch...
Thank you so much!
Amazing video. Very clear explanations. I started my homelab projects with proxmox, pfsense and etc two years ago but never came across your channel before. However, hats off the way you have made everything clear with examples. I will def. be recommending you to the communities I know.
Great video. Finally sat down to re-do my 10G networking and figured it was time to setup active-backup and vlan awareness. When I did the initial setup, my VMs were fine on another 10G bridge I created but my NFS and iSCSI shares were capped at the 1G speeds - Not anymore! Covered exactly what I needed.
Just stepping into proxmox with a QNAP TS-470pro converted to pve. This is perfect for helping get the networks setup! Much appreciated.
Best video I've seen on Proxmox network configurations so far. You cover the details that are lacking in many of other videos that led me to this one. 👍
Glad you like it!
You explained a complex subject so simply that even I could understand. Thank you!
Glad it was helpful!
Thank for this "advanced" information on Proxmox networking. I am new to Proxmox and I appreciate your video explaining this.
Excellent tutorial. Informative, calm style, easy to follow. Simply perfect.
A really excellent detail oriented tutorial on networking in the larger sense with Proxmox as the implementation - as well as where to actually *find* all the configuration bits. I've been doing Linux a long time (started with Unix) and this video refreshed and educated me about some networking details that had grown hazy over the years (I let the younger guys deal with it at work 😃).
Much appreciated, and subscribed. 👍
Glad it was helpful!
Yo dawg. Nice video 😉
Proxmox GUI has come a long way since v4. Was great to see you showing off the possibilities and no config editing.
I have the task of creating a bond with 10GbE and 1GbE backup, so your video was perfect to help me dry run and visualise how to achieve this without config editing 👍 no doubt this will save me a bunch of time.
You've done a great job of making more advanced network topics accessible to a lot of folks. Bravo.
Glad you liked it! Working on tutorials for some of the more complex parts of the networking GUI (SDN and Firewall)
Hi, thanks for your video, that´s very interesting and helpful. One question: why is your 2,5 gbit interface marked as half duplex (at 09:32)?
Best proxmox' network concepts explanation so far. Good job!
The name's Bond... Bonded Bond
😂😂😂
This was quite helpful in configuring proxmox for a pfSense VM that has 3 vlans on a trunk. I missed it in your video, but there was a hint to what I needed to complete the configuration. My host PC has 4x 2.5gb ports, and I wanted to have pfSense serve both the trunk vlans and the local ports with their respective DHCP pools. The bridge was the answer! I was able to bridge the vlan to the local port, with the bridge having the IP address and DHCP server, and the vlan and local port having no IPs.
Glad it helped!
THANK YOU! This is such a specific thing that is really hard to find instruction on anywhere else. At least that is this detailed.
I'm still learning about Proxmox, though I've used it for several years now. I really appreciate your videos. You've helped me a learn a great deal.
Glad to help!
Magic 🪄 what a wonderful video. This needs to go viral.
Thank you for such a detailed explanation of Proxmox networking.
thanks a lot for that. now i definitely will dive into proxmox again - turned away from it about 2 years ago because of not looking more into the bridge setup
also +10 for the mikrotik switch. love their stuff
Thanks for the demo and info, now my proxmox is speedier! Have a great day
Only just come across your channel and I'm hooked, keep up the great work and a tickle under the chin to Sherlock!
The perfect deep level of detail I was looking for. your vids are amazing.
That was super cool, now you've got me shopping for managed switches so I can get goofy my home network. I've got 4-port cards in all of my infrastructure boxes already...
Managed switches are a ton of fun!
Thanks for the video! but for your next videos could you please use diagram software to illustrate complex concepts, it definitely helps the community as all other youtubers use it and it's a must in the networking world :)
What a great video. I am just starting with Proxmox and this helps a ton!
This is an absolutely awesome video!! Thank you!! I am now making use of all these nics on my Gen9s!
This is absolutely outstanding. I read a book with similar content, and it was truly outstanding. "The Art of Meaningful Relationships in the 21st Century" by Leo Flint
Wow, [mention specific thing you liked about the video]! I especially found [mention specific part you enjoyed] interesting. [Ask a question related to the video]. Keep up the great work! # [relevant hashtag]
awesome, thanks for the indepth explanations. Very good to understand and follow! Even though i was looking for a tutorial on the GUI VLAN and subnet configuration options that apparently came with proxmox 8 (?).
edit: nvm, you have another video on that, awesome!
Very good video. Thanks!
If you using bonding bonds check that you are not using VLANs on bond0.
I have bond0 (LAGG 10G) and bond1 (backup to 1G). And bond1 not working until I remove all VLANs on bond0.
Thankyou so much for the management vlan trick @20 mins. Cheers
Great video. I do wish you could have gone down the cluster rabbit hole a bit. I'd like to see how that gets setup.
Hi when you talk about clustering tips, what if my server only has one nic, were you referring to setting up a virtual bridge so three node can talk to each other in that bridge?
For example, I have three nodes, they are all in 30 subnet. My initial thought is to just let managed switch pass all vlan traffic to my three nodes nic, then make then VLAN aware. And let the three nodes communicate in 30 subnet, for the VM I can put vlan tag and put them on 10 subnet or 20 subnet. Is this the dedicated interface you were referring to? THank you!
Great video, I'm definitely ready for the OVS follow-up!
I'm actually not sure if I'll do OVS 'manually' in the Network menu, or go straight to the software-defined networking system. It's a *really nice* gui for cluster-wide networking for VMs. But one of those two is coming up.
I enjoyed much this video and it was so clear that now I understand well how to take full advantage of these features!!. I also use Mikrotik switches and in my case I had to disable VLAN aware on the vmbr0 as it didn't let the pass traffic or talk with Mikrotik switch. I got IP assignment from DHCP Server but traffic didn't passed through. Disabling the VLAN aware solved my problem!
I enjoyed your video. Regarding proxmox networking in general, what is the best approach to reduce latency? For example, if you're working with video or audio where timing is important.
That's a good question, and I suspect SR-IOV will get you the least jitter as the software bridge will be more dependent on CPU load.
@@apalrdsadventures Thanks. I was thinking that might be the case, since it would use the asic on the nic. I just started using a Connectx-4 card and it can break out multiple devices for use in SR-IOV. I just need to figure out how to best utilize that functionality across multiple VMs/Containers.
Thanks for this great Proxmox HowTo. There is one question I still have: Is it possible to receive a Trunk with Proxmox and split the different VLAN of the Trunk to separate Bridges which act like Access Ports? Meaning, that I can simply add a VM to different Bridges to have it connected to the different VLANs?
You can use Linux VLANs off the interface, and set those as the bridge ports on each bridge. Then the bridges are not vlan-aware and only carry traffic for the VLAN of their bridge port.
Shoot, did I miss it? What about bonding tlb and alb? Will watch again. Very informative!
Absolutely stunning video. Thank you!
QQ. If I want to have a VM that is a router using vLANs, is it more efficient to have multiple virtual NICs on the VM with different vLANs tagged in the Proxmox config, or pass it through to a single virtual NIC and then do the tagging on the router? (I hope that makes sense!)
Top notch. Very detailed and informative. Thank you!
Thank you for this Great video. I managed to setup bond interface on my server just by watching this video and referring to official documentation 😎
yup, this cleared up so much in so little time. Thanks for helping on my journey brah.
Happy to help!
Very comprehensive video, thank you so much!
Glad it was helpful!
Excellent video! Learned how I can have VM's on the same host communicate with each other. Is it possible to have them communicate over a cluster or is that more complicated?
I have been thinking about this as well. Did you find an answer?
Great explanation of the various options. Thanks!
Glad you enjoyed it!
I love this video. Explained a lot. I wonder if you might cover where IOMMU configuration and where it is useful and where it's not.
IOMMU for networking or in general? It's a bit of a different topic
Well, maybe a general overview of PCIe devices where IOMMU would be useful.
Like maybe (I'm guessing here) IOMMU can be necessary with GPUs used for transcoding or compute; might be useful with some HBA scenarios; and wondering if it could be useful with NICs at all, like when using pfSense OS in a VM.
In general it's needed for PCIe passthrough, be that a GPU, NIC, or HBA. There are other types of passthrough though (bridged NIC, block device, USB) which don't require IOMMU.
I'm working on a video on this topic, not pfSense but passthrough methods in general
Excellent Analysis!
this is a great tutorial. I have often struggled with this fumbling till it works. The only thing that would have been more helpful is if you went in a little more on the trunk for the vm... I didn't quite follow that.
I am new at proxmox and I will most likely need to look at this video multiple times. So first of all: Thank you. What I have difficulties to grasp at this point is: why would you set an ip address to a bridge? If I should see the bridge as a swithc: a switch does not have an IP address. Is the VM not supposed to set it's own IP address internally, or get one through DHCP? Or is this the fixed IP address, the address for the Proxmox server itself, on this bridge?
Setting an IP address on the bridge is essentially plugging in the Proxmox server itself to the bridge, in one step.
Do not tuch the stuff in the video if you dont have the keys for the server room at 15:45 on Friday. Don't ask me why and how I know
I'm a little confused here.. when making a vlan on proxmox, do you need a physical switch for the Vlan nic to work, or is this a virtual switch? Thanks in advance.. and much thanks for the tutorial.
You do not need a physical switch for VLANs to work. You do need a switch that supports VLANs if you want your vlans to leave the Proxmox box.
Can you point me to what you used to choose the LACP Bond0 hashing algorithm? I've seen people choose 2+3 and others (like you) choose 3+4 in their guides.
I can speculate about it, but in this journey I've embarked on, this is a learning opportunity as much as it is a chance to play with some cool self-hosted homelab stuffs.
Realistically hash based bonds are about the probability of two packets being assigned to different interfaces. So, do you expect to see the most variability in the L2 (MAC), L3 (IP), or L4 (TCP/UDP) headers? In practice it doesn't really matter and the choice is often driven by what hardware offload supports, especially on switches.
Just what the Dr Ordered. Thanks!
Hi ! Good informative video. Wanted to know how much reliable is the usb to ethernet. I came to know from the PFSENSE forums, that these power down automatically and cause the link to be down. Let me know your findings on this? It would be helpful
BSD has worse drivers in general than Linux, so you might have a better time with a Linux-based solution than BSD-based if you aren't using very common PCIe NICs.
Thanks for this brilliant video!
Great video, thanks man!
@apalrdsadventures How do you diagnose issues when a seemingly simple change breaks this? I have trunked VLANs on 1G (pfSense) and a (GS748Tv5) smart switch. I also have a working bonded LACP link between the switch and a NAS, so I am pretty confident the switch is ok.
On proxmox, as soon as I convert the trunk from a NIC port to a bonded NIC (even one), nothing goes through. :( I DID notice that you had to tear it down and build it again to get it to work. I've done that but no joy.
Ideas? THANKS!
I bonded a bond to a bond and removed the vmbr so now I have to remake the whole thing as I wrecked it again. Whahoo!
Thank you!
But I don't understand how to make a access from Inet to my virtual machine, and make my VM isolated from all other my network. Yes, maybe VLANs.... But, o my Gos, am I have to block traffic by all to all (subnets) for every new one subnet (group of virtual machines)?
Thanks for your explanation About of All.
Glad it was helpful!
If I remember correctly on a regular switch if the two devices talking to each other is physically connected to it the traffic is never pushed further to for instance a router and is just forwarded directly by the switch from port A to port B. I assume the virtual bridges behave the same way and traffic that never needs to leave the bridge interface to reach it's destination is never sent out to the connected switch although the packets themselves will off course be visible to anyone on the network. So unless it's important to hide traffic between VM's there is no need to actually set up dedicated bridges right?
Traffic from a new source will initially flood the network until the switch 'learns' the MAC addreses on each port. This should happen really quickly before the node even has an IP address due to the DHCP / RA process. But it's still part of the same layer 2 broadcast domain, shares the same DHCP server and RAs, layer 3 subnets, ...
So creating multiple vmbr's isn't really to 'hide' traffic between VMs, but to create a unique layer 3 subnet for a special purpose. You might do this to simulate a physical topology where two VMs are directly connected instead of via the main netework, or if you have a virtual router and want the downstream network(s) to be isolated from each other and the upstream network(s).
Thanks for doing this!! Phew...
nice you went extra mile and added 2.5! much appreciated #james bond0
lol thanks! USB NICs aren't ideal, but at least it shows the difference from real 2+G to aggregated 2+G
I never got an IP on my bond, that I made for my internal network, and I set a subnet just like you did.
Sometimes quirky stuff like this about proxmox pisses me off.
Hey Bud, great video once again!
Question 1: How come you could not get 2Gb/s on the LACP with the 2 x 1GBb ports?
Question 2: Can you do a demonstration on OpenVSwitch (i'm also gonna be experimenting with this soon)
With LACP, it determines which link to send a packet on based on the hash of the packet header. So, packets with the same destination will always go through the same link. It's deterministic, fast, and can be massively parallel, but it means a single socket will never reach 2G on 2x 1G links.
@@apalrdsadventures Lets say if I did LACP on 4 x 1GBs and configure LACP on those ports on my switch, would I ever be able to get 3 - 4 Gb/s?
@@zparihar Aggregated, yes. But it will still only allow 1Gb between two specific endpoints This means you can have 4 machines concurrently accessing the server at 1Gb each, but each will still only get 1Gb at any given time. the server side will be able to serve up 4 1Gb connections, just not to the same client. The hash component is what allows this (L2 works from mac address, L3 would be the IP, and L3/4 is IP and Application protocol). Using the L4 hash would allow you to use two different services (FTP download and something like streaming a video simultaneously), each tapping into a different connection. I personally never used that high level of a hash. If you're looking for a point to point high speed connection and don't want to spring for multiGig or 10Gb+ switching hardware, you could do a direct 10Gb link, 10Gb network adapter prices (Used) have fallen off a cliff in recent years. could do both NICS and an SFP+ DAC cable for under $100 easily (that would be for dual port cards and two cables under 10m - always thinking redundancy). I'm actually planning on using bridging and cascading cabling in this way for clients (smaller businesses) starting next year to save them the cost (and power requirement - my Cisco 5548s are about 450 watts a pop!) of 10Gb switches.
Think of lacp as a reception desk with 2 receptionists. one receptionist talks at 1gbps (normal conversation speed between 2 people). a guest converses with one receptionist at 1 gbps. if another guest shows up, they will talk to the 2nd receptionist, therefore the reception desk is now talking at 2gbps in aggregate to two different guests. but one guest cannot converse at 2x the normal conversation speed.
This is excellent. Thank you soo so much.
Excellent video.. If you can please post a video with evpn vxlan in proxmox. Great video!!!
I just did SDN basics, so it will be next in the SDN list (unicast vxlan and evpn vxlan)
@@apalrdsadventures Fantastic!! Hope soon because SDN is very very great technology in Proxmox. Thanks!!
Nice work
Hello, could you create a tutorial on cluster removal of a node and adding another node to that cluster? also, it would be nice to hace a tutorial on how to improve rdp capabilities of the3 vm so full HD video be played on windows via RDP on proxmox?
Awesome video. Some diagrams would make it even better. Saved it on my networking list and I subscribed to the channel. Thanks for your work.
Glad you like it!
I have a 5 node cluster setup with HA. Each has one, 1GB NIC that is shared for cluster/VM/PVE GUI. When I attempt to live migrate a VM or Restore from my SAN. I run into the issue where the cluster tires to fence that node. My guess has come down to what you said here. It seems I may need to have a dedicated NIC for corosync. I am assuming the cluster isn't getting the heartbeat do to high latency and is fencing the node. Have you seen this as a problem before?
It could potentially be. I haven't seen it fence a node that was running properly, but I'm also not pushing the network super hard in testing
Thanks for the explanations!!!!
Question, have you every connecting a zero client like the EVGA PD05 PCoIP Zero Client
The VLAN are working wehn I don't have VLAN aware checked. As soon as I check it, it quits working. Also if I migrate the VM to the same node in the cluster as teh pfsense, it quits working. I actually have to shut it down, reboot the node that the pfsense vm is running in for it to start working properly again. I've spent many hours today trying to figure out what's wrong lol. I have 3 vlans and native and pfsense is routing it all properly. But as soon as I migrate a vlan tagged VM to the same node as pfsense or set VLAN aware = yes then the doesn't route the traffic.
EXCELLENT VIDEO!!!!!
I already knew most of the things in the video, but don't ask me how many times I broke my network and locked myself out of proxmox ^^
I never used active-backup and trunks though, very interesting indeed
Bonding is mostly useful in larger networks anyway, but it's still fun to play with if you don't have 10G (or if you have 10G an wish you had 25G).
You mean, that I can have multiple ISP merged together just like a Loadbalancer? And I could use this proxmox as my point of access to the home network with a LB of ISP? thats sounds great. I thought I had to buy complex and expensive routers and configure them somehow. Something like virtual router... Even tho, I'm not sure if VRRP works as LB(that merge multiple ISP speed connextion).
All of this is at layer 2, not layer 3, so no IP routing / load balancing, just MAC.
Fantastic. Thank you.
Hello, any idea why the 2.5gb interface shows as half-duplex?
Great video, really appreciate you sharing your knowledge on this subject.
I was wondering if I could aggregate two ports that are connected straight through to reach other, I have two proxmox machines with a dual 10gig on each with no switch in between (aside from the gig interface I use for proxmox to get out to the rest of the network) I got the servers to communicate to each other over the dual aggregated pair of nics, but I'm not sure I'm using double the bandwidth.
You've definitely honored your neck beard.
LACP should have no issues in an active/active scenario (Proxmox defaults to LACP Active, so it will initiate the lacp bond, but it's fine if both sides do this). With LACP you still have the limit of one nic for a single connection. LACP is usually the best bet since it's standardized and you shouldn't have to deal with any vendor oddities across all of your equipment.
Without a switch, you have the option of using balance-rr where it will accept packets on any port and send them out alternating ports, meaning you get truly double bandwidth for a single connection. You can really only use this mode if the other end also supports manually configured link aggregation groups, which is extremely switch dependent and not usually recommended since LACP is guaranteed to work without odd behavior, but since you're going direct Linux-Linux it will work.
If you're using a cluster with more than just the two nodes, they will need to do cluster communication (including VM migration) on a network they all have access to. If it's 2 + a QDevice, you can move migration traffic to the bond but still need to keep corosync on the public network - see pve.proxmox.com/pve-docs/pve-admin-guide.html#_guest_migration for more info on this. You would then need to assign a private /30 IPv4 on the link with static IPs on the two ends so the nodes can communicate across the bond (presumably on the bridge you build on top of the bond). Or, you can just let VMs communicate across the bond and Proxmox doesn't need an IP on that link at all.
Hi There,
I have 2 dell r720 running pve 8.2.4, connected by 2 1gig links to a cisco 3750 switch. I can get one pve to connect to the switch using lacp for the bond0. the other pve won't connect. same configuration for both bond0s. If you want more info let me know
thx
Hayward
Great video, thank you very much! I encoutered a proxmox server where vmbr0 bridges the slaveport "eno1.100". Is this the same config as "vmbr0.100" bridging "eno1"?
Not exactly. It changes how untagged traffic on the bridge behaves (and also breaks vlan-awareness on the bridge)
If the slave port is eno1.100, then the untagged traffic on vmbr0 becomes vlan 100 on the wire. Tagging a vlan on the bridge would then nest the tags on the wire (although I don't believe this config would be vlan-enabled at all on the bridge).
The other way around creates a tagged interface on vlan 100 off of the vmbr *for the proxmox system*, but VMs using vmbr0 directly aren't vlan tagged, but they could be tagged if you set the vlan id on the net interface.
@@apalrdsadventures I see, thank you for your reply! If I understand it correctly, eno1.100 is like a "linux thing" while vmbr0.100 is a "proxmox-thing"?
It changes where in the layering the VLAN tags are added/removed, which changes if vmbr0 can use vlan tags at all.
Taking eno1 and bridging to vmbr0 means that the bridge itself is now vlan-aware (can handle any vlan), and vmbr0.100 creates an interface for the system to use vlan 100, but VMs can still use any vlan as the bridge is not confined to only vlan 100.
Taking eno1 and making a vlan-tagged interface eno1.100 and bridging *that* to vmbr0 means vmbr0 is now a member of vlan 100 via its parent interface, even if you select no vlan / default vlan on the bridge.
@@apalrdsadventures Ah ok, now I understand.That makes a lot of sense, thank you!
Im trying to follow along but cant seem to get my 2 windows machines to ping each other. I tried using the tag and it didnt work. I have them connected with a bridge and they're still not talking. Not sure what im doing wrong.