Great video, I ran into a situation identical to this with an ISP. Their solution was to adjust the MTU. I didn't understand how that worked, but I do believe you just explained it. Is this something that ISPs that use EVCs don't have to deal with since they aren't using VXLANs?
Thanks. It isn't so much that the carriers are using VXLAN, in my example, the carrier transit network would be 6300-3 (the switch in the middle), the VXLAN VTEP would be customer side devices connecting to the carrier network. Moreover, one of the main use cases of VXLAN is that the VTEP is even further into the customer network, right on the server, so the carrier is merely carrying the VXLAN encapsulated in IP, their devices just see an IP packet. For the example you give, of EVC, the carrier is merely an L2 transit network then, in this instance, their devices would have to be able to carry frames that are larger than the standard 1514 / 1518 bytes 'on the wire' (or 1522 if we count the trailer). I can't speak for ISPs but I think they would just enable 'jumbo frames' L2 frames up to 9000 bytes on their network, it can't be cost effective or good for customer service if an ISP is having to negotiate their transit network MTU with every customer, just because the customer wants to run protocol X,Y,Z (in this case, VXLAN) on their links. We do get queries in where carriers say they have a fixed MTU of 1500 bytes and push the problem to the customer, hence why I included the 'Alternatives' section. But then VXLAN is standard networking now, the RFC is from 2014, so if the carrier is saying the link is fixed time to negotiate a new contract, or look to a different carrier.
I appreciate the reply. This particular case was a bit different. It's a point to point with an ISP upstart, they leased the firber from another startup. So many variables. I do not believe that they were using EVCs, so I assume either VXLAN or a VRF configuration.
@@Scotty_J. Ok, yes, that does sound like a setup that starts to complicate things. If the ISP themselves are using additional encap., like VXLAN, then they would have to be aware of the overhead they are adding themselves. Traffic MTU concerns are not new to ISPs though, there are such concerns when it comes to MPLS, another technology used by ISPs that adds overhead. Any ISP worth your business will have a clear idea and strategy around this, and be able to communicate it to customers.
First off, great series on VXLAN, very useful. QQ on this one. After you showed all routers were default 1500 MTU, your first ping with a request that was too large made it to 6300-3 and then got dropped as a giant. So is MTU only inbound on CX, since this packet successfully exited 6300-1 1/1/1 vtep interface which was still set at default 1500 mtu?
Thanks. In answer to your question, that's not what I observed but it was hard to prove where the traffic is being dropped. The issue I had with the VXLAN traffic that exceeded the MTU on 6300-1 was to show any indication that the packet was being dropped. Looking around the 16:30 mark, I pinged with 1423 bytes and that failed, the 6300-3 transit switch doesn't increment its Giants counter with that but then I couldn't find any counter that incremented on the 6300-1. I remembered back to the RFC, that VTEPs must not fragment, and they can silently discard. Rather than go deep researching this, I just accept that the 1423 ping is being dropped on 6300-1 (it doesn't hit 6300-3) then move on to pinging 1422 bytes with 6300-3's MTU decremented.
@@null_zero Just finished up explainer 3 and again nice job (explains a lot on how our current vendor fabric is setup and am ready for evpn :) ). Thanks for the explanation to my reply (might have missed that portion of the video), but your reply is in total line with what I would expect in the environment.
Great video, thanks. If you increase the MTU in one switch at a time in a network, is there a chance some flosw stop working before you finished to configrue all the switches with the new MTU?
Thanks. Ok, so assuming that the MTU is too low, then traffic will be already being dropped or running at a lower MTU (if Path MTU is in play), raising the MTU needs to be completed along the path before the traffic can flow. For example, see in my video how the pings didn't work until I configured MTU on each device, the point at which the drop occurred merely changed (those Giant Errors that moved from 6300-3 to 6300-2 after I configured the higher MTU on 6300-3.) Changing MTU can also be an impactful change in some instances. For example, if OSPF is running on the underlay and interface MTU is significant in the OSPF neighbour build state, which it is on AOS-CX, then altering the MTU on one end of an link will bring down OSPF until both ends of the link are identical. In conclusion, MTU changes should be planned to be completed in the network path as a whole and considered to be impactful changes that require outage windows. If you are investigating this in the lab, just configure one switch at a time and observe the results, in most case I found there was a hit, or everything needed to be configured along the path for success.
Great series. Thanks to you I'm starting to finally understand VXLAN and EVPN.
This has been a great series, thank you so much!
Thank you so much for these video series
Great series; learned a ton!
Great video, I ran into a situation identical to this with an ISP. Their solution was to adjust the MTU. I didn't understand how that worked, but I do believe you just explained it. Is this something that ISPs that use EVCs don't have to deal with since they aren't using VXLANs?
Thanks. It isn't so much that the carriers are using VXLAN, in my example, the carrier transit network would be 6300-3 (the switch in the middle), the VXLAN VTEP would be customer side devices connecting to the carrier network. Moreover, one of the main use cases of VXLAN is that the VTEP is even further into the customer network, right on the server, so the carrier is merely carrying the VXLAN encapsulated in IP, their devices just see an IP packet.
For the example you give, of EVC, the carrier is merely an L2 transit network then, in this instance, their devices would have to be able to carry frames that are larger than the standard 1514 / 1518 bytes 'on the wire' (or 1522 if we count the trailer). I can't speak for ISPs but I think they would just enable 'jumbo frames' L2 frames up to 9000 bytes on their network, it can't be cost effective or good for customer service if an ISP is having to negotiate their transit network MTU with every customer, just because the customer wants to run protocol X,Y,Z (in this case, VXLAN) on their links.
We do get queries in where carriers say they have a fixed MTU of 1500 bytes and push the problem to the customer, hence why I included the 'Alternatives' section. But then VXLAN is standard networking now, the RFC is from 2014, so if the carrier is saying the link is fixed time to negotiate a new contract, or look to a different carrier.
I appreciate the reply. This particular case was a bit different. It's a point to point with an ISP upstart, they leased the firber from another startup. So many variables. I do not believe that they were using EVCs, so I assume either VXLAN or a VRF configuration.
@@Scotty_J. Ok, yes, that does sound like a setup that starts to complicate things. If the ISP themselves are using additional encap., like VXLAN, then they would have to be aware of the overhead they are adding themselves.
Traffic MTU concerns are not new to ISPs though, there are such concerns when it comes to MPLS, another technology used by ISPs that adds overhead. Any ISP worth your business will have a clear idea and strategy around this, and be able to communicate it to customers.
First off, great series on VXLAN, very useful. QQ on this one. After you showed all routers were default 1500 MTU, your first ping with a request that was too large made it to 6300-3 and then got dropped as a giant. So is MTU only inbound on CX, since this packet successfully exited 6300-1 1/1/1 vtep interface which was still set at default 1500 mtu?
Thanks. In answer to your question, that's not what I observed but it was hard to prove where the traffic is being dropped.
The issue I had with the VXLAN traffic that exceeded the MTU on 6300-1 was to show any indication that the packet was being dropped. Looking around the 16:30 mark, I pinged with 1423 bytes and that failed, the 6300-3 transit switch doesn't increment its Giants counter with that but then I couldn't find any counter that incremented on the 6300-1. I remembered back to the RFC, that VTEPs must not fragment, and they can silently discard. Rather than go deep researching this, I just accept that the 1423 ping is being dropped on 6300-1 (it doesn't hit 6300-3) then move on to pinging 1422 bytes with 6300-3's MTU decremented.
@@null_zero Just finished up explainer 3 and again nice job (explains a lot on how our current vendor fabric is setup and am ready for evpn :) ). Thanks for the explanation to my reply (might have missed that portion of the video), but your reply is in total line with what I would expect in the environment.
Hi 6300-3 is managed by you. If we want to connect 2 vtep switch via internet how can i do ?
Can't we enable jumbo frames as a solution to fragmentation.
Great series Joe. QQ: Why were the packets dropped when they exceeded the MTU? By default Pings from Ubuntu come with DF bit set?
Great video, thanks. If you increase the MTU in one switch at a time in a network, is there a chance some flosw stop working before you finished to configrue all the switches with the new MTU?
Thanks. Ok, so assuming that the MTU is too low, then traffic will be already being dropped or running at a lower MTU (if Path MTU is in play), raising the MTU needs to be completed along the path before the traffic can flow. For example, see in my video how the pings didn't work until I configured MTU on each device, the point at which the drop occurred merely changed (those Giant Errors that moved from 6300-3 to 6300-2 after I configured the higher MTU on 6300-3.)
Changing MTU can also be an impactful change in some instances. For example, if OSPF is running on the underlay and interface MTU is significant in the OSPF neighbour build state, which it is on AOS-CX, then altering the MTU on one end of an link will bring down OSPF until both ends of the link are identical.
In conclusion, MTU changes should be planned to be completed in the network path as a whole and considered to be impactful changes that require outage windows. If you are investigating this in the lab, just configure one switch at a time and observe the results, in most case I found there was a hit, or everything needed to be configured along the path for success.
What is the IP MTU? Is this all after the IP-Header?