I once got an IT job working in a car dealership company in the UK. 1 Particular location was reporting really poor "internet access", and internal app access. No one knew anything about the site, so i set about an Audit end to end. I found in total 21 "100mb Hubs", not switches. There were collisions everywhere, a literal IT disaster. There was no real budget so spend, so i managed to get a job lot of Netgear Prosafe switches from a company that had closed down cheap. Then 1 weekend me and a assistant re-patched the entire site. Ran like a dream after that.
While daisy-chaining switches may function without issues in a low-stress, small-scale environment, this practice can wreak havoc in a corporate setting where the switches are under heavy load. Typically, each switch needs to offload data through 1Gbps or 10Gbps links, which can easily become saturated in a daisy-chain configuration. In a modern corporate setting, it's common practice to connect each set of 48 ports back to aggregation switches using fiber links. This approach minimizes the hop count to local resources and backup servers, net circuits, enhancing network performance and reliability.
I agree with this as well- for work environments, and for home networks where there are a lot of devices and/or a lot of traffic to deal with. For my home network I currently have a total of three Netgear 5-port 2.4 Gbit switches that are daisy-chained (for now), with the first switch connected to a 2.4Gbit port on our internet router/firewall. All of our internal devices that plug into these switches support up to 1Gbit (nothing more)… so in essence we have a 2.4Gbit backbone that stretches from the inside interface of our internet router to all of the switches. I realize this network design is not standard practice in commercial environments… but for us right now, it’s working pretty well.
There are switches that can handle 25Gbps through 25G SFP fiber interconnects but yes, you can still saturate these links if enough devices connected to these switches try to communicate all at once. This can happen with even 2 or 3 layer deep switches when large storage servers are doing backups or 100s of PCs all need to do updates at the same time. I have personally saturated 2 layer deep (layer 3) switches simply doing Veeam backups.
Thats more of a bottlenecking issue than daisychaining. Obviously if each vlan had 254 clients trying to compete then its just going to impinge on its neighbours. Having said that in the stadium you'd just get the bandwidth of the initial connection shared amoung all.
So in my experience, you are asking for trouble after about the 4th switch. This is using unmanaged switches. Mostly it comes down to ARP table issues with clients on the first hub not being able to talk to clients on the last hub... sometimes. It looks like everything is working, then weird networking issues start popping up, lost packets, being able to ping one client but not the one the next port over, etc.
@@kreuner11 each switch needs to maintain a table of which port maps to which MAC address. Daisy chain 4 switches together and plug a device in switch D, in an ideal world, the ARP broadcast would make its way back to switch A and everyone is happy. In the real world, with heavy network traffic, cheap switches each hosting many devices, this is not the case. The example in the video is cool, but these are industrial managed switches about 10 to 20 times more expensive than their unmanaged counterparts. Plus each VLAN is only hosting 2 ports. I'm kind of an old timer at this I remember when a 10 Mb/S hub with a 100 Mb/s backbone port was state of the art and we looked forward to replacing the coax lines. Spend a couple days troubleshooting weird network issues and dealing with pissed off users teaches long lasting lessons. With better processers and more memory in newer switches, this may not be the issue it once was, but I still get a nervous when daisy chaining more than 4 switches together.
@@kreuner11 switches have a buffer inside to store a cache of the ARP responses of devices connected to the various ports. This allows if a packet arrives at the switch directed to MAC address X to know that it needs to send packets out of port 5. If the MAC address is not found in the table the switch has to send the message to all the ports of the switch, and then cache the eventual response. Now a swtich, expecially a cheap one, doesn't have a lot of memory. And with a lot of devices this tables gets full quickly, and generates the sort of errors that the other user pointed out. By the way it's a problem if you use small unmanaged switches, modern switches have a lot of memory so this is not a big deal.
I did check the arp table, filmed it and nearly included it in the video but I didn't think enough people would understand what I was showing. Maybe I'm underestimating my average viewer. In any case, the arp table looks exactly like you would expect, the same mac learned on half the ports.
Switches don’t care about ARP tables, which map IP addresses to MACs. You mean MAC address tables which map MACs to ports. MAC address tables are per VLAN on almost all switches that correctly support VLANs (almost all switches that claim VLAN support), so it works fine. Many switches don’t explicitly support VLANs but will still forward VLAN encapsulated frames. They likely would have MAC address table issues.
When the MAC table is full what happens. The switch does know which port a frame is supposed to send to. So your switch becomes a hub and it sends the frame to all ports. Common trick to reduce performance is to fill the MAC table on a switch.
Answer: How many switches do you have? 🙂 The default spanning-tree settings limits rings to 7 - beyond that loops will not be detected. If you tweak the timers, maybe 14-15 hops. Of course, if you don't care about spanning-tree (as you didn't), there's no physical limit. As others have pointed out, this sort of arrangement rapidly creates traffic bottlenecks (esp. for broadcast.) (Industrial ethernet can end up with daisy chains in the dozens, but they're low bandwidth applications. I've setup a Sun v20z cluster [rack] with 32 machines; the management interfaces is a 3-port cascaded switch, though it does not run or interfere with stp... you're supposed to chain them and connect to the network from the top and bottom of the string.)
@@tyttuut How much current can flow in an electrical circuit depends on the line resistance. The higher the resistance, the lower the resistance. In a short circuit situation, a fuse can only disconnect quickly when there is enough current flowing. When you limit current by daisy chaining power strips, the fuse might not disconnect in time and cables get damaged (melting, burning).
But if you talk to any inspector, chaining power strips leads to an immediate massive fire because you can overload the first strip, even if it has a fuse. Yes they are this stupid!
I think it’s more of an issue in the US than countries with fused plugs. In the US, you can buy (unfused) extension flexes with different gauges of wire and supported loads, which is nuts in itself.
Yes, in the US with their wild extension cords this could be scary. However, in any civilized country you can chain as many as you want. In EU the extension cords are rated for 16A, the largest residential breaker is 16A so you will pop the breaker before overloading the extension cord. It's almost like someone though about it and wrote a standard, amazing right. Worst that can really happen is you get too much voltage drop and whatever you plug in doesn't work.
I never really get that argument, as long as total load doesn't exceed the capacity of initial link it doesn't matter. I suppose the reason is people will always consume all plugs and most have no clue what uses what power. So for breakfast you stick your kettle on, microwave and toaster thats a good 30Amps!
My guess is the original limit of 5 was based on hub model and was a limit of CSMA/CD where every device was on the same physical collision segment. There was a limit to the maximum amount of CSMA/CD response time. With switches, every switch creates a new collision segment.
That's where the 100m limit comes in. Cascaded _HUBS_ do introduce an additional limit because of processing delay; switches create different broadcast domains, so that isn't a concern.
@@jfbeam in the very old time of thick concentric cable, there was a limit of 5 segment of 500m each. 2500m was limit, where delays made collisions not working. IMO - the same applies for hubs, hovever - that was for 10M ethernet, I think 100M shortens distance to 250m. But above only applies to real hub, not to switches.
You could say that each hop will be a level lower/deeper into the network. Therefore you would change the STP priority. That would make it up to 16 diferent levels. That could also be an answer.
For Enterprise switches like the Dell PowerConnect I usually just "stack" them together. I used to be able to stack up to 12 switches as one super switch then latest firmware update dwarfed it to 8 switches. Ah well. If the switches aren't able to stack then I would pick the top best switch as the home run switch and then plug the other switches into that rather than daisy chaining them together. Less problems that way.
sure, switch stacking is a solution, but in the setting presented in the video (the stadium) would you be able to stack them across that big of a distence ?
This was like running in OS on a VM versus bare metal. You just create a switch in a VM and then call it good....the two are not equivalent. Despite this, I did enjoy the video and I found it interesting. Thank you for your time....
There are no technical limitations to daisy-chaining to any standard, but most modern switches use the spanning tree protocol. The advantage is that this protocol prevents loops, which can easily be created accidentally and take down your entire network. It also allows for intentional redundant paths, so if one path goes down, another will take over and the network will continue to operate normally. The protocol works automatically, there is nothing you need to configure, just turn it on (unless turned on by default) and the switches do the rest. After an automatic learning phase, they know the layout of your network and will automatically break any loops and thus redundant paths, which are basically always loops. Yet the spanning tree protocol has a 7-hop limit, meaning that every switch must be reachable from every other switch within a maximum of 7 hops, otherwise the protocol cannot work properly. The only thing you can do if your network needs to be larger is to segment it and restrict the spanning tree protocol to each of those segments, but not use it between the segments.
I had about 20 daisy chained to provide WiFi to a large area, but the last few switches would randomly stop pinging and strange things would happen. Since they were daisy chained with fiber and I had 6 strands per box, I linked the strands at the halfway point and took the last half straight back to the main fiber switch that the others were on, problem solved. These were Cisco 3560-CX switches for reference.
The depth is actually not the problem. You should theoretically be able to chain an infinite number of them together. However, width is the problem; the number of devices you connect to a single port. If you daisy chain more than 5 switches with each 48 ports, with at each end a computer wanting to stream content. This will give problems. (Even when we ignore the bandwidth to your ISP)
It's worth mentioning most of the added delay end to end you are seeing here is all the added Ethernet SerDes delay and not actually table lookups done by the switch per hop, although there is some delay for this as well. Some other comments talking about address spaces or CAM table exhaustion are a little uninformed. In a layer 2 switch you do not need the switch to have an IP address per vlan or at all. It also does not technically need a MAC address although in practice these boxes have a chassis MAC address. Reason being that the switch simply inspects the Ethernet header and forwards it out the interface that this destination was previously learned through (assuming post convergence and arping and whatnot). The switch does not attach or sub out its own mac address into the frame and thus never technically needs one for operation. So at most in this setup you would have two MAC address entries per VLAN in the CAM table of each switch. One for the Laptop one for the NAS. That's it, and if these were individual switches instead of VLANS (no technical difference here for the purposes of testing since spanning tree isn't a factor in a daisy chain where there intrinsically are no loops) there would be 2 MAC addresses per switch which is obviously never hitting close to any real world limit. In this test there is no technical limitation to how many switches you could chain together. CAM table size isn't going to limit you unless your table holds one entry (lol), Spanning tree isn't a factor, flooding behavior isn't a factor. The only thing that would eventually stop test results would be ICMP timeouts or TCP RTT issues once you eventually hit these sorts of delays.
That comment from the stadium guy probably explains some of the reliability issues on public networks. It probably works fine enough, but im sure the people using it still suffer from it being suboptimal.
Dang, modern networking hardware is kinda crazy if you think about it. Most of the latency really is in the physical distance the signal has to traverse. No wonder fiber is so drastically faster.
a 64 byte ping is about a 796 bit ethernet frame including vlan header. the whole ping exchange is going to be about 1592 bits divided by 100 million bits per second (100mbps ethernet), the total forwarding time into the switch's buffer is 15.92 microseconds. since we have to forward it back out, that's 31.84 microseconds. Which is very, very close to what you actually measured. Please replay your experiment with different ping payload sizes. Also, I should point out that modern datacenter switches that do cut-through-forwarding do not wait for the entire frame to make it into a buffer before the output interface can begin forwarding it, vastly reducing that forwarding delay. Very cool video!
Each switch adds a tiny inter-frame delay. How many switches is "how much inter-frame delay can Ethernet tolerate". No simple answer really ... it is not infinite but it is very large. In practice adding inter-frame delay can increase collisions under load, but the impact of that type of issue depends on many variables. Switches are store and forward, so how many you can have is like asking how many people can pass a bit of paper along in a chain, the answer is lots, and assuming you don't need an ACK for the note and just fire out the next one in sequence then the chain length does not matter from a performance perspective, It is more fault tolerance issue than a performance issue.
had about 200x 24 port managed switches daisy chained for a LAN party with ~4k computers, no where near DREAMHACK's ~12k computers , the imprtant thing is not getting cables mixed up and going back to the same device's ports or else you get a burnt out switch , think we lost 5 because of that happening from end users getting their cables crossed and then just ended up plugging the patch cable back into the switch instead of their PC's , after that we started hot gluing them in and not allowing acess to the switches and at the time they were almost $1k AUD each to buy
STP gets a bit messy past 7 hops. It can trigger false positives/not relese properly without manually restarting the port. 7 hops are usually too many for in building deployments. This turns into something else on a city scale/regional ISP scale. There the rule is ignored but care is taken to split the network up so STP still sees it as sub 7 switch between Client and Router. There are deployments where you'll see upwards of 30 switches in a row that work just fine especially in multi gig scenarios where routers may not be fast enough to carry such traffic across. Tho these are no longer L2 and it takes carefull planning to ensure nothing goes wrong duing operation. In a case of a L3 switch you do decrement the TTL value so in theory in an IPv4 network you can have max 255 L3 switches (and probobly some L2 as well) before you hit a hard limit. As to the speed. Switches preform any tasks at sub ms speeds so the biggest delay to your ping is the PC and the NAS. You can hardly excede 5ms with a 30 switch chain.
Switches maintain caches of destination MAC address and the port that they are directly or indirectly connected to. Once a switch observes too many MAC addresses and fills the cache (which size depends on quality of the switch), it will be forced to send the packets to all ports (to relearn the missing MAC address when it observes a response. So, it's not a limit on the number of switches...it is a limit on the number of local endpoints on a network and the switch's ability to benefit from the internal caching. Otherwise, the switch will just act like a hub and send packets to all ports.
that's awesome of course. But the fact that you used commercial/enterprise switches may have something to do with the speeds you're getting. What if you try this on consumer hardware as you had initially wanted to do...
I recall some rule about 5-4-3 (which I googled) - The rule mandates that there can only be a maximum of five segments, connected through four repeaters, or concentrators, and only three of the five segments may be mixing segments. This last requirement applies only to 10BASE5, 10BASE2, and 10BASE-FP Ethernet segments. This was back about 25-30 years ago when we were running 10Base2 and token ring networks with Novell. With regards to your cisco - I wonder if that is a little misleading as the switch cpu/ram etc would dictate the capacity and throughput.
I think you might get different results if you had much longer cables as well. You would have cables everywhere but your latency would increase and maybe things just start not being able to handle it. Maybe the frames just get forwarded from switch to switch and all is well. Combine that testing with a filled MAC table and could have absolute carnage. I'd love to see that but that's a lot of cable.
If I used 100 metre patch cables I would be adding 4800 metres to the distance. Signals travel a 5 millionths of a millisecond per metre so an extra 24,000 millions of a millisecond or .0024 ms extra. It would make a difference it's not enough to matter. I do agree that chaining 48 real, fully populated switches with devices generating traffic would give us a different outcome.
I think that a lot of the concerns about how many switch hops you can have come from much further back...there WAS a limit to how many HUBs you could daisy chain before it just plain broke. This was before switches were readily accessible or affordable. Nice test scenario, though!
There is no technical limit, and you would probably need to get a hundred switches together for an empty test environment to have issues at all. Its just a practice that is likely to lead to issues in the real world. Like being unable to tell where a problem is being caused as its impacting everything, or a bottle neck you dont know you have. Much easier to assure performance and service a model with mostly a single hub/core. But if your aware of what you are doing and what bandwidth you'll see and have good reason then go for it.
Fiber isn't really that expensive. Especially if you're okay with generic SFPs, a cheaper brand switch, and multimode fiber. Which, to be fair, you should be okay with if your alternative is daisy chaining some cheap TP-Link switches.
On the stadium story: Running Cat6 to a switch then to another switch over and over again (somehow with power for each switch too) instead of running a single OF cable is doing my head in. :P I'm thinking it would be less fuss to run OF instead of all the switches, especially since cable would have to be run either way.
Daisy chaining power strips is totally safe unless you pull more than like 6kW through them. If they are fused correctly you never will see anything go up in smoke. I've seen a video here on youtube where they created a string of hundreds. There will be a voltage drop due to wire length.
It was never about latency, or even uplink capacity. It was always about the spanning tree. Change those switches to traditional spanning tree and make a loop and watch it melt down. About 15 years ago I was trying to stress test two 10g links and used something similar. I routed the traffic back and forth across the links 15 times or so with vlans and cables crossing back and forth and was able to max out the 10g link with my laptop and iperf.
routers not switches this is why you see multiple hops when using something like trace route to something on the internet and no hops when going to something on your own lan. its layer 2 vs 3
I definitely 'feel' like a more complex setup with devices plugged in in the chain as well as at either end would cause issues, but have never explored beyond the hub and spoke environment. I wonder if you could simulate it in GNS3?
given that the internet is basically a lot of switches, partially daisy chained, sometimes with some branches, sometimes with a loop in them, it shouldn't be that surprising that you ca, in fact, daisy chain a lot of switchtes
The latency is because each vlan has to receive the complete packet and forwards in software. For dumb switches theres no limit because once the packet leaves its as good as the first packet. People here are saying that in large corporate networks under heavy load they do see degredation. This is because broadcast traffic or traffic the switch doesnt know where to route gets copied to all ports and this leads to storms which then impinges on everything else. But daisy chaining them for a stadium would work to a huge scale, and its a reasonable approach, each switch is then a repeater. I think performance will degrade at large scale because theres a limit to how many macs each switch can learn before it starts forgetting the first leading to increased broadcasts. I'm going to suggest the limit is about 254 switches before you start needing to be creative. The internet is basically 4bn switches daisychained.
Unless he's making them Layer 3 vlans they are not counted as hops. The max on that would be 255. The main thing you'd run into, and what Cisco is talking about is that spanning tree has issues when you go wider than 7 switches away from the root of the tree. Spanning tree is supposed to be unreliable after that.
A router is a hop. This is a single broadcast domain. The unicast Ethernet packets are not processed but are either forwarded as is out all ports except the port received or, if the switch already knows what port the destination is downstream is connected to, forwards it out only that port. If the frame is a multicast or broadcast Ethernet frame, it is forwarded out all ports. The frames are not decapsulated, and the IP (if the enveloped protocol is IP) TTL is not decremented.
Managed do. Unmanaged just show up as a bunch of devices on the same port. Much the same as having a bunch of VMs each with their own IP plus the host machine. A unmanaged switch is just a dumb bridge.
my guess is slightly over 2 billion, if you tell all of them to act as routers (so possibly using custom firmware for them), don't care about internet connectivity, and use a /32 address space for each of them
I can give you the real answer, which is, until u max the cam table space of any 1 switch. switches operate in 2 modes, switch mode, which is normal operation and hub mode. hub mode you will most likely never see, its when the switch sends ALL the packets out on ALL the ports. hub mode is entered when the cam table of a switch goes OOM. the cam table stores the mac address to port number schema. this is how switches determine where data is sent from an incoming packet. it looks for the destination mac address in the cam table, finds the port and bob's your uncle. when a cam table is filled with nonsense data like from an arp flood. the switch may switch modes into hub mode, the switch can no longer determine where packets are supposed to go so it can in essence just broadcast them to the whole network. when it does this, other switchinges being in the same mode will do same, causing a packet storm, and crippling the network. for corporate switches, cam table sizes are specified as part of the spec. bigger badder switches have huge cam tables. end point switches keep them small. preventing corporations from using small cheap switches as core switching elements.
"hub mode" is unicast flooding. Some switches will flood, some will drop once the table is full. (some can be configured to drop unknown unicast) By disabling mac-learning, or setting the table size to zero, some switches can be turned into a hub. (but you really don't want to)
probably the switch did some optimisation and skipped ports...if you do this with physical switches then please expect a latency of 0.5 ms to 1 ms per switch..I have tried with 5 physical soho switchs
@@matthewdaley7535 This is my assumption without any proof -> Skipping hops means - the software inside the router optimizes the route because as far it sees start and end is the same machine, vlan is software so it can 'route' it if it can decide it is safe... | 5ms per switch -> Its 0.5ms to 1 ms per swich...so 5ms is total for 5 physical home/small office switches (TP link green gigabit) I have done it myself.
@@nid274 Stop making assumptions. These are SWITCHES not ROUTERS. Switching happens in microseconds (and there are thus that operate in nanoseconds!) The only way to "bypass hops" is through odd, proprietary technology like Cisco's Multi-Layer Switching (mls) - which tags frames with ingress/egress information. (as far as I know, that has been dead for years.)
@@nid274 Yes. It will route what it is CONFIGURED to route. Setting up 24 VLANs does not automatically create 24 routed interfaces. (if he did, it would add about 1ms each... the 2960 isn't a very good layer-3 device.) Does he need to post the switch configurations for you to understand it's all switching and no routing?
Maybe smartphones didn't exist when the stadium was built and current management does not have the money to run fiber cabling all over. This happens all the time in the real world that we have to work with a non-ideal, non-cisco budget and just make things work.
You neglected to account for max segment length. Ethernet is rated to 100 meters per LAN segment (broadcast domain) for CDMA to be reliable. Sure it can work but what will probably happen on chatty networks is that retransmissions will start flooding as CDMA collisions increase across the segment.
It’s fun, but I don’t think it’s a valid test. The managed switch you were using only has to hold a single MAC table. It also doesn’t have to deal with ever increasing numbers of BPDUs from other switches or several other network awareness protocols that might be in use such as VLAN discovery or simply CDP other LLDP. Most importantly it doesn’t have to deal with multiple STP connections to other switches which would be vital in a large extended network so you could close the loop without making a broadcast storm. As I recall the recommended STP hop limit is only 8 hops. It would have been much more interesting to see how the background traffic climbed as you added more switches and to see at what point it swamped out host traffic.
I bought two for this project thinking there is no way it would work across 48 hops. The switches were only $35US and the cables were $1 each. But it worked so well that I figured even 8 would not be enough to make it fail and I didnt want to end up with 8 100Mbit switches that I would just have to sell.
In the real world, the best senario would be a daisy chain, and then a cable back from the last switch to the first, so incase any switch in the chain dies, the network stays up, and that's maybe what that reddit user did
Seems to me all you've done is make a really long connection. You haven't emulated a real world scenario because you don't have multiple computers sending and receiving data between these daisy chained LAN hubs. Once these LAN hubs have to start dealing with multiple traffic sources, you have something that can be tested.
I once got an IT job working in a car dealership company in the UK. 1 Particular location was reporting really poor "internet access", and internal app access. No one knew anything about the site, so i set about an Audit end to end. I found in total 21 "100mb Hubs", not switches. There were collisions everywhere, a literal IT disaster. There was no real budget so spend, so i managed to get a job lot of Netgear Prosafe switches from a company that had closed down cheap. Then 1 weekend me and a assistant re-patched the entire site. Ran like a dream after that.
Yes, hubs are not scalable. But a switch is akin to a single connection for each port, so thats why you no longer have collision lights.
Classic.
Damn, I was not even aware that they made 100 millibit hubs.
@@voioxunderrated comment lmao
While daisy-chaining switches may function without issues in a low-stress, small-scale environment, this practice can wreak havoc in a corporate setting where the switches are under heavy load. Typically, each switch needs to offload data through 1Gbps or 10Gbps links, which can easily become saturated in a daisy-chain configuration. In a modern corporate setting, it's common practice to connect each set of 48 ports back to aggregation switches using fiber links. This approach minimizes the hop count to local resources and backup servers, net circuits, enhancing network performance and reliability.
Agreed.
I agree with this as well- for work environments, and for home networks where there are a lot of devices and/or a lot of traffic to deal with. For my home network I currently have a total of three Netgear 5-port 2.4 Gbit switches that are daisy-chained (for now), with the first switch connected to a 2.4Gbit port on our internet router/firewall. All of our internal devices that plug into these switches support up to 1Gbit (nothing more)… so in essence we have a 2.4Gbit backbone that stretches from the inside interface of our internet router to all of the switches. I realize this network design is not standard practice in commercial environments… but for us right now, it’s working pretty well.
There are switches that can handle 25Gbps through 25G SFP fiber interconnects but yes, you can still saturate these links if enough devices connected to these switches try to communicate all at once. This can happen with even 2 or 3 layer deep switches when large storage servers are doing backups or 100s of PCs all need to do updates at the same time.
I have personally saturated 2 layer deep (layer 3) switches simply doing Veeam backups.
Thats more of a bottlenecking issue than daisychaining. Obviously if each vlan had 254 clients trying to compete then its just going to impinge on its neighbours. Having said that in the stadium you'd just get the bandwidth of the initial connection shared amoung all.
Mind to do a test?
So in my experience, you are asking for trouble after about the 4th switch. This is using unmanaged switches. Mostly it comes down to ARP table issues with clients on the first hub not being able to talk to clients on the last hub... sometimes. It looks like everything is working, then weird networking issues start popping up, lost packets, being able to ping one client but not the one the next port over, etc.
How would that occur in a modern setting? An arp packet is a regular broadcast packet, a switch generally shouldn't be rejecting broadcast packets
@@kreuner11 each switch needs to maintain a table of which port maps to which MAC address. Daisy chain 4 switches together and plug a device in switch D, in an ideal world, the ARP broadcast would make its way back to switch A and everyone is happy. In the real world, with heavy network traffic, cheap switches each hosting many devices, this is not the case. The example in the video is cool, but these are industrial managed switches about 10 to 20 times more expensive than their unmanaged counterparts. Plus each VLAN is only hosting 2 ports. I'm kind of an old timer at this I remember when a 10 Mb/S hub with a 100 Mb/s backbone port was state of the art and we looked forward to replacing the coax lines. Spend a couple days troubleshooting weird network issues and dealing with pissed off users teaches long lasting lessons. With better processers and more memory in newer switches, this may not be the issue it once was, but I still get a nervous when daisy chaining more than 4 switches together.
@@kreuner11 switches have a buffer inside to store a cache of the ARP responses of devices connected to the various ports. This allows if a packet arrives at the switch directed to MAC address X to know that it needs to send packets out of port 5. If the MAC address is not found in the table the switch has to send the message to all the ports of the switch, and then cache the eventual response.
Now a swtich, expecially a cheap one, doesn't have a lot of memory. And with a lot of devices this tables gets full quickly, and generates the sort of errors that the other user pointed out. By the way it's a problem if you use small unmanaged switches, modern switches have a lot of memory so this is not a big deal.
@@alerighi ahh, my bad. Yeah running out of space in the MAC table will cause problems
Number of switches is not the limiting factor in that case, number of client devices is, which isn’t what this video was asking.
Nice solution using vlans!
I would have guessed that the arp table would have gotten a bit screwy
I did check the arp table, filmed it and nearly included it in the video but I didn't think enough people would understand what I was showing. Maybe I'm underestimating my average viewer. In any case, the arp table looks exactly like you would expect, the same mac learned on half the ports.
@@matthewdaley7535 We do appreciate the technical details, just saying.
@@matthewdaley7535 You underestimated your viewers. We are very interested in that stuff :)
Switches don’t care about ARP tables, which map IP addresses to MACs. You mean MAC address tables which map MACs to ports.
MAC address tables are per VLAN on almost all switches that correctly support VLANs (almost all switches that claim VLAN support), so it works fine.
Many switches don’t explicitly support VLANs but will still forward VLAN encapsulated frames. They likely would have MAC address table issues.
When the MAC table is full what happens. The switch does know which port a frame is supposed to send to. So your switch becomes a hub and it sends the frame to all ports. Common trick to reduce performance is to fill the MAC table on a switch.
Answer: How many switches do you have? 🙂
The default spanning-tree settings limits rings to 7 - beyond that loops will not be detected. If you tweak the timers, maybe 14-15 hops. Of course, if you don't care about spanning-tree (as you didn't), there's no physical limit. As others have pointed out, this sort of arrangement rapidly creates traffic bottlenecks (esp. for broadcast.)
(Industrial ethernet can end up with daisy chains in the dozens, but they're low bandwidth applications. I've setup a Sun v20z cluster [rack] with 32 machines; the management interfaces is a 3-port cascaded switch, though it does not run or interfere with stp... you're supposed to chain them and connect to the network from the top and bottom of the string.)
You can *theoretically* daisy chain as many power strips as you want as long as you don't overload the circuit.
You also need to consider line resistance.
@@jor7137 can you elaborate?
@@tyttuut How much current can flow in an electrical circuit depends on the line resistance. The higher the resistance, the lower the resistance. In a short circuit situation, a fuse can only disconnect quickly when there is enough current flowing. When you limit current by daisy chaining power strips, the fuse might not disconnect in time and cables get damaged (melting, burning).
@@jor7137 why would anything melt if there isn't enough current flowing to blow the fuse?
@@tyttuut a circuit breaker can take up to 30 minutes to trip at it's rated current and only trips immediately at 1,5 to 2 times of it's rated current
1:50 uhm yeah mate, not really an issue when you're pulling 5 watts per switch lol
But if you talk to any inspector, chaining power strips leads to an immediate massive fire because you can overload the first strip, even if it has a fuse. Yes they are this stupid!
I think it’s more of an issue in the US than countries with fused plugs.
In the US, you can buy (unfused) extension flexes with different gauges of wire and supported loads, which is nuts in itself.
@@bdavbdavbdavbdav these are definitely Australian plugs, 240v makes this even less of a concern
Yes, in the US with their wild extension cords this could be scary. However, in any civilized country you can chain as many as you want. In EU the extension cords are rated for 16A, the largest residential breaker is 16A so you will pop the breaker before overloading the extension cord. It's almost like someone though about it and wrote a standard, amazing right. Worst that can really happen is you get too much voltage drop and whatever you plug in doesn't work.
I never really get that argument, as long as total load doesn't exceed the capacity of initial link it doesn't matter. I suppose the reason is people will always consume all plugs and most have no clue what uses what power. So for breakfast you stick your kettle on, microwave and toaster thats a good 30Amps!
That CCNA stuff was about hubs. And yes, as you mention in the card, when running spanning tree.
My guess is the original limit of 5 was based on hub model and was a limit of CSMA/CD where every device was on the same physical collision segment. There was a limit to the maximum amount of CSMA/CD response time. With switches, every switch creates a new collision segment.
This makes perfect sense. Just knowing this makes you old. :-)
@@matthewdaley7535 11 years ago my college networking classes covered collision and broadcast domains.
That's where the 100m limit comes in. Cascaded _HUBS_ do introduce an additional limit because of processing delay; switches create different broadcast domains, so that isn't a concern.
Yes from that point of view its unlimited. Every packet is as good as the first one.
@@jfbeam
in the very old time of thick concentric cable, there was a limit of 5 segment of 500m each.
2500m was limit, where delays made collisions not working.
IMO - the same applies for hubs, hovever - that was for 10M ethernet, I think 100M shortens distance to 250m.
But above only applies to real hub, not to switches.
You could say that each hop will be a level lower/deeper into the network. Therefore you would change the STP priority. That would make it up to 16 diferent levels. That could also be an answer.
For Enterprise switches like the Dell PowerConnect I usually just "stack" them together. I used to be able to stack up to 12 switches as one super switch then latest firmware update dwarfed it to 8 switches. Ah well. If the switches aren't able to stack then I would pick the top best switch as the home run switch and then plug the other switches into that rather than daisy chaining them together. Less problems that way.
sure, switch stacking is a solution, but in the setting presented in the video (the stadium) would you be able to stack them across that big of a distence ?
@@razorr3751 Generally a good idea to try home run each switch to the main one. Sometimes you can't.
This was like running in OS on a VM versus bare metal. You just create a switch in a VM and then call it good....the two are not equivalent.
Despite this, I did enjoy the video and I found it interesting.
Thank you for your time....
There are no technical limitations to daisy-chaining to any standard, but most modern switches use the spanning tree protocol. The advantage is that this protocol prevents loops, which can easily be created accidentally and take down your entire network. It also allows for intentional redundant paths, so if one path goes down, another will take over and the network will continue to operate normally.
The protocol works automatically, there is nothing you need to configure, just turn it on (unless turned on by default) and the switches do the rest. After an automatic learning phase, they know the layout of your network and will automatically break any loops and thus redundant paths, which are basically always loops.
Yet the spanning tree protocol has a 7-hop limit, meaning that every switch must be reachable from every other switch within a maximum of 7 hops, otherwise the protocol cannot work properly. The only thing you can do if your network needs to be larger is to segment it and restrict the spanning tree protocol to each of those segments, but not use it between the segments.
All I could hear in my head was Xzibit saying: “We heard you like switches, so we got switches for your switches’ switches!!” 🤪 cool video mate 👍🏻
I had about 20 daisy chained to provide WiFi to a large area, but the last few switches would randomly stop pinging and strange things would happen. Since they were daisy chained with fiber and I had 6 strands per box, I linked the strands at the halfway point and took the last half straight back to the main fiber switch that the others were on, problem solved. These were Cisco 3560-CX switches for reference.
eevblog2 has a video where he daisy chains a heap of power point double adaptors
The depth is actually not the problem. You should theoretically be able to chain an infinite number of them together. However, width is the problem; the number of devices you connect to a single port. If you daisy chain more than 5 switches with each 48 ports, with at each end a computer wanting to stream content. This will give problems. (Even when we ignore the bandwidth to your ISP)
It's worth mentioning most of the added delay end to end you are seeing here is all the added Ethernet SerDes delay and not actually table lookups done by the switch per hop, although there is some delay for this as well.
Some other comments talking about address spaces or CAM table exhaustion are a little uninformed. In a layer 2 switch you do not need the switch to have an IP address per vlan or at all. It also does not technically need a MAC address although in practice these boxes have a chassis MAC address. Reason being that the switch simply inspects the Ethernet header and forwards it out the interface that this destination was previously learned through (assuming post convergence and arping and whatnot).
The switch does not attach or sub out its own mac address into the frame and thus never technically needs one for operation. So at most in this setup you would have two MAC address entries per VLAN in the CAM table of each switch. One for the Laptop one for the NAS. That's it, and if these were individual switches instead of VLANS (no technical difference here for the purposes of testing since spanning tree isn't a factor in a daisy chain where there intrinsically are no loops) there would be 2 MAC addresses per switch which is obviously never hitting close to any real world limit.
In this test there is no technical limitation to how many switches you could chain together. CAM table size isn't going to limit you unless your table holds one entry (lol), Spanning tree isn't a factor, flooding behavior isn't a factor. The only thing that would eventually stop test results would be ICMP timeouts or TCP RTT issues once you eventually hit these sorts of delays.
That comment from the stadium guy probably explains some of the reliability issues on public networks. It probably works fine enough, but im sure the people using it still suffer from it being suboptimal.
Doesn’t really matter in that situation since you’re gonna be paying attention to the game or performance or whatever.
Dang, modern networking hardware is kinda crazy if you think about it. Most of the latency really is in the physical distance the signal has to traverse. No wonder fiber is so drastically faster.
a 64 byte ping is about a 796 bit ethernet frame including vlan header. the whole ping exchange is going to be about 1592 bits divided by 100 million bits per second (100mbps ethernet), the total forwarding time into the switch's buffer is 15.92 microseconds. since we have to forward it back out, that's 31.84 microseconds. Which is very, very close to what you actually measured. Please replay your experiment with different ping payload sizes. Also, I should point out that modern datacenter switches that do cut-through-forwarding do not wait for the entire frame to make it into a buffer before the output interface can begin forwarding it, vastly reducing that forwarding delay. Very cool video!
Each switch adds a tiny inter-frame delay. How many switches is "how much inter-frame delay can Ethernet tolerate". No simple answer really ... it is not infinite but it is very large. In practice adding inter-frame delay can increase collisions under load, but the impact of that type of issue depends on many variables. Switches are store and forward, so how many you can have is like asking how many people can pass a bit of paper along in a chain, the answer is lots, and assuming you don't need an ACK for the note and just fire out the next one in sequence then the chain length does not matter from a performance perspective, It is more fault tolerance issue than a performance issue.
had about 200x 24 port managed switches daisy chained for a LAN party with ~4k computers, no where near DREAMHACK's ~12k computers , the imprtant thing is not getting cables mixed up and going back to the same device's ports or else you get a burnt out switch , think we lost 5 because of that happening from end users getting their cables crossed and then just ended up plugging the patch cable back into the switch instead of their PC's , after that we started hot gluing them in and not allowing acess to the switches and at the time they were almost $1k AUD each to buy
That should not fry a switch.
It will however probably jam that network segment
STP gets a bit messy past 7 hops. It can trigger false positives/not relese properly without manually restarting the port. 7 hops are usually too many for in building deployments. This turns into something else on a city scale/regional ISP scale. There the rule is ignored but care is taken to split the network up so STP still sees it as sub 7 switch between Client and Router.
There are deployments where you'll see upwards of 30 switches in a row that work just fine especially in multi gig scenarios where routers may not be fast enough to carry such traffic across. Tho these are no longer L2 and it takes carefull planning to ensure nothing goes wrong duing operation.
In a case of a L3 switch you do decrement the TTL value so in theory in an IPv4 network you can have max 255 L3 switches (and probobly some L2 as well) before you hit a hard limit.
As to the speed. Switches preform any tasks at sub ms speeds so the biggest delay to your ping is the PC and the NAS. You can hardly excede 5ms with a 30 switch chain.
Switches maintain caches of destination MAC address and the port that they are directly or indirectly connected to. Once a switch observes too many MAC addresses and fills the cache (which size depends on quality of the switch), it will be forced to send the packets to all ports (to relearn the missing MAC address when it observes a response. So, it's not a limit on the number of switches...it is a limit on the number of local endpoints on a network and the switch's ability to benefit from the internal caching. Otherwise, the switch will just act like a hub and send packets to all ports.
Thanks, this is very helpful. So, it's a myth to really care about it.
Good to know when I design a network next time.
that's awesome of course. But the fact that you used commercial/enterprise switches may have something to do with the speeds you're getting. What if you try this on consumer hardware as you had initially wanted to do...
I recall some rule about 5-4-3 (which I googled) - The rule mandates that there can only be a maximum of five segments, connected through four repeaters, or concentrators, and only three of the five segments may be mixing segments. This last requirement applies only to 10BASE5, 10BASE2, and 10BASE-FP Ethernet segments. This was back about 25-30 years ago when we were running 10Base2 and token ring networks with Novell. With regards to your cisco - I wonder if that is a little misleading as the switch cpu/ram etc would dictate the capacity and throughput.
I think you might get different results if you had much longer cables as well. You would have cables everywhere but your latency would increase and maybe things just start not being able to handle it. Maybe the frames just get forwarded from switch to switch and all is well. Combine that testing with a filled MAC table and could have absolute carnage. I'd love to see that but that's a lot of cable.
If I used 100 metre patch cables I would be adding 4800 metres to the distance. Signals travel a 5 millionths of a millisecond per metre so an extra 24,000 millions of a millisecond or .0024 ms extra. It would make a difference it's not enough to matter. I do agree that chaining 48 real, fully populated switches with devices generating traffic would give us a different outcome.
I think that a lot of the concerns about how many switch hops you can have come from much further back...there WAS a limit to how many HUBs you could daisy chain before it just plain broke. This was before switches were readily accessible or affordable. Nice test scenario, though!
great video.
but for longer connections i just learned about GPeR .
thanks
There is no technical limit, and you would probably need to get a hundred switches together for an empty test environment to have issues at all. Its just a practice that is likely to lead to issues in the real world. Like being unable to tell where a problem is being caused as its impacting everything, or a bottle neck you dont know you have. Much easier to assure performance and service a model with mostly a single hub/core. But if your aware of what you are doing and what bandwidth you'll see and have good reason then go for it.
Fiber isn't really that expensive. Especially if you're okay with generic SFPs, a cheaper brand switch, and multimode fiber. Which, to be fair, you should be okay with if your alternative is daisy chaining some cheap TP-Link switches.
On the stadium story: Running Cat6 to a switch then to another switch over and over again (somehow with power for each switch too) instead of running a single OF cable is doing my head in. :P
I'm thinking it would be less fuss to run OF instead of all the switches, especially since cable would have to be run either way.
ltt recently did a lan party and they said for them 12 was the max that after that things broke
Daisy chaining power strips is totally safe unless you pull more than like 6kW through them. If they are fused correctly you never will see anything go up in smoke.
I've seen a video here on youtube where they created a string of hundreds. There will be a voltage drop due to wire length.
It was never about latency, or even uplink capacity. It was always about the spanning tree. Change those switches to traditional spanning tree and make a loop and watch it melt down.
About 15 years ago I was trying to stress test two 10g links and used something similar. I routed the traffic back and forth across the links 15 times or so with vlans and cables crossing back and forth and was able to max out the 10g link with my laptop and iperf.
Isn't the internet just millions of switches
routers not switches this is why you see multiple hops when using something like trace route to something on the internet and no hops when going to something on your own lan. its layer 2 vs 3
no, you have routers not switches
Millions of MANAGED switches
@@ideegenialia router is just a level 3 switch lol
just take my like
Not as exciting a video as if you'd actually gone and bought a lot of switches, but I'm now a tiny bit more knowledgeable, so thank you
I definitely 'feel' like a more complex setup with devices plugged in in the chain as well as at either end would cause issues, but have never explored beyond the hub and spoke environment. I wonder if you could simulate it in GNS3?
The latency increase is most likely not even due to the switch hops, but due to baseT being relatively laggy compared to say, sfp and its variants
Good stuff.
All the Switches!
given that the internet is basically a lot of switches, partially daisy chained, sometimes with some branches, sometimes with a loop in them, it shouldn't be that surprising that you ca, in fact, daisy chain a lot of switchtes
The question is how many switches can you go before you need to insert a router
The latency is because each vlan has to receive the complete packet and forwards in software. For dumb switches theres no limit because once the packet leaves its as good as the first packet. People here are saying that in large corporate networks under heavy load they do see degredation. This is because broadcast traffic or traffic the switch doesnt know where to route gets copied to all ports and this leads to storms which then impinges on everything else. But daisy chaining them for a stadium would work to a huge scale, and its a reasonable approach, each switch is then a repeater. I think performance will degrade at large scale because theres a limit to how many macs each switch can learn before it starts forgetting the first leading to increased broadcasts. I'm going to suggest the limit is about 254 switches before you start needing to be creative. The internet is basically 4bn switches daisychained.
Does it show 48 hops? Or do switches not count as a hop?
Unless he's making them Layer 3 vlans they are not counted as hops. The max on that would be 255. The main thing you'd run into, and what Cisco is talking about is that spanning tree has issues when you go wider than 7 switches away from the root of the tree. Spanning tree is supposed to be unreliable after that.
A router is a hop. This is a single broadcast domain.
The unicast Ethernet packets are not processed but are either forwarded as is out all ports except the port received or, if the switch already knows what port the destination is downstream is connected to, forwards it out only that port.
If the frame is a multicast or broadcast Ethernet frame, it is forwarded out all ports. The frames are not decapsulated, and the IP (if the enveloped protocol is IP) TTL is not decremented.
Managed do. Unmanaged just show up as a bunch of devices on the same port. Much the same as having a bunch of VMs each with their own IP plus the host machine.
A unmanaged switch is just a dumb bridge.
I think the results would be different if the cables were longer, say 10 meters, and many different manufacturers and models of switches.
my guess is slightly over 2 billion, if you tell all of them to act as routers (so possibly using custom firmware for them), don't care about internet connectivity, and use a /32 address space for each of them
I think you've predicted my future network. 😁
I can give you the real answer, which is, until u max the cam table space of any 1 switch. switches operate in 2 modes, switch mode, which is normal operation and hub mode. hub mode you will most likely never see, its when the switch sends ALL the packets out on ALL the ports. hub mode is entered when the cam table of a switch goes OOM. the cam table stores the mac address to port number schema. this is how switches determine where data is sent from an incoming packet. it looks for the destination mac address in the cam table, finds the port and bob's your uncle. when a cam table is filled with nonsense data like from an arp flood. the switch may switch modes into hub mode, the switch can no longer determine where packets are supposed to go so it can in essence just broadcast them to the whole network. when it does this, other switchinges being in the same mode will do same, causing a packet storm, and crippling the network.
for corporate switches, cam table sizes are specified as part of the spec. bigger badder switches have huge cam tables. end point switches keep them small. preventing corporations from using small cheap switches as core switching elements.
The packet storm is only if there is a loop in the network.
@@Douglas_Gillette how often do you not see double linked switches in a corporate environment?
"hub mode" is unicast flooding. Some switches will flood, some will drop once the table is full. (some can be configured to drop unknown unicast) By disabling mac-learning, or setting the table size to zero, some switches can be turned into a hub. (but you really don't want to)
probably the switch did some optimisation and skipped ports...if you do this with physical switches then please expect a latency of 0.5 ms to 1 ms per switch..I have tried with 5 physical soho switchs
There is no chance the switch is skipping hops. Doing so would break many real world setups. What makes you think each switch adds .5ms per hop?
@@matthewdaley7535 This is my assumption without any proof -> Skipping hops means - the software inside the router optimizes the route because as far it sees start and end is the same machine, vlan is software so it can 'route' it if it can decide it is safe... | 5ms per switch -> Its 0.5ms to 1 ms per swich...so 5ms is total for 5 physical home/small office switches (TP link green gigabit) I have done it myself.
@@nid274 Stop making assumptions. These are SWITCHES not ROUTERS. Switching happens in microseconds (and there are thus that operate in nanoseconds!) The only way to "bypass hops" is through odd, proprietary technology like Cisco's Multi-Layer Switching (mls) - which tags frames with ingress/egress information. (as far as I know, that has been dead for years.)
@@jfbeam did you know L3 switches can ROUTE and SWITCH!!
@@nid274 Yes. It will route what it is CONFIGURED to route. Setting up 24 VLANs does not automatically create 24 routed interfaces. (if he did, it would add about 1ms each... the 2960 isn't a very good layer-3 device.) Does he need to post the switch configurations for you to understand it's all switching and no routing?
Boy whichever stadium didn’t do it right the first time was cutting corners.
Maybe smartphones didn't exist when the stadium was built and current management does not have the money to run fiber cabling all over. This happens all the time in the real world that we have to work with a non-ideal, non-cisco budget and just make things work.
@@mrfrenzy. who said anything about Cisco
You neglected to account for max segment length. Ethernet is rated to 100 meters per LAN segment (broadcast domain) for CDMA to be reliable.
Sure it can work but what will probably happen on chatty networks is that retransmissions will start flooding as CDMA collisions increase across the segment.
300m? I think I only get 100m cable before the signal degrades too much. Unless that wasn't what you meant?
@@xTerminatorAndy Good catch. I was thinking in feet. Fixed.
thanks for doing gods work.
It’s fun, but I don’t think it’s a valid test. The managed switch you were using only has to hold a single MAC table. It also doesn’t have to deal with ever increasing numbers of BPDUs from other switches or several other network awareness protocols that might be in use such as VLAN discovery or simply CDP other LLDP. Most importantly it doesn’t have to deal with multiple STP connections to other switches which would be vital in a large extended network so you could close the loop without making a broadcast storm. As I recall the recommended STP hop limit is only 8 hops. It would have been much more interesting to see how the background traffic climbed as you added more switches and to see at what point it swamped out host traffic.
Isn't the TTL like 255? So that must be the theoritical maximum
Switches don't reduce the TTL, routers do.
why would one have the slightest doubt that daisy chaining switches doesn't work?
Are you using a blue yeti microphone?
Yes! How can you tell?
2 because I only own 2 right now and don't have access to any others.
I bought two for this project thinking there is no way it would work across 48 hops. The switches were only $35US and the cables were $1 each. But it worked so well that I figured even 8 would not be enough to make it fail and I didnt want to end up with 8 100Mbit switches that I would just have to sell.
Switchecption
In the real world, the best senario would be a daisy chain, and then a cable back from the last switch to the first, so incase any switch in the chain dies, the network stays up, and that's maybe what that reddit user did
I could do a video to see how long spanning tree takes to work out that a link has broken.
And then spanning tree turned on one do the switches or else there will be a broadcast storm.
The answer is 2^8^2. Because you have the whole 10.x.x.x range
Seems to me all you've done is make a really long connection. You haven't emulated a real world scenario because you don't have multiple computers sending and receiving data between these daisy chained LAN hubs.
Once these LAN hubs have to start dealing with multiple traffic sources, you have something that can be tested.
I want to see it with the real deal, the cheap trash dlink switches.
now ...... that is kinda smart but stupid in the same time 🤔
This is not daisy chaining at all.