Thanks so much for this. Had been struggling to get my Mellanox cards working with the Mikrotik switch and that FEC mode change did the trick! Hadn't found that information anywhere else - very much appreciated.
Glad I was able to help! It took me a few days to figure that out. I have no idea why auto-mode isn't working through the options. It's not like there are many to try...
I ran into the same issues with iperf3 testing (not able to get the full 100Gb/s). I found in my case I had to run three separate instances of iperf3 to hit about 97.5Gb/s. Although iperf3 has a threading mode built-in, it's network threads and not CPU threads and it seems to hit the CPU threshold around 38-50Gb/s depending on the CPU you're using. It was smart to use two instances of iperf3 in the video for that reason. It's crazy how networking is hitting the limits of even our artificial benchmarking tools.
Yeah right. What also always helps is enabling Jumbo Frames and if you do a lot of SDN or hight network throughput stuff is DPDK. These switches need to be tested with something like TRex
Edit: ok nevermind. Iperf2 is much better for testing 100gbe + Were you testing a Mikrotik CRS504? If so, I'd love to see the iperf3 setting you're using, or MTU size if it's necessary to change. With my best settings, my throughput has topped out at a little over 92 Gbps on my CRS504 four processes, with two iperf threads each. Buffer length (lowercase "L") 1M iperf3 -c "$SERVER" -T s1 -p 5101 -l 1M -P 2 & iperf3 -c "$SERVER" -T s2 -p 5102 -l 1M -P 2 & iperf3 -c "$SERVER" -T s3 -p 5103 -l 1M -P 2 & iperf3 -c "$SERVER" -T s4 -p 5104 -l 1M -P 2
In the future, I would recommend trying iPerf 2 for 100 Gbps testing. iPerf 2 is still actively maintained and is unrelated to iPerf 3, so iPerf 3 is not just a newer/better replacement and they each have their strengths and weaknesses. iPerf 2 includes 100 Gbps testing in their test suit for every build, so you shouldn't need to run multiple instances.
The chip that this switch is using uses 16x25Gib lanes. 4 go to each SFP port and that is why you can do the breakout and the reason for the weird view in the interface. You can see more data on it in Mikrotik's own channel: ruclips.net/video/rE6fnmbOMD0/видео.html
Crazy find on that switch. Looks like youre testing iperf with TCP - perhaps part of why you weren't hitting line rate - I'd imagine that and/or buffering and/or traffic bursts in general; regardless - impressive! I've been binge watching your electronics channel and also do networking stuff so found this interesting also!
Nice to see the lower wattage usage vs 10 base-t ports (The whole managed switch can consume up to 60watts based on power supply rating). That would be deciding factor for my home use. Crazy how fast adding up 10w ports in electricity over a few years can justify buying a better/faster setup. I do wish it had two 10gbe ports without consuming 1 of it's 4 100gbit ports for connecting to the rest of a traditional network and proper router. But as you infer, $800 is probably outside the scope of most home networks. Note: Bell ISP in Canada is starting to roll out 8gbit up and down internet for reasonable cost vs other packages. Would be amazing to have, but the price to upgrade home network would be crazy to match.
I can't thank you enough for this video! I found it after trying to get a link up for some connetcx-4 cards for the last days and the FEC mode = fec91 was the only issue.
@@HomeSysAdmin omg Im so glad I came across your channel. this was the exact problem I had. spent way too many days ordering and waiting on cables to arrive
You know with the CAPsMAN (and also Wireless - I think) you can provision several access points in the Mikrotik ecosystem similar to like Unify does it with there WiFi stuff. Yes RouterOS is a VERY big bucket of almost everything (and then multiple kitchen sinks) of what Mikrotiks can do! 😉
hell yeah - wise choice my friend it is not that much more than 10g and has a good upgrade path from 25 to 100 - it matches up well for smb where they want a ws and a nas and 4 cluster nodes - a real time saver and network efficiency - i could see some people running a couple of these - hopefully there will be more 100g switches in this price range as 23 unfolds - mikrotik makes some good product #connectx the dots
if you set up one of these switches as as a L3 switch (router) definitely don't do it through the quick setup, you need to manually do the config and enable L3 hardware offloading, otherwise it'll try to run from the slow CPU. it can also offload NAT connections if you need to, but requires additional configuration.
I understand that at the ends of the DAC cable there is an EEPROM and it says who I am. Question - does the NIC card ports reading the eeprom compare it to the eeprom on the other end of the cable? I connected 2 DAC cables through a docking board (QSFP28 QSFP28) and can't get a connection between NIC cards I don't understand what's wrong.
@@HomeSysAdmin No its not, SFP28 backwards compatible to 10GB and even 1GB, so it can be plugged to a 10GB router SFP port. BUT if you put the breakout cable from 100GB port to a 10GB port, then all the other cables from the 100GB port also are 10GB, so totalling 40GB.
It's amazing what they are doing for the price, but their GUI's are too convoluted for my tastes. They could also stand to use terms that make better sense ... something other than fec91, etc...
I am a fan of its GUI interface, Webfig and CLI. I understand that you have many options but not all of them are used, and it depends a lot on the level of requirement.
I call it a switch because that's what it is. The name of the software installed is RouterOS, which is pretty much what all Mikrotik devices run. It has routing functionality, but does that technically make it a router? :)
the device has the RouterOS operating system, the product line is CRS, Cloud Router SWITCH, which means that it has RouterOS operating system but its most specific function is focused on switch activities
CRS504 Switch... amzn.to/3Xt0Ean (Amazon)
CRS504 Switch... ebay.us/RvWmxb (Ebay)
10GTek DAC Cables... amzn.to/3ZCynQq
Mellanox ConnectX-4 NIC... ebay.us/yT26fp
(affiliate links)
Hi, the link to the DAC cables is pointing to the switch instead of the cables.
@@BennyTygohome Ooops, it's fixed. Thanks!
Thanks so much for this. Had been struggling to get my Mellanox cards working with the Mikrotik switch and that FEC mode change did the trick! Hadn't found that information anywhere else - very much appreciated.
Glad I was able to help! It took me a few days to figure that out. I have no idea why auto-mode isn't working through the options. It's not like there are many to try...
I ran into the same issues with iperf3 testing (not able to get the full 100Gb/s). I found in my case I had to run three separate instances of iperf3 to hit about 97.5Gb/s. Although iperf3 has a threading mode built-in, it's network threads and not CPU threads and it seems to hit the CPU threshold around 38-50Gb/s depending on the CPU you're using.
It was smart to use two instances of iperf3 in the video for that reason. It's crazy how networking is hitting the limits of even our artificial benchmarking tools.
Yeah right. What also always helps is enabling Jumbo Frames and if you do a lot of SDN or hight network throughput stuff is DPDK. These switches need to be tested with something like TRex
IPERF3 is single thread IPERF has multi thread so use the old version or run Multiple instances of IPERF3
Edit: ok nevermind. Iperf2 is much better for testing 100gbe +
Were you testing a Mikrotik CRS504? If so, I'd love to see the iperf3 setting you're using, or MTU size if it's necessary to change. With my best settings, my throughput has topped out at a little over 92 Gbps on my CRS504
four processes, with two iperf threads each. Buffer length (lowercase "L") 1M
iperf3 -c "$SERVER" -T s1 -p 5101 -l 1M -P 2 &
iperf3 -c "$SERVER" -T s2 -p 5102 -l 1M -P 2 &
iperf3 -c "$SERVER" -T s3 -p 5103 -l 1M -P 2 &
iperf3 -c "$SERVER" -T s4 -p 5104 -l 1M -P 2
In the future, I would recommend trying iPerf 2 for 100 Gbps testing. iPerf 2 is still actively maintained and is unrelated to iPerf 3, so iPerf 3 is not just a newer/better replacement and they each have their strengths and weaknesses. iPerf 2 includes 100 Gbps testing in their test suit for every build, so you shouldn't need to run multiple instances.
Thanks for the tip. I did find this out along the way - rather unfortunate/confusing naming there.
The breakout to 16x 25GbE is really interesting, especially for the price.
The chip that this switch is using uses 16x25Gib lanes. 4 go to each SFP port and that is why you can do the breakout and the reason for the weird view in the interface. You can see more data on it in Mikrotik's own channel: ruclips.net/video/rE6fnmbOMD0/видео.html
Crazy find on that switch. Looks like youre testing iperf with TCP - perhaps part of why you weren't hitting line rate - I'd imagine that and/or buffering and/or traffic bursts in general; regardless - impressive! I've been binge watching your electronics channel and also do networking stuff so found this interesting also!
he is using FEC91 which is only 91% effective giving a line rate of 91 Gbps. He was fairly close to that.
Nice to see the lower wattage usage vs 10 base-t ports (The whole managed switch can consume up to 60watts based on power supply rating). That would be deciding factor for my home use. Crazy how fast adding up 10w ports in electricity over a few years can justify buying a better/faster setup. I do wish it had two 10gbe ports without consuming 1 of it's 4 100gbit ports for connecting to the rest of a traditional network and proper router. But as you infer, $800 is probably outside the scope of most home networks.
Note: Bell ISP in Canada is starting to roll out 8gbit up and down internet for reasonable cost vs other packages. Would be amazing to have, but the price to upgrade home network would be crazy to match.
I can't thank you enough for this video! I found it after trying to get a link up for some connetcx-4 cards for the last days and the FEC mode = fec91 was the only issue.
Glad I could help! It took me a few days to figure this out as well. I kept going in circles "maybe it's the cable" or "maybe I got a bad card". Nope!
@@HomeSysAdmin omg Im so glad I came across your channel. this was the exact problem I had. spent way too many days ordering and waiting on cables to arrive
In my case it was ancient firmware on my Intel e810 100Gb NICs that I was using that would fail auto- negotiation.
Cool little switch. The MPO to 4x LC break-out fiber cables are not cheap. Interesting review. Thanks for sharing
This is awesome. Thanks for the overview on setup and configuration . This is very flexible setup especially being able to use breakout cables.
I'm waiting for mine to arrive, to include it in a network of approx 12Gb to 28Gb traffic.
You know with the CAPsMAN (and also Wireless - I think) you can provision several access points in the Mikrotik ecosystem similar to like Unify does it with there WiFi stuff.
Yes RouterOS is a VERY big bucket of almost everything (and then multiple kitchen sinks) of what Mikrotiks can do! 😉
hell yeah - wise choice my friend it is not that much more than 10g and has a good upgrade path from 25 to 100 - it matches up well for smb where they want a ws and a nas and 4 cluster nodes - a real time saver and network efficiency - i could see some people running a couple of these - hopefully there will be more 100g switches in this price range as 23 unfolds - mikrotik makes some good product #connectx the dots
What happen if you just use "-P 16" souldnt this do the trick instead of starting multiple iperf3?
Iperf -P 10 for multi parallel connections
What would a 100Gbit broadcast storm look like?
I'd rather not find out lol
if you set up one of these switches as as a L3 switch (router) definitely don't do it through the quick setup, you need to manually do the config and enable L3 hardware offloading, otherwise it'll try to run from the slow CPU. it can also offload NAT connections if you need to, but requires additional configuration.
Nice Walk thru on the set up !
Thanks 🙂
you are tha BEST 2nd time you Help Me Alot
I understand that at the ends of the DAC cable there is an EEPROM and it says who I am. Question - does the NIC card ports reading the eeprom compare it to the eeprom on the other end of the cable?
I connected 2 DAC cables through a docking board (QSFP28 QSFP28) and can't get a connection between NIC cards
I don't understand what's wrong.
can you enable hardware macsec on this device? will it run at 100G or near it? have you tried MPLS?
What gave the test without iperf3 parameters?
Older iPerf2 is likely faster. I got 67Gb/sec with 4 threads, but one host was fully CPU loaded with 4 cores (3GHz).
Hello, does this switch support rdma?
If using a break out cable 100G to 4x 25G, is it possible to put the 25GB to a 10GB sfpf port, so is it backwards supported?
No, 25Gb is typically a different formfactor SFP28 from 10Gb SFP+
@@HomeSysAdmin No its not, SFP28 backwards compatible to 10GB and even 1GB, so it can be plugged to a 10GB router SFP port. BUT if you put the breakout cable from 100GB port to a 10GB port, then all the other cables from the 100GB port also are 10GB, so totalling 40GB.
@@jossushardware1158 Oh you're right, didn't know that - never used SFP28 myself. Thanks for the info!
It's amazing what they are doing for the price, but their GUI's are too convoluted for my tastes. They could also stand to use terms that make better sense ... something other than fec91, etc...
I agree regarding the GUI. I am not a fan of RouterOS.
I am a fan of its GUI interface, Webfig and CLI.
I understand that you have many options but not all of them are used, and it depends a lot on the level of requirement.
Just configure a skin for it so you just see the menus you use. Cleans it up good
Why do you keep calling this device a switch when it's OS says router?
I call it a switch because that's what it is. The name of the software installed is RouterOS, which is pretty much what all Mikrotik devices run. It has routing functionality, but does that technically make it a router? :)
the device has the RouterOS operating system, the product line is CRS, Cloud Router SWITCH, which means that it has RouterOS operating system but its most specific function is focused on switch activities
@@HomeSysAdmin I asked because it confused me is all, thank you for the explanation.
@@camilopaspi thank you for you’re explanation.
it's skipping too fast. i dont even know where to open the console
I don’t have a comment but be careful with the Apollo has a major electrical shock hazard just a FYI.
I saw that video. I was planning to review one but never had the time and kindof forgot to reply. Guess that's a good thing!
@@HomeSysAdmin As far as I know two people testing have gotten shocked. There a lot of people disappointed.
@@HomeSysAdmin Please post the video link.
@@rayk32 It's on Will Prowse's channel ruclips.net/video/zHeRBG0nOIk/видео.html
@@HomeSysAdmin thanks!
overuse of "thEE"
Huh? Overuse of what?
the / "thee" ... just too much. otherwise, nice review
Oh, didn't even realize I was doing that lol. Thanks for the feedback.
@@HomeSysAdmin you are welcome.
do you have the CRS still in use? tried MLAG with it?