Alright. This is nuts. You're killing it lately. I stop everything and listen to your videos the moment they come out! That's bad! QA was down and I was listening to you while working on it cause I wasn't going to stop fixing QA! Great content. Great presentation. very fun topics! Keep up the excellent work!
These videos you make, they remind me why I fell in love with Linux and computer tinkering over 20 years ago now. I absolutely love the way you try out things, research a lot, test various settings, and how your experiments get more optimization as you upload your videos. I have a great time watching your videos, thanks!
I’m really hoping for getting some “router boards” for the compute module soon! Having a Pi based router with something between 2 & 4 gigabit Ethernet ports would be amazing! OpenWRT already has support for the Pi and the throughout it has for VPN’s is amazing! I’m using a Pi 4 for bonding together 2 different ISP’s and I’m getting a full gigabit. But using usb Ethernet cards is a bit hacky.
This is one use case I hope to explore further once I get a few more network cards tested. There are already a couple concept boards (one in production that should be available soon!) with more than one 1 Gbps port. I would like to see one with at least one 2.5 Gbps port too!
Re your $65 10G Copper SFP: I used to work at a network equipment development company, and we used fs.com for all of our fiber and transceivers if we couldn't find what we needed quickly enough through our cheaper contacts, and they've always been really nice and cheaper than most other major online retailers. They have 10G Copper SFPs for $59, so you may can save a few bucks with your 10G upgrade!
So as to "why am I doing this?" I guess "because it's there," a common mountain climbing adage, isn't enough. You could explain it to your wife as, "this is a good example to show techniques that one goes through to diagnose hardware of any type: testing, knowing all the variables, and recompiling the kernel." Plus you're an *effective* teacher, which means you will get to more people with lower skill sets showing a hypothesis (network speed), data metrics, and conclusion based on the results, even if it doesn't agree with your hypothesis. This is PURE scientific method, right here. Which is why I watch and subscribe. :D
Re: Enabling Jumbo frames - not only the devices you want to use Jumbo frames on need to be configured to it, but all devices in the Ethernet Segment, ie all devices in the same IP Subnet need to have the same MTU or you get into 'network randomly breaks' territory.
I'm always trying to find a happy medium for the time I upload videos-surprisingly (to me!) only 30% of my video views are from North America... so I want to try to put the videos up (and do live streams) when more people can watch!
@@JeffGeerling i don't think it is not that important for videos, the main benefit of video on demand is that you can watch it whenever you like and how often you like.
I kept watching this and wanting to say, "but you need to.." and then Jeff does it. Well done Jeff. I think you're the only one pushing the CM4 to these kind of limits. MTU and jumbo frame change for sure will help the most, even with 1Gbps. It's like trying to move to a new house with a Kia vs a U-Haul. Tons more room in the frame, so fewer trips. A lot less work. Of course everything in the path has to be able to fit this figurative U-Haul (9000 byte frame). Jeff, Could you please share your iperf/iperf3 options/switches with the class?
Hey, me again, saw that you cut the kernel compilation time, that's awesome! If you want to go even lower, you can use CCACHE. The tradeoff is that it's gonna take some space on your disk. It's basically a cache of pre-compiled .o files from .c/cpp files. It's automatic and reliable. First time you're compiling with it, it's gonna take the same ammount of time as usual. Second time, you better put on your seatbelt.
Should note that you can do that easily by changing the Makefile and replacing the line github.com/raspberrypi/linux/blob/rpi-5.4.y/Makefile#L424 by CC = ccache $(CROSS_COMPILE)gcc . I also recommand that you set the cache size higher than the default 1GB (set to 15GB for kernel only should be good). I used to have over 200GB ccache to build android A LOT faster.
To give you an idea, cloned the kernel and compiled. ccache first run real 12m25.409s user 110m10.196s sys 11m7.344s ccache (clean with cache built) real 1m45.049s user 13m45.032s sys 2m7.171s Once you modify some files, only these modified files will get rebuilt. Mostly everything else will be in the cache. Side note, 1GB ccache might be more than enough for that kernel, it ended up using around 350MB on my system. I used to compile for a few devices which takes more space for the cache. Running out of cache space is not optimal + it's not reserving the space so setting it higher wouldn't be a bad idea.
Hehe... you have cracked the code! Though sometimes between steps 3-4 I put in "try the manufacturer's driver download and get disappointed that it doesn't compile on ARM, so skip to step 4 anyways".
If you often need to recompile the kernel, instead of cross compiling, what about compiling on your cluster ? That would be a very nice usecase : building a cluster of pi so that pi related dev gets faster ^^
There's a company called 10gtek based in california that makes and sells spf transceivers for a really good price. They also oem code the transceivers to a lot of popular brands. I bought an sfp fiber for my mikrotik hap ac and one for my netgear gc110 switch and they work as expected. They're on ebay and amazon. Both transceivers cost $18 total. I believe rj45 transceivers are slightly more expensive but still less than half from whatever brand you're buying from.
Hello Jeff, great video! You have became my favorite rpi chanel :) About your compiles, can't you just compile the driver as modules and modprobe those to your pi? do you have to recompile the whole kernel each time? (it has been a long time since i compiled a kernel) Also are you still using your pi zero + hq camera for shooting, or was it only for the video on the HQ cam?
Did you do a retest of the initial setup but with switch os? Might've been routeros just playing tricks on you. Sometimes those irq percentages don't really show bottlenecks.
OMG Jeff you had me laughing out loud! Thanks for another budget busting video and for sharing your production related minor mishaps. Keep up the great work. It won't be long until you get your 100k subscriber plaque. Sub count currently standing at 98.4k. Anyone who reads this comment and isn't already subscribed please do so. Let's get Jeff over the 100k mark for something positive to remember in this year we'd all like to forget.
Subscriber count now at 98.8k come on we only have 30 hours left to make 100K!! Again if you haven't already take that mouse, finger, or stylus and mash that subscribe button!!
this is really useful, recently upgraded my server to have 2.5GB ethernet but couldn't get the driver to work, pretty sure though i need the exact same driver
Another trick you can do for recompiling kernel is to create a droplet with high cpu and memory, clone and compile kernel there, then destroy the droplet. I used to do that for compiling android rom and kernels. Not only it saves the compile time, it also saves git clone time, as digitalocean's internet is obviously many times fast than mine :P You can probably automate the whole creating droplet, destroying droplet thing with scripts :D
A little suggestion for 10G transreceivers: don’t put them in adjacent ports 😉 leave a port in between for a copper tail or a fiber module, that helps the thermal distribution
Perhaps you could publish a full install of the OS with all the drivers for all the cards you have gotten working since you started the series on the CM4 integrated into it?
Haha, you know I thought about doing that exact same thing earlier today. It's a possibility. I would like to see maybe Ubuntu for Pi or some distribution focus on adding more driver support so it behaves more like a standard desktop OS where you don't have to recompile the kernel every hour or so to get missing drivers :)
@@JeffGeerling yes but we are counting on you, because you have been gracious enough to figure out what works, it would be the crown jewel to your series. Here is everything I tested that works......and here is the full install that supports them all. Thus making you a hero.
Hello Jeff, thanks for all the great content, one quick question, is there already a Raspberry Pie that comes built-in with at least two NICs? 1G speed will be enough for the time being, however, dual 10G NICs Raspberry pie will be super awesome. Thank you in advance, and happy holidays!
Good question! There is one board in development that Seeed studios will be selling soon with two 1 Gbps ports (one goes through a USB 3.0 interface): github.com/geerlingguy/raspberry-pi-pcie-devices/issues/25#issuecomment-751566018
@@JeffGeerling Oh cool, thanks for the prompt response, I will definitely check it out. Another quick question though, there is one small board (not a Raspberry Pie) built by a group called Kobol link here: kobol.io/ it is a NAS, seems to have the two 1G NICs but I am more interested in using it as a router/firewall. I own the previous generation of NAS called Helios from the same group Kobol and I won't change it for the time being, this one comes with a single 1G nic. Thanks again, let me know your thoughts on the Helios64. Looking forward to hearing your opinion.
It would be nice to plug cm4 in and it auto run at 1gb or 2GB. If there is hardware out there for cm4, they should automatically have downloads to update OS as an auto - switch for their adapter. If Raspbian / Raspberry foundation wants value in compatibility of different vendor adapters they gotta communicate well as Microsoft seems to with hardware adapter vendors. I know the foundation is working wide open. However YOU ALONE keep coming up with ideas / videos on new things for PI4 /cm4. I sit up to 2a-3A a lot of times going back over your videos in enthusiastic wonderment! I want to do them all!
With how many times you've had to recompile and add drivers for all the various knick-knacks (or should that be NIC-nacs :P), I wonder what would happen if you just recompiled with every driver on the list lol
I see a T shirt in the works for ,,, Life is really Simpler with a BIG FAT PIPE ! ... When Red shirt Jeff shows up, is the fire department and paramedics alerted ???
Came back to this for “gotchas” as I had an idea. CM4 “steam links”. Basically the CM4 on a carrier board that gives it up to 2.5Gb/s running steam OS3.0 (steam deck os) maybe a NVMe SSD or an m.2 SATA. While it would have steam os it wouldn’t be playing games directly by streaming them. Haven’t play with steam os 3.0 much yet as it’s not official, but if you can have different user logins and that login streamed the game from that player’s library. Idk. I think I am on to something, but maybe not
Hi. Amazing content. Here is an interesting experiment: Have pi have 2 x 1gig ethernet ports, configured to be LACP, and pass through the combined traffic via 3rd 2.5G ethernet dongle. Use Case - to make a 2.5G adapter for NASes, that have 2 x 1G etherner and LACP support, but no support for multi Gig dongles via USB, such as PR4100
I'm not sure why you bother with cross compiling kernel with BCM2711. I mean, I did attempt compile the whole android-rpi with 8GB pi 4. While task failed successfully (curse google closed source binaries), native kernel compile performance is decent. I haven't time it against my Ryzen 3 1200 (4c/4t) but compiling android-rpi kernel (forked from rpi) pretty decent. Thought I had to wait around 2 hors but it finished earlier. My machine uses external HDD over USB3 so I can compile everything fyi
The next step is to use PCI-E switch to connect two 2.5Gbps adapters and use dpdk or xdp to bridge them. I guess quad core Cortex-A72 could handle the load. RPi 4 as the future-proof routing platform is a quite interesting concept.
I have a 2-port Syba 2.5 Gbps card on the way-github.com/geerlingguy/raspberry-pi-pcie-devices/issues/46 ... I'll be interested to see if I can get more bytes/sec through that with both interfaces.
Great video! Do you happen to have a video on how to build a distributed mysql cluster on multi node kubernetes using raspberry pi? There are so much things to consider and I am quite lost! Thanks in advance!
That's something I intend to test soon. I want to get a few more cards individually tested before I get back into testing PCIe switches (I have two already, one more on the way).
Have you tried using distcc for any of your kernel compiles? Works pretty well on my rpi cluster with 4 pi4 8gb's and 2 pi3b+'s. It looks like it supports cross compiling as well, just be sure to exclude the master node from the hosts list when you have more than 5 or 6 nodes. Just got my feet wet with kernel compiling and only did it once on a single pi before needing to find a better way. But now with the iSCSI modules I have a k3s cluster up and running with the only storage being 2 usb ssd's on one of the pi's, good riddance micro sd cards!
I haven't tried it yet, but mostly I like trying to make the compile operation as self-contained as possible since I sometimes hop from computer to computer, and don't always have a fleet of Pis running and at the ready :)
Is it really necessary to build the whole kernel? I noticed that you enabled the driver as a module [M] so you'd only have to build that and copy to /lib/modules and stuff. But maybe I don't know what I'm talking about...
I could, I'm just lazy and just do the whole kernel and copy everything over since I have it scripted now. Rebuilds of the kernel only take a few seconds usually, and the copy takes like 20-30 seconds over the network, so it's about as quick as just doing one module separately.
Interrupt load is like 3-5% at 9000 MTU, barely a blip. I'm excited to see what the Pi will be able to do with a proper 10 Gbps card... I haven't tested DPDK or XDP yet, there are so many other cards I have stacked up yet to test first :)
Wondering how PCIe compares to the flurry of USB 3.0 to 2.5G Ethernet adapters that have come out 2020 for £30-£40 and potentially can be used on Raspberry Pi 4(00)?
So how much did throughput change in the CM4 or Pi4’s built-in Ethernet with JUST Jumbo frames enabled? Did you cover that in a previous video & I missed it? Thx.
One question I have regarding the GPU-Project: Could this enable hardware-transcoding for Plex? I have a Quadro P400 laying around and would like to make a mini-plex-server with it
2 years later, and Linux support for 2.5Gbit ethernet is still flaky. With LMDE 6 I got it working plug-and-play with 2x pcie x1 cards, but only getting about 1/2 rated speed - even with sysctl tweaks. Still, ~180MB/sec is still noticeably better than gigabit. Setting the mtu to 9000 on both sides resulted in dismal performance tho
Still working on it. So far all four GPUs I've tested lock up at some point during driver initialization, and right now I'm working on getting remote GDB debugging working so I can try to troubleshoot the drivers.
@@JeffGeerling wouldn't it be nice if we saw some of ARM based DGPU? Like a physical Mali card or Adreno card? By the way....as you said there is an x86 specific feature which is the problem for RPi gpu. How will this be addressed? I don't think x86 emulation will be possible at this low level
Excellent suggestion. @Jeff Geerling checks that out. What good is high network bandwidth if it bogs down your storage read/write speeds if you are doing a NAS.
That's a good question. It seems like until the CM4 there was little work done on making PCIe on the BCM2711 work better, but things are slowly turning around. I don't think we'll see a ton more drivers in the Pi OS kernel soon, but maybe one of the other Pi distros like Ubuntu Server could have more baked in at some point?
Yes, I either use that or Ubuntu 20.04 Server for Pi, as both of those 64-bit OSes seem to play _much_ better with device drivers and software that's typically tested only on ARM64 and not ARMv7 / 32-bit.
They sorta are. I've seen 15k packets on at least one switch manufacturer - might've been either Extreme or Arista).. and IPv6 has jumbograms, which can be up to 4GiB in size. :o
consider bonding or bridging links and get 3.5 with some sort of link agg lacp? i tried bonding myself and only got a couple more mb/s - there is a way to do it maybe try a usb2.5gbe and bridge or bond 2 2.5 - also consider usb nvme raid - pls test a few of these in quest for optimized pi - it may go on forever as a work in progress, it is all good - the next version of pi is liable to be a barn burner if they do it rite - more pci lanes is what it needs - and usb4 would be fantastic - sep controllers maybe and 16gb option would be fantastic also - that is really all it needs - when you uncork the io you can realize the full speed of the chip #bottlenecks #gluster node #io computer
props to red shirt jeff - the reenactment was so realistic (realtekish) i almost felt like i was there smelling the crab magic smoke myself #grammy award for extemporaneous industrial smoke and magic
this was great content and will be of interest to anyone running gluster on the pi - nvme and fast network really help performance - the pi folks should be getting a die shrink with the next iteration - that should help with power and they will get ipc gains - more pci lanes and more ram would be game changers - they should just do one more tier, they can respin for usb4 and ddr5 #odroid #sbc mkt metrics #rockpi #better easier clustering #instead of soldered on ram just have slots - you could have a pi with 64gb ram #nuc competitor the sbc space is liable to be intense in the next few years and a space to watch #outperform #format the butt or root partition as it is otherwise known #sonic on arm #L2/3 potatoe routers/switch #connectx and opnsense for your 100g network - what you really need to connect ws/devbox/server and nas and 2 vmservers (4 cards 2 dualport) if you nat chain or 8 cards if you want to diy switch which would give you big advantages - short fiber links are cheap - i think 100gbe solutions will be popular for smb going fwd - 10g is just scratching the surface - the newer nvme raid arrays match up well with the 100g and nvme scales great for db ops - refurb mkt is hot but the fast network with fast nvme to go with it to compliment are the big upsells along with gpu passthrough (this would enable pi to do compute) #techmikeny bargain hunter #reseller ready #resellers panel #don't break debian #no warranty #america's volume opnsense dealer #underclocked potato routers
2.5gb switches r still way too expensive and its been a year since u made the vid, if i wanna buy a pcie card on ebay from my own country its 111$ and china i can get 4 cards for 100$ then wait 2 months to get here with all this bullshit going on. Anyway i scored a 16 sata expansion card PCIe 1x slot for 45$ which is a bargain so I cant wait to test that out when it gets here and eventually get a raspi to test it on.
cwn you belive it happend the same thing to me today but with inte nuc. iv been iperfing all day long, imstall, tweek, bios, bla bla .. ethtool did the job
Oreck XL. Had a beastly Dyson for years, but I realized it's actually harder to vacuum with the Dyson because it sucks so hard (in a good way)-you kind of had to pull it everywhere. An Oreck seems to be pretty good between 'sucks things up' and 'doesn't suck itself into medium-pile-carpet so much that it's hard to use'. I do miss all the attachments of the Dyson though. And it did last me about 10 years before part of the plastic on the head cracked off.
@@TomOConnor-BlobOpera Haha, yes, back in the day. It's a darn reliable vacuum though. The reason I bought one is I know two people who bought them in the 90s and they're still going strong. Bagged vacuums are annoying, sure, but at least I can just vacuum a ton over the year and replace the bag once every now and then instead of emptying out the tornado chamber (that shows the nasty buildup) every time I vacuum :D
I'm just going to come out and say it, "Life simpler with a big, fat pipe" is, imo, the best quote of 2020.
Except you misquoted, you forgot the is.
@@robbyandrews223 lol yep, I do that sometimes.
I guess Im kinda randomly asking but do anybody know of a good website to stream newly released movies online ?
Jeff was absolutely on one in this video 🤣
Alright. This is nuts. You're killing it lately. I stop everything and listen to your videos the moment they come out! That's bad! QA was down and I was listening to you while working on it cause I wasn't going to stop fixing QA! Great content. Great presentation. very fun topics! Keep up the excellent work!
QA can wait. Production is all that matters :D
Have you tried setting the MTU to "Over 9000!"
Heh... did you see the bloopers at the end? ;)
@@JeffGeerling bloopy bloop
@@JeffGeerling Yes, I did XD hence the comment. (in case people don't watch the bloopers; they should)
@@JeffGeerling The magic smoke dramatic reenactment by Red Shirt Jeff got me busting out laughing before the bloopers.
The Linux kernel has consumed Jeff. Soon he will become one with it and directly manipulate it via telepathy.
He's not running Gentoo, impossible
bwhahahaha
As a Gentoo user I don't get idea why he even talk about it :D
These videos you make, they remind me why I fell in love with Linux and computer tinkering over 20 years ago now. I absolutely love the way you try out things, research a lot, test various settings, and how your experiments get more optimization as you upload your videos. I have a great time watching your videos, thanks!
I have a list longer than a few page scrolls on Trello of the things I'd _like_ to try, too :D
@@JeffGeerling looking forward to watching literally every second of it. 🐧
I come to this channel for the cutting edge tech. I stay for the dramatic recreations of vacuuming accidents.
I understand 10 percent of the tech and still enjoyed this! Yay ithernet!
I’d have set the MTU to 9001 just so I could say it’s OVER 9000!!!
Should have blamed the vacuum accident on red shirt jeff 😋
"Life simpler with a big, fat pipe" - the wife says you have a point!
I'll buy that T-shirt ..
I’m really hoping for getting some “router boards” for the compute module soon!
Having a Pi based router with something between 2 & 4 gigabit Ethernet ports would be amazing!
OpenWRT already has support for the Pi and the throughout it has for VPN’s is amazing! I’m using a Pi 4 for bonding together 2 different ISP’s and I’m getting a full gigabit. But using usb Ethernet cards is a bit hacky.
This is one use case I hope to explore further once I get a few more network cards tested. There are already a couple concept boards (one in production that should be available soon!) with more than one 1 Gbps port. I would like to see one with at least one 2.5 Gbps port too!
Re your $65 10G Copper SFP: I used to work at a network equipment development company, and we used fs.com for all of our fiber and transceivers if we couldn't find what we needed quickly enough through our cheaper contacts, and they've always been really nice and cheaper than most other major online retailers. They have 10G Copper SFPs for $59, so you may can save a few bucks with your 10G upgrade!
Keep it up! This sort of testing is really helping the Raspberry Pi become even more useful, just keep Red-Shirt Jeff away!
This takes me back to setting up Linux servers before drivers were good.
So as to "why am I doing this?" I guess "because it's there," a common mountain climbing adage, isn't enough. You could explain it to your wife as, "this is a good example to show techniques that one goes through to diagnose hardware of any type: testing, knowing all the variables, and recompiling the kernel." Plus you're an *effective* teacher, which means you will get to more people with lower skill sets showing a hypothesis (network speed), data metrics, and conclusion based on the results, even if it doesn't agree with your hypothesis. This is PURE scientific method, right here. Which is why I watch and subscribe. :D
Re: Enabling Jumbo frames - not only the devices you want to use Jumbo frames on need to be configured to it, but all devices in the Ethernet Segment, ie all devices in the same IP Subnet need to have the same MTU or you get into 'network randomly breaks' territory.
another CM4 pcie video? niceeeee
I like how this video released at 11.30pm in my country. Keep up the good work Jeff!
I'm always trying to find a happy medium for the time I upload videos-surprisingly (to me!) only 30% of my video views are from North America... so I want to try to put the videos up (and do live streams) when more people can watch!
@@JeffGeerling i don't think it is not that important for videos, the main benefit of video on demand is that you can watch it whenever you like and how often you like.
I love the style of your videos Jeff, keep it up!
I kept watching this and wanting to say, "but you need to.." and then Jeff does it.
Well done Jeff. I think you're the only one pushing the CM4 to these kind of limits.
MTU and jumbo frame change for sure will help the most, even with 1Gbps. It's like trying to move to a new house with a Kia vs a U-Haul. Tons more room in the frame, so fewer trips. A lot less work.
Of course everything in the path has to be able to fit this figurative U-Haul (9000 byte frame).
Jeff, Could you please share your iperf/iperf3 options/switches with the class?
Looks like you used iperf3. I'm wondering if you forced no fragmentation? Like what swtiches on iperf3 did you use?
I would have never thought of doing the overclocking.
Hey, me again, saw that you cut the kernel compilation time, that's awesome! If you want to go even lower, you can use CCACHE. The tradeoff is that it's gonna take some space on your disk. It's basically a cache of pre-compiled .o files from .c/cpp files. It's automatic and reliable. First time you're compiling with it, it's gonna take the same ammount of time as usual. Second time, you better put on your seatbelt.
Should note that you can do that easily by changing the Makefile and replacing the line github.com/raspberrypi/linux/blob/rpi-5.4.y/Makefile#L424 by CC = ccache $(CROSS_COMPILE)gcc . I also recommand that you set the cache size higher than the default 1GB (set to 15GB for kernel only should be good). I used to have over 200GB ccache to build android A LOT faster.
To give you an idea, cloned the kernel and compiled.
ccache first run
real 12m25.409s
user 110m10.196s
sys 11m7.344s
ccache (clean with cache built)
real 1m45.049s
user 13m45.032s
sys 2m7.171s
Once you modify some files, only these modified files will get rebuilt. Mostly everything else will be in the cache.
Side note, 1GB ccache might be more than enough for that kernel, it ended up using around 350MB on my system. I used to compile for a few devices which takes more space for the cache. Running out of cache space is not optimal + it's not reserving the space so setting it higher wouldn't be a bad idea.
an average video from jeff:
1.get a pci-e device
2.plug it in
3.lspci and dmsg
4.recompile the Linux kernel
5.excitement
Hehe... you have cracked the code! Though sometimes between steps 3-4 I put in "try the manufacturer's driver download and get disappointed that it doesn't compile on ARM, so skip to step 4 anyways".
If you often need to recompile the kernel, instead of cross compiling, what about compiling on your cluster ? That would be a very nice usecase : building a cluster of pi so that pi related dev gets faster ^^
You're a great educator.
Damn, I love these kind of videos so much
There's a company called 10gtek based in california that makes and sells spf transceivers for a really good price. They also oem code the transceivers to a lot of popular brands. I bought an sfp fiber for my mikrotik hap ac and one for my netgear gc110 switch and they work as expected. They're on ebay and amazon. Both transceivers cost $18 total. I believe rj45 transceivers are slightly more expensive but still less than half from whatever brand you're buying from.
You should have blamed the vacuum accident on Red Shirt Jeff. It sounds like something he would do.
"But hold on a second"... Linus! I have PTSD from that phrase :D
Oh no, it's leaking. I gotta stop watching LTT to let my brain reset!
For faster compile times, check out ccache or sccache :)
Wooo Hoooo Nice work dude.
Hello Jeff, great video! You have became my favorite rpi chanel :)
About your compiles, can't you just compile the driver as modules and modprobe those to your pi? do you have to recompile the whole kernel each time? (it has been a long time since i compiled a kernel)
Also are you still using your pi zero + hq camera for shooting, or was it only for the video on the HQ cam?
Did you do a retest of the initial setup but with switch os? Might've been routeros just playing tricks on you. Sometimes those irq percentages don't really show bottlenecks.
OMG Jeff you had me laughing out loud! Thanks for another budget busting video and for sharing your production related minor mishaps. Keep up the great work. It won't be long until you get your 100k subscriber plaque. Sub count currently standing at 98.4k. Anyone who reads this comment and isn't already subscribed please do so. Let's get Jeff over the 100k mark for something positive to remember in this year we'd all like to forget.
Subscriber count now at 98.8k come on we only have 30 hours left to make 100K!! Again if you haven't already take that mouse, finger, or stylus and mash that subscribe button!!
this is really useful, recently upgraded my server to have 2.5GB ethernet but couldn't get the driver to work, pretty sure though i need the exact same driver
Another trick you can do for recompiling kernel is to create a droplet with high cpu and memory, clone and compile kernel there, then destroy the droplet. I used to do that for compiling android rom and kernels. Not only it saves the compile time, it also saves git clone time, as digitalocean's internet is obviously many times fast than mine :P
You can probably automate the whole creating droplet, destroying droplet thing with scripts :D
Ansible!
All of those network cards after the raids, are you trying to make a compact NAS ? Like for a backup NAS ?
A little suggestion for 10G transreceivers: don’t put them in adjacent ports 😉 leave a port in between for a copper tail or a fiber module, that helps the thermal distribution
Perhaps you could publish a full install of the OS with all the drivers for all the cards you have gotten working since you started the series on the CM4 integrated into it?
Haha, you know I thought about doing that exact same thing earlier today. It's a possibility.
I would like to see maybe Ubuntu for Pi or some distribution focus on adding more driver support so it behaves more like a standard desktop OS where you don't have to recompile the kernel every hour or so to get missing drivers :)
@@JeffGeerling yes but we are counting on you, because you have been gracious enough to figure out what works, it would be the crown jewel to your series. Here is everything I tested that works......and here is the full install that supports them all. Thus making you a hero.
Hello Jeff, thanks for all the great content, one quick question, is there already a Raspberry Pie that comes built-in with at least two NICs? 1G speed will be enough for the time being, however, dual 10G NICs Raspberry pie will be super awesome. Thank you in advance, and happy holidays!
Good question! There is one board in development that Seeed studios will be selling soon with two 1 Gbps ports (one goes through a USB 3.0 interface): github.com/geerlingguy/raspberry-pi-pcie-devices/issues/25#issuecomment-751566018
Think you'd hit the limit of the PCIe interface at that point, a single gen 3 lane tops out at 8Gb
@@JeffGeerling Oh cool, thanks for the prompt response, I will definitely check it out. Another quick question though, there is one small board (not a Raspberry Pie) built by a group called Kobol link here: kobol.io/ it is a NAS, seems to have the two 1G NICs but I am more interested in using it as a router/firewall. I own the previous generation of NAS called Helios from the same group Kobol and I won't change it for the time being, this one comes with a single 1G nic. Thanks again, let me know your thoughts on the Helios64. Looking forward to hearing your opinion.
It would be nice to plug cm4 in and it auto run at 1gb or 2GB. If there is hardware out there for cm4, they should automatically have downloads to update OS as an auto - switch for their adapter. If Raspbian / Raspberry foundation wants value in compatibility of different vendor adapters they gotta communicate well as Microsoft seems to with hardware adapter vendors.
I know the foundation is working wide open. However YOU ALONE keep coming up with ideas / videos on new things for PI4 /cm4.
I sit up to 2a-3A a lot of times going back over your videos in enthusiastic wonderment!
I want to do them all!
With how many times you've had to recompile and add drivers for all the various knick-knacks (or should that be NIC-nacs :P), I wonder what would happen if you just recompiled with every driver on the list lol
Don't think I haven't contemplated doing this!
holy boogered caviar Robin!! U didn't get played by the Joker this time....
cheap trick to try later to bypass the CM4 PCIe bandwidth is to compress the data on the fly.
I see a T shirt in the works for ,,, Life is really Simpler with a BIG FAT PIPE ! ... When Red shirt Jeff shows up, is the fire department and paramedics alerted ???
Now that you have LightSpeed, whats the next project for the 2021?
Came back to this for “gotchas” as I had an idea. CM4 “steam links”. Basically the CM4 on a carrier board that gives it up to 2.5Gb/s running steam OS3.0 (steam deck os) maybe a NVMe SSD or an m.2 SATA. While it would have steam os it wouldn’t be playing games directly by streaming them.
Haven’t play with steam os 3.0 much yet as it’s not official, but if you can have different user logins and that login streamed the game from that player’s library.
Idk. I think I am on to something, but maybe not
May sound stupid, but I don't know so here it goes. Why not install all the drivers when compiling linux? Would that make it easier in the long run?
his bet bits are the outros. Hilarius.
Good stuff!
Hi. Amazing content.
Here is an interesting experiment:
Have pi have 2 x 1gig ethernet ports, configured to be LACP, and pass through the combined traffic via 3rd 2.5G ethernet dongle.
Use Case - to make a 2.5G adapter for NASes, that have 2 x 1G etherner and LACP support, but no support for multi Gig dongles via USB, such as PR4100
Hint: There's cheaper modules that do the same multi-bitrate stuff as that Mikrotik
I'm not sure why you bother with cross compiling kernel with BCM2711. I mean, I did attempt compile the whole android-rpi with 8GB pi 4.
While task failed successfully (curse google closed source binaries), native kernel compile performance is decent. I haven't time it against my Ryzen 3 1200 (4c/4t) but compiling android-rpi kernel (forked from rpi) pretty decent. Thought I had to wait around 2 hors but it finished earlier. My machine uses external HDD over USB3 so I can compile everything fyi
The next step is to use PCI-E switch to connect two 2.5Gbps adapters and use dpdk or xdp to bridge them. I guess quad core Cortex-A72 could handle the load. RPi 4 as the future-proof routing platform is a quite interesting concept.
I have a 2-port Syba 2.5 Gbps card on the way-github.com/geerlingguy/raspberry-pi-pcie-devices/issues/46 ... I'll be interested to see if I can get more bytes/sec through that with both interfaces.
@@JeffGeerling BTW, have you tried to use RSS? You have to enable ENABLE_RSS_SUPPORT in driver's Makefile for that.
With RSS and multiple TCP or UDP streams (-P option in iperf client), you will be able to use multiple cores to process network load.
Great video!
Do you happen to have a video on how to build a distributed mysql cluster on multi node kubernetes using raspberry pi?
There are so much things to consider and I am quite lost!
Thanks in advance!
I would love to see a video about getting zfs working on the pi compute module.
Thanks in advance
Seconded.
Now it’s interesting how much of that bandwidth is usable if you have an SSD connected over USB3 and do a copy over NFS.
That's something I intend to test soon. I want to get a few more cards individually tested before I get back into testing PCIe switches (I have two already, one more on the way).
Why did you dial back the MTU 9000 Jumbo frames when you did the overclock?
Why not both??
because the card WAS maxed out... so he wanted to see if he could get max speed just with the OC and not jumbo frames
More re-enactments by the Red Shirt Jeff Players thanks.
i was just wondering if you would have recompiled the kernel again .... ahahahah
I prefer netperf instead of iperf. Also the interrupts shouldn’t be that terrible because of napi?
Have you tried using distcc for any of your kernel compiles? Works pretty well on my rpi cluster with 4 pi4 8gb's and 2 pi3b+'s. It looks like it supports cross compiling as well, just be sure to exclude the master node from the hosts list when you have more than 5 or 6 nodes. Just got my feet wet with kernel compiling and only did it once on a single pi before needing to find a better way. But now with the iSCSI modules I have a k3s cluster up and running with the only storage being 2 usb ssd's on one of the pi's, good riddance micro sd cards!
I haven't tried it yet, but mostly I like trying to make the compile operation as self-contained as possible since I sometimes hop from computer to computer, and don't always have a fleet of Pis running and at the ready :)
Referencing an AVE vih-jeh-oh?! +100,000,000 internet points good sir.
Tappy tap taaaaap right on that like button, sir!
“Oh I need the cards”, Jeff I thought my computer died but I forgot to put in the cpu.
Is it really necessary to build the whole kernel? I noticed that you enabled the driver as a module [M] so you'd only have to build that and copy to /lib/modules and stuff.
But maybe I don't know what I'm talking about...
I could, I'm just lazy and just do the whole kernel and copy everything over since I have it scripted now.
Rebuilds of the kernel only take a few seconds usually, and the copy takes like 20-30 seconds over the network, so it's about as quick as just doing one module separately.
Would you consider trying TCP over USB 3.1 aka RNDIS?
Any hope of DPDK or XDP? Also what’s the cpu interrupt load at 9000MTU?
Interrupt load is like 3-5% at 9000 MTU, barely a blip.
I'm excited to see what the Pi will be able to do with a proper 10 Gbps card...
I haven't tested DPDK or XDP yet, there are so many other cards I have stacked up yet to test first :)
Wondering how PCIe compares to the flurry of USB 3.0 to 2.5G Ethernet adapters that have come out 2020 for £30-£40 and potentially can be used on Raspberry Pi 4(00)?
I'd be curious to see a multigig usb ethernet device throughput on a pi4. I'm curious if they would be supported on the pi.
Could have just used a 2.5gb usb adapter for the mac and connected the pi with it with a utp cable. Ugreen or sabrent 2.5 adapters work great
So how much did throughput change in the CM4 or Pi4’s built-in Ethernet with JUST Jumbo frames enabled? Did you cover that in a previous video & I missed it? Thx.
I touched on it in a video on the Intel I340-T4 - the built-in interface goes from ~940 Mbps to ~980 Mbps.
One question I have regarding the GPU-Project: Could this enable hardware-transcoding for Plex? I have a Quadro P400 laying around and would like to make a mini-plex-server with it
2 years later, and Linux support for 2.5Gbit ethernet is still flaky. With LMDE 6 I got it working plug-and-play with 2x pcie x1 cards, but only getting about 1/2 rated speed - even with sysctl tweaks. Still, ~180MB/sec is still noticeably better than gigabit. Setting the mtu to 9000 on both sides resulted in dismal performance tho
Hey man! Love your videos. But is there an update for eGPU for rpi cm4?
Still working on it. So far all four GPUs I've tested lock up at some point during driver initialization, and right now I'm working on getting remote GDB debugging working so I can try to troubleshoot the drivers.
@@JeffGeerling wouldn't it be nice if we saw some of ARM based DGPU? Like a physical Mali card or Adreno card? By the way....as you said there is an x86 specific feature which is the problem for RPi gpu. How will this be addressed? I don't think x86 emulation will be possible at this low level
I would buy a T-shirt that says “So I recompiled the Raspberry Pi Kernel”
Why not compile kernel with all as modules ;)
Red Shirt Jeff agrees, he wants ALL THE MODULES.
Now... if only there were a way to also get an 802.11ad card in there, along with a second CM4, to get a Wi-Fi extender with a 60 GHz backhaul.
If using a USB 2.5 GbE Adapter alongside a USB SATA SSD (500 MB/s read/write) how would that effect the iperf3 bandwidth tests?
Excellent suggestion. @Jeff Geerling checks that out. What good is high network bandwidth if it bogs down your storage read/write speeds if you are doing a NAS.
You only have one PCIe lane, so I don't really see a way to do this.
So TNSR has an ARM Version wonder if it'd be possible to do line rate 2.5gbit using DPDK or VPP via Fi.Do or TNSR?
Hehe I can just imagine my Mom coming to this video and reading this comment, and being like "yeah I don't understand any of that" :D
What u do , is so cool..
Why not connect raspberry pi to mac directly, set static ip on both, then run iperf3?
When will the pi support all this stuff out of the box?
That's a good question. It seems like until the CM4 there was little work done on making PCIe on the BCM2711 work better, but things are slowly turning around.
I don't think we'll see a ton more drivers in the Pi OS kernel soon, but maybe one of the other Pi distros like Ubuntu Server could have more baked in at some point?
Is it possible to include all the drivers so that you dont need to compile the kernal every time?
Can you please try to use pi as a only layer 3 switch ? And test it on mac again with two virtual interface.
I kinda want to do something like this for a pi nas. 1 gigabit per second just doesn't seem fast enough for a nas.
Are you running the Raspberry Pi OS 64bit beta?
Yes, I either use that or Ubuntu 20.04 Server for Pi, as both of those 64-bit OSes seem to play _much_ better with device drivers and software that's typically tested only on ARM64 and not ARMv7 / 32-bit.
What about the GPU's?
How was the CPU usage with jumbo frames without overclock?
Very minimal (like 3-5%). It really only hits a wall when it runs into the queueing.
Can't wait for Super Jumbo frames to be a thing
They sorta are. I've seen 15k packets on at least one switch manufacturer - might've been either Extreme or Arista).. and IPv6 has jumbograms, which can be up to 4GiB in size. :o
@@TomOConnor-BlobOpera jumbogram. Pretty sure that was a thing in the 80s - to do with a larger lady arriving at your party and dancing...
🤔
Can you do a video on Linux kernel compiling
Planning on it!
Keep doing pi videos...
Did you figure out why using the riser smoked the first card?
Not yet. I plan on taking a longer look soon, and maybe testing things out more directly.
consider bonding or bridging links and get 3.5 with some sort of link agg lacp? i tried bonding myself and only got a couple more mb/s - there is a way to do it maybe try a usb2.5gbe and bridge or bond 2 2.5 - also consider usb nvme raid - pls test a few of these in quest for optimized pi - it may go on forever as a work in progress, it is all good - the next version of pi is liable to be a barn burner if they do it rite - more pci lanes is what it needs - and usb4 would be fantastic - sep controllers maybe and 16gb option would be fantastic also - that is really all it needs - when you uncork the io you can realize the full speed of the chip #bottlenecks #gluster node #io computer
props to red shirt jeff - the reenactment was so realistic (realtekish) i almost felt like i was there smelling the crab magic smoke myself #grammy award for extemporaneous industrial smoke and magic
this was great content and will be of interest to anyone running gluster on the pi - nvme and fast network really help performance - the pi folks should be getting a die shrink with the next iteration - that should help with power and they will get ipc gains - more pci lanes and more ram would be game changers - they should just do one more tier, they can respin for usb4 and ddr5 #odroid #sbc mkt metrics #rockpi #better easier clustering #instead of soldered on ram just have slots - you could have a pi with 64gb ram #nuc competitor the sbc space is liable to be intense in the next few years and a space to watch #outperform #format the butt or root partition as it is otherwise known #sonic on arm #L2/3 potatoe routers/switch #connectx and opnsense for your 100g network - what you really need to connect ws/devbox/server and nas and 2 vmservers (4 cards 2 dualport) if you nat chain or 8 cards if you want to diy switch which would give you big advantages - short fiber links are cheap - i think 100gbe solutions will be popular for smb going fwd - 10g is just scratching the surface - the newer nvme raid arrays match up well with the 100g and nvme scales great for db ops - refurb mkt is hot but the fast network with fast nvme to go with it to compliment are the big upsells along with gpu passthrough (this would enable pi to do compute) #techmikeny bargain hunter #reseller ready #resellers panel #don't break debian #no warranty #america's volume opnsense dealer #underclocked potato routers
That smell...
There are two smells you will never forget, that of a backed up sewage pipe, and that of electronical magic smoke.
2.5gb switches r still way too expensive and its been a year since u made the vid, if i wanna buy a pcie card on ebay from my own country its 111$ and china i can get 4 cards for 100$ then wait 2 months to get here with all this bullshit going on. Anyway i scored a 16 sata expansion card PCIe 1x slot for 45$ which is a bargain so I cant wait to test that out when it gets here and eventually get a raspi to test it on.
smb direct on Raspberry pi?
Love your show, you remind me of someone, hint: The Chad is going down
cwn you belive it happend the same thing to me today but with inte nuc. iv been iperfing all day long, imstall, tweek, bios, bla bla .. ethtool did the job
Stupid off-topic comment, what make is that vacuum cleaner?
Oreck XL. Had a beastly Dyson for years, but I realized it's actually harder to vacuum with the Dyson because it sucks so hard (in a good way)-you kind of had to pull it everywhere.
An Oreck seems to be pretty good between 'sucks things up' and 'doesn't suck itself into medium-pile-carpet so much that it's hard to use'.
I do miss all the attachments of the Dyson though. And it did last me about 10 years before part of the plastic on the head cracked off.
@@JeffGeerling was Oreck the one who used to advertise by holding a bowling ball above their ceo held only in place by the vacuum power?
@@TomOConnor-BlobOpera Haha, yes, back in the day. It's a darn reliable vacuum though. The reason I bought one is I know two people who bought them in the 90s and they're still going strong.
Bagged vacuums are annoying, sure, but at least I can just vacuum a ton over the year and replace the bag once every now and then instead of emptying out the tornado chamber (that shows the nasty buildup) every time I vacuum :D
Why don't you just build one kernel with a whole lot of modules for the cards you're going to use?
Jeff already stated he was considering it.
The CM4 is one of the ultimate tinkering platforms ever built. Change my mind.