Real science tries to disprove it's self, that's how you can tell a real science from fake science. Global warming caused my man and CO2 is fake science. It's all cherry picked data there is no antithesis,
I can't believe you're even trying this! There's a reason people buy giant PC cases and mount all their hardware inside. This is so dumb, why am I even watching this video?!
@@KienTran a 1GB Pi3 is more than enough to run basic storage ZFS without dedup or anything too fancy. We run 8TB+ pools 1GB x86 boxes and have done for a couple of years without a single bit lost (sample size 300+ units, 6PB total) The (old Atom 330) boards in use are constrained by 1 slot of RAM so the max installable is 2GB - but we cannot renew or upgrade the hardware as they are all deployed in very remote locations and run 24/7 True, they suffer from bottlenecking on intense random workloads but the boxes are fine for typical NAS use. We also run 20 or more RasPi 3's on ZoL doing the same...
Brilliant, nothing like cobbling together a raid array from parts. I've been waiting for something like this for the Raspberry PI for ages,thanks for doing this.
Jeff, I love your videos! You have genuine content rather than rehashed stuff that 50 other people have published. I've done a lot of the same things you have...using mdadm, hacking microcontrollers, and building Pi projects. So its great to see you being so organized and sharing your findings with the world. And I love your bloopers at the end of the videos. Thanks!
Very interesting video. I would love to see a video on compiling the Pi Kernel for stuff like this. Thanks for being so informative. People like you are what make the Raspberry Pi community so wonderful.
I have a list of some of the 'accessory' videos that I would like to make, including how to compile / cross-compile the kernel, how to flash the eMMC on the CM4, etc. - it's just there isn't enough time in my day to get to all this :( Maybe someday...
@@JeffGeerling Not so, hence the mistake as was £100 part in larger sale. But SAS card once I get around to it seemed good idea and did like the specs heat tollerance wise as had idea's of loft NAS. Still probably go with old x86 pc I have, though the RPi or arm based avenue would be my choice. WHY oh why don't they make some SAS to USB3 adapter that are fair price, think found one and that was half a grand :(. Certainly a market here for a fair priced one.
My favourite part about this video is how honest it was about all the little "gotchas" that arose during the project. To me, that's where the value is --- learning about where things can go sideways, as opposed to watching a video where everything goes according to the plan. Thanks for sharing!
I used to think you were really smart. Maybe even "perfect". Then I started thinking: you always see bloopers for your videos but Red Shirt Jeff never seems to have bloopers. Maybe Red Shirt Jeff is smarter and never makes mistakes? I wonder if someday he will start making your videos for you!! HAHAHA
I am not a storage nutcase anymore, but used to collect old hard drives and this project brings back memories! Makes me want to see if my giant old external SCSI Mac drive still spins up!
Its not a firmware issue, it is a PCIe lane issues. Some of those cards just won't work on 1 PCIe lane. Deal with the LSI cards all the time with Dell and also other RAID cards and i have tired them in 1x to 16x adapters and they just won't work. I offered him some cards that I do know work though. Would love to see what the Pi 4 can do now!
I would love for there to be a MiniITX form factor "motherboard" that you could slot a pi 4 computer module into instead of a CPU and RAM, would be interesting/cool and handy for some projects Alas I do not have the schematic/board layout knowledge to do something like that myself
I've considered working on this, if for no other reason than I've had a few close calls with the crazy contraptions I've been putting together on my desk.
This is such an interesting concept for me, I really look forward to the next video. I have a couple projects I dont need a large server for so this is really intriguing for me, thank you!
Omg Jeff keep me hanging. You know you can go over 10 min? 😁. Anyway how about you take this one further and actually mount the raspberry pi in a 24 drive server rack and have the Pi run all 24 drives. Heck I watched this video I probably will watch that one too! Thanks for your excellent content. One more suggestion. Make a video on how you compile the pi kernel. I am sure there are viewers that have never seen it done you could talk about all the options where to get the source code and all that lovely stuff.
these videos take so long to produce; I was originally planning on everything in one video, but I ended up having too much to be able to get it in one video this week. plus the streaming series starts Wed!
So that’s what SAS is. With regards the cards not working, could it be due to needing a minimum amount of lanes, eg x4? Have you tried in an x86 to confirm they work? :)
Jeff, I think you maybe have just been unlucky with the LSI cards, some of my bits and pices have just arrived and I plugged in an LSI 9211-8i I had laying around in the drawer and got this: pi@raspberrypi:~ $ sudo lspci -v 00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0, IRQ 55 Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 00000000-00000fff Memory behind bridge: f8000000-f80fffff Capabilities: [48] Power Management version 3 Capabilities: [ac] Express Root Port (Slot-), MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=028 Capabilities: [240] L1 PM Substates Kernel driver in use: pcieport 01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02) Subsystem: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] Flags: fast devsel I/O ports at [disabled] Memory at 6000c0000 (64-bit, non-prefetchable) [disabled] [size=16K] Memory at 600080000 (64-bit, non-prefetchable) [disabled] [size=256K] [virtual] Expansion ROM at 600000000 [disabled] [size=512K] Capabilities: [50] Power Management version 3 Capabilities: [68] Express Endpoint, MSI 00 Capabilities: [d0] Vital Product Data Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [c0] MSI-X: Enable- Count=15 Masked- Capabilities: [100] Advanced Error Reporting Capabilities: [138] Power Budgeting Capabilities: [150] Single Root I/O Virtualization (SR-IOV) Capabilities: [190] Alternative Routing-ID Interpretation (ARI) I'll let you know how I get on, I'm assuming I have to compile the LSI drivers into the kernel now.
Nice! Can you post your findings to this GitHub issue? pipci.jeffgeerling.com/cards_storage/ibm-servraid-br10i-lsi-sas3082e-r-sas-raid.html (or start a new one for that card in particular)
@@JeffGeerling You've actually motivated me to order a RockPro64, this device has a PCIEx4 potentially giving it much more potential: www.pine64.org/rockpro64/ it also seems someone has the LSI 9211-4i working on it: wiki.pine64.org/wiki/ROCKPro64_Hardware_Accessory_Compatibility#PCIe_devices I am going to have a look through their forums and see if I can find anything helpful that could help with the Pi.
BTW. You can boot Rasberry Pi using UEFI, and load something like generic Debian, Ubuntu or Centos images and kernels. They are usually for arm64 servers that have UEFI (often called ServerReady in ARM speak, now ARM SystemReady SR), but with RPi4 firmware from pftf repo, it just works. These kernel usually have almost all modules compiled in as modules, so no need to recompile the kernel. Usually you will have access to all same modules as on x86 kernel on these distros (minus some x86 specific stuff, like some motherboard sensors, some ISA sounds drivers, etc), but almost all PCI / PCIe stuff (including hard drive controllers, raid cards, network cards, wifi, audio and video related cards, GPUs, etc) will be there. Anyway great video. I love experiments like that myself. As of the not-working RAID card, it is probably related to bios on these cards. They often need to initialize at boot, and their bios is often x86 only. Not all of them tho. Some of them can be initialized by linux drivers and firmware loaded by Linux. More investigation is required definitively. Also you do not need explain or apologize for your ideas, that some will consider silly. There is plenty of uses for a setup like this, and even if there is none, it is still fun, and you can learn many things from it.
I did this but with a rockpro64, an LSI 9211-4i HBA, 4x SAS 4TB drives, all in a 1U server chassis with hot swap bays. The rockpro64 has a 4x PCIE port and it all works together nicely running on Armbian OS and openZFS.
For those thinking "go with a server": Server fans have a tendency to sound like vacuum cleaners. I have a HP Proliant that sounds like 3 carpet shampooers and that's normal for the model, not fan malfunction. Using 32 sticks of RAM, 4 CPU's, 8 HDD's, 11 expansion cards, and 2 power supplies, it can generate too much heat for a quiet set of fans. HP Proliant DL585 G2 server, BTW.
I am hoping with the rise in popularity ARM that more quiet and power efficient servers become available. Right now there aren't many choices still for ARM servers for sale. In the meantime, Supermicro makes some embedded servers that run on Intel Atom with passive cooling. I used to have one for my home lab but it had the infamous C2000 clock signal death and doesn't boot anymore. The newest generation of Atom processors (C3000) do not have this defect AFAIK. There are also the EPYC 3000 and Xeon-D choices in the embedded options but I don't know how those are in heat and noise.
Hi Jeff, interesting video. I wanted some network Pi based storage that I could store my files on and stream videos / music via Samba. I opted for OMV, 4x SSD, 4xUSB3 to SSD cables, a vertical USB3 powered hub, a single metal vertical SSD enclosure to store the 4xSSDs all screwed in. All told less than 10w electricity consumption for both Pi 4 2GB and USB hub together. I definitely did not want a fan running 24x365 for both power consumption and noise reasons. Everything stores on a small bookcase shelf. Sure SSDs are more expensive but I am happy with my 3.2 TB of network storage that I can upgrade at any time as SSDs become cheaper. Your choice of kit will kill mine anyday and I look forward to your next video.
Hey, if it works, it works! Just make sure you have backups (rclone to Amazon Glacier is my basic weapon of choice these days). My primary NAS that I use for video storage right now is a janky old Mac mini with four 2TB SSDs that I paid a few arms and legs for last year (used to be a couple giant HDDs but those things were noisy and slow to spin up).
Buying a 3 pack so you have a single spare??? You need to buy in bulk and order 20 extras, you know you are going to need them ! Great video as always I just did this same project but on my R730xd so it will support 44 drives. The lsi controllers with the firmware mod are fantastic. 10tb sas drives are 99 bucks now!
Hi Jeff. What command you use to mesure the performance of your raid? I'm using a RockPro64 just like you're using. I want to compare with the RaspberryPi 4 to check if there's any difference. Thanks.
Hi Jeff, could you do a video on setting up and experimenting with 2 pi cameras and the CM4 / CM4 breakout board? It was an absolute naus to get working on the CM3.
Great Video. Love the bloopers... "4 SATA cards?" You seem to be the most knowledgeable person out there on the Pi and Linux. I appreciate your input immensely. I'd love to have you make a video on how you can use TRIM support on a m.2 SSD. SATA and/or NVME with the Pi 4. And how to setup and run RAID1 for redundant data integrity. Is this possible? Pretty please... Can't wait for the next video adventure. Thanks!
I'll be talking RAID 0, RAID 1, and RAID 10 (and why I don't particularly trust RAID 5/6/zRAID with parity) in the next video :) And I am far from the most knowledgeable person on Pi + Linux! I just document everything I learn, whereas many people don't seem to like putting all the stuff they learn out on the Internet :)
@@JeffGeerling Wow Jeff, really looking forward to your RAID video with baited breath! Can you please touch on the TRIM aspect as well? I think this would really touch on the reliability aspect about using SSD's running on linux partitions.
@@AntiqueRadioandTV For TRIM, check out this blog post: www.jeffgeerling.com/blog/2020/enabling-trim-on-external-ssd-on-raspberry-pi - I won't be covering it in my next video, but likely at some point in the future.
@@farhanyousaf5616 haha! If you're near San Jose in California, Excess Solution is where I got them for $10 apiece. They're a massive Walmart-sized electronics surplus shop so it might take a bit of searching down aisles to find them but they still had some last time I was there. Unfortunately they are closing soon thanks to Amazon buying the property (after Google shut down Weirdstuff, and then the oldest one Halted closing more recently).
I have, and it works great with a floppy power adapter-the problem I have is that the floppy connector is a pain to disconnect (compared to the barrel plug). So for a more permanent install, I would wire up a power switch for the PSU and use the 4-pin floppy connection on the CM4 IO Board. But for day-to-day testing, when I need to pull power often, I like the barrel plug separate from the bulky PSU.
The PCI slot for RPI is just what I have been waiting for, for a "one node - one disk" CEPH storage, though just having a SATA port directly on the PI would be so much easier.
First of all thanks for your great vidz and output, also rock on with your hacking pi's/it's not meant but that's why I do it attitude. Second, SAS != RAID RAID != SAS SAS != SATA Meaning SAS is not equal to RAID or the other way around. Maybe as I'm so familiar with legacy/modern enterprise HW stuff, I wanted to point that out for others (as you've probably have a good crasp on it already). SAS (serial-attached-scsi) is modern serial SCSI as old legacy SCSI was parallel. SCSI is a protocol as is (P)ATA or (S)ATA. SAS controller can be either HBA without RAID or it might have RAID functionality. Even though majority probably are also raid controllers. Also not all SAS controllers talk ATA, there were few first gen SAS controllers which didn't have the support. Either it's embedded to the same SAS controller or the card also needs an ATA chip (this i'm unaware of but more likely it's the same). It's cheap to embed SATA but less so with SCSI. That's why majority of SAS cards support SATA, but not the other way around. Also they do not support both protocols simultaneously as I am aware of as I haven't worked with the newest stuff...
How did you compile your kernel with Ubuntu , last time I checked (and long since gave up) Ubuntu doesn't give directions for custom compiling the kernel (of course I stand to be corrected) anyway would love if you could share your direction sources, other distros like debian, no problem!
Yes (usually) you might have to block off the smbus pin with kapton tape when you have a broken bios on your motherboard that acts randomly when a raid card is plugged in
This is interesting... The Pi doesn't POST like a normal x86 box so how would it load the option ROM and provide the POST configuration options to set up RAID?
4 года назад
Is a good idea to use SSDs instead for lower power usage?
Hey, @jeffgeerling, on a side note (Your SAS video is why I follow, your why not? attitude is awesome) could you do a video on your environment that you use for re-compiling the kernels? I use a Ryzen 7 1700/ 32GB / 10 TB / 4 1GBit Ethernet Ubuntu VirtualBox Host running many dedicated VMs for dev use, including a Pi-Gen machine (for building custom RPiOS images. Cross-compiling is something I'd consider if I had a good example to follow...
I promise I'll try to get to a video on the process (especially as I'm starting to tweak it for better performance / easier work on the overall process). Right now the basic environment I have set up is documented here: github.com/geerlingguy/raspberry-pi-pcie-devices/tree/master/extras/cross-compile
Pi OS currently only looks for eMMC (if present on Compute Modules), microSD, and USB. I'm hopeful they'll make it so it looks for NVMe or SATA storage if any is attached.
@@JeffGeerling You can boot Pi from ANYTHING addressable if you configure boot partition on SD Card properly. Pi boots from SD card which points that system is in /dev/sdX
"I had to rig up [a janky-ass jumper cable bridging power connections]"... "Now that I have a much more professional looking rig" 8-D Congrats! In your next video please highlight where you keep your fire extinguisher :)
You said Dell LSI card's? To let them link up you need to disable a connection to the pci-e connector. Because the firmware searches for a dell mobo :( "The trick is just to physically disable the SMBus signal. It is composed of just two pins B5 (SMCLK, SMBus clock) and B6 (SMDAT, SMBus data). These two pins need to be covered by tape or nail polish. On the top side of the card, they are the 5th and 6th PCIe pins from the left."
Would REALLY like to see the SAS cards work on the Pi. Sad they didn't. Been struggling with SAS drives on a sever build recently and need to find an alternative way to interface with them.
Interesting to see the initial plan was for hardware raid. Due to the more limited CPU resources on a pi? I've heard warnings against hw raid, mainly SMART monitoring being "broken" by the card hiding it, and when the card dies your disks are useless unless you have spare card of the same make/model.
It's definitely important to have your hardware raid monitoring properly configured. A lot of people kind of skimp on the configuration side of things and then, yeah, it gets hidden when certain things in the array fail until it's too late, it's 2 a.m. Monday morning and your entire system is going to need rebuilding.
@@JeffGeerling true, the only important goal I see is "can it be done" I found an adapter to plug an eMMC module into the SD card slot. Not fast but super cool for swapping out projects and storing them. Love your channel!
Could it have been due to the SAS raid card trying to run it's BIOS on startup i.e. to setup the drives? As it's for AMD64 it wouldn't have been able to run on the Arm? There are some which run in HBA mode(i.e. don't handle the raid but let the OS have direct access to the disks, I've got one in my DL380e. Perhaps then it would then allow the PI to see the drives? In essence it would be the same as the card you ended up with but with the ability to add more than 4 drives in the future.
They sell power/reset switches for these psus. I wonder if LED PSUs would work (probably not, as they don't have both 12 and 5V outputs; though a 12V to 5V buck converter is pretty cheap).
Does the card actually need more than a 1x link? Also, for the PSU, how about connecting that to the pi? You could either have that power the pi (and thus not need any other cables), and/or have the pi drive the PS_ON connecter low to switch it on.
I actually considered this, but then after testing it with a floppy power connector (which worked fine to power on the Pi), I realized how much more annoying it is to *unplug* the floppy connector than a barrel plug. For a more permanent installation I may stick with the floppy connector (especially if I wire up a PSU power switch somewhere). But for testing the barrel plug is a lot easier.
@@JeffGeerling Glad I could help get red shirt more fun. Did you have any thoughts on if the original card needed more than a 1x pcie slot? I have been looking to replace my current NAS with a more customisable one, and that card looks a lot nicer than a bunch of wires.
I love where Jeff's head is with this. Here's hoping a future Pi will include at least one modern PCI Express slot. I have no idea about this, but are there any older HBA or RAID cards that might fit into the slot natively? What I want to see is a mass market of low-cost, low-power, elegant, Pi-based devices...including NAS. I'm not much of a tinkerer. I want something that just works well and looks presentable with minimal fuss. I know there are pre-built solutions out there, and I have a Synology NAS myself, but I want to drive prices further down. My Pi 3 B+ really let me down when I tried to use it as a RAID5 NAS. Even if it had worked reliably, it was a rat's nest of cables.
A lot of SAS or RAID cards need to be configured when booting up a pc. Could that be causing an issue? Maybe you can find a Linux until to configure them? Also, for the power supply you could use a GPIO pin on the Pi to power it on once the pinks booted?
-Thanks, wonderful content .... !!! -So, my dream is to assemble much cheaper and DIY a FreeNAS with raspberry and HDs. -Now with this working perfectly we can go to the Freenas staff and demand that we have an image for Raspberry ... !!! -With OpenMediaVault it was already possible, but now with these advances you can have a Freenas (I want the pluggins of frenas running). -You are evolving exactly as I needed the project, thank you ... !!!
mdadm is pretty sweet. My main NAS died and I moved the array to a backup machine. I have hot swappable drives and I put them in one at a time while tail-ing syslog. I watched each of the 9 drives get detected and when the last one went in I saw a msg from mdadm that said "hey I see an array here, you're all set!" Super painless and I didn't lose my 56TB of data!
I wonder if the issue has something to do with it running on a non x86 system, I'm pretty sure the raid cards have specific ROMs on them for the architecture
![Seungmin "그렇게, 천천히, 우리(As we are)" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/kAzmhLHePqU/mqdefault.jpg)
Love it when people can show their own mistakes without being ashamed.
That is why I like your trailers.
Well he is adorably nerdy 🤣
Mistakes are learning opportunities.
Real science tries to disprove it's self, that's how you can tell a real science from fake science. Global warming caused my man and CO2 is fake science. It's all cherry picked data there is no antithesis,
Yay! My new favorite channel. Another tinkering fool, just like me.
It's a good thing we don't shame him
I'm laughing with a dell R900 under my bed :))
But I can't even hear myself laughing tho...
😂
This made me laugh xD
HP DL380 G6 for me, but I feel your pain brother 🙏😭
I assume you use it to warm up the bed?
@@Vanplay92 and a good chunk of the house
I can't believe you're even trying this!
There's a reason people buy giant PC cases and mount all their hardware inside.
This is so dumb, why am I even watching this video?!
xd
:D
when you leave a dumb comment on your own video just to see how people will react to it
Wat
Maybe make a custom storage case for this kit, imagine the nice Pi board on top of it all?
ZFS or it didn't happen :)
This guy
You, I like you
Can you install OpenZFS on raspian?
Lol there’s no way ZFS is going to run on such constraints....at least not usably anyway
@@KienTran a 1GB Pi3 is more than enough to run basic storage ZFS without dedup or anything too fancy.
We run 8TB+ pools 1GB x86 boxes and have done for a couple of years without a single bit lost (sample size 300+ units, 6PB total)
The (old Atom 330) boards in use are constrained by 1 slot of RAM so the max installable is 2GB - but we cannot renew or upgrade the hardware as they are all deployed in very remote locations and run 24/7
True, they suffer from bottlenecking on intense random workloads but the boxes are fine for typical NAS use.
We also run 20 or more RasPi 3's on ZoL doing the same...
Brilliant, nothing like cobbling together a raid array from parts. I've been waiting for something like this for the Raspberry PI for ages,thanks for doing this.
Jeff, I love your videos! You have genuine content rather than rehashed stuff that 50 other people have published. I've done a lot of the same things you have...using mdadm, hacking microcontrollers, and building Pi projects. So its great to see you being so organized and sharing your findings with the world. And I love your bloopers at the end of the videos. Thanks!
Very interesting video. I would love to see a video on compiling the Pi Kernel for stuff like this. Thanks for being so informative. People like you are what make the Raspberry Pi community so wonderful.
I have a list of some of the 'accessory' videos that I would like to make, including how to compile / cross-compile the kernel, how to flash the eMMC on the CM4, etc. - it's just there isn't enough time in my day to get to all this :(
Maybe someday...
Me too!!! 🧐
Having a 6TB SAS drive purchased by mistake a year ago still dusting, I was pondering this the other day, THIS is epicly timed Sir.
That's a pricey-and highly performant-mistake!
@@JeffGeerling Not so, hence the mistake as was £100 part in larger sale. But SAS card once I get around to it seemed good idea and did like the specs heat tollerance wise as had idea's of loft NAS. Still probably go with old x86 pc I have, though the RPi or arm based avenue would be my choice.
WHY oh why don't they make some SAS to USB3 adapter that are fair price, think found one and that was half a grand :(.
Certainly a market here for a fair priced one.
@@paulgray1318 Haha, true, I was digging around to find any way to get SAS without a SAS card and I found that adapter too, like $700!
My favourite part about this video is how honest it was about all the little "gotchas" that arose during the project. To me, that's where the value is --- learning about where things can go sideways, as opposed to watching a video where everything goes according to the plan. Thanks for sharing!
Things rarely go according to plan around these parts!
I used to think you were really smart. Maybe even "perfect". Then I started thinking: you always see bloopers for your videos but Red Shirt Jeff never seems to have bloopers. Maybe Red Shirt Jeff is smarter and never makes mistakes? I wonder if someday he will start making your videos for you!! HAHAHA
Your videos are getting better, seems like you are having fun making them.
Love your nerd talk....it has quickly shown me how much I have missed in the few years I have been off grid
7:36 Scary one - for a moment, I thought it read "SMR"
r/datahoarder leaking!
@@JeffGeerling , get the tape!
Or move out the way, Tape Master needs room.
"...four SATA cards. Four SATA cards?" Dat face. Hilarious. Excellent work, sir, much enjoyment and appreciation of your work being had here.
You always do the most RAIDICULOUS projects with your Raspberry Pis!
Oh you...
@@JeffGeerling I try... and am trying. Very trying.
off topic,the Silverstone CP11 SATA cables are really nice and easy to manage, but yeah, SAS working would have been nicer
Ooh nice, I'd never seen those before.
@@JeffGeerling they're slightly more pricey than standard SATA cables, but as long as I have to keep using SATA cables, I'll be using CP-11s
Yea I had to get some as they were the only ones that would fit under my gpu, the price is "special".
I am a storage nutcase and I love that these crazy Raspi videos are a gateway drug to talk about other topics, such as SAS/SATA RAID/HBA cards.
I am not a storage nutcase anymore, but used to collect old hard drives and this project brings back memories!
Makes me want to see if my giant old external SCSI Mac drive still spins up!
You must read my mind Jeff! Every time I think of a project, within a day or two, you release a video on it!
Thank you for doing this! I was expecting to see SAS on RPi. :D
It could still happen... might just have two dud cards, don't have a PC to test them in right now :/
From memory, I think some Dell RAID controllers are tied to a Dell RAID key which sits on the motherboard. I think I have one on my Desk somewhere.
00:49 - 🍌 for scale
I wonder what RAID type you’ll create. I’m hoping 1+0.
RAID 0 is so fast, and so stripey.
But anything besides RAID 10 and it feels like I should be playing the clip "Do you feel lucky, punk?"
@@JeffGeerling what about raid 5? it would overload the PI cpu during write though, or would it?
RAID-Z-something, because on anything else your data is just on an ephemeral /tmp with /dev/random appended in occasionally
Linus Tech Tips should hire this man and throw some money at a regular Raspberry Pi show.
This is what I've been long waiting for to make on Raspi. Gonna save this to my video list. Thank you Jeff.
Your joyous demeanor and terrible humor is delightful. Thank you for doing ridiculous stuff like this and sharing it in such a pleasant way
Jeff, you are the best!
The video is so ironic and fascinating at the same time :)
This is something I have been waiting for a loooong time! Now we can build Linux NAS drive with pico PSU.
Nice video Jeff. Well done. can't wait to see the rest.
I wonder if you could flash the LSI cards into HBA mode and use them that way
Its not a firmware issue, it is a PCIe lane issues. Some of those cards just won't work on 1 PCIe lane. Deal with the LSI cards all the time with Dell and also other RAID cards and i have tired them in 1x to 16x adapters and they just won't work. I offered him some cards that I do know work though. Would love to see what the Pi 4 can do now!
@@drtweak87 ah right, that makes sense. I have also tried pcie risers with adaptec and lsi cards, with mixed results
I would love for there to be a MiniITX form factor "motherboard" that you could slot a pi 4 computer module into instead of a CPU and RAM, would be interesting/cool and handy for some projects
Alas I do not have the schematic/board layout knowledge to do something like that myself
I've considered working on this, if for no other reason than I've had a few close calls with the crazy contraptions I've been putting together on my desk.
Turing pi 2 Will do this no?
Also check out github.com/geerlingguy/raspberry-pi-pcie-devices/issues/19
I think Level 1 Tech did a B550M motherboard recently that looked good for something like this.
Sounds like you want a 4x stack of Pi 400's with duck tape around them
Out of curiosity, what was the cost of this build?
Great videos Jeff, keep them coming!
This is such an interesting concept for me, I really look forward to the next video. I have a couple projects I dont need a large server for so this is really intriguing for me, thank you!
Why can I only give you one thumbs up? You deserve more!
I gave you a thumbs up, so now there's two.
Omg Jeff keep me hanging. You know you can go over 10 min? 😁. Anyway how about you take this one further and actually mount the raspberry pi in a 24 drive server rack and have the Pi run all 24 drives. Heck I watched this video I probably will watch that one too! Thanks for your excellent content.
One more suggestion. Make a video on how you compile the pi kernel. I am sure there are viewers that have never seen it done you could talk about all the options where to get the source code and all that lovely stuff.
these videos take so long to produce; I was originally planning on everything in one video, but I ended up having too much to be able to get it in one video this week. plus the streaming series starts Wed!
Don't you have to flash those cards in IT mode to get them working without a traditional bios?
Jeff, eres un fenómeno y un fuera de serie. Buen video.
So that’s what SAS is.
With regards the cards not working, could it be due to needing a minimum amount of lanes, eg x4?
Have you tried in an x86 to confirm they work? :)
You’re such a boss man! Love this channel.
Need 10 hour loop of that HDD ASMR now... or maybe just put a server under my bed.
@Pașca Alexandru has you covered!
Underbed heating?
Jeff, I think you maybe have just been unlucky with the LSI cards, some of my bits and pices have just arrived and I plugged in an LSI 9211-8i I had laying around in the drawer and got this:
pi@raspberrypi:~ $ sudo lspci -v
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 55
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: f8000000-f80fffff
Capabilities: [48] Power Management version 3
Capabilities: [ac] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=028
Capabilities: [240] L1 PM Substates
Kernel driver in use: pcieport
01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)
Subsystem: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
Flags: fast devsel
I/O ports at [disabled]
Memory at 6000c0000 (64-bit, non-prefetchable) [disabled] [size=16K]
Memory at 600080000 (64-bit, non-prefetchable) [disabled] [size=256K]
[virtual] Expansion ROM at 600000000 [disabled] [size=512K]
Capabilities: [50] Power Management version 3
Capabilities: [68] Express Endpoint, MSI 00
Capabilities: [d0] Vital Product Data
Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [c0] MSI-X: Enable- Count=15 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [138] Power Budgeting
Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
Capabilities: [190] Alternative Routing-ID Interpretation (ARI)
I'll let you know how I get on, I'm assuming I have to compile the LSI drivers into the kernel now.
PS, this device will be in IT mode already from its days in a ZFS server.
Nice! Can you post your findings to this GitHub issue? pipci.jeffgeerling.com/cards_storage/ibm-servraid-br10i-lsi-sas3082e-r-sas-raid.html (or start a new one for that card in particular)
@@JeffGeerling You've actually motivated me to order a RockPro64, this device has a PCIEx4 potentially giving it much more potential: www.pine64.org/rockpro64/ it also seems someone has the LSI 9211-4i working on it: wiki.pine64.org/wiki/ROCKPro64_Hardware_Accessory_Compatibility#PCIe_devices I am going to have a look through their forums and see if I can find anything helpful that could help with the Pi.
I literally laughed out loud when you said what I was thinking, “...but wait, there’s still time left in this video...” lol
can i ask you a question?
where do you get the money to buy everything you showcased every single video?
BTW. You can boot Rasberry Pi using UEFI, and load something like generic Debian, Ubuntu or Centos images and kernels. They are usually for arm64 servers that have UEFI (often called ServerReady in ARM speak, now ARM SystemReady SR), but with RPi4 firmware from pftf repo, it just works. These kernel usually have almost all modules compiled in as modules, so no need to recompile the kernel. Usually you will have access to all same modules as on x86 kernel on these distros (minus some x86 specific stuff, like some motherboard sensors, some ISA sounds drivers, etc), but almost all PCI / PCIe stuff (including hard drive controllers, raid cards, network cards, wifi, audio and video related cards, GPUs, etc) will be there.
Anyway great video. I love experiments like that myself. As of the not-working RAID card, it is probably related to bios on these cards. They often need to initialize at boot, and their bios is often x86 only. Not all of them tho. Some of them can be initialized by linux drivers and firmware loaded by Linux. More investigation is required definitively.
Also you do not need explain or apologize for your ideas, that some will consider silly. There is plenty of uses for a setup like this, and even if there is none, it is still fun, and you can learn many things from it.
I did this but with a rockpro64, an LSI 9211-4i HBA, 4x SAS 4TB drives, all in a 1U server chassis with hot swap bays. The rockpro64 has a 4x PCIE port and it all works together nicely running on Armbian OS and openZFS.
For those thinking "go with a server": Server fans have a tendency to sound like vacuum cleaners. I have a HP Proliant that sounds like 3 carpet shampooers and that's normal for the model, not fan malfunction. Using 32 sticks of RAM, 4 CPU's, 8 HDD's, 11 expansion cards, and 2 power supplies, it can generate too much heat for a quiet set of fans.
HP Proliant DL585 G2 server, BTW.
I am hoping with the rise in popularity ARM that more quiet and power efficient servers become available. Right now there aren't many choices still for ARM servers for sale.
In the meantime, Supermicro makes some embedded servers that run on Intel Atom with passive cooling. I used to have one for my home lab but it had the infamous C2000 clock signal death and doesn't boot anymore. The newest generation of Atom processors (C3000) do not have this defect AFAIK. There are also the EPYC 3000 and Xeon-D choices in the embedded options but I don't know how those are in heat and noise.
yes servers are loud but I'm not going to wait days to resilver a drive on a pi when it takes minutes on my server.
I'm only guessing, but most SAS cards have bios for management, could it be that it's required to run for it to boot the card?
That could definitely be the case :(
Just a quick tip, I’ve noticed that the performance is much better when using XFS for the disks instead of EXT4
better yet ZFS
Hi Jeff, interesting video. I wanted some network Pi based storage that I could store my files on and stream videos / music via Samba. I opted for OMV, 4x SSD, 4xUSB3 to SSD cables, a vertical USB3 powered hub, a single metal vertical SSD enclosure to store the 4xSSDs all screwed in. All told less than 10w electricity consumption for both Pi 4 2GB and USB hub together. I definitely did not want a fan running 24x365 for both power consumption and noise reasons. Everything stores on a small bookcase shelf. Sure SSDs are more expensive but I am happy with my 3.2 TB of network storage that I can upgrade at any time as SSDs become cheaper. Your choice of kit will kill mine anyday and I look forward to your next video.
Hey, if it works, it works! Just make sure you have backups (rclone to Amazon Glacier is my basic weapon of choice these days).
My primary NAS that I use for video storage right now is a janky old Mac mini with four 2TB SSDs that I paid a few arms and legs for last year (used to be a couple giant HDDs but those things were noisy and slow to spin up).
Thanks Jeff! Totally agree about backups.
Buying a 3 pack so you have a single spare??? You need to buy in bulk and order 20 extras, you know you are going to need them ! Great video as always I just did this same project but on my R730xd so it will support 44 drives. The lsi controllers with the firmware mod are fantastic. 10tb sas drives are 99 bucks now!
Hi Jeff. What command you use to mesure the performance of your raid? I'm using a RockPro64 just like you're using. I want to compare with the RaspberryPi 4 to check if there's any difference. Thanks.
That would be, "Don't count your chickens before they are hatched." It's easy to count the eggs, as eggs are eggs! :-)
Hi Jeff, could you do a video on setting up and experimenting with 2 pi cameras and the CM4 / CM4 breakout board? It was an absolute naus to get working on the CM3.
I wonder if you can use the poe hat and the usb c power port to power the pi at the same time for devices with high power draw
Interesting RPi exploration.
With the "To Be Continued" ending, might this have been better titled with a "Part 1" ?
Maybe so; I'm not used to splitting things up, but due to time constraints this week I had to!
Geekworm pi accessories reviews?
What terminal did you use?
Built-in Terminal app on my Mac. I have a custom theme and prompt though: github.com/geerlingguy/dotfiles
@@JeffGeerling thank you
This man does the Flying Spaghetti Monster's work, naysayers be damned!
This is so awesome. I was wondering about this very thing 😀
Great Video. Love the bloopers... "4 SATA cards?"
You seem to be the most knowledgeable person out there on the Pi and Linux. I appreciate your input immensely.
I'd love to have you make a video on how you can use TRIM support on a m.2 SSD. SATA and/or NVME with the Pi 4. And how to setup and run RAID1 for redundant data integrity. Is this possible? Pretty please...
Can't wait for the next video adventure.
Thanks!
I'll be talking RAID 0, RAID 1, and RAID 10 (and why I don't particularly trust RAID 5/6/zRAID with parity) in the next video :)
And I am far from the most knowledgeable person on Pi + Linux! I just document everything I learn, whereas many people don't seem to like putting all the stuff they learn out on the Internet :)
@@JeffGeerling Wow Jeff, really looking forward to your RAID video with baited breath! Can you please touch on the TRIM aspect as well? I think this would really touch on the reliability aspect about using SSD's running on linux partitions.
@@AntiqueRadioandTV For TRIM, check out this blog post: www.jeffgeerling.com/blog/2020/enabling-trim-on-external-ssd-on-raspberry-pi - I won't be covering it in my next video, but likely at some point in the future.
Great, I have to try!
Would not building the marvell sata controller as seeprate module work? Does one really need to build whole kernel for raspberrypi?
Cause it would, but the guy is a noob !
aw, no 15k sas drive? looking forward to the next iteration of "why not!"
Been wanting to do something like this for some time. Have a bunch of Addonics 5-drive SAS/SATA enclosures and loads of various SAS controllers.
If you want to donate any... I'd love to help out. :)
@@farhanyousaf5616 haha! If you're near San Jose in California, Excess Solution is where I got them for $10 apiece. They're a massive Walmart-sized electronics surplus shop so it might take a bit of searching down aisles to find them but they still had some last time I was there. Unfortunately they are closing soon thanks to Amazon buying the property (after Google shut down Weirdstuff, and then the oldest one Halted closing more recently).
@@rbus I've been to Weird Stuff. Was so sad to see them shutdown... :(
Have you considered using the 3.3v standby on the psu to power the pi?
I have, and it works great with a floppy power adapter-the problem I have is that the floppy connector is a pain to disconnect (compared to the barrel plug). So for a more permanent install, I would wire up a power switch for the PSU and use the 4-pin floppy connection on the CM4 IO Board. But for day-to-day testing, when I need to pull power often, I like the barrel plug separate from the bulky PSU.
Are you referencing banano coin ?)
Are those harddrives SAS or SATA?
The PCI slot for RPI is just what I have been waiting for, for a "one node - one disk" CEPH storage, though just having a SATA port directly on the PI would be so much easier.
Jeff, most LSI cards do require PCIe x8 or they won’t even boot. In some rare cases x4 does work but it depends on the board.
First of all thanks for your great vidz and output, also rock on with your hacking pi's/it's not meant but that's why I do it attitude.
Second,
SAS != RAID
RAID != SAS
SAS != SATA
Meaning SAS is not equal to RAID or the other way around.
Maybe as I'm so familiar with legacy/modern enterprise HW stuff, I wanted to point that out for others (as you've probably have a good crasp on it already).
SAS (serial-attached-scsi) is modern serial SCSI as old legacy SCSI was parallel. SCSI is a protocol as is (P)ATA or (S)ATA. SAS controller can be either HBA without RAID or it might have RAID functionality. Even though majority probably are also raid controllers.
Also not all SAS controllers talk ATA, there were few first gen SAS controllers which didn't have the support. Either it's embedded to the same SAS controller or the card also needs an ATA chip (this i'm unaware of but more likely it's the same). It's cheap to embed SATA but less so with SCSI. That's why majority of SAS cards support SATA, but not the other way around. Also they do not support both protocols simultaneously as I am aware of as I haven't worked with the newest stuff...
How did you compile your kernel with Ubuntu , last time I checked (and long since gave up) Ubuntu doesn't give directions for custom compiling the kernel (of course I stand to be corrected) anyway would love if you could share your direction sources, other distros like debian, no problem!
Bro, You are already breaking every limits of raspberry pi....
Love your work bro...❤️
Great video (as always). But, Do the SAS cards work in a "normal" pc?
Yes (usually) you might have to block off the smbus pin with kapton tape when you have a broken bios on your motherboard that acts randomly when a raid card is plugged in
Nice video!
Can I use a SAS port on the motherboard to plug 4 SATA HDD with a cable adaptor??
This is interesting... The Pi doesn't POST like a normal x86 box so how would it load the option ROM and provide the POST configuration options to set up RAID?
Is a good idea to use SSDs instead for lower power usage?
Great video!
Small tip, if you don't mind: look at Icydock products to connect all those drives
They have some neat (but slightly pricey) gear!
Hey, @jeffgeerling, on a side note (Your SAS video is why I follow, your why not? attitude is awesome) could you do a video on your environment that you use for re-compiling the kernels? I use a Ryzen 7 1700/ 32GB / 10 TB / 4 1GBit Ethernet Ubuntu VirtualBox Host running many dedicated VMs for dev use, including a Pi-Gen machine (for building custom RPiOS images. Cross-compiling is something I'd consider if I had a good example to follow...
I promise I'll try to get to a video on the process (especially as I'm starting to tweak it for better performance / easier work on the overall process). Right now the basic environment I have set up is documented here: github.com/geerlingguy/raspberry-pi-pcie-devices/tree/master/extras/cross-compile
Can you boot of these drives? Or can the Pi only boot from the SD card?
Pi OS currently only looks for eMMC (if present on Compute Modules), microSD, and USB.
I'm hopeful they'll make it so it looks for NVMe or SATA storage if any is attached.
@@JeffGeerling You can boot Pi from ANYTHING addressable if you configure boot partition on SD Card properly.
Pi boots from SD card which points that system is in /dev/sdX
Can the SAS card work on a Jetson Xavier?
"I had to rig up [a janky-ass jumper cable bridging power connections]"... "Now that I have a much more professional looking rig" 8-D Congrats! In your next video please highlight where you keep your fire extinguisher :)
I bought a NAS and SAS hard drive but unable connect it to SATA interface is there a way?
You said Dell LSI card's? To let them link up you need to disable a connection to the pci-e connector. Because the firmware searches for a dell mobo :( "The trick is just to physically disable the SMBus signal. It is composed of just two pins B5 (SMCLK, SMBus clock) and B6 (SMDAT, SMBus data). These two pins need to be covered by tape or nail polish. On the top side of the card, they are the 5th and 6th PCIe pins from the left."
ZFS on a RPi?
Would REALLY like to see the SAS cards work on the Pi. Sad they didn't. Been struggling with SAS drives on a sever build recently and need to find an alternative way to interface with them.
Have you found a good way to power both the HDDs and CM4?
I'm experimenting with a few different options currently. Another video will be coming soon with details!
@@JeffGeerling awesome, I love the idea of setting up the CM4 as a small NAS
Interesting to see the initial plan was for hardware raid. Due to the more limited CPU resources on a pi?
I've heard warnings against hw raid, mainly SMART monitoring being "broken" by the card hiding it, and when the card dies your disks are useless unless you have spare card of the same make/model.
It's definitely important to have your hardware raid monitoring properly configured. A lot of people kind of skimp on the configuration side of things and then, yeah, it gets hidden when certain things in the array fail until it's too late, it's 2 a.m. Monday morning and your entire system is going to need rebuilding.
Don't count your chickens before there are hatched, not eggs.
Oh haha, I completely missed that-I even remember reading what I wrote and thinking something was wrong there... Nice.
they are , not there are ;-)
@@jyvben1520 Thank you, reason 14 why not to giggle while typing.
@@gregoryturner1505 well you had fun, so all is well !
Eat your cake and have it, too
Did you find out why the link was down on the LSI card?
For redundancy and backup, but not speed, right?
It depends on how many drives you have and what your goals are.
@@JeffGeerling true, the only important goal I see is "can it be done"
I found an adapter to plug an eMMC module into the SD card slot. Not fast but super cool for swapping out projects and storing them. Love your channel!
You can use the psu 12v for rp4 but needs more converting cables or cut the jack entirely
Could it have been due to the SAS raid card trying to run it's BIOS on startup i.e. to setup the drives?
As it's for AMD64 it wouldn't have been able to run on the Arm?
There are some which run in HBA mode(i.e. don't handle the raid but let the OS have direct access to the disks, I've got one in my DL380e. Perhaps then it would then allow the PI to see the drives?
In essence it would be the same as the card you ended up with but with the ability to add more than 4 drives in the future.
Would be nice to hear whether you've tried a card in a PC when it fails in the pi but is under suspicion itself.
I don't currently have access to a PC where I could test it :(
They sell power/reset switches for these psus.
I wonder if LED PSUs would work (probably not, as they don't have both 12 and 5V outputs; though a 12V to 5V buck converter is pretty cheap).
Does the card actually need more than a 1x link?
Also, for the PSU, how about connecting that to the pi? You could either have that power the pi (and thus not need any other cables), and/or have the pi drive the PS_ON connecter low to switch it on.
I actually considered this, but then after testing it with a floppy power connector (which worked fine to power on the Pi), I realized how much more annoying it is to *unplug* the floppy connector than a barrel plug.
For a more permanent installation I may stick with the floppy connector (especially if I wire up a PSU power switch somewhere). But for testing the barrel plug is a lot easier.
@@JeffGeerling There is always the option of cutting cables and rewiring, no reason the ATX power supply can't have a barrel connector.
@@jeffreyblack666 True, true. I'll maybe task Red Shirt Jeff on that.
@@JeffGeerling Glad I could help get red shirt more fun.
Did you have any thoughts on if the original card needed more than a 1x pcie slot?
I have been looking to replace my current NAS with a more customisable one, and that card looks a lot nicer than a bunch of wires.
I love where Jeff's head is with this. Here's hoping a future Pi will include at least one modern PCI Express slot.
I have no idea about this, but are there any older HBA or RAID cards that might fit into the slot natively?
What I want to see is a mass market of low-cost, low-power, elegant, Pi-based devices...including NAS. I'm not much of a tinkerer. I want something that just works well and looks presentable with minimal fuss.
I know there are pre-built solutions out there, and I have a Synology NAS myself, but I want to drive prices further down. My Pi 3 B+ really let me down when I tried to use it as a RAID5 NAS. Even if it had worked reliably, it was a rat's nest of cables.
A lot of SAS or RAID cards need to be configured when booting up a pc. Could that be causing an issue? Maybe you can find a Linux until to configure them? Also, for the power supply you could use a GPIO pin on the Pi to power it on once the pinks booted?
you could save some space and wait. . . and maybe money with an SFX Psw. my question would be. . . . could you use the sfx to power everything?
-Thanks, wonderful content .... !!!
-So, my dream is to assemble much cheaper and DIY a FreeNAS with raspberry and HDs.
-Now with this working perfectly we can go to the Freenas staff and demand that we have an image for Raspberry ... !!!
-With OpenMediaVault it was already possible, but now with these advances you can have a Freenas (I want the pluggins of frenas running).
-You are evolving exactly as I needed the project, thank you ... !!!
mdadm is pretty sweet. My main NAS died and I moved the array to a backup machine. I have hot swappable drives and I put them in one at a time while tail-ing syslog. I watched each of the 9 drives get detected and when the last one went in I saw a msg from mdadm that said "hey I see an array here, you're all set!" Super painless and I didn't lose my 56TB of data!
Sometimes simple is best. And I like mdadm because it's (relatively) simple.
I wonder if the issue has something to do with it running on a non x86 system, I'm pretty sure the raid cards have specific ROMs on them for the architecture