I made the Petabyte Raspberry Pi even faster!
HTML-код
- Опубликовано: 16 май 2024
- Sometimes you have to go slow to go fast-with 60 hard drives on 1 Raspberry Pi, I found a way to make it run in RAID!
It's not the fastest storage server in town, but it's good for about 70 MB/sec write speeds and sucks down about 500 Watts of power.
Many questions viewers have about the PetaPi are answered in this video.
Support me on Patreon: / geerlingguy
Sponsor me on GitHub: github.com/sponsors/geerlingguy
Merch: redshirtjeff.com
2nd Channel: / geerlingengineering
Special thanks to 45Drives for sending the Storinator and hard drives used in the production of this video!
#PetabytePi #PetaPi #RaspberryPi
Contents:
00:00 - The sound of a spaceship landing
00:43 - HBA firmware woes
01:42 - Firmware updates
03:02 - Btrfs RAID 0 - more stable
03:26 - Forcing PCIe Gen 1
03:55 - What's the Gen 2 breaking point?
04:58 - Overclocking for speeeeed
05:28 - Power to the Pi
06:14 - Price? Weight? Hardware RAID?
07:12 - Multiple Pis? 10 Gigabit?
08:16 - What's the name?
08:35 - Deploying the Petabyte 
Наука
Coming out of warp speed when those drives spin down. What a fantastic sound.
It's unreal! so cool
It's also the absoutly worst, what could happen. If all drives spin down at the same time, it causes vibrations so hard, it can damage the disks so they die...
I feel like you are helping the community by testing the niche cases and thus help the main cases along the way.
That's the goal! If someone wants to build a large(ish) storage array with a Pi, at least with the CM4... they can avoid doing things that are definitely dead ends :)
yesss!
Yeah using old versions of broadcom/LSI tools to flash firmware images because they don't check signatures or card type is a somewhat well-known thing in the crossflashing community (turning old crappy RAID cards in HBAs). Good to see that this time-honored tradition is still alive and well with more modern cards
Yeah, also useful for crossflashing cards (e.g. to turn HP-branded models back to the vanilla LSI variant, which sometimes removes restrictions). The HP H220/H221/H222 variants, for example, are often significantly cheaper than the original LSI version. You can't crossflash the HP variants using the P20 sas2flash tool, but the P14 version works perfectly well instead.
So what is supposed to be the real way to update the firmware then....? Or are these older cards not supported by the new flashing utility?
Having just converted an old HP DL380 G7 into a TrueNAS box, this is all too familiar to me.
@@kyoko703 Jeff in the video is flashing the right firmware for his cards so the tool should NOT fail like that. It is probably a bug.
Crossflashing is another thing, where you flash a different firmware to convert a crappy RAID card to HBA (because of cost reasons, there is plenty of old low end SAS RAID cards in the used market and their price is much lower than the price of HBAs)
@@marcogenovesi8570 The problem was, that the firmware was way to old and the LSI tool wasn't able to verify the firmware. LSI verifies every version, so you won't brick the card. Older version of LSIcli probably doesn't do this. Sometimes you've to do a upgrade way from version 1.6 to 2.4 and from 2.4 to 3.9 and so on, till you're at the latest version 5.0.4.
Thats, cause some firmware versions have major changes, which are only able to flash fine in a special way and the vendor doesn't guarantee a working card, if you don't flash in this particular steps.
Sounds like a turbine shutdown. 😬
No. like Supra
On a MH-65 Dolphin helicopter 🚁
@@gus473 Or an S-3 Viking
'All passengers, please remain at your seat until the seatbelt fasten sign disappears'
And death drives 😬
0:24. The Seagate Sound....On steroids!
Right now there's a bunch of sound engineers at Industrial Light & Magic facepalming that they could have gotten the best space ship power down sound just by turning off a bunch of hard drives 🤣
Thats funny. The other day, a tram drove by us, and a kid commented "that sounded just like the X-fighter zooming by in starwars", and it totally did. So much so, I wonder if that perhaps is how those sounds were made.
What does it sound like when you turn off 60 disks?
The THX noise, apparently.
Exactly what I thought. "THX, the audience is listening"...
My mother-in-law went to the pet store to get what's essentially a dremel for dog/cat nails. It was called pedi-paws or something like that. Instead she asked the worker for a pedi-file. Good call on the name you picked.
hahaha
Oh the joy of juggling different storcli versions and notes on which one works with what for unknown reasons - I know that game.
lol the scars of a sysadmin
*twitches*, i still have a usb drive somewhere with a mass load of storcli and megacli versions on it (along with bunch other tools for dell and hp variations cause screw us right)
The fact that this is possible at all blows my mind
Very nice! I'd like to see the test with the SAS extender and only one HBA: the system PCI should be more stable (no PCI switch, Gen2 speed) so better local speed (for array rebuild...)
Although the process of debugging and ground breaking is stressful, just how much fun was this.
It looks like, to me, that this project was a blast.
Amazing Jeff. I love your endeavors. Thank you for all your hard work
sadly at this time it's easier to buy 1.2 petabytes than it is 1 pi
there are lots of pi$$$ on ebay
@@flethacker I do not like to be robbed. :-)
I think I got lucky, bought an RPI4 on a whim a couple of weeks ago and it just happened to be in stock I guess. Good little nas/server and hardly uses any electricity.
Firmware flashing HBA's is the gift that keeps giving. The gift is frustration.
Or you read the manuals of the vendor (at least Broadcom has for the most cards a well documentation) and follow the instructions of the way to flash the cards, especially from which version to which version the upgrade is supported and use the "easiest" way the suggest, for example, flashing directly the firmware from the bios of the cards, from the EFI shell, from the EFI/UEFI itself (this depends on the EFI/UEFI you has and which support the PCIe cards has themselves for EFI/UEFI interaction) or use a flashingtool from the OS itself. Some ways are easier, some are harder, some are brutal^^
Thanks Jeff for the update of this wild experiment which is weird and pretty much fun, and I do wanna know more about ceph cluster with several Pis.
Thank you so much for taking the hits for us :)
Impressive feat!
Thanks for sharing your expirence with all of us 😀
Jeff , I’m so jealous of this!
Awesome work and thanks for letting us in on the magic
Yay! You showed my comment (7:00). In other news, I was surprised at how poorly the power consumption scaled when you spun down some of the drives. 200 watts for 15 drives seems a lot worse than 500 watts for 60 drives. I'm curious where all the draw is coming from.
The fans alone on this thing probably eat up 30-40W. They're pretty massive, and spin at high RPM (each one is like 1 or 2A at 12v)
@@JeffGeerling If you did a smaller 15 drive rack, could one use less airflow? Or is it needed to push past the rows of drives?
I've got a PSU that draws 60W with no load. So you might find a lot of bare minimum kind of things in that 200W. Also PSU efficiency should increase as load goes up and then probably dives at some point before the maximum power.
The Rockchip RK3588 looks so sweet. Been keeping my eye on those. Thay would make great servers and high performance routers/switches.
That is fricking amazing Jeff !!
Jeff and Wendel collab, nice :D
Awesome project! I'm very surprised how well you got this to run! Can't wait for Wendell guest episode.
Congrats Jeff! I knew you could do it. OMG, the hoops you've jumped through to get there is mind boggling.
Is 'Winner Jeff" another one of your characters? The "thrill of victory' is a pleasure to behold. Even better to feel, especially after those "agonies of defeat", eh Jeff?
Just goes to prove that it's all relative. We rarely appreciate the good times, until we've had to struggle for it. 😎
Great job, Jeff. Thanks for sharing.
Love the editing, keep it up!
That sound of the drives spinning down! Jeez thats cool!!
That's insane I'm so happy to have heard this
60drives spinning down
It's really neat seeing you push these tiny SBCs to such extremes!
I love your ambitions projects!
Always wholesome to see Jeff shouting out smaller RUclips Channels like LinusTechTips.
Really cool video. I admire your perseverance 🙂!
What you really want is multiple Pi hosts with smaller groups of drives running as a Ceph cluster. (Oh man, as soon as I started typing that, I got to the part where you say that you're doing exactly that. Good job!)
7:20 I’ve been waiting for this Ceph video for so long. I’m pretty sure it’s gonna be awesome 🥳😛😍
Thankfully the one 8TB 7,200RPM HD my Pi NAS has is way more than enough for my use case.
Is fantastic to see it can be done if I ever needed it. Or had the cash, for that matter.
I ran a 60 SCSI disk Linux software RAID array in 2005. I was up to /dev/sdbc or something. 🤣 I had to backport kernel SW RAID code because the older kernel only had 4KB of memory to store the dev names. When I’d start the array the kernel would panic because the RAID info overwrote the buffer in memory to keep track of the array info. Fun times. Great video.
I love watching Wendall when we're talking storage, RAID, ZFS, Linux, etc. good call
Great video dude 👍
lovvvve this ideas with pis :) thanks dear jeff!
Wooo CEPH Cluster FTW! Really looking forward to seeing how you did that and how it compares to mine.
Watching on my flight back to STL. Keep it up and I’ll wave while landing at Lambert!
0:25 that sounds a little bit like the Prague metro C slowing down
So want that sound bite of the spin down!
Fascinating!!
One word... Amazing!
Thx 4 overclocking ! 10% Note throughput is quite nice.
You could Test against an older Desktop Low end CPU ( less Power Draw For comparison.)
Pretty impressive!
Thank you for all the hard work for our entertainment and learning. My question is what about multiple Pi's as a Cluster for controller?
0:21 Love it!
No ECC memory was the first deal breaker to come to mind.
Cool and determined though, so much so that I watched it entirely!
Thanks Jeff I need a new Apple keyboard now 🤣 Peta File made me spill my tea 💖🤣
Oh god yes, a Jeff Geerling video, finally something to watch!
I really wonder how nice and warm that is gonna be for winter and if it'll keep Red Shirt Jeff warm.
Honestly tho, mega project and world needs more people willing to do projects/challenges like this - just to see if it can be done 🤘🤘🤘😉
Awesome video as usual! 500w of constant power draw.... wooo yeah. I have a pair of QNAP TR-004 and those are loud enough. Only have them running when I'm writing or reading.
500 watts is about half the draw of a typical small window air conditioner.
bruh .... the amount of testing and effort .... bruh, we appreciate you!
I wish we paid .13 the kilowatt where Im at.... its almost at .28 here. Thats around $130 to run this a month
Ouch. If you would like to deploy a petabyte at my house, I will offer you the low low rate of 0.23 and skim the profit :D
Of course I only have 35 Mbps up, so if you ever need to access data from it, that'll be a couple megabytes per second :D
Hi Jeff
I'm currently working on my own Ceph video / installation on raspberry pi.
I'm trying to get version 16 to run on Raspbios but sadly the only packages I found was version 14 for bullseye and the packages had missing builds in the ceph repository.
So my last resort was creating an emulated pi and build my own Ceph packages.
Have you found another reliable way to install Ceph?
Best regards
Daniel
That shut down noise was exactly what I wanted
Astonishing!
Thanks!
Really looking forward to the Ceph cluster. It would be interesting to see if you can increase performance incrementally with one Pi at a time.
Ceph on Pis, omg 😂😂 .... can't freaking wait for it! XD Cephadm is great btw ;) But I'm not sure if they have aarch64 images. If not you should build them and use that, as every other deployment method is much more cumbersome
Is the issue of using 3 HBAs due to limitations with the interrupt controller on the Pi SoC? I used to have to flash firmware with older builds of the tools and then apply the updates sequentially, but that was way back in the day (like 15 years ago) so I was shocked to see that was still a thing! Thanks for taking the time to jump through all of the hoops!
Nevermind, I made it to 07:45
@Jeff Geerling you can use pi cluster supercomputer as parallel computing so that cluster can handle many operation in raid 0
ngl - shutting down sounds reminds me of my old vacuum cleaner i used to have. Noise it made was straight from hell
The power down sounds like a Transformer laying down for a nap.
Yay for getting Wendell into this :D
Hey Jeff, it might be that pice switch that is having the problem with having 3 of the cards at the same time. If I’m remembering correctly that card was made mostly for mining. So it would be one video card a time. I would look there.
good to storage school work
I’d be interested in how a cluster of pi’s would handle the 60 drives. It’s probably a massive task and you’ve already mentioned your working towards that. Anyway, thanks for making these videos!
I just attached the 4th 10TB HDD to my NAS. I'm rebuilding the RAID right now.
But I think you won.
Waiting for RAID to rebuild, my favorite pasttime! I have 3.3 days(!) left on an initial sync on an array on one of my other NASes.
@@JeffGeerling I guess I will have to wait until tomorrow morning to use the NAS again.
So what do you use all that storage for??
@@raghu5038 Somebody has to keep a backup of the internet.
Flash the HBAs to IT mode and compile in ZFS to the kernel if you want decent throughput and reciliency.
Awesome!
So you immediately ruled out "Peta-file",
but *not* "The Big PP" ?
I want to talk to HR.
😂😂😂😂
Sounds familiar. I was testing a Thunderbolt Fibre Channel HBA on a Monterey (intel) Mac today. The card seemed to work, but the config tool reported IO errors when attempting to check the firmware version. Thankfully, three year old drivers worked fine on Monterey to detect and update the firmware and now I can run up to date drivers and access our giant tape library from an M1 Mac Mini.
So maybe that should be your next video? Talk someone into lending you a 3573 (Dell TL4000/IBM TS3200) tape library with a couple of LTO8 drives and a FC HBA? ;)
"This time, I only have half a petabyte, but peak power consumption is only 150W!"
This thing would cost ~1350€/y in electricity bills alone here in Europe...
Still peanuts in comparison to the initial investment cost, though.
If you actually need one Petabyte of storage, you can probably afford the power bill for this
Yeah as a German (one of the highest energy prices in the world) i wasn't sure whether to laugh or cry at those US$ 0.1296/kWh
In my area it would be around 2100€ a year.
5:42 High peak power consumption during boot-up (the experts among us call this "inrush current") once made the electricity supply during a trade show I attended collapse. I had a 15-bay server case with 600 W redundant PSU with me and wanted to show it to potential customers. Instead, they only saw pitch black darkness. :(
Oopsie!
Jeff, you're an inspiration! I find it fascinating how you get all those niche projects working. It got me wondering if I could do something similar.
I'm wondering if an M1 mac mini running asahi linux could become a decent NAS + self hosted application box, similar to how some NAS's allow you to spin up some dockerized apps on your home network besides simply storing and serving your files. I have no clue how to make it all work optimally yet, though. Similar to how you said adding 10GB ethernet on a raspberry pi doesn't really add any speed benefits, I am unsure if things like 10GB ethernet, thunderbolt 3/4 external harddrive enclosure, hardware / software raid, etc. would be worth spending money on or if the performance gains are minimal. Having HDDs for storage and an SSD for caching also sounds good, but not sure how I would implement that.
I do think the energy efficiency and performance of m1 will make for a great base for this project, just not sure how to do the cost benefit analysis of the other hardware and software decisions I would have to make. I've got a lot to think about. What's your first impression of the idea? Do you think it's viable? Any tips? Have you considered doing something like this before?
Thank you for your videos and always being so passionate in explaining your projects.
I don't envy you on getting those HBAs flashed. I had one where I had to take it from Linux, to Windows, to FreeBSD to UEFI multiple times before I finally got it flashed and working.
So i had bought old SAS server with only 6 drives, and it is consuming, according to idrac 175W, never gave it a thought, that drives can accualy take a big part in this. Also in Poland, electricity costs are bigger that u have so i pay for that almost 20 USD monthly...
But i like the fact that i can run on it surveillance server, game servers (Valhaim for example), nextcloud ect.
It is still cool project, i just like that i can just dump many things at my server, and it costed me 336 USD by todays course.
How good is that sound at 00:26 🥰
I don't know why you didn't choose Peta-FIle...seems like a great name!
When he said that I heard AvE's soice say "aaaannnnd DEMONITIZED"
I feel you
How's the MTBF on those drives when all placed in one chassis and sharing all that vibration? That's a real issue in case and rack design for storage arrays.
Love the Wendel shirt.
PetaPi is a good name. Though Petaberry Pi, Pi Byte, or Raspeta could also work.
Why do the project?
Because you can.
Love it.
Hey Jeff! I was actually wondering if you have any other sbc's that might have more pcie lane's to test projects like this against the pi 4? something like a jetson nano/tx2/xavier module
0:21 sounds like a reverse audio of the THX intro.
I wonder if there was some IRQ sharing issues when you get to 4 HBA devices. I know it's a weird one, but I've had this happen with PRI T1 cards in the past and had to make sure they weren't sharing IRQs on the PCI-Express bus
Hey Jeff, it would be interesting to see how you’d setup raid with a few SSDs for retropie. I was following another one of your blog posts hoping to combine the two concepts of creating a simple RAID1 Array and booting from the SSD. Definitely been a struggle on my end. I’ve been getting a “wrong fs type, bad option, bad superblock on /dev/md0, missing codepage or helper program, or other error” Long story short: I just wanted to know if you had any more information or tips on solving this (BTW I’m a cmdline noob 🙃)
that is completely bananas 😀
I genuinely think Pi-tabyte sounds pretty good for this :)
Those spinning down drives sound like a Jubilee line tube train coming into a station
Are you going to run a more typical threadripper type setup? And are you planning to do a software or hardware raid. For faster drives i know that a hardware raid card is much faster than doing a software raid especially even a threadripper gets bogged down with not too much through put with the software raid.
Yeah I've seen things like this now and then. Firmware updaters are a pain. I have done similar for fiberchannel cards. I used the dos based updaters.
Great video, Jeff. I know this shouldn't be my takeaway but wow your electricity prices are insanely cheap! We're now up at 28p/34c per kWh in the UK (almost 2x what they were) and that's the capped price (would otherwise be higher!)
Sound effects engineers would pay for those sounds.
When Wendell gets involved, you know you're in for a good time.
That was the most interesting video about a subject that i can't forsee being useful to me for a long time :)
" you can't run more than 30 drives on a pi in pcie gen2". Noted :D, just in case i happen to find a pallet of drives and a pi :D
All joking aside, that still shows how capable the pi is; i don't think i would have expected things like that when i bought the first version B of the pi some years ago.
I wonder where the pi 5 will enable us to go...
Haha, right? I mean, in production I'm not going to use the Pi either, but it does give me some ideas for smaller projects!