HomeLab MEGA Storage - Home Data Center Guide | 60 Bay JBOD Disk Shelf | NetApp DE6600 / MD3060e
HTML-код
- Опубликовано: 6 окт 2024
- Written article digitalspacepo...
Cheap HDDs shop.digitalsp...
FEATURED GEAR
Netapp de6600 geni.us/netapp...
Dell MD3060e geni.us/dell_3...
DE6600 SAS2 Modules geni.us/6600_S...
DE6600 PSU X-48564-00-R6 geni.us/DE6600...
DE6600 Replacement Tray X-48566-00-R6 geni.us/DE6600...
DE6600 Fan Canister X-48565-00-R6 geni.us/DE6600...
👇HOMELAB GEAR (#ad)👇
RACK - StarTech 42U Rack geni.us/42u_Rack
DISK SHELF (JBOD) + CABLE
Netapp ds4246 geni.us/netapp...
Netapp ds4243 geni.us/netapp...
QSFP to 8088 (SAS Cable needed for 4246 & 4243 JBODs) geni.us/mCZCP
HARD DRIVES shop.digitalsp...
RAM
DDR4 RAM geni.us/DDR4_E...
SERVER
Dell r720 geni.us/OAJ7Fl
Dell r720xd geni.us/5wG9n6
Dell t620 geni.us/dell_t...
SERVER RAILS + CABLE MANAGEMENT
APC Server Rails geni.us/APC-SE...
Cable Zip Ties geni.us/Cable_...
Monoprice 1U Cable Mgmt geni.us/Monopr...
Cable Mgmt Tray geni.us/Server...
Dymo Label Maker geni.us/DYMO_L...
HBA
LSI 9207-8e geni.us/LSI-92...
ENCLOSURE
Leviton 47605-42N geni.us/levito...
SWITCH
Dell 5548 Switch geni.us/Dell_5548
Mellanox sx6036 Switch geni.us/Mellan...
Brocade icx6610 Switch geni.us/Brocad...
UPS
Eaton 9PX6K geni.us/Eaton9...
Eaton 9PX11K geni.us/Eaton9...
Be sure to 👍✅Subscribe✅👍 for more content like this!
Join this channel to get Store discounts + more perks www.youtube.co...
Shop our Store (receive 3% or 5% off unlimited items w/channel membership) shop.digitalsp...
Please share this video to help spread the word and drop a comment below with your thoughts or questions. Thanks for watching!
☕Buy me a coffee www.buymeacoff...
🔴Patreon / digitalspaceport
🛒Shop
Check out Shop.DigitalSp... for great deals on hardware.
DSP Website
🌐 digitalspacepo...
Disclaimers: This is not financial advice. Do your own research to make informed decisions about how you mine, farm, invest in and/or trade cryptocurrencies.
*****
When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network.
*****
******
As an Amazon Associate I earn from qualifying purchases.
******
#homelab #datacenter #servers
Love that old style NetApp disk shelf on the left of that DE6600 so many Datacenters still use them. Those SFF 8088 connections on the back are rock solid.
The DS4246 is the world's best 24 bay JBOD. No question about it, amazing little machines.
@@DigitalSpaceport totally agree. Especially coupled wit a FAS and OnTap it’s simply one of the best storage solutions
Worked a lot with things like dell ISCI, eg. MD3220i and you actually need 3 things to get those if-6 working, first is out of band management going (special software, somewhat easy to find), then because they're pulled from god knows where you need to telnet and do a "lemClearLockdown" on each module, finally you need to let the controllers know there are 2 of them and that's "redundancyMode=duplex". You need the so called "root" password (it's not gonna be called root just fyi), which I can not share (but you do find it on the internet with enough searching). If you're really interested, you're going to find the answers. After that, you need to actually go in the management software, define your FC or IB, make same modifications on switches and be sure you configure the FC or IB adapters accordingly in linux (multipath is a b***h on linux). Heck of a headache, but the speeds are just like running the drives locally... PS: if the controllers still have passwords on them for the out of band management you're going to need to reset them "sysWipeAllConfigData"
oh hello there! I have a unit for testing without data on it that I can play with here still. Will be testing this all out this week. TY very much for these cmds. When you say "special software, somewhat easy to find" do you mean off dells site or something open source like openfiler?
@DigitalSpaceport MDSM software I think
Great content! This was actually on my wish list. Thank you!
Excellent video, thanks for all your hard work.
So with the I/F-6 IOM/EMM, they are the raid controller / brains of the SAN. They do run Santricity or whattever rebanded version your vendor calls it. They are basically full servers running VxWorks. The extra connector on them just links to the second module for failover / sync purposes. I've been working on getting the different versions working with different shelves, it takes some doing but it can be done. It's much easier if you have drives with firmware from the same vendor as the software running on them - eg if its a Netapp E5600 like what looks like you have, by default you need to run Netapp branded disks with the netapp firmware. You can bypass this check tho (bit more tricky). You can also flash eg a Netapp E2700 EMM to run the Dell MD3800 firmware, which dell makes publicly available, but didnt update it as much as the netapp firmware. Still locked, but dell is more laid back on flashing drives etc. Im currently working on putting together a big document of all part numbers of the LSI/Engenio shelves and controllers and firmwares from all the vendors and what works with what so you can build your own SAN's.
Personally my favroite at the moment is probably the Netapp E2700 / Dell MD3800 since they are more recent, more powerfull, and more popular to find, but I'm really wanting to get my hands on some Netapp E5600 which can have 96GB cache/DRAM instead of the 8GB from the E2700.
The newer ones are the E2800, which can support 25gbit ethernet, but seems like netapp may have made some physical changes to the pinouts, and they are still quite expensive, but thats next on my list to try and make work. Ping me if you want me to send a heap more info, and if you're open to id love to get remote access to some of what you have to validate my findings on my equipment
I'm not sure how this comment escaped me and I hope you are still alive! YES. social@digitalspaceport.com and let's collab.
That damaged power supply is an industry standard "quarter brick" module - you may be able to replace it. It may be press-fit or soldered, but in any case it's a much easier repair than the fiddly transformer bit by itself.
Yeah they are cheap on fleabay also. Love ex enterprise hw that sold a lot for that reason.
The modules with the RAM and heatsink in them are basically just Intel Servers, they do iSCSI/FC generally, meaning that if you could get your own OS installed on them you could use the array completely standalone.
I've been working on this but man, VxWorks really has this bootloader locked down.
Hey buddy, well done on a run through of the magnificent DE6600 :)
Just a FYI, there is no measurement called "Amperage", the correct term is current.
An AMP certainly has a measurement. 6.241*10^18 electrons/sec/point
16:50 that expander is for their san systems. Netapp DE like stands for disk expansion so that is made for their own san systems so the fiber optic connects to the san box and it controls the disk arrays (wich are raid controllers) and they connect to eac other using the qsfp connections so they require their san to work. Tho netapp made some expanders to work on normal pcs, it is not really ment to do that. The DS line stands for disk shelf and it is for connecting as das to a normal pc. That's why they can also be used as san expansion.
4:20 THANK YOU FOR DOING dBa VOLUME MEASUREMENT! (not enough equipment demos do this)
Yeah I need to do this for all hardware demos
@@DigitalSpaceport Yes please!
Great video! Especially the part about the EMMs came in quite handy as the last video about these JBODs left me wondering. The only thing that isn't quite clear to me yet is the exact power consumption. Do you have some in depth numbers like: what's the power consumption using a singe power supply and a single EMM without any drives? How much additional power consumption for adding the redundant power supply, or for the redundant EMM?
You have to run this with all the EMMs and power supplies installed. If you don't, the fans will run up really high and hurt your wattages a lot. Also without the full load of disks, the fans will run a bit higher. The front 20 drives (4 per each tray) does to be loaded. These run around 2.2-2.4 amps at 245VAC which works out to around 560 ish watts loaded
Your contents are gold, thank you so much! Can I ask your opinion, costs aside, do you think it is less headache to go for 12 x 10 bay usb enclosures, or to go for 2 x DE6600? Noise is also a concern as I'll be putting this inside my house 😅
I would NOT put DE6600 inside my house. They are not super loud outside startup, but they are not live with it level noise either. The NetApp DS4246 is a live inside with it level of noise in a JBOD
@@DigitalSpaceport Understood, thank you for your insight I might go for the DS4246 when I expand
Wow. Super detailed video. I didn't catch if you stated what the power usage tended to be for 1 fully loaded unit. Those power connectors on the back of it are C19/C20. Those are more and more common in datacenters now, but usually are plugged into C19/C20 sockets on the PDUs to make sure it can handle the heat and current. I don't often see them cabled up to standard C13/C14 plugs. Not sure that there is any actual risk to doing it the way you did. (as long as the current doesn't exceed the cord or the plug)
He should’ve let you hear the initial startup sound. Those fans can really go berserk on those NetApp devices. I truly love them.
Also don’t see any OnTap configuration in the first 10 minutes. Oof reacted to early. The startup sound was right after the first message
@@ernestoditerribile I'm not surprised. I've been a sysadmin for 25 years, so I've seen how devices/systems spin up the fans to their highest setting when powered on. I especially love rebooting management modules in server blade chassis remotely while a new sysadmin is doing work near them. Those things sound like an airplane taking off.
OnTap configuration would be pretty expensive at the size I've got running here 😭
@@DigitalSpaceport yeah I know. It’s largely used in Datacenters. Your setup looks like a Small to medium business solution.
:(( Why do you have such awesome prices for electronics in the US vs Europe? The HDD listed there are around 9-10 Dollar per Terrabite
We ship to some European countries. What country are you in?
Thank you. Always look forward to your content.. ...
I have 1 PB and other than Hard drives cost I only have to spend an extra 600 for 33 drives, that's only $19 per drive and that's with good cooling and vibration-dampening, Power supply, HBA, cables, and expansion cards. I use an empty 4U JBOD with strips of foam in between drives, and I still think I'm spending too much in order to make an ok return at today's prices. I have no idea how you are going to make your money back. If you think the price is going to go up that much, then just buy XCH
That jbod is commonly around 500 (it's 500 right now also) and is needed for ten+ years of operational lifespan with proper airflow draw on 7200 RPM drives. I paid less then that for mine also but had to get a pile of other stuff. Folks assume I have spent a fortune, quite the opposite.
I don't think the point of this video is to save money. I would like to hear more about your solution tho. I personally have scaled out using Supermicro JBOD's. Less density 45 vs 60 in 4U's, but I feel like the Supermicro offering with features etc is better long term for me vs these Netapp JBOD's.
I am aligned with Digital Spaceport in the need to deploy my storage in an enterprise fashion. I have no issues with all the creative bare bone solutions people have come up with to build huge storage arrays, its just not my style and I don't these deployments will hold up long term.
Something is not clear to me, all this in your house? How much electricity do you pay?
Around 225-250 for entire house ea month
Hi Jeremy and congratulations for all the videos you make, we already spoke a few days ago about the DE6600 but with another account but the same logo and I decided to support your channel. I received 3 NETAPP DE6600 but with I/F-6 controller (11-03426), I have a sas3 9300 8e HBA card, it doesn't read my hard disks and I get the yellow signal in front of me. Were you able to find a solution for these controllers? A thousand thanks
Have you tried to look under the copper heatsink in the "active" EMM/IOM ? Also have you tried to see what does comes from it's IOIO (miniUSB) connector? I guess it could be some sort of service port meaning serial console. I think, it could be x86 based (socketed Intel CPU) and you probably could install different OS on it.
Few years ago I got my HB-1235 JBOD and as I was working a bit around Dell Compellent storages at that time, I got myself Dell SCv2020 controller marked as "for parts" (it got only broken SAS connector, that was probably causing POST issues and not passed manufacturing control)... and I knew, that especially Dell Compellent line is Intel CPU based (SCv2000/SCv2020 or SC4020 are E3-1265L v2 with 8 or 16 GB DDR3L RAM respectively in 2 slots and "management" ports are actually Intel X540-T2 10GbT chipset, it also has PCIe 3.0 x8 slot for different I/O types).
I managed to thinker with it a bit, as I saw at my job, how they update SCOS in SC4020, it has IPMI built in - yaay and also adapter from 3,5mm jack to serial console was handy. I wasn't able to successfully install SCOS on it without original drives, but I managed to install TrueNAS and even VMware ESXi on it :D (it has 32GB SATA DOM inside), I just needed to reflash the firmware on the integrated LSI 2308 controller to non-OEM one to see the drives in the HB-1235 shelf. Now few years later, I still run it as off-site backup unit.
My observation is, that most of those Dell Compellent / MDxxxx / EqualLogic / NetApp / HPE 3Par / IBM and probably others (except EMC afaik) disk enclosures/controllers are OEM/ODM manufactured by former Xyratex (now Seagate) and use same Xyratex SBB format of controller I/O (as you mentioned one missing pin array in the controller I/O, I guess it could be for inter-controller communications e.g. cluster link).
Another interesting controller to look for is Flextronics SATIT2 (for Microsoft StorSimple arrays), I was looking for it for a while, but not needed to purchase one yet. It looks interesting as it is Intel socket 2011 based (=E5 v1 or v2 CPUs), has 4 memory slots and ConnectX-3 onboard with QSFP+ (probably 40Gb, but marked as 10GbE and not mentioned anywhere what are the real speed capabilities).
This is the most intense hope I have had for those controllers yet! I have no idea if this will work, but I may try here. Only one way to find out!
IOIO Micro port is on the same controller as the IOIO serial port. It serves as console port. 9600 speed. So as a TPM/Field Engineer, we use either port to directly connect to those ports for direct access.
Great Video, Great content!
I'm a Newb to server based equipment and I've recently acquired 3 of these for free to mess around with.. The power cables that came with it seem to be flat female to power supplies and standard 3-prong that fits in a wall socket, but I would assume that it needs a special outlet to run enough / proper power? Would something like a 240v dual 120v x 120v socket maybe work? I do have a 50 AMP capable NEMA plug in my garage that I could use a pig-tail with if that helps..
Any assistance is much appreciated.
Too late, I already got myself a DE6600 delivered today. It is much heavier than I anticipated! I noticed some HDD indicator lights are not working, but all drives are detected in Ubuntu. Can I ask please what is the best way to manage the drives, for example how do I find out which drive is bad should one goes faulty?
Hey this is a bit of an old video, but how do you protect your harddrives for power losses etc. I mean i know you run an UPS, but let's say that you're running out of juice in the battery - how does it safely power down seeing you are using that physical power button in the back.
Thanks for great videos btw, it's been inspiring me to get a DE6600 too, very close to pulling the trigger tbh.
Luckily its a massive UPS. I do have truenas set to shutdown with 5 mins of no power, which has happened only twice so far. This protects the state of the data in ZFS adequately in case of extended power loss. Have had no corruption detected in scrubs.
The jbods themselves only ever power up and down directly. There is no soft off powerdown button on any enterprise JBOD I have seen so far if that is what you are asking. I do usually switch them off from the PDU however and not the physical button, but its essentially the same.
In short it powering down is its safe powering down. You protect your data at the controller level and of course should not physically be powering up and down your JBODS frequently. JBODs are intended to stay running 24/7/365 but powering them off is reality also, even in datacenters.
Hi Buddy, thanks for the excellent tutorial, Please can u clarify for me if the ''Dell PowerVault MD3860f'' will work with Truenas?
When you see and at the end of a storage product, that indicates it's a fiber channel part. I personally do not think you want to screw around with fiber channel. It's not "easy" and working with the storage concepts around it is something I have tried and failed to get working a few times. Maybe you can replace the Fiber Channel controllers with SAS controllers however and then it should work no issues with TrueNAS.
Incredible content!!! Epic
Hey, that's a cool shirt, bro 😎
I would really hate to see your monthly power bill. Cool product regardless, I use NetApp hardware at my current job and did not know about this product line.
It's like 425 bucks with AC running and all this. Not really that bad.
I've never seen any hard drive in stock on your store, how often do you get stock replenishments?
I've got more in Friday or Monday this week. I give channel paying members first notification and a few hours window and they have been selling extremely fast recently. I also have an email sign up on the website that goes out typically faster then RUclips notifications of restock for nonpaid members.
Wrt your stated minimum 20 drives for operation. Is that a "McSpecial NetApp" fw thing? I have a few MD3060E units and before I knew anything about them I used a single drive in the front top tray just to test and it worked fine.
Also using them without trays seems like begging the universe to wreck your life. On the 3060E units there is a switch the trays depress. Further it makes replacement a bit of a pita when any of the central drives need swapped and vibrational backing out is more likely to occur.
Well these are netapp branded vs dell so im not sure. Ive not gotten into their terminal settings either. Mine ran fans like LOUD without drives in the front row. Def a FW issue but you know iow loud these things can get.
@@DigitalSpaceport Indeed. When I first hooked them up after determining they worked, since I only had one circuit I didn't figure it would make much difference with both supplies connected. The noise when only one supply is connected is the loudest thing ever! Annoyingly loud inside the house (they are in the garage) through a few walls and rooms of separation. My wife was seriously displeased. I promptly acquired a y-cable. :)
Do you, by any chance, have the 12Gb SAS EMM part number? I have also noted, as you mentioned, the Dell variant is significantly more spendy. Unclear to me as to why beyond "because Dell".
Lastly, in your video when cautioning people about messing with the chain cables, and then later about the potential for cuts and whatnot. My second one arrived with 2 broken chain cables...blood was demanded. I recommended gloves, real gloves, not the nitrile things.
Hah I can relate! Im cant suggest you can plug and play the SAS 3 upgrade emm, and get a perf boost. the trays have expanders in them, im guessing but likely capped at SAS2. I never tried it, but if you do lmk how it turns out! Sry dont have P/N
Great video and information. Thank you for sharing. I had a few questions i just got a POWERVAULT MD3460 and a few Dell 640 server as the main. Would love to see how to fully connect and configure to a server. What PDU are you are using? You also stated that theses arrays work on 220V i currently have 208 would not work correct? thank you
208/220/240 are all the same really as far as what's wired, which is dual hots to the receptacle and circuit. If you check what the actual running VAC is, it likely is higher than 208 by a decent amount.
Hi Jarod, I saw that when you daisy chain (bridge between jbods) you use two controllers. I noticed that it works well for me even with just one controller per JBOD, is there an explanation why you use it with two controllers for each jbod? It's better? A thousand thanks
"Homelab", no my good sir. You have a private data center. A nice looking one, at that.
Agree 💯
I’m new to this stuff can I get one of these and fit it in any server rack ? Wanting to get one of these to just fill for my plex or storage in general
These are in my Dell 4220 standard depth rack and also use the APC rail trick and work fine like that. They do not have a ton of clearance on the back, but they do not protrude from the rear of the racks.
All your drives show as sold out :( Any idea of when they will be back?
This looks crazy expensive to run for home lab. Do you have cheap electricity?
10 cents US per kWh is pretty cheap I think.
Damn if I can find those sas esm modules in Australia all my 4 of my de6600 are fibre channel. 😢
Have you ever considered running fiberchannel? I think its a horrible idea but I have so many of those damn ESMs I would love to figure out how to run FC or IB. I originally had hoped to run IB for it all.
@DigitalSpaceport Well I do have a Mellanox sx6036 so might try hooking up to that
What rail mounts did you use ?
Imagine the Jellyfin or Plex Servers you could run with this 🤣🤣🤣🤣🤣
Nice. I have 8 DS4246 JBODs with caddies and I0M6s if you know anyone looking for some. In Las Vegas
if you dont mine me asking what is your power bill monthly
It's around 225-250/mo. My electric rate is .9875 per kWh
Doesn't matter the power bill, its the Partner/Wife Approval rating that is important.
Thanks g
Did you do any more with these with FC? I have a bunch of these with the FC controllers I'm going to try to get working before I give in and source SAS controllers at huge cost.
I tried FC but didnt have the right speed FC switch. I would suggest you try openfiler software and ensure you have the correct FC speeds. Im more interested in IB, but no I ran out of patience testing all these. Only SAS could I get working.
@@DigitalSpaceport Did it show up in the SAN fabric?
Hello, what controller and hba card are you using? I use 9207-8e and IBM 90Y8595 ESM controller, but can not recognize sata hard disk, I see yours can use sata hard disk, may I ask how to do it?
With SATA you need to plug into the bottom controller furthest left port IIRC
@@DigitalSpaceport thanks，it working。Your video helps me a lot。
I’ve got a DE6600 and noticed that I cannot use SATA drives. How did you get your DE6600 see SATA drives?
plug in both lower left and upper left SAS 8088 cables to the EMM.
@@DigitalSpaceport Thanks! I'm considering buying DE6600 and I'm worried about same thing. If I plug it as you say, will it work without interposers? Thanks!
@@petopeto6540 you do not need interposers but try to not mix SAS and SATA disks in the same tray
Thanks for reply. Sorry to bother you again, I found Dell MD3060e with 2 x 08X4HH. Have you tried SATA drives with it? Will it work if I plug cables as you advised? Thanks!@@DigitalSpaceport
SATA drive on 3060e works for me.
I would love to repurpose some of the obsolete Netapp shelfs at my work behind Linux, but there is one problem to wich I cannot find a solution: How can i physically identify a disk that needs replacing when I only have an HBA and Linux ? That is the prerequisite to any open source storage project. Anybody have an idea?
Keep a note with all the serial numbers of the disks and their slot.
А ю крейзи!