As a former mechanic, the dirt remind me the kind of dirt that you can find in automobile workshop, a mix of dust (from brake pad etc) and old grease/oil sludge.
The fact that it says ADP DMS on the front is a good indication it was definitely installed in a dealership setting. That software group is created specifically for dealer management/repair order management systems.
i think so too. i have seen enough servers put into cnc-shops, dripping with oil. even ibm decided to no longer waranty exchange the fans, it took them 2 full sets^^
@@dstrickner123timeline works. The company that is now “CDK” used to be part of ADP. They were (and are) notorious for requiring equipment leasing and not letting the Dealer own anything.
This is a server from "CDK" (formerly ADP). DMS = Dealer Management System. They are often found at car dealerships in North America. Past few years they have moved 'serverless' now relying on SD-WAN. I encounter these all the time. As far as the IBM 3650m2 - it was a solid machine. I still have one running at my office, we will decommission it. The IP address on the front of the server is likely not reachable via the Internet. CDK (ADP) used a Cisco WAN to extend that IP address out to the dealership. Controls printers and communication to/from the CDK data center.
@@clabretro I think ADP sold all of their DMS stuff to CDK. CDK has now dropped their Cisco router based stuff and switched to Cisco Meraki. Those IP addresses are only accessible via the Velocloud SD-WAN, not the Internet. I’m not 100% certain when ADP/CDK stopped using the servers (depends on the dealership) but pretty sure everything went SD-WAN about 5 years ago so I doubt there is anything newer on that machine than 2016/2017 best case scenario (if the dealership upgraded)
100% car dealership server for ADP's dealer management service (DMS). That exhaust dust gets everywhere and sticks. They spun off the DMS under their subsidiary, CDK.
My first thought with this is smoke damage. While this server obviously wasn’t burning, I wonder if something near it caught fire and had to be extinguished. That might explain the minor moisture damage and soot (maybe?) inside. Enjoying the restoration and looking forward to part 2.
My hands started drying up uncomfortably while I was watching you clean the thing with IPA and contact cleaner. Wear gloves my man! Awesome job, regardless of the hand-drying horrors haha :)
That server looks like it was though a fire. The grime is soot and smoke residue and the water damage is from sprinklers. That would be my guess at least.
Hey, me again... So what you have here is a server that ran a DMS, dealer management system. I am not sure what the product was called when ADP owned it, but nowadays the division was sold off to a company called CDK Global, and the new iteration of the software is called Drive. I haven't watched the video yet but it will be running a heavily customized version of CentOS (or maybe RHEL) I believe. This was absolutely located in a dealership of some kind, I am intimately familiar with the software. The software manages the entire dealership from service department, to part sales and inventory, HR and payroll...all text menu (green screen) driven. There is almost certainly data on those drives. I actually work for a dealership (as IT) and one of the first things we did during my employment was migrate away from this CDK drive into a platform called Excede. The dealership had migrated away from autonet years prior in favor of SD-WAN. Autonet was really just T1. I still have a working instance of drive running if you wanted a closer look, but its not very exciting stuff.
Hey! Yeah agreed, all signs point towards that. Interesting to hear the dealerships were actually using autonet as well, I guess that all adds up then.
@@clabretro Pretty funny you make videos on two obscure things I'm most familiar with... old pSeries hardware and DMS software... Thinking this channel may just be made by an AI trained on my browsing habits over the last 15 years. Now thats a scary thought.
@@clabretroweird little blue thing is a "feature key" by the way... All it does is unlock extra features of the "IMM2" module, IBM's IPMI. I know it unlocks virtual media support. It might be needed to allow remote KVM console, I cant remember. Without it I think the IPMI is limited to power control and reading vital product data, event logs, temperatures and stuff.
Neat. ADP sold a lot of these to Dealerships. They provided their software to service and body shop, estimate software for certified dealers. Incredible estimating software.
AHHH, I love this! When I was working, (part time) when I was 12, the company was using X3650 M4's for their file servers, and even Active Directory :P on Server 2012 R2, in 2018... It's a small company but janky... They got a Synology iSCSI with only 1 PSU running, which was like damn... Great video btw. Awesome job.
This is the server I use for my home NAS, the main difference being its branded by Cisco. Happy to see you restoring it, it's a nice piece of hardware.
Yep, that's my vote too. Servers tend to pull air through from the front to the back. Likelihood a small electrical fire in this, but more likely an adjacent rack would have had the soot and combustion products suckered into the box for the seconds / minutes the power remained on. extinguishant products (possibly foam for "electrical safety"... don't guarantee the safety OF your electrical equipment and you'll find a number of racks end up being "collateral damage" depending on the incident size. Most data floors don't use sprinklers due to all the electronics, but the fire service might use foam or water, again, depending on the severity and risk of further damage. Losing a rack full of servers isn't as bad as losing the whole floor and possibly floors either side! I would definitely recommend against breathing it in though, that's for sure. Using a wet cleaning method was for the best, though as others have said..... Gloves!!! Lol
It might also have been in an auto dealership, specifically in the repair area. It smelled musty, rather than any burnt smell, but it certainly got wet!
For that gunk I probably would've chanced an "Adrian's Digital Basement" style wash with dish soap, water, and a very soft bristled brush for that mainboard, then an IPA dunk for drying. Yikes! Either way, nice cleanup job, that's a big board with a lot of things to go wrong.I think the theories of a datacenter fire are likely true, either that or a long stay in a stack in a humid environment with a lot of other crap in the air (my first thought was also that that grime looks a lot like auto shop/garage grime). Very happy to have this channel recommended to me by the YT algo (it gets it right every once in a while!). Excellent mix of expertise, curiosities, and that liitle bit of jank that every good homelab should have.
"I dunno what that is" at 6:20 It's an upgrade key, usually either for the better RAID modes or the IPMI. Without it, the associated module will not be as fully functional, or maybe not work at all. Think of it as a hardware DLC.
Never have I felt smarter than watching you take the tab off the LPD, thinking 'you should cut the ramp off the latch and put the tab back on as a puller' then seeing you do exactly that haha
I had been feeling like this machine had spent some time as a homelab in a basement or garage from the looks of the residue in there, and I feel like an auto plant (or even a larger dealership) tracks with that, like you're seeing a decade of buildup from oil changes and radiator flushes.
Man, that server was grungy! I've pulled a few servers apart that had a little bit of that filth with close to 10 years of service, but nothing like that! Wow!
thats the younger brother to systems we still have in daily prod, yes the support contract is expensive to cover them. we have x3550s I have in my rack although i rarely power them on to noisy and slow compared to my other old junk, also have 3 dell pweredge6600? quad xeon cpus (still 32bit) and a few p3 era 1u servers. preiously had some old proliant and a p90 prosignia. my favorite server i have ever had though was an ibm netfinity pentium pro200 with 4GB ram for its age it was a decently quick machine
We have these servers (and their predecessor x3650 (without an „M“ designation) and successor Generations M3 - M5) at work. I used to service them but apart from swapping Hard Disks, Power Supplies, RAM, CPUs, PCIe Cards and the occasional SAS controller I never got to taking one apart this far. Thanks for taking us on this journey! In terms of diagnostics you might check the MM (Management Module) of this beast. It‘s IBM‘s way of Lights Out Management and if you manage to access it it might provide more diagnostic output than the LEDs. It should have a Web Interface (terribly slow and not exceedingly informative (you may want to check the event log, though)) but also telnet / ssh access which lets you execute diagnostic commands. As a firefighter I‘d say that gunk is very probably smoke particles of a major fire incident nearby and if water was used to extinguish it could explain the water damage mainly on the outside, too.
Thanks for watching and the extra info! I haven't actually tried to access the IMM yet, that's one of my next steps. I suspect it's from an auto shop... the dirt/grime didn't smell burnt and basically the only thing that could get through it was water and brake clean lol.
Careful with that brake cleaner, it eats rubber / plastics. I believe the ABS sensor safe versions are better for this but I would still avoid it near electronics.
When cleaning old boards, I like submerging in hot soapy water and letting it soak for a while, and then scrubbing with a toothbrush. Then I give it a bath in alcohol while hitting it with a toothbrush again. For rust, I sand what I can off and then dump some vinegar on it and clean up again with alcohol. After it dries for a day or so, it's good to go. I'm not sure how that would fair with that black gunk though.
At least it looks way better than my Sun X4500, that was in good condition, when ordered, but then the delivery guy let it fell from a pallet , even cpu socket was ripped off
I've had quite a few IBM systems over the years, and had a few of them blow transistors on the power rail. So that would be my guess of what happened. Check the board LEDs to see how far the power gets and do a smell test.
6:14 That is the feature enable hardware key for the IMM advanced license. Without that, you don't get access to some features in the management system, I think including virtual media support. As for the BRD error, I had that with my 3650m3..... I spent days working on it until finding out that one of the risers just wasn't all the way into the slot. It damn near SNAPS down into the slot. When you think you have it in, then *lean* on it more. Well. You have part two up, so I'll have to see if you figured that out by then. :)
A fun new video from the handsome Clabretro guy! Seeing the grime in that system makes me glad I don’t work at a dealership. Imagine breathing that every day. Now I know what that dealership smell is. Regardless, thanks for sharing. I’m curious if you’ll be able to resurrect it. Maybe this system failed while in service and they moved it to some leaky storage area before disposing of it. Your research showed what looked like the name of a Chrysler dealership. Could ADP hunt them down for not destroying the drives? Hmm. Suspense, drama, intrigue!
Recently got one of the 3650 M3 versions, interesting seeing the differences and similarities, mine did have a hyper visor key in that USB slot on the rider so that is what it’s for
You ever thought about getting one of them electric cleaners? Would be good for the smaller boards and things. They really clean things good. They probably have ones for the bigger board but I am sure they are very expensive. Or put it in your dish washer (I hear people do that with motherboards). Also I hear BW100 is great for cleaning electrical stuff. It's non conductive and drys like IPA if not faster.
This experience made me really think about it. I should've either hosed it down or ran through the dish washer in hindsight, really would've sped things up.
I used to own a X3650 M5 and that steady power light blink is normal...if the server is "not ready to power on" it blinks faster at you lol...BTW the IMM messages you saw are the IBM version of Hp ilo Dell Idrac etc. IMM needs to be happy otherwise the fans go full tilt like they did in the video. if you plug it in and wait about 3-5 minutes or so for IMM to fully boot it shouldn't blast off like that-that would be something else to check try to connect to the IMM webUI it think it defaults to DHCP...
This thing probably was in the shop area of the car dealership (probably mounted on a rack that hung from the ceiling) so this is malaise era Chrysler dust
Defiantly a hit to SLA's. This is one of my biggest complaints with IBM/Lenovo hardware. Takes forever to boot up. Dell is faster, HP is faster (at least the machines in our DC)
Next time if you encounter a such dirty motherboard, just put in the dishwasher. Before that remove the batteries and drain any leftover electricity from the board. Dont use any detergent, just water. Without any electrcity in the board it wont corrode. Der8auer use this method after his sub-zero OC sessions to remove vaseline from his boards and cards.
if you want a way to protect the steel after rust and cleaning has taken the plating off, and don't want to bother with painting, get a wax-based anticorrosive coating (like boeshield). most of that class of products have matte finishes when dry that won't look too weird, beside the visual texture of the stripped zinc plating.
oh, also, if you want a more durable version of the CHP-170 (read; survives abuse like cutting 16ga braided stainless wire rope), CHP/Hakko now makes a version with hardened toolsteel blades, the TR-30-58-R.
This has more than a slight resemblance to the Lenovo RD220 which is "the server" where I work. Turns out it wasn't being used for much so I shut it down this year and nobody was affected. As part of a Wi-Fi upgrade I then installed a same-generation HP DL320 G6 with Proxmox to run the UniFi controller and an OpnSense instance. These old servers run the exact same software as new servers so I don't consider them retro. If they supported software (or hardware gizmos) not usable on modern systems then I would call them retro. Actually then I'd call it vintage but whatever. It's nice to see stuff come back from being junk no matter what you call it.
20:22 - the connector is there for reasons same as the USB one (and this is likely SAS not SATA) - internal tape drives usually come with SAS (LTO and stuff like that) or USB interface (for DAT tape drives).
I have one of these minus the ADP branding in working condition heading for the scrap heap in 3 weeks time. If there happens to be any parts you need I'm sure I can arrange shipping.
ADP. Dealer Management System. It is car dealership server managed by ADP Dealer Services. ADP DMS would be the "IT department" or supplement support for the ADP hardware like in my case for the in house IT I used to manage the worlds largest dealership deployment back in the day. (98-03) Had cool toys like Robot WORM drive storage. Vertical conveyor belt/elevator storage for car parts. (like a giant Carvana vending machine) WRQ Reflection custom programming.. Y2K... I met Ralph, the HMFIC of ADPDS at the time. It was fun times until they gave the IT Director job to a buffoon con-man car salesman.. Back stabbing industry. Almost every conceivable function of a car dealership/service is ran from a server like that. We used a dual CPU DEC Alpha and some Unix to run over 1.2k VT200 terminals for all the individual dealerships we managed. From sales/accounting to the technicians looking up parts in the service department. Multiple locations.
YIKES!!!! Looks like it was used in an auto shop and was stored in the back room under the engine cores and used oil tanks! That's alot of filth and schmutz!
Brakes, Tires and Lubes. Each having their own effect on air quality in an environment such as a mechanics shop. Used oil even when it is cold has significant off gassing of everything from micro particles of wear parts metals, oil additives to unburnt gasoline. anti-freeze and coolants evaporate. brake cleaner rapidly evaporates let alone the break pad dust. tire rubber new and used off gasses constantly, and the anti sun deteriorate chemical has its own special kind of funk. exhaust soot. heck even rust dust. and to top it all off the normal skin, hair, and dust from outside. If I can recall there was a university study of mechanics shops and the effect on health it has for the mechanics. it can take off 10 to 15 years of life working a mechanic job. Even a lube tech, tire tech, even service advisors are affected. fresh air is mandatory.
Maybe it suffered from "basement/warehouse damage" high humidity enviroment, atleast I'd think that real waterdamage would look worse? Also for next time cleaning a dirty machine: Nitril gloves, the ones for carmechanics that offer extra grip, don't wanna risk a cut from sharp metal with all that grime on your hands.
I have a slightly older 3650 that complains about the raid battery being dead and wont boot past that even with raid disabled. It is aggravating as a battery is more expensive than the thing is worth. .
Yea ADP and the Okidata branded ADP printers too. If anybody wants to go down a costly rabbit hole. The upside was all the equipment, software, etc etc. was already per-configured to plug in and mostly go. But, if you was just to purchase the hardware separate in some cases. It was $$$ less. If they would give you that option. I don't know about ADP but most of those type of systems started out on Mini computers and moved into x86 systems due to cost saving. Well for the company mfg. the software / hardware. Not really towards the end customer.
what a beaut of a server! Is using contact cleaner safe on CPU sockets? Those things are so fragile, anything will bend a pin. Regardless, nice to see IBM including those handy little diagnostic lights, it's just cool to have for what ever reason even when the system works.
Definitely a good looking server. I actually didn't spray down the sockets, I left them covered to make sure nothing got in there. They were actually really clean inside, I forgot to film that.
Is it possible this server was in the vicinity of a fire, and maybe got splashed with water? Or has someone just run it in their garage for a few years?
Looks like it was running at a dealership. Their server closet probably was just set it in the back with the mechanics where theres soot and brake dust all over. Probably a carcinogen.
haven't gotten anything out of the serial port yet but it does have an IMM which appears to function well, even though the machine won't boot anymore. so that'll be the next investigation point!
@@clabretro the IMM should be functional after power initialization at least and might have more error or diag details. What is displaying on the LPD now?
The IMM appears to be functional (in fact it has that ADP IP address assigned to it), and I get the slow 1 blink per second on the front power light and on the IMM LED. But when I press the power button the button blinks quickly and nothing happens. The LED on the LPD is just scrolling as you see towards the end of the video. *Occasionally* the LPD will show the BRD LED again. I'm thinking I need to clean contacts + reseat everything. LEDs on the power supply are all green.
it definitely got wet but I'm thinking autoshop based on the grime. I can't remember if I mention in the video but it doesn't smell burnt... just very musty
Have you removed the CPU and cleaned beneath it? All that washing gunk around probably washed some into the LGA socket... something might be shorting out under there.
@@clabretro f4 with brd usually a indicator there is a problem as you changed hte battery and have the riser slots installed but i am curious what would happen if you tried to put the CPU in the other socket....ive had a server pretty much die myself as the x16 slot started malfunctioning and onboard graphics on that server would not work with plex at all likely but alteast not everything will go to waste
Yeah cleared the CMOS and all that, experimented with different RAM sticks. It doesn't power up at all anymore, which could indicate some voltage regulators maybe having gone bad during that first boot up (hopefully not though). Going to re-seat the CPU next.
At this point I'm thinking it was in an auto shop for a long time and also got hit with some water damage. It smelled really musty rather than burnt or sooty.
I got one HP proliant dl 360p gen 8 and 2 Proliant dl380P gen 8 but rgey say that IBM is more easy to work whiy i dont know but i will get one some day...What do people do whit there computers/servers????....where have this server been ? in a factory?...brrrr....do people know abiut taking care of there servers? clean them etc...they work longer if they take care of them..By the way do you know about ILO? My 360 does not find the ILO and the IP adress and i have try every thing but stil and i am not a technican so i dont know what to do or search...
I have a DL380p Gen 8. If you are having trouble getting into the iLO 4 you can try entering BIOS setup (or RBSU as HP calls it) where you should have options to reset the iLO password and network configuration. You can also perform some resets via the DIP switches on the motherboard in case you can't get into the configs. IIRC details on the DIP switches are printed on the lid.
I have try the bios but when i start the server it says something about not started i will check again i dont remember but thanks for answer...@@eDoc2020
As a former mechanic, the dirt remind me the kind of dirt that you can find in automobile workshop, a mix of dust (from brake pad etc) and old grease/oil sludge.
that's also what I was thinking, and that tracks if maybe it was at an actual dealership running some management software
The fact that it says ADP DMS on the front is a good indication it was definitely installed in a dealership setting. That software group is created specifically for dealer management/repair order management systems.
@@clabretro Probably shoved the server rack in the garage because it's loud out there anyway
i think so too. i have seen enough servers put into cnc-shops, dripping with oil. even ibm decided to no longer waranty exchange the fans, it took them 2 full sets^^
@@dstrickner123timeline works. The company that is now “CDK” used to be part of ADP. They were (and are) notorious for requiring equipment leasing and not letting the Dealer own anything.
This is a server from "CDK" (formerly ADP). DMS = Dealer Management System. They are often found at car dealerships in North America. Past few years they have moved 'serverless' now relying on SD-WAN. I encounter these all the time. As far as the IBM 3650m2 - it was a solid machine. I still have one running at my office, we will decommission it. The IP address on the front of the server is likely not reachable via the Internet. CDK (ADP) used a Cisco WAN to extend that IP address out to the dealership. Controls printers and communication to/from the CDK data center.
Interesting! Yeah it's definitely looking like exactly what you're describing. Cool to learn a little more about it.
@@clabretro I think ADP sold all of their DMS stuff to CDK. CDK has now dropped their Cisco router based stuff and switched to Cisco Meraki. Those IP addresses are only accessible via the Velocloud SD-WAN, not the Internet. I’m not 100% certain when ADP/CDK stopped using the servers (depends on the dealership) but pretty sure everything went SD-WAN about 5 years ago so I doubt there is anything newer on that machine than 2016/2017 best case scenario (if the dealership upgraded)
Ahhh this is perfect love this older server hardware as a 23 year old in a data center
100% car dealership server for ADP's dealer management service (DMS). That exhaust dust gets everywhere and sticks. They spun off the DMS under their subsidiary, CDK.
I think you're right, all signs are pointing towards that
Just came here to say this. At least this one doesn't have pigeon poop or rat droppings on it...
Former CDK/ADP User here crazy how nothing has changed on the software side
My first thought with this is smoke damage. While this server obviously wasn’t burning, I wonder if something near it caught fire and had to be extinguished. That might explain the minor moisture damage and soot (maybe?) inside. Enjoying the restoration and looking forward to part 2.
someone else in the comments said it might be exhaust pipe smoke which would make a lot of sense
My hands started drying up uncomfortably while I was watching you clean the thing with IPA and contact cleaner. Wear gloves my man!
Awesome job, regardless of the hand-drying horrors haha :)
haha I know
Who gives a fuck about dry hands? LOL
sometimes you gotta cover your hands in volatile cleaning chemicals. keeps the hair growin’ on your chest
i still have one of these on service for a customer, the CISCO MCS 7800 / IBM X3650 M2, the cisco label one, what a realible machine
That server looks like it was though a fire. The grime is soot and smoke residue and the water damage is from sprinklers. That would be my guess at least.
I'm thinking maybe an auto shop, it smells musty rather than burnt. Definitely got wet though!
Hey, me again... So what you have here is a server that ran a DMS, dealer management system. I am not sure what the product was called when ADP owned it, but nowadays the division was sold off to a company called CDK Global, and the new iteration of the software is called Drive. I haven't watched the video yet but it will be running a heavily customized version of CentOS (or maybe RHEL) I believe. This was absolutely located in a dealership of some kind, I am intimately familiar with the software. The software manages the entire dealership from service department, to part sales and inventory, HR and payroll...all text menu (green screen) driven. There is almost certainly data on those drives. I actually work for a dealership (as IT) and one of the first things we did during my employment was migrate away from this CDK drive into a platform called Excede. The dealership had migrated away from autonet years prior in favor of SD-WAN. Autonet was really just T1. I still have a working instance of drive running if you wanted a closer look, but its not very exciting stuff.
Hey! Yeah agreed, all signs point towards that. Interesting to hear the dealerships were actually using autonet as well, I guess that all adds up then.
@@clabretro Pretty funny you make videos on two obscure things I'm most familiar with... old pSeries hardware and DMS software... Thinking this channel may just be made by an AI trained on my browsing habits over the last 15 years. Now thats a scary thought.
Haha I was thinking the same thing.@@willross6468
@@clabretroweird little blue thing is a "feature key" by the way... All it does is unlock extra features of the "IMM2" module, IBM's IPMI. I know it unlocks virtual media support. It might be needed to allow remote KVM console, I cant remember. Without it I think the IPMI is limited to power control and reading vital product data, event logs, temperatures and stuff.
Neat. ADP sold a lot of these to Dealerships. They provided their software to service and body shop, estimate software for certified dealers. Incredible estimating software.
The fact you didn't wear gloves really proves there is a thin line between madness and bravery
Had to do it 😆
I appreciate you courting old-age cancer for our entertainment today. Your workshop is now a superfund candidate.
I do what I can 😆
AHHH, I love this! When I was working, (part time) when I was 12, the company was using X3650 M4's for their file servers, and even Active Directory :P on Server 2012 R2, in 2018... It's a small company but janky... They got a Synology iSCSI with only 1 PSU running, which was like damn... Great video btw. Awesome job.
A RUclips channel called My Playhouse is an awesome channel, messed with hardware such as the M1, M2 and other servers.
oh yeah I watch all his videos!
@@clabretroquality content!
@@LeeZhiWei8219I bought my x3650 m5 cause of him.
This is the server I use for my home NAS, the main difference being its branded by Cisco. Happy to see you restoring it, it's a nice piece of hardware.
Looks like there was a datacenter fire, and the rust on top was from sprinklers.
I thought the same. The black stuff looks like soot from the smoke. Surely absolutely perfekt for your skin 😅
could be!
Yep, that's my vote too.
Servers tend to pull air through from the front to the back.
Likelihood a small electrical fire in this, but more likely an adjacent rack would have had the soot and combustion products suckered into the box for the seconds / minutes the power remained on.
extinguishant products (possibly foam for "electrical safety"... don't guarantee the safety OF your electrical equipment and you'll find a number of racks end up being "collateral damage" depending on the incident size.
Most data floors don't use sprinklers due to all the electronics, but the fire service might use foam or water, again, depending on the severity and risk of further damage.
Losing a rack full of servers isn't as bad as losing the whole floor and possibly floors either side!
I would definitely recommend against breathing it in though, that's for sure. Using a wet cleaning method was for the best, though as others have said..... Gloves!!! Lol
It might also have been in an auto dealership, specifically in the repair area. It smelled musty, rather than any burnt smell, but it certainly got wet!
The dirt in it makes sense to be related to a dealership, when I saw DMS at first I thought Document Management System.
For that gunk I probably would've chanced an "Adrian's Digital Basement" style wash with dish soap, water, and a very soft bristled brush for that mainboard, then an IPA dunk for drying. Yikes! Either way, nice cleanup job, that's a big board with a lot of things to go wrong.I think the theories of a datacenter fire are likely true, either that or a long stay in a stack in a humid environment with a lot of other crap in the air (my first thought was also that that grime looks a lot like auto shop/garage grime).
Very happy to have this channel recommended to me by the YT algo (it gets it right every once in a while!). Excellent mix of expertise, curiosities, and that liitle bit of jank that every good homelab should have.
Thanks! Yeah I was initially going to wash it in soapy water, in hindsight I should've just gone ahead and done that.
IPA is great for breaking down chemicals, but for general dirt and grime you really need water to break things down.
@@xDownSetx yeah should've stuck with my original sink wash plan
"I dunno what that is" at 6:20 It's an upgrade key, usually either for the better RAID modes or the IPMI. Without it, the associated module will not be as fully functional, or maybe not work at all. Think of it as a hardware DLC.
Ah, interesting. thank you!
ADP had a sewerage pipe burst at a data center back in 2011 in Ohio. Caused an outage if I remember correctly.
I hope that's not what happened to this one 😆
The excitement I am feeling, unmatched, new clabretro video, let’s go
I love the industrial design of servers
agreed!
Loving the content. As a former techie that now rarely gets to touch the hardware part of the environment, this really is the stuff.
thanks for watching!
Never have I felt smarter than watching you take the tab off the LPD, thinking 'you should cut the ramp off the latch and put the tab back on as a puller' then seeing you do exactly that haha
haha. great minds think alike.
I had been feeling like this machine had spent some time as a homelab in a basement or garage from the looks of the residue in there, and I feel like an auto plant (or even a larger dealership) tracks with that, like you're seeing a decade of buildup from oil changes and radiator flushes.
that's what I'm thinking too
Man, that server was grungy! I've pulled a few servers apart that had a little bit of that filth with close to 10 years of service, but nothing like that! Wow!
Easily the worst one I've ever dealt with, nothing even comes close haha
perfect timing for my food
That looks like 1000 people smoked a pack of smokes near that server. ACK. Great video
Man, I love enterprise gear! Everything is so thought through and modular.
thats the younger brother to systems we still have in daily prod, yes the support contract is expensive to cover them. we have x3550s
I have in my rack although i rarely power them on to noisy and slow compared to my other old junk, also have 3 dell pweredge6600? quad xeon cpus (still 32bit) and a few p3 era 1u servers. preiously had some old proliant and a p90 prosignia.
my favorite server i have ever had though was an ibm netfinity pentium pro200 with 4GB ram for its age it was a decently quick machine
In 2015, I got 3 of this machine and install Proxmox VE 5 (I guess) at that time, to create a Proxmox Cluster. It worked pretty well till recently.
We have these servers (and their predecessor x3650 (without an „M“ designation) and successor Generations M3 - M5) at work. I used to service them but apart from swapping Hard Disks, Power Supplies, RAM, CPUs, PCIe Cards and the occasional SAS controller I never got to taking one apart this far. Thanks for taking us on this journey!
In terms of diagnostics you might check the MM (Management Module) of this beast. It‘s IBM‘s way of Lights Out Management and if you manage to access it it might provide more diagnostic output than the LEDs. It should have a Web Interface (terribly slow and not exceedingly informative (you may want to check the event log, though)) but also telnet / ssh access which lets you execute diagnostic commands.
As a firefighter I‘d say that gunk is very probably smoke particles of a major fire incident nearby and if water was used to extinguish it could explain the water damage mainly on the outside, too.
Thanks for watching and the extra info! I haven't actually tried to access the IMM yet, that's one of my next steps. I suspect it's from an auto shop... the dirt/grime didn't smell burnt and basically the only thing that could get through it was water and brake clean lol.
That‘s entirely possible, too - either way it must have been neglected for a very long time. I‘m happy to see you bringing it back to life.
Careful with that brake cleaner, it eats rubber / plastics. I believe the ABS sensor safe versions are better for this but I would still avoid it near electronics.
Oh yeah definitely, I only used it on the metal chassis.
When cleaning old boards, I like submerging in hot soapy water and letting it soak for a while, and then scrubbing with a toothbrush. Then I give it a bath in alcohol while hitting it with a toothbrush again. For rust, I sand what I can off and then dump some vinegar on it and clean up again with alcohol. After it dries for a day or so, it's good to go. I'm not sure how that would fair with that black gunk though.
Totally agree, in hindsight I should've washed the mainboard that way. I washed all the other stuff with water and it cleaned up much better.
At least it looks way better than my Sun X4500, that was in good condition, when ordered, but then the delivery guy let it fell from a pallet , even cpu socket was ripped off
That's the worst. Happens to so many of these servers.
I've had quite a few IBM systems over the years, and had a few of them blow transistors on the power rail. So that would be my guess of what happened. Check the board LEDs to see how far the power gets and do a smell test.
yeah I'm hoping it's something power related early on
6:14 That is the feature enable hardware key for the IMM advanced license. Without that, you don't get access to some features in the management system, I think including virtual media support.
As for the BRD error, I had that with my 3650m3..... I spent days working on it until finding out that one of the risers just wasn't all the way into the slot. It damn near SNAPS down into the slot. When you think you have it in, then *lean* on it more.
Well. You have part two up, so I'll have to see if you figured that out by then. :)
Great video and channel! The m4 series boxes are great. I have a x3550 m4 (the 1U version of this box) and it’s running great to this day.
Thanks! Yeah those M4s look nice!
A fun new video from the handsome Clabretro guy! Seeing the grime in that system makes me glad I don’t work at a dealership. Imagine breathing that every day. Now I know what that dealership smell is. Regardless, thanks for sharing. I’m curious if you’ll be able to resurrect it.
Maybe this system failed while in service and they moved it to some leaky storage area before disposing of it. Your research showed what looked like the name of a Chrysler dealership. Could ADP hunt them down for not destroying the drives? Hmm. Suspense, drama, intrigue!
Recently got one of the 3650 M3 versions, interesting seeing the differences and similarities, mine did have a hyper visor key in that USB slot on the rider so that is what it’s for
You ever thought about getting one of them electric cleaners? Would be good for the smaller boards and things. They really clean things good. They probably have ones for the bigger board but I am sure they are very expensive. Or put it in your dish washer (I hear people do that with motherboards). Also I hear BW100 is great for cleaning electrical stuff. It's non conductive and drys like IPA if not faster.
This experience made me really think about it. I should've either hosed it down or ran through the dish washer in hindsight, really would've sped things up.
Such a dog to work on, you have my sympathies mate!
I used to own a X3650 M5 and that steady power light blink is normal...if the server is "not ready to power on" it blinks faster at you lol...BTW the IMM messages you saw are the IBM version of Hp ilo Dell Idrac etc. IMM needs to be happy otherwise the fans go full tilt like they did in the video. if you plug it in and wait about 3-5 minutes or so for IMM to fully boot it shouldn't blast off like that-that would be something else to check try to connect to the IMM webUI it think it defaults to DHCP...
thank you, very useful! was wondering about the slow vs. fast blink. my next step is indeed trying to talk to the IMM
Evaporust would work great on that big metal sheet.
Unfortunately I was out! ha
This thing probably was in the shop area of the car dealership (probably mounted on a rack that hung from the ceiling) so this is malaise era Chrysler dust
Yup, I think you're right!
I used to deploy these systems... My main gripe was always the absolutely ridiculous BIOS boot time
Defiantly a hit to SLA's. This is one of my biggest complaints with IBM/Lenovo hardware. Takes forever to boot up. Dell is faster, HP is faster (at least the machines in our DC)
Next time if you encounter a such dirty motherboard, just put in the dishwasher. Before that remove the batteries and drain any leftover electricity from the board. Dont use any detergent, just water. Without any electrcity in the board it wont corrode. Der8auer use this method after his sub-zero OC sessions to remove vaseline from his boards and cards.
yeah I will, I should've at least started with the sink on this one!
yay a new video
the systems are always cool but i do agree that that is a bio hazard! Great video!
thanks!
if you want a way to protect the steel after rust and cleaning has taken the plating off, and don't want to bother with painting, get a wax-based anticorrosive coating (like boeshield). most of that class of products have matte finishes when dry that won't look too weird, beside the visual texture of the stripped zinc plating.
oh, also, if you want a more durable version of the CHP-170 (read; survives abuse like cutting 16ga braided stainless wire rope), CHP/Hakko now makes a version with hardened toolsteel blades, the TR-30-58-R.
Yeah that's a great idea, and thanks for the tip on the TR-30-58-R, I do need a better one.
Got 2 R410’s one was a little dirty the other one was something like this machine but without the rust.
6:15 That is an xclarity license key and may be the only part worth anything from that machine. Good luck!
Ha yeah this board might be toast. Thanks!
Typical IBM. When they work, they almost bullet proof. But when they break, it's ugly. Beautiful machine.👍
I've pulled stuff from a sewer before, but this server was just dirty. What does an ADP datacenter look like? :) Great videos by the way, I'm hooked!
haha. thanks!
This has more than a slight resemblance to the Lenovo RD220 which is "the server" where I work. Turns out it wasn't being used for much so I shut it down this year and nobody was affected. As part of a Wi-Fi upgrade I then installed a same-generation HP DL320 G6 with Proxmox to run the UniFi controller and an OpnSense instance. These old servers run the exact same software as new servers so I don't consider them retro. If they supported software (or hardware gizmos) not usable on modern systems then I would call them retro. Actually then I'd call it vintage but whatever. It's nice to see stuff come back from being junk no matter what you call it.
yeah I wouldn't call this one "retro" quite yet either
20:22 - the connector is there for reasons same as the USB one (and this is likely SAS not SATA) - internal tape drives usually come with SAS (LTO and stuff like that) or USB interface (for DAT tape drives).
I have one of these minus the ADP branding in working condition heading for the scrap heap in 3 weeks time. If there happens to be any parts you need I'm sure I can arrange shipping.
Oh yeah... if you don't mind, hit me up at the email in the channel's about page.
Since this server is the nastiest I've EVER seen. I didn't catch on on vid...but check CPU the Xeon itself and CPU socket 1 for gime-ness lol
I forgot to show that, CPU was perfectly clean!
ADP. Dealer Management System. It is car dealership server managed by ADP Dealer Services. ADP DMS would be the "IT department" or supplement support for the ADP hardware like in my case for the in house IT I used to manage the worlds largest dealership deployment back in the day. (98-03) Had cool toys like Robot WORM drive storage. Vertical conveyor belt/elevator storage for car parts. (like a giant Carvana vending machine) WRQ Reflection custom programming.. Y2K... I met Ralph, the HMFIC of ADPDS at the time. It was fun times until they gave the IT Director job to a buffoon con-man car salesman.. Back stabbing industry. Almost every conceivable function of a car dealership/service is ran from a server like that. We used a dual CPU DEC Alpha and some Unix to run over 1.2k VT200 terminals for all the individual dealerships we managed. From sales/accounting to the technicians looking up parts in the service department. Multiple locations.
YIKES!!!! Looks like it was used in an auto shop and was stored in the back room under the engine cores and used oil tanks! That's alot of filth and schmutz!
definitely in an auto shop, it was really grimed on there
You get the award for the dirtiest hands from working on a computer. Cool restoration project
Brakes, Tires and Lubes. Each having their own effect on air quality in an environment such as a mechanics shop. Used oil even when it is cold has significant off gassing of everything from micro particles of wear parts metals, oil additives to unburnt gasoline. anti-freeze and coolants evaporate. brake cleaner rapidly evaporates let alone the break pad dust. tire rubber new and used off gasses constantly, and the anti sun deteriorate chemical has its own special kind of funk. exhaust soot. heck even rust dust. and to top it all off the normal skin, hair, and dust from outside.
If I can recall there was a university study of mechanics shops and the effect on health it has for the mechanics. it can take off 10 to 15 years of life working a mechanic job. Even a lube tech, tire tech, even service advisors are affected. fresh air is mandatory.
well this thing was hit with all of the above haha
Yup, 13 years ago feels like yesterday. When I had hair.
Maybe it suffered from "basement/warehouse damage" high humidity enviroment, atleast I'd think that real waterdamage would look worse? Also for next time cleaning a dirty machine: Nitril gloves, the ones for carmechanics that offer extra grip, don't wanna risk a cut from sharp metal with all that grime on your hands.
Much to surprised, this was quite older.
Bling-Bling. That was so non-IBM-like thing to see. :) Funny
It looks like it was in a data center fire and the damage on the case looks like it got hit by water instead of halon like most data enters use
I'm thinking it was in an auto shop and then got some water damage at some point as well.
Great video
Thanks!
When doing restoration like that it is good to use vinegar and baking soda
I have a slightly older 3650 that complains about the raid battery being dead and wont boot past that even with raid disabled. It is aggravating as a battery is more expensive than the thing is worth. .
I bet if you would turn it on before cleaning it would work flawlessly :)
Unfortunately I think you're right lol
I would have tried to JB weld the button for the light path module
Oh yeah that would've been a good option. I really need to have some of that on hand.
i just use a hammer each time my server bricks it self surprisingly it boots right after.
this one might need that
This could well have been the result of a building fire. Some fire suppressions can leave residue dunno though obviously
Yea ADP and the Okidata branded ADP printers too. If anybody wants to go down a costly rabbit hole. The upside was all the equipment, software, etc etc. was already per-configured to plug in and mostly go. But, if you was just to purchase the hardware separate in some cases. It was $$$ less. If they would give you that option. I don't know about ADP but most of those type of systems started out on Mini computers and moved into x86 systems due to cost saving. Well for the company mfg. the software / hardware. Not really towards the end customer.
what a beaut of a server! Is using contact cleaner safe on CPU sockets? Those things are so fragile, anything will bend a pin. Regardless, nice to see IBM including those handy little diagnostic lights, it's just cool to have for what ever reason even when the system works.
Definitely a good looking server. I actually didn't spray down the sockets, I left them covered to make sure nothing got in there. They were actually really clean inside, I forgot to film that.
Contact cleaner is definitely safe on CPU sockets. I've done it a few times myself when troubleshooting.
👍👍
These ADP servers used to run a software called DMS which was programmed in I think Kerridge KCML which was a terrible programming language.
never heard of KCML... looks rough! haha
Is it possible this server was in the vicinity of a fire, and maybe got splashed with water?
Or has someone just run it in their garage for a few years?
Thinking an auto shop for a long time, then some water damage on top of that.
Water damage and a black soot like dust?…..perhaps it was in a building that had a fire ?
I'm thinking auto shop then some sort of water damage, it didn't smelly sooty or burnt at all, just really musty.
Looks like it was running at a dealership. Their server closet probably was just set it in the back with the mechanics where theres soot and brake dust all over. Probably a carcinogen.
Yeah I'm thinking it was in an auto shop or something
Other than the LPD providing a code, doesn't this model have a management port and interface? The default userID and password can prob be reset.
haven't gotten anything out of the serial port yet but it does have an IMM which appears to function well, even though the machine won't boot anymore. so that'll be the next investigation point!
@@clabretro the IMM should be functional after power initialization at least and might have more error or diag details. What is displaying on the LPD now?
The IMM appears to be functional (in fact it has that ADP IP address assigned to it), and I get the slow 1 blink per second on the front power light and on the IMM LED. But when I press the power button the button blinks quickly and nothing happens. The LED on the LPD is just scrolling as you see towards the end of the video. *Occasionally* the LPD will show the BRD LED again. I'm thinking I need to clean contacts + reseat everything. LEDs on the power supply are all green.
@@clabretro maybe just start with cpu and mem and see if it will initialize ok.
I start looking for part two and then I noticed that this video is just one day ago. I mistakenly overlooked and thought it was one year ago. Crap!
Insurance Loss Adjuster here.. This machine has been in a fire and then sat for a month or so in a place with no or partial roof.
it definitely got wet but I'm thinking autoshop based on the grime. I can't remember if I mention in the video but it doesn't smell burnt... just very musty
This server has seen some shit.
Have you removed the CPU and cleaned beneath it? All that washing gunk around probably washed some into the LGA socket... something might be shorting out under there.
It's actually very clean (visually) but I think I'm going to clean it anyway when I troubleshoot some more.
@@clabretro Ehh, well if it looks clean, maybe it's better to leave it be then... Bending those LGA pins would be a nightmare...
9:39 not the connectors😩
heh yeah they survived. got the brake cleaner right off of them with IPA after I was done. They were stuck in there anyway 😆
It must of been in some sort of Fire
Thinking an auto shop for a long time
shocked it only had a single CPU and 2 1gb dimms. ddr 3 or 4 at least? :p
DDR 3 at least, but still pretty sad 😆
that server might be toast with a error code of e7 not shown anywhere while there is a possibility of voltage regulator has failed
yeaaaah it's definitely possible a voltage regulator is bad
@@clabretro f4 with brd usually a indicator there is a problem as you changed hte battery and have the riser slots installed but i am curious what would happen if you tried to put the CPU in the other socket....ive had a server pretty much die myself as the x16 slot started malfunctioning and onboard graphics on that server would not work with plex at all likely but alteast not everything will go to waste
Yeah cleared the CMOS and all that, experimented with different RAM sticks. It doesn't power up at all anymore, which could indicate some voltage regulators maybe having gone bad during that first boot up (hopefully not though). Going to re-seat the CPU next.
@@clabretro fidning wich ones that are bad will be a pain
usually they are dusty this is just gross.
IBM has always over engineered their hardware and it is always built like a tank. Can’t say that about most server hardware these days.
looks like fire damage.
what is the blue part next to the battery?
Another commentor mentioned some sort of hardware DRM module to unlock/lock features.
Hi @clabretro, i have a clean and running x3650 M3 System, if you are interrested. The only downside is the shipping from germany^^"
Thank you! Unfortunately yeah the shipping will be too much, but I really appreciate it!
@@clabretro if you need any help with IBM or Lenovo enterprise equipment, let me know... I repair those for living
Thanks! This M2 is going to be tricky... won't boot anymore. Only done basic troubleshooting but hoping it's not voltage regulators!
almost looks like its been in a fire
At this point I'm thinking it was in an auto shop for a long time and also got hit with some water damage. It smelled really musty rather than burnt or sooty.
that makes sense I guess@@clabretro
My guess would be smoke/melted plastic particulates that get sucked up by fans during a fire. Horrible stuff.
I got one HP proliant dl 360p gen 8 and 2 Proliant dl380P gen 8 but rgey say that IBM is more easy to work whiy i dont know but i will get one some day...What do people do whit there computers/servers????....where have this server been ? in a factory?...brrrr....do people know abiut taking care of there servers? clean them etc...they work longer if they take care of them..By the way do you know about ILO? My 360 does not find the ILO and the IP adress and i have try every thing but stil and i am not a technican so i dont know what to do or search...
Not sure about the ILO, I've never actually used any HP machines newer than Gen 1 haha.
I have a DL380p Gen 8. If you are having trouble getting into the iLO 4 you can try entering BIOS setup (or RBSU as HP calls it) where you should have options to reset the iLO password and network configuration. You can also perform some resets via the DIP switches on the motherboard in case you can't get into the configs. IIRC details on the DIP switches are printed on the lid.
I have try the bios but when i start the server it says something about not started i will check again i dont remember but thanks for answer...@@eDoc2020
look like it was in a fire
Thinking it was in an auto shop for a long time and also got hit with some water damage, it smelled really musty rather than burnt or sooty.
i see hope for the best keep it up
take contact,i have system x experience...