My Server Flooded
HTML-код
- Опубликовано: 7 сен 2024
- lmg.gg/secretl...
Thanks to Secretlab for being the title sponsor of LMG Clips this year! Check out their ergonomic gaming chairs at the link above. Your back will thank you!
Linus’ server racks flooded.
Watch the full WAN Show: • Elon Musk Admitted I W...
► GET MERCH: lttstore.com
► GET EXCLUSIVE CONTENT ON FLOATPLANE: lmg.gg/lttfloa...
► SPONSORS, AFFILIATES, AND PARTNERS: lmg.gg/partners
► OUR WAN PODCAST GEAR: lmg.gg/wanset
FOLLOW US ON SOCIAL
---------------------------------------------------
Twitter: / linustech
Facebook: / linustech
Instagram: / linustech
TikTok: / linustech
TikTok (LMG Clips): www.tiktok.com/@_lmgclips_
Twitch: / linustech
Didn’t know about his sister. Sorry to hear that Mr Sebastian
Damn. He had his sister on the show once on a build video. Is that her?
@@FrietjeOorlogno, he told it wasn't her
@@FrietjeOorlog No, he addressed it in one or two WAN's ago, that it wasn't the one in the video
yeah that hit me like a gut punch. My condolences, if you read this Linus
@@misterpuff797 thank god, the one in the show was so young, hope it wasn't premature tho
Sorry to hear about your sister.
Can someone explain?
@@DRSDavidSoft Linus had an unexpected death in his immediate family that he talked about a little in a previous WAN show, turns out it was his sister. He mentions it pretty quickly in this video.
@@TheSencatShit is that the sister we saw on video?
@@TheSencat People need to acknowledge why this is happening so frequently so it can finally STOP!
@@dakoderii4221 It's too late to stop it. At best, we can place the blame where it belongs.
My condolences Linus. I can't imagine the pain of losing a sibling.
May she rest in peace.
I'd watch that "resurrecting my water damaged UPS" video. It actually sounds super cool
Sounds more like a fire hazard, so super hot actually.
Something like a UPS you just dont risk, same goes for a PSU. Its safer, depending on why it died, to just get a new one. Water damage can make stuff do fucky things, and a UPS flowing quite a lot of power can still fail after some time even if its "resurrected"
There's not much to a UPS. If it's just him swapping internals from a donor there's no real risk as long as the chassis is completely dry.
@@renderedpixels4300 As long as you are competent you can repair a UPS, its functionally more like working as a mains electrician than a computer tech so you do need to know some things and the repair in most cases for anything now is replace larger assemblies - one FET died, replace the whole control board...
Multiple submarines have been lost due to water flooding into battery banks... I would not risk it. Just e-waste recycle it too be honest. If Navy folks cannot even figure out how to handle water inudation into battery banks then that UPS is a waste of time to try to salvage and dangerous.
Sorry for your loss Linus, and well done for having the restraint in that moment to not rip the server rack apart and smash it all into tiny pieces.
I understand the whole point is for this to not happen to begin with but… there’s homeassistant compatible devices you can put on the floor to detect water leaks, then you can use that to fire off a critical emergency shut-down
I would like to see some kinda of device that can detect a change in flow or pressure, out side of a window, and fire off events
@@DarkAxi0m can’t remember what it’s called but there’s an LTT video about a device that keeps the entire water cooling loop under a bit of vacuum, so that if it has a leak the pressure will change and it can sound an alarm- but no water will get out because of the vacuum.
But this has giant, open reservoirs so that wouldn’t work
@@PentrossAquacomputer LeakShield
@@DarkAxi0m Or, a level detector in the tank itself. If it drops too low then somethings wrong and it needs to kill everything. As well as CPU temp monitoring if the pump fails
@@Pentross That's a bit over killed and would require the pump to be at the end of the loop etc.
Because they use a static flow rate pump, a single flow sensor on the return line would be enough. If the flow decrease even a little bit, you know either your pump is bad, or you have leak (be it small or big). You could complement that with a water leak sensor at the bottom of the rack and use them to turn of the pump in case of a problem.
In 1988 I was working at IBM, in a brand new (unfinished) 56 level tower near Paris. And one day, we saw water falling from the ceiling, slowly at start, and becoming more and more like the Niagara’s fall. The building had 4 reservoirs of 30000l on the roof, for water usage in the building, and as a fire suppression, and one brand new valve broke, letting 10000 gallons of water, going down the 56 floors, devastating everything in its path. Fun time.The worst was the water getting more and more dirty at each floor, with the particles of ceilings tiles, and other crap.
that sounds like a spectacle
Why store underground and use pumps if you could just play the world's worst game of Jenga!
Where in this were you?
We had the same thing happen, but in an 8 story residential building and with the water heater. We were running all over with every bucket, bowl, and trashcan we could find trying to save musical instruments, tech, and furniture
I remember watching a video of a building, I believe in Paris, that had a giant fish tank in the lobby. The tank was huge, I'm guessing 100' tall?? (I'm just guessing, I know it had a staircase that went around it to get to the upper floors. This was in a fairly new building. Whoever designed the tank didn't do so properly and the entire tank came apart. People were flushed across the floor in a giant wave of water. Was pretty crazy to see. I just looked, it was in Berlin and the video can be found online.
sorry for your lost linus best wishes
Rip Linus sister I hope you're doing well Linus I know it's not easy to lose family that close.
Sorry for your sister first of all.
Your story really helped me today. Had a IT crisis today. Came back to a dead SSD after whole day of work and dealing with the gov for my ID. Saw this clip while frustrated. I got hope back and, following linus' precedence, left to get food instead.
Thanks for the motivation. Hopefully it'll work out for me too
Everything looks better on a full stomach
Manifesting everything working out Man U got this
Thanks for the likes! My SSD is completely dead and I had to reinstall my OS on a new one, but my backup setup worked and I got all the data back from the cloud. So everything worked out in the end
"...I'm not saying it went well, I'm just saying that I'm not a f*cking idiot." -Linus 2024
Linus we watched the home build series.
maybe a voltage differential between 2 metallic parts in the same loop?
next time make shure every part is eighter grounged ot completly isolated.
There's some universe where Linus just said F it and cooled directly with the pool water, and his entire basement filled with water.
The corroded connector sounds like it was caused by electrolysis induced by a grounding loop.
What if in the construction someone took a slightly identical connector made by a different material? It could be an human error
Al vs. copper
Oh, this brought back memories! I work for a large ISP. A few years back we were upgrading the incoming fiber to one of our contracted housing associations. This required drilling up and re-sealing the concrete floor of the server room, after which the association hired another contractor to sand and polish the floors. During this time, the existing hardware was still up and running.
When we came back the day after, we realized that the contractors hadn't covered the racks AT ALL. Every single system was CAKED in concrete dust, and they'd all self-terminated due to overheating. The entire room was covered in dust. We were less than pleased with the association, and told them that they would have to get the contractor to clean up their mess before we started to try to get everything up and running again.
Some of you just had a chill run up your spine, and yes, it is exactly as you feared. When we came back, they had literally pressure washed the entire room, including our racks. There was water dripping from the switches.
I still remember vividly how my boss got the call, stood up, and exclaimed incredulously - "They pressure washed the fucking racks.". I don't think I've ever seen a more apt example of someone having lost their faith in humanity...
They are 100% going to pay for damages
My condolences for your sister Linus, may she rest in peace
0:41 Damn.. Our condolences Linus.
Have you tried putting it in rice?
He means well 😅
to summon Dennis?
The sister?
@@joaopedroalbernazoof
@@SignalJones Lmao
In the aquarium hobby there is a sensor that sits on the floor and alerts you to water leaks.
That it the FIRST ff thing you do when doing shit with water.
So sorry for your loss Linus, wishing you and your family some peace in these times❤
Just knew about your sis, Linus. Rest in peace, and my soul and heart goes to your family
I'm always surprised at how resistant tech is to water. I'm still sticking to air cooling.
Yeah, I also got lucky with hardware and water: a few years back I was moving, my desktop was already outside to be loaded into a car, I did not notice it rained for a bit while I was inside finishing up. Once in the new apartment I connected everything and fired up the PC, which returned a black screen after a while I noticed water was coming out the bottom of the case, I my heart jumped, I let it dry for a day or so, tried again and everything was working just fine.
Yes, never had an air cooler kill a PC in almost 40 years of owning systems.
@@mastroitek Even if its not water the number of times I have had a issue with a piece of tech whether mine or my friends and I replaced it or set it aside intending to throw it away, check a warranty, evaluate possible fixes Etc only to get busy or forget about it and have it work perfectly fine once I got back around to dealing with it is kinda astonishing.
Maybe you need some sort of "drip tray" every few machines in the rack, which could divert the water away somehow. Like a rack shelf but with a fall towards one side and a drain hose away from the rack. This would isolate any failures to only a portion of the rack rather than losing a whole rack. If you also added a water detection into the tray then you'd know there was a fault. I didn't see anything with quick googling but I'd guess some 3D printing could produce something.
I think you could just make trays from sheet metal for that. Also yeah having flood sensors should be a no brainer as he's already running home assistant anyway, so all he really needs to do is to buy some.
I would add drain holes and tubes to all the systems which end in a single drip tray/container to limit them dripping on each other. Finally add a water sensor to the drip tray.
@@Iisakki3000 Home assistant is a terrible way to deal with cutting power because you're relying on so much to work right; Best way to deal with it is to simply have a hardware water cutoff that cuts power, a close second is to use the tools that are likely built into the PDU's/batteries and have a graceful shutdown.
MY condolences go out to your whole family. I am sorry to hear that you lost a family member. It hurts so much, I can't imagine what you're feeling and totally respect you for coming in and still doing the things you do. Keep being a reason for me to have pride in the people of this city.
Dissolved solids in the water, and probably leaking current from the pump. I have zapped myself on a few occasions with hydroponic gardening and pumps leaking stray current into the reservoirs. Quickest way to find out put a bucket in the loop and stick your hand in there after a manicure if you can feel tingling on your cuticles its stray current.
Im so sorry to hear about your sister. My condolences to you and your family.
Sorry to hear about your sister, thats heartbreaking
My condolences and hope yall get the opportunity to properly mourn your lost
A few months back I had a water pipe break in my house right above my lounge room where all my electronics and my desk are and water ran through everything in there for hours before I came home to turn off the water, went through a similar state of mind when diagnosing everything but was also extremely surprised that damn near everything survived, even batteries(I watched em like a hawk), after letting everything dry for a week. modern tech is surprisingly durable
I'm sorry to hear you lost your Sister Linus. I know it is poor consolation, but I'm sending my best wishes and condolences for your loss.
I do love when people forget that the IP rating isn't just about water. You tell them something is IP rated dustproof and they just look at you confused.
Lost a best friend and an uncle in the past, my condolences. Losing someone close to you never gets easy.
I'm so sorry about your sister, I lost one of my brothers a few years ago and know how much it hurts. Losing a loved one isn't something you get over, it's something you learn to live with. My condolences and prayers to you and your family.
My condolences to you and your family about the loss of your sister.
That's amazing most of the equipment survived. Way back when, my buddy had a 7990XT in a Mountain Mods Horizon case. He spilled a drink that ran off the edge of his desk right in to the open top, onto his 7990XT. I didn't see him online for a few months until he recounted this story to us. Lost the mobo and GPU to a cup of water.
Sorry to hear about your sister.
Instal moisture sensors throughout the rack.
A video fixing this and having monitoring sensors. Bonus points for having a video camera so you can check up on it remotely.
pretty sure he already as a surveillance camera in his server room
Cameras only give you a view of what happened after. And may not work if the nvr recording it all is soaking wet
It all made sense when he mentioned EK for his water cooling
The company I work runs on a 2006 Dell Poweredge server, that got flooded, with a ton of mud, we pressure washed it, replaced the thermal paste, left to dry for a couple days on a room with the AC on the dry setting, and it's been working 24/7 ever since...
I wonder if there’s a way to add a mechanical shut off that runs from a flow rate monitor that triggers an emergency shut off when the flow rate spikes indicating a leak? Sounds like a Pii project.
Could do what factories do and put audio sensors that listen for out of place noises.
I'd say a pair of flow sensors before and after the rack, with logic that if there's less flow on the after than the before, you have a leak, and therefore shut down the pump and fire off a notification.
Sorry for your loss Linus
Add a flow meter than alerts you when the debit is over a set value.
And as someone points out, it's probably an electrolysis because of a grounding short.
Check the other systems for possible grounding problem and electric tape them
Sorry about your loss Linus, I wish you and your family well.
Brass and copper are still susceptible to galvanic corrosion. Galvanic corrosion happens when two dissimilar metals are in contact with each other AND have some sort of electrical current running through them. It can even happen to the same metal with different levels of impurity or different alloys.
You need to install humidity detectors, alarms, emergency shutdown valves etc
Ouch man sorry to hear about that very hard day
bro called xeon v3 ancient. My entire homelab runs on v3s and v4s
It is ancient.
Mine is also entirely on v2s and v3s.
Well, they're 10 years old
10 years is pretty ancient for tech
My deepest condolences for your loss to you and the family, sorry for the server flood
I had a server room flood caused by one of the cassette ac's which had an electrical fault which triggered the fire sprinklers (because it's in an office building, you're not allowed to remove them). We immediately killed power to the server room, removed all the servers from the racks... Rinsed them under running water (because the insides were dusty before the flood), stored them on their sides to drain and hit them with heat guns and hair dryers. I think our survival rate was around 85 to 90% as well. With all of them going for another 4 or 5 years until they were upgraded and replaced during the next refresh.
Pretty much all houses in North America will use 120v/240v two phase power, some apartment buildings and a lot of industrial sites will use 3-phase, where you can get 120v between neutral and a phase, or 208v between two of the 3 phases. So in theory appliances that use the ~200+ volt power could either get 208v or 240v, and there are usually fluctuations in the power grid so voltage can skew from those numbers as well.
Condolences on your sister. Losing family is never easy (I know I still have to properly deal with the death of my father over a decade ago). Best of luck to you and your family, immediate and extended and otherwise through this.
Have you considered reaching out to Jay (JayzTwoCents) about the water cooling issue? He may also have some suggestions including if you want to go with a different company rather than EK for whatever reason.
What I do is use a heat exchanger for each machine to my cooling loop and use Koolance 705 non-conductive coolant. Highly recommend, been running like this for years.
first thing I always do is spray on another conformal coating on all PC parts pre install and after assembly , helps stop dust causing corrosion in humid area's and protects against any other liquid damage that might happen , amazed you got away with a flooded server rack , I normally use water to water exchanged loops so they liquids them selves are isolated per machine and the main loop just exchanges the heat from 1 block to another , sometimes impurities just get into the loops , I also nickle plate everything myself
At my last job, an internal gutter broke. So under heavy rain, half of the roof water was piuring down... Through a switch!
There was an actual waterfall coming through the air holes in the front panels.
However, switch was still 100% working. We only noticed when the carpet in the office next to the rack room started being wet.
I could only imagine losing one of my siblings and how I'd take that. My dad passed away right in the beginning of the whole covid crap. He died from congestive heart failure, he was only 63. I visited his grave every single day after work and sat at his grave and cried for 45m straight. Did that for 2 years. I dread the day 1 of my siblings dies. I still cry like a baby once a week thinking about my dad. Ugh
Sounds like electrolytic corrosion, something on that one system was sending voltage through the loop, causing the corrosion.
Could be different Earth potential between the pool and the server. Significant direct current flowing into the domestic Earth can cause the house Earth to shift by several volts.
@nicholasvinen the PCs are on a seperate loop that trades heat to the pool loop via a heat exchanger. At most I could think was the pump was somehow inducing a current in the water loop, or pump is copper and fitting was aluminum
A theory. It sounds like you have a fitting in the loop that's "pretending" to be copper or brass. Basically a chunk of plated aluminum. Coating gets scratched a little and boom, you have a mixed metal loop.
My condolences to you, the family and possibily her family and friends.
Why is it stuff like this always happens after an awful day.
Air Cooler: I will overheat this cpu for fun.
Water Cooling: I will end this man whole career.
"I once used a flimsy intel stock cooler and now I think all air coolers suck." /s
@@xXRenaxChanXx technically,
@@xXRenaxChanXxthey suck and blow, its an air cooler thats how they work :^) Badumtss
reasonable AC will cause your CPU to throttle if fans stop but you still will be able to finish your work for the day, just a bit slower
if a pump stops... oh well
and if it leaks...
My deepest condolences. Never have I ever lost a family member as close as a sister, and I can't imagine how it might feel. I hope you have and will find strength in your loved ones in these trying times.
I feel your pain Linus. I lost my brother when he was 24, I was 25. It hurts like nothing else. I’m sorry for your loss.
buy the thing, wanna see you fix it
Thin sheet of plastic between machines can help direct the water off the machines below!
I’m soo sorry to hear about the loss of your sister my condolences to you and family
You should maybe build and fit some drain pans between systems. Easy to design and build a 1U drain pan to catch water that might drain from a water cooled system. ;)
Also, here is a most outrageous video idea.
Step 1) Gather 2 identical motherboards and build identical systems with them using inexpensive components and test them.
Step 2) Disassemble one system and its components, spray some clear enamel spray paint on the circuit boards of each component (including PSU) on just one of the builds. Be careful not to coat heat spreaders, connectors or contact pads that might be needed for them function. Coat enough of each board so that it might be considered water resistant.
Step 3) Reassemble the now "water resistant" build with components and spray the system again while it is assembled so the paint will coat PCI-e slots, power connectors etc... Test that it still works like in step 1.
Step 4) Get wet! Moisten both systems, coated and not coated, until they crash. How much water can they each take before they are completely unusable. Try some number of times to dry each system, test its stability and soak it again. Be sure to do this safely and away from any mains electrical outlets or cords. Do not touch any power cords with wet hands or touch any wet power cords near either end, they will shock you... ask me how I know! [insert random employee laying on the ground saying, "I'm okay. He told me not to..."] 😂😂
Step 5) If any system is still running mask off the CPU heat spreader, spray some clear coat around the CPU and socket edges and then try to directly water cool the CPU with a garden hose.
DIRECT WATER COOLING... WITH A GARDEN HOSE ! ! !
Profit and have a laugh. ;)
sounds like a upgrade video, of adding in a fail safe. so if the water gets low in one of the tanks an alert goes out. if it gets lower it starts killing power to pumps and rails to keep a fire from happening.
Sorry to hear about your loss
We had a freak weather event here on Friday. Rain at a rate greater than 4 inches per hour. Overwhelmed my massive curtain drain for the first time in 20 years since installation. Water was 3 inches up the basement windows but thankfully didn't blow out the windows. Stood in 8 inches of water pumping out the basement window wells. My neighbor had his basement windows blown out and his finished basement flooded AND his pool full of muddy water. Have a whole pile of lithium batteries on the basement floor for the solar system - that was my first concern. Also have an unpopulated rack with servers on the floor waiting to be installed.
Damn Linus, super sad about your sister. I am very sorry for your loss. You talked about it as if it was common knowledge, you share so much about your life yet you didnt say anything about your sister, you could have had so many messages of support towards or family, so many prayers going towards them, you will have them now... I am so sorry
I'm very sorry for your loss Linus ❤
watch out on the lithium ups if water got into the bms it can cause a fire after a few charge cycles depending on how good the bms is once water gets in it can cause it to get stuck on losing the over charge and cell balance protection (why most ebike /scooter fires happen )
haha... just don't watercool, air is suffucient, especially if you have a server room in the basement - noise isn't even an issue!
You can buy a flow rate measurement device and a couple solenoid valves, hook up the flow rate detector to all the valves with an independent system like an arduino, and if flow rate spikes, it can send out alarms and close the solenoid valves. Could isolate all the different water cooling loops in the machines, kill the power and keep this from happening again.
Man, RIP to your sister and my condolences to you!
The first sentence is pretty much my experience with AMD as well. I only run a RX 6700 XT, but the eight months I have had with it so far were pretty uneventful. I had one minor bug that saw me roll back drivers to the last version and wait for the next one and that was it. Nothing I haven't experienced in the decade of Nvidia before.
A few weeks ago I remember that Linus wore a t-shirt made by his sister and he seemed a little down.
I'm sorry for your lost, Linus. My condolences to you and your family
Condolences to the Family Linus. Also is the sister is the same sister that build a pc with him on one of the LTT video?
he said last wan show that it wasn't
Water cooled components at the bottom of the rack going forward. Maybe cover them with a slim plastic sticky-back cover or something too (to get water past them, rather than through them).
And a powder based automatic fire extinguisher hanging on the ceiling on top - just in case. (It’s going to cause a heck of a mess, but it might save your house.)
I deleted all my social media apps and just have RUclips for really any social media. I'm sorry about your loss...it was kind of casually mentioned here and I had to do a double take to make sure that I heard it correctly. Condolences to the entire family. Be sure to find the time and energy to heal. Everything else can and will wait.
5:49 now thats what i call open loop water cooling
All the fittings should be of the same metal, or be made of plastic. You can still have bimetallic corrosion between brass and copper if they are in contact with each other or submerged in an electrolyte such as water.
You could add water leak detectors inside the rack, that would instantly shut down the power in case of leak to prevent damage
Hate to break it to you but that lithium UPS is probably toast. It might run but you want to open it up and check that there isn't any moisture inside.
I wasn't able to watch the WAN show, so my late condolences to Linus and his Family. It's one thing to lose a family member, but a sibling at that is even worse not just for Linus, but his parents are well. Hope you're all doing well.
Many years ago, I accidentally left the roof window, directly above a PII computer, open when it rained.
Even the power supply didn't break (I still preferred to replace it), although the system was running even while it was raining into it.
So I wouldn't say that old mainboards can't withstand that :)
I am really sorry for your loss Linus. I lost my sister a couple years back and its not easy.
Sorry to hear thoughts go out to you & your family. That's sad
Linus I lost my little brother 2 years ago I know what you are going through this is something you will never get over the emotions will rush once in a while ( I’m tearing up thinking about it) I’m so sorry you have to deal with this.
I'm sorry for your loss
Hey Linus. Sorry to hear about your sister.
About your leak situation, an online filter would be a good idea. Aquacomputer makes one with valves included for service.
sorry for your loss Linus!
You need some sort of static pressure sensor, inlet vs outlet so if you see a dip in water pressure, the pump shuts down instantly. maybe even shut down the machines...
I clicked not because Linus was on the left of the thumbnail, but because I liked the illustration showing the server on the bottom of the ocean. ^^
Sorry to hear about your loss, Linus.
I wish your family the best
❤
Thinking about an automatic fault detection. Something measuring the flow of water going into the loop and back out of the loop so you can measure the difference and if there is a difference, shut everything down.
Fitting corroding and exploding is terrifying. Looking forward to update video because my water cooled server is now making me anxious lol
9:55 The Cobbler’s Children Have No Shoes… very very familiar with that contradiction.
I am sorry for your loss Linus, take as much time as you need.
Much love to you and your fam. Sorry for your loss, Linus.
Condolences on your sister Linus, greetings from 🇩🇰