Aww yes, the scream of hard drives as they self destruct. Been there. My server room monitor called me at 3:00 am and the synthesized voice said "Warning, your computer room temperature is 132Deg F". I could hear the scream when I came in the front door of my building.
Luckily for us we just lost a few drives in our ZFS arrays. Now we have each machine's OS scripted for thermal shutdown limits. You never think you'll need those scripts until something like this happens lol.
@Android Developer Whats funny is about two years ago we had an issue where the entire school district’s internet went down where someone hacked in and released some sort of a virus. It was some sort of ransomware and I think they shut off the internet connections to prevent any further spread.
Don't worry, everything survived bar 2 3TB hard drives in the NAS box. Good job we have off site backups too! Thanks for the appreciation on this vid guys, I work hard to help run this network smoothly, it isn't perfect but then which network is? It runs well and I love network and IT support in general. Thanks!
We had something like this. our 2 big ACs in the small company datacenter went down, we were wondering why some servers are down. we go to check, "wow the servers are really loud today", went in, wall of 70-80° heat came out, he runs over to the windows, opens all up and checks everything. the servers at the top were off because it was too hot. Then we developed a monitoring sensor system for our monitoring software with a raspi and 10 sensors over 1wire.
You're remarkably calm considering the situation. How long did it take to get the AC restored, was anything permanently cooked, did you look into any backup cooling plans for future possibilities? I've been in enough failed cooling comms cupboards to know the instant reaction of sweat pouring off your face.
@@toranine09there is though. Rush over to get the door open to let out the hot air. Bring in exhaust fans immediately instead of making a video nonchalantly. Shut everything down. Find out why the air conditioner was off.
I personally would of installed a fail-safe system that when the A/C failed it would power down all non-crucial functions, start a off-site back up, after the back-up it would slowly shut down everything and alert the IT manager of the pending doom.
Some Guy is that really in “””industry””” standard? so every small company to every large company is expected to run two ac units. And how would that help in a power outage that shuts the ac down for some reason. You could argue that they could use backup power for the ac but that’s an immense amount of power.
@@UlrikBrun-ir7mh that is not how it works. It's a matter of the compressor compressing refrigerant which increases the refrigerant temperature, which is pumped to the outdoor unit (in this case) to dump the heat. Then the refrigerant pressure is dropped, resulting in cold refrigerant which goes to the indoor unit (in this case) to absorb heat. Repeat the cycle. Basic science principle. If you increase pressure of a gas under constant volume, the temperature increases, or decrease pressure and temperature decreases.
Experienced this a few times before. Worst case caused the insulation around the beefy power cables from our old PABX UPS... to melt. After we got temporary cooling up and running and aircon techs in to provide the longer term fix, we performed a full inspection, including under the floor, which is when we found the bare copper of the PABX power cables, where melted plastic dripped off them and settled onto the concrete floor. Scary stuff. And... lots and lots of hard drives dying in the days following, with us frantically replacing them and hoping they'd rebuild before the next imminent failure. Never knew that I'd be facing significant risks of heatstroke as an infrastructure admin!
I love everyone saying how they would have done this or that, had redundant whozits, etc. I would have done what my bosses told me and what they gave me a budget for, after I informed them of the risks. Which is what I assume happened here. But man, you gotta give it to the servers, they were trying everything they could to cool themselves down.
I do what my bosses tell me, unless what they tell me is stupid. In which case I tell them and why. If they still don't budge, I go above my boss and sometimes to mutually trusted and respected equity partners in the firm. And if that doesn't work, I find a new job at a workplace which has not become dysfunctional.
We had a similar situation overnight in our server room. Air temp got to 65C and we had to shut things down to cool the room. It took 2 full days to restore everything and to get the temp back to normal. We had ot ditch the all UPS batteries for new as they were cooked but the server hardware was all fine. No doubt it's shortened the life on some parts though...
@@TravisWeir Yeah UPS, i think the AC tripped a breaker when it blew, probably the normal room breaker while the servers are on their own PDU+UPS, so the watchdog caught em when they overheated, or when they rebooted. Is why my PDU has luascripting in it for overheating scenarios to trigger EPO, or alternate cooling solutions
That is good that everything survived, I use to have my server in an ATX case with zero cable mgmt and it was on the third floor in the middle of July and what a mess that created, BSOD's, system lock ups and of course drive failure.
Rough situation. If it was me I would start shutting down whatever non-critical equipment that I could and get every fan in the building in that room as I have someone call it in
Thats what you get for turning a broom closet into a "server room" . Wall mount A/C units barely work in a bed room, never mind with a real heat source like your million pizza boxes racked up there.
Yes and three is highly recommended, because with only two, the instant that one fails... the load on the remaining unit instantly doubles! Dramatically increasing the risk of it suddenly dying too. The number of times I've needed to tell colleagues of this, only to have them claim that the risk of two failures in quick succession is extremely unlikely, has been infuriating for me. More so when I was then proven correct on a number of occasions. One time we had a PSU fail in a SAN disk chassis and as a consequence all the fans in the unit then went to 100% speed (increasing fan failure risk AND increasing remaining PSU power usage) and the load on the remaining PSU thus more than doubled. I warned our storage guy (I was the netsec guy, former server admin) that the faulty PSU needed to be replaced ASAP, because of the dramatic and sudden increased load on the PSU and he insisted that the likelihood of a quick succession failure was low (just based on very rudimentary ideas of probability which completely ignored the age of the equipment and the sudden more than doubling of load for a prolonged period of time). So he told EMC (our SAN provider) that we did not need to use the highest priority callout replacement and that 4 hours response would be fine........ Shortly after, the fan in the remaining PSU died, so that PSU shut down and thus *ALL* SAN LUN's associated with the disks in that chassis went offline and all associated VMs were suspended due to write failures. A major disaster for the firm, due to so many systems going offline and staying offline until they could be brought back up, pending writes written and filesystem and database consistency checked. If they had heeded my warnings and kept cold spares for key items, like I recommended... 🙄
True, however this is a school by the looks and they dont have the budget for that lol. My high school didn't even have A/C, there was an open window and a fan wedged into it and that was it.
He’s so nonchalant with getting that door open. No idea why they didn’t do everything they could with bringing in large exhaust fans to get out the hot air.
THIS IS WHY YOU MUST INVEST IN A ENVIRONMENTAL MONITOR.... Had this happen at my workplace and while it didn't immediately kill a box, within 3 days one of our critical servers was dead due to a motherboard failure. While this is an older video there is no good reason not to have an environmental monitor. APC makes a decent unit (Netbotz 250) that can send alerts via SNMP or SMTP for $500 or so. As an IT people we always have to ask ourselves "What is my companies data worth"? If your managers or finance people balk at spending $500 for an environmental monitor you gotta ask them what is their data worth. While it has happened to me as well I will be the first to say there is no acceptable reason this should ever happen.
And if you dont want to spend so much money, a small diy build with an esp microcontroller would cost you less than 15$ in total, and has everything to monitor temps, send it over wifi to your network and if necessary even shut down your servers. And for redundancy you can built 2 of them, link them so if one goes "alarm 1000 degrees in here" it can check the other one to prevent accidents.
So what happened later on? Was the air conditioning fixed in time? Did any of the server die or have to be shutdown while the air con was fixed? How did this disrupt the school? (If it disrupted the school?)
Hahah who fault was the A/C failure? Who designed this room?? Whoever did, clearly should not be in IT and if they worked for me, they would be looking for a new job
You Might Want To Install a Fail Safe For AC Failure Might Want To Use High Power CFM Fans That You Can Buy For Duct Work with Thermometer to turn on if it get warmer then you would like also If You Have To Push The Hot Air Outside Warm The School For Cheaper
i would of programmed all the servers as soon as they where to get near critical to send a message that says servers are shutting down i wouldnt let them get near critical which is 131*F (for HDD's) (not sure in celsius) i only have a small personal file server for media and backup of my computers it is pretty much a standard PC repurposed
At least all the cables aren't stiff anymore.
Aww yes, the scream of hard drives as they self destruct. Been there. My server room monitor called me at 3:00 am and the synthesized voice said "Warning, your computer room temperature is 132Deg F". I could hear the scream when I came in the front door of my building.
I've had this happen before, it's not fun. You can literally see heat pouring out of the door like asphalt on a sunny day.
Absolutely, I could hear the fans going berserk from down the hall! Not fun at all, we lost a server that day!
Luckily for us we just lost a few drives in our ZFS arrays. Now we have each machine's OS scripted for thermal shutdown limits. You never think you'll need those scripts until something like this happens lol.
SIN3R6Y Scripting my personal home server for thermal shutdown probably saved its life, like, 3 times!
Do server CPUs not turn themselves off when they reach maximum temperature?
Really proves viability of data furnaces to heat buildings
The schools in my town all have servers, but the thing is, very few have air conditioning. The servers run pretty hot.
@Android Developer Whats funny is about two years ago we had an issue where the entire school district’s internet went down where someone hacked in and released some sort of a virus. It was some sort of ransomware and I think they shut off the internet connections to prevent any further spread.
@Android Developer agreed
That’s where probably the grading system that teacher entered sits in the school server system.
Don't worry, everything survived bar 2 3TB hard drives in the NAS box.
Good job we have off site backups too!
Thanks for the appreciation on this vid guys, I work hard to help run this network smoothly, it isn't perfect but then which network is? It runs well and I love network and IT support in general.
Thanks!
Jake Billing did she get it all in her house to keep warm? Lol
Are you the IT department or the third party consultant??
We had something like this. our 2 big ACs in the small company datacenter went down, we were wondering why some servers are down. we go to check, "wow the servers are really loud today", went in, wall of 70-80° heat came out, he runs over to the windows, opens all up and checks everything. the servers at the top were off because it was too hot. Then we developed a monitoring sensor system for our monitoring software with a raspi and 10 sensors over 1wire.
The sound of a dying stack
You're remarkably calm considering the situation. How long did it take to get the AC restored, was anything permanently cooked, did you look into any backup cooling plans for future possibilities? I've been in enough failed cooling comms cupboards to know the instant reaction of sweat pouring off your face.
How in the world did you not receive temperature threshold notifications? Anywhere else, this would be just cause for disciplinary action.
I love this comment!
How do you know they didn't? It doesn't seem like he's going to do anything about it at the moment besides record video anyway.
@@NillKitty He appears to have turned the thermostat back on.
@@NillKitty one of the psus died, there’s not much he can do in the moment, especially considering the hardware needs to actually cool off first
@@toranine09there is though. Rush over to get the door open to let out the hot air.
Bring in exhaust fans immediately instead of making a video nonchalantly.
Shut everything down.
Find out why the air conditioner was off.
60 degrees? Time to shut down asap before equipment is damaged!
I personally would of installed a fail-safe system that when the A/C failed it would power down all non-crucial functions, start a off-site back up, after the back-up it would slowly shut down everything and alert the IT manager of the pending doom.
Or you follow industry standards by installing a redundant AC...
Some Guy is that really in “””industry””” standard? so every small company to every large company is expected to run two ac units. And how would that help in a power outage that shuts the ac down for some reason. You could argue that they could use backup power for the ac but that’s an immense amount of power.
@@UlrikBrun-ir7mh uh, that's a giant pile of bull manure
@@UlrikBrun-ir7mh LMFAO
@@UlrikBrun-ir7mh that is not how it works. It's a matter of the compressor compressing refrigerant which increases the refrigerant temperature, which is pumped to the outdoor unit (in this case) to dump the heat. Then the refrigerant pressure is dropped, resulting in cold refrigerant which goes to the indoor unit (in this case) to absorb heat. Repeat the cycle.
Basic science principle. If you increase pressure of a gas under constant volume, the temperature increases, or decrease pressure and temperature decreases.
Experienced this a few times before. Worst case caused the insulation around the beefy power cables from our old PABX UPS... to melt. After we got temporary cooling up and running and aircon techs in to provide the longer term fix, we performed a full inspection, including under the floor, which is when we found the bare copper of the PABX power cables, where melted plastic dripped off them and settled onto the concrete floor. Scary stuff. And... lots and lots of hard drives dying in the days following, with us frantically replacing them and hoping they'd rebuild before the next imminent failure. Never knew that I'd be facing significant risks of heatstroke as an infrastructure admin!
0:19 The gut-wrenching sound of computers overheating
My servers sound like that aswell... But I put them there so i don't need air conditioning and also mine have Gpus in them that need a lot of cooling
I love everyone saying how they would have done this or that, had redundant whozits, etc. I would have done what my bosses told me and what they gave me a budget for, after I informed them of the risks. Which is what I assume happened here. But man, you gotta give it to the servers, they were trying everything they could to cool themselves down.
I do what my bosses tell me, unless what they tell me is stupid. In which case I tell them and why. If they still don't budge, I go above my boss and sometimes to mutually trusted and respected equity partners in the firm. And if that doesn't work, I find a new job at a workplace which has not become dysfunctional.
Those switches alone put out a surprising amount of heat. Wonder if extra cooling on the A/C condenser outside would help it along.
The sound of the servers made me cringe.
Good luck...
We had a similar situation overnight in our server room. Air temp got to 65C and we had to shut things down to cool the room. It took 2 full days to restore everything and to get the temp back to normal. We had ot ditch the all UPS batteries for new as they were cooked but the server hardware was all fine. No doubt it's shortened the life on some parts though...
Holy shit that's hot.
Holy shit, I keep the heated plate on my 3D printer that hot, it's enough to deform plastics
The Aluminum Macintosh sitting amongst the PCs was refreshing, my school had a few next to old HP compaqs back in the day
I think the only room temp that can compare is an attic in Texas over the summer at noon. Also, what was the thing that was beeping?
Travis Weir Sounds like a U.P.S. was the beeping thing.
Ok, I thought that's what it was but wasn't sure.
@@TravisWeir Yeah UPS, i think the AC tripped a breaker when it blew, probably the normal room breaker while the servers are on their own PDU+UPS, so the watchdog caught em when they overheated, or when they rebooted. Is why my PDU has luascripting in it for overheating scenarios to trigger EPO, or alternate cooling solutions
I miss videos like this. I wish you did more of them.
That is good that everything survived, I use to have my server in an ATX case with zero cable mgmt and it was on the third floor in the middle of July and what a mess that created, BSOD's, system lock ups and of course drive failure.
Who's here to see how CollegeBoard servers are going to be this year?
bruh, this is why you have proper monitoring.
Rough situation. If it was me I would start shutting down whatever non-critical equipment that I could and get every fan in the building in that room as I have someone call it in
A nightmare I had to live for a long time especially when the company didn't like to pay money for a good maintenance!
Yikes 140 Deg Fahrenheit!. surprised the equipment wasn't toasted.
Thank you so much for doing the conversions for me. :D
no problem, My pleasure :)
Thats what you get for turning a broom closet into a "server room" . Wall mount A/C units barely work in a bed room, never mind with a real heat source like your million pizza boxes racked up there.
How is it that we are able to hear your voice over all those fans berserking? :O
Imagine this in the age of Nvidia...
I've experienced this on more than one occasion. It's HORRIBLE!
Did you lose any kit?
2 3TB HDDs
@@Nevexo287 they lost one server
Wicked space heaters
Anyone else notice the 2 Apple Xserve and the apple Xserve Raid and Apple g5 powermac
The only time the fans in an Apple product have been audible haha
it's a G5. Mac Pros have 2 drive slots in the front.
@@jamescollins6085 x serves and g5 powermacs are absolute screamers even at idle at room temp
@@jhonjhon1740 True. I've seen videos on the G5 systems, and they sound like vacuum cleaners.
You're telling me it was 140of in there??
I recommend using two to three air conditioners so that when there is a break down of anyone of these, at least the other one is still working!
Yes and three is highly recommended, because with only two, the instant that one fails... the load on the remaining unit instantly doubles! Dramatically increasing the risk of it suddenly dying too.
The number of times I've needed to tell colleagues of this, only to have them claim that the risk of two failures in quick succession is extremely unlikely, has been infuriating for me. More so when I was then proven correct on a number of occasions.
One time we had a PSU fail in a SAN disk chassis and as a consequence all the fans in the unit then went to 100% speed (increasing fan failure risk AND increasing remaining PSU power usage) and the load on the remaining PSU thus more than doubled. I warned our storage guy (I was the netsec guy, former server admin) that the faulty PSU needed to be replaced ASAP, because of the dramatic and sudden increased load on the PSU and he insisted that the likelihood of a quick succession failure was low (just based on very rudimentary ideas of probability which completely ignored the age of the equipment and the sudden more than doubling of load for a prolonged period of time).
So he told EMC (our SAN provider) that we did not need to use the highest priority callout replacement and that 4 hours response would be fine........
Shortly after, the fan in the remaining PSU died, so that PSU shut down and thus *ALL* SAN LUN's associated with the disks in that chassis went offline and all associated VMs were suspended due to write failures.
A major disaster for the firm, due to so many systems going offline and staying offline until they could be brought back up, pending writes written and filesystem and database consistency checked.
If they had heeded my warnings and kept cold spares for key items, like I recommended... 🙄
True, however this is a school by the looks and they dont have the budget for that lol. My high school didn't even have A/C, there was an open window and a fan wedged into it and that was it.
He’s so nonchalant with getting that door open. No idea why they didn’t do everything they could with bringing in large exhaust fans to get out the hot air.
One would assume the airconditioning was replaced with a new model that has auto restart after power failer. Clean the filters monthly.
The CPU's are now toasted to death... This is sad...
We have a Mitsubishi Air con at work in the Staff room,Its always playing up or failed. :D
Uh yeah a meltdown alert would be accurate.
THIS IS WHY YOU MUST INVEST IN A ENVIRONMENTAL MONITOR.... Had this happen at my workplace and while it didn't immediately kill a box, within 3 days one of our critical servers was dead due to a motherboard failure. While this is an older video there is no good reason not to have an environmental monitor. APC makes a decent unit (Netbotz 250) that can send alerts via SNMP or SMTP for $500 or so. As an IT people we always have to ask ourselves "What is my companies data worth"?
If your managers or finance people balk at spending $500 for an environmental monitor you gotta ask them what is their data worth. While it has happened to me as well I will be the first to say there is no acceptable reason this should ever happen.
And if you dont want to spend so much money, a small diy build with an esp microcontroller would cost you less than 15$ in total, and has everything to monitor temps, send it over wifi to your network and if necessary even shut down your servers. And for redundancy you can built 2 of them, link them so if one goes "alarm 1000 degrees in here" it can check the other one to prevent accidents.
Even at home I have a fail safe system and a backup air conditioner installed. I get alerts and can control both remotely from anywhere in the world.
60 degrees holy shit
Whoever shut the AC off is getting fired.
I think the channel is HVAC, its an AC repair men (or what you call it), and from his storys you see how often they fail if nobody looks after it
pov: roblox server rooms
Maybe their servers are good but probably faulty air conditioners? Well it could be.
And thats why you have 2 aircondioners ina server room
this is why you use networked hvac controls, or analog if you can't, last thing you need is a dead t-stat battery killing dozens of servers
This is why you don’t use comfort cooling and have no redundancy
I hope you wear ear plugs there. You'll get permanent hearing loss if not
Should of bought a watchdog temperature probe. 140 degree F ambient!? Thats insane!
So what happened later on? Was the air conditioning fixed in time? Did any of the server die or have to be shutdown while the air con was fixed? How did this disrupt the school? (If it disrupted the school?)
He purposely turned it off.
+Channel Tech it sounds like the school lost power overnight turning off the AC's, but the servers were still on from and UPS's
+Elliott Gamer Would be my guess aswell.
140 degrees? jesus
Damn Americans with our SI units, it even took me a second 😂
Scary situation to be in.
The stupidity of installing a single domestic air conditioner in a critical room.
Stupidity knows no limits.
Ok american
More AC units are better, plus it also cools off areas that are otherwise unreachable.
It's cool some DC runs at 50C normally to save on AC costs
What a janky server room
Use a proper datacenter air conditioning system for the room xD
60 degrees in that room, it's like a hot summer day in Iraq. And that's just air temperature, imagine hardware temperatures.
Nope. Not going there.
UPS squelching for help
You might want to shut the servers down if they overheat.
The beeps sound like a bird.
I didn't expect to see Apple Xserves here 0:44
My hear sank a bit hearing those fans scream. Is it normal to feel sympathy over machinery?
Give me my RTX 3090 back!
someone definitely getting fired that day lol
You from England?
how come those server rooms dont shutdown if it gets hot like on a pc
Dont matches light at 60°C?
Like that could have been alot worse if somebody left matches there for whatever reason
Cigarettes is a danger too.
I see you are Running Windows XP Professional on that desktop
This place looks like it might be a hospital. Nothing but the best technology is used in healthcare!
@@redsquirrelftw it's a school
@@redsquirrelftw ok american
The servers in my school use a centralised sub zero cooling system which I have never heard of
thats sounds way too overkill for a school XD
@@legominimovieproductions it's more of a engineering college/university
Cryogenic chamber as a server room?
Lol I always had a few portable air conditioners on hand for when this happened. render farms..
Wow that's just insane!
This is my ps4 at 3am
how much temperature can be raised in critical conditions?
Hahah who fault was the A/C failure? Who designed this room?? Whoever did, clearly should not be in IT and if they worked for me, they would be looking for a new job
0:19 when u launch forza horizon 5 in a celeron
When you run EA SPORTS F1 2023 with a Pentium and Windows XP
You Might Want To Install a Fail Safe For AC Failure Might Want To Use High Power CFM Fans That You Can Buy For Duct Work with Thermometer to turn on if it get warmer then you would like also If You Have To Push The Hot Air Outside Warm The School For Cheaper
my computer do this sound and is small... also with the seagate 7200 rpm blue caviar disk is more loudly xD
I thought it was 60 F not C, so this whole video I was thinking *Why is it such a big deal? Should be able to cool just fine*
American moment 🐴🐴
Remember to always use C when dealing with electronics temperature, even if you're American.
i sweated from here 🥵
Open up some windows to let the warm air out.
Recorded: July 2013
jk...
what? no redundant aircon no monitoring system? sheesh
it's a school
Would make for a great sauna
You sound like Pyrocynical.
What is the Name from the rack screen/keyboard in labtopformat?
Elias Müller it's a KVM
Thx you
This was pop up to my yt
Is that sever 2003
what was the pc with the error code?
They gonna break its too hot. I think they can handle with 40c but not 60+
40c is normal temperature
Can't imagine how hot the actual equipments are? When it reaches 70-80 degrees it's high risk of overheating.
2:13 nice face
Almost a 150f
lovely a sauna
Seriously, even with the door opened it was unhealthful for you to be in there alone.
happens all the time
ROBLOX sever be like
i would of programmed all the servers as soon as they where to get near critical to send a message that says servers are shutting down i wouldnt let them get near critical which is 131*F (for HDD's) (not sure in celsius) i only have a small personal file server for media and backup of my computers it is pretty much a standard PC repurposed
Jeez
What was wrong? Was there lots of storage that caused a fire? Who turned off the AC?
Hit the epo
Anyone hear slenderman at 1:36
60 degrees? That's a little chilly wtf... oh wait, metric sh!t
:)
That’s not a server room a shit hole and a museum. Have some pride and clean it