The sheer volume and sound is scary enough. But if you also image the amount of energy, enough to vaporize you in a bright flash, is being transferred and switched, its terrifying.
It's only a danger if you opened the enclosure door. It's not enough to vaporize you, but it definitely could give you serious third-degree burns. The enclosure is designed to handle the worst-case arc incident.
He did just say that the generator is "only" 500kw. I think you're thinking of powerstation where they put on protective suits to throw a switch, they handle a bit more power than these single sections of a datacenter do. It's still a scary amount, but peanuts compared to what the big boys do.
Same here! It's so dumb! I thinks I'm only interested in the last thing I accidentally clicked on and ONLY that, there's people I'm still subscribed to that I've forgotten existed because it's showing me Tiktok and a UPS in a data center failing.
It is good to see that some companies do a proper full test of the whole system as i have found that some computer bosses will not allow this full changeover but only to run the generators for a short amount of time that does not do them any good without a load and does not give any indication that the whole system will work as designed when a mains power failure occurs,Foolhardy!
Not sure how your large UPS systems work in the UK but here in the USA, UPS systems of this size and class are always online. We call it double conversion where the utility (mains) AC is rectified and float charges the battery bank. The inverter is always feeding the load. As the rectifier and battery are DC, they are simply spliced together. There is no switch involved. The rectified mains DC is normally about 10 volts above the battery string voltage to keep the batteries in charge mode. Of course there is a static and maintenance bypass switch on each UPS but normally the UPS output is never interrupted during a grid failure. Also at this size, we normally run at 480 volts which is a US standard distribution voltage, that is stepped down to 120/208 volts for the loads.
Hearing a Gen-Set under load ... the most "Beautiful Sound on the Planet" .. during a prolonged power outage. Cheers ........... thanks for sharing this
I was a UPS technician before in a DC and now I am applying as a BMS tech in a tier 3 DC. good thing i came across to your channel and it helps me refresh and learn a lot as well. that i can use on my next job.. "hopefully" i will get the job.
Don't skimp. Get a bit larger. Did you include everything you need for the stability (monitor, remote modem or KVM)? Does it allow extra batteries for more run timeor do you pick the biggest VA so you are far DOWN on the load vs runtime curve
5 лет назад+3
Depends on where you are we have a power failure roughly every ten years for fifteen minutes to half an hour. In Thailand it's a few times a day and brown outs at random.
The transfer switch we had was a 3 pump up then turn the knob. Them BAM! It made me jump too like you feel the amps. Then the generator diesel pumped out black smoke for a few seconds while picking up the load. Zero automation by the way.
A friend of 20 years works at Services Australia, Centrelink, Medicare, Many more Data Centre.. he blew my mind when he told me about a Direct Fuel line that runs from a Petrol Service Station close by, Directly to there Generators..... ( He said the system must never go down )
Be interesting to know if it’s unmetered at the servo , direct pipe from main diesel tank , or possibly other fuels , And at the government generator plant room , they have a meter for whatever fuel that is pumped from the service station (the servo might have a auxiliary tank that is rented to the Australian government owned (services Australia ) That way re filling the emergency fuel tanks can be done at the (BP , or Caltex ) Better than having a massive fuel tanker enter the complex
Just found the channel and love it. What a great idea shows transparency within the industry, I work for NTT Global EMCA at my site with have 40 Gen-set at 1250kw total load. all of our network is via PLC so we can export back to the grid. keep it up guys
Cool to see this being done. I work in a DC as a hall engineer. Changing Server motherboards etc. So i hear the tests happening but we never get to see whats being done.
A loaded test is required to meet NFPA 99 testing requirements. Many newer installations that have an iso-parallel bus will have an on-site load bank so that individual gensets can be tested at full design load, without placing any undue risks on customer equipment. In the example here, one of the parallel generators is turned down in order to increase the load on the remaining units, but doing this presents the risk of running without your engineered redundancy, and still falls short of design loading. The extremely long ATS neutral delay also serves no purpose other than unnecessarily damaging UPS batteries.
I have a question, if you guys are still watching comments. Why put the UPS outside of the load for normal operation conditions? What's the benefit of doing so? It almost seems like it'd be much more expensive to do it that way, but I'm sure there's some benefit to it. I'm more used to mains/gen > UPS > load (with a automatic switch between the mains and the generators in case of mains failure), since the company I work for uses the UPS not only as a immediate backup but also to clean up the power going into the server racks.
I don't understand you. A UPS is always in the load and always out of the load. The batteries are always outside on the backup line. And depending on the configuration (mostly due to specs like load), the basic components (filters, inverters and static switches) are on the main or on the backup lines.
I would say efficiency, most server PSU's have capacitors that will keep the server going when switching the relays. The correct term is an On-line UPS or Off-Line like shown here.
@@adpmol that an online Riello UPS, the generator supply will be filtered through the UPS as the only way for the load to be diverted from the system would be to place the unit into static or external bypass, in which case the UPS would not support the transfer. He has just simplified the process as when the UPS is online its only function is to clean the sinewaves.
Please please please please make these more often :D I watched one video and I HAD to watch those others too! I absolutely LOVE these videos! Thanks for the entertaiment!
at 12:21 why did you guys blur out the stats of the gensets doesn't make much senseI wouldn't think to know what RPM and temperature and frequency is sensitive information
Well you just gotta go to the generator then find out what the alarm about....if you can clear the alarm...then you should start the generator manually...then let the ATS/STS take over when power is Stable from Generator, otherwise you should hustle to fix the cause of alarm....
these sites are amazing. I take care of a couple of unmanned ILA's for a national data line and the system there is entirely automatic, we do a test every 6 months, even though i've requested a minimum 3 months due to the nature of these sites on the network.... BUT its absolutely terrifying hearing the ATS click, especially since mine is a 3 way between mains, all open and gen. the click is powerful enough to shake the hut and you can feel the transfer through the earth.
Would like to see a video on the type of, and how you test you're fire suppression system(s). Love these video's as they give us more insight on the daily workings in a DC, and see that the people working there enjoy their jobs.
i know this may be a bit of a secret topic, but i would love to see a video about some of the security of the DC, both network and physical, monitoring and detection, prevention and response
Question. So Power output of your generator does not provide charge/power to UPS & its batteries when it comes online? Meaning it bypasses the UPS System entirely? Thank you!
@@deanmarsh1452 Having done an IPVT of our system. The generator does supply power to the UPS. The UPS momebtsrioy goes on bttery as the ATS does its job. The UPS doesnt care where the power comes from (mains or generator) it cleans the energy through double conversion anyways.
I have recently carried out a series of load-bank tests on mission-critical sets from 40kVA to 2000 kVA. The customer used to run the sets up once a month to make sure they started, but never ran on full load. A lot of them were over-specified, resulting in carbon build-up, sticky valves, clogged silencers etc. My job was to run the sets up to 100% of plate rating for 4 hours, to make sure that they do not fail under operating conditions. After the usual checks and inspection, staged increases of loading, (5 minutes warm-up, 5 minutes 10%, 10 minutes 25%, 10 minutes 50%, then the balance of 4 hours at 100%), were implemented. One set would only run to 80% without overheating, another had a failed turbo-charger bearing, another had a radiator leak that only showed on full load, etc, etc. Once the test is complete, a 10 minute no-load cool-down period is done. Once the customer was aware of the issues, and they were dealt with, their decision was to load-bank the sets bi-weekly. (It's a great job for a winter's day; parking 20 feet in front of a 400kW fan-heater certainly keeps the van warm!)
@@TheManLab7 I am guessing because overloading is generally not a situation the generator is designed to handle on a bi-weekly schedule and might result in damage more than detect it.
@@TheManLab7 I imagine they were accurately specified and would never run at 110%. The control systems should not be designed to ask for a damaging output from the genset. Another assumption here but in the case of less than N working generators during an outage I would program the control software to keep the generators capped at 100% and use the UPSes to hold up the rest of the loads while a decision is made about which loads to cut or throttle down if it is not automatic. If you need 3 gensets and you only have 2 running it is better to keep your 2 at 100% by cutting loads (assuming you can't hold the excess load until an emergency genset can arrive) than it is to go dark because you broke them trying to keep everything running. If I was running a NOC all my major services/applications would need to be locally throttleable, at least manually. Your question isn't silly, I just don't think that's the way NOC administrators and operators think.
Man, that's a lot of power out of that tiny engine. Ours is .9 megawatt and is easily three times that size. It's the one part of my job that I truly loathe - monthly engine runs. It is incredibly foolish to stand in front of those ATS panels when the transfers takes place. I flip the switch and take off. I can get out of the room into a safe spot before the transfer takes place. Hopefully this guy won't end up in some arc flash accident video later on down the road. it scares me just watching him in this video.
Question: do you contact your power supplier before starting the test? They will see a significant change in their power output the moment you switch off
No the power company is usually not notified. The load change is not as much as one would think, he is switching one ATS at a time, not dumping the entire building onto the generators simultaneously. I worked in a descent size hospital, we had 3200KW worth of generator power and would switch over the entire building once a month, never called anyone outside the facility to notify of what we were doing.
At my company we use a manual diesel generator which means we have to manually swap all distribution boards one by one to the gen which is fun lol, jealous at how simple and automated a data centre can do it which makes sense when you cant afford any downtime whatsoever
If the gennys don't get the signal to start, how long do you have on the UPS to get them started manually? Also, It would be great to see a split-screen version of this, watching the generators start up in real time with the ATS switching over
Don't know if this will help, but generally, you'd be expecting at least 15 minutes of autonomy time although I reckon that these folks have at 20+ minutes.
In a datacenter like that i would expect that they can run the complete 2 hours until a new generator gets there, on batteries. A datacenter cannot afford any downtime and batteries are cheap compared to the cost if they run out of power. A single rack loosing power for less than a second might take over 20 minutes to restart and booting up completely, which means thousands of people not being able to work, and that's just one rack, and assuming everything boots up normally
@@MarioGaricTravel Based on the video, it looked like 10-20 minutes from extrapolating off the time the UPS was waiting for the genset to engage and the 10% drop in battery SOC displayed.
Thank you, Rich...I was wondering why he was so jumpy (Yep, I looked it up)...it's not the noise, it's the danger of arc flash. Protective gear would be a good idea.
Interesting to see this being done this is something I get involved in . Just wondering what if the electronics don’t work are you able to start and sync the generators by hand and switch the system by hand ?
I was surprised to see the hold up time seem so short, what was that? Maybe 60 seconds to 90%, maybe it was rounding down to 90% so 20 minutes at best?
@@kingrpriddick And I was surprised it's so long. 10-20 times the switch over to the generators is a huge safety margin. But that's probably needed to have enough of a buffer for the 2 hours to get a mobile generator when too many of the fixed ones fail.
The frequency drop at 8:53 doesn't sound good for the equipment. Is there no possibility for something like a soft starter (such as slowly powering down the inverter while in parallel with the generator) for the changeover, or is it impossible for the UPS to sync with the generator so this can happen? Edit: explanation makes it sound like UPS is cutting out and dumping the entire load onto the generator, which sounds like it slows down momentarily while the governor reacts.
with a double conversion UPS which these certainly are, there isn't really any need to sync the frequency exactly, but the ATS still needs to cut over on a zero crossing.
What you hear has no relation to the AC frequency, it's just the UPS PFC/rectifier circuitry losing power and DC bus being loaded which makes the UPS adjust the inverter switching duty cycle. This is normal. The output AC frequency is going to be exactly the same as was the input frequency since most double conversion units sync the output to the input (except for some rare special cases like frequency conversion 50 -> 60 and vice versa) to be able to safely switch into bypass at any time. It would have to freerun for much more time to have the frequency wander off a bit and when the input power is restored, the UPS will first sync the output back to the input before switching back from the battery.
Good video - I used to do monthly genset testing on several large data centres (From 3 to 12 Meg). Once had the mains actually fail just before I conducted the test - saved me some work. 👍😁. Gensets on the 3MVA setup were V12 50 litre engines, the 12 meg site had 6 x 2MVA - 78 litre, V20’s. 👍
Regarding the implemetantations of the generators, was there a handover, regarding what procedures to follow when it comes to testing, periodic checks, and switching over from the mains to the generators?
I swear it was "Not yet" and then he did say okay and turned it lol. I guess if it was "not yet" and he did that, we would of heard some non so nice of oxford english.
I like your videos, I am just wondering if 10v1600 series is enough to get the energy for the hole data center. They have only 700kva and for example 20v4000 series has 4000kva output. Maybe it makes sense to replace all small once to 2 bigger once :)
someone's going to argue there's very little reason to move but for me I don't wanna be the very unlucky soul to experience the arc flash from one of those 😂
i believe that it does both at the same time, because power is needed in the data center and ups' need to be charged as well to make it safe to switch back to mains
@@myRAYgame-Disk2 I think normal switching time on the "usual consumer UPS" is around 15ms. Minimum Hold up time for a Powersupply on a Computer/Server has to be 20ms, IIRC. So there is no chance, that a good quality PSU on a Server actually even gets "drained", even under full load, if the UPS Switches from/to Mains, which is actually pretty insane. I wonder what the Voltage levels look like while that happens tho.
i love how this all comes across as a walk in the park during the test. I've been on site during a power failure waiting for a generator to kick in before the UPS dies and i can tell you its one of the most stressful times in an IT guys life....
So if you have 3 generators each with 0.5MW Output and you do a load test on the entire data centre, do you have to schedule a day and time to do the load test with National Grid? I can't imagine they would be happy with 1.5MW suddenly dropping off the grid in the space of 10 minutes, that would likely cause a frequency change as well as a voltage change, neither of which are particularly good for the grid and anythng connected to it.
In reality the although the generators may be 0.5MW each, they should/would be overspec'd on purpose, for the case of generators failing while on load. I would guess that it may be the case that the whole site may only actually require a fraction of the total available output.
@@bsmith2053 That may be so, but imagine the entire Data Centre is using 0.5-1MW at any one time and maybe 2MW max, if you take that loading off the grid without alerting them (and without there being a failure to the supply in the area) then they might wonder what's going on. With that sudden decrease in loading, that could cause the National Grid to have to turn off a windfarm or slow down a power plant.
@@CoolSteve08 I still don't believe that even at 2MW the network would need to do too much in terms of compensation. I think the power stations would see this as a small blip in the scheme of things. And if that meant though that sources were taken offline/diverted so be it.
Definitely heard what you heard -- god forbid the guy meant 'no', what would he have said? "Yes, no?!!?" ... Hopefully, however, the system is robust enough that no true level of 'readiness' is actually necessary, since as we all know, the power companies are more often than not unable to give a warning before the mains power goes out!
Belleville Drivers if there was a failure of the gen during testing, you’d flick the key switch back to mains and it would retracted the ATS back to the mains. It’s a controlled environment doing these tests when you know the mains is stable.
Pff small gensets and breakers, on my previous work they have breakers of 3kV/150A on the small blowers they make a BANG if they switch. The gen sets are automated synced with the grid, 1 genset is 3kV/2000A. This is nice to :)
How often do you run the generators in a year? Just thinking the feels won't last forever, and the engines needs to run atleast 4-5 times a year at a minimum...
Every month 2 of them run and one is off. So at worst case they run every 2 Months for 4+ Hours which is sufficient for these Engines to warm up und lube everything and get rid of condensation in the oil. I guess Engine Oil swap is also done regularly, because these become the heart of the Datacenter, as soon as Power drops, which is why they test these every Month for an extensive time.
The longest wait of your life when doing mains fail, 30 seconds waiting for generator contactor /ACB to close, a million possibilities go through your mind 😂😂
How many gensets do you have?? I saw 3 of course, but do I understand you have 3 more?? I have run outdoor concerts on 10 Mw but this is very nice indeed, ty for the info
I remember one time I was working in some 33kV circuit breakers in a switchroom (dead testing) and we agreed with the guy testing them that he would shout switching as a warning but he just mumbled it quietly and hit the button anyway. Loud enough and strong enough to shake the CB and and the room. We exchanged words as you can imagine. Anyway this just reminded me of it.
Anyone know thr names of those ATS'. What type of ATS' those are. Or just what I'd have to slap in google to find ATS' like those. I've been looking for a while and i can only find 'small' ones. I only see ATS' like those in diagrams. Also, is each generator connected to each ATS or just one gen per ATS? Might seem like a dumb question but i was a little confused when the second ATS was aware that some generators were on.
These are not traditional ATSs made by someome such as Asco. Due to size and protection this is low voltage switchgear with electrically operated breakers
you are absolutely right, i've seen an ATS hang up and fail to transfer and shorted out a dc rectifier and exploded the controller and blew the door open. Anyone switching breakers or ATS with load should either be standing outside the arc flash boundary when the actual switch happens or wearing an arc flash suite. There is enough current that when shit goes wrong, it goes wrong big time.
I guess there would be a extremely big natural catastrophe if it'd take this long to restore power to the grid. But, I guess they just use those generators and get new fuel.
Pushing the E stop on a Genset equipped with a deep sea panel is not an acceptable way of isolating an engine from starting. You should be turning off the battery isolator !
Good video; you think it’s loud when it works properly; I was doing a test one time and a a switch jammed; so loud, and you could feel the thud; I jumped off the ground; I had to abort the test that time.
The UPS is the “middle man” between the grid and generator power. When grid power goes out the generator needs to start (it’s just an engine), and get it’s power output up to an acceptable level (pretty much the same as starting a car and putting it into gear). That process takes around a minute, the UPS is required to keep both the data centre and the systems that manage the generator startup process for the minute or so it takes for the generator to start providing output. So the answer is yes, if the UPS fails the data centre would go offline, BUT there is more then one ups, and many built in redundancies (so it would be very very unlikely for that to happen). If the generator fails the UPS can continue providing power, albeit not for long, but long enough to identify an issue and potentially fix or call for an emergency generator.
A video of a data center power failure sent me here
Literally same
Same
What’s up brudda!
x2
Same
7:03 "Sorry was that 'yes' or 'not yet'?"
"Nawts... Yest"
"OK just gonna do the thing"
Lmfao!!!
He said "Sorry, yes."
...in a mildly frustrated tone of voice
The sheer volume and sound is scary enough. But if you also image the amount of energy, enough to vaporize you in a bright flash, is being transferred and switched, its terrifying.
It's only a danger if you opened the enclosure door. It's not enough to vaporize you, but it definitely could give you serious third-degree burns. The enclosure is designed to handle the worst-case arc incident.
He did just say that the generator is "only" 500kw. I think you're thinking of powerstation where they put on protective suits to throw a switch, they handle a bit more power than these single sections of a datacenter do. It's still a scary amount, but peanuts compared to what the big boys do.
I see youtube algorithm has chosen this channel
Yes
Same here! It's so dumb! I thinks I'm only interested in the last thing I accidentally clicked on and ONLY that, there's people I'm still subscribed to that I've forgotten existed because it's showing me Tiktok and a UPS in a data center failing.
It is good to see that some companies do a proper full test of the whole system as i have found that some computer bosses will not allow this full changeover but only to run the generators for a short amount of time that does not do them any good without a load and does not give any indication that the whole system will work as designed when a mains power failure occurs,Foolhardy!
This is a channel I never knew I needed in my life.
Very good that you test the generators under full load, diesel loves to be run hard.
Not sure how your large UPS systems work in the UK but here in the USA, UPS systems of this size and class are always online. We call it double conversion where the utility (mains) AC is rectified and float charges the battery bank. The inverter is always feeding the load. As the rectifier and battery are DC, they are simply spliced together. There is no switch involved. The rectified mains DC is normally about 10 volts above the battery string voltage to keep the batteries in charge mode. Of course there is a static and maintenance bypass switch on each UPS but normally the UPS output is never interrupted during a grid failure. Also at this size, we normally run at 480 volts which is a US standard distribution voltage, that is stepped down to 120/208 volts for the loads.
Hearing a Gen-Set under load ... the most "Beautiful Sound on the Planet" .. during a prolonged power outage. Cheers ........... thanks for sharing this
I love that too ❤️
Why did you put quotes around Beautiful Sound on the Planet?
I was a UPS technician before in a DC and now I am applying as a BMS tech in a tier 3 DC. good thing i came across to your channel and it helps me refresh and learn a lot as well. that i can use on my next job.. "hopefully" i will get the job.
I am just about to get a small UPS for a FreeNAS Server - Now if feels like a joke 😂
Get the 1400 not the 700.
@ sorry i don't unterstand
@@Felix-ve9hs 1400VA
Don't skimp. Get a bit larger. Did you include everything you need for the stability (monitor, remote modem or KVM)? Does it allow extra batteries for more run timeor do you pick the biggest VA so you are far DOWN on the load vs runtime curve
Depends on where you are we have a power failure roughly every ten years for fifteen minutes to half an hour. In Thailand it's a few times a day and brown outs at random.
Just found this channel yesterday and it has quickly become one of my favorite channels. Keep up the good work and the videos coming!
The transfer switch we had was a 3 pump up then turn the knob. Them BAM! It made me jump too like you feel the amps. Then the generator diesel pumped out black smoke for a few seconds while picking up the load. Zero automation by the way.
Nice job, just a quick clarification,why ATS -6 gen takes too long to be up. usually it's less than 12-15 sec. but here it took more than 30 sec .
Do you inform you "grid" about that test? I wonder how they compensate the loss of the load in the grid when you're doing the tests. thx
A friend of 20 years works at Services Australia, Centrelink, Medicare, Many more Data Centre.. he blew my mind when he told me about a Direct Fuel line that runs from a Petrol Service Station close by, Directly to there Generators..... ( He said the system must never go down )
Be interesting to know if it’s unmetered at the servo , direct pipe from main diesel tank , or possibly other fuels ,
And at the government generator plant room , they have a meter for whatever fuel that is pumped from the service station (the servo might have a auxiliary tank that is rented to the Australian government owned (services Australia )
That way re filling the emergency fuel tanks can be done at the (BP , or Caltex )
Better than having a massive fuel tanker enter the complex
"their", please.
Just found the channel and love it. What a great idea shows transparency within the industry, I work for NTT Global EMCA at my site with have 40 Gen-set at 1250kw total load. all of our network is via PLC so we can export back to the grid. keep it up guys
oh wow you work at NTT, does NTT offer single U colo or so in their locations
Cool to see this being done. I work in a DC as a hall engineer. Changing Server motherboards etc. So i hear the tests happening but we never get to see whats being done.
Great to see this being done properly! Far too many places run the generators unloaded as a test.. :/
Management is always too scared.
@@sjf4405 always ends up biting you in the ass!
A loaded test is required to meet NFPA 99 testing requirements. Many newer installations that have an iso-parallel bus will have an on-site load bank so that individual gensets can be tested at full design load, without placing any undue risks on customer equipment. In the example here, one of the parallel generators is turned down in order to increase the load on the remaining units, but doing this presents the risk of running without your engineered redundancy, and still falls short of design loading. The extremely long ATS neutral delay also serves no purpose other than unnecessarily damaging UPS batteries.
That's not a test, just exercising the engine. Absolutely foolish not to test under load on a regular basis.
Nice facility - very clean.
So, can those guys say they drive a V10 at work?
I have a question, if you guys are still watching comments.
Why put the UPS outside of the load for normal operation conditions? What's the benefit of doing so? It almost seems like it'd be much more expensive to do it that way, but I'm sure there's some benefit to it.
I'm more used to mains/gen > UPS > load (with a automatic switch between the mains and the generators in case of mains failure), since the company I work for uses the UPS not only as a immediate backup but also to clean up the power going into the server racks.
I don't understand you. A UPS is always in the load and always out of the load. The batteries are always outside on the backup line. And depending on the configuration (mostly due to specs like load), the basic components (filters, inverters and static switches) are on the main or on the backup lines.
I would say efficiency, most server PSU's have capacitors that will keep the server going when switching the relays.
The correct term is an On-line UPS or Off-Line like shown here.
I'm pretty sure that's what they're doing.
@@adpmol that an online Riello UPS, the generator supply will be filtered through the UPS as the only way for the load to be diverted from the system would be to place the unit into static or external bypass, in which case the UPS would not support the transfer. He has just simplified the process as when the UPS is online its only function is to clean the sinewaves.
Please please please please make these more often :D I watched one video and I HAD to watch those others too! I absolutely LOVE these videos! Thanks for the entertaiment!
at 12:21 why did you guys blur out the stats of the gensets doesn't make much senseI wouldn't think to know what RPM and temperature and frequency is sensitive information
My inner "What if it goes wrong? scenario" and "intermittent beeping noises" nerds are going crazy for this video
Well you just gotta go to the generator then find out what the alarm about....if you can clear the alarm...then you should start the generator manually...then let the ATS/STS take over when power is Stable from Generator, otherwise you should hustle to fix the cause of alarm....
@@alfaheston i think you misunderstand the person’s comment
Awwww those are some cute little generators!
I work in a DC myself as an engineer and it was cool to see the differences and similarities between my site and yours. Wish we had radios though!
these sites are amazing. I take care of a couple of unmanned ILA's for a national data line and the system there is entirely automatic, we do a test every 6 months, even though i've requested a minimum 3 months due to the nature of these sites on the network.... BUT its absolutely terrifying hearing the ATS click, especially since mine is a 3 way between mains, all open and gen. the click is powerful enough to shake the hut and you can feel the transfer through the earth.
That's really great to get info straight from datacenter guys like you, keep up the good work !
Would like to see a video on the type of, and how you test you're fire suppression system(s).
Love these video's as they give us more insight on the daily workings in a DC, and see that the people working there enjoy their jobs.
i know this may be a bit of a secret topic, but i would love to see a video about some of the security of the DC, both network and physical, monitoring and detection, prevention and response
Thank you, this is absolutely golden information
3:47 Subtitles say:
"James you're sleeping"
"Yes"
😂😂
QUALITY CONTENT
Interesting video. Thank you!
What are the green dongles in the background at 12:00+?
probably battery packs for their radio devices
Question. So Power output of your generator does not provide charge/power to UPS & its batteries when it comes online? Meaning it bypasses the UPS System entirely? Thank you!
You'd think that would be hitting the UPS. You'd want that between the hardware and generator.. Unless there are other UPS's downstream.
@@deanmarsh1452 Having done an IPVT of our system. The generator does supply power to the UPS. The UPS momebtsrioy goes on bttery as the ATS does its job. The UPS doesnt care where the power comes from (mains or generator) it cleans the energy through double conversion anyways.
I have recently carried out a series of load-bank tests on mission-critical sets from 40kVA to 2000 kVA. The customer used to run the sets up once a month to make sure they started, but never ran on full load. A lot of them were over-specified, resulting in carbon build-up, sticky valves, clogged silencers etc. My job was to run the sets up to 100% of plate rating for 4 hours, to make sure that they do not fail under operating conditions. After the usual checks and inspection, staged increases of loading, (5 minutes warm-up, 5 minutes 10%, 10 minutes 25%, 10 minutes 50%, then the balance of 4 hours at 100%), were implemented. One set would only run to 80% without overheating, another had a failed turbo-charger bearing, another had a radiator leak that only showed on full load, etc, etc. Once the test is complete, a 10 minute no-load cool-down period is done.
Once the customer was aware of the issues, and they were dealt with, their decision was to load-bank the sets bi-weekly. (It's a great job for a winter's day; parking 20 feet in front of a 400kW fan-heater certainly keeps the van warm!)
How come you didn't do an overload test like running them at 110%?
@@TheManLab7 I am guessing because overloading is generally not a situation the generator is designed to handle on a bi-weekly schedule and might result in damage more than detect it.
@@TheManLab7 I imagine they were accurately specified and would never run at 110%. The control systems should not be designed to ask for a damaging output from the genset. Another assumption here but in the case of less than N working generators during an outage I would program the control software to keep the generators capped at 100% and use the UPSes to hold up the rest of the loads while a decision is made about which loads to cut or throttle down if it is not automatic. If you need 3 gensets and you only have 2 running it is better to keep your 2 at 100% by cutting loads (assuming you can't hold the excess load until an emergency genset can arrive) than it is to go dark because you broke them trying to keep everything running. If I was running a NOC all my major services/applications would need to be locally throttleable, at least manually.
Your question isn't silly, I just don't think that's the way NOC administrators and operators think.
That must have been a nice smokeshow as the exhaust got up to temperature for the first time a in a long while.
Man, that's a lot of power out of that tiny engine. Ours is .9 megawatt and is easily three times that size. It's the one part of my job that I truly loathe - monthly engine runs. It is incredibly foolish to stand in front of those ATS panels when the transfers takes place. I flip the switch and take off. I can get out of the room into a safe spot before the transfer takes place. Hopefully this guy won't end up in some arc flash accident video later on down the road. it scares me just watching him in this video.
Given the lack of high voltage warnings anywhere, I imagine that they're safe from arc flash with the cabinets closed.
A V10 is pretty much never a tiny engine.
great video, why is it taking so long to switch from primary to secondary input?
does it need to sync frequenzy or something?
Very informative ! Glad I found this channel.
I felt like George Russell was giving me the tour
Question: do you contact your power supplier before starting the test? They will see a significant change in their power output the moment you switch off
No the power company is usually not notified. The load change is not as much as one would think, he is switching one ATS at a time, not dumping the entire building onto the generators simultaneously. I worked in a descent size hospital, we had 3200KW worth of generator power and would switch over the entire building once a month, never called anyone outside the facility to notify of what we were doing.
Depending on the size of the power grid. Really large ones won't even notice too much of a power change as it will rebalance automatically.
Thank you very much for fantastik video👍👍👍Greate job!
Just a question: Why you choose to run the gen-set for 4 hours?
Is related to your particular SLA or is based on some norm or recomendation ?
Thanks..
The tier of the data center. It’s mandatory by the tier you’re set at.
@@kylelee639 Thanks
linus tech tips data centre edition
The dude on the radio sounds like a imperial droid
hahahahahahah
At my company we use a manual diesel generator which means we have to manually swap all distribution boards one by one to the gen which is fun lol, jealous at how simple and automated a data centre can do it which makes sense when you cant afford any downtime whatsoever
If the gennys don't get the signal to start, how long do you have on the UPS to get them started manually? Also, It would be great to see a split-screen version of this, watching the generators start up in real time with the ATS switching over
Don't know if this will help, but generally, you'd be expecting at least 15 minutes of autonomy time although I reckon that these folks have at 20+ minutes.
In a datacenter like that i would expect that they can run the complete 2 hours until a new generator gets there, on batteries. A datacenter cannot afford any downtime and batteries are cheap compared to the cost if they run out of power. A single rack loosing power for less than a second might take over 20 minutes to restart and booting up completely, which means thousands of people not being able to work, and that's just one rack, and assuming everything boots up normally
@@MarioGaricTravel Based on the video, it looked like 10-20 minutes from extrapolating off the time the UPS was waiting for the genset to engage and the 10% drop in battery SOC displayed.
@@Newspaperman57 Definitely not that long under the load we saw.
Lookup what arc flash is. You should have protection gear. Good reason it makes you jump.
Thank you, Rich...I was wondering why he was so jumpy (Yep, I looked it up)...it's not the noise, it's the danger of arc flash. Protective gear would be a good idea.
Who came from the youtube algorithm?
Me
5:30 "bang"
There's always the question.... was it a good bang or or did something explode?
I guess that one good thing is that you've heard it.
Interesting to see this being done this is something I get involved in . Just wondering what if the electronics don’t work are you able to start and sync the generators by hand and switch the system by hand ?
Very nice to see this kind of procedure. Just curious what capacity are those UPS ?
I was surprised to see the hold up time seem so short, what was that? Maybe 60 seconds to 90%, maybe it was rounding down to 90% so 20 minutes at best?
@@kingrpriddick And I was surprised it's so long. 10-20 times the switch over to the generators is a huge safety margin. But that's probably needed to have enough of a buffer for the 2 hours to get a mobile generator when too many of the fixed ones fail.
great work guys!! scaring during CB operation happens to me everytime too :)
The frequency drop at 8:53 doesn't sound good for the equipment. Is there no possibility for something like a soft starter (such as slowly powering down the inverter while in parallel with the generator) for the changeover, or is it impossible for the UPS to sync with the generator so this can happen?
Edit: explanation makes it sound like UPS is cutting out and dumping the entire load onto the generator, which sounds like it slows down momentarily while the governor reacts.
with a double conversion UPS which these certainly are, there isn't really any need to sync the frequency exactly, but the ATS still needs to cut over on a zero crossing.
What you hear has no relation to the AC frequency, it's just the UPS PFC/rectifier circuitry losing power and DC bus being loaded which makes the UPS adjust the inverter switching duty cycle. This is normal. The output AC frequency is going to be exactly the same as was the input frequency since most double conversion units sync the output to the input (except for some rare special cases like frequency conversion 50 -> 60 and vice versa) to be able to safely switch into bypass at any time. It would have to freerun for much more time to have the frequency wander off a bit and when the input power is restored, the UPS will first sync the output back to the input before switching back from the battery.
you should've said "bravo Six, going dark" on the radio, the other guy sounded like makarov on the radio. You getting scared gets me all the time 🤣🤣🤣🤣
Over by the ups was the vent on top for battry off gass or for heat offput
Hi, can you show a video of the power distribution, Switchgear, UPS, PDU and connection to the racks.
Good video - I used to do monthly genset testing on several large data centres (From 3 to 12 Meg). Once had the mains actually fail just before I conducted the test - saved me some work. 👍😁. Gensets on the 3MVA setup were V12 50 litre engines, the 12 meg site had 6 x 2MVA - 78 litre, V20’s. 👍
do you want a medal or a cookie?
Regarding the implemetantations of the generators, was there a handover, regarding what procedures to follow when it comes to testing, periodic checks, and switching over from the mains to the generators?
I swear I thought I heard him say "not yet" and then you say okay, let's turn the key. lol
I swear it was "Not yet" and then he did say okay and turned it lol. I guess if it was "not yet" and he did that, we would of heard some non so nice of oxford english.
I heard "I don't see any reason not to" instead of "not yet"
@@Timooooooooooooooo in other Words: Whatever happens after you turn the key off. Im not responsible for what happens next :D
You can tell this dude is on the verge of a nervous breakdown.
@@TAK-YON_ well, yeah. I mean, game recognizes game
I like your videos, I am just wondering if 10v1600 series is enough to get the energy for the hole data center. They have only 700kva and for example 20v4000 series has 4000kva output. Maybe it makes sense to replace all small once to 2 bigger once :)
It looks like each UPS has a fume hood. I guess that is for hydrogen from lead-acid batteries if they are being charged too fast for whatever reason.
12:38 what are the three green beans?
I wouldn't stand in front of those breakers when they close.
someone's going to argue there's very little reason to move but for me I don't wanna be the very unlucky soul to experience the arc flash from one of those 😂
At least, don't be standing in front while wearing synthetic clothing (unless it's nomex), and look away.
One question... When you are on generators, do you charge the UPS' or is all the power going to the floor to keep those awake?
i believe that it does both at the same time, because power is needed in the data center and ups' need to be charged as well to make it safe to switch back to mains
@@myRAYgame-Disk2 I think normal switching time on the "usual consumer UPS" is around 15ms. Minimum Hold up time for a Powersupply on a Computer/Server has to be 20ms, IIRC. So there is no chance, that a good quality PSU on a Server actually even gets "drained", even under full load, if the UPS Switches from/to Mains, which is actually pretty insane. I wonder what the Voltage levels look like while that happens tho.
Great video , very well explained ⚡️
i love how this all comes across as a walk in the park during the test. I've been on site during a power failure waiting for a generator to kick in before the UPS dies and i can tell you its one of the most stressful times in an IT guys life....
So if you have 3 generators each with 0.5MW Output and you do a load test on the entire data centre, do you have to schedule a day and time to do the load test with National Grid? I can't imagine they would be happy with 1.5MW suddenly dropping off the grid in the space of 10 minutes, that would likely cause a frequency change as well as a voltage change, neither of which are particularly good for the grid and anythng connected to it.
Hmm yes of course. I was thhinking the same thing
In reality the although the generators may be 0.5MW each, they should/would be overspec'd on purpose, for the case of generators failing while on load. I would guess that it may be the case that the whole site may only actually require a fraction of the total available output.
@@bsmith2053 That may be so, but imagine the entire Data Centre is using 0.5-1MW at any one time and maybe 2MW max, if you take that loading off the grid without alerting them (and without there being a failure to the supply in the area) then they might wonder what's going on. With that sudden decrease in loading, that could cause the National Grid to have to turn off a windfarm or slow down a power plant.
@@CoolSteve08 I still don't believe that even at 2MW the network would need to do too much in terms of compensation. I think the power stations would see this as a small blip in the scheme of things. And if that meant though that sources were taken offline/diverted so be it.
I mean Imagine they got some monster Power Factor Correction Caps attached to the building power in so that problem doesn't happen.
Guy on radio: no, yes!
You should agree on clearly speaking. Look to aviation or military.
Definitely heard what you heard -- god forbid the guy meant 'no', what would he have said? "Yes, no?!!?" ... Hopefully, however, the system is robust enough that no true level of 'readiness' is actually necessary, since as we all know, the power companies are more often than not unable to give a warning before the mains power goes out!
If the Gens fail during your test you have to put grid power back manually?
Belleville Drivers if there was a failure of the gen during testing, you’d flick the key switch back to mains and it would retracted the ATS back to the mains. It’s a controlled environment doing these tests when you know the mains is stable.
@ 5:27...get used to it man...LOL!!!
2:00 i think it run at night by itself and return back to not get cought, smart energy working.
Love your videos guys. Keep up the good work!
Great to see this stuff. Thanks for showing us this testing.
That's a nice PHP weathermapper graph you have there :)
i need this for my off grid cabin data server
Pff small gensets and breakers, on my previous work they have breakers of 3kV/150A on the small blowers they make a BANG if they switch. The gen sets are automated synced with the grid, 1 genset is 3kV/2000A. This is nice to :)
5:26 yes that contact made sure you knew it was kicking in.
How often do you run the generators in a year? Just thinking the feels won't last forever, and the engines needs to run atleast 4-5 times a year at a minimum...
0:10 on a monthly basis
Every month 2 of them run and one is off. So at worst case they run every 2 Months for 4+ Hours which is sufficient for these Engines to warm up und lube everything and get rid of condensation in the oil. I guess Engine Oil swap is also done regularly, because these become the heart of the Datacenter, as soon as Power drops, which is why they test these every Month for an extensive time.
The longest wait of your life when doing mains fail, 30 seconds waiting for generator contactor /ACB to close, a million possibilities go through your mind 😂😂
How many gensets do you have?? I saw 3 of course, but do I understand you have 3 more??
I have run outdoor concerts on 10 Mw but this is very nice indeed, ty for the info
Those big ATS auto breakers scare the $#@^& out of me too! every time!!
extremely interesting video guys, keep it up 😉
I remember one time I was working in some 33kV circuit breakers in a switchroom (dead testing) and we agreed with the guy testing them that he would shout switching as a warning but he just mumbled it quietly and hit the button anyway. Loud enough and strong enough to shake the CB and and the room. We exchanged words as you can imagine. Anyway this just reminded me of it.
Anyone know thr names of those ATS'. What type of ATS' those are. Or just what I'd have to slap in google to find ATS' like those. I've been looking for a while and i can only find 'small' ones. I only see ATS' like those in diagrams. Also, is each generator connected to each ATS or just one gen per ATS? Might seem like a dumb question but i was a little confused when the second ATS was aware that some generators were on.
These are not traditional ATSs made by someome such as Asco. Due to size and protection this is low voltage switchgear with electrically operated breakers
This may look simple but testing all these generators, ATSs, UPSs, and alarm setpoints WITH IT LOAD is very ballsy. Well done.
you are absolutely right, i've seen an ATS hang up and fail to transfer and shorted out a dc rectifier and exploded the controller and blew the door open. Anyone switching breakers or ATS with load should either be standing outside the arc flash boundary when the actual switch happens or wearing an arc flash suite. There is enough current that when shit goes wrong, it goes wrong big time.
I mean if you're testing it without load, you're not really testing it, so I question the value of the test.
Hey, what will happen when a catastrothic power failure strikes the Data Center and a restore of power will be in 1 week or so ?
I guess there would be a extremely big natural catastrophe if it'd take this long to restore power to the grid.
But, I guess they just use those generators and get new fuel.
@10:50 why is that AC outside the building? ಠ_ಠ
Thats the second floor or rooftop of the main entrance , you can see theres a transparent roof , but theres a window open :D
Tis not.
I feel like slamming a door would give this dude ptsd
Ikr.
🤣😂🤣😂 Totally he's too jumpy
Needs to clean out his shorts after this video!
How long it takes to change over to genset source from utilities source?
*stares in procedure use & adherence and place keeping*
Is this just a continual promotion for this Data center to pickup prospective employees?
Ayyy, Edgar’s Water, the same water dispenser fairy that we have at work!
Really cool video guys. Thank you!
Well in my country we don't need a generator test, as the test runs automatically at least 5 times a day 😂
Are you saying that you have unreliable power?
@@jamescollins6085 sure, in summer and winter we have 50% power time. Else we get 70-80% power time..
Lol
You must live in California!
No I live in Baghdad
Only 500kw? We have twin 2 megawatt gens with a flywheel cutover just for our office lab :)
Pushing the E stop on a Genset equipped with a deep sea panel is not an acceptable way of isolating an engine from starting. You should be turning off the battery isolator !
Good video; you think it’s loud when it works properly; I was doing a test one time and a a switch jammed; so loud, and you could feel the thud; I jumped off the ground; I had to abort the test that time.
Okay, you test it what if one of the things fail, will the data center go offline? Such as the UPS?
The UPS is the “middle man” between the grid and generator power. When grid power goes out the generator needs to start (it’s just an engine), and get it’s power output up to an acceptable level (pretty much the same as starting a car and putting it into gear). That process takes around a minute, the UPS is required to keep both the data centre and the systems that manage the generator startup process for the minute or so it takes for the generator to start providing output.
So the answer is yes, if the UPS fails the data centre would go offline, BUT there is more then one ups, and many built in redundancies (so it would be very very unlikely for that to happen).
If the generator fails the UPS can continue providing power, albeit not for long, but long enough to identify an issue and potentially fix or call for an emergency generator.