What Really Happened with the Substation Attack in North Carolina?
HTML-код
- Опубликовано: 16 янв 2023
- An overview of the substation attack in Moore County, North Carolina in December 2022.
📺 Sign up for Nebula to support the channel and watch my videos ad-free! go.nebula.tv/practical-engine...
Sorry I sound a bit stuffy in this one! We had to shoot this before everyone left for the holiday break, and I was still getting over a cold.
This event highlights the need for making critical substations more secure and also making the grid more robust so that someone can’t rob tens of thousands of people of their lights, heat, comfort, and livelihood for four days with just a few well-placed bullets.
Signed copies of my book (plus other cool stuff) are available here: store.practical.engineering/
Practical Engineering is a RUclips channel about infrastructure and the human-made world around us. It is hosted, written, and produced by Grady Hillhouse. We have new videos posted regularly, so please subscribe for updates. If you enjoyed the video, hit that ‘like’ button, give us a comment, or watch another of our videos!
CONNECT WITH ME
____________________________________
Website: practical.engineering
Twitter: / hillhousegrady
Instagram: / practicalengineering
Reddit: / practicalengineering
Facebook: / practicalengineergrady
Patreon: / practicalengineering
SPONSORSHIP INQUIRIES
____________________________________
Please email my agent at practicalengineering@standard.tv
DISCLAIMER
____________________________________
This is not engineering advice. Everything here is for informational and entertainment purposes only. Contact an engineer licensed to practice in your area if you need professional advice or services. All non-licensed clips used for fair use commentary, criticism, and educational purposes.
SPECIAL THANKS
____________________________________
This video is sponsored by Nebula.
Stock video and imagery provided by Getty Images.
Music by Epidemic Sound: epidemicsound.com/creator
Tonic and Energy by Elexive is licensed under a Creative Commons Attribution License
Source: • Elexive - Tonic and En...
Producer/Writer/Host: Grady Hillhouse
Editor: Wesley Crump
Production Assistant: Josh Lorenz
Script Editor: Ralph Crewe
Background Painting: Josh Welker
Graphics: Nebula Studios
Drone Videography: Sandhills Aerial
Bullet hole photos: Jonathan Drake
![Natanael Cano - Ya Te Olvidé [Official Video]](http://i.ytimg.com/vi/xLig8I91SJQ/mqdefault.jpg)
📺Watch your favorite creators ad-free on Nebula: go.nebula.tv/practical-engineering
📚What do you think? Are you willing to pay higher rates for increased physical security, or are you okay with the risk of a few outages to get cheaper electricity?
😷Sorry I sound a little stuffy in this one! Wanting to get this out as possible meant having to shoot with a cold before everyone left for the holidays.
Civilization seems to be in a declining phase, so paying for addition security would be prudent.
You sure about Nebula though? That link lists it under the premium plans, not the $15 standard one.
Why isn't the transformers manufactured on a standardised spec to reduce cost and increase interchangeability?
@@imag0r I am pretty sure that it links to Curiosity Stream.
You need to apologize for terror-washing. Investigators linked what happened here to the western board substation attacks and they're *not* struggling to call it terrorism. They ARE calling it terrorism.
The "proud boys" had a plan to do this and low & behold, it happened. Just them too. Not a bunch of mixed groups, just them. So you need to apologize because you insulted the intelligence of a lot of people with this one. We already know what this was.
In fact you've got to be purposeful to be covering up their actions as a mystery.
Highly inappropriate
Grady quietly dragging the shooter for reading the grid diagram wrong is low-key hilarious.
I would guess they just looked at the powerlines
@@AUTgriesbrei then they would of known it was not a connection to the 230KV line
Could’ve just done that to be safe that it would take that area out. Better safe than sorry, as they say
@@pyropulseIXXI you keep using that word. I don't think it means what you think it means.
@@dr.kraemer bol
Someone I know worked on putting these 2 subsations back online , pretty much everything in this video is correct and actually happened. The other mess he told me about was the sewage treatment plant backing up and causing their pumps to trip off and he had to wade through raw sewage to fix the jammed up pumps. Props to the electrical engineers and all the hard workers that fixed this mess in a couple of days
Electrical engineers got their hands dirty?!
@@TheCastedone ?
@@norbertnagy5514 it's a joke that electrical engineers are very hands off in practicing their craft. Really any engineer will get a bit of ribbing from a lot of tradespeople because even if you do get in the field and get your hands dirty, you also spend a fair amount of time sitting at a desk. At the end of the day it's just banter.
A buddy of mine works at a local sewage treatment plant. One of his jobs is to clean out the digestors when they get clogged by people flushing junk down the toilet. He gets dressed up in an old fashioned diver's suit with a brass helmet and oxygen hose and his descends into the goop. When he emerges a little later, he's covered in hypodermic needles and tampon applicators and someone has to carefully remove everything before he can remove his suit.
@@JeffS96 i understand now thx.
I’m a Duke employee and my dad and brother are too. They both worked this outage. I was pleasantly surprised that everything you said in this video was 100% accurate
Someone (or several someones) shot up electrical equipment at a substation in California too several years ago, exact same MO as this attack
@jonslg240 someone? Just like that huh, drink a 12 pack and grab a rifle lookin fer some fun......couldn't possibly be a state actor? Nooooo way
@@thelugoffgamecock792why do some people never miss an opportunity to show everyone who they are?
Thirty years ago I worked for a company that built and repaired transformers, especially the massive ones. Each one is unique, and they are not built by automated machines, both the coils and the stacks are made by hand with mechanical assists; these are skilled and time consuming jobs. The big ones are not something that can be replaced in a month, let alone in days, plus they require special transportation just to move them due to size. And Grady is right, there are no replacements sitting around in storage anywhere to be swapped in if something like this happens on a larger scale.
How well are they hardened against EMP? If the Carrington event happened again would it destroy the grid?
Well that’s yet another example of bone-headed engineering. What a cluster.
Why are they unique, though? It stands to reason that thousands of them are needed and they do roughly the same thing (converting from a standardized high voltage to standardized lower voltage), so it should be possible to settle on a few sizes for different power requirements and have standardised designs.
@@1Maklak Exactly. Doing it any other way is the absolute height of stupidity.
@@1Maklak They are unique because setting up a production line where thousands are produced every day, the way cars are made, doesn’t make sense because you don’t need to make that many.
They are huge and complex structures, with a small but very critical market, and as such it makes more financial sense to have engineers make the transformers themselves, instead of making the many machines required to automate the making of transformers.
In addition, they may do the same thing but they can have wildly different loads required. The transformer in a power station for NYC deals with wildly different electrical loads than that of London, Kentucky.
At a power plant where I worked, the main output transformers were tested every outage. When testing showed they were nearing their end of life, replacements were ordered. Built and shipped from overseas, took over a year to get them. When one started developing further issues, we had to derate (reduce output about 50%) until the replacements arrived and could be installed. These monsters are pretty much custom and built-to-order.
It's a major weakness these terrorists are aware of. They know if they hit enough of them they'll overwhelm our replacement ability in short order. Its being actively discussed among the groups being monitored. Just a FYI.
Overseas?
That's awful. They should be produced here.
Out of curiosity what do you measure to determine it is near their EOL? Leakage for degradation of the isolation?
@@LeviathantheMightybeing built overseas likely had very little to do with the time, and getting the equipment to manufacture them here would either have no impact, or a negative impact on time to produce.
Overseas. Yeah. Lower costs, lower pay and benefits for the employees.
Vital construction should be "in country".
When I did my military engineering training, we had a special course on "Urban Denial." The basic concept was how to make a large urban center uninhabitable. We had experts from the electric companies, water and sewer departments of the local city, and the natural gas company come to speak to us. They walked us through step-by-step how to shut down electricity, water and sewer, and gas in a modern city. If you have inside knowledge, this is disturbingly simple. The key is to attack items that are crucial to the system but hard to repair or replace (like the transformers in the video above). The electrical expert said, with a single box of TNT, he could disable electricity to the entire city in an hour or two. Power is one thing, but if you cut off water, sewer and heat too, things get very bad very fast. Most key services (hospitals, police, fire) have generator backups, but they usually only have a week or so of fuel. They can be resupplied, but millions of people can't be. Look at what happened in 2003, a software bug caused a cascade failure that knocked out power across multiple states in north-east USA and Ontario, Canada leaving millions without power. That only lasted a few hours. Now, multiply that chaos by weeks or even months. I am pleased to see Grady bringing this risk to people's attention. You need to be prepared to take care of yourself and your family in the event of an emergency. Everyone should be prepared for at least 72 hours with no outside assistance. If you are able, you should consider extending that to 2 - 3 weeks. Camping gear (camp stove, lanterns, sleeping bags) and preserved food are a good start.
It's happening right now in Ukraine. Suicide drones/loitering munitions hit transformers and other key nodes at a rate that will be impossible to repair without enormous time resources and effort
I know of a transformer station in my country that if attacked would knock out power for millions and is in such a place it's power distribution can't be replaced.
It is heavily fortified including lethal electric fencing which is unheard of where I live , military and government installations don't even have this protection.
You posses extremely dangerous knowledge.
@Peters6221 I don't believe in coincidences for the most part, so I'd say it's definitely related. The way the "authorities" are being so tight-liped about it also stinks of something more. The idea of someone disgruntled is just too simple when you look at all the other implications of the attack- Special Operations military base, renowned golf course and housing community for the elite... this was either a test, or a distraction for something else. Theft? Data breach? Something... I mean hell, the "authorities" STILL don't know who, how and when the anthrax was stolen from the US Army's obscure chemical & biological weapons "research" facility in MD back in the 90s... you know the same anthrax that was being mailed to a bunch of important people around the country. Weird times and getting weirder...
What's the military application for making a city uninhabitable?
My family company designs high voltage transformer stations in Slovenia. What we do now is we build walls around transformers. It is a fire and anti explosion safety feature. It is becoming a standard.
Transformers are in mass production and are possible to repair. Few months ago we moved from active service to a museum a transformer made in 1920. They are made to last!
I live in Richmond County, which is the next county to the south of Moore. These outages are always especially dangerous to those residents who rely on electrical power for life-sustaining medical equipment. For anyone who is reliant on electrical power to remain healthy, take the time to invest in a backup generator of sufficient capacity to power your medical equipment. Keep at least 5 gallons of clean fuel on hand at all times, treat this fuel with stabilizer, and become proficient at starting and properly setting up your generator. Letting the engine run for at least 15 minutes once a month will help to prevent stale fuel from gumming up the carburetor, and ensures the engine will start and run when it is needed. Yes, I practice this method I preach at home myself, albeit with an old Navy surplus carburetor-less diesel generator
I work for a local small engine repair shop, and tried to help as much as possible by expediting repairs to everyone's portable generators who lived in Moore Country when this happened. I appreciate the coverage of this vandalism, and hope such events don't become a new trend. Power outages are an inconvenience during the best weather, but can quickly turn deadly during periods of extreme heat or cold.
Propane is a fantastic fuel for backup generators because it stores forever, no issues with gummed up carburettors etc.
As a former Diesel Generator operator &. mechanic, if you use Oil Heating I recommend getting a diesel generator that will run on #2 Fuel Oil (which is a "kissing cousin" to winterized #2 Diesel fuel. Basically you have a large tank of fuel that will last you through even a major disaster!
@@timengineman2nd714An absolutely terrific idea! And don't neglect to upgrade your fuel oil storage capacity to make sure you have ample #2 oil on hand to run both the furnace and the generator!
Substations are vulnerable, but high voltage transmission line towers in remote locations are completely indefensible.
They are, but they're also pretty sturdy. It would take a fair amount of explosives to topple one and it would be quite difficult not to draw attention. You'd also need to get away far enough that you won't be enclosed by a lockdown of the area.
Then you probably also need to hit the second part of the loop which is going to be hundreds of miles away. Quite difficult to organize and likely impossible to do yourself.
The repairs will most definitely be costly, but the parts for it are standardized and the hotfix will likely be relatively quick.
If I was thinking like someone that wants to do the most damage with the highest emotional impact at the least amount of risk, I wouldn't really see it as a top tier option.
True, but overhead transmission lines are exposed to weather, and that presents a more frequent risk. So the transmission grid must be designed to withstand the loss of individual lines.
@@paulelderson934 lol you could take down a 700kv tower with less than $100 of tannerite
@@paulelderson934 it could be that the sabotaging equipment is set to fire off a couple days after installation. It's highly unilkeliy for someone to be around to find it and alert the police.
@@besacciaesteban It's especially unlikely to be noticed if it doesn't look like a homebrew device and instead could be confused with something the provider could add themselves by a lay person.
I work as a consulting PM for a major utility. For the last year I have been managing security projects to harden sub stations. Some stations are pretty much fortresses while others are lightly protected. That is quickly changing, they can track ballistics and drones.
Also my latest material lead times say transformers are in excess of 60 week lead times from transmission transformers
@@tribalismblindsthembutnoty124 turn everything off and it will be fine
@@tribalismblindsthembutnoty124 Basically nothing against a powerful CME, it would basically send humanity to the dark ages. An EMP would probably take out a lot, but it wouldn't be as completely catastrophic as a CME. In the event of a EMP generated by a nuke, I think the fallout is a lot more of a concern than the EMP.
Such a mean-spirited attack; I cannot guess the motivations for it. My grandfather spent a lifetime in power distribution.
Solar storms.... Earth weakening magnetic field....
@@nikkiofthevalley Not sure I agree. CME is generally predicable, and utilities have standard operating measures they take to prevent events from causing damage. And there are also mitigation steps that can be included in transmission infrastructure design.
Correction. Those are not ballistic resistant walls. They are fire walls. They are built to contain a fire to keep it from spreading instead of using a fire suppression system(Sprinkler System).
Idk 8in of concrete is pretty resistant to most ballistics
Well tbh bro we have more guns than people and most of the time no one is shooting at the stations so i can see why it would seem more exoensive than its worth to litterally bulletproof the whole system
You are sadly mistaken.
That was my immediate thought. It is what it would look like though. Plenty of videos of 50 cal going right through 10" of concrete on RUclips.
@@powderdropzone 50 BMG is an Anti-armour round. Saying something is not ballistic resistant because it doesn't resist something made to counter it is kind of putting the cart before the horse.
Also just because something isn't used in a manner that is meant to be ballistic resistant doesn't stop it from being ballistic resistant.
6:59 The mobile transformer looked really neat, almost menacing. Would have been cool to see going down the road, but I bet it had a cover.
One up for all the great people who have over-thought infrastructure emergencies.
What is your opinion of those in Congress who have consistently refused to harden our Grid against EMP attacks?
Three nukes detonated 120 miles above the Continental USA will put our power grid out of action for years. The population dieoff from exposure, starvation, thirst, and breakdown in the rule of law is reliably estimated at 90%.
Our ICBM defenses have assumed an over-the-pole attack.....There is no defense against missiles launched via the gulf coast...... NO DEFENSE!
One up for all the great people who have over-thought infrastructure emergencies??? Seriously??????
I agree, it's a great bit of kit to have on standby
They don't have a cover for them. Some of the equipment on the portable might fold up for transportation.
Another youtuber - Bobsdecline - has a video about a mobile substation his utility uses. It's smaller than the one Duke used in this case (138/69kV vs 230kV) but should be the same concept. ruclips.net/video/hh23MTPrvKc/видео.html
@@matthewroutt3938
Thank you Matt for sharing, checking it out now! I believe it’s an excellent concept and praise whom created it.
WOW Ive been an electrician for 40yrs and Ive never seen a mobile substation. That was impressive.
I've been all over the country driving truck and I've never seen one either. I think I might try building a model. Looks cool.
Sounds like a motive. Real sus.
@@eligebrown8998 a model would be awesome
Here in Ontario they have several , they are used when it's necessary to upgrade or renovate electrical substations. The person who damaged this small substation is a terrorist, especially since they knew what they were doing.
I built a temp line for a sub crew to rebuild part of a station. Their mobile was awesome i have pictures of it still if anyone would like to see.
Hi Grady! I am a substation engineer, and I've specifically worked on designing substation security in the past. It is true that ballistic barriers are becoming more common, however they are very expensive. This is because the massive height requirement of the walls themselves. When you are considering protection from firearms, you must do a line of sight calculation from the highest advantage point available. Do some basic trig, and there can easily be 40-50ft high walls to protect some of the bigger transformers in higher kV substations!
Why don't they bury substations?
@@12101DyM Cost. Same reason pylons are favored over buried cables except in built-up areas: Digging big holes is expensive.
@@12101DyM Then you would also need costly mecanical ventilation plus tons of permit to dig, plus tons of permits to dispose of the contaminated dirt and how many sub station is required ?
What about constructing a building around the substation, similar to how a landline telephone company has a "central office?"
@@Eternal_Tech There is such a building in Charleston, WV, but I'm not sure if it was to protect the substation, or so it wouldn't look so ugly in a busy part of town. It's concrete and red brick, and it has heavy wooden doors.
I got stuck in Moore county overnight on a trip back from the Lantern Festival. My Girlfriend and I stopped at a gas station outside of the town,where she didn't want to get gas at, prices were too high and she was sketched out. So we left and drove into Moore County where we found a gas station in a dark part of town. After gas station hopping, we decided to call Family for help, and had to wait a few hours for them to get there. Meanwhile we were reading reports of how everything was all unfolding. Originally we thought something must have fell on a powerline.. and services would be restored soon. We were wrong. Met some friendly people. Police officers, a lady gave us a gas jug... in case someone would have let us siphon some of their out of their car.
Grady, you’re the best. Please keep this style and take your time with this amazing content!!!!
Any 230KV transformer requires about 5-7 days period to bowser (multiple vacuum pulldowns, vacuum fill the oil from the bottom up, heat the oil, circulate the oil through water separation filtration). There is careful testing done to assure moisture levels are safe for energization. Once the envelope is penetrated by moist ambient air, there is no recourse. You have to perform this process. This time doesn't include draw down of the remaining oil, repair of the radiators, etc. Typically, the radiators can be repaired by welding. Also, utilities used to keep wooden pegs to drive into the holes to 1) stop leaking and 2) possibly prevent tripping, 3)refill oil if the envelope still has a nitrogen blanket. The low oil tripping schemes are pretty high up in the tank. The float will go to the alarm level first and then trip afterwards. The goal is to trip before any sensitive components are above the oil. I have first hand experience with this type failure (low tank leaks) and generally there is a small vacuum formed by the escaping oil and it does leak but not at a tremendous rate. Probably the worst of the leak is due to the flow rate of the nitrogen regulator trying to maintain the blanket. So it takes a while for the transformer to go into alarm and then trip. Luckily all of this happens before serious damage is done. If the radiators are penetrated in a way that damages multiple layers, those have to be removed, cut apart, repaired and rewelded/layered to a full assembly. That takes a while. Many utilities have a full repair facility for power equipment and that would be done there and returned to the substation. In some cases, it might be possible to get spare radiators from other transformers. For instance, the 230/115KV Autobank(s) in a substation could have all good radiators put on one transformer if they are in fact the same dimensions and mounts. Also, radiators could be removed, the flanges blocked off and operate the transformer at reduced load until scheduled repairs can occur. In a situation like this, you do whatever you have to to get the lights on.
That location is unusual with a looped 115 system and no alternate feeds. Generally there is a large 115 network and 230 network with multiple dispersed 230/115 locations where those networks tie.
A side note these 230/115 autotransformers also play an important role as a ground current source. A ground fault on the 115kv will be seen as a 3 phase event on the 230kv network due to the closed delta tertiary windings. This allows the generators and 230kv system & up to function as a three phase source for any 115kv problems. A useful feature, it works the same way in either direction.
Another note, the low oil tripping systems are usually hardwired. No operator intervention required. The nature of these gauges, microswitches and the fact that the transformer is a grounding source plays into the design of the system. Most use a blocking method where the tripping coil is shorted by one contact which must open before the contact that makes causes the coil to operate. So a contact must part and one must make for the trip to occur. There is usually a current limiting resistor in there in case both contact are made at the same time. It is pretty hard to have a misop of that system unless you are doing maintenance on the indicator and forget to block it. Yep, it happens. It's not hard for everyone to 'know your name'.
Thank you for that. Very interesting and tidbits someone like myself not in the field would never know but love to learn. 🍻
I thought it was super odd that they have 115kV acting as a sub-transmission grid. What I am thinking is that the 115kV was existing, then built out 230kV when they had the load pocket grow, and just converted the 115kV to a de facto sub transmission loop. Would somewhat explain the fly by at the Carthage station as they would have built out one side of the ROW with 230 while the 115 was either far enough away or under something like daily outages, then cutover the 115 to the 230 station at West End.
@@fensoxx No problem. 230kv and higher transformers require a lot of time to prep for energization. I thought it was important to express what the utility was up against. The transformers may have had a good blanket, no way to know w/o being there. With the investment involved they would and should be conservative and assume water contamination no matter. Believe me, no utility wants customers offline. Kudos to the utility for working through this mess.
I don't understand electricity that much but I was growing marijuana about 15 years ago oh, and I had all 120-volt ballast running throughout the house in the power bill was like 2500 bucks a month
My buddy told me I could cut my power bill in half if I switch to 240-volt ballast and daisy- chained them to each other and he said Each one that had extra power would just give it to the next one and they would run super-efficient like that- He was RIGHT. MY BILL DROPOED ABOUT 48% IMMEDIATELY
Long story short one day the power went out and I heard a loud explosion outside,
The power guy showed up at my house
Said
"YOU GOT 2 CHOICES"
1- YOU CUT ME A CHECK RIGHT NOW FOR $5800 CASH TO REPLACE THE TRANSFORMER
2- WE CALL OUR INVESTIGATION UNIT OUT TO FIND OUT 'WHY IT BLEW'??????
Needless to say I grabbed my checkbook and disconnected 32lights
(Leaving 46 connected - around 188amps total)
@@gigaWUTT Ya know, I need to watch it again but that looked like a 230 to distribution level voltage (15kv or 27kv) class Mobile unit. It is very common to install those temporary out on a right away and tie into the 230 and one or more distribution feeders or the original distribution low side bus. I would be floored to find out any utility had a 230/115 mobile solution. Those can't be shipped with the bushings installed due to height. Refer back to the processing time to place bushings and fill with oil. Also, those Autobanks are very heavy dressed. Typically shipped on rail stripped down (as the frame of the car with wheels applied to the tank). They can be shipped by truck but it is one of those huge things with crazy numbers of wheels that all steer. It is possible to backfeed the 115 by going 230/12kv then 12kv to 230 with another mobile. That's not much of a source though. Impedance would be high. They also have to deal with the delta highside of the step up transformer being w/o reference. This usually requires Vo relaying to protect the line from a downed or grounded conductor. A mess at 230kv but capacitive bushing taps is the way I have seen that done. Wire them in an ungrounded wye and connect the voltage relay between that wye and system ground/neutral. I have done that very thing with a 115/12 and 12/46 kv mobile unit to keep a radial 46 kv system energized during a planned outage. Certainly an easier feat when the components are large enough to manage the entire load. No way that was possible in this case. they maybe could have picked up a little load elsewhere but that line would have been w/o typical distance protection and most utilities will not hedge that bet due to liability. From my experience I was noodling all sorts of tricks and fixes but it is very hard to manage a forced outage on an isolated network. the utility had their hands full.
Moore County local here, so cool to see a video on this! 4 days without power was quite an unpleasant shock, but luckily it was not very cold that week. I had no idea we were so vulnerable.
Talk around town is that some locals who worked with Duke for a long time got laid off in a bad way and now took revenge…
I imagine that why they hit Carthage too, if they were familiar with backup plans.
That's my first theory rather than terrorism. I'm betting some local rednecks felt like Duke did them wrong somehow and this was their response. I would think investigators should be able to review lists of layoffs or contractors that were fired/let go and come up with some suspects.
During that time period, I've also heard it was in retaliation for a drag show that occured in the county and the bad actors wanted to disrupt the show. All we know is based on several cryptic messages from (legally) potential bad actors, hopefully we'll get some concrete investigations coming out soon
@@TrebleSketch Amazing that even in 2022 there is still useless hate and bigotry around
@@Birthold Grooming to be caring and respectful? Something you wouldn't understand.
@@Birthold Groomers? Like... hair-brushing? Please explain.
I moved to moore county just after this attack, our house signing was delayed because our realtor had no power. Almost everybody in our neighborhood has generators now and I had NO IDEA someone actually attacked the substation. So cool seeing my little town of wispering pines on a practical engineering video!
Reminds me of Y2k in port Stanley Ontario, it actually happened, but only because idiots damaged the substation at midnight
"Security through Obscurity":
I once interviewed at a place that managed the power network for the PA, DE, NJ tristate area. When I got to the address I was given I looked around, saw a brick wall around an unlabeled building on one corner, a strip mall style office park on the other, and a sign at the office park that listed the name of the place I was going to.
So I pulled in and started looking for the right door. Went all the way around the building and never found it.
Called the phone number I had to ask where it was that I was supposed to be going.
Across the street behind the brick wall that had literal drawbridges at the driveway entrances.
@Kevin Wittwer Depends on how paranoid you are.
Grady explaining how easy it is has me wondering about his situational awareness.
Did the wall come down and turn into a bridge?
It's all about executive compensation. When the executives want a second yacht or third home they do away with hardening these critical sites and put the money in their pockets.
People would be shocked if they knew how badly these things are protected.
Also, I think these were done for informational purposes. They wanted to see how the grid operators would react. There was another attack either yesterday or the day before taking out another substation.
Eventually, we will see coordinated attacks at multiple stations where the aim will be real damage.
I used to work for the utility supplying London, UK during the height of the IRA activities. An attack on the network was our biggest fear. As Grady points out these bits of kit are often bespoke and don't just sit around in the store. We had a number of contingences and the network is fairly robust, consisting of a number of loops in loops but with a bit of knowledge it would have been relatively easy to cause severe disruption that would have taken months to restore. There's only so much switching you can do before you reach the limitations of the network.
We role played such an event a number of times. Fortunately it never happened.
I remember the attempted attack at Warrington gasworks, that would have been bad.
sounds like it's time to stockpile backups...
@@thetruthserum2816 You'd think. Indeed, for the lower kVA, most utilities have plenty of kit laying around the place. This is often decommissioned equipment or redundant kit following an upgrade and so on, as well as new. Also the manufacturers have loads of it knocking about too. In any case they could probably build a new whatever in a couple of weeks.
The issue is with the really big stuff. Apart from it just being very expensive, and no business can afford to have that much capital tied up doing nothing but sit in a yard somewhere, it often needs to be made for that specific site, with a particular set of characteristics. That can take months.
I live in Florida where we get smashed with hurricanes like every year. It's amazing how quickly they repair the power grid. I don't know if we use those fancy bespoke transformers, but as soon as the power goes out from the wind, helicopters start delivering new equipment and money immediately. It's an impressive operation. Must be expensive. Florida profits big time from hurricanes.
@@alexbarnett8541 That's a really good point actually. I do wonder how Florida manages to seemingly "shrug" off hurricanes that happen as frequently as they do
Your jab at the end about it's vulnerability... *Chef's kiss*
Thank you, Grady, for this video! I live less than an hour from Carthage (and less than 20 miles from the Shearon Harris nuclear plant). I've been wondering exactly what happened in Moore County. Your video answered a lot of questions that I had.
I've just sent you an email.
When I was a radio tech in the late 90's, I got called out to a remote repeater site and discovered that someone had shot the air conditioning unit, which completely shut down the site. Just a small building with several radios in it but when the cooling system stops working, the radios must shut down or overheat. This one was mostly commercial systems but the same vulnerability exists in public service and emergency communications systems. Thanks for all your great videos!
Fortunately every state is on digitally trunked mesh systems. Much more resilient than the old analog systems.
@@borkborkfoxxo279 You're going to hear from a number of people about this, but I'll get in first and let you know that, if you like, you have a lot of educational reading ahead of you.
@@borkborkfoxxo279 digital doesn't meant it's less vulnerable... Just not at the same areas as analog tech.
The house of cards is the same even if the card materials are different; everything falls just the same.
@@WyvernYT No. Bad. Digitally trunked systems are linked in multiple ways, not just by the range of their transmitters. They are definitionally more resilient.
@@PrograError God forbid someone think of that and put multiple transmission routes between towers. That would be ridiculous, right?
I retired after 44 years with an investor owned utility. The control center probably got oil level alarms and dispatched someone, but then temperature alarms too. so they would have opened breakers and high side switches to de-energize and protect equipment. Nuisance alarms are commonplace. Similar alarms from adjacent stations might signal intruders which would lead to law enforcement being dispatched. In my opinion they did a good job since it didn’t look like the transformers had to replaced.
At my utility, our transformers are all covered by what we call bullet hole protection. It's a combination relay consisting of the outputs of low oil and gas detection relays. The idea is to protect against, well, bullet holes (and other acute leaks) by automatically de-energizing the transformer before the windings get exposed. You don't want to wait for temperature, or worse, differential protection to operate. Using the two relays together avoids nuisance trips. You can have low oil (we measure at the conservator) when the temperature drops or due to slow leaks, and you can have a bit of gas due to non-critical hotspots. But if you see both, it probably means the transformer is losing oil rapidly, and you don't want to wait for operator input.
I don't know if other utilities use this idea... but people shoot at our transformers all the time, so it's pretty useful here.
@@glenm99 That's really good use of logic on the relays, you see similar 2 of 3 channelized logic at a lot of power plants for the same reasons. Thanks for sharing, this stuff is fascinating.
@@glenm99 Do you think they also shoot out the protective relaying communications to the breaker?
@@Carl_in_AZ Not sure what you mean. Like, are hunters and drunk kids wandering by the substation and putting small caliber slugs through the concrete walls of the control building, into the relay panels? No. Are they aiming at the tiny control cables running from the trench up into the breaker cabinets? Also no. They're aiming for the big, fancy things that go "plink" and "ting" and sometimes spark when they score a hit.
In terms of what the protection shoots out.... The bullet hole protection typically triggers one of the transformer's protection schemes (which may also take as input the other non-electrical trips, and some simple overcurrent elements, stuff like that). What outputs that scheme asserts varies by station, according to the station configuration.
- If the transformer has high or low side breakers (or circuit switchers), it'll trip all of those.
- If there is no high side breaker, but there is communication with adjacent substations available, it'll send a signal asking remote stations to isolate the transmission line (and connected transformer) from their end.
- If there is no high side breaker, there may be a solenoid-operated ground. This closes to introduce a deliberate, hard fault onto the line, which in turn causes the protection at remote stations to operate and isolate the transmission line.
- There are some other things that might happen if there's connected generation, wider area transmission considerations, etc. It can get complicated.
- And there's usually some backup relays triggered, meant to de-energize a wider portion of the power system if a breaker fails to open for some reason. (We call this "breaker fail" protection.)
In any case, if the transformer has motor-operated disconnects, the protection scheme will wait a couple of seconds, check that the transformer is de-energized, then open the disconnects (and usually trigger a lockout relay). It's so the operator or an electrician can't accidentally try to re-energize a faulted transformer, and also it's helpful in figuring out which protection scheme operated.
If a bus was de-energized as a result of the protection operation, there may be an automatic transfer scheme that gets triggered, picking up the load from another transformer.
@@glenm99 Before I retired as an electrical power generation engineer I worked for AEP in substations in Michigan. I also designed power generation systems for a couple mfgs in switchgear and another company producing standby power generation for data centers. Everything you said is right on. A person who really knows what they are doing could shoot out the SCADA. You try to design around a single point of failure but there is always one way around the protection. Is difficult to design around sabotage. I was wondering if the person that knew how to isolate the grid communications and knew what to shoot out in what order. If they did then this could explain why authorities are not talking. A few years ago this happen in an Uptime Tier 4 Enterprise data center. The forensics shared during a 7x24 meeting found out that a disgruntled employee knew the order to take out the standby power's SCADA followed by a utility power outage during a storm in their triple redundancy system.
What I can not understand is why Duke's power sectionalizers didn't kick in.
I love in Moore County, have for most of my life. I got to experience the outage first hand(oh joy). I have to drive past this exact station everyday on my way to work. It's weird seeing it on a channel I've been subscribed to for years.
I think I may have drove past there while you were filming the shots around 4:04. I remember seeing a Sheriff's car out there visible from nearby 211 highway, and was wondering what they still had going on there weeks later. Haven't seen one out there since either.
Review the experience gained in Colombia and Peru. The was a multi-year coordinated effort to disrupt the power grid at local and national level in both countries.
Some hard lessons can be learned without having to re-live them in the US.
The video is light on details, and avoids pandering to panic and fear. 👍
Yeah and since all of the criminals from those countries are crossing into the US, this is very relevant now.
It also lessens the use of this video as an educational tool for copycats
@@jean-pierredeclemy7032 There was an actual engineering office in Bogota, paid by the FARC, to study the public data, do field surveys of the transmission lines, and "bid" on spare parts and equipment to gather intelligence and recommend where to strike.
I was amazed by how deep these actors can burrow into detailed technical matters.
Regards.
That mobile power station is pretty impressive.
It seems to have been able to act as a transformer for the area while the other ones were still under repair.
The people in that area would probably have had to wait much longer for power if that truck was not available. It likely also assisted in speeding up the repairs by providing power to the substation itself, making work easier.
I’d be friends with both you guys. Nerd out friends.
Thank god for diesel...
Most people have more than one.
This has happened about 6-7 years ago out here in CA. Those substations now have 15ft walls around them with sniper sensor towers every 100ft. If someone shoots at them it can pin-point where the shot came from with 3ft. It's the same tech the army uses in battlefields. They also have security vehicles roaming the hills surrounding the substation now 24/7. They shoot the transformers, then the oil leaks out of the transformers, causing them to overheat then they pop. It happened more than once, which is why they have the new walls and sensors installed now.
I was hypothesizing microphones stationed around substations to triangulate in on where gunshots came from about a month ago. I figured surely someone had thought of that ages ago. Good to hear I was right about that. The beautiful part is that those installations would be very inexpensive and have very low power requirements similar to an Arduino. Tiny as well, to where attackers would never feel confident they disabled all of them.
@@jeffreymorris1752 the power company probably doesn’t want to pay $ for it
What's the point of the sensors if the shooter just leave immediately after
@@ciskokid5936 if you couple it with some video, or direct contact with police forces, this may help catching the moron.
@@ciskokid5936 I can think of four benefits off the top of my head
1) Simple acoustic detector to backup the alarm system. It alerts you that there was an attack, so you can take action to mitigate damage before the transformer fails.
2) Aid in the investigation. If you know where the shot came from, you can look for bullet casings, footprints, etc
3) Make the attackers feel less safe. Make them worry, "what if the FBI found out we were planning this, and is waiting around the corner, and going to receive real-time notification of our exact location?"
4) Couple the acoustic sensors with an IR/LLTV camera to possibly catch footage of the perpetrators
You have to show us how you got that drone footage - it’s beautiful!
Thank you for the educational content and channel recommendations
The main feed station for the town I live in is about 15km north of us on the main highway into town; theoretically it could be struck by an errant semi and we would be without power for quite a while. The simplest security measure on it is simply a sign that says "no copper on site, all wiring is aluminum," addressing the main threat to the local grid.
Transformer windings are definitely copper but accessing them would prove to be more difficult than it would be worth
@@1992djg don't underestimate the desperate.
@@PrograError Or the greedy
Wait... Yes, aluminium is cheaper than copper, but not by orders of magnitude... Why would one steal the copper but not the aluminium?
And is it even technically possible to steal elements of such a substation while it is energized? I guess the risk is only limited to the moments when it's shut down for maintenance.
15 kV power lines, the railway catenary etc. sometimes get indeed stolen. But at such a low voltage it's not that difficult to get enough insulation to do it in a more or less safe way. But with 110 kV power lines and very thick, heavy wiring that's not even easy to move?
I maintain the grounds for my county’s electric supplier, 22 substations and the work center in total. One of the substations is located just behind a college. About once every few years, typically the start of the school year, the director of the power company, NEW head coach of the golf team and the president of the college will have to meet because the incoming freshman golf team members use the substation for target practice. No one ever thinks to tell the new coach to tell the new team members to not do that.
Doesn’t the outgoing coach think to do that?
According to the local newspaper most coaches don’t leave “happy” so I assume they don’t care and everyone else assumes that “not my job”. That’s just my personal opinion on why it continues to happen.
Must live in a fantasy world if you think telling college freshman not to do something won’t make them want to do it more. Just build a taller wall.
@@HailMary888 Hey, he’s got a side gig selling used balls on eBay 😀
@Under7Cs yes it takes me about an hour but my contract states that I can charge for extra “liter removal”. The only catch is I have to turn the golf balls into my boss at the power company. I don’t mind he’s a cool guy and makes sure I get paid well since he bills the college for “damages”.
I’m always surprised how close those substations are to roads. There’s a substation on a major highway about 10 miles from me.. nothing but a fence blocking it and a clear shot from a passing car.
Thank you. Good, basic, engineering report. Especially thank you for keeping technical and not editorializing. As many of us being challenged today, you too will have to decide on balance between information and security. Most of us will vote for information. But that’s because your viewers are good character practical engineering people. Thank you.
So glad you covered this.
We're in the affected area and getting solid news on what actually happened has been extremely difficult.
What do you think about the fact that you don't get to hear what happened?
Normally news/government channels would be all over this...
Don't you think it is suspiciously quiet over there?
@@blinking_dodo While it was happening I didn't have access to the news because even the 4G for the phones was down. After the power was back it had fallen off the news.
What news I did get was long on human drama and short on technical analysis.
@@mbvoelker8448 Wait, cell towers should have 48 hour backup power, right?
@@blinking_dodo Phone calls and text worked, but 4G didn't.
@@blinking_dodo everyone in the nation heard about this, what are you talking about?
Great video Brady! Just a comment on the wall shown at 9:50 between 2 transformers. This is not a ballistic wall, but a blast/fire wall that protects one transformer if the other has a catastrophic failure. In a nutshell degraded oil can loose its insulative properties which can cause internal arcing, leading to an over-pressure situation inside the transformer, and then it can go boom. The wall is designed to protect the second transformer.
Yes, thanks for pointing this out. Unfortunately, there aren't a lot of good images of ballistic barriers available to license from stock footage libraries.
@@PracticalEngineeringChannel CrashBoomBank is right that the image shows a fire wall, but I would thing that strategically placed firewalls in combination with better perimeter protection would go a long way toward preventing this kind of sabotage. The basic weakness is that chain-link fences have holes that anyone can poke a gun through, and if you can see the target, you can shoot it.
@@monophoto1 Yes, and i would think the radiators were targeted over the tank itself, simply because those tanks are constructed from very thick steel plates, simply to withstand the forces involved in a failure they have to safely contain, and also to survive for decades in use. Even the smaller transformers of around 100kVAhave substantial tanks, easily able to be classed as bullet proof, and the most fragile parts are the radiators and insulators, as pretty all other parts are made very heavy and strong, as they have to contain the expected pressure pulse coming from internal arcing, and not deform at all.
As to keeping spares, typical utility will have around 10 of the smaller ones on hand as units for swap out for service, and probably the same number of MV switches as well. Larger they might have one or two per district, just for unexpected loss due to weather, but the bigger units and switchgear is pretty much going to be one per utility spare, possibly even one only in the whole country, there to go for scheduled work.
By me power was lost due to vandalism to steal $5 worth of copper, and destroying the outgoing cables and the incoming MV switchgear. Took 4 days to restore as well, mostly because they had to bring in the spare parts and the crews from all over to do the work.
@@SeanBZA One of reasons for the thick sides of the transformer tank is also the weight of the oil as well as the gas pressure from an internal fault
wild seeing one of my favorite youtube channels doing a video on my hometown and where i live. i rent a spot in a factory opposite of the carthage substation that was hit and had cameras facing the substation and attack. got to have fun talking to all kinds of local and federal agents.
I was surprised there weren't concrete walls around the transformers, not primarily for their protection but for protecting the rest of the equipment from them if they fail.
Actual local here, there was a drag show (adults only) at the local theatre, there were protests and counter protests. most I've talked to suspect that it was an individual or small group who could not handle the implications of individual liberty and decided to commit a terror attack. If they get caught they'll probably spend their whole life in prison, as at least one person died as a result of the outage. Even if you oppose individual liberty and freedom, there's at least 50 countries you could move to that agree with you. Go there, don't attack Americans. The cost of freedom is bigger than just defending it from foreign threats, it's also domestic.
The county is very conservative but most conservatives understand that if you can force someone to dress a particular way, that the other side can also force you to dress a particular way. That individual liberty is the only thing that protects your values. Remove it at your own peril.
If they hate drag show so much why don't they move to islamic conservative country lol, those country also hate gay, trans, etc everything they hate and believe very well aligned with them hahahahh 🙄
Thank you for mentioning this. I’d heard of this in connection to the recent protests and attacks on drag shows around the US and was surprised almost no one (including Grady) brought this up as a potential motive. The coordination of the two attacks and the timing, as well as the existing undercurrent and talks of violence make it a pretty plausible motive for this. Some mainstream news coverage does mention this as well as online extremists talking about taking out infrastructure as a sort of pillaging, although honestly I don’t think the two potential motives are necessarily mutually exclusive. I hope the attackers are caught and brought to justice although it’s looking more and more unlikely.
@@Anya-Prime nobody mentions it because it's total nonsense. It's attention-grabbing people. These attacks had nothing to do with the drag show. People will go and stop groomers in person. It's absolute speculation and nobody cares.
@@Anya-Prime I kinda don't blame him for not bringing it up since it could be seen as encouraging it on the sly. The show went on uninterrupted as the lights had battery backup. Though the kind of person to shoot up a power station, is likely to have some inability to fully reason about cause and effect. I suspect they will in fact get caught as theres real financial incentive and they are domestic terrorists. I doubt you'll see it in the news though it makes it easier to bag em.
Read "Lights out" by Ted Koppel. He very completely explained this exact vulnerability years ago. The transformers used at key grid interties are even harder to replace (unique to the site, and very large) and not made in the USA any more. Disabling these could cause outages in months, not weeks.
There was that attack in cali like 20 years ago too
18 month lead time, made DHS come up with Transformer Spare program. Leak the Dioxin by bullet or hand , game over
@@MrSolLeks en.wikipedia.org/wiki/Metcalf_sniper_attack
@@billclancy4913 ah, 2013. Feels like 20 years ago. Thx for the link
based
I really like your videos. You;ve got a cool lilt to the way you speak. I enjoy the experiments that are set up in the garage. I have been modeling trains for a long time.
I don't really watch much on Nebula, but I still have the sub to support all of you guys. Thanks :)
7:11 That is the coolest looking truck. Like giant spark plugs
As a guy who knows people in Moore county, I'd like to commend Duke energy for finally getting something done on time. I lived there for a bit, and not once did Duke get the power fixed on time.
Probably cos they (town folks) are gonna freeze their arse off... And they'd be on the hook...
That on time fix if just a best case estimate. The computation will give a average restoration time for an outage based on recent outages in the area that’s out compared to what has happened in the past. The trouble man and or system operator that is evaluating will adjust the time based on trouble found. We try to estimate the time as close as posable but there are things you can’t anticipate. Then again that’s not my company.
@baronweber6801 they took like 5 hours to fix a squirrel chewing on a line. How does that take 5 hours
@@thezackast2752 well… it’s not a “brownout situation “
@@thezackast2752 crew has to come out, get the lines de-energized, install a splice, then re-energize the line. Have you ever touched a 15KV 600A line before?
As a industrial electrician with power systems experience. I would agree, the perp had similar knowledge. You nailed it.
i really like your videos grady, you have a good speaking voice and are easy to listen to.
This really demonstrated how vulnerable our power grid is.
To the sun.
It’s honestly not as vulnerable as people think it is. I’ve been in grid controls for a decade and the overwhelming majority of our issues could be solved simply by building more generation.
@@titleloanman Better still, massively decentralizing generation
If every residence is a producer you eliminate this risk almost entirely.
@@dorvinion better still a solar panel on every person's back. Dumb
Wait till you find out how vulnerable they are to electronic attack from thousands of miles away
I first learned about how vulnerable transformers at substations were a decade ago when talking about CMEs and how difficult they are to replace en masse. Crazy we’re in a situation now where direct attacks are actually taking them out
What if it was an inside job by security companies. They created a need to have security around transformers.
and the question is who is doing it?
@@trinydex not to be a smart arse, but has anyone considered Russian or Chinese supporters or even domestic terrorists and how the hell do you stop the Israel wire method?
Inside job??
@@stefanblumhoff2744 you can not. One would think those same amount of bullet's could cripple the whole state or south east like what happen when the whole north east went dark. If you notice it's awful close to a base that's Full of Yes Men, and women, that follow Every Order. We will see what that 75,000.00 brings out of the woodwork that we get to learn about.
This just reinforces the fact that modern society is held up by impossible infrastructure and preppers are not as crazy as everyone makes them put to be.
Great video, Grady!
Learn to live like your great grandparents did....
I've lived my entire life prepared for power outages and "grid failures," whatever that grid may be, but man, when people call me a prepper, I want to smack them. Nothing wrong with a big stack of split wood and a well stocked cold storage room.
Nah, preppers CAN be that positive feedback loop crazy and I don't find it irrational that a "prepper" could be the culprit. A good segment of prepper culture is pro-prepping effort at the cost of societal effort.
Prepper MAGA folks are probably the ones doing this.
Great video, as always!
I work at a large power plant near the confluence of 2 500kv transmission lines. The switchyard connecting them just got (starting last year) upgraded perimeter walls around it to obscure the equipment from view. They aren’t ballistic protection but they do hinder someone from viewing the specific equipment. Love the channel btw
Unless that person owns a drone lol
@@ST3ADYxKICKS thats illegal to fly by those stations.
@@pkwithmeplease so is shooting transformer. Don’t think they care too much champ
Sometimes concealment can be just as effective as cover. If you don't know _where_ to shoot, why expose yourself by doing so?
@@cryptoskid117, Willing to bet that’s the next security upgrade, some type of jammer that keeps drones out of the area.
Where I live in Europe, large grid transformers are mostly encased in concrete.
The main concerns here were vehicle strikes, falling equipment during storms and enemy infantry fire like machine gunfire, RPGs, mortars and shrapnel from artillery shelling.
It is also cheaper than burying them under ground, which was done for the first ones on the grid.
Maybe wrong here, but isnt Europe grid also much more interconnected? Giving extra redundancy. I think that is best way to prevent such incidents.
@@belisarian6429 Idk if more or less interconnected, but yes it is..
Yeah redundancy is good for protection against accidents and natural disasters and
I guess if one site is attacked one can alert security forces to respond to all others..
@@belisarian6429 USA is over two times bigger than EU. 4.3 mil sq/km versus 10.4. Your countries are tiny as hell.
@@lll9107 What is your argument? Surface area of USA is big therefore they cannot enclose transformers with concrete to prevent damage from bullets.
@Cancer McAids Yeah the grid is owned by the government, regions and municipalities over here in some areas local grids are privately owned..
Baltimore had a near miss. Just watched Beau's video on this type of attack and how to be prepared.
Great vid. There is a documentary about the vulnerabilities of the power grid, "Grid Down, Power Up." Very cool. Very free.
While this is a little bit scary, as an engineer myself, I love seeing the hard work put into protective devices as well as emergency plans(the substation truck) come together and save money and get the power back on faster. No doubt engineers will eventually figure out better ways to protect the substations all togethher in the future.
Just think of this:
You need only one crazy techie that buys some drones and "heavy fireworks" to drive around targeting substations.
I could probably darken the entire Netherlands myself, if i wanted.
(Not anymore though, since this message will get me on a list 😂)
How often do you eat beans?
Nearly everything needs to be underground in rebar-reinforced and faraday-shielded concrete vaults. Put 500+ kilovolt transmission lines underground, suspended in the center of bored tunnels that are 15-25 feet in diameter. The grid would be protected from terrorism, solar storms, snow and ice, hurricanes and tornadoes, bird guano on insulators, flooding...
@@DMahalko Dammit that is a great idea.
Usually straightforward to eyeball the voltages given insulator sizes and spacing between conductors. Here in Arizona there’s a solid high voltage interconnect around PHX, but outlying areas aren’t so fortunate.
As you state, “security through obscurity” is typical in many engineering disciplines, primarily due to cost and the customer claiming “oh, that can’t happen”
That's common in the old telephone business. The switching and wire centers would take months to repair or replace if seriously damaged. But there were in nondescript buildings which had no indication aside from a very small sign that they were part of the bell system. Moreover, the cables and came in underground despite the fact that most of the distribution was on poles.
Reminds me of the mobile switch we prepared to roll out when building 7 went down. It had been sitting so long unused we had to fire it up at the office first to ensure it was still operable.
Thanks for the video.
I work in the industry and finding bullet holes in substation gear is on the rise in CA as well, very frustrating.
Because most of them are in the forest, I thought sub stations were shot at all the time by hunters who miss their targets and by drunks.
@@MrNeptunebob true lol but not just that. DHS has sent memos warning domestic terrorists “continue to plot and encourage physical attacks against electrical infrastructure.”
Excellent - very well done. My experience as a power system planner is that major transmission infrastructure is designed to account for at least N-1 failures, but the same is not true for local area distribution systems like Moore County, and that's got to change. This was a highly publicized event, but it was not the first - there was a similar shooting incident at a 500kV PG&E substation near San Jose a few years ago that got less attention because the local transmission grid was designed to withstand a single-point failure. But in addition to the lesson regarding system planning, it also shows the need for both physical protection of critical infrastructure (a chain-link fence is, well, porous), and better spares planning. Mobile transformers are great in theory, but in reality, limitations on road weight and size prevent them from having the capacity to be very effective. Ultimately, this should lead utilities to move away from customization to use of more modular designs so that a small number of spares can backfill a larger number of applications.
There has always been a few random incidents of frustrated hunters shooting insulators during deer season, but the nature of the threat is changing, and the consequences are potentially far more severe.
I think part of this is also that in general N-1 redundancy doesn't always cover the same type of failures. The Moore county electrical grid probably was "N-1 redundant" if you would have asked someone in Duke energy, they almost certainly could have opened up breakers and switches to route around any reasonable failure within the main targeted substation. Geographic redundancy isn't always considered.
N-1 wouldn’t have made a difference here. N-1-1 would have been the better design, but assuming the attacker knew what they were doing, they would have likely shot at the third transformer. They needed another 115kv feed to that loop to alleviate some of this but who knows if the load could have been supported by the single line.
Also mobile transformers have less ground pressure than typical semi-tractor trailers on roads.
Duke Energy likely has spares but due to these being 230kV, they are likely stored with oil, so you need to de-process the transformer, ship it, and reprocess which could take upwards of a month. Plus you can see in the arial clips they are remediating the soil since it’s contaminated with the transformer oil. I would predict that the one that’s out of service will be getting replaced with a new unit.
Interesting comment, i‘m planning transmission networks for germany. TSOs must always calcilate by n-1. Also we have to secure the so called n-1* criteria, which means if we shut down one piece of eqipment and then an error occurs we still have to maintain an operational grid without an outage. On top of that, we have to secure our transmission system against common-mode or common-cause failures. So a complete destruction of a substation, or destroying towers don‘t cause any outage.
For the distribution network we just maintain n-1 and most of the time n-1*. That‘s how we achive a mean outage time of ~10min/ year for every customer.
I remember that one well. That was truly a competent effort by a motivated group. They preposition rock piles to find their shooting locations in the dark, cut the comm lines and then took out the transformers.
@@gigaWUTT Agree fully - the basic design of a 115kV look with power entering at only one point was flawed. Before there were terrorists there were tornadoes, and a single tornado would take down that design.
When we were kids we used to go to the sub station and swing a big chain in huge explanation was awesome 👌
In 2013 the Metcalf substation outside San Jose was attacked by a sniper targetting the transformers. Before the attack they cut some AT&T fiber trying to disrupt substation communications. The perpetrators were never caught. The substation is now surrounded by a concrete wall and a lot of perimeter security cameras and sensors.
Our towns main sub station was taken down for almost a day by a squirrel causing an arc
We need to ban squirrels
I'm surprised Grady didn't talk about the wee little mungers. Squirrels cause more damage to the electrical grid every year than all hurricanes in the last 30 years combined. Squirrels are terrorists! :D
@mbox314 high capacity assault squirrels are a threat to our society!
It's funny, at a site I used to work which happened to have a tokamak living there the power safety video included a squirrel hopping along a power line, a day or two after they switched videos a squirrel became an unintentional conductor and tripped the main transformer
Must be those politically extreme squirrels.
That portable substation is incredible.
A similar incident happened in Pocahontas Couny, WV on Thankgiving Day in the 1980s. The transformer got shot by an over eager hunter on opening day of deer season.
Hellos from moree county and that whole power outage was terrible especially since classes were online at college
"Security through obscurity" is very dangerous to rely on.
anonymity is the best defence
Obfuscation
Seems like it is still better value for money than security through big walls.
@@SocialDownclimber That is, until the secret gets out, and in this case, the secrets got out
True, there is ZERO security in obscurity or obfuscation. It's common for people to intuitively believe it can work (look at the comments here), but it is a falsehood.
Another interesting part to add to this sabotage. Just a few days after this attack, hand written bomb threats were sent to many power plants in neighboring states. We treated the letter we got at our plant in KY very cautiously knowing that just a few days prior this event had happened in North Carolina.
"hand written" Really?
@@quintrankid8045 No one ever said wanna be terrorists were smart enough to not write notes in their own hand writing.
@@quintrankid8045 Probably written on the back of a dollar general receipt knowing NC
@@quintrankid8045 what else would you write it with? a printer can easily be traced
Did you notice anything about the language use that struck you as non-native English speakers?
I've recently heard that apparently the CIA is responsible (through local partners) for a lot of those industrial fires that are happening in Russia.
This attack, and letters like that, seem like the exact same sleeper-cell counter-moves that Russia might make as a "back off, for two can play this game" kinda message....
YES!,,..you make several very good vital points in this video. especially the one about how the electrical stations works and how are somewhat vunarble to attacks such as those types and how they are restored in an such event that occurred. or how hard it is for restoration of device. I can appreciate why the electrical company in that state do not revealed most of it its findings as to not let these bad actors know as to do to it again.
I like his videos because they are full of facts, no fluff or flair
In general, protections are not designed to protect equipment from failing - they are designed to protect the rest of the power system from failed equipment.
Similar to ones in your home really, the breaker or fuse isnt there to stop a device from releasing the magic smoke. Its there to stop the home wiring from releasing its magic smoke when the device goes into a dead short on failure.
Not true, it’s literally called transformer protection 😂 Look up SEL-487E relays
@@gigaWUTT I spent 15 years in protection design, including participating in IEEE standards development. I've written papers published at TAMU, Georgia Tech, and WPRC relay conferences.
Protections only operate after the associated equipment has developed a fault - i.e. it has failed.
@@StevenHodderIs this the part where we start comparing degrees, PE certifications, and W2’s? All I was saying your over generalization is false. A true academic would acknowledge that.
Just because a relay trips when it sees a “fault” doesn’t automatically mean it’s failed. It only means that there was a data point for a set amount of time that was outside the constraints, which are typically set to not damage equipment beyond repair.
@@StevenHodder that is not true anymore. Many electrical protection systems are designed for a whole bunch of issues, including tripping out on transformer temperature. Only some relay functions are due to a fault, and modern substations monitor all sorts of parameters. I'm an active PE in multiple states, to say we only protect after failure is completely false.
Thank you for your book. It is wonderful. I evangelize about it to all my friends. Now onto the meat. The trailer mounted portable substation by EFACEC of Portugal is a marvel unlike no other. Much like a massive carnival ride, what it took to make such a heavy and complex piece of equipment travel over the road is nothing short of Burj Khalifa level engineering. Stop frame at 7:11 and look at that. Besides the chassis, the weight distribution, the height and width restrictions, and leveling jacks and everything that had to be done to control arcing, this is truly a complete package that can apparently respond to a crisis at a moments notice anywhere it is needed. ON A FREAKING TRAILER! Does Duke own something like this? Or is there a third party, "high voltage SWAT team" that keeps this gear on standby ready to deploy at a moment's notice? (Obviously very useful in tornado/hurricane country.) I do realize that you specialize in civil engineering, but you gotta be wowed by such a magnificent all-in-one, fast deploying, (voltage adaptable?) piece of gear that is SO specialized you could live your entire life and NEVER see something as cool as this. That's seven axles on the ground BEFORE it gets to the hauler's fifth wheel. I did intentionally give props to the traveling carnival ride industry because if you ever want to see some seriously over-the-top mechanical engineering, watch a three trailer giant ferris wheel get set up. ruclips.net/video/a1g_XB1mb5k/видео.html Thanks for everything you do. People should be much more aware of the incredible things that make our lives as we live them possible.
The mobile sub is most likely Duke's. Most IOU's own a fleet of these, usually made by either US-based Delta Star or EFACEC (as you mentioned). Mobile subs are designed to be deployed quickly so they must comply with state transport regulations meaning one pound over and they'd require a special permit (which would defeat the purpose). They are limited in their output, around 80 MVA at 230kV and 65C temperature rise because of the weight restrictions. Some IOUs deploy mobile transformers, more for maintenance purposes, which can be rated higher as the rig is devoted to just the transformer and not the other equipment.
Not to downplay the amazingness of the engineering, but that articulated trailer is a standardized design that you can see hauling all kinds of super-heavy equipment.
Using it for the transformer like that is genius to be sure, especially mounting the cooling and control equipment on the front and rear sections! But while the _utilization_ of that trailer type is impressive engineering, I just wanted to be clear that they didn't _design_ the trailer, get it road-certified, etc. It's an off-the-shelf trailer just _used_ extremely cleverly.
most substations here in the UK have a brick wall around them. the small one at the end of my street is in a shed about the size of a single car garage.
Love your content!
I used to live in Moore County. Grew up there until college. It's scary when I heard it, and I know people who were affected, but its a little strange to see one of my favorite RUclips channels talking about my home. Keep up the good work, Grady!
Had this same feeling when he talked about the 2003 eastern US power outage and talking about the Cleveland grid where it all started :3 Like, hey that's us! We caused a whole region of the country to go out of power! :D
This page is great
Uh, yeah it's not that strange. It's on the earth. Something happened there of importance. Ergo, youtubers talk about it. It can happen to any place on the earth
@@gxlorp you must be really fun at parties
Same. I have a lot of family in the area.
I work for a transformer manufacturer and distributor and I sent this to all my co workers. Very well made!
I had an issue at a large remote storage lot. One Hikvision auto tracking dome camera got all we needed to catch and prosecute bad actors. That one camera covered 250 yards and zommed and followed anything that moved within that 250 yard square.
Great video ! You just outlined how to disrupt any substation !
Obviously they already know how to disrupt substations, and this knowledge has been known for decades. My uncle used to rant about this potential and he passed away in 1975.
I worked in a substation in a different country for more than a year before moving to US. Substations are constantly targeted by thieves looking to steal copper snd other metals. They are very knowledgeable about the inner workings of the grid and know how to trip lines so that they can go in and cut out wires. I am pretty sure it's the same in US also.
Also the design of the grid can be improved to avoid single point entry failure. Local loops should be connected to the grid in multiple ways so that even if 1 station goes out, others should keep supplying.
Correct. Also the audacity of these copper thieves to risk their lives around high voltage equipment just to collect a few hundred dollars of scrap metal is utterly terrifying to me.
@@sahanavica.5574 pls search for the phrase "substation theft" in RUclips.
@@AndogaSpock Oh no, I'm well aware, I use to work adjacent to this field too.
No. The metal thieves don’t go after in-use conductor much in the US. They will steal every piece of copper you leave out overnight at a job site, though.
@@dzerkle The ones near me seem partial to highway lighting wiring though. There’s always sections of roads unlit at night due to the thieves.
Happened in Puyallup Washington to. Around Christmas morning.
Back during a tornado outbreak there was a substation that had a transformer fail. It sent power to a large power plant to run their plant. Then that plant tripped out. There were several counties in the dark. The transformer had to be replaced and that had to be built. They do not have spares. It was a week before they put a temporary sub in place. Then several more days till the power plant was back on line. I went by that substation three months or so later and the temp was still in place.
The grid is rock solid till it suffers a catastrophic failure. Then it seems more like a fragile egg shell. Scary stuff.
While you are focused on daily comforts that electricity provides(and making this video gentle), there are many who are dependent on medical equipment that needs dedicated electrical supply. Batteries are there for emergencies, to get to a power supply. I don't think you forgot, but it is important for me to highlight. Thanks for sharing your expertise!
Another fantastic video, Grady. Well done and bonus points for getting drone footage.
In the 10 years since the attack on Metcalf and Keo substations (of which I was part of the substation security task force back then), the challenge will be how to improve hardening, robustness, and response to smaller substations. While this attack didn't affect the bulk electric system, it still mattered to the citizens of Moore County.
The Metcalf sniper attack seemed to be to be a "proof of capability" of a criminal organization. They specifically targeted equipment that was shut down, but proved that they could do massive damage. Had they shut down a city, they would have had a hundred times the press coverage. This was to-the-point and relatively quiet.
A guy a few comments above said stand multiple steel plates around the transformers. Easy cheap and you can put them up pretty quick and it wont cost anywhere near what a huge cement wall would. It might not be perfect but would make it a lot tougher to damage a transformer with a bullet.
Being someone who isn't from the US, it seems like either people or things getting shot are a pretty consistent pattern.
It really just doesn't happen anywhere else.
Maybe it doesn’t happen where you live, it also is extremely rare occurrence here in the US.
Except the rates of gun violence in Sweden, New Zealand and Australia are rising quickly and Sweden has also had a lot of grenade attacks over the past several years. It wont be long until is evens out across the globe.
I live in Portland, Oregon and the substation near me has had people on site 24 hours a day since early December.
Excellent work once again Grady. This really brings to light many of the problems these “bad actors” are causing. I have a feeling most of these attacks are just practice for a more coordinated attack. I don’t think we have seen the last of this.
Auto Transformers are a very specialized piece of equipment and these people know that. Unprotected equipment, overloaded equipment, and supply chain issues are creating a trifecta that will have severe consequences in the future. I urge people especially if you live near or pass by a substation regularly to keep an eye out. If you see something that doesn’t look right or normal, say something. We don’t usually know things have happened until after the person has left.
Love,
Your local transmission operator.
Really well said, thank you! I jogged by a 500kV SS a few years ago and noticed the gate open, no bucket truck around so I called the EMC. This was in rural Georgia.
Where I live, we don't have much motivation to turn off the power on the state.
However, I disagree with these individuals only in their sense of timing. Well... and lack of proper demolition tactics... but not everyone can develop those skills.
Perhaps our government should stop funding domestic terrorists to further its own political agenda before those of us who toppled nations for them decide the evil empire has to fall at any cost.
It's amazing how little it takes to mess something up and have major consequences.
Internet is much more fragile. We had a transformer fire on a light pole in our little town. It took out the internet for the entire end of our state and into the bordering states around near us.
yeah the perps could never even come close to paying back the damage. If they had any morals, they would self punish with the only reasonable outcome.
Another problem is there's somewhat of a shortage on distribution transformers as well. It's been a struggle just keeping normal stuff in the warehouse stocked.
One good storm and my local utility is gonna be screwed. This is a widespread problem.
I was only 9 years old but I remember the blackout of 2003 in Southeast Michigan. The power rapidly went on and off several times while I was watching cartoons on tv. We were without power for several days.
I worked at a utility truck company for a few years. Some of the fiberglass serial platforms have ballistic plating between the plastic liner and the platform.
Where were they sold? Afghanistan?
@@richardcranium3579 Illinois. East St. Louis.
Thanks for covering this Brady. Your work to educate the public on infrastructure is valuable and appreciated.
Substations in VA now have shot locators installed around the perimeter of the station.
The best way to add security would be to shift away from a tree like structure of the power grid, to an interconected type. Like connecting the 115 kV loop to next 115 kV loop, so one can supply the other if one gets diconnected from the 230 kV lines.
re: "The best way to add security would be to shift away from a tree like structure of the power grid, to an interconected type."
Um, did you notice the topology in the ring around Moore County? What do you suppose the purpose of that 'ring' was?
To be honest when I was a kid I asked my dad what electrical substations are and in my strange child hood mind I had the thought, "What if someone targeted all the substations across the country?" I may or may not have been reading my dad's Tom Clancy novels at the time. If a coordinated attack on the nation's electrical substations was carried out it could cripple the country. This isn't at simple thing to coordinate in secret, but it's scary how vulnerable the US's power grid actually is.
that's why most utilities have rings (redundant) lines.
@@KENFEDOR22 Ken, I just wish we could have this information, without helping those that would hurt us. Semper Fi
One more reason to get rooftop solar with a power pack and be independent.
agree I think about this situation all the time
@@TechNextLetsGo my camper van would sustain me
Can you do a video on why what seems like the majority of manhole covers are placed in the tread of roads. Asking for my balljoints.
And why two inches below the rest of the roadway?
@@Brian-L Because the lowest bidder gets the job even if they are worthless.
And why don't ours have nice floral designs on them like the ones in Japan?
@@Brian-L I believe that's the result of repaving.
@@valcaron , because people in the US would steal them, we can't be trusted with nice things.
We learn about these transformers and power distribution centers from your videos!
Media's been silent on this, great video!
The video is 6 min old and 13 min long you couldnt have possible watched the whole video
almost like theres no confirmed information. ik you americans love conspiracies so just blame edp445 on this one
@feuerhai557 i watched it at 2x speed. Gotta get on my level bruh
Yes, to prevent inspiring copycats.
@@TriNguyen-he7xk media should have at least still said something about it, then gave an update on the situation when more news come out.