🏭 Watch more of my power grid videos here: ruclips.net/video/v1BMWczn7JM/видео.html 📫 Big announcement coming in March! Stay up to date here: practical.engineering/email-list
I was living in downtown Ottawa, Ontario, Canada. I remember the blackout vividly. Not sure where you got your info, but my power was out for 3 days. My house in lowertown was literally one of the last to get power back. My roommates and I all took shrooms, wandered downtown in the pitch dark, then almost burned our house down trying to cook a full pack of bacon on our little camping bbq. Good times.
The power plant in the very first frame, next to the United Nations building in Manhattan, that's where I was... in the control room as a very young Engineer. I was actually giving a tour of our control room and flipping through some system graphics, when all 3 units tripped offline at nearly the same time, and the plant went completely dark, which before then was thought to be impossible. One of the operators actually yelled at me because he thought it was my fault, lol... I was relieved when the system operator called a few minutes later, and told us to, "Standby, we just lost the entire Northeast." We overcame a lot of challenges that evening, and for a young Engineer, it was exciting. I slept on the roof that night because it was too hot inside. I remember looking up at the night sky and seeing the stars, and thinking... when was the last time anyone in Manhattan looked up at the night sky and saw the stars like this? Great job on the video, and thanks for bringing back such great memories.
This happened when I was 4 and I thought I caused it by watching too much TV. I've lived with the guilt for many years, thank you for explaining what really happened and for clearing my conscience.
When you learn about the complexity of managing the power grid, it feels like a miracle that we consumers are actually allowed to do whatever we want and use as much power in whatever way we want in our homes. Something to be thankful for, since all the heavy lifting and problem solving is being done by someone else on our behalf.
On the opposite side of the world there's also USSR power grid, which is kind of unique in that it covered 12 timezones. Would be interesting to hear more about it. Also there is recent outage in Middle East (from what i know conclusion was reached and it was caused by short circuit in one of coal plants and lack of water in hydroaccumulating power plants because of dry year), and there might be another one coming up with Ukraine disconnecting from Russia and Belarus. Power usage is mostly quite stable, there's morning peak and evening peak and other than that it's on similar levels. Also it's mostly related to "lifestyle" of whole community, and in lesser degree to how much an individual uses. As even if one person consumes 10kw more, it will not be noticeable on a scale of 1000 people, other than maybe causing some questions to that person. Bigger problem for power grids is solar generation, as it's very unstable (it depends on weather after all) and it doesn't really help with two peaks that often. Also as in many places there are laws giving preference to local "green" generation, so people can "give back to the grid", which, when happening on a large scale, is lowering midday demands, but only sometimes. So companies running power plants have to accommodate these power spikes somehow, which means generators can't always work in optimal mode, reducing their lifespan. Overall I think that there's a better solution to climate change than current solar/wind craze.
Not entirely true. The amount of current we can use in our homes is limited by breakers. But yea, there's a lot of freedom for consumers. So I agree with the gist :)
I was working in midtown near Times Square when the black out occurred. I lived in NJ at the time and found myself stranded. I had only $4 in my wallet and spent that buying a couple bottles of water as it was indeed a warm day. I heard that there were buses leaving from the Port Authority so I walked over there only to find a massive crush of humanity there. Had to be well over 100,000 people around there trying to get out. Then I heard the ferries were running across the Hudson and walked to them only to find many more thousands of people trying to get on a ferry. I gave up, and walked back to the office building where I worked and eventually got a ride late that night in a car service sedan with someone seated literally in my lap. We drove alone through the pitch black Lincoln tunnel. So eerie. The lesson learned: I always carry $300 in emergency cash ever since. When the electricity goes out, cash rules all, as I found out the hard way.
Damn...I just remembered how amazing that outage was. A bunch of us kids in the neighborhood had sleepovers, people were just going to each other's houses. We were playing in the streets. There was such a strange and unique coming together of humans where I was. As soon as the power came back, people started to go on about their usual business and that special moment was gone
I grew up in a third world country. Blackouts like this are the norm, it will be just another regular week. It’s crazy how the 1st stands still if power goes out just for a few hours
The Lift Pump Station located near Ludington, Michigan helped save the entire eastern half of the US and Canadian power grids. This facility maintained grid frequency by absorbing most of the nearly 5 GW load imbalance as the disconnects around Lake Erie engaged. The pump station filled itself to the brim, well past its safe design water surface elevation, then ran itself nearly dry, again outside of safe operating levels and at great risk of causing cavitation in the turbines to provide power afterwards. These actions helped prevent catastrophic damage cascading all the way to Denver and gave some time for other stations to reconnect. If a power station is scrammed, a rapid shutdown, and depending on its type it can take days if not weeks to get a power station back up and running. A pump station can provide nearly immediate power generation. Thank you to all of the station operators, line workers and service crews who struggled on that day, your actions and quick thinking, kept the lights on east of the Rockies and prevented a far darker outcome. Both the 2003 and the Texas outages could have been much worse if the load imbalances had been allowed to progress a few minutes or even seconds longer. The control systems in place now are night and day more robust than what was present in 2003.
This sounds like an amazing story. I did some quick googling, but I didn't find any additional information. Would you mind sharing any articles about the Ludington Pump Station operators?
I was working at a 2000 MW power plant that day. I recall hearing air blast breakers operating frantically in the switchyard just after 4 pm. Lights blinked and then moments later the plant was black. From my office, I looked across the river and saw two other large power plants also fully tripped. Our plant generators were spinning, but there was no grid to connect to. We hoped to keep the units ready to resynch, but we could not keep up with demin water consumption, and one by one, lost the plant. The grid was partially restored by the next day when we were able to fire our units and resynch, thus rebuilding the grid. Certainly a day etched in my mind!
What kind of plant was that? In the hydro world we just keep one or two groups spinning and exited but asynchronous to maintain our auxiliary transformers indefinitely. What consumes demineralized water and why can't it run on local power?
@@gregorymalchuk272 We built the grid backwards from our hydro electric plant at Niagara Falls (their station service requirements are virtually zero) and when we had power at our station we were able to restart our units normally via the reserve station service transformer. We could aid have used standby gas turbine generators to do the same.
I was a kid in Brooklyn during this blackout. It was one of the most memorable nights of my life because we got a full night sky due to no light pollution. You usually never see more than 2-3 stars at night in my neighborhood.
Humans are moving towards a certain direction and there are no brakes on this ride. The days of genuine, distraction free human companionship might be long gone.
New Rochelle over here I was 9 at the time, and I can confirm, saw my first shooting star with my brother. We didn't have power for almost a week it was one of my favorite memories ngl.
My brother and a friend were in Manhattan that night and said it was the most bizzare thing to see a full night sky from the streets of Manhattan. People were generally good too, handing out beers and cooking food that they didn't want to go to waste.
I'm not gonna lie, as someone who was in NYC during this, Manhattan specifically, it was actually really fun. Gas service was still going so restaurants were cooking and effectively giving away all the food they had built up, everyone was out on the street ( who wasn't trudging across a bridge) was drinking. Driving...yes driving...was actually really easy because traffic was self moderating. It was a very surreal experience, all told. Spent the day after that first night out in Brooklyn by the water drinking with friends and then hung out at my friend's place on the west side and watched them switch the super blocks back on one by one. Looking up 8th avenue, it was kind of amazing. From the horizon you could see entire sections of Manhattan switch on, I wanna say like 10 block squares at a time coming towards where we were down in the 30's.
Down in the Philly area, we watched the entire hot mess on the news. With our jaws on our chests. We're a 21st century society with a 1950's electrical grid. :/
When first learning grid operations it was explained to me thus: Imagine the ceiling in a room has a wide variety of hooks embedded. Each hook can only support a different weight and will pull out if overstressed. Now imagine you have a wide variety of weights that you must suspend from these hooks. You attach them with rubber bands of varying strengths. You link a bunch of hooks and weights with bands. But if one hook pulls out, the weights stretch rubber bands and shift and you need to keep things suspended. Now, plan things out so you never drop a weight, regardless if a hook pulls out, or a band breaks.
Whoops, you used too many rubber bands and nobody wants to pay for that. Try again but with less rubber bands. I don't care how safe it is as long as none of the weights ever fall. (Yes, that last sentence is supposed to be contradictory.)
I love this explanation! It puts it in complex and simple to understand way to think about the grid, it's various load capacity factors, and the various loads on it, that nobody can really control. SCE in California can shut off all the air conditioner compressors for a huge number of customers by sending out a radio signal to a box on each A/C unit. They could do this before 1986, when I found out about the system. This can shed 100's of MW of power load from the grid instantly.
I remember this quite well. I was working in Solon, OH at the time, and the power tripped off at around 4:00. Some phone calls from our facilities manager confirmed it wasn't just us, but pretty much everyone in NE Ohio. They sent us home since we couldn't do much except twiddle our thumbs and it was clear it wasn't coming back on anytime soon. Got home, pulled the rope to open my garage door by hand, turned on the camp lantern for light when it got dark, and kept listening to the battery powered radio for news. Lit my gas stove with a lighter and made dinner, and read a book by candlelight. Quite peaceful, actually. When I heard that water pressure was going down I filled my bathtub so I had some water. Power came back on around 10 PM. Next day, I found out this was huge problem for lots of folks. I didn't have a generator back then, but it was certainly eye opening.
that summer was so hot, we live in a 12 floor building in Manhattan. everyone that night went to the roof to sleep, and everyone on the rooftop brought their flashlight and start playing chess and other board games. it was pretty fun for kids like us. you get to see every neighbors and kids.
I loved this day. I lived beside a valley with a river. After so many hours, and seeing that everything was down, i set up a campsite beside the river. Will never forget how bright the stars were. I was actually sad when the city glow returned.
The power went out above my head as I was boarding a plane in the Toronto airport. Ended up sitting on the tarmac for two hours, not knowing what the cause of the delay was. We were shuffled off the plane and I, a thirteen year old about to embark on their first solo journey, was left alone in an airport. I was too young for a cell phone at that time and the pay phones weren't always working. Managed to get a hold of my parents who brought me home. I can remember the sweltering heat. My elderly dog ended up passing away that night because of this heat. No vets were open because of the blackout. We didn't get power back in my house until 7am the next morning. It was...definitely a bad day for me.
I'll always remember that day fondly. I was the admin of our companies servers and all of the battery backups were going nuts! First the brownouts, then blackouts, then when power stayed on within that hour all of the battery backups stayed on battery because of an over-voltage (it stayed over 130 volts for almost an hour!). Confusion set in since I had power, but they refused to switch back to online power and the batteries would not recharge. On a whim I turned on the fans on all three AC units, along with all the lights since everyone had gone home by then. That managed to get the power draw down below 130 so the battery backups went off battery and back on-line. From that point all I had to do was monitor the voltages until they stabilized below 130 volts and I could shut off the lights and fans on the AC units. Thanks to your video I now know what was causing the over-voltage issue, and now most newer UPS are built so they won't go completely onto battery when over-voltage issues occur.
That’s a cool story! I have worked in a datacenter however thankfully haven’t experienced anything like this. Good job using the AC to drop the line voltage - it doesn’t take much current flow to drop 5-10v from a building supply.
I remember this. Me and my friends were swimming when the pool tank shut off... we went home and realized we had lost power. My house didn't have any power for 4 days. We lost everything in the fridge and had to use flashlights to go about the house. It was dark everywhere and we couldn't go anywhere. It was a crazy few days... I was 11 going on 12 when it happened. I'm now 31. Amazing how much time has passed since then. Thanks for this
@@napalmstickylikeglue I was 9 also in the Detroit area. My brother and I were watching TV and I thought he was turning the TV on and off like 4 or 5 times before going completely dark.
This is fascinating to me. That spring, I had began my public safety career as a police officer at the Cedar Point Police Department in Sandusky, OH. For those unfamiliar, Cedar Point is a MASSIVE amusement park about an hour east of Toledo right on Lake Erie. It was an incredibly busy day that day, right in the middle of the peak season, and the park was nearly at capacity. When the outage hit, we obviously had to shut everything down and close the park. You want to talk about insanity...try getting 50k upset people out of a 350 acre area located on a peninsula, with only one causeway going in and out. I got a lot of opportunities to work on my people skills that day, haha. I never knew the details of how that blackout happened until now. Thanks Grady!
Holy shit. I live in Cleveland so I know exactly what Cedar Point is......and I never even considered the people there on a hot summer day, when suddenly all power completely shuts down. I guess I always assumed they had backup generators for safety. Maybe not to run the park, but enough that people don't get stuck hundreds of feet in the air on something like a ferris wheel that isn't coming back down. Or imagine being on a roller coaster, going up, where it's slowly inching it's way up.......and then it just stops. Luckily the outage happened right around 4pm, so it was still light outside. Imagine if the outage had happened at 10pm.....
@@fitter70 I can't speak for their current setup as this was nearly 20 years ago, but back then there were generators that provided backup power for the hotels and emergency services facilities/comms, and just enough juice to operate the ride control systems and bring everything safely to a halt. We couldn't keep them running normally on backup power though. Most of the rides need an enormous amount of power to run properly, far more than a generator could provide. For example, the Top Thrill Dragster had just opened up that year, and that ride alone took an absolutely INSANE amount of electrical power to generate the hydraulic pressure needed to launch the trains. An impressive sight, honestly...but it absolutely required main-line high voltage to work. There certainly are generators big enough to put out that kind of power, but the size and cost just weren't worth the investment considering a grid failure like this one wasn't even thought to be possible before that incident. Much cheaper and safer to just shut it down until power was restored. You're correct though, it was a mess. Luckily the park engineering team is very good at their jobs and they had everyone down off of the rides within an hour or so. Most of the rides (especially the coasters) used pneumatic brakes, so there was plenty of air stored up in the tanks. The operators allowed the trains to coast in, and then manually stopped them to get everyone off safely without putting them in danger. All things considered, it was a pretty successful operation.
I was at the park that day. A park police officer told me it was a terrorist attack in New York. I’ll never forget the chaos of leaving Sandusky for home.
My friends in Toronto recounted stories of how traffic during rush-hour became total chaos. There were several people standing in the middle of intersections marshalling traffic for hours on end. I've always wondered who relieved these samaritans when they became tired and needed to go home.
I encountered one of these Samaritan’s at the intersection of Bayview & Steeles - 2 lanes + left turn lane in each direction. This guy demonstrated the most incredible skill in effectively directing the traffic. Although cars stretched forever behind the intersection (not much different from a normal Toronto rush hour), I think we were all grateful for this guy’s assistance. Everyone followed his direction with no one trying to cheat.
Please make a video about the process of bringing back power after a blackout. How do you bring power plants online again without them tripping again immediately? How are areas sectioned off to ensure load is consistent with supply as it is being brought online again? How are different "islands" reconnected between them ensuring frequency and phase sincronization after a blackout?
I watched the system at a switch station here. In a event of a feeder overload the system dumped all the circuits leaving the switch station. When they had three of the feeders back online they just powered on each outgoing circuit one by one with a delay of up to 10 minutes. For The backbone grid they would also have the other issue of matching generation with demand.
You do in stages creating islands of generation and consumption. Then slow sync and join in more bits of the grid back together. Old school method of sync is to use. Light bulb for connecting each phase and then slow/speed up the generator till the light bulb goes out(no voltage difference means no difference between the phase) Modern day uses GPS timing signals that aliign the phases.
There is a very interesting article I read where the DoE setup an isolated grid and had to restart it from scratch without using any Diesel generators; only using the solar setup. It was a great article. I can't find the original source.
i was born in 2001, so i was only 2 years old (i.e. too young to remember) when all of this was going on. but my mum told stories of how the whole street stepped up and helped everyone out - there was a small (probably 20-30 units) apartment building on the street with mostly disabled and elderly people, i.e. the people most adversely-affected by the power loss. some people went around, handing out burgers and hotdogs cooked on their propane BBQs, my mum had a bunch of ice cream in the freezer that was now melting so she held me in one arm, grabbed the ice cream, bowls, and spoons, put it all into a little buggy cart thing (not like a shopping cart, but a wagon), and walked up and down the block handing out bowls of half-melted ice cream. there was this older fellow who had a couple generators (just little things, the sort of generator you'd have with you when camping) and a few people dragged their freezers over (helped out by someone's dolley), apparently the entire street ate a big ol BBQ with ice cream bars, ice-cold beer, etc etc in front of this guy's place. even the old and disabled ppl from the apartment came out! my mum still has a picture of me, enjoying a 'magic' show put on by one of the ppl from that apartment! it's really interesting how communities, people who really don't know eachother very well, can band together in tough times. sure, it would've been easier to just ignore my mum and i, she's never been terribly healthy, nevermind those poor ppl in the apartment block - but how everyone pulled together to maintain civility, community, and got through the crisis together? gives you just a lil bit of faith in humanity, don't it?
I recently retired from the FERC and one thing that didn't get noticed is that typical fossil plants do not have black start capability and therefore cannot get the grid back up on their own (generally speaking). That means they need power fed into them before they can power up and generate back onto the grid. However, our hydro units in upstate NY, specifically the Raquette River projects do have that capability and were one of the main components that brought the grid back up. I know hydro has gotten a bad name environmentally but there are certain attributes such as black start, spinning reserve and voltage stabilization that should be worth more than acknowledged.
I've heard an anecdotal story wherein one ingenious set of engineers at a coal-fired plant somewhere in NY used their car batteries, wired together, to run the coal supply conveyor belt (ordinarily powered from the grid) long enough to get the system re-lit. Simple AC motors can be run on DC, and vice versa. AC motors will always spin the correct direction on DC as long as the VA input is enough, DC motors on AC have a 50/50 change of running in reverse (if they run at all). Them getting running in turn helped other plants restart.
This is not correct. Some universal motors with brushes can run on ac or dc. But normal industrial Ac squirrel cage asynchronous motors will not run in dc!
@@xheralt. You cannot run a regular AC induction or synchronous motor on DC. It won’t spin. What you’re probably referring to with the car batteries is using them to “flash the field.” An AC synchronous motor uses an active magnet field (usually the rotating part) to excite (generate voltage) across the stator. That field gets established using a DC source when the generator is getting brought online before the breaker is closed. The generator has a device called a voltage regulator that monitors AC voltage and current output to determine how much DC field energy to put on the field rotor. More field current means a greater flux across the stator. This can increase or decrease either the voltage (if the generator isn’t online but spinning up) or reactive power (VARs) if the machine is online connected to the grid. There is a rough direct relationship between voltage and vars based on the electromagnetic inductance of the generator and the electrical system it’s connected to. The greater the DC field, the greater the VARS, which can make the voltage go up (and vice versa). The voltage regulator only has AC voltage and current to determine how much DC field energy to put to the rotor. If the machine is spinning but offline, it only has voltage. The AC voltage doesn’t appear on its own because until there is DC energy introduced to the field, there is no voltage. A lot of older machines used something called a “pilot exciter” to initially excite the field. Think of jump starting your car. This is very similar to what a pilot exciter does and lights the DC “pilot light.” This is called “flashing the field” by giving that initial bump. A lot of plants did not maintain their pilot exciters over the years. So it is out of service a lot of times. So what they can do is bring a car battery in during start up and put the DC voltage across the collector or slip rings (the apparatus that transfers the DC energy from the main exciter to the DC field) on the DC field. This makes the AC voltage immediately start across the generator winding and the main exciter and voltage regulator can take off for the remainder of startup and while the machine is online. The battery is only needed for a brief moment on startup and the machine can do the rest. Modern voltage regulators with power electronics don’t have this problem as the solid state voltage regulator and exciter can do this on its own with no battery needed.
I've worked at the Indian River plant in Delaware and currently at Vienna plant in Maryland. Both of these plants have black start capability with an auxiliary jet that ties into the grid and we can get our start up power from it. They're also used as a kick in the pants for when demand is up but starting the main unit isn't neccessary.
The sound of nearly the entire neighborhood cheering when the lights came back on, man you had to be there to hear it. A timeless sound that will forever be etched in my head. I remember how it all began and how it ended. We definitely were all still pretty shook thinking the worse because of 9/11. My parents scared the hell out of me saying ppl were likely going to loot houses (we lived in the suburbs in bumblef*ck nowhere) I as a dumb 12 year old, imagined it all going down. We all slept in one room burning UP that summer, lol. Anyway! Very interesting information here! I wonder if anyone lost their job. Salute to those who put their back into fixing this.
Yeah people definitely thought it was a terror attack at first. I was 17 when it happened and my mom called me a few minutes after the power went out and told me she that something major was happening and it might be a terror attack. She was working at LaSalle Bank headquarters (Detroit Michigan area) and she said all of the branches were calling into headquarters to report power outages and many of those branches were hundreds of miles away.
@@Chris-hn4lp that’s actually wild because that’s where I was when this happened. Well, further north in Mt Clemens, MI. Small world, nonetheless! But yeah, I definitely heard people speculating the worse too. I had just came back from an acting class I was attending back then. I just remember going to a Walmart and it being pitch black inside. Interesting times.
@@SirPrincemarsallis In Manhattan too-not a BBQ, but the neighbors in my apartment building all brought whatever they thought might spoil without refrigeration down to the lobby, and we had a potluck. The pizzerias stayed open by candlelight (electricity was out, but gas for the pizza ovens wasn't) and so did about half the bars. Some people took it upon themselves to stand the middle of Broadway and direct traffic. I love this city.
I was not even 10 at the time. My dad had just walked in the door with 2 weeks worth of groceries when it tripped. I can still remember his “You have GOT to be kidding me!” As we hoped nothing would go to waste.
@@danieldanielson2650 For us it was about 24 hours, but there were people around us that were out for days. We also had family that got power back hours before we did so we took our food to their place to save what we could.
This blackout was my first day as a volunteer firefighter. Now I have the privilege of being a professional firefighter and had to deal with the Blizzard and blackout in Buffalo NY. Great subject!
Thanks for bringing back the memory. I was working at a printing shop in Toronto and the press ground to a halt, we had no idea what happened. Thankfully radio stations had backup power and within 30 minutes or so it became clear that it's not just a squirrel that got into a transformer. Getting home was brutal, the traffic was nuts. But the evening/night was cool, with zero light pollution and clear sky, it was an amazing full moon casting shadows at midnight, and people walking around with flashlights. We got power back the next day and life resumed... Much worse power outage came in 2013 ice storm, we had no power for 7 days during really cold temps in the winter just before Christmas.
In my small town, I was having a nap after school, and woke up about 10 minutes after the outage started (maybe because of the eerie silence?) I turned on a portable radio like usual to listen for news about the blackout, and heard the entire country was dark. (Thankfully, an exaggeration.) We ended up walking across town to stay with a friend, because safety in numbers, I guess. There was a single store still open, because they were the only ones who could manage sales using pen, paper, and solar/battery calculator. Naturally there was a line to buy bottled water (batteries were sold out almost immediately) leading halfway down the street. We passed the time talking, drinking, not using our battery powered radios and video games because who knows how long this will last? We might need those batteries for more important things... At this time we still had no idea what caused the outage or even how far it extended, so we didn't know if it was a system failure, terrorist attack (9/11 was still fresh in everyone's mind), natural disaster, etc, that could take weeks to repair. At one point we watched someone break into the liquor store across the street, but we couldn't do anything about it. Not even call police, since all the phone lines had gone down by then; although they had battery backup, The systems had been overloaded and the batteries had failed after several hours. The lights came back on at about 4AM, and things went right back to normal...
@@maximilian19931 That and the free food downtown restaurants and sidewalk cafés were giving to diners. But seeing the Milky Way from my urban back yard was the sweetest icing on the cake. If annoying neighbours weren't walking around with flashlights thereby wrecking my night vision, the experience would have been *perfect.*
Also a great place to share "where were you?" stories! I was in a hospital a day after jaw reconstruction surgery at 17. They had just moved me to a two-person room for further recovery when the blackout happened and they told me to "not get comfortable" and that I might be going home that day because of resources. Sure enough, my parents arrived two hours later and the hospital discharged me. I remember the main highways were so backed up so my parents drove the back roads (I had to go out of town for my surgery) half hour to get home. There was no ice in the stores but one of parent's neighbours gave us some so i could keep icing my swollen jaw. I was still coming off the morphine IV so I think I slept through the most of the blackout; when I woke, everything was working again. Hospital was in Hamilton; I lived in Brantford, Ontario, Canada
i was 9 at the time and at walmart with my mom. the power went just as she was about to cash out. it was a real weird day or two without power and having to listen to the car radio for any updates. atleast we had a pool at the time to beat the heat
Drove home from Toronto that night back to Hamilton, my buddies and I caught wind of Grimsby still having power, we stocked up on ice and beer, filled the bathtub up and partied lol great memories for sure!
I was a young operator (3 years experience) at a 700 mw coal fired plant in Alabama. I remember the voltage swings that day and our dispatcher frantically telling us to maintain voltage schedule and field breakers in automatic. I never saw anything before or since. I had a high "pucker factor" for a little while that is for sure. To affect us this far south tells me it could have been a lot worse.
Cascade through a much larger area. Almost happened in Australia when some plant owners thought it a bright idea to operate at constant output with no proportional frequency response. The regulator learned their lesson from that one.
Moral of this story, every time you flip the light switch on and the lightbulb illuminates, appreciate the hundreds of engineers and operators that make that little thing possible!
I was 8 years old when this happened and thought it was the coolest thing ever. It's crazy I've been ignorant of the fact so much work has been put into warming my home, keeping the water flowing and much more. Thank you to all those hard working people who created this system and keep it running!
Isn't it weird how it was so exciting to us kids? We just remember hanging outside and having barbeques. I remember being excited I found batteries for my radio so we could put on the news.
For additional context to anyone who isn’t hip to this story. The companies who were responsible to manage things (FirstEnergy and others) also had plenty of shady activities going on internally. So as you follow along with this wonderful walkthrough, keep in the back of your head the vision of your neighbor who has that old lawnmower that barely works. But it works so he won’t replace it even though he spends 5 hours every time just to get it running. Apply that exact mental visual to the scale of a company that is responsible for managing the grids as you listen to him talk about how these tiny pockets of problematic issues slowly domino effect
This was honestly the first nail in the coffin. There was a significant amount of outdated infrastructure that had just been ignored up until this point
I remember this blackout in Jersey City and NYC. Surprisingly, the blackout took so many people by surprise that there were zero reports of crime because everything came to a standstill. Instead, everyone tried to help each other when everyone tried to figure out what happened.
@@Wendifur_ yeah I definitely remember that. I feel like that mentality kept everyone unified and willing to help one another. One thing I also remember was my dad sleeping on a chair in the garage with a baseball bat because of ALL the days a long blackout would happen had to be while the automatic garage was still open 😄
I’m from Louisiana but was visiting NY in August of 2003 with a friend..was my last full day there before a long drive back. We were staying in rock away beach but had just finished visiting the Met and we’re on the subway when this happened. We had to walk 20 miles to get back. Seeing millions of people line the street, seeing the dark silhouette of Manhattan as the sun set while we were crossing into queens and then seeing the stars come out over New York were images I will never forget. The people who had made it home (this sss 2 years after 9/11 as well) were literally handing out food and water to those who still had to walk. It was incredible. We left the next day and it took us 8 hours to get out of the city via the belt Parkway. Absolutely nuts.
I'm a power systems protection engineer and confirm that this is a very informative and accurate representation of how protection would react in this situation. Really pleased that he got distance protection correct. I wonder whether Load Blinders on the distance relay characteristics would have prevented the cascade or would the reduction in frequency and voltage have led to the cascade being inevitable.
I was at a nuclear plant in upstate NY at the time. Sometimes it wasn't even the protection relaying. We started experiencing wider and wider frequency fluctuations as loads/gen units were tripping and lines tripped and reclosed. Afterwards it was determined that even though our main turbine governor wasn't in 'speed control mode' (nuc plants aren't used as 'regulating units' ) it just couldn't keep up with the wide swings, hydraulic pressure was lost and we went down.
@@mikefochtman7164 makes sense. I'm aware Nuclear is for base load only as takes a large amount of time to ramp up and down. In the UK we use mostly oil, gas and coal for reacting to increases/decreases in load. Basically, the gist I get is that this was a ticking time bomb with only one possible outcome. I am impressed that the grid actually coped with so many line and generator losses before the final collapse. From a planning perspective I only planned for an N-1 event. I think that the grid in question was at N-4 before the final collapse. One generator + three lines down.
@@stephenrowley4171 I don't know that much about this particular incident, however the grid was much larger than the area lost. I think that effectively this was load shedding to try and save the rest of the grid and prevent a black start event whether it was intentional or not. Load shedding is very hard to manage when not initiated by control. As I understand it from Mike above lines were autoreclosing which would cause huge power swings and make it very hard to try and balance the load & generation. Also bulk supply points usually have underfrequency and undervoltage protection to 'cut off an arm to save the body.' This is a kind of automated load shedding to try and save the grid from collapsing.
@@mikefochtman7164 what happens when a nuke plant has to shut down? Does it take a while or can you just SCRAM it? Do you have a bunch of excess heat to deal with? Is it difficult to restart? Nuclear is really interesting to me, but the only design I know much about is the RBMK for obvious reasons...
I remember this day very very well and I was so proud of my friends group. We had a contingency plan that if anything ever happened to the point where it was a panicky situation that required everybody to meet up at one place they would all meet at my place. I want to go pick up my friend Lindsey because she was the only one without a car and when I came home there was literally about 20 of them just waiting outside. They had generators and gas and propane generators and they brought their grills and their coolers and we started just cooking up all the food that was going to be going bad eventually and we basically had a three day long party. When we were on the third day we started siphoning gas from our cars so that we can continue to power our generators so that we can keep people's medications cold that needed it. And ever since then we all now use propane generators and we usually have around six propane tanks, each, at all times.
I was in my twenties, living downtown Toronto when this happened. I remember walking home from what I thought was a day off of work that turned into a week without power. It was surreal how quiet such a large city became, I remember some really hot nights while trying to sleep, the only source of light was the moon. It was pretty cool
Great summary. I was living and working in Connecticut that day. We lost power, and I had to help my co-worker manhandle a heavy centrifuge over to an outlet that was on emergency power feed so she could remove her samples rather than have her study destroyed by ambient temperature overnight. On the way home, I stopped off at a Mom and Pop grocery to buy a couple tins of cat food - the store was using hand calculators to sell food - and for their meats in the coolers, they were selling huge amounts to anyone who wanted for a dollar for what people could carry. I was between grills at that time, so my dinner was yogurt and the last of the backyard raspberry crop.
It's strange I remember living through this in Mississauga, Ontario and I remember the power being out for several days with rolling blackouts, but Wikipedia says the power was back within a day. Does anyone else remember it taking longer? Another thing is New York Mayor Michael Bloomberg and state governor George Pataki immediately blamed Canada for the whole thing.
I was wondering the same thing while watching this. When it said the power was back on quickly, that does not line up with my experiences in Michigan. Power was out for almost 3 days if memory serves me.
I was just coming into Saul Saint Marie when the power went out. We stopped at a gas station and paid cash for peanut butter and bread in case it was a long outage. We had PB sandwiches for supper after the owners showed us a hotel room by flashlight that evening. The next morning we drove to Sudbury where the power was on and we bought gas.
I was in Brantford, ON - we had power back that evening, lost it again the next morning, and it came on for good that second evening (~28 hours after the outage). But there was also about a week of dramatically lowered consumption, because the nukes were offline. I don't recall any rolling blackouts, but there was definitely a period of aggressive power conservation, with a lot of big factories and other heavy power users closing down to conserve power for necessities. That might be what you're remembering.
I remember this… no warning, no reason given, the power just went out. You expect power outages in storms and the like. But on a gorgeous New England day, in the late afternoon, all the sudden… nothing. I remember it being the first time that I really thought about disaster preparation. The confusion of the event was interesting. The radio was the hero of that day! Great episode, dude!
Since when power outages are forewarned? It is out, and nobody can predict it early enough to warn someone. It is out, and it is no reason to expect anything bad from it.
A very good reason for keeping radio going. Today's reliance on the internet and most TVs needing mains power, means that radio is absolutely critical in blackouts. With developments like RUclips and Spotify, and the internet in general, radio is increasingly becoming redundant. Most independent and local stations here in the UK, for instance, are struggling financially and many have given up their medium wave frequencies. And young people almost entirely don't listen to radio any more. Left to itself, radio could easily fade into one of yesterday's technologies - but this cannot be allowed to happen.
@@yaroslavpanych2067 well, usually you can expect that the power is likely to go out because it's storming or such... You don't expect them on a pleasant sunny day. Of course air conditioning and car accidents are still a thing, so you can never truly predict them.
@@maximilian19931 As someone who grew up with a single window airconditioner for the entire 1000 sq ft house, I can tell you that people won't be happy without ALL THAT AIR CONDITIONING. Not just be unhappy, some people will die because over the last 50 years, we've built houses and apartments around ALL THAT AIR CONDITIONING and now we have a lot shelter that relies on it!
I remember this, power was out in Manitoba for a couple of hours as well. Only time I have ever seen a true brownout, where the voltage drooped so low the incandescent lamps burned a deep orange for a short while before the grid went offline. Hydro here had our grid isolated and restarted within a couple hours and we saw just how big it was when the news came on tv later that night. What we thought was just a minor local outage ended up stretching across Ontario, into Quebec and sweeping south into the US and across the eastern seaboard and lasted 24 hours or more in some places.
Now that I think about it I remember a time years ago where the lights were super dim. I know it was during the summer, so it might have been this blackout.
I'm from the West Coast, but I remember this blackout very well. I had won a trip to Boston to see the Oz Fest and was on a plane on final approach to Logan Int. Our plane had to hold for maybe 30 mins. Boston still had full power, so we really didn't know what was happening until later drinking at a bar.
I was just starting as an electrician for a New England utility in '03. It's amazing to me how personnel have been eliminated even since then. If we had a significant blackout again like we did in 2003, it would take significantly longer because we do not have the boots on the ground like we used to. Automation is fantastic when it works but you need skilled, knowledgeable people at the generators and at the substations to coordinate things. In 2003 we had over 20 electricians in my shop, now we have less than half. Reliance on SCADA is a mistake that will turn to bite us one day. Grady, I really enjoyed your video, a suggestion for the next one, do one on recovery from a blackout called Black start. That should really help people understand the complexity and coordination required to bring back the grid once it is collapsed. Utilities that prioritize profit and stock price over reliability are the Achilles heel of the grid. You need skilled people. When things go sideways, relays and computers are not going to help you.
I wonder what you think about the hacking risk that apperantly many were saying existed in the grid, would it that vulnerable to recreate this but way wider and far damaging... I'm presuming the automatons would make it way worse.
Ppl used to say the exact same thing before the 2003 blackout happend. And will continue to do so till after the AIs take over everything. Maybe someday they will be right but i hope not.
I remember that day well. I live in the Rochester, NY area and power went out in most of the area. One generating plant that was located on the shore of Lake Ontario (known as Russell Station) was able to separate from the grid and stayed online. Since that plant had direct transmission lines running to the county's water pumping station and sewage treatment plant those were able to remain in operation during the outage. A few towns and villages that also had direct lines from that plant also still had power. Russell Station was seen as a hero during that event but due to it being a coal fired power plant it was closed a few years later.
I live in Eastlake, and know full well about our contribution to this event. I hadn't realized the estimated deaths that were caused by, either directly or indirectly, this major outage. It's always a bit of a joke whenever the major blackout comes up, though it rarely does. Knowing just how devastating it truly was, however, is crazy. Especially with hindsight. Thanks for covering this story.
I worked as a Control Room operator for 36 years in a Fossil fired plant on LI ( 4 Units 375MW each) I had just left a day shift that day when this happened. In the ensuing weeks I read the Interim Report from the Federal Government on the cause(s) of the blackout. This video is an accurate account of what went on. Good job.
As someone in the business, I just want to say all of your information in this video is excellent and spot on! Even showing the old regions, nice touch. It must of taken a while to research/ learn the lingo for this video!
my dad used to be an elected official and would tell me about working with engineers of all kinds for setting up, maintaining, and repairing all types of things. he respected this type of work but always had troubles understanding it. your videos are super important for the layman's knowledge of systems we all rely on every day. thank you for making this readily available and easy to digest
I just love how you, as an engineer yourself, defines the problem and gives a simple to understand solution to negate any possible reoccurring problems like this from happening again. Good job!
I was working as a welder at the time it happened. I was doing spray-transfer on some heavy steel plate when everything went dark. I lifted my welding shield, and still couldn't see, I briefly panicked, because I briefly thought I'd spontaneously gone blind...
I’m a control room operator at a plant in the northeast. Until I started there, I had no idea about, nor really gave it a thought past, how I flick the light switch & the light comes on. It still amazes me at how many people involved & tools/machines there are that give me the ability to flick that switch & light my room.
These kind of events are happening all the time and we don’t even notice. I’m fascinated how reliable the power grids are made, and how resilient they are. It took so many consecutive failures to actually end up in a blackout. Thanks for this video this is very interesting and well explained as usual!
Actually it was only one failure, providing power to Cleveland with their high demand. Stopping this wiuld have prevented the full blackout and only pull Cleveland off grid, keeping the rest on grid. Disconnect high consumers and save the rest.
Money in the pocket drives everthing. Top money thieves will always have back up as they can. The rest of society who pays for everthing suffer. America and its fat greed.
Seems like it mainly boils down to a computer glitch. If they had worked correctly and alerted operators about the faults, they probably could have dealt with them properly and prevented the failures from spreading beyond the first one.
@@renakunisaki Or if the power company actually cut the trees that kept taking down their lines. Preventative maintenance costs money though so it's better to let things collapse and fall apart.
I was 23 years old when this happened. I remember how the cascade failure stopped right at grid disconnect between Minnesota and the Canadien side. I remember for the first time being aware that a problem on the east coast could affect me in the upper midwest and how bad the grid system is. I think this saying puts it best “We (America) are a first world nation with a third world power grid.”
I ran an observatory here in New Zealand and we used to run dark sky night tours with highly powerful telescopes. We had a New York couple who loudly proclaimed "The last time we saw stars was 16 years ago (2003). We didnt know what they were at first!" and they shared a lot of stories about being mesmerized at the sky back then, as well as telling that infamous story that their friend rung 911 and told the operator that UFOs were coming (only because they have never seen that many stars)
That was such an amazing time as a teenager. It really brought the community together. My parents had a really good generator and every morning would take the coffee maker out front and make coffees for anyone who wanted one. I spent a lot of time outside with friends instead of playing video games with them.
I remember this, I was working 3rd shift stock at Officemax at the time in Michigan, our supervisor had the nerve to tell us we couldn't go home, and had to continue stocking pens and pencils and paperclips in the dark, with flashlights. That was when I walked out, hit the road and went north on I-75 until I finally saw lights on billboards and luckily my dad lived in a powered area, I never went back to that job or city since.
That is more a political story than an Engineering story, but still might be interesting. An preventable tragedy that will have affects for decades, and maybe generations.
@@stewartthompson72 it would highlight another critical dependency our modern way of life has: the pipes buried in the ground. Because they’re buried, we tend to easily forget about them.
Flint did not fail the water supply. State of Michigan politicians thought Flint didn't deserve clean water because the people of Flint didn't deserve it.
I remember this. It was actually pretty great. EVERYONE was out grilling for days and there was a non-stop road hockey tournament going on in basically every parking lot ♥️ Met so many neighbours for the first time ever.
I remember this happening. I was at work in CT. and everyone was confused and progressively got more and more freaked out the more we learned how far away the outage was, including Canada. Only two years after 9/11, you know where everyone's head was going. It got super eery trying to go home with zero power. The streets were a mess.
I was in Newport Michigan when this happened in view of the nuclear plant. I couldn't understand why we didn't have power despite there being a nuke plant within looking distance. I remember all of the theories people were coming up with. At first the obvious it was terrorism with 9/11 less than 2 years prior. The second was massive fires in New York. Third was we were bring invaded. Fourth the government purposely shut us down because we were using too much power.
As a system protection engineer, this is a great video! I was only in elementary school when this happened, but we still talk about it at least a couple of times a year during different conferences and seminars. You do an amazing job of taking hundreds of pages of documentation and translating into an interesting and digestible video.
Having worked in the power industries for a long time, I can tell you that the power grid is the single most important utility in the country. As you touched on, without power, traffic becomes gridlocked, water systems fail and on and on. It is a national security risk and should be on a level of importance with the military. Consider that if Bin Laden had really been trying to cripple us, he would have flown the airliners into the Hoover Dam and/or the massive power generation facilities along the Niagara river. Taking those facilities out would have crippled us for years, not a few weeks or months. Our grid is at risk from weather, terrorist attack, lack of maintenance and most importantly, political hubris. How many 2003s or Texas outages must we have before we wake up?
Lets be thankful terrorists are too dumb to realize that coordinated stiles on dams to stack flood surge (or possibly just the top dam breaks the rest, but personally i wouldn't take the risk it doesn't) is the most devastating thing they can do. Countless cities are down stream of major dams. Hell they could literally just shot a bunch of the glass & porcelain insulators on major transmission lines simultaneously and take the grid down. The grid isn't exactly guarded, and this says nothing of the cyber attack risk from nations like Russia or China.
@@jasonreed7522 I'd say they are not dumb, just like big flashy stuff... what makes the eyes pop for the target audience. Hostage a plane? sure, it get the news but ya dead or placed on trial. but big bangs? now that what we are talking about; let the surprise, anger of the target audience flow... and then the government have to fly those hellfires and kill some baddies. But what if, the civillians are within the OA? Hey... it's 2 in 1 now... we get another angry to-be jihadis... while we get rid of the current top wanted... even better if it's the wrong Tango.
@@jasonreed7522 A report came out years ago that names 4 transformers that could be shot that would take the eastern interconnection down. Granted it assumed no operator intervention. What really made people mad was as I said....it NAMED the transformers. In reality it would take more than 4 to get a collapse, but it still could be done. A substation in CA was shot up years back and caused an issue. Even more fun, not like there are back ups for the big transformers just sitting in a shed.
@@andrewt.5567 The report NAMED the critical targets, just how dumb do you have to be to do that. You can say something like just 4 transformers could be attacked and bring down NYC without opperator intervention and get just as effect a message, or have the names only in the report for people with clearance. I choose to measure my words and not name dams directly above major cities, even if it takes less then 5 min of effort to identify one that could hurt a lot of people. You don't do the bad guys job for them. (Someone tell our media that)
@@cmmartti Correct. Consider the 'Dam Busters' raid on the German dams in WW2 - they had to go to great lengths to get specially designed bombs to detonate deep under water on the upstream side of the dams, and it took several bombs to break each dam. (The water acts to confine the explosion and massively increases the effectiveness of the bomb). Flying a 767 full of fuel into the Hoover Dam is unlikely to scratch the concrete much, in fact it would be hard to think of a structure less likely to be inconvenienced by having an aircraft hit it. Crashing on the switchyard (or a substation) would likely do far more damage, though collateral damage (other than the consequent power cut) would not be that extensive.
I remember when this happened. We were leaving town for an important meeting in Toronto, and since both the Tim Hortons and the Gas station in town both had emergency generators, people were stopping by our township for them. However, the lineup at the Tim Hortons was far longer than the lineup for the gas station! Only in Canada.
2003 was the year of a big blackout in Europe too, it happened on 28th September 2003 in Italy. A 380 kV power line located in Switzerland was overloaded, the conductors sag and one of them touched a tree, creating a short circuit to ground and causing the line to trip. That line, which still exists today and is part of the swiss national electric backbone, starts from a big substation in the inner Switzerland near the city of Lucern and ends in a substation in the southern part of the country, in the canton Ticino. From the same busbar in that substation where the line is connected, another one starts and ends to a substation in Italy, near the swiss border. That line is being used to export energy from Switzerland to Italy, and the italian substation where it ends is located at the west of the canton Ticino's border. The interruption of the swiss trunk shut off the line going to Italy, and all the load that was on that line was took over by another 380 kV international export line, which starts in the canton Grigioni and again ends in a substation in Italy, located at the east of the canton Ticino's border. Those two lines in fact are part of a loop which starts from the big substation in the inner Switzerland, crosses a substation in Ticino, goes to Italy by crossing the canton Ticino's west border, in Italy crosses several substations located at the north of Milan until it reaches the other italian substation located near the canton Ticino's east border, then re-enters back in Switzerland, crosses the substation in the canton Grigioni, then goes to another substation in inner Switzerland and finally goes back to the big substation near the city of Lucern, where the loop starts. That second line on the east side overloaded too, conductors sag, another tree was touched and the line shut off automatically. At this point, all the export power stream that was going from Switzerland to Italy was missing. Loads redistributed and were took over by a couple of lines from France (the european grid is all interconnected), they overloaded shortly after and gone off-line. All the import load then moved over systems from Slovenia and Austria, which at that time were absolutely undersized to carry such amount of power, and they immediately shut off. Meanwhile sensors at grid control centers all around Europe were detecting heavy load transients, this to give an idea of how big was the thing that was happening in that moment. At that point all the power which Italy usually imported from the nearby countries was missing. Generators in italian power plants overloaded, slowed down, grid line frequency started to decrease until reaching about 48 Hz, power plants gone off-sync, disconnected from the grid one after another, the whole system collapsed and suddenly the whole Italy was left without electricity supply. It all happened in about 15 minutes, and it took nearly 2 days to restore the power supply in the whole country. The main cause of this nation-wide blackout was erratic/incomplete communication between each country's grid management companies. Swiss personnel, while unsuccesfully trying to reconnect the faulty line (phase angle too wide at the line's sides), asked their italian counterpart to reduce imports immediately after the first line trip, but they underestimated the amount. On the other side, Italy was importing from Switzerland more energy than the amount that was planned, this led to the first line overload that started it all. And when the second line tripped the cascaded effect begun. The reports from both Italy and Switzerland are available on the Internet, they're very interesting and fascinating.
I vaguely remember this. Wasn't most of Switzerland's almost completely electrified train system offline for about a day too, stranding a few hundred thousand passengers?
Watching the video I also remembered this incident and thought "well, this might be worth investigating, too". Then I found your reply and was happy to learn more about this. Here in Germany we've heard of the incident but didn't experience any outages due to it. Thank you for summarizing what happend then!
@@uschurch No, at least as far as I can remember that day there weren't particular issues with the railways. Swiss railways company (SBB/CFF/FFS in german, french and italian) runs its own grid, which is a 2-phase (180° phase angle between the two) 16.67 Hz system, with its own power plants and substations. There are some substations where both the regular 50 Hz 3-phase and the railways' 16.67 Hz 2-phase grids are connected, with equipment that act as interface between the two systems, so there are some junctions. But essentially the railways' grid is an independent system. From your name I guess you're from Germany or Austria, if I remember correctly in your country there's a similar configuration as I remember that, when visiting those countries, I clearly saw poles carrying wires in multiple of 2 instead of 3, like it is by us for the railways grid. Some years later there were a couple of accidents where important parts of the railways had issues with power supply. Maybe you're remembering about these, but the disruptions lasted some hours, not days, and they weren't nation-wide events. On that day of 2003 instead, here in Ticino, where I live, we were affected too by Italy's power outage for some hours during the night, but we had power restored by the sunrise. I actually remember that I saw all the streetlights still turned on during the late morning and I was wondering why. Then I heard of what happened during the night and I understood, as the timers stopped during the power outage and when power was restored they were all off-time. This happened because the two 380 kV lines coming from inner Switzerland that tripped, carried electricity not only for export to Italy, but to supply our canton too, as they power our local 220 kV grid. When they went off-line canton Ticino was actually nearly completely isolated from the rest of the swiss grid, with the exception of just some secondary and quite old and undersized 220 kV links which failed pretty soon too, and for a certain moment we were actually powered by the italian grid, I think just by the east-side international line (the one which is connected to the substation located in the canton Grigioni that I mentioned in my first comment). Once the italian system gone offline it pulled us down too.
@@DavidSallge You're welcome, it's an event that fascinated me and I like to talk about it. If you search in Google you'll find the reports that describe all that happened that night. The problem is that I don't know if you'll find them in german or english (maybe for the swiss' one there's a german version too), as I found documents only in italian.
I worked maintenance for a company in northern NJ when this happened. I was replacing light bulbs in ceiling fixtures. I was turning the lights on and off, checking as I replaced each. A co-worker came running in screaming and demanding to know what I did because the whole building's power flickered several times (while I had the lights off)and blamed me, I had no idea it was happening until we lost power completely. Co-worker still was blaming me. 🙄 I was having company that night and it was the hottest day and so humid. We sat around sweating our a.. off only to find out 3 hours later that someone's, at the party, house less then a mile away had power and a/c. I often think about that day because although we were physically very uncomfortable, it was a very enjoyable time
As a power systems protection/relaying engineer, I find this quite interesting. This blackout has been documented in many IEEE technical publications. Interesting to see how trees cause ground faults on power networks. 16 lines here tripped on ground fault !!!. I love protection and control , challenging and exciting speciality in electrical engineering. I love the way you speak of zones of protection. Distance protection is the work horse of transmission line protection. I encourage electrical/power systems graduates to go into P&C as protection engineers are always in demand, both in operations and consulting practices
I lived in midtown Manhattan when this happened. Looking back,it was amazing how everyone helped each other. The elevators were not working so we helped the elderly walk up the stairs in a 20 floor apartment building. The restaurant’s were giving out free food
I wish you would do a story on the New York City blackout in the early '70's. I remember being stuck because electric subways were not able to work, but never really learned what actually happened.
Are you referring to the blackout in 1977? I wasn't born till the 80s but my mom was living in NYC for the 77 blackout and was playing bingo with about 100 other people in a church basement. She realized that she had won and yelled "Bingo!" and poof, the lights went off, lol. She was still somehow able to receive her winnings ($500) but she remembers getting home that night was chaos.
I always find, as the daughter of an engineer (chemical), sister of an engineer (mechanical) and wife of an engineer (civil), that your videos are very interesting and informative to a person who trained as a biochemist and economist and has a strong interest (as a civil servant) in economic infrastructure for enterprise. And having watched the impact of the 2003 blackout in real time, this was a particularly interesting explanation.
I grew up in Toledo and I was maybe 4 or 5 when this happened. I read the title and I vaguely remembered it, then I remembered everyone panicking over a “blackout” and me being so confused that nothing worked. Thanks for the video and filling in a lost memory for me lol
This is a very good video on that outage. I worked at a control center in Florida and I remember seeing to frequency effects all the way down in Florida. We had telemetry with the MISO EMS system and the operators there were freaking out that day with good reason!
This was a really complicated topic and you did a great job explaining it! Thank you! I'd be interested in a follow-up video going into more detail on the changes to the grid since 2003, and what vulnerabilities have/haven't been fixed.
The more I learn about electricity the more impressed I am with how we even managed to build these types of huge, complicated networks. It's incredible what humans have managed.
Very well explained! I live in Cleveland and i'll never forget that day as it was the day before i went off to college! Was in Strongsville and we needed gas and every station had their driveways blocked off...thats when we noticed stop lights were out and thus it began
This happened in summer and not gonna lie as a teenager this was kinda fun lol. We could see the stars at night, hang out in the neighborhood until sunset, had bbq’s because all our stoves were electric. Only thing that sucked was cold showers since my house didn’t use gas. This video brought back so many memories.
I work at a public aquarium and have experienced the cascading effect of minor problems in the network infrastructure stacking to collapse the whole. It's scary how quickly your world can go dark around you if you're not paying attention or especially if you are paying attention, but you don't have reliable data... I feel for those operators that got blindsided by bad data coming at them. Keep up the good work for all the other days that you kept it running smoothly! ⚡
I do some electronics circuit design myself. We are "intelligent" and design systems not randomly but with the components they NEED. conversely this means any failure was something that was NEEDED and will have a cascading effect somewhere else. I was thinking about this a while ago and its actually scary how one component failure will make the entire system inoperable, and this holds true for not just electrical systems. it also extends to society and human body :( . downside of intelligent design i suppose
@@monsterhunter445 you can have protections but you cant really design for fault tolerance because any component can fail in a number of different ways, in best case it will make the design 4X more complex and then its exponentially more prone to faults. Best you can do is design a duplicate system as backup. Like i said, its scary.
Grady, I wish I had a teacher like you in school. I would have become an engineer for sure. You explain things so that everyone can understand. and you dress very nice too..
I very vaguely remember all of this. I live in Nevada now but am from Cleveland and was young then and I recall it being nuts. Who knew trees, and a relay could cause so much chaos. The people who basically run our lives by helping keep electricity flowing in nearly the entire country are folks to be appreciated! Well done and amazing work on this video.
This has been very enlightening! I lived through this event in upstate NY, at the time we all shrugged, thought it was a local outage. It's the boonies, it happens. Then, over the radio scanner heard a fire call go out for a transformer on fire - which was a false call, which just further made us think, "pfffffft, no big deal..." It wasn't until the hours started passing and turning on a battery powered radio and tuning into one of the AM stations broadcasting out of NYC that we learned of the magnitude after it all. And until watching this video the take away was "Yeah, a transmission line in Ohio sagged into a tree because it was overloaded carrying another transmission line's load as well." This video has been a big eye opener into all the moving parts that really went into the whole thing. It really was the perfect storm of cascading faults and errors.
A couple of interesting events that happened in Australia that would be worth looking into and doing videos on are the south Australian blackout and subsequent black start of 2016 where the entire state went down, and the Callide C accident, where one of the units (from my interpretation of the report) essentially suffered a turbine trip. but due to a loss of both AC and DC control voltage systems was unable to disconnect from the grid and started motoring
I was right on the fringe of this blackout in Michigan. Our area still had power but we had a brownout for a couple of hours. The lights were dim, the TV (analog) hardly worked and the air conditioner was making a funny whiny noise. The power company must have borrowed power from other utilities further out West when the brownout ended.
I live on the east side of Cleveland. I was 7 years old when this happened. We were out in the swimming pool all afternoon, and when we came inside around 4 PM, the TV was off--which is odd, considering we typically kept it on for the dogs. I remember the absolute silence in the house with all the appliances off, and thinking it was weird, but cool. My grandfather and step-grandmother were visiting from Arizona and staying in my house at the time, and I remember later in the night all of us crowding around my dad's crank-operated TV watching the news. I remember going to bed by candlelight, and flushing the toilet with water from the pool, as we live in an area higher in elevation that relied on a pump for water. I didn't think anything of that day until I got older and got my radiological mitigation certification from FEMA. Part of that course was learning how a nuclear power plant operates, and I started researching Perry NPP, which is only a 30 minute drive from my house. It's then that I realized why FirstEnergy is so aggressive about hacking our trees to bits every year, as their lack of tree trimming service was a leading cause of this event. Since FirstEnergy filed for bankruptcy, they'd been lax about tree maintenance, so I fear it's only a matter of time until this happens again.
I was living on Long Island, NY during this black out. That was one of those problem areas you mentioned in NY that took a couple days to get power back. I was young at the time, but I remember feeling excited that my radio could be battery powered and we could listen to the news. I remember hearing how it was a massive black out but not much else. My parents made the most of it, we had a barbeque with the neighbors and spent most of the day playing in the yard.
The precursor to this event was East Lake Unit 6 being taken off line the evening before for repairs. This caused the need to import about 600 more megawatts into the Cleveland area that weren't there the day before. Importing power, in this case, put an additional reactive strain on the transmission system. This was made worse by the East Lake unit being off and not being able to provide the reactive support - VARS to go along with the 600MW import. Also, when the blackout occurred, much more load was lost than generation. This caused the highest frequency spike ever seen on the Eastern Interconnection due to more generation remaining on line than load to serve.
@@FOH3663 The frequency is maintained at 60.00HZ unless the interconnection is on a time correction. Typically an excursion from 60 is negative and it's due to a large loss of generation, resulting in too much load for the remaining generation. This happens fairly routinely, but the frequency usually doesn't decline to much below 59.94HZ. It's quite unusual for a large enough loss of load to drive the frequency up any significant amount. This system upset did that. Frequency recording charts (now electronic, but I sure remember when they were paper!) are scaled to show declines from scheduled 60.00 to about 55.00HZ. This is for posterity, as after a decline to about 58.50HZ, generators have come off line due to protective relays opening their breakers to prevent turbine damage. These same charts don't go much over 60.20HZ because the frequency just doesn't usually move in that direction any significant amount.
Eastlake 6 is a 35MW combustion turbine. You're thinking of a different unit or a different plant. I know this because I did the siting engineering for EL6 working for CEI in the 70's. What you say is true if you change the 6 to a 5.
Story time. I was in the Army stationed at Camp Yongsan in 2018-2020. Camp Yongsan is in the middle of Seoul. Its being decommissioned, so there is very little oversight. Seoul, and Yongsan are on different electrical grids because of US/Korea military protocols. One cold morning, the whole base lost power, and since I work at the data center on base, that was a big deal. We have two 1.2 Megawatt rated generators on site specifically for the data center. The problem is, no one knew hoe to start them, and cut over from shore power, to the gennys. I step in to figure it out. Having no formal background in such things. I got the Gennys running, but the control systems were in Korean. Eventually I got the building powered on, and everything seemed ok. The gennys run on a 400V system, and I did notice the ammeter dial was nearly maxxed out. And when I went outside, the street lights were back on, and traffic signals too. It was very early, and everything seemed ok, so I went about my business. You might be able to see where this is going.. Within an hour they called me back, and told me both gennys breakers tripped. Well.. I dunno about you but 2.4 Megawatts seemed like a lot for one building. Plus, the street was dark again. Weird. Come to discover, my building was basically providing power for the entire outage area for about an hour, courtesy of myself. Figured out that the shore/genny switch wasnt automatic, and I switched it over to genny only, and everything was fine. The ammeter showed only about 10% utilization. Actually, we only needed one genny running for our building. Oh well. Thats one way to learn. I never told anyone, they were just glad it worked. My own personal 2003 grid failure scenario. Here is a link to a video of the actual generators from my story I filmed on that day. photos.app.goo.gl/twmp7zAhXm3JeXCq9
I was 13 years old and we lived near Boston at the time of the great Northeast Blackout of Nov 1965. I learned a valuable lesson that lasted to this day regarding keeping a good portable radio and flashlights on hand and a supply of spare batteries.
As a EEE grad; WOOOOOOW; amazing summary of events, and great explanation of the complicated dynamics of a cascading blackout. Your very brief, but approachable explanation of Real vs Reactive Power is to be envied! Really concerning that the Right-of-ways are not well maintained enough to prevent the trees shorting the lines. I wonder how much the computer system was to blame, and whether or not over-reliance on it was the primary cause. Seems like they need to do more 'paper days' where they can become accustomed to fallback methods of monitoring and control
Most importantly they now have dedicated signals telling them if and when the computer fails, so they can focus their brains and eyes on the raw data instead of starting at the main panel stuck on "no problems detected".
I used to write software for a lines company, I would to get annoyed at the paranoia of the control room guys whenever we wanted to roll out something new. Having seen this, I have a lot more respect for how I approach these systems. In IT a cascading failure is usually relatively easy to resolve, so I can fully understand that naive approach when it comes to the real-world effects of our critical systems failing from developers.
@@Simcitywok I just did a case competition related to cybersecurity and this was the bulk of the points we ran into. No one wants to upgrade IT because it's one of those things that no one notices when it's actually working properly.
Its politics, not the controllers or the developers of the grid, or the people coding the circuit boards routines. You need a party that is controlling rehabilitee, controlling all the operators on the grid. We need a dominate party here, or some Gorvenerment agency controlling the independent operators.
@@lucasrem1870 I lived on a remote Alaskan military base for a year. The power plant kept the system stable for the main buildings, but the TV station often saw wild swings in voltage, or power outages. I was working one say when the 120 volt power to the equipment dropped to 20 volts. Then it pegged the voltmeter at 210 volts before almost every breaker in the building tripped. It damaged some equipment, but it shredded the film that I was running of the start of the World series. It was so unstable after that, that I was pissed. The power plant manager just laughed and told me that there wasn't a damned thing that I could do about it I thought, "Hold my Mountain Dew, and strap yourself in!" I put up a 'Technical Difficulties' slide, then had or DJ record a broadcast cart that stated, "AFRTS is experiencing power problems. We will return to our schedule, when it is resolved. If you have any questions, call:XXX-XXXX" The phone number was the unpublished number to the power plant manger's office. He called me and ranted, "What the hell are you doing to me?" I just laughed and told him, "There's not a damned thing you can do about it!" and hung up. Never piss off the Engineer! 😆
My parents and I were on our first trip on QE2, a Westbound Transatlantic, when this happened. We stayed at the Helmsley Hotel in Manhattan for 3 days from the 17th August and I got the water equivalent of food poisoning from using the tap water in our room on the 29th floor despite my parents checking with the staff and being told the water was safe to drink. We still had a good time in Manhattan, including going to the Central Park boating lake which I've wanted to go back to for many years!
Bravo! This has been the best explanation and the first explanation that actually makes sense to me. Thank you for putting the effort into this. I was playing Diablo on my computer in upstate NY at the start of this. My UPS started arbitrarily beeping so I turned around and looked at my dining room light and saw it strobing slowly. I shut everything down and turned off my main disconnect, then I got out my meter and checked my lines coming in. It was usually 230V nominal. It was slowly going from 170 to 270 volts about twice a minute. Clearly a phasing issue. Nothing said up until now has justified that to me, but this explains it perfectly. We were down for about 12 hours. Thank you again. Keep up your amazing videos.
2 года назад+6
This brings back some memories. I was a graduate student in Ontario at the time. One afternoon our building on campus lost power. We didn't think much of it as there was some construction and the crews had cut a cable just the week before. After a few minutes we decided to go home. That's when we realized that the entire campus was without power, and thought that was bad. The whole magnitude of the situation only hit us when I got home and saw that there was no power anywhere in the city.
I’m a retired utility engineer and was working that day as a consulting engineer. You gave the best explanation of the August 2003 blackout that I’ve heard. One thing that needs to be said is that the power grid in the US is itself the largest and most complex“machine” ever built by man. While it’s reliability has been amazingly good, it’s really limited to our understanding of every conceivable event. It still requires the human mind to interpret events and make on the spot decisions. If this human intervention is not present, a cascading outage is the results. All major outages are a result of the lack of timely operator intervention. One major factor is that the system itself is composed of protection equipment that ranges in age from current technology to devices over 100 years old. It all has to operate together. It’s simply not possible economically to upgrade the entire system at one time. The best that can be done is to plan for every contingency we can identify and have an operating plan that addresses those contingencies. As computers and Artificial Intelligence become more capable, we can analyze more contingencies and anticipate more events. However we then become more vulnerable when the technology itself fails. It’s a balancing act. Most of the time we get it right. Sometimes we run into a situation we never anticipated. I doubt it will ever be possible to completely remove the human factor from the operation of the power grid, nor would that be in our best interest.
I was working on the eastbound Amtrak Capitol Limited and we were halfway across Indiana when the power went down. The signals that control the rail lines went dark and operations were immediately impacted. We could only advance at about 15 mph between the dead signals and we only made it to Cleveland by 0830 rather than around 0200. We didn’t reach power until we got to Pittsburgh in mid-afternoon where we were due around 0800. A trip from hell…
"Only a few places took more than a day." Detroit was out for 3 days! This was super interesting to finally understand why I lost power for 3 days as a kid. I'd heard tons of rumors, including historical markers that claim it was a squirrel on the line in a key location. Thank you once again for making these videos covering these failure events and explaining how and why they occured. They're incredibly interesting and I've been sharing them with my other engineering friends.
I was on the edge of the blackout area, in Ypsilanti, MI. I could tell the outage was an unusual one because it was preceded by several minutes of voltage sags, voltage surges, and frequency deviations as the grid became uncontrollable.
Yeah, I was living in South Ypsi at the time, I only really remember hearing some stuff about power grids failing and not having AC/electricity for about 2-3 days, but admittedly i was 9-10 yrs old around then.
I was at the University of michigan, I had just gone home and my home was at the boundary of DTE snd Consumers Power- we had power but I saw the stoplights a mile to our East out, I called in and got no answer, so I turned around and went back. Spent the next 60 plus hours at eork. The UM power plant lost two of three generators but due to luck the last one stayed up so it didn’t have to restart lack. We had absorption cooling at critical buildings which let us save electrical power and due to the incredible competence of our trades staff we saved all the critical labs. Ive never been more aware of how complex our civilization is- or of what irreplaceable an asset human skilled people are.
![[#2024MAMA] G-DRAGON - 무제(Untitled, 2014)+POWER+HOME SWEET HOME+뱅뱅뱅+FANTASTIC BABY | Mnet 241123 방송](http://i.ytimg.com/vi/Ox29z5Nf1Uk/mqdefault.jpg)
🏭 Watch more of my power grid videos here: ruclips.net/video/v1BMWczn7JM/видео.html
📫 Big announcement coming in March! Stay up to date here: practical.engineering/email-list
Perhaps you could make a video on how to put the grid online after catastrophy like this. Why it's difficult and how long it actually tookb this time.
Hey I had just turned 3 a week and a half earlier. Cool
I was living in downtown Ottawa, Ontario, Canada. I remember the blackout vividly. Not sure where you got your info, but my power was out for 3 days. My house in lowertown was literally one of the last to get power back.
My roommates and I all took shrooms, wandered downtown in the pitch dark, then almost burned our house down trying to cook a full pack of bacon on our little camping bbq. Good times.
@@Stevie-J 🕌🕌🕌🕌🕌🕌
@@Stevie-J Me, too. But that unbuttoned pocket was a distraction. ;-)
The power plant in the very first frame, next to the United Nations building in Manhattan, that's where I was... in the control room as a very young Engineer. I was actually giving a tour of our control room and flipping through some system graphics, when all 3 units tripped offline at nearly the same time, and the plant went completely dark, which before then was thought to be impossible. One of the operators actually yelled at me because he thought it was my fault, lol... I was relieved when the system operator called a few minutes later, and told us to, "Standby, we just lost the entire Northeast." We overcame a lot of challenges that evening, and for a young Engineer, it was exciting. I slept on the roof that night because it was too hot inside. I remember looking up at the night sky and seeing the stars, and thinking... when was the last time anyone in Manhattan looked up at the night sky and saw the stars like this? Great job on the video, and thanks for bringing back such great memories.
Your "fault", lol
The whole NE went dark in the 60s too IIRC.
Thats a beautiful story!
That is a hell of a story, something you tell your Grandkids.
Hmmm.... what's this E-Stop button do?
Signed; Homee
This happened when I was 4 and I thought I caused it by watching too much TV. I've lived with the guilt for many years, thank you for explaining what really happened and for clearing my conscience.
@@mellohi6175 the last bit was a joke
@@cosmosisrose inb4 they try to save face by claiming they were joking too
Guys i think it was just supposed to be a light hearted comment, it’s okay.
Man was watching too many Saturday morning cartoons and caused 10 billion dollars in economic damage 😔
@@brandonhuynh4528 he must’ve been watching too much Tom and Jerry 💀
When you learn about the complexity of managing the power grid, it feels like a miracle that we consumers are actually allowed to do whatever we want and use as much power in whatever way we want in our homes. Something to be thankful for, since all the heavy lifting and problem solving is being done by someone else on our behalf.
We pay for it so why not? They're there for us, not the other way around.
We should consume less to keep those guys from those problems. With load sheading, like reduce power consumption on peak times.
On the opposite side of the world there's also USSR power grid, which is kind of unique in that it covered 12 timezones. Would be interesting to hear more about it. Also there is recent outage in Middle East (from what i know conclusion was reached and it was caused by short circuit in one of coal plants and lack of water in hydroaccumulating power plants because of dry year), and there might be another one coming up with Ukraine disconnecting from Russia and Belarus.
Power usage is mostly quite stable, there's morning peak and evening peak and other than that it's on similar levels. Also it's mostly related to "lifestyle" of whole community, and in lesser degree to how much an individual uses. As even if one person consumes 10kw more, it will not be noticeable on a scale of 1000 people, other than maybe causing some questions to that person. Bigger problem for power grids is solar generation, as it's very unstable (it depends on weather after all) and it doesn't really help with two peaks that often. Also as in many places there are laws giving preference to local "green" generation, so people can "give back to the grid", which, when happening on a large scale, is lowering midday demands, but only sometimes. So companies running power plants have to accommodate these power spikes somehow, which means generators can't always work in optimal mode, reducing their lifespan. Overall I think that there's a better solution to climate change than current solar/wind craze.
@@wumi2419 Good points!
Not entirely true. The amount of current we can use in our homes is limited by breakers. But yea, there's a lot of freedom for consumers. So I agree with the gist :)
I was working in midtown near Times Square when the black out occurred. I lived in NJ at the time and found myself stranded. I had only $4 in my wallet and spent that buying a couple bottles of water as it was indeed a warm day. I heard that there were buses leaving from the Port Authority so I walked over there only to find a massive crush of humanity there. Had to be well over 100,000 people around there trying to get out. Then I heard the ferries were running across the Hudson and walked to them only to find many more thousands of people trying to get on a ferry. I gave up, and walked back to the office building where I worked and eventually got a ride late that night in a car service sedan with someone seated literally in my lap. We drove alone through the pitch black Lincoln tunnel. So eerie. The lesson learned: I always carry $300 in emergency cash ever since. When the electricity goes out, cash rules all, as I found out the hard way.
Damn...I just remembered how amazing that outage was. A bunch of us kids in the neighborhood had sleepovers, people were just going to each other's houses. We were playing in the streets. There was such a strange and unique coming together of humans where I was. As soon as the power came back, people started to go on about their usual business and that special moment was gone
I grew up in a third world country. Blackouts like this are the norm, it will be just another regular week. It’s crazy how the 1st stands still if power goes out just for a few hours
@@simba8665 idk how hanging out with others equals standing still for you this is what happens when you rely on technology
@@sparklight0964 I think it’s more about the liminality of the entire event that caused time to stand still. Not the technology.
Some people still do this till nów You know ?!
I was born that year 💀
The Lift Pump Station located near Ludington, Michigan helped save the entire eastern half of the US and Canadian power grids. This facility maintained grid frequency by absorbing most of the nearly 5 GW load imbalance as the disconnects around Lake Erie engaged. The pump station filled itself to the brim, well past its safe design water surface elevation, then ran itself nearly dry, again outside of safe operating levels and at great risk of causing cavitation in the turbines to provide power afterwards. These actions helped prevent catastrophic damage cascading all the way to Denver and gave some time for other stations to reconnect.
If a power station is scrammed, a rapid shutdown, and depending on its type it can take days if not weeks to get a power station back up and running. A pump station can provide nearly immediate power generation.
Thank you to all of the station operators, line workers and service crews who struggled on that day, your actions and quick thinking, kept the lights on east of the Rockies and prevented a far darker outcome.
Both the 2003 and the Texas outages could have been much worse if the load imbalances had been allowed to progress a few minutes or even seconds longer.
The control systems in place now are night and day more robust than what was present in 2003.
Sounds like an entire video could be done on their actions that day
@T.J. Kong Now imagine a nation state deciding to infiltrate the intranet these stations connect on.
This was the early thousands, when communication systems were less robust
Living in Mt Pleasant, MI at the time, all I knew was that we had power and Detroit didn't. Now I know why! Thanks for that.
This sounds like an amazing story. I did some quick googling, but I didn't find any additional information. Would you mind sharing any articles about the Ludington Pump Station operators?
I was working at a 2000 MW power plant that day. I recall hearing air blast breakers operating frantically in the switchyard just after 4 pm. Lights blinked and then moments later the plant was black. From my office, I looked across the river and saw two other large power plants also fully tripped. Our plant generators were spinning, but there was no grid to connect to. We hoped to keep the units ready to resynch, but we could not keep up with demin water consumption, and one by one, lost the plant. The grid was partially restored by the next day when we were able to fire our units and resynch, thus rebuilding the grid. Certainly a day etched in my mind!
What kind of plant was that? In the hydro world we just keep one or two groups spinning and exited but asynchronous to maintain our auxiliary transformers indefinitely. What consumes demineralized water and why can't it run on local power?
I'll bet
@@DickCheneyXX Almost certainly the boiler make-up water for thermal power station.
If there was a station blackout, where did you get the power to run the pumps and the draft fans?
@@gregorymalchuk272 We built the grid backwards from our hydro electric plant at Niagara Falls (their station service requirements are virtually zero) and when we had power at our station we were able to restart our units normally via the reserve station service transformer. We could aid have used standby gas turbine generators to do the same.
I am a software guy and I LOVED this video. Really shows how much our work, when done improperly, can have an impact.
Would have been NICE for the operators to have known none of the alarms were working! Communicate.
Or, improperly - Crowdstrike.
I was a kid in Brooklyn during this blackout. It was one of the most memorable nights of my life because we got a full night sky due to no light pollution. You usually never see more than 2-3 stars at night in my neighborhood.
The whole neighborhood came out, and everyone was hanging out. It was cool. Definitely could use some more nights like this nowadays
Humans are moving towards a certain direction and there are no brakes on this ride. The days of genuine, distraction free human companionship might be long gone.
New Rochelle over here I was 9 at the time, and I can confirm, saw my first shooting star with my brother. We didn't have power for almost a week it was one of my favorite memories ngl.
My brother and a friend were in Manhattan that night and said it was the most bizzare thing to see a full night sky from the streets of Manhattan. People were generally good too, handing out beers and cooking food that they didn't want to go to waste.
That's sad, you must come to South Africa 🌍 beautiful night skies
I'm not gonna lie, as someone who was in NYC during this, Manhattan specifically, it was actually really fun. Gas service was still going so restaurants were cooking and effectively giving away all the food they had built up, everyone was out on the street ( who wasn't trudging across a bridge) was drinking. Driving...yes driving...was actually really easy because traffic was self moderating. It was a very surreal experience, all told. Spent the day after that first night out in Brooklyn by the water drinking with friends and then hung out at my friend's place on the west side and watched them switch the super blocks back on one by one. Looking up 8th avenue, it was kind of amazing. From the horizon you could see entire sections of Manhattan switch on, I wanna say like 10 block squares at a time coming towards where we were down in the 30's.
Goddanm that sounds so great..
phuturephunk: As Mr.Spock would say,"Fascinating"
I was in Brooklyn when the blackout occoured in 2003
Same! I actually think that blackouts are fun but of course dangerous for some but still fun for me personally.
Down in the Philly area, we watched the entire hot mess on the news.
With our jaws on our chests.
We're a 21st century society with a 1950's electrical grid. :/
When first learning grid operations it was explained to me thus:
Imagine the ceiling in a room has a wide variety of hooks embedded. Each hook can only support a different weight and will pull out if overstressed. Now imagine you have a wide variety of weights that you must suspend from these hooks. You attach them with rubber bands of varying strengths. You link a bunch of hooks and weights with bands. But if one hook pulls out, the weights stretch rubber bands and shift and you need to keep things suspended. Now, plan things out so you never drop a weight, regardless if a hook pulls out, or a band breaks.
Whoops, you used too many rubber bands and nobody wants to pay for that. Try again but with less rubber bands. I don't care how safe it is as long as none of the weights ever fall.
(Yes, that last sentence is supposed to be contradictory.)
I love this explanation! It puts it in complex and simple to understand way to think about the grid, it's various load capacity factors, and the various loads on it, that nobody can really control.
SCE in California can shut off all the air conditioner compressors for a huge number of customers by sending out a radio signal to a box on each A/C unit. They could do this before 1986, when I found out about the system. This can shed 100's of MW of power load from the grid instantly.
I remember this quite well. I was working in Solon, OH at the time, and the power tripped off at around 4:00. Some phone calls from our facilities manager confirmed it wasn't just us, but pretty much everyone in NE Ohio. They sent us home since we couldn't do much except twiddle our thumbs and it was clear it wasn't coming back on anytime soon. Got home, pulled the rope to open my garage door by hand, turned on the camp lantern for light when it got dark, and kept listening to the battery powered radio for news. Lit my gas stove with a lighter and made dinner, and read a book by candlelight. Quite peaceful, actually. When I heard that water pressure was going down I filled my bathtub so I had some water. Power came back on around 10 PM. Next day, I found out this was huge problem for lots of folks. I didn't have a generator back then, but it was certainly eye opening.
that summer was so hot, we live in a 12 floor building in Manhattan. everyone that night went to the roof to sleep, and everyone on the rooftop brought their flashlight and start playing chess and other board games. it was pretty fun for kids like us. you get to see every neighbors and kids.
idk if anyone still remember, post-power shortage, people starting to get free TV channels where these channels usually need to get payed.
I lived in Elyria at the time and was at work, as well. Thankfully, I lived within walking distance and my power was restored around 9pm
Bowling Green reporting in! I remember being sent home early as well. we went out to the bars and had candle lit beers. cash only! lol
I'm from Akron didn't it start there?
I loved this day. I lived beside a valley with a river. After so many hours, and seeing that everything was down, i set up a campsite beside the river. Will never forget how bright the stars were. I was actually sad when the city glow returned.
Seeing a sky full of stars, especially the Milky Way, really changes you
@@mop9091 First world aspirations. sigh.
People died from no electricity to hospitals and elderly care homes and people on medical machines, go to hell.
@@painless4785 wym
ok
The power went out above my head as I was boarding a plane in the Toronto airport. Ended up sitting on the tarmac for two hours, not knowing what the cause of the delay was. We were shuffled off the plane and I, a thirteen year old about to embark on their first solo journey, was left alone in an airport. I was too young for a cell phone at that time and the pay phones weren't always working. Managed to get a hold of my parents who brought me home. I can remember the sweltering heat. My elderly dog ended up passing away that night because of this heat. No vets were open because of the blackout. We didn't get power back in my house until 7am the next morning. It was...definitely a bad day for me.
Ah man, sorry to hear about your dog. Talk about making a difficult situation a 1000X worse. :(
Sorry about your dog :(
Sorry bout doggie ❤
Aww I am so sorry. I remember by the time I got to Queens from Manhattan, my cat was overheating, panting like a dog.
I'm so sorry to hear about your dog. My heart goes out to you.
I'll always remember that day fondly. I was the admin of our companies servers and all of the battery backups were going nuts! First the brownouts, then blackouts, then when power stayed on within that hour all of the battery backups stayed on battery because of an over-voltage (it stayed over 130 volts for almost an hour!). Confusion set in since I had power, but they refused to switch back to online power and the batteries would not recharge. On a whim I turned on the fans on all three AC units, along with all the lights since everyone had gone home by then. That managed to get the power draw down below 130 so the battery backups went off battery and back on-line. From that point all I had to do was monitor the voltages until they stabilized below 130 volts and I could shut off the lights and fans on the AC units.
Thanks to your video I now know what was causing the over-voltage issue, and now most newer UPS are built so they won't go completely onto battery when over-voltage issues occur.
Boost buck ups are the best.
But they all have limits on what they can do.
That’s a cool story! I have worked in a datacenter however thankfully haven’t experienced anything like this. Good job using the AC to drop the line voltage - it doesn’t take much current flow to drop 5-10v from a building supply.
All I remember is that it took me 3 hours to get home a couple miles away.
Road rage 😡
All this because IT guys failed to or refused to notify anyone of a problem and led to things escalating.
I remember this. Me and my friends were swimming when the pool tank shut off... we went home and realized we had lost power. My house didn't have any power for 4 days. We lost everything in the fridge and had to use flashlights to go about the house. It was dark everywhere and we couldn't go anywhere. It was a crazy few days... I was 11 going on 12 when it happened. I'm now 31. Amazing how much time has passed since then. Thanks for this
I had just got done swimming. Was down in the Detroit area visiting my aunt. Same age. Weird no? Lol
@@napalmstickylikeglue I was 9 also in the Detroit area. My brother and I were watching TV and I thought he was turning the TV on and off like 4 or 5 times before going completely dark.
This is fascinating to me. That spring, I had began my public safety career as a police officer at the Cedar Point Police Department in Sandusky, OH. For those unfamiliar, Cedar Point is a MASSIVE amusement park about an hour east of Toledo right on Lake Erie. It was an incredibly busy day that day, right in the middle of the peak season, and the park was nearly at capacity. When the outage hit, we obviously had to shut everything down and close the park. You want to talk about insanity...try getting 50k upset people out of a 350 acre area located on a peninsula, with only one causeway going in and out. I got a lot of opportunities to work on my people skills that day, haha. I never knew the details of how that blackout happened until now. Thanks Grady!
Holy shit. I live in Cleveland so I know exactly what Cedar Point is......and I never even considered the people there on a hot summer day, when suddenly all power completely shuts down. I guess I always assumed they had backup generators for safety. Maybe not to run the park, but enough that people don't get stuck hundreds of feet in the air on something like a ferris wheel that isn't coming back down.
Or imagine being on a roller coaster, going up, where it's slowly inching it's way up.......and then it just stops.
Luckily the outage happened right around 4pm, so it was still light outside. Imagine if the outage had happened at 10pm.....
I've visited Cedar Point several times. That must have been a mess getting people off of rides is there any backup power generation at Cedar Point?
@@fitter70 I can't speak for their current setup as this was nearly 20 years ago, but back then there were generators that provided backup power for the hotels and emergency services facilities/comms, and just enough juice to operate the ride control systems and bring everything safely to a halt. We couldn't keep them running normally on backup power though. Most of the rides need an enormous amount of power to run properly, far more than a generator could provide. For example, the Top Thrill Dragster had just opened up that year, and that ride alone took an absolutely INSANE amount of electrical power to generate the hydraulic pressure needed to launch the trains. An impressive sight, honestly...but it absolutely required main-line high voltage to work. There certainly are generators big enough to put out that kind of power, but the size and cost just weren't worth the investment considering a grid failure like this one wasn't even thought to be possible before that incident. Much cheaper and safer to just shut it down until power was restored. You're correct though, it was a mess. Luckily the park engineering team is very good at their jobs and they had everyone down off of the rides within an hour or so. Most of the rides (especially the coasters) used pneumatic brakes, so there was plenty of air stored up in the tanks. The operators allowed the trains to coast in, and then manually stopped them to get everyone off safely without putting them in danger. All things considered, it was a pretty successful operation.
Thanks for the response 👍
I was at the park that day. A park police officer told me it was a terrorist attack in New York. I’ll never forget the chaos of leaving Sandusky for home.
My friends in Toronto recounted stories of how traffic during rush-hour became total chaos. There were several people standing in the middle of intersections marshalling traffic for hours on end. I've always wondered who relieved these samaritans when they became tired and needed to go home.
I encountered one of these Samaritan’s at the intersection of Bayview & Steeles - 2 lanes + left turn lane in each direction.
This guy demonstrated the most incredible skill in effectively directing the traffic. Although cars stretched forever behind the intersection (not much different from a normal Toronto rush hour), I think we were all grateful for this guy’s assistance. Everyone followed his direction with no one trying to cheat.
"Look for the helpers" - Mr Roger's mom.
P
Driving down the gardener that night, seeing T.O completely dark was creepy weird!! But a lot of fun at the same time!
Please make a video about the process of bringing back power after a blackout. How do you bring power plants online again without them tripping again immediately? How are areas sectioned off to ensure load is consistent with supply as it is being brought online again? How are different "islands" reconnected between them ensuring frequency and phase sincronization after a blackout?
I watched the system at a switch station here.
In a event of a feeder overload the system dumped all the circuits leaving the switch station.
When they had three of the feeders back online they just powered on each outgoing circuit one by one with a delay of up to 10 minutes.
For The backbone grid they would also have the other issue of matching generation with demand.
+
You do in stages creating islands of generation and consumption.
Then slow sync and join in more bits of the grid back together. Old school method of sync is to use. Light bulb for connecting each phase and then slow/speed up the generator till the light bulb goes out(no voltage difference means no difference between the phase)
Modern day uses GPS timing signals that aliign the phases.
There is a very interesting article I read where the DoE setup an isolated grid and had to restart it from scratch without using any Diesel generators; only using the solar setup. It was a great article. I can't find the original source.
@@stephenrowley4171 synchroscopes are pretty neat bit of engineering
i was born in 2001, so i was only 2 years old (i.e. too young to remember) when all of this was going on. but my mum told stories of how the whole street stepped up and helped everyone out - there was a small (probably 20-30 units) apartment building on the street with mostly disabled and elderly people, i.e. the people most adversely-affected by the power loss. some people went around, handing out burgers and hotdogs cooked on their propane BBQs, my mum had a bunch of ice cream in the freezer that was now melting so she held me in one arm, grabbed the ice cream, bowls, and spoons, put it all into a little buggy cart thing (not like a shopping cart, but a wagon), and walked up and down the block handing out bowls of half-melted ice cream.
there was this older fellow who had a couple generators (just little things, the sort of generator you'd have with you when camping) and a few people dragged their freezers over (helped out by someone's dolley), apparently the entire street ate a big ol BBQ with ice cream bars, ice-cold beer, etc etc in front of this guy's place. even the old and disabled ppl from the apartment came out!
my mum still has a picture of me, enjoying a 'magic' show put on by one of the ppl from that apartment!
it's really interesting how communities, people who really don't know eachother very well, can band together in tough times. sure, it would've been easier to just ignore my mum and i, she's never been terribly healthy, nevermind those poor ppl in the apartment block - but how everyone pulled together to maintain civility, community, and got through the crisis together? gives you just a lil bit of faith in humanity, don't it?
I recently retired from the FERC and one thing that didn't get noticed is that typical fossil plants do not have black start capability and therefore cannot get the grid back up on their own (generally speaking). That means they need power fed into them before they can power up and generate back onto the grid. However, our hydro units in upstate NY, specifically the Raquette River projects do have that capability and were one of the main components that brought the grid back up. I know hydro has gotten a bad name environmentally but there are certain attributes such as black start, spinning reserve and voltage stabilization that should be worth more than acknowledged.
I've heard an anecdotal story wherein one ingenious set of engineers at a coal-fired plant somewhere in NY used their car batteries, wired together, to run the coal supply conveyor belt (ordinarily powered from the grid) long enough to get the system re-lit. Simple AC motors can be run on DC, and vice versa. AC motors will always spin the correct direction on DC as long as the VA input is enough, DC motors on AC have a 50/50 change of running in reverse (if they run at all). Them getting running in turn helped other plants restart.
This is not correct.
Some universal motors with brushes can run on ac or dc.
But normal industrial Ac squirrel cage asynchronous motors will not run in dc!
@@xheralt. You cannot run a regular AC induction or synchronous motor on DC. It won’t spin. What you’re probably referring to with the car batteries is using them to “flash the field.” An AC synchronous motor uses an active magnet field (usually the rotating part) to excite (generate voltage) across the stator. That field gets established using a DC source when the generator is getting brought online before the breaker is closed. The generator has a device called a voltage regulator that monitors AC voltage and current output to determine how much DC field energy to put on the field rotor. More field current means a greater flux across the stator. This can increase or decrease either the voltage (if the generator isn’t online but spinning up) or reactive power (VARs) if the machine is online connected to the grid.
There is a rough direct relationship between voltage and vars based on the electromagnetic inductance of the generator and the electrical system it’s connected to. The greater the DC field, the greater the VARS, which can make the voltage go up (and vice versa).
The voltage regulator only has AC voltage and current to determine how much DC field energy to put to the rotor. If the machine is spinning but offline, it only has voltage. The AC voltage doesn’t appear on its own because until there is DC energy introduced to the field, there is no voltage.
A lot of older machines used something called a “pilot exciter” to initially excite the field. Think of jump starting your car. This is very similar to what a pilot exciter does and lights the DC “pilot light.” This is called “flashing the field” by giving that initial bump. A lot of plants did not maintain their pilot exciters over the years. So it is out of service a lot of times. So what they can do is bring a car battery in during start up and put the DC voltage across the collector or slip rings (the apparatus that transfers the DC energy from the main exciter to the DC field) on the DC field. This makes the AC voltage immediately start across the generator winding and the main exciter and voltage regulator can take off for the remainder of startup and while the machine is online.
The battery is only needed for a brief moment on startup and the machine can do the rest.
Modern voltage regulators with power electronics don’t have this problem as the solid state voltage regulator and exciter can do this on its own with no battery needed.
I've worked at the Indian River plant in Delaware and currently at Vienna plant in Maryland. Both of these plants have black start capability with an auxiliary jet that ties into the grid and we can get our start up power from it. They're also used as a kick in the pants for when demand is up but starting the main unit isn't neccessary.
All Nuke plants have black start capability since they all have multiple, large, emergency generators.
The sound of nearly the entire neighborhood cheering when the lights came back on, man you had to be there to hear it. A timeless sound that will forever be etched in my head. I remember how it all began and how it ended. We definitely were all still pretty shook thinking the worse because of 9/11.
My parents scared the hell out of me saying ppl were likely going to loot houses (we lived in the suburbs in bumblef*ck nowhere) I as a dumb 12 year old, imagined it all going down. We all slept in one room burning UP that summer, lol. Anyway! Very interesting information here! I wonder if anyone lost their job. Salute to those who put their back into fixing this.
Yeah people definitely thought it was a terror attack at first. I was 17 when it happened and my mom called me a few minutes after the power went out and told me she that something major was happening and it might be a terror attack. She was working at LaSalle Bank headquarters (Detroit Michigan area) and she said all of the branches were calling into headquarters to report power outages and many of those branches were hundreds of miles away.
@@Chris-hn4lp that’s actually wild because that’s where I was when this happened. Well, further north in Mt Clemens, MI. Small world, nonetheless! But yeah, I definitely heard people speculating the worse too. I had just came back from an acting class I was attending back then. I just remember going to a Walmart and it being pitch black inside. Interesting times.
@@spadeyspacely lol I was in taylor michigan in Kmart when the lights went out I was honestly so surprised alot of looting did not take place
I remember the BBQ the neighborhood had! Good times in the city! (Detroit)
@@SirPrincemarsallis
In Manhattan too-not a BBQ, but the neighbors in my apartment building all brought whatever they thought might spoil without refrigeration down to the lobby, and we had a potluck.
The pizzerias stayed open by candlelight (electricity was out, but gas for the pizza ovens wasn't) and so did about half the bars. Some people took it upon themselves to stand the middle of Broadway and direct traffic.
I love this city.
I was not even 10 at the time. My dad had just walked in the door with 2 weeks worth of groceries when it tripped. I can still remember his “You have GOT to be kidding me!” As we hoped nothing would go to waste.
@carolinepahlhow is the documentary coming along?
Lmfao
Lol something similar just happened to me last tuesday. Went and go groceries then soon as I get home the power went out for the next 14 hours.
How Long did the Blackout Last?
@@danieldanielson2650 For us it was about 24 hours, but there were people around us that were out for days. We also had family that got power back hours before we did so we took our food to their place to save what we could.
This blackout was my first day as a volunteer firefighter. Now I have the privilege of being a professional firefighter and had to deal with the Blizzard and blackout in Buffalo NY. Great subject!
Thanks for bringing back the memory. I was working at a printing shop in Toronto and the press ground to a halt, we had no idea what happened. Thankfully radio stations had backup power and within 30 minutes or so it became clear that it's not just a squirrel that got into a transformer. Getting home was brutal, the traffic was nuts. But the evening/night was cool, with zero light pollution and clear sky, it was an amazing full moon casting shadows at midnight, and people walking around with flashlights. We got power back the next day and life resumed...
Much worse power outage came in 2013 ice storm, we had no power for 7 days during really cold temps in the winter just before Christmas.
The only good thing of this blackout is to see all that light pollution gone.
We had no power here in Windsor for like 4 days
@@maximilian19931 that happened to my city last summer after the derecho storm. It was really cool to see the stars in the city
In my small town, I was having a nap after school, and woke up about 10 minutes after the outage started (maybe because of the eerie silence?)
I turned on a portable radio like usual to listen for news about the blackout, and heard the entire country was dark. (Thankfully, an exaggeration.)
We ended up walking across town to stay with a friend, because safety in numbers, I guess. There was a single store still open, because they were the only ones who could manage sales using pen, paper, and solar/battery calculator. Naturally there was a line to buy bottled water (batteries were sold out almost immediately) leading halfway down the street.
We passed the time talking, drinking, not using our battery powered radios and video games because who knows how long this will last? We might need those batteries for more important things... At this time we still had no idea what caused the outage or even how far it extended, so we didn't know if it was a system failure, terrorist attack (9/11 was still fresh in everyone's mind), natural disaster, etc, that could take weeks to repair.
At one point we watched someone break into the liquor store across the street, but we couldn't do anything about it. Not even call police, since all the phone lines had gone down by then; although they had battery backup, The systems had been overloaded and the batteries had failed after several hours.
The lights came back on at about 4AM, and things went right back to normal...
@@maximilian19931 That and the free food downtown restaurants and sidewalk cafés were giving to diners. But seeing the Milky Way from my urban back yard was the sweetest icing on the cake. If annoying neighbours weren't walking around with flashlights thereby wrecking my night vision, the experience would have been *perfect.*
Also a great place to share "where were you?" stories! I was in a hospital a day after jaw reconstruction surgery at 17. They had just moved me to a two-person room for further recovery when the blackout happened and they told me to "not get comfortable" and that I might be going home that day because of resources. Sure enough, my parents arrived two hours later and the hospital discharged me. I remember the main highways were so backed up so my parents drove the back roads (I had to go out of town for my surgery) half hour to get home. There was no ice in the stores but one of parent's neighbours gave us some so i could keep icing my swollen jaw. I was still coming off the morphine IV so I think I slept through the most of the blackout; when I woke, everything was working again. Hospital was in Hamilton; I lived in Brantford, Ontario, Canada
i was 9 at the time and at walmart with my mom. the power went just as she was about to cash out. it was a real weird day or two without power and having to listen to the car radio for any updates. atleast we had a pool at the time to beat the heat
I was 2 months old and probably at home or something idk I'd have to ask my mom lmao. I live in central Ohio btw.
Drove home from Toronto that night back to Hamilton, my buddies and I caught wind of Grimsby still having power, we stocked up on ice and beer, filled the bathtub up and partied lol great memories for sure!
I'm sure there's a jaw-dropping pun in here somewhere but I don't have the wattage to make it. :P
I was a young operator (3 years experience) at a 700 mw coal fired plant in Alabama. I remember the voltage swings that day and our dispatcher frantically telling us to maintain voltage schedule and field breakers in automatic. I never saw anything before or since. I had a high "pucker factor" for a little while that is for sure. To affect us this far south tells me it could have been a lot worse.
Interesting
Realistically, how bad could it have gotten, and what consequences are we talking about?
Cascade through a much larger area. Almost happened in Australia when some plant owners thought it a bright idea to operate at constant output with no proportional frequency response. The regulator learned their lesson from that one.
Pucker factor. I love it 😂😂😂😂
Moral of this story, every time you flip the light switch on and the lightbulb illuminates, appreciate the hundreds of engineers and operators that make that little thing possible!
I was 8 years old when this happened and thought it was the coolest thing ever.
It's crazy I've been ignorant of the fact so much work has been put into warming my home, keeping the water flowing and much more.
Thank you to all those hard working people who created this system and keep it running!
Heyyy, I was 8 when that happened, too!
Or well, 4 days away from 8. I was at the Bronx zoo for my birthday when it hit. Weird day.
Damn we all born in 95 too? xD I was also eight, but I was in FL when it happened, never knew about something like this happened.
Warming your home? It was summer.
Isn't it weird how it was so exciting to us kids? We just remember hanging outside and having barbeques. I remember being excited I found batteries for my radio so we could put on the news.
@@FaBB10_FS24 Just a generalization of everything they do around the clock. Not just the black out :p I'm from Canada so it gets very cold! LOL
For additional context to anyone who isn’t hip to this story. The companies who were responsible to manage things (FirstEnergy and others) also had plenty of shady activities going on internally. So as you follow along with this wonderful walkthrough, keep in the back of your head the vision of your neighbor who has that old lawnmower that barely works. But it works so he won’t replace it even though he spends 5 hours every time just to get it running.
Apply that exact mental visual to the scale of a company that is responsible for managing the grids as you listen to him talk about how these tiny pockets of problematic issues slowly domino effect
You mean to tell me First Energy Ohio is a corrupt unreliable organization *gasp* no way 😮 thats just silly!
This was honestly the first nail in the coffin. There was a significant amount of outdated infrastructure that had just been ignored up until this point
I remember this blackout in Jersey City and NYC. Surprisingly, the blackout took so many people by surprise that there were zero reports of crime because everything came to a standstill. Instead, everyone tried to help each other when everyone tried to figure out what happened.
I was in connecticut. It somewhat felt the same because we were all still on edge from 9/11.
@@Wendifur_ yeah I definitely remember that. I feel like that mentality kept everyone unified and willing to help one another.
One thing I also remember was my dad sleeping on a chair in the garage with a baseball bat because of ALL the days a long blackout would happen had to be while the automatic garage was still open 😄
If this were to happen today I don't think people would react the same way this time around sadly.
@@Mortal-Monk the 13% will robb everything
I live in Jersey City
Is absolutely amazing that something as small as electricity in Cleveland pretty much shut down the entire northeast. Wow. Thanks for this breakdown
I’m from Louisiana but was visiting NY in August of 2003 with a friend..was my last full day there before a long drive back. We were staying in rock away beach but had just finished visiting the Met and we’re on the subway when this happened. We had to walk 20 miles to get back. Seeing millions of people line the street, seeing the dark silhouette of Manhattan as the sun set while we were crossing into queens and then seeing the stars come out over New York were images I will never forget. The people who had made it home (this sss 2 years after 9/11 as well) were literally handing out food and water to those who still had to walk. It was incredible. We left the next day and it took us 8 hours to get out of the city via the belt Parkway. Absolutely nuts.
I'm a power systems protection engineer and confirm that this is a very informative and accurate representation of how protection would react in this situation. Really pleased that he got distance protection correct. I wonder whether Load Blinders on the distance relay characteristics would have prevented the cascade or would the reduction in frequency and voltage have led to the cascade being inevitable.
I was at a nuclear plant in upstate NY at the time. Sometimes it wasn't even the protection relaying. We started experiencing wider and wider frequency fluctuations as loads/gen units were tripping and lines tripped and reclosed. Afterwards it was determined that even though our main turbine governor wasn't in 'speed control mode' (nuc plants aren't used as 'regulating units' ) it just couldn't keep up with the wide swings, hydraulic pressure was lost and we went down.
What about load shedding as well? Or was beyond the capability of load shedding?
@@mikefochtman7164 makes sense. I'm aware Nuclear is for base load only as takes a large amount of time to ramp up and down. In the UK we use mostly oil, gas and coal for reacting to increases/decreases in load. Basically, the gist I get is that this was a ticking time bomb with only one possible outcome. I am impressed that the grid actually coped with so many line and generator losses before the final collapse. From a planning perspective I only planned for an N-1 event. I think that the grid in question was at N-4 before the final collapse. One generator + three lines down.
@@stephenrowley4171 I don't know that much about this particular incident, however the grid was much larger than the area lost. I think that effectively this was load shedding to try and save the rest of the grid and prevent a black start event whether it was intentional or not. Load shedding is very hard to manage when not initiated by control. As I understand it from Mike above lines were autoreclosing which would cause huge power swings and make it very hard to try and balance the load & generation. Also bulk supply points usually have underfrequency and undervoltage protection to 'cut off an arm to save the body.' This is a kind of automated load shedding to try and save the grid from collapsing.
@@mikefochtman7164 what happens when a nuke plant has to shut down? Does it take a while or can you just SCRAM it? Do you have a bunch of excess heat to deal with? Is it difficult to restart?
Nuclear is really interesting to me, but the only design I know much about is the RBMK for obvious reasons...
I remember this day very very well and I was so proud of my friends group. We had a contingency plan that if anything ever happened to the point where it was a panicky situation that required everybody to meet up at one place they would all meet at my place. I want to go pick up my friend Lindsey because she was the only one without a car and when I came home there was literally about 20 of them just waiting outside. They had generators and gas and propane generators and they brought their grills and their coolers and we started just cooking up all the food that was going to be going bad eventually and we basically had a three day long party. When we were on the third day we started siphoning gas from our cars so that we can continue to power our generators so that we can keep people's medications cold that needed it.
And ever since then we all now use propane generators and we usually have around six propane tanks, each, at all times.
That’s awesome, thanks for inspiring me to start a plan!! Lol
good fake story bro.... the power went out during noon and came back at 4 am
@@bonefishgrill6382 it lasted between 2 hours to 4 days depending on the area
Thats way better than the vandalism and robbing who happening in that time
@@bonefishgrill6382 imagine calling someone out then being wrong 🤣🤣
I was in my twenties, living downtown Toronto when this happened. I remember walking home from what I thought was a day off of work that turned into a week without power. It was surreal how quiet such a large city became, I remember some really hot nights while trying to sleep, the only source of light was the moon. It was pretty cool
Great summary. I was living and working in Connecticut that day. We lost power, and I had to help my co-worker manhandle a heavy centrifuge over to an outlet that was on emergency power feed so she could remove her samples rather than have her study destroyed by ambient temperature overnight. On the way home, I stopped off at a Mom and Pop grocery to buy a couple tins of cat food - the store was using hand calculators to sell food - and for their meats in the coolers, they were selling huge amounts to anyone who wanted for a dollar for what people could carry. I was between grills at that time, so my dinner was yogurt and the last of the backyard raspberry crop.
It's strange I remember living through this in Mississauga, Ontario and I remember the power being out for several days with rolling blackouts, but Wikipedia says the power was back within a day. Does anyone else remember it taking longer? Another thing is New York Mayor Michael Bloomberg and state governor George Pataki immediately blamed Canada for the whole thing.
Blaming Canada? That's some Red Forman governor if I've ever seen one.
I remember being at my grandparents' place in Perth, ON, and I recall the power being out for a few days
I was wondering the same thing while watching this. When it said the power was back on quickly, that does not line up with my experiences in Michigan. Power was out for almost 3 days if memory serves me.
I was just coming into Saul Saint Marie when the power went out. We stopped at a gas station and paid cash for peanut butter and bread in case it was a long outage. We had PB sandwiches for supper after the owners showed us a hotel room by flashlight that evening. The next morning we drove to Sudbury where the power was on and we bought gas.
I was in Brantford, ON - we had power back that evening, lost it again the next morning, and it came on for good that second evening (~28 hours after the outage).
But there was also about a week of dramatically lowered consumption, because the nukes were offline. I don't recall any rolling blackouts, but there was definitely a period of aggressive power conservation, with a lot of big factories and other heavy power users closing down to conserve power for necessities. That might be what you're remembering.
I remember this… no warning, no reason given, the power just went out. You expect power outages in storms and the like. But on a gorgeous New England day, in the late afternoon, all the sudden… nothing. I remember it being the first time that I really thought about disaster preparation. The confusion of the event was interesting. The radio was the hero of that day! Great episode, dude!
Since when power outages are forewarned? It is out, and nobody can predict it early enough to warn someone. It is out, and it is no reason to expect anything bad from it.
All the air conditioning was the reason for the blackout.
ALL THAT AIR CONDITIONING.
A very good reason for keeping radio going. Today's reliance on the internet and most TVs needing mains power, means that radio is absolutely critical in blackouts. With developments like RUclips and Spotify, and the internet in general, radio is increasingly becoming redundant. Most independent and local stations here in the UK, for instance, are struggling financially and many have given up their medium wave frequencies. And young people almost entirely don't listen to radio any more. Left to itself, radio could easily fade into one of yesterday's technologies - but this cannot be allowed to happen.
@@yaroslavpanych2067 well, usually you can expect that the power is likely to go out because it's storming or such... You don't expect them on a pleasant sunny day. Of course air conditioning and car accidents are still a thing, so you can never truly predict them.
@@maximilian19931 As someone who grew up with a single window airconditioner for the entire 1000 sq ft house, I can tell you that people won't be happy without ALL THAT AIR CONDITIONING. Not just be unhappy, some people will die because over the last 50 years, we've built houses and apartments around ALL THAT AIR CONDITIONING and now we have a lot shelter that relies on it!
Big failures like this are most often a perfect storm of unlikely events happening all together. It's really fascinating.
I remember this, power was out in Manitoba for a couple of hours as well. Only time I have ever seen a true brownout, where the voltage drooped so low the incandescent lamps burned a deep orange for a short while before the grid went offline. Hydro here had our grid isolated and restarted within a couple hours and we saw just how big it was when the news came on tv later that night. What we thought was just a minor local outage ended up stretching across Ontario, into Quebec and sweeping south into the US and across the eastern seaboard and lasted 24 hours or more in some places.
Very little effect in Quebec, and brief at that.
Now that I think about it I remember a time years ago where the lights were super dim. I know it was during the summer, so it might have been this blackout.
Quebec wasn't affected. I know, I'm from Montreal. I watch everything on TV with my AC on...
I'm from the West Coast, but I remember this blackout very well. I had won a trip to Boston to see the Oz Fest and was on a plane on final approach to Logan Int. Our plane had to hold for maybe 30 mins. Boston still had full power, so we really didn't know what was happening until later drinking at a bar.
I was just starting as an electrician for a New England utility in '03. It's amazing to me how personnel have been eliminated even since then. If we had a significant blackout again like we did in 2003, it would take significantly longer because we do not have the boots on the ground like we used to. Automation is fantastic when it works but you need skilled, knowledgeable people at the generators and at the substations to coordinate things. In 2003 we had over 20 electricians in my shop, now we have less than half. Reliance on SCADA is a mistake that will turn to bite us one day.
Grady, I really enjoyed your video, a suggestion for the next one, do one on recovery from a blackout called Black start. That should really help people understand the complexity and coordination required to bring back the grid once it is collapsed.
Utilities that prioritize profit and stock price over reliability are the Achilles heel of the grid. You need skilled people. When things go sideways, relays and computers are not going to help you.
It's almost as if capitalism shouldn't be the driving force behind critical infrastructure.
I wonder what you think about the hacking risk that apperantly many were saying existed in the grid, would it that vulnerable to recreate this but way wider and far damaging...
I'm presuming the automatons would make it way worse.
Google Ukraine power grid hack.
Ppl used to say the exact same thing before the 2003 blackout happend. And will continue to do so till after the AIs take over everything. Maybe someday they will be right but i hope not.
@@someonespotatohmm9513 They're _already_ right.
I remember that day well. I live in the Rochester, NY area and power went out in most of the area. One generating plant that was located on the shore of Lake Ontario (known as Russell Station) was able to separate from the grid and stayed online. Since that plant had direct transmission lines running to the county's water pumping station and sewage treatment plant those were able to remain in operation during the outage. A few towns and villages that also had direct lines from that plant also still had power.
Russell Station was seen as a hero during that event but due to it being a coal fired power plant it was closed a few years later.
I’m sure they didn’t replace it with anything. Wait until next time. We don’t stand a chance
I live in Eastlake, and know full well about our contribution to this event. I hadn't realized the estimated deaths that were caused by, either directly or indirectly, this major outage. It's always a bit of a joke whenever the major blackout comes up, though it rarely does. Knowing just how devastating it truly was, however, is crazy. Especially with hindsight. Thanks for covering this story.
Lol, I remember the back and forth between canada about which side was to blame
I worked as a Control Room operator for 36 years in a Fossil fired plant on LI ( 4 Units 375MW each) I had just left a day shift that day when this happened. In the ensuing weeks I read the Interim Report from the Federal Government on the cause(s) of the blackout. This video is an accurate account of what went on. Good job.
So it was your fault
@@youtubeuser206 No, not at all. I guess you have not yet watched the video in full. It's worth your while. The presenter did a good job on it
which plant? I live on LI
@@jamescannone1425 Northport Power Station
@@kurtbjem777 he's kidding, never mind
As someone in the business, I just want to say all of your information in this video is excellent and spot on! Even showing the old regions, nice touch. It must of taken a while to research/ learn the lingo for this video!
my dad used to be an elected official and would tell me about working with engineers of all kinds for setting up, maintaining, and repairing all types of things. he respected this type of work but always had troubles understanding it. your videos are super important for the layman's knowledge of systems we all rely on every day. thank you for making this readily available and easy to digest
I just love how you, as an engineer yourself, defines the problem and gives a simple to understand solution to negate any possible reoccurring problems like this from happening again. Good job!
I was working as a welder at the time it happened. I was doing spray-transfer on some heavy steel plate when everything went dark. I lifted my welding shield, and still couldn't see, I briefly panicked, because I briefly thought I'd spontaneously gone blind...
I’m a control room operator at a plant in the northeast. Until I started there, I had no idea about, nor really gave it a thought past, how I flick the light switch & the light comes on. It still amazes me at how many people involved & tools/machines there are that give me the ability to flick that switch & light my room.
These kind of events are happening all the time and we don’t even notice. I’m fascinated how reliable the power grids are made, and how resilient they are. It took so many consecutive failures to actually end up in a blackout. Thanks for this video this is very interesting and well explained as usual!
Actually it was only one failure, providing power to Cleveland with their high demand. Stopping this wiuld have prevented the full blackout and only pull Cleveland off grid, keeping the rest on grid.
Disconnect high consumers and save the rest.
Money in the pocket drives everthing. Top money thieves will always have back up as they can. The rest of society who pays for everthing suffer. America and its fat greed.
Seems like it mainly boils down to a computer glitch. If they had worked correctly and alerted operators about the faults, they probably could have dealt with them properly and prevented the failures from spreading beyond the first one.
@@renakunisaki Or if the power company actually cut the trees that kept taking down their lines. Preventative maintenance costs money though so it's better to let things collapse and fall apart.
@@renakunisaki Like most big problems, it comes down to a perfect storm of many failures at the same time.
I was 23 years old when this happened. I remember how the cascade failure stopped right at grid disconnect between Minnesota and the Canadien side. I remember for the first time being aware that a problem on the east coast could affect me in the upper midwest and how bad the grid system is. I think this saying puts it best “We (America) are a first world nation with a third world power grid.”
That's amazing! Even ELECTRICITY doesn't want to go to Cleveland! thanks, Grady
If it did, it would want to immediately flee
Jesus Christ, that's savage. But if I interpret this comment correctly, so is Cleveland.
Underrated comment
I ran an observatory here in New Zealand and we used to run dark sky night tours with highly powerful telescopes.
We had a New York couple who loudly proclaimed "The last time we saw stars was 16 years ago (2003). We didnt know what they were at first!" and they shared a lot of stories about being mesmerized at the sky back then, as well as telling that infamous story that their friend rung 911 and told the operator that UFOs were coming (only because they have never seen that many stars)
That was such an amazing time as a teenager. It really brought the community together. My parents had a really good generator and every morning would take the coffee maker out front and make coffees for anyone who wanted one. I spent a lot of time outside with friends instead of playing video games with them.
I remember this, I was working 3rd shift stock at Officemax at the time in Michigan, our supervisor had the nerve to tell us we couldn't go home, and had to continue stocking pens and pencils and paperclips in the dark, with flashlights. That was when I walked out, hit the road and went north on I-75 until I finally saw lights on billboards and luckily my dad lived in a powered area, I never went back to that job or city since.
I'm liking this "when infrastructure fails" series! how about Flint, Michigan's mismanagement of their water supply piping next?
That is more a political story than an Engineering story, but still might be interesting. An preventable tragedy that will have affects for decades, and maybe generations.
@@stewartthompson72 it would highlight another critical dependency our modern way of life has: the pipes buried in the ground. Because they’re buried, we tend to easily forget about them.
Flint did not fail the water supply.
State of Michigan politicians thought Flint didn't deserve clean water because the people of Flint didn't deserve it.
@@vickiegrant3325 and banks were betting. on municipal bonds that the gov was gonna save money and boost bond value
I remember this. It was actually pretty great.
EVERYONE was out grilling for days and there was a non-stop road hockey tournament going on in basically every parking lot ♥️
Met so many neighbours for the first time ever.
awesome
I remember this happening. I was at work in CT. and everyone was confused and progressively got more and more freaked out the more we learned how far away the outage was, including Canada. Only two years after 9/11, you know where everyone's head was going. It got super eery trying to go home with zero power. The streets were a mess.
I was in Newport Michigan when this happened in view of the nuclear plant. I couldn't understand why we didn't have power despite there being a nuke plant within looking distance.
I remember all of the theories people were coming up with. At first the obvious it was terrorism with 9/11 less than 2 years prior. The second was massive fires in New York. Third was we were bring invaded. Fourth the government purposely shut us down because we were using too much power.
Major props for the work you put into making such high quality voiceovers. I’d imagine it involves numerous takes and patience.
As a system protection engineer, this is a great video! I was only in elementary school when this happened, but we still talk about it at least a couple of times a year during different conferences and seminars. You do an amazing job of taking hundreds of pages of documentation and translating into an interesting and digestible video.
Having worked in the power industries for a long time, I can tell you that the power grid is the single most important utility in the country. As you touched on, without power, traffic becomes gridlocked, water systems fail and on and on. It is a national security risk and should be on a level of importance with the military. Consider that if Bin Laden had really been trying to cripple us, he would have flown the airliners into the Hoover Dam and/or the massive power generation facilities along the Niagara river. Taking those facilities out would have crippled us for years, not a few weeks or months. Our grid is at risk from weather, terrorist attack, lack of maintenance and most importantly, political hubris. How many 2003s or Texas outages must we have before we wake up?
Lets be thankful terrorists are too dumb to realize that coordinated stiles on dams to stack flood surge (or possibly just the top dam breaks the rest, but personally i wouldn't take the risk it doesn't) is the most devastating thing they can do.
Countless cities are down stream of major dams.
Hell they could literally just shot a bunch of the glass & porcelain insulators on major transmission lines simultaneously and take the grid down. The grid isn't exactly guarded, and this says nothing of the cyber attack risk from nations like Russia or China.
@@jasonreed7522 I'd say they are not dumb, just like big flashy stuff... what makes the eyes pop for the target audience.
Hostage a plane? sure, it get the news but ya dead or placed on trial. but big bangs? now that what we are talking about; let the surprise, anger of the target audience flow... and then the government have to fly those hellfires and kill some baddies. But what if, the civillians are within the OA? Hey... it's 2 in 1 now... we get another angry to-be jihadis... while we get rid of the current top wanted... even better if it's the wrong Tango.
@@jasonreed7522 A report came out years ago that names 4 transformers that could be shot that would take the eastern interconnection down. Granted it assumed no operator intervention. What really made people mad was as I said....it NAMED the transformers. In reality it would take more than 4 to get a collapse, but it still could be done. A substation in CA was shot up years back and caused an issue. Even more fun, not like there are back ups for the big transformers just sitting in a shed.
@@andrewt.5567 The report NAMED the critical targets, just how dumb do you have to be to do that. You can say something like just 4 transformers could be attacked and bring down NYC without opperator intervention and get just as effect a message, or have the names only in the report for people with clearance.
I choose to measure my words and not name dams directly above major cities, even if it takes less then 5 min of effort to identify one that could hurt a lot of people. You don't do the bad guys job for them. (Someone tell our media that)
@@cmmartti Correct. Consider the 'Dam Busters' raid on the German dams in WW2 - they had to go to great lengths to get specially designed bombs to detonate deep under water on the upstream side of the dams, and it took several bombs to break each dam. (The water acts to confine the explosion and massively increases the effectiveness of the bomb). Flying a 767 full of fuel into the Hoover Dam is unlikely to scratch the concrete much, in fact it would be hard to think of a structure less likely to be inconvenienced by having an aircraft hit it.
Crashing on the switchyard (or a substation) would likely do far more damage, though collateral damage (other than the consequent power cut) would not be that extensive.
I remember when this happened. We were leaving town for an important meeting in Toronto, and since both the Tim Hortons and the Gas station in town both had emergency generators, people were stopping by our township for them.
However, the lineup at the Tim Hortons was far longer than the lineup for the gas station!
Only in Canada.
People always seem to get hungry when things like this happen
I had to check if you were Canadian because your pronunciation of Toronto was perfect!
Love the video, it earned my subscription.
2003 was the year of a big blackout in Europe too, it happened on 28th September 2003 in Italy. A 380 kV power line located in Switzerland was overloaded, the conductors sag and one of them touched a tree, creating a short circuit to ground and causing the line to trip. That line, which still exists today and is part of the swiss national electric backbone, starts from a big substation in the inner Switzerland near the city of Lucern and ends in a substation in the southern part of the country, in the canton Ticino. From the same busbar in that substation where the line is connected, another one starts and ends to a substation in Italy, near the swiss border. That line is being used to export energy from Switzerland to Italy, and the italian substation where it ends is located at the west of the canton Ticino's border.
The interruption of the swiss trunk shut off the line going to Italy, and all the load that was on that line was took over by another 380 kV international export line, which starts in the canton Grigioni and again ends in a substation in Italy, located at the east of the canton Ticino's border. Those two lines in fact are part of a loop which starts from the big substation in the inner Switzerland, crosses a substation in Ticino, goes to Italy by crossing the canton Ticino's west border, in Italy crosses several substations located at the north of Milan until it reaches the other italian substation located near the canton Ticino's east border, then re-enters back in Switzerland, crosses the substation in the canton Grigioni, then goes to another substation in inner Switzerland and finally goes back to the big substation near the city of Lucern, where the loop starts.
That second line on the east side overloaded too, conductors sag, another tree was touched and the line shut off automatically. At this point, all the export power stream that was going from Switzerland to Italy was missing. Loads redistributed and were took over by a couple of lines from France (the european grid is all interconnected), they overloaded shortly after and gone off-line. All the import load then moved over systems from Slovenia and Austria, which at that time were absolutely undersized to carry such amount of power, and they immediately shut off. Meanwhile sensors at grid control centers all around Europe were detecting heavy load transients, this to give an idea of how big was the thing that was happening in that moment.
At that point all the power which Italy usually imported from the nearby countries was missing. Generators in italian power plants overloaded, slowed down, grid line frequency started to decrease until reaching about 48 Hz, power plants gone off-sync, disconnected from the grid one after another, the whole system collapsed and suddenly the whole Italy was left without electricity supply. It all happened in about 15 minutes, and it took nearly 2 days to restore the power supply in the whole country.
The main cause of this nation-wide blackout was erratic/incomplete communication between each country's grid management companies. Swiss personnel, while unsuccesfully trying to reconnect the faulty line (phase angle too wide at the line's sides), asked their italian counterpart to reduce imports immediately after the first line trip, but they underestimated the amount. On the other side, Italy was importing from Switzerland more energy than the amount that was planned, this led to the first line overload that started it all. And when the second line tripped the cascaded effect begun.
The reports from both Italy and Switzerland are available on the Internet, they're very interesting and fascinating.
I vaguely remember this. Wasn't most of Switzerland's almost completely electrified train system offline for about a day too, stranding a few hundred thousand passengers?
Watching the video I also remembered this incident and thought "well, this might be worth investigating, too". Then I found your reply and was happy to learn more about this. Here in Germany we've heard of the incident but didn't experience any outages due to it. Thank you for summarizing what happend then!
@@uschurch No, at least as far as I can remember that day there weren't particular issues with the railways. Swiss railways company (SBB/CFF/FFS in german, french and italian) runs its own grid, which is a 2-phase (180° phase angle between the two) 16.67 Hz system, with its own power plants and substations. There are some substations where both the regular 50 Hz 3-phase and the railways' 16.67 Hz 2-phase grids are connected, with equipment that act as interface between the two systems, so there are some junctions. But essentially the railways' grid is an independent system. From your name I guess you're from Germany or Austria, if I remember correctly in your country there's a similar configuration as I remember that, when visiting those countries, I clearly saw poles carrying wires in multiple of 2 instead of 3, like it is by us for the railways grid.
Some years later there were a couple of accidents where important parts of the railways had issues with power supply. Maybe you're remembering about these, but the disruptions lasted some hours, not days, and they weren't nation-wide events.
On that day of 2003 instead, here in Ticino, where I live, we were affected too by Italy's power outage for some hours during the night, but we had power restored by the sunrise. I actually remember that I saw all the streetlights still turned on during the late morning and I was wondering why. Then I heard of what happened during the night and I understood, as the timers stopped during the power outage and when power was restored they were all off-time. This happened because the two 380 kV lines coming from inner Switzerland that tripped, carried electricity not only for export to Italy, but to supply our canton too, as they power our local 220 kV grid. When they went off-line canton Ticino was actually nearly completely isolated from the rest of the swiss grid, with the exception of just some secondary and quite old and undersized 220 kV links which failed pretty soon too, and for a certain moment we were actually powered by the italian grid, I think just by the east-side international line (the one which is connected to the substation located in the canton Grigioni that I mentioned in my first comment). Once the italian system gone offline it pulled us down too.
@@DavidSallge You're welcome, it's an event that fascinated me and I like to talk about it. If you search in Google you'll find the reports that describe all that happened that night. The problem is that I don't know if you'll find them in german or english (maybe for the swiss' one there's a german version too), as I found documents only in italian.
I worked maintenance for a company in northern NJ when this happened. I was replacing light bulbs in ceiling fixtures. I was turning the lights on and off, checking as I replaced each. A co-worker came running in screaming and demanding to know what I did because the whole building's power flickered several times (while I had the lights off)and blamed me, I had no idea it was happening until we lost power completely. Co-worker still was blaming me. 🙄 I was having company that night and it was the hottest day and so humid. We sat around sweating our a.. off only to find out 3 hours later that someone's, at the party, house less then a mile away had power and a/c. I often think about that day because although we were physically very uncomfortable, it was a very enjoyable time
As a power systems protection/relaying engineer, I find this quite interesting. This blackout has been documented in many IEEE technical publications. Interesting to see how trees cause ground faults on power networks. 16 lines here tripped on ground fault !!!. I love protection and control , challenging and exciting speciality in electrical engineering. I love the way you speak of zones of protection. Distance protection is the work horse of transmission line protection. I encourage electrical/power systems graduates to go into P&C as protection engineers are always in demand, both in operations and consulting practices
I lived in midtown Manhattan when this happened. Looking back,it was amazing how everyone helped each other. The elevators were not working so we helped the elderly walk up the stairs in a 20 floor apartment building. The restaurant’s were giving out free food
I worked at an ice-cream shop so we just started handing out free ice cream
I wish you would do a story on the New York City blackout in the early '70's. I remember being stuck because electric subways were not able to work, but never really learned what actually happened.
Are you referring to the blackout in 1977? I wasn't born till the 80s but my mom was living in NYC for the 77 blackout and was playing bingo with about 100 other people in a church basement. She realized that she had won and yelled "Bingo!" and poof, the lights went off, lol. She was still somehow able to receive her winnings ($500) but she remembers getting home that night was chaos.
@ntfx_org7603 Cuckoo.
I remember that outage, too. I was a kid, and was encouraged to study my homework by candlelight. Just like Abraham Lincoln.
I always find, as the daughter of an engineer (chemical), sister of an engineer (mechanical) and wife of an engineer (civil), that your videos are very interesting and informative to a person who trained as a biochemist and economist and has a strong interest (as a civil servant) in economic infrastructure for enterprise. And having watched the impact of the 2003 blackout in real time, this was a particularly interesting explanation.
Only one major engineering professional (electrical) is missing in your family, or until some point in future;)
@@organicfarm5524 I sure hope that someday she becomes related to an EE! It's like she's a Pokemon trainer, but for engineers. :D
It's politics, not engineering!
We need a dominant party that controls all the independent parties that operate on the Grid
I grew up in Toledo and I was maybe 4 or 5 when this happened. I read the title and I vaguely remembered it, then I remembered everyone panicking over a “blackout” and me being so confused that nothing worked. Thanks for the video and filling in a lost memory for me lol
This is a very good video on that outage. I worked at a control center in Florida and I remember seeing to frequency effects all the way down in Florida. We had telemetry with the MISO EMS system and the operators there were freaking out that day with good reason!
Hats off to the linesmen, engineers, and developers that keep our lights on. They're heros too.
⚡👷
Having been through many hurricanes, I can confirm that the visiting linemen are very appreciative of cold beer and pizza at the end of their shift.
This was a really complicated topic and you did a great job explaining it! Thank you! I'd be interested in a follow-up video going into more detail on the changes to the grid since 2003, and what vulnerabilities have/haven't been fixed.
The more I learn about electricity the more impressed I am with how we even managed to build these types of huge, complicated networks. It's incredible what humans have managed.
now just think of having to basically rebuild those but to make it 2-way more capable... on most grids...so the sun can go and come.
@@PrograError the hell are you talking about? The grid doesn't have to change.
And we have barely scratched the surface.
Very well explained! I live in Cleveland and i'll never forget that day as it was the day before i went off to college! Was in Strongsville and we needed gas and every station had their driveways blocked off...thats when we noticed stop lights were out and thus it began
I genuinely really enjoy this postmortems of infrastructure failures. Incredibly interesting and great form of storytelling event by event.
This happened in summer and not gonna lie as a teenager this was kinda fun lol. We could see the stars at night, hang out in the neighborhood until sunset, had bbq’s because all our stoves were electric. Only thing that sucked was cold showers since my house didn’t use gas. This video brought back so many memories.
It was hot though, a cold shower wasn’t bad…..
I work at a public aquarium and have experienced the cascading effect of minor problems in the network infrastructure stacking to collapse the whole. It's scary how quickly your world can go dark around you if you're not paying attention or especially if you are paying attention, but you don't have reliable data... I feel for those operators that got blindsided by bad data coming at them. Keep up the good work for all the other days that you kept it running smoothly! ⚡
I do some electronics circuit design myself. We are "intelligent" and design systems not randomly but with the components they NEED.
conversely this means any failure was something that was NEEDED and will have a cascading effect somewhere else.
I was thinking about this a while ago and its actually scary how one component failure will make the entire system inoperable, and this holds true for not just electrical systems. it also extends to society and human body :( . downside of intelligent design i suppose
Amen
@@fortheregm1249 that's why you need fault tolerance in your design?
@@monsterhunter445 you can have protections but you cant really design for fault tolerance because any component can fail in a number of different ways, in best case it will make the design 4X more complex and then its exponentially more prone to faults. Best you can do is design a duplicate system as backup. Like i said, its scary.
Grady, I wish I had a teacher like you in school. I would have become an engineer for sure. You explain things so that everyone can understand. and you dress very nice too..
I very vaguely remember all of this. I live in Nevada now but am from Cleveland and was young then and I recall it being nuts. Who knew trees, and a relay could cause so much chaos. The people who basically run our lives by helping keep electricity flowing in nearly the entire country are folks to be appreciated! Well done and amazing work on this video.
This has been very enlightening! I lived through this event in upstate NY, at the time we all shrugged, thought it was a local outage. It's the boonies, it happens. Then, over the radio scanner heard a fire call go out for a transformer on fire - which was a false call, which just further made us think, "pfffffft, no big deal..." It wasn't until the hours started passing and turning on a battery powered radio and tuning into one of the AM stations broadcasting out of NYC that we learned of the magnitude after it all. And until watching this video the take away was "Yeah, a transmission line in Ohio sagged into a tree because it was overloaded carrying another transmission line's load as well." This video has been a big eye opener into all the moving parts that really went into the whole thing. It really was the perfect storm of cascading faults and errors.
A couple of interesting events that happened in Australia that would be worth looking into and doing videos on are the south Australian blackout and subsequent black start of 2016 where the entire state went down, and the Callide C accident, where one of the units (from my interpretation of the report) essentially suffered a turbine trip. but due to a loss of both AC and DC control voltage systems was unable to disconnect from the grid and started motoring
This is really interesting, I would love to see a documentary about it! :)
That would be interesting. Sounds like it happened too recently for the to be any detailed public info though
I was right on the fringe of this blackout in Michigan. Our area still had power but we had a brownout for a couple of hours. The lights were dim, the TV (analog) hardly worked and the air conditioner was making a funny whiny noise. The power company must have borrowed power from other utilities further out West when the brownout ended.
I live on the east side of Cleveland. I was 7 years old when this happened. We were out in the swimming pool all afternoon, and when we came inside around 4 PM, the TV was off--which is odd, considering we typically kept it on for the dogs. I remember the absolute silence in the house with all the appliances off, and thinking it was weird, but cool. My grandfather and step-grandmother were visiting from Arizona and staying in my house at the time, and I remember later in the night all of us crowding around my dad's crank-operated TV watching the news. I remember going to bed by candlelight, and flushing the toilet with water from the pool, as we live in an area higher in elevation that relied on a pump for water.
I didn't think anything of that day until I got older and got my radiological mitigation certification from FEMA. Part of that course was learning how a nuclear power plant operates, and I started researching Perry NPP, which is only a 30 minute drive from my house. It's then that I realized why FirstEnergy is so aggressive about hacking our trees to bits every year, as their lack of tree trimming service was a leading cause of this event.
Since FirstEnergy filed for bankruptcy, they'd been lax about tree maintenance, so I fear it's only a matter of time until this happens again.
I was living on Long Island, NY during this black out. That was one of those problem areas you mentioned in NY that took a couple days to get power back.
I was young at the time, but I remember feeling excited that my radio could be battery powered and we could listen to the news. I remember hearing how it was a massive black out but not much else. My parents made the most of it, we had a barbeque with the neighbors and spent most of the day playing in the yard.
Things certainly weren't as connected as they are now. Strange to think about.
The precursor to this event was East Lake Unit 6 being taken off line the evening before for repairs. This caused the need to import about 600 more megawatts into the Cleveland area that weren't there the day before. Importing power, in this case, put an additional reactive strain on the transmission system. This was made worse by the East Lake unit being off and not being able to provide the reactive support - VARS to go along with the 600MW import. Also, when the blackout occurred, much more load was lost than generation. This caused the highest frequency spike ever seen on the Eastern Interconnection due to more generation remaining on line than load to serve.
ruclips.net/video/OGGAMaBGzRs/видео.html
Nah, just blame it on not having enough regulation...
"highest frequency spike" ... ?
Will you elaborate please?
@@FOH3663 The frequency is maintained at 60.00HZ unless the interconnection is on a time correction. Typically an excursion from 60 is negative and it's due to a large loss of generation, resulting in too much load for the remaining generation. This happens fairly routinely, but the frequency usually doesn't decline to much below 59.94HZ. It's quite unusual for a large enough loss of load to drive the frequency up any significant amount. This system upset did that. Frequency recording charts (now electronic, but I sure remember when they were paper!) are scaled to show declines from scheduled 60.00 to about 55.00HZ. This is for posterity, as after a decline to about 58.50HZ, generators have come off line due to protective relays opening their breakers to prevent turbine damage. These same charts don't go much over 60.20HZ because the frequency just doesn't usually move in that direction any significant amount.
Eastlake 6 is a 35MW combustion turbine. You're thinking of a different unit or a different plant. I know this because I did the siting engineering for EL6 working for CEI in the 70's.
What you say is true if you change the 6 to a 5.
Story time.
I was in the Army stationed at Camp Yongsan in 2018-2020. Camp Yongsan is in the middle of Seoul. Its being decommissioned, so there is very little oversight.
Seoul, and Yongsan are on different electrical grids because of US/Korea military protocols. One cold morning, the whole base lost power, and since I work at the data center on base, that was a big deal.
We have two 1.2 Megawatt rated generators on site specifically for the data center. The problem is, no one knew hoe to start them, and cut over from shore power, to the gennys.
I step in to figure it out. Having no formal background in such things. I got the Gennys running, but the control systems were in Korean. Eventually I got the building powered on, and everything seemed ok.
The gennys run on a 400V system, and I did notice the ammeter dial was nearly maxxed out. And when I went outside, the street lights were back on, and traffic signals too. It was very early, and everything seemed ok, so I went about my business. You might be able to see where this is going..
Within an hour they called me back, and told me both gennys breakers tripped. Well.. I dunno about you but 2.4 Megawatts seemed like a lot for one building. Plus, the street was dark again. Weird.
Come to discover, my building was basically providing power for the entire outage area for about an hour, courtesy of myself. Figured out that the shore/genny switch wasnt automatic, and I switched it over to genny only, and everything was fine. The ammeter showed only about 10% utilization. Actually, we only needed one genny running for our building.
Oh well. Thats one way to learn. I never told anyone, they were just glad it worked.
My own personal 2003 grid failure scenario.
Here is a link to a video of the actual generators from my story I filmed on that day.
photos.app.goo.gl/twmp7zAhXm3JeXCq9
Thanks for the story.
I was 13 years old and we lived near Boston at the time of the great Northeast Blackout of Nov 1965. I learned a valuable lesson that lasted to this day regarding keeping a good portable radio and flashlights on hand and a supply of spare batteries.
There where a lot of children born 9 moths after the 1965 blackout. I wonder if that happened after the 2003 blackout?
As a EEE grad; WOOOOOOW; amazing summary of events, and great explanation of the complicated dynamics of a cascading blackout. Your very brief, but approachable explanation of Real vs Reactive Power is to be envied!
Really concerning that the Right-of-ways are not well maintained enough to prevent the trees shorting the lines. I wonder how much the computer system was to blame, and whether or not over-reliance on it was the primary cause. Seems like they need to do more 'paper days' where they can become accustomed to fallback methods of monitoring and control
Most importantly they now have dedicated signals telling them if and when the computer fails, so they can focus their brains and eyes on the raw data instead of starting at the main panel stuck on "no problems detected".
I used to write software for a lines company, I would to get annoyed at the paranoia of the control room guys whenever we wanted to roll out something new.
Having seen this, I have a lot more respect for how I approach these systems. In IT a cascading failure is usually relatively easy to resolve, so I can fully understand that naive approach when it comes to the real-world effects of our critical systems failing from developers.
The IT team never agrees with the other teams. :)
@@Simcitywok I just did a case competition related to cybersecurity and this was the bulk of the points we ran into. No one wants to upgrade IT because it's one of those things that no one notices when it's actually working properly.
Its politics, not the controllers or the developers of the grid, or the people coding the circuit boards routines.
You need a party that is controlling rehabilitee, controlling all the operators on the grid. We need a dominate party here, or some Gorvenerment agency controlling the independent operators.
@@lucasrem1870 I lived on a remote Alaskan military base for a year. The power plant kept the system stable for the main buildings, but the TV station often saw wild swings in voltage, or power outages. I was working one say when the 120 volt power to the equipment dropped to 20 volts. Then it pegged the voltmeter at 210 volts before almost every breaker in the building tripped. It damaged some equipment, but it shredded the film that I was running of the start of the World series. It was so unstable after that, that I was pissed. The power plant manager just laughed and told me that there wasn't a damned thing that I could do about it
I thought, "Hold my Mountain Dew, and strap yourself in!" I put up a 'Technical Difficulties' slide, then had or DJ record a broadcast cart that stated, "AFRTS is experiencing power problems. We will return to our schedule, when it is resolved. If you have any questions, call:XXX-XXXX" The phone number was the unpublished number to the power plant manger's office. He called me and ranted, "What the hell are you doing to me?" I just laughed and told him, "There's not a damned thing you can do about it!" and hung up.
Never piss off the Engineer! 😆
@@michaelterrell Man you basically doxxed him, that ain't cool
My parents and I were on our first trip on QE2, a Westbound Transatlantic, when this happened. We stayed at the Helmsley Hotel in Manhattan for 3 days from the 17th August and I got the water equivalent of food poisoning from using the tap water in our room on the 29th floor despite my parents checking with the staff and being told the water was safe to drink. We still had a good time in Manhattan, including going to the Central Park boating lake which I've wanted to go back to for many years!
Bravo! This has been the best explanation and the first explanation that actually makes sense to me. Thank you for putting the effort into this. I was playing Diablo on my computer in upstate NY at the start of this. My UPS started arbitrarily beeping so I turned around and looked at my dining room light and saw it strobing slowly. I shut everything down and turned off my main disconnect, then I got out my meter and checked my lines coming in. It was usually 230V nominal. It was slowly going from 170 to 270 volts about twice a minute. Clearly a phasing issue. Nothing said up until now has justified that to me, but this explains it perfectly.
We were down for about 12 hours.
Thank you again. Keep up your amazing videos.
This brings back some memories. I was a graduate student in Ontario at the time. One afternoon our building on campus lost power. We didn't think much of it as there was some construction and the crews had cut a cable just the week before. After a few minutes we decided to go home. That's when we realized that the entire campus was without power, and thought that was bad. The whole magnitude of the situation only hit us when I got home and saw that there was no power anywhere in the city.
I’m a retired utility engineer and was working that day as a consulting engineer. You gave the best explanation of the August 2003 blackout that I’ve heard. One thing that needs to be said is that the power grid in the US is itself the largest and most complex“machine” ever built by man. While it’s reliability has been amazingly good, it’s really limited to our understanding of every conceivable event. It still requires the human mind to interpret events and make on the spot decisions. If this human intervention is not present, a cascading outage is the results. All major outages are a result of the lack of timely operator intervention.
One major factor is that the system itself is composed of protection equipment that ranges in age from current technology to devices over 100 years old. It all has to operate together. It’s simply not possible economically to upgrade the entire system at one time.
The best that can be done is to plan for every contingency we can identify and have an operating plan that addresses those contingencies.
As computers and Artificial Intelligence become more capable, we can analyze more contingencies and anticipate more events. However we then become more vulnerable when the technology itself fails. It’s a balancing act. Most of the time we get it right. Sometimes we run into a situation we never anticipated. I doubt it will ever be possible to completely remove the human factor from the operation of the power grid, nor would that be in our best interest.
I was working on the eastbound Amtrak Capitol Limited and we were halfway across Indiana when the power went down. The signals that control the rail lines went dark and operations were immediately impacted. We could only advance at about 15 mph between the dead signals and we only made it to Cleveland by 0830 rather than around 0200. We didn’t reach power until we got to Pittsburgh in mid-afternoon where we were due around 0800. A trip from hell…
Thank you for explaining this outage. We were moving that day, (in the Toronto area) and what a mess!! Have never heard it explained. Great Job!
"Only a few places took more than a day."
Detroit was out for 3 days!
This was super interesting to finally understand why I lost power for 3 days as a kid. I'd heard tons of rumors, including historical markers that claim it was a squirrel on the line in a key location.
Thank you once again for making these videos covering these failure events and explaining how and why they occured. They're incredibly interesting and I've been sharing them with my other engineering friends.
I was on the edge of the blackout area, in Ypsilanti, MI. I could tell the outage was an unusual one because it was preceded by several minutes of voltage sags, voltage surges, and frequency deviations as the grid became uncontrollable.
I was in Novi! Had the same experience
Also in Novi, small world
Yeah, I was living in South Ypsi at the time, I only really remember hearing some stuff about power grids failing and not having AC/electricity for about 2-3 days, but admittedly i was 9-10 yrs old around then.
I wouldn't be surprised if it happens Again
I was at the University of michigan, I had just gone home and my home was at the boundary of DTE snd Consumers Power- we had power but I saw the stoplights a mile to our East out, I called in and got no answer, so I turned around and went back. Spent the next 60 plus hours at eork. The UM power plant lost two of three generators but due to luck the last one stayed up so it didn’t have to restart lack. We had absorption cooling at critical buildings which let us save electrical power and due to the incredible competence of our trades staff we saved all the critical labs. Ive never been more aware of how complex our civilization is- or of what irreplaceable an asset human skilled people are.