How did the old telephone plant get its power? How much power does the museum use? How do we manage our power usage? All these questions answered in the video!
I had almost completely forgotten that during storms when power went out (a frequent occurrence in the rural spot where we lived when I was little) the phones would all still work and we used to routinely call others in neighborhood towns to ask about conditions there... and even sometimes call the power company to get updates about repairs. ⛈️☎️
Yeah, I really miss when phones kept working with the power out, ever since telcos in my area switched to fiber-to-the-node they never change the batteries in the node boxes so after the first LONG outage in the winter, all the batteries froze when they discharged too far. Since then, phone and internet go out with the lights. We've tried telling em about the bad batteries, but they don't wanna spend the money, and it's a great reminder that just cause a system is newer, it isn't necessarily better. It's especially bad cause most of the cell towers in the area are ALSO on battery backups, and while they last for about 6 hours when the lights go out, they either don't HAVE auxiliary gensets or they never bothered to fuel them up after the ice storm so once they're dead you're cut off from the world. It sucks that we've somehow REGRESSED in terms of disaster preparedness, it used to be that the landline was THE most reliable utility and would keep on working after everything - record setting ice storms, tornadoes, the literal nuclear apocalypse, no matter what you could still call the emergency numbers on the fridge. Now, you gotta have a Baofeng with an illegal linear amp and a big aerial, and KNOW what FRS channel and squelch code the volunteer search-and-rescue dispatches on cause police/fire/EMS are all on encrypted digital radios, even CB channel 9 isn't monitored at dispatch like it used to be for emergencies.
Many years ago here in Worcester, England, the main exchange office had power problems and service was failing. A fuse of the grasshopper style failed however for some reason did not set the alarm off! That stopped emergency calls in the area. Not good.
That actually played in to a natural disaster I experienced about 10 years ago (flood). The power for the whole town was out for days. The cell towers died after a few hours. One person I knew had a land line that worked the whole time, and they were a major communication link for a group of us that worked together.
I spent 16 years as a C.O. tech (a so called 4 wall tech). When it was time for the yearly BOD test the part I always was concerned about was the coup de fouet readings. Watching the voltage drop and waiting for the bottom and the slow rise back to the operating voltage was always stressful regardless of the fact that it always did come back up. A 100 Gig circuit down was less stressful than almost any power issue. Thanks for the video. Most folks don’t realize how sophisticated a communication network was and is today. A cell phone call in a car on the interstate that doesn’t drop is a modern miracle that people take totally for granted.
I remember i once tried to call home with uncle's cell phone in 1998 but network coverage wasn't good back then in our city. I only heard noise and static (like what you hear in a radio). It feels strange and uncanny when i remember it and compare with what we have today.
As an electrical engineering student in the early 1990's I went on a field trip to visit a telephone exchange. We were shown into a huge room with many channels in the floor designed to carry many cables. Most of the room was empty except for a couple of racks of equipment in one corner and only a handful of cable channels had a couple of wires in them. We students assumed that this room had been designed for future expansion. In reality, it was just the opposite: the exchange had been upgraded to all-solid state equipment and subscriber lines were multiplexed in decentralized equipment cabintes, so all that was coming and going into the exchange were a handful of fiber optic cables!
@@mitchbayersdorfer9381 -48v was distributed through Electronic Switching System (ESS) machines but all circuits I can think of used negative to ground circuits except for line feed and ringing. The 4 ESS ran on +130v with a lot of +3V logic.
oh yes, I remember the huge building that was the exchange when I was a kid, now its mostly empty, because its just a couple of optic fiber routers, all of the POTs signaling is generated near the consumer unit on small metal boxes you can find on the walkway every block.
Working for one of the independent telcos in FL many years ago, I was working in a CO when the power went out, the diesel generator attempted to crank over, and the starter battery promptly exploded. I thought the building had been struck by lightning. When the system gave up trying to crank that one, the turbine secondary generator whined to life a minute later. Trying to figure out the location where we’d been hit, we walked through the whole building, and when we came to the generator room, the carnage of the blown starter battery with shards of plastic all over the room and freshly sprayed acid made me very glad we didn’t typically have any reason to be in there.
I never heard of anyone dropping a wrench in a distribution unit but I saw a pair of HVAC techs dropping a duct over the bus bars. Molten galvanized steel smells awful. Keep up your excellent work!
There's an exchange near me in Edinburgh that allegedly has the word CHLORIDE debossed into the battery room ceiling after a misoperation of the plant caused the batteries to gas severely and explode, firing the cell lids upwards. I've never remembered to check for myself.
Not a telco power story but amusing nonetheless. Back in the 80s I was working in DMERT development at Bell Labs. There were several labs, each one consisting of a room with a 3B20D processor and an adjoining room containing a DEC PDP-11/70 computer, which was used to control and monitor the 3B20. Feeding these two rooms were aluminum busbars carrying 208 VAC 3-phase at some horrendous number of amps. One day I was in the lab and a couple of the busbars decided to end it all. There was a deafening ka-BOOM! and I jumped about three feet into the air. I made a beeline for the door, hitting the EPO button on the way out. (The power was already off but, hey, I hit the switch anyway.) The electricians began work on replacing the busbars and temporarily left the old ones lying in the hallway. I was standing around with a few co-workers when two guys walked by. One glanced at these melted, gnarled-up busbars, turned to his friend, and said, "Obviously a software problem."
Listening to you talk about the level of traffic the exchange could carry, reminded me that in the 80s I was part of the team that were in the Guinness book of world records for carrying IIRC 1.6 million busy-hour call attempts (BHCA) on our model System-X local exchange. Reed relays and silicon do not have the same beauty and sound as a Strowger exchange with its motor selectors and two motion switches!
A story I heard (take with a grain of salt). At a central office the power (utility) went out. OK, fine we start up the standby generator. At the time this was a gas powered turbine. To start said turbine, a blast of compressed air was sent through to start the blades running, and it was followed by the burning gas. That's fine. Generator up to speed. Now flip the switch to supply the load. Well, over the years, (since the last power failure) loads were added and the generator couldn't handle it very well, so the turbine stalled and died. Not thinking very well, the procedure was tried again (the compressed air supply was good for two starts). Rinse, repeat. So now there is no more air to start the turbine, and all they could do is watch the CO slowly die. The procedure was changed to make the charging of the compressed air supply a top priority, and the loads on the emergency supply side of things was more closely monitored. No, I was not a witness to this, it is a second (or third) hand story. I was told that this happened in San Francisco. Live and learn!
A friend of mine worked for AT&T / Ameritech. He told me there was big fiasco at I believe it was the Bowery Street central office in Akron, Ohio. The building is a fairly tall multistory office. HVAC chillers occupy the top floor and the next floor down has a bank of turbine generators. A chiller failure dowsed the turbine generators and switch gear, bringing down 9-1-1 service and large part of the CO switching. Truck based generators were brought in from an AT&T equipment yard, but the portable generators did not want to start in cold winter weather which was making the fuel gel. It apparently took several days to get everything sorted out and running again. The moral is: don’t mount your generators where they could get wet because they eventually will get wet. My old employer learned that my putting the 50 KVA MG set in the basement under a fire main which broke. TEPCO learned that at the nuke plant in Fukushima where they put the Diesels in the basement. Another fiend had an Internet business site which was hosted at an independent telco in Doylestown, Ohio (not far from Akron, Ohio) in the 1990s. A utility power failure caused a an automatic transfer switch try to start and cut in a Diesel generator, but the transfer switch exploded, knocking his Internet servers out for almost a week. I was an engineer at small data center and we test ran our Diesel every two weeks. The first test of the month was no load. The second test transferred the entire data center to the Diesel for 30 minutes. I always got the classic adrenaline rush every time I pressed the transfer button, remembering the story I was told about Doylestown. When I first started the data center job, there was not a program in place to exercise and test the Diesel generator, and very little preventative maintenance had been done. It was about 2:00 AM on a February night with an ice storm, and I was awakened by my pager. I looked at the pager and saw some power transfer notices from the Symmetra UPS. A little while later, there was another pager message from the Symmetra. Sleepy, I assumed the last notice was about the bump which occurs when the automatic transfer switch goes back to mains from Diesel. When I looked at my pager as I got up to go to work, I was horrified to see the message was the two minute warning from the Symmetra that it was about to shut down. Luckily in that last two minutes, the utility power had come back and saved my bacon. What had happened is that the Diesel had shut off almost immediately because the seal on its water pump blew out. The data center had been running on its UPS batteries for about 45 minutes. Bringing the data center up from being completely off is probably a several hour job and we would have had customer complaints. Karma was kind to us that night.
Decades ago, heard one about the graveyard technician who went to the head: first alarm: the mains went down, the second alarm: the first automatic generator failed to start, the 3rd alarm: the second automatic generator failed to start. He was moving fast by then, as the third and final generator failed to start. (you can also take this one with a grain of salt) LOL Same source; ‘The Painter and his Bucket of Paint’. Bus bar and bucket. Didn’t event cause a blip! Won’t say much about the painter (shaken, but alive). That was the kind of crowd that I was running around with back then. 80’s
One of the problems with the turbines was that once the turbine shut down and the power went out again an exhaust gas temperature sensor would not let the turbine restart if the power went off a second time a short time later. If I remember correctly, the two turbines at Anaheim Lemon (ANHMCA01) were removed and diesel generators put back in because of this.
@@edwinsinclair9853 Was that one of the Solar Gas Turbines? In the late 1960's , we had a Solar 200kw gas turbine set at ARCDCA11 that blew up ... I was on duty when the turbine, running a monthly full load test ate a gland seal (we think), failed to shut down on over speed because fuel was being supplied by way of the gear oil in the planetary gear box, air damper was slammed shut but that only crushed the flexible joint in the plenum, splitting it open and allowing room air to supply the turbine. Over speed failed so the beast just continued until it shutdown, I guess the bearings failed when speed reached some ungodly speed. I was not in the room when all of this happened, but knew something was amiss when all the building lights went off, every alarm known to man came on. Auto transfer switch failed, stuck between commercial and emergency positions so there was no emergency power, or commercial power available. I had to manually restore commercial power breaker. Total down time was about 30 minutes, just enough to get the fire department next door to extract the smoke from the power room, before it worked it's way to the #5xbar switch room on the second floor. Of course, the end cell switch operated, probably because 48 volt plant voltage had dropped down low enough. Long day, Lots of alarms that day, but no real damage to customer service or the switch machines......
All of our MG sets were 400 amp, 52 volts They were all Western Electric /GE open frame sets, similar to what you guys have, only painted gray instead of black.. We also had 400 amp solid state rectifiers (NOISEY!) . We had 4 strings of batteries made up of 23- 2.17 Volt cells each, and at least one end cell per string. Emergency power was supplied by a 200 KW Solar Gas Turbine. Our switching system was 2 units of SXS, and 1 unit of #5 XBar. We also had a 130 volt string of cells for ring generator, and Before Ma Bell got rid of them, teletypes requiring 130 V battery. Those cells were I think 2180 AH rating each, and were manufactured by either Excide or CD . If the battery voltage dropped below 48 volts, the emergency end cell switch would cut in extra cells as needed, to keep the voltage at 49.91 volts. Battery strings were connected to overhead copper bus bars hanging from the ceiling on large red insulators. Our main switch panel was all knife switches, some rated at 600 amps. Might be interesting to tell the people how those high amperage fuses were changed, with the knife switches! Dozens and dozens of fuses were used to supply the various switch frames on the 1st and second floors. We were a small local end office CO, but up the street a couple of miles, we had a 4 story CO that was also a major toll hub with lots of microwave. Their emergency power was 2-750 KW Solar gas turbines. I worked at both of these CO's, nearly 10 years before moving east, all the COEM's worked power at some time, we all trained,..I can still remember using that Weston single cell voltmeter, and taking Sp. gravity readings as a weekly routine. I ran the Emergency gas turbine once per week with no load, and once per month with full load. . That little office today is all 5ESS.
I'm always surprised at how much useful (or at least fascinating) information you pack into these videos. For example, the prevention of electrolysis with a positive ground.... Thank you to you and all your team!
When I was a telephone guy at Kapaun AB (1990s), we had these scheduled PMIs we had to do. One of them was to stress test our batteries, so every Friday we would disconnect the backup generators and commercial power, make sure we stayed up for 4 hours, and then reconnect everything. Well, we are people right, and eventually a mistake was made, and someone forgot to reconnect the power and went home for the weekend. Well...I know everybody here is smart enough to figure out what happened, next lol. So, after all of us had to stand tall on the Commander's carpet getting ripped new ones on Saturday morning, we scratched that stress test from out PMI schedule and never did it again.
In the early '80s, I worked for a company that built video switching systems. We built one system for Pacific Bell; it was to be installed in a CO in Los Angeles (I don't know which one), where it would handle all of Pac Bell's West Coast TV network switching. Because it was to be installed in a CO, it had to run on 48V. We didn't have a 48V system at the assembly plant, so we improvised one with four car batteries in series. An old welding transformer that produced a 50V output charged them. To limit the current, someone went to Sears and got a clothes dryer heating element. The whole thing was built onto a pallet, with the "ballast resistor" standing on ceramic insulators. Although it did not get red hot, it did get too hot to touch, and I always worried that it would ignite the hydrogen offgassing from the batteries. We kept a box fan blowing on the rig when the batteries were charging.
The story about Kennedy's assassination causing the switch overload reminds me of an incident when I was a teenager living in Memphis, TN. In mid-August of 1977 I was home working on my mini telephone office I had setup in a metal building in my parent's back yard. At some point I went into the house and mom was watching TV. At that moment, a very somber news bulletin interrupted her show to announce that Elvis Presley had died after being found slumped over in the master suite bathroom at Graceland. After the initial shock, mom and I both headed to our separate phones to call friends. Both phone lines were served by a crossbar 5 system and getting dial tone was difficult. It was mostly the hum of the office power over the phone until the dial tone finally kicked in. Even then, every single number dialed was met with a fast busy signal indicating busy trunks. I eventually got my call through via persistence, but mom had to wait a while to call anyone. So many people in Memphis were trying to call friends, all at once, who were Elvis fans to give the news and it seemed to overwhelm switches all over Memphis.
Another very interesting video. While I was not able to locate this scene quickly, there are two spots in the video where you show the batteries themselves. The second scene, the one with a woman standing in front of the racks, shows batteries manufactured by the company I worked at for 19 years - C&D Technologies. In earlier times they were known as C&D Battery. Batteries in that configuration normally had multiple cells in them and there was all kinds of strapping (series and/or series/parallel) to deliver 48 volts. Remember that the chemistry of lead-acid batteries tells us that an individual cell only produces 2 volts. This means that you need to wire cells in series to increase the output voltage. I was surprised that there was no mention of AT&T's contribution to the battery industry. I would expect that many of the old timers commenting here will remember the round batteries that they must have encountered during their careers. AT&T as "Ma Bell" was always concerned with longevity and reliability in anything they produced. AT&T engineers set out to create a lead-acid battery that would last longer than traditional batteries. They did this by literally changing the entire configuration. Typical lead-acid batteries are in rectangular "jars". Think of your car battery or the wastebasket sized units on those racks. The plates are all vertical. What happens over time is that the oxide paste that is held by the plates (they are actually lead grids that are filled with paste) ages and some falls to the bottom. Once there is enough to form a connection between plates the battery shorts out and fails. The Round Cell tried to eliminate that by using horizontal plates and then using a cylindrical jar. C&D was the only manufacturer licensed by AT&T to produce the Round Cell. They were in production in various configurations until about 2015. When I hired on at C&D in 2002 I saw a huge room full of these at C&D's Leola, PA manufacturing facility charging away. The other variant AT&T had was a much longer "formation" step. The first application of charging energy to a freshly created battery "forms" the positive and negative paste. The Round Cell took about 30 days to complete this step; normal lead-acid batteries take much, much less time.
Even if it was in Europe they blew the fuse, that would be in excess of 12 kW. 4800 A @ 48 V is 230.4 kW (!) That’s 20 to 40 times the Stones extravagances…
You're so good at explaining and so knowledgeable! It warms my heart whenever young people put great interest in old tech, making sure that the knowledge will be kept for generations to come.
I had a summer job for P.T.& T. at 555 Pine St. in San Francisco in 1962. One of the switchmen there told me about the earthquake of 1957. He was working in the crossbar no.5 office. When the earthquake hit the office went silent. Everyone who was on a call stayed on the phone. Those who were about to make a call froze. Then, all of a sudden, the office went crazy with everyone trying to make calls. In short order it locked up as the registered filled up.
I worked in AT&T's Integrated Test Network (ITN) as a UNIX SA in the 90's. At the time, it was the 3rd largest network behind AT&T's "real" network and MCI. We had a full battery -48V power plant. I remember walking by the control unit (not sure what it was called) one day and it was supplying -51V at 5600 Amps!
I remember in the 80s (and maybe even early 90s) sometimes when making a landline call, you could hear faint tones leaking into your line whilst waiting for things to connect or whatever was happening. And definitely remember phones working in power outages
whats best is when that wrench blows up the A side power and the B side blows up when trying to switch, causing the CO and the community to disappear from the map. those are fun days at work indeed
I remember getting a tour of our local CO as a kid and was always impressed by the big batteries and big DC cabling. Eventually got a job and worked in that building for a good 10 years as a NOC tech. We don't really touch anything we just monitor it though. When there are storms we have to keep track of all the COs that have no power so we can deploy techs with generators if the batteries get too low and it can get quite hectic. Cell towers work the same way, -48v as well. Our office got moved to the call centre building about a year ago though, I miss being in the CO, it was neat in there even though I never really got to touch anything. Interestingly enough I've been setting up a CO power plant at home to keep all my server stuff running, as well as my workstation. It's a work in progress but I have the basics, just saving up for the batteries now, and proper cabling as what I have now is temporary.
Excellent! Funny that you have a Power Designs Inc. model 5015T power supply just like I have around the 7 minute mark! I got it from a ham radio friend of mine who has passed away. He worked for Bell Labs back in the 1970's. It is a great 50 volt 1.5 amp current and voltage limited power supply. Great job on all of these videos, I really enjoy them.
Another super vid thanks Sarah! I worked in a large exchange in the UK in 1975 that had a battery stack with 60 gallons (72 US gal) of acid per cell. The Big Switch at the end of the string was scary large, SO tempting to pull the lever...
I really like your presentation style! I found this entire video really interesting. It's sometimes hard to believe that all of this stuff was designed and manufactured, often by hand, before anything resembling a computer existed. Some day I would like to visit your museum. The fact that you have the knowledge and expertise to keep a sample of such old equipment in operation is actually really remarkable. 💖
Very interesting. During WWII Bell Labs and Western Electric worked together on a top secret weapon called a Mark 24 "Mine". It was an acoustic homing torpedo developed to sink submerged submarines. Bell Labs developed the electronics that used 4 underwater microphones (Hydrophones) to steer the torpedo towards the swishing sound of the U boats propeller. Western Electric used a standard 48 volt telephone battery for power. General Electric supplied a standard 48 volt commercial wash machine motor to power the propeller. The batteries and laundry motor were chosen because they would fit inside a standard torpedo case. Code name for this weapon was "FIDO". It was first used in March, 1943 and was hugely successful. This weapon allowed the supplies for the Normandy invasion to begin being sent to England with less fear they would be sunk. This weapon was classified Top Secret for 40 or 50 years after the war ended.in case it needed to be used during the Cold War. There are RUclips videos on the Mark 24 "FIDO".
I think you hinted at it - there is no master power switch. There are also faults in the power bus that occur before fuses. You just wait for these faults to burn through and use fans to blow away the hydrogen cloud.
As an electrical engineer and master electrician in New York City, it's pretty funny seeing some of these switch gear in a museum, yet. I still see them in major Telecom COs to this day.
You see negative voltage all the time in analog amplifiers and filters, its very convenient to have two power rails, one above and the other below ground, that way you can sink current as well as source it while keeping the signal around 0 volts, avoiding introducing a DC component. In cases where its not practical to have two supplies, it's common to create a virtual ground, using a high current op-amp or similar to create a voltage exactly between the two power rails, and then distribute that through out the circuit. You also find it in some other niche applications, like flash memory, where a rather high positive or negative voltage is generated using charge pumps, and creating a negative bias voltage on the substrate of integrated circuits to improve transistor performance.
The collective knowledge you all have on these beautiful systems is just astonishing and makes me feel all warm and fuzzy! Then again, it is Summer in Texas so I’m I’ll upgrade that to prickly and sun-stroked. Either way, I really hope to come visit the museum some day and get a chance to meet all of you.
The interesting thing about dropping a spanner on busbars is it makes more of a BAMPF noise than a bang per se. The most memorable story I heard of telco power though, was a team of painters who decided that the busbars were a handy place to put a full and sealed can of paint. Did you know UK style System X style switches will work perfectly happily even when covered with white emulsion?
Having reliable power systems for telecom stuff is so important and some companies just don't take it seriously. You can add all the redundancies you want at higher levels, but they don't do much if your telephone switches/router/switch/fiber transport/microwave equipment is sitting there, powered down for silly reasons. The WISP/Fiber provider I work for generally takes it seriously and it's great how little maintenance you'll need when things are properly grounded, you have redundant modular rectifier shelves, and battery banks with a full day (or more) of runtime, and everything is on DC power. We've done acquisitions where the previous owners used standard office-grade UPSes to backup site equipment or simply nothing at all. It all falls apart quickly when a storm or a power grid issue rolls through, needing a lot of manpower for what should be preventable issues. The telephone companies learned that lesson a *long* time a ago and it frustrates me to watch new telecoms have to learn it the hard way.
It was wonderful to see End Cell switching. That has to be the craziest part of telecom power. I’m not sure you mentioned that power is not disrupted while adding a cell. This means the cell is first shorted out, then put in series with the other cells, and then the short removed. I’m assuming all end cells work the way I was taught.
Thanks again Sarah for the great presentation, the march of technological progress is inevitable and sadly the old type of CO's are vanishing to be replaced by a few racks of routers and firewalls... It's great that your facility is preserving some portion of that history :)
Another great video Sarah. I spent a lot of time around many of these old power boards in the 80’s. My job was design of the AC side especially for the very large COs. Redundancy was key and no single point failure modes were acceptable. Some central offices were so large that they had as many as 7 AC substations and 6 emergency generators! And there were still a few small substations with M-G sets still running. The museum is an invaluable record of what was achieved in the 20th century. Sadly, it is disappearing day by day. Wire line services are going away at an incredible rate. Even cellular will likely become obsolete within 20 years in favor of an all satellite network. In 50 years, people will have to visit the museum to see what the telephone system looked like.
9th grade field trip was to our local CO - my biggest memory was the power - the rows of lead acid batteries, and then the giant IH Solar gas turbine generator that had notices that it could start automatically without notice - Probably this field trip is what sparked my interest in telco - Also apparently it was rare for a smaller CO to have one of the Solar generators Looking forward to my visit to your museum next month!
Back in the late 70's and the early 80's, I handled the power rooms in Hartford, CT (SNET) for many years. In the 01 and 02 offices, we had the standard Excide batteries that most offices use. But in the newer 03 office, along with the Excide batteries, they had 12 strings of submarine batteries. These were about 4 feet tall but were solid black so you couldn't see into them. Everyone hated them and in my last few years, many of them started to swell. That's not good for any battery but even more dangerous for the submarine batteries. I learned from a friend in the Navy that these batteries only had a "safe" lifespan of about 14 years. After checking the records, I determined that these batteries in the 03 office were in their 19th year! This explains why we had to "float" these more often as they wouldn't hold that charge properly. Yeah, "floating" submarine batteries is another world in itself. You need twice the airflow to mitigate the hydrogen gas as with the Excide batteries. The risk of explosion was terrible when I had to charge these monsters. Worst of it all, for all the reports that I filed about the batteries and the danger, the submarine batteries were never replaced while I was there. I found out that they were taken out of service about 5 years after I left Hartford because one of the submarine batteries finally split open due to the swelling.
I love coming across your page! I. found one of these rooms in an old hospital that was abandoned in 2019. They had a newer telephony room that ran off of the newer grid, but It was wild seeing this and learning how it worked!
It's really amazing how much inertia is in telecom. While individual components have changed out over the years, the topology of a modern CO's power plant really hasn't changed all that much. It's still negative 48V. There's still a string of batteries in the basement, although the rectifiers are indeed all solid state. Each rack still has its own set of fuses, complete with monitoring, although that's all electronic these days. Also worth mentioning is why 48V was chosen. It was a balance of safety and conductor size, while still being high enough to reliably break through oxidation on relay contacts.
In the 1960s, where I live was served by an SxS office. It was neat to hear the electromechanical sounds after going off hook, dialing and ringing. It got quieter when the talk path was cut in at the last selector, but faint electromechanical sounds could still be heard. When making long distance calls, periodic bursts of multi frequency tones could be heard in the background.
24:00 battery plus on chassis/ground was used in cars until they managed to properly protect wires&contacts from water and found the electrolytic corrosion on the chassis more problematic on the long term.
This was very thoroughly scripted and well presented. I had never heard the rationale of using MINUS 48v being tied to avoiding electrolysis of the twisted pair cables in the outside plant.
Early lines were often 19 gauge open wire on glass insulators, so corrosion protection was needed and well understood fairly early on. In fact, negative line voltage with respect to earth goes back to telegraph days which began in 1844. In the 1840s, the only power source was wet cell batteries because dynamos had not yet been invented. The telegraph and telephone systems had battery vaults full of crow’s foot cells and battery boys to remove spent electrolyte snd consumables, then replenish the cells. As Sarah noted, early telephone sets had their own batteries, usually two wet cells. The early cells were crow’s foot with copper sulfate electrolyte. The later cells were LeClanche cells with zinc rods and carbon blocks with acid electrolyte. Union carbide in Cleveland, Ohio made carbon blocks for cells. The telephone company sent around a wagon once or twice a year to service customer batteries. In about the 1890s, no. 6 dry cells, often Blue Bell branded, replaced wet batteries. About the same time common battery office design made customer premise batteries unnecessary. None the less customer premise dry cells were still in use into at least the 1950s in some places. I have a couple of WECo metal battery boxes still in their cardboard containers which look about 1950 vintage. I also have a station service manual from about 1975, a condensed from of the applicable Bell System Practices, with a section on customer premise batteries.
Thank you for these lovely explainers, which I know take considerable time. I watch a lot, but the complexity of the phone system even pre-digital era is just staggering to me. Down the road, if you can do a large general explainer with some graphics flowcharts on how all these components link together, that would be golden. The engineering here just blows my mind.
your story telling style reminds me of Tim Hunkin and The Secret Life of Machines... if you haven't seen that old series something makes he think you would like it.
Sarah - you are the Museum’s Tim Hunkin. Thank you for bringing all these systems to life for us. It really is gratifying to see that the art of technical storytelling is alive and well.
This is fantastic. I'm due to visit Seattle in a couple of weeks, and I seriously hope to see the museum. I'm in IT now, but electromechanical, analog stuff still blows my mind, and I would love to see it in action. Thanks so much!
Wrench go bye-bye, but screwdrivers have a chance to get launched! A change of pants follows. Much more modern than the museum, but there are little packaged power plants that fit entirely within a cabinet. Marconi Lorain, Vertiv, and Valere have or had systems like this. In the typical style, BDFB up top and equipment down low. The "fully enclosed" ones are kind of fun with the ventilation slots on the lid - sized quite appropriately so that anything that should fall through would be small enough to be instantly vaporized! :) Keep tabs on your washers, folks Not quite common enough to be a rite of passage, but a good reminder to wrap your tools in electrical tape.
I definitely remember POTS working during power outages. When we moved to Santa Maria, CA in 1979 the phone numbers for the city were 5 digits. Probably the 5 digits were the local numbers just for Santa Maria. Then one day the phone company told us that we must use 92 in front of our 5 digit numbers. Some years later I had a job at a college - THEY had their own 48 volt lead-acid batteries in their basement for their elaborate 70's style phone system. So cool.
Honestly, I have been trying to get more detailed information on the design of these very early open-frame ITE breakers for years, as they were once quite ubiquitous in panel boards. I have a few drawings in 1915-1930 electrical reference books and some spurious bits of sales literature, but not a concrete working description of how the actuator actually ties in mechanically. Yours is the first video where I've ever seen one operate. And yours is a very elegant version, both because it's DC (which is harder to electrically break), and because it has the complicated secondary mechanism for automated reclosing which I had never seen before at all. I know telephony is the main focus of the channel, but if you ever decide to revisit this in a future video, you'd probably be the first to clearly document the overload-adjustable coil-dashpot assembly that made the ITE (inverse time element) design remarkable in its day.
That's a very good point, thank you. I was also trying to learn more about these breakers, and I found very little information online. All I know is what I've been able to piece together from the various bits of sales literature. Maybe I should put together a shorter video showing how these work in more detail.
I remember when the phone guy came to change the batteries for my families phone. They were in a wooden box mounted underneath the basement stairs. I remember helicoptering the worker. I think that had to be in the late 60's. I ended up being in the CATV business. We dealt with large battery banks too. Marcos went to put a labeler on top of the DC breakers box. The box didn't have a top. He quickly opened the door and grabbed it. Not noticing that he hit a couple of breakers and knocked out some critical gear that caused an outage (ONS).
You forgot to mention a little ditty when you spoke about the grasshopper fuses and the alarms. The big, cartridge fuses had no way of tripping the alarms so they wired a grasshopper fuse in the circuit. Now when a 400 amp cartridge fuse blows, the 1/4 amp grasshopper blows along with it thus tripping the alarm. But you don't know how often I have seen a "tech" standing at a BDFD with a pile of blown 1/4 amp fuses at his feet. "The damn thing keeps blowing every time I replace it!". Then I would show him that the fuse he was trying to replace was actually a "tell tail fuse" for a cartridge fuse!
Yeah, I'm calling bs on that last part. Sure, one fuse blows then replace it. The replacement immediately blew? You have a real problem and you should not try again until the problem is found and fixed. If the answer to a blowing fuse is to blindly keep jamming replacements in then why is the fuse there at all?
@@uzlonewolf unfortunately, these people should have never been CO techs. Yes it's logical for you and I to say that is a stupid thing to do but believe me, these were people with long records of being brainless. These are people who didn't care about their jobs and the union protected them.
That "motor starter" is a Manual Reduced voltage soft starter (RVSS). These have pretty much died with the electronic RVSS because most people don't learn about them. Its usually cheaper to eliminate them instead of fixing them.
Very interesting throughout, and interesting to hear about the grasshopper fuses. At the BBC in the UK the 50V fuses were very similar but were called butterfly fuses . . .
My father worked in a 5X5 crossbar office from cutover in Nov of 1949 till retirement in Dec 1973 and their power plant changed over the years. When cut into service they replaced operators, and the locals couldn't figure out the "new" system. They had three GE motor generator sets the were 200 amp which evolved into one being replaced with a Westinghouse 400 amp machine, it was a screamer sound wise compared to the GE generators. Finally in the last few years of service they installed a solid-state rectifier that became the number one in the string and ran at full output of around 500 amp. Originally there was no standby power for AC, later they installed a 100kw diesel unit that had to be manually started, just after Dad's retirement a Solar gas turbine unit was installed a 250kw unit. At retirement the exchange had two full units 232 and 439, now there are five exchange units and are all ESS and are in the basement, whole first floor has nothing left but the main frame. Haven't been in the place since ESS as there is no regular staff. Remotely monitored and "nothing can possibly go wrong!"
I spent about 10 years maintaining DC power plants and backup generators. At one site we had 3 2.2 megawatts Solar Generators operating in parallel 400 gallons of fuel per hour I also saw what happens when plumbers working on a water sprinkler system above the bus bar shorted out a ladder from the bus bar to ground. It put a hole in the 1/4 “bus bar. And destroying the ladder When working on these power plants a good safety reminder is to keep one hand in your pocket
A benefit of -48 V that could never have been foreseen by the inventor is that you can switch it with N-MOSFETs which are much more efficient than P-MOSFETs. When I did a design for a manufacturer of modern battery backups for cell phone base stations, I was quite happy that I didn't need complicated high-side switches. (Yes, -48 V is still the telecom standard.) But I have a hard time rewiring my brain for positive GND.
5:30 interesting; I haven't seen autotransformers used for starting motors, but here in Europe, with 400/230V 3-phase AC used for most motors up to tens of kilowatts (and higher voltages like 6kV if you need even more power), we use wye/delta switches; wye puts 230V on the windings and is used for startup at less power, and delta puts the full 400V working voltage on the windings. Slip rings and shunt resistors are sometimes used, they also used to be used as speed control devices, but this function has largely been taken over by variable frequency drives. Give my best to Jay - she's a friggin' genius (how else can I call someone who makes their own vacuum tubes?), next generation Glasslinger coming up :) That's one absolutely gorgeous breaker! Multi-stage switching, remote control... A thing of beauty and a joy for ever. "Scientist: It's much too unpredictable - don't let it overcharge! Barney: What do you mean, overcharge?" Mercury vapor rectifier... I miss PhotonicInduction a LOT. DDoS attack on an old phone switch, eh? Might be one of the first recorded examples. It's very interesting to learn that the negative voltage has a real deal reason to it. All the fuses look interesting too. Thanks a lot for the insightful tour!
Yep. I saw firsthand the MASSIVE detroit diesel gensets used to power the Orlando FL telephone system. It's on Robinson Ave. If you care to look it up. They are spectacular!
Good to see old equipment kept running and in good condition .Get rid of the green hair go to blue 💙.You are dealing with electricity you need to match the arc flash ! Be careful stay safe and thank you .Sent this video to a retired Bell employee .
Thanx for the history lesion. Back in the 1960s when I was a trainee technician in a large central office that consisted mainly of Western electric Rotary systems and a large step office on another floor one of our weekly duties was the power room. There we had two 24 volt batteries for long distant toll switchboards , two 48 volt CO batteries with the end cells permanently connected due to the high load of about 1200 amps and two 130 volt carrier long distant batteries. Every Monday we would set out to read all the cells that were open top not fully enclose and the the Specific Gravity readings This would take the best part of the day . The rest of the week we would just do the pilot cells. These batteries were charged from silicon rectifiers with 3 phase AC generators that were manually brought online during the peak morning and afternoon. The 130 volt batteries were changed with AC to DC generators with a "No Break" generator that had a humugas flywheel that would keep the system going until the diesel generators got going incase of a power failure. I soon leaned that sulfuric acid soon eats thru a pair of jeans.
I always thought that there was a pair of CRAY-1 super computers in the basement and the waste heat was used to boil water into steam. That steam was then used to power a pair of steam turbines that in turn spun the big generators. Or maybe I was thinking of somewhere else. Love all of your videos.
The negative 48v makes a lot of sense. One other major user of negative DC voltage is London Underground's fourth rail system, with the third rail at +420v and the fourth at -210v.
Наука
I had almost completely forgotten that during storms when power went out (a frequent occurrence in the rural spot where we lived when I was little) the phones would all still work and we used to routinely call others in neighborhood towns to ask about conditions there... and even sometimes call the power company to get updates about repairs. ⛈️☎️
Yeah, I really miss when phones kept working with the power out, ever since telcos in my area switched to fiber-to-the-node they never change the batteries in the node boxes so after the first LONG outage in the winter, all the batteries froze when they discharged too far. Since then, phone and internet go out with the lights. We've tried telling em about the bad batteries, but they don't wanna spend the money, and it's a great reminder that just cause a system is newer, it isn't necessarily better. It's especially bad cause most of the cell towers in the area are ALSO on battery backups, and while they last for about 6 hours when the lights go out, they either don't HAVE auxiliary gensets or they never bothered to fuel them up after the ice storm so once they're dead you're cut off from the world.
It sucks that we've somehow REGRESSED in terms of disaster preparedness, it used to be that the landline was THE most reliable utility and would keep on working after everything - record setting ice storms, tornadoes, the literal nuclear apocalypse, no matter what you could still call the emergency numbers on the fridge. Now, you gotta have a Baofeng with an illegal linear amp and a big aerial, and KNOW what FRS channel and squelch code the volunteer search-and-rescue dispatches on cause police/fire/EMS are all on encrypted digital radios, even CB channel 9 isn't monitored at dispatch like it used to be for emergencies.
Many years ago here in Worcester, England, the main exchange office had power problems and service was failing. A fuse of the grasshopper style failed however for some reason did not set the alarm off! That stopped emergency calls in the area. Not good.
That actually played in to a natural disaster I experienced about 10 years ago (flood). The power for the whole town was out for days. The cell towers died after a few hours. One person I knew had a land line that worked the whole time, and they were a major communication link for a group of us that worked together.
In the wild! Fancy seeing you here!
@@Alexis-lt3zy hiya! 👋☺️👍
I spent 16 years as a C.O. tech (a so called 4 wall tech). When it was time for the yearly BOD test the part I always was concerned about was the coup de fouet readings. Watching the voltage drop and waiting for the bottom and the slow rise back to the operating voltage was always stressful regardless of the fact that it always did come back up. A 100 Gig circuit down was less stressful than almost any power issue. Thanks for the video. Most folks don’t realize how sophisticated a communication network was and is today. A cell phone call in a car on the interstate that doesn’t drop is a modern miracle that people take totally for granted.
I remember i once tried to call home with uncle's cell phone in 1998 but network coverage wasn't good back then in our city. I only heard noise and static (like what you hear in a radio). It feels strange and uncanny when i remember it and compare with what we have today.
Back when people cared about the customer, the physical plant, and reputation.
As an electrical engineering student in the early 1990's I went on a field trip to visit a telephone exchange. We were shown into a huge room with many channels in the floor designed to carry many cables. Most of the room was empty except for a couple of racks of equipment in one corner and only a handful of cable channels had a couple of wires in them. We students assumed that this room had been designed for future expansion. In reality, it was just the opposite: the exchange had been upgraded to all-solid state equipment and subscriber lines were multiplexed in decentralized equipment cabintes, so all that was coming and going into the exchange were a handful of fiber optic cables!
Those negative voltage brainworms also infect you if you work on old Volkswagens.
@@mitchbayersdorfer9381 -48v was distributed through Electronic Switching System (ESS) machines but all circuits I can think of used negative to ground circuits except for line feed and ringing. The 4 ESS ran on +130v with a lot of +3V logic.
oh yes, I remember the huge building that was the exchange when I was a kid, now its mostly empty, because its just a couple of optic fiber routers, all of the POTs signaling is generated near the consumer unit on small metal boxes you can find on the walkway every block.
"red button for operation, green for disconnect" sounds like powerstation logic. "green" is always the secure "shutdown" state of each pipe or device.
Working for one of the independent telcos in FL many years ago, I was working in a CO when the power went out, the diesel generator attempted to crank over, and the starter battery promptly exploded. I thought the building had been struck by lightning. When the system gave up trying to crank that one, the turbine secondary generator whined to life a minute later. Trying to figure out the location where we’d been hit, we walked through the whole building, and when we came to the generator room, the carnage of the blown starter battery with shards of plastic all over the room and freshly sprayed acid made me very glad we didn’t typically have any reason to be in there.
I never heard of anyone dropping a wrench in a distribution unit but I saw a pair of HVAC techs dropping a duct over the bus bars. Molten galvanized steel smells awful.
Keep up your excellent work!
Metal + bus bars = **bad**
@@EricLikness But I bet its quite exciting!
There's an exchange near me in Edinburgh that allegedly has the word CHLORIDE debossed into the battery room ceiling after a misoperation of the plant caused the batteries to gas severely and explode, firing the cell lids upwards. I've never remembered to check for myself.
Not a telco power story but amusing nonetheless.
Back in the 80s I was working in DMERT development at Bell Labs. There were several labs, each one consisting of a room with a 3B20D processor and an adjoining room containing a DEC PDP-11/70 computer, which was used to control and monitor the 3B20.
Feeding these two rooms were aluminum busbars carrying 208 VAC 3-phase at some horrendous number of amps.
One day I was in the lab and a couple of the busbars decided to end it all. There was a deafening ka-BOOM! and I jumped about three feet into the air. I made a beeline for the door, hitting the EPO button on the way out. (The power was already off but, hey, I hit the switch anyway.)
The electricians began work on replacing the busbars and temporarily left the old ones lying in the hallway. I was standing around with a few co-workers when two guys walked by. One glanced at these melted, gnarled-up busbars, turned to his friend, and said, "Obviously a software problem."
Listening to you talk about the level of traffic the exchange could carry, reminded me that in the 80s I was part of the team that were in the Guinness book of world records for carrying IIRC 1.6 million busy-hour call attempts (BHCA) on our model System-X local exchange.
Reed relays and silicon do not have the same beauty and sound as a Strowger exchange with its motor selectors and two motion switches!
A story I heard (take with a grain of salt). At a central office the power (utility) went out. OK, fine we start up the standby generator. At the time this was a gas powered turbine. To start said turbine, a blast of compressed air was sent through to start the blades running, and it was followed by the burning gas. That's fine. Generator up to speed. Now flip the switch to supply the load. Well, over the years, (since the last power failure) loads were added and the generator couldn't handle it very well, so the turbine stalled and died. Not thinking very well, the procedure was tried again (the compressed air supply was good for two starts). Rinse, repeat. So now there is no more air to start the turbine, and all they could do is watch the CO slowly die. The procedure was changed to make the charging of the compressed air supply a top priority, and the loads on the emergency supply side of things was more closely monitored.
No, I was not a witness to this, it is a second (or third) hand story. I was told that this happened in San Francisco. Live and learn!
A friend of mine worked for AT&T / Ameritech. He told me there was big fiasco at I believe it was the Bowery Street central office in Akron, Ohio. The building is a fairly tall multistory office. HVAC chillers occupy the top floor and the next floor down has a bank of turbine generators. A chiller failure dowsed the turbine generators and switch gear, bringing down 9-1-1 service and large part of the CO switching. Truck based generators were brought in from an AT&T equipment yard, but the portable generators did not want to start in cold winter weather which was making the fuel gel. It apparently took several days to get everything sorted out and running again.
The moral is: don’t mount your generators where they could get wet because they eventually will get wet. My old employer learned that my putting the 50 KVA MG set in the basement under a fire main which broke. TEPCO learned that at the nuke plant in Fukushima where they put the Diesels in the basement.
Another fiend had an Internet business site which was hosted at an independent telco in Doylestown, Ohio (not far from Akron, Ohio) in the 1990s. A utility power failure caused a an automatic transfer switch try to start and cut in a Diesel generator, but the transfer switch exploded, knocking his Internet servers out for almost a week. I was an engineer at small data center and we test ran our Diesel every two weeks. The first test of the month was no load. The second test transferred the entire data center to the Diesel for 30 minutes. I always got the classic adrenaline rush every time I pressed the transfer button, remembering the story I was told about Doylestown.
When I first started the data center job, there was not a program in place to exercise and test the Diesel generator, and very little preventative maintenance had been done. It was about 2:00 AM on a February night with an ice storm, and I was awakened by my pager. I looked at the pager and saw some power transfer notices from the Symmetra UPS. A little while later, there was another pager message from the Symmetra. Sleepy, I assumed the last notice was about the bump which occurs when the automatic transfer switch goes back to mains from Diesel. When I looked at my pager as I got up to go to work, I was horrified to see the message was the two minute warning from the Symmetra that it was about to shut down. Luckily in that last two minutes, the utility power had come back and saved my bacon. What had happened is that the Diesel had shut off almost immediately because the seal on its water pump blew out. The data center had been running on its UPS batteries for about 45 minutes. Bringing the data center up from being completely off is probably a several hour job and we would have had customer complaints. Karma was kind to us that night.
Decades ago, heard one about the graveyard technician who went to the head: first alarm: the mains went down, the second alarm: the first automatic generator failed to start, the 3rd alarm: the second automatic generator failed to start. He was moving fast by then, as the third and final generator failed to start. (you can also take this one with a grain of salt) LOL
Same source; ‘The Painter and his Bucket of Paint’. Bus bar and bucket. Didn’t event cause a blip! Won’t say much about the painter (shaken, but alive).
That was the kind of crowd that I was running around with back then. 80’s
One of the problems with the turbines was that once the turbine shut down and the power went out again an exhaust gas temperature sensor would not let the turbine restart if the power went off a second time a short time later. If I remember correctly, the two turbines at Anaheim Lemon (ANHMCA01) were removed and diesel generators put back in because of this.
@@edwinsinclair9853 Was that one of the Solar Gas Turbines?
In the late 1960's , we had a Solar 200kw gas turbine set at ARCDCA11 that blew up ... I was on duty when the turbine, running a monthly full load test ate a gland seal (we think), failed to shut down on over speed because fuel was being supplied by way of the gear oil in the planetary gear box, air damper was slammed shut but that only crushed the flexible joint in the plenum, splitting it open and allowing room air to supply the turbine. Over speed failed so the beast just continued until it shutdown, I guess the bearings failed when speed reached some ungodly speed. I was not in the room when all of this happened, but knew something was amiss when all the building lights went off, every alarm known to man came on. Auto transfer switch failed, stuck between commercial and emergency positions so there was no emergency power, or commercial power available. I had to manually restore commercial power breaker. Total down time was about 30 minutes, just enough to get the fire department next door to extract the smoke from the power room, before it worked it's way to the #5xbar switch room on the second floor. Of course, the end cell switch operated, probably because 48 volt plant voltage had dropped down low enough.
Long day, Lots of alarms that day, but no real damage to customer service or the switch machines......
Another brilliant video from Sarah and the team. One of the best technical channels on youTube.
All of our MG sets were 400 amp, 52 volts They were all Western Electric /GE open frame sets, similar to what you guys have, only painted gray instead of black.. We also had 400 amp solid state rectifiers (NOISEY!) . We had 4 strings of batteries made up of 23- 2.17 Volt cells each, and at least one end cell per string. Emergency power was supplied by a 200 KW Solar Gas Turbine. Our switching system was 2 units of SXS, and 1 unit of #5 XBar. We also had a 130 volt string of cells for ring generator, and Before Ma Bell got rid of them, teletypes requiring 130 V battery. Those cells were I think 2180 AH rating each, and were manufactured by either Excide or CD . If the battery voltage dropped below 48 volts, the emergency end cell switch would cut in extra cells as needed, to keep the voltage at 49.91 volts. Battery strings were connected to overhead copper bus bars hanging from the ceiling on large red insulators. Our main switch panel was all knife switches, some rated at 600 amps. Might be interesting to tell the people how those high amperage fuses were changed, with the knife switches! Dozens and dozens of fuses were used to supply the various switch frames on the 1st and second floors. We were a small local end office CO, but up the street a couple of miles, we had a 4 story CO that was also a major toll hub with lots of microwave. Their emergency power was 2-750 KW Solar gas turbines. I worked at both of these CO's, nearly 10 years before moving east, all the COEM's worked power at some time, we all trained,..I can still remember using that Weston single cell voltmeter, and taking Sp. gravity readings as a weekly routine. I ran the Emergency gas turbine once per week with no load, and once per month with full load. . That little office today is all 5ESS.
As an electrical engineer, that was an awesome video.
I'm always surprised at how much useful (or at least fascinating) information you pack into these videos. For example, the prevention of electrolysis with a positive ground.... Thank you to you and all your team!
When I was a telephone guy at Kapaun AB (1990s), we had these scheduled PMIs we had to do. One of them was to stress test our batteries, so every Friday we would disconnect the backup generators and commercial power, make sure we stayed up for 4 hours, and then reconnect everything. Well, we are people right, and eventually a mistake was made, and someone forgot to reconnect the power and went home for the weekend. Well...I know everybody here is smart enough to figure out what happened, next lol. So, after all of us had to stand tall on the Commander's carpet getting ripped new ones on Saturday morning, we scratched that stress test from out PMI schedule and never did it again.
In the early '80s, I worked for a company that built video switching systems. We built one system for Pacific Bell; it was to be installed in a CO in Los Angeles (I don't know which one), where it would handle all of Pac Bell's West Coast TV network switching. Because it was to be installed in a CO, it had to run on 48V. We didn't have a 48V system at the assembly plant, so we improvised one with four car batteries in series. An old welding transformer that produced a 50V output charged them. To limit the current, someone went to Sears and got a clothes dryer heating element. The whole thing was built onto a pallet, with the "ballast resistor" standing on ceramic insulators. Although it did not get red hot, it did get too hot to touch, and I always worried that it would ignite the hydrogen offgassing from the batteries. We kept a box fan blowing on the rig when the batteries were charging.
Now that's some hillbilly shit right there
I love it!!! (and i'd probably do the same...)
Love these explainer videos. Amazing to see all the details of a centrals office's power systems.
The story about Kennedy's assassination causing the switch overload reminds me of an incident when I was a teenager living in Memphis, TN.
In mid-August of 1977 I was home working on my mini telephone office I had setup in a metal building in my parent's back yard. At some point I went into the house and mom was watching TV. At that moment, a very somber news bulletin interrupted her show to announce that Elvis Presley had died after being found slumped over in the master suite bathroom at Graceland. After the initial shock, mom and I both headed to our separate phones to call friends. Both phone lines were served by a crossbar 5 system and getting dial tone was difficult. It was mostly the hum of the office power over the phone until the dial tone finally kicked in. Even then, every single number dialed was met with a fast busy signal indicating busy trunks. I eventually got my call through via persistence, but mom had to wait a while to call anyone. So many people in Memphis were trying to call friends, all at once, who were Elvis fans to give the news and it seemed to overwhelm switches all over Memphis.
Neat to learn why -48 volts. I really enjoyed this video & look forward to more deep dives.
That is some cool old equipment being preserved! Never would have thought such massive power equipment was in a telephone office.
Another very interesting video. While I was not able to locate this scene quickly, there are two spots in the video where you show the batteries themselves. The second scene, the one with a woman standing in front of the racks, shows batteries manufactured by the company I worked at for 19 years - C&D Technologies. In earlier times they were known as C&D Battery. Batteries in that configuration normally had multiple cells in them and there was all kinds of strapping (series and/or series/parallel) to deliver 48 volts. Remember that the chemistry of lead-acid batteries tells us that an individual cell only produces 2 volts. This means that you need to wire cells in series to increase the output voltage.
I was surprised that there was no mention of AT&T's contribution to the battery industry. I would expect that many of the old timers commenting here will remember the round batteries that they must have encountered during their careers. AT&T as "Ma Bell" was always concerned with longevity and reliability in anything they produced. AT&T engineers set out to create a lead-acid battery that would last longer than traditional batteries. They did this by literally changing the entire configuration. Typical lead-acid batteries are in rectangular "jars". Think of your car battery or the wastebasket sized units on those racks. The plates are all vertical. What happens over time is that the oxide paste that is held by the plates (they are actually lead grids that are filled with paste) ages and some falls to the bottom. Once there is enough to form a connection between plates the battery shorts out and fails. The Round Cell tried to eliminate that by using horizontal plates and then using a cylindrical jar. C&D was the only manufacturer licensed by AT&T to produce the Round Cell. They were in production in various configurations until about 2015. When I hired on at C&D in 2002 I saw a huge room full of these at C&D's Leola, PA manufacturing facility charging away. The other variant AT&T had was a much longer "formation" step. The first application of charging energy to a freshly created battery "forms" the positive and negative paste. The Round Cell took about 30 days to complete this step; normal lead-acid batteries take much, much less time.
I Love the AvE warning sticker on the fuseboard.
The Rolling Stones blowing a 50 amp fuse got nothing on that 4,800 amp fuse!
Even if it was in Europe they blew the fuse, that would be in excess of 12 kW.
4800 A @ 48 V is 230.4 kW (!) That’s 20 to 40 times the Stones extravagances…
You're so good at explaining and so knowledgeable! It warms my heart whenever young people put great interest in old tech, making sure that the knowledge will be kept for generations to come.
I had a summer job for P.T.& T. at 555 Pine St. in San Francisco in 1962. One of the switchmen there told me about the earthquake of 1957. He was working in the crossbar no.5 office. When the earthquake hit the office went silent. Everyone who was on a call stayed on the phone. Those who were about to make a call froze. Then, all of a sudden, the office went crazy with everyone trying to make calls. In short order it locked up as the registered filled up.
I worked in AT&T's Integrated Test Network (ITN) as a UNIX SA in the 90's. At the time, it was the 3rd largest network behind AT&T's "real" network and MCI. We had a full battery -48V power plant. I remember walking by the control unit (not sure what it was called) one day and it was supplying -51V at 5600 Amps!
I remember in the 80s (and maybe even early 90s) sometimes when making a landline call, you could hear faint tones leaking into your line whilst waiting for things to connect or whatever was happening. And definitely remember phones working in power outages
whats best is when that wrench blows up the A side power and the B side blows up when trying to switch, causing the CO and the community to disappear from the map. those are fun days at work indeed
I remember getting a tour of our local CO as a kid and was always impressed by the big batteries and big DC cabling. Eventually got a job and worked in that building for a good 10 years as a NOC tech. We don't really touch anything we just monitor it though. When there are storms we have to keep track of all the COs that have no power so we can deploy techs with generators if the batteries get too low and it can get quite hectic. Cell towers work the same way, -48v as well. Our office got moved to the call centre building about a year ago though, I miss being in the CO, it was neat in there even though I never really got to touch anything. Interestingly enough I've been setting up a CO power plant at home to keep all my server stuff running, as well as my workstation. It's a work in progress but I have the basics, just saving up for the batteries now, and proper cabling as what I have now is temporary.
Excellent! Funny that you have a Power Designs Inc. model 5015T power supply just like I have around the 7 minute mark! I got it from a ham radio friend of mine who has passed away. He worked for Bell Labs back in the 1970's. It is a great 50 volt 1.5 amp current and voltage limited power supply. Great job on all of these videos, I really enjoy them.
r/endcells is a WAY better community than r/incels 😂
Omg Dev lmao 😂
I could bet my ass on that!
great video! thanks! I remember when I canceled my POTS line to get VOIP, it was a sentimental day for me. My friends were like, what the heck???
Another super vid thanks Sarah! I worked in a large exchange in the UK in 1975 that had a battery stack with 60 gallons (72 US gal) of acid per cell. The Big Switch at the end of the string was scary large, SO tempting to pull the lever...
I really like your presentation style! I found this entire video really interesting. It's sometimes hard to believe that all of this stuff was designed and manufactured, often by hand, before anything resembling a computer existed. Some day I would like to visit your museum. The fact that you have the knowledge and expertise to keep a sample of such old equipment in operation is actually really remarkable. 💖
Really happy I had the chance to visit in person last month!
Very interesting. During WWII Bell Labs and Western Electric worked together on a top secret weapon called a Mark 24 "Mine". It was an acoustic homing torpedo developed to sink submerged submarines. Bell Labs developed the electronics that used 4 underwater microphones (Hydrophones) to steer the torpedo towards the swishing sound of the U boats propeller. Western Electric used a standard 48 volt telephone battery for power. General Electric supplied a standard 48 volt commercial wash machine motor to power the propeller. The batteries and laundry motor were chosen because they would fit inside a standard torpedo case. Code name for this weapon was "FIDO". It was first used in March, 1943 and was hugely successful. This weapon allowed the supplies for the Normandy invasion to begin being sent to England with less fear they would be sunk. This weapon was classified Top Secret for 40 or 50 years after the war ended.in case it needed to be used during the Cold War. There are RUclips videos on the Mark 24 "FIDO".
I think you hinted at it - there is no master power switch. There are also faults in the power bus that occur before fuses. You just wait for these faults to burn through and use fans to blow away the hydrogen cloud.
As an electrical engineer and master electrician in New York City, it's pretty funny seeing some of these switch gear in a museum, yet. I still see them in major Telecom COs to this day.
You see negative voltage all the time in analog amplifiers and filters, its very convenient to have two power rails, one above and the other below ground, that way you can sink current as well as source it while keeping the signal around 0 volts, avoiding introducing a DC component. In cases where its not practical to have two supplies, it's common to create a virtual ground, using a high current op-amp or similar to create a voltage exactly between the two power rails, and then distribute that through out the circuit.
You also find it in some other niche applications, like flash memory, where a rather high positive or negative voltage is generated using charge pumps, and creating a negative bias voltage on the substrate of integrated circuits to improve transistor performance.
Incredibly educational. Thank you.
The collective knowledge you all have on these beautiful systems is just astonishing and makes me feel all warm and fuzzy! Then again, it is Summer in Texas so I’m I’ll upgrade that to prickly and sun-stroked. Either way, I really hope to come visit the museum some day and get a chance to meet all of you.
The interesting thing about dropping a spanner on busbars is it makes more of a BAMPF noise than a bang per se.
The most memorable story I heard of telco power though, was a team of painters who decided that the busbars were a handy place to put a full and sealed can of paint. Did you know UK style System X style switches will work perfectly happily even when covered with white emulsion?
Having reliable power systems for telecom stuff is so important and some companies just don't take it seriously. You can add all the redundancies you want at higher levels, but they don't do much if your telephone switches/router/switch/fiber transport/microwave equipment is sitting there, powered down for silly reasons. The WISP/Fiber provider I work for generally takes it seriously and it's great how little maintenance you'll need when things are properly grounded, you have redundant modular rectifier shelves, and battery banks with a full day (or more) of runtime, and everything is on DC power.
We've done acquisitions where the previous owners used standard office-grade UPSes to backup site equipment or simply nothing at all. It all falls apart quickly when a storm or a power grid issue rolls through, needing a lot of manpower for what should be preventable issues. The telephone companies learned that lesson a *long* time a ago and it frustrates me to watch new telecoms have to learn it the hard way.
The information about negative voltage was enlightening.
It was wonderful to see End Cell switching. That has to be the craziest part of telecom power. I’m not sure you mentioned that power is not disrupted while adding a cell. This means the cell is first shorted out, then put in series with the other cells, and then the short removed. I’m assuming all end cells work the way I was taught.
Thanks again Sarah for the great presentation, the march of technological progress is inevitable and sadly the old type of CO's are vanishing to be replaced by a few racks of routers and firewalls...
It's great that your facility is preserving some portion of that history :)
Another great video Sarah. I spent a lot of time around many of these old power boards in the 80’s. My job was design of the AC side especially for the very large COs. Redundancy was key and no single point failure modes were acceptable. Some central offices were so large that they had as many as 7 AC substations and 6 emergency generators! And there were still a few small substations with M-G sets still running.
The museum is an invaluable record of what was achieved in the 20th century. Sadly, it is disappearing day by day. Wire line services are going away at an incredible rate. Even cellular will likely become obsolete within 20 years in favor of an all satellite network. In 50 years, people will have to visit the museum to see what the telephone system looked like.
Satellite will only replace cellular in rural areas, urban areas are just too dense even for the new LEO constellations they're launching.
9th grade field trip was to our local CO - my biggest memory was the power - the rows of lead acid batteries, and then the giant IH Solar gas turbine generator that had notices that it could start automatically without notice - Probably this field trip is what sparked my interest in telco - Also apparently it was rare for a smaller CO to have one of the Solar generators Looking forward to my visit to your museum next month!
Back in the late 70's and the early 80's, I handled the power rooms in Hartford, CT (SNET) for many years. In the 01 and 02 offices, we had the standard Excide batteries that most offices use. But in the newer 03 office, along with the Excide batteries, they had 12 strings of submarine batteries. These were about 4 feet tall but were solid black so you couldn't see into them. Everyone hated them and in my last few years, many of them started to swell. That's not good for any battery but even more dangerous for the submarine batteries. I learned from a friend in the Navy that these batteries only had a "safe" lifespan of about 14 years. After checking the records, I determined that these batteries in the 03 office were in their 19th year! This explains why we had to "float" these more often as they wouldn't hold that charge properly. Yeah, "floating" submarine batteries is another world in itself. You need twice the airflow to mitigate the hydrogen gas as with the Excide batteries. The risk of explosion was terrible when I had to charge these monsters. Worst of it all, for all the reports that I filed about the batteries and the danger, the submarine batteries were never replaced while I was there. I found out that they were taken out of service about 5 years after I left Hartford because one of the submarine batteries finally split open due to the swelling.
Another wonderful video.... thank you.
Awesome video and well-explained.
I love coming across your page! I. found one of these rooms in an old hospital that was abandoned in 2019. They had a newer telephony room that ran off of the newer grid, but It was wild seeing this and learning how it worked!
It's really amazing how much inertia is in telecom. While individual components have changed out over the years, the topology of a modern CO's power plant really hasn't changed all that much. It's still negative 48V. There's still a string of batteries in the basement, although the rectifiers are indeed all solid state. Each rack still has its own set of fuses, complete with monitoring, although that's all electronic these days.
Also worth mentioning is why 48V was chosen. It was a balance of safety and conductor size, while still being high enough to reliably break through oxidation on relay contacts.
In the 1960s, where I live was served by an SxS office. It was neat to hear the electromechanical sounds after going off hook, dialing and ringing. It got quieter when the talk path was cut in at the last selector, but faint electromechanical sounds could still be heard. When making long distance calls, periodic bursts of multi frequency tones could be heard in the background.
24:00 battery plus on chassis/ground was used in cars until they managed to properly protect wires&contacts from water and found the electrolytic corrosion on the chassis more problematic on the long term.
THIS WAS ABOUT 1955 OR 1956, WHEN CAR MFGS CHANGED OVER TO 12 VOLT SYSTEMS, also about the time that generators were changed over to alternators.
@@henrythompson7595"+ on GND" was kept in cars from Japan well into the 1980ies, resulting many oopies for people doing car audio retrofits.
Excellent presentation.
This was very thoroughly scripted and well presented. I had never heard the rationale of using MINUS 48v being tied to avoiding electrolysis of the twisted pair cables in the outside plant.
Early lines were often 19 gauge open wire on glass insulators, so corrosion protection was needed and well understood fairly early on. In fact, negative line voltage with respect to earth goes back to telegraph days which began in 1844. In the 1840s, the only power source was wet cell batteries because dynamos had not yet been invented. The telegraph and telephone systems had battery vaults full of crow’s foot cells and battery boys to remove spent electrolyte snd consumables, then replenish the cells.
As Sarah noted, early telephone sets had their own batteries, usually two wet cells. The early cells were crow’s foot with copper sulfate electrolyte. The later cells were LeClanche cells with zinc rods and carbon blocks with acid electrolyte. Union carbide in Cleveland, Ohio made carbon blocks for cells. The telephone company sent around a wagon once or twice a year to service customer batteries. In about the 1890s, no. 6 dry cells, often Blue Bell branded, replaced wet batteries. About the same time common battery office design made customer premise batteries unnecessary. None the less customer premise dry cells were still in use into at least the 1950s in some places. I have a couple of WECo metal battery boxes still in their cardboard containers which look about 1950 vintage. I also have a station service manual from about 1975, a condensed from of the applicable Bell System Practices, with a section on customer premise batteries.
Thank you for these lovely explainers, which I know take considerable time. I watch a lot, but the complexity of the phone system even pre-digital era is just staggering to me. Down the road, if you can do a large general explainer with some graphics flowcharts on how all these components link together, that would be golden. The engineering here just blows my mind.
your story telling style reminds me of Tim Hunkin and The Secret Life of Machines... if you haven't seen that old series something makes he think you would like it.
I *love* The Secret Life of Machines! Being mentioned in the same sentence as Tim is quite an honor :)
Well, you’re definitely up there my technical friend. You have the original technical skills to be the advisor.
We need to get Sarah a TV budget and a Rex Garrod to drive the stunt cars etc
Sarah - you are the Museum’s Tim Hunkin. Thank you for bringing all these systems to life for us. It really is gratifying to see that the art of technical storytelling is alive and well.
ALL of Tims Shows are on RUclips.. Most with bonus footage at the end....
This is fantastic. I'm due to visit Seattle in a couple of weeks, and I seriously hope to see the museum. I'm in IT now, but electromechanical, analog stuff still blows my mind, and I would love to see it in action. Thanks so much!
Wrench go bye-bye, but screwdrivers have a chance to get launched! A change of pants follows.
Much more modern than the museum, but there are little packaged power plants that fit entirely within a cabinet. Marconi Lorain, Vertiv, and Valere have or had systems like this. In the typical style, BDFB up top and equipment down low. The "fully enclosed" ones are kind of fun with the ventilation slots on the lid - sized quite appropriately so that anything that should fall through would be small enough to be instantly vaporized! :) Keep tabs on your washers, folks
Not quite common enough to be a rite of passage, but a good reminder to wrap your tools in electrical tape.
I definitely remember POTS working during power outages. When we moved to Santa Maria, CA in 1979 the phone numbers for the city were 5 digits. Probably the 5 digits were the local numbers just for Santa Maria. Then one day the phone company told us that we must use 92 in front of our 5 digit numbers. Some years later I had a job at a college - THEY had their own 48 volt lead-acid batteries in their basement for their elaborate 70's style phone system. So cool.
Honestly, I have been trying to get more detailed information on the design of these very early open-frame ITE breakers for years, as they were once quite ubiquitous in panel boards. I have a few drawings in 1915-1930 electrical reference books and some spurious bits of sales literature, but not a concrete working description of how the actuator actually ties in mechanically.
Yours is the first video where I've ever seen one operate. And yours is a very elegant version, both because it's DC (which is harder to electrically break), and because it has the complicated secondary mechanism for automated reclosing which I had never seen before at all.
I know telephony is the main focus of the channel, but if you ever decide to revisit this in a future video, you'd probably be the first to clearly document the overload-adjustable coil-dashpot assembly that made the ITE (inverse time element) design remarkable in its day.
That's a very good point, thank you. I was also trying to learn more about these breakers, and I found very little information online. All I know is what I've been able to piece together from the various bits of sales literature.
Maybe I should put together a shorter video showing how these work in more detail.
Came for the power.... Stayed for the color!!! Love Sarah!!!
I remember when the phone guy came to change the batteries for my families phone. They were in a wooden box mounted underneath the basement stairs. I remember helicoptering the worker. I think that had to be in the late 60's. I ended up being in the CATV business. We dealt with large battery banks too. Marcos went to put a labeler on top of the DC breakers box. The box didn't have a top. He quickly opened the door and grabbed it. Not noticing that he hit a couple of breakers and knocked out some critical gear that caused an outage (ONS).
Good video ! Lots of great info, very well presented!
Sarah, that was fascinating! Thanks for the deep dive :)
You forgot to mention a little ditty when you spoke about the grasshopper fuses and the alarms. The big, cartridge fuses had no way of tripping the alarms so they wired a grasshopper fuse in the circuit. Now when a 400 amp cartridge fuse blows, the 1/4 amp grasshopper blows along with it thus tripping the alarm.
But you don't know how often I have seen a "tech" standing at a BDFD with a pile of blown 1/4 amp fuses at his feet. "The damn thing keeps blowing every time I replace it!". Then I would show him that the fuse he was trying to replace was actually a "tell tail fuse" for a cartridge fuse!
Yeah, I'm calling bs on that last part. Sure, one fuse blows then replace it. The replacement immediately blew? You have a real problem and you should not try again until the problem is found and fixed. If the answer to a blowing fuse is to blindly keep jamming replacements in then why is the fuse there at all?
@@uzlonewolf unfortunately, these people should have never been CO techs. Yes it's logical for you and I to say that is a stupid thing to do but believe me, these were people with long records of being brainless. These are people who didn't care about their jobs and the union protected them.
That "motor starter" is a Manual Reduced voltage soft starter (RVSS). These have pretty much died with the electronic RVSS because most people don't learn about them. Its usually cheaper to eliminate them instead of fixing them.
The hair is giving Mrs Slocombe from “Are You Being Served?”. I love it!
Are you unanimous in that?
i love the look of old electromechanical equipment..
Thanks for a really interesting and insightful video on a topic that isn't often covered. Really appreciate it 😃
This was an awesome freaking video! More like this. Thanks Sarah
Very interesting throughout, and interesting to hear about the grasshopper fuses. At the BBC in the UK the 50V fuses were very similar but were called butterfly fuses . . .
Yall are so freaking cool. This is fascinating
A place for the girls like us! I'd absolutely love to go there someday, and hang out with Sarah, Astrid, Jay and others.
Awesome! Thanks!
Absolutely love these videos! So fascinating and well presented! (Also loving the hair!)
This is so freaking cool and I cannot wait to visit!
This is so facinating. Your content has gotten better and better and this, and the power plant, probably my favorite.
Thanks so much. It's all really interesting. If I could clone a younger version of myself, she'd definitely join your team!
Great to see a new video, love your hair.
My father worked in a 5X5 crossbar office from cutover in Nov of 1949 till retirement in Dec 1973 and their power plant changed over the years. When cut into service they replaced operators, and the locals couldn't figure out the "new" system. They had three GE motor generator sets the were 200 amp which evolved into one being replaced with a Westinghouse 400 amp machine, it was a screamer sound wise compared to the GE generators. Finally in the last few years of service they installed a solid-state rectifier that became the number one in the string and ran at full output of around 500 amp. Originally there was no standby power for AC, later they installed a 100kw diesel unit that had to be manually started, just after Dad's retirement a Solar gas turbine unit was installed a 250kw unit. At retirement the exchange had two full units 232 and 439, now there are five exchange units and are all ESS and are in the basement, whole first floor has nothing left but the main frame. Haven't been in the place since ESS as there is no regular staff. Remotely monitored and "nothing can possibly go wrong!"
I spent about 10 years maintaining DC power plants and backup generators. At one site we had 3 2.2 megawatts Solar Generators operating in parallel 400 gallons of fuel per hour
I also saw what happens when plumbers working on a water sprinkler system above the bus bar shorted out a ladder from the bus bar to ground. It put a hole in the 1/4 “bus bar. And destroying the ladder
When working on these power plants a good safety reminder is to keep one hand in your pocket
Darn it Sarah, I don't know how you're able to make things so interesting, but you do it! Thanks for the great content everyone!
Excellent finale music.
A benefit of -48 V that could never have been foreseen by the inventor is that you can switch it with N-MOSFETs which are much more efficient than P-MOSFETs. When I did a design for a manufacturer of modern battery backups for cell phone base stations, I was quite happy that I didn't need complicated high-side switches. (Yes, -48 V is still the telecom standard.) But I have a hard time rewiring my brain for positive GND.
5:30 interesting; I haven't seen autotransformers used for starting motors, but here in Europe, with 400/230V 3-phase AC used for most motors up to tens of kilowatts (and higher voltages like 6kV if you need even more power), we use wye/delta switches; wye puts 230V on the windings and is used for startup at less power, and delta puts the full 400V working voltage on the windings. Slip rings and shunt resistors are sometimes used, they also used to be used as speed control devices, but this function has largely been taken over by variable frequency drives.
Give my best to Jay - she's a friggin' genius (how else can I call someone who makes their own vacuum tubes?), next generation Glasslinger coming up :)
That's one absolutely gorgeous breaker! Multi-stage switching, remote control... A thing of beauty and a joy for ever.
"Scientist: It's much too unpredictable - don't let it overcharge!
Barney: What do you mean, overcharge?"
Mercury vapor rectifier... I miss PhotonicInduction a LOT.
DDoS attack on an old phone switch, eh? Might be one of the first recorded examples.
It's very interesting to learn that the negative voltage has a real deal reason to it. All the fuses look interesting too. Thanks a lot for the insightful tour!
That stuff is so cool. I had to zoom in on the danger warning sticker. Highly recommended. 😆
Cool equipment nice explanation 👍
An interesting video. Thanks.
Fascinating video about electrical power and distribution. Do like the Mercury ARC rectifier they mesmerizing to watch when operating. 😎
Yep. I saw firsthand the MASSIVE detroit diesel gensets used to power the Orlando FL telephone system. It's on Robinson Ave. If you care to look it up. They are spectacular!
Good to see old equipment kept running and in good condition .Get rid of the green hair go to blue 💙.You are dealing with electricity you need to match the arc flash ! Be careful stay safe and thank you .Sent this video to a retired Bell employee .
Thanx for the history lesion. Back in the 1960s when I was a trainee technician in a large central office that consisted mainly of Western electric Rotary systems and a large step office on another floor one of our weekly duties was the power room. There we had two 24 volt batteries for long distant toll switchboards , two 48 volt CO batteries with the end cells permanently connected due to the high load of about 1200 amps and two 130 volt carrier long distant batteries. Every Monday we would set out to read all the cells that were open top not fully enclose and the the Specific Gravity readings This would take the best part of the day . The rest of the week we would just do the pilot cells. These batteries were charged from silicon rectifiers with 3 phase AC generators that were manually brought online during the peak morning and afternoon. The 130 volt batteries were changed with AC to DC generators with a "No Break" generator that had a humugas flywheel that would keep the system going until the diesel generators got going incase of a power failure. I soon leaned that sulfuric acid soon eats thru a pair of jeans.
this is a great video and so much information thanks,.
I always thought that there was a pair of CRAY-1 super computers in the basement and the waste heat was used to boil water into steam. That steam was then used to power a pair of steam turbines that in turn spun the big generators. Or maybe I was thinking of somewhere else. Love all of your videos.
The negative 48v makes a lot of sense. One other major user of negative DC voltage is London Underground's fourth rail system, with the third rail at +420v and the fourth at -210v.
nice job
I was missing you videos from inside the museum. Can’t wait to visit. I’m coming soon
Another amazing video. so interesing especially Volunteer Johns tale of What were you doing when president keneddy was shot.
I love this! subbed!!