A thermal camera might have helped locate the fault, if the short was causing the wiring and components to heat up -- but you'd probably consider that to be cheating!
I have scars from reaching into server racks with zip-ties... One cut almost needed stitches. I like my Velcro straps, but eventually I would love to learn wax lacing. Great job as usual! It makes me happy seeing people take pride in a hobby, especially the electro-mechanical types of hobbies.
I learned how to lace in 1998 when I was installing muxes... and then when I started to work in resi POTS a few years later I completely forgot how to do it!
I've not done lacing, but one thing that I think is important is the wire ties or lacing can put too much pressure on a bundle and cause the insulation on the wires to deform. It's not a problem with these wires because they're cotton covered. But the plain pvc insulation could be deformed. This isn't a problem with velcro strips plus it's really great to be able to remove a strip and find the wire and then put the same strip back on without having to replace it with a new one.
I've learned wax lacing in the 90s. Never used it in ma professional career. And I'm glad about that. I don't really need the blisters on my fingers again.
A housekeeping hint for all the cut cabling in the back of the panel. Lace them back into their own form then you can move them as a bundle. All the wires look to be color coded, but it appears that all the wires in the "Buss" are coded the same, or nearly. That is troubleshooting heck. The wires are bundled at the old Western Electric plant to aid in placing the laced loom in the frame. There was a wire board operator who spent his entire day running wire around a bunch of nails according his instructions. Then he would lace them all up and cut the wires. That whole assembly went to the operator that soldered all the terminals, and he would check his work with a "Beeper". Labor intensive but "Made in America".
Thanks Sarah for showing a real time trouble shooting sequence really brought back gobs of old memories from the 1970's. I can't count the number of faults exactly like that, shorted and or bent terminals, or a solder ball sitting on the banjo that I have cleared in my 31 years with just a quick visual inspection. Many a time I came in at 4pm and was handed a trouble that they had been working on the day shift with no resolution and I cleared it in a few minutes as I always did that visual inspection prior to getting out the books and tools. Bent contacts in connectors was another quick fix that everyone had overlooked. On trunk troubles I would sort them out as to trouble types and work on only one type at a time and there were only a few causes to that type and I would clear bunches at a time because I was lazy and made it easy on myself. I was actually mediocre at best but got a good reputation by being sneaky and fixing the easy stuff first then had plenty of time to spend on the head scratchers. My eyesight was always lousy so I was crap on the backplane, especially the newer wire-wrapped backplanes. Looking at all those close wire-wrapped terminals gave me virago. We had guns that tightened up and clipped off the cable ties at a specific pressure so as to not damage the wires. I still use one they are not that expensive.
If you're worried about blowing a fuse, grab one of the 24v halogen clippin lights and use it as a fuse. Instead of consuming fuses you'll just turn the light on 😊
@@steveschulte8696 That's what the headset is for. It has a resistor built in so that when I touch things with the probe, it will click in my ear, but not blow the fuse. But if the probe touches ground *and* the thing I'm testing at the same time (because I was careless with my probe) then the resistor won't be in the circuit, and I will short to ground through the metal part of the probe. I could have put a resistance lamp in place of the fuse, but I use this probe so much, and I can't be bothered to do that every single time I use it.
Hey Sarah, thanks for that! I enjoyed every minute, the complexity of central offices from back then never ceases to amaze me. Sheer engineering marvels and I wish I'd learned about this stuff 40 years ago. Cheers from the UK.
Thanks Sarah. What a great video. You really laid out how to think and work like a veteran trouble shooter. I got a call one day from a guy who had been using an IBM PC XT for close to 20 years (yeah, I know). The PC had recently started acting like someone had tuned the big red power switch (remember those on IBM PCs?) off and quickly back on. The power supply appeared to have been undisturbed and the cover was installed with security Torx screws. With a little gentle rocking I heard something. I got my security Torx driver, took the cover off and found an extra cover screw lying between a heat sink and an inductor lead which had just a little solder tinning exposed above the circuit board. That screw had apparently been there since the power supply was manufactured and only recently made firm enough contact against the inductor lead to cause trouble. It is interesting how a latent short can sit around for years, in a panel frame, or PC power supply, and just the right temperature shift, vibration, or whatever, will finally push the short together… ensuring job security for a repair person.
Hey, in my most recent work as a radio Chief Engineer before fully retiring, we had a transmitter with redundant control power supplies. Dead, dead, dead. Both Polyfuses (one immediately physically above the other) burnt up. Replaced them and one of the two redundant supplies worked. Troubleshot the other and eventually found a factory wiring error, two wires swapped in a connector, shorting the supply to ground. How it got through factory final inspection and out the door like that rather a mystery. But after 20 years of opening to protect against the permanent short, the poor polyfuse burnt up and took out the one above it. Not the first time I found things like that over the years!
@@jumbie6 This one isn’t a short circuit, but a strange repair request. I got a call from a neuroscience lab where a hydraulic micro manipulator was being used to position electrodes. The control box had suddenly quit responding. I opened it up and found a green Radio Shack alligator clip had fallen off of two pieces of wire that it had been bridging. It was apparently a low volume piece of equipment and it had been sent from the factory without remembering to make a temporary engineering change permanent.
Panel machines amaze me. I can still remember the very first time I walked into the PArkway central office in Seattle for a T-Carrier class and was met by the odd smell of hot grease and burnt cork. Smelled nothing like the Step-by-Step machine I worked. 😉Spent some time just watching the the machine work. Fascinating.
That false ground would have caused an issue to use a signal tracer, it might have bled through a bit, but it could have just been a time waster if tried.
If someone were to place a call when this fault was happening, what would happen to their call? Hangs up? Some sort of busy signal? Stuck? Also why not trace that abandoned wire bundle and eliminate stuff like that? Is there a concern that moving it around too much may cause other faults?
I choose whether or not to eliminate extra wires depending on what they are, and how they look. If they were original to the switch, I leave them. If they were added after, in a haphazard way, I remove them. These were original. A call during this failure would just get stuck.
You know. In a hundred years time, in the year of 2123, there will be more working 200 year old windup grammophones around than working CRT-televisions, AM/FM radios, videorecorders and analog telephones from the second half of the 1900's. Thinking of you guys keeping generations of telephone centrals alive and working is therefore of utmost importance for our common technological history due to their incredible rarity.
Great job of troubleshooting! @ 31:36 just before you said "send it around the back" , you needed to take that short tag end and tie a half hitch with the long lacing end to lock it in. I've done a good amount of lacing as an AT&T tech, and that half hitch will lock that loop in very well.
It's amazing how similar the debugging process is to what I do today in web development. Tracing through what works, what doesn't, how far the signal gets, where it doesn't, or where it gets corrupted, until you find a single parenthesis out of place. xD I love watching you at work. Great fun!
@ 24:15 That is a standard "frame iron" , its plugged into an outlet that connects to AC power on a sliding track connection along with that ladder. It's powered on with a switch at the end of the aisle, along with any other iron plugged into an outlet on that aisle. They are more commonly used to solder the connections from the Vertical / Cable pair, to the horizontal / equipment connections, and are left on all day in a working central office.
You don't need a special tool to cut zip ties properly, small cutters as you were using to remove them will work just fine. My personal pet peeve is when they cut them on an angle leaving a sharp point just waiting to rip the back of my hand open.
I manage a team of maintenance technicians in manufacturing. You would be amazed at how frequently we run into similar situations with lack of documentation and needing to troubleshoot hardware the way you had to in the video. Good job!
May fav part of this video was the cable lacing. Those zip ties were really bugging me. Cable lacing really is a lost art and I would love to see it used more often.
I just wanted to let you know that I've been trying to find the time to offer my self to volunteer but due to mt own obligations, job and personal issues I've not ben able to do so Sarah. But seeing you and your passion in maintaining these systems bring me great joy and awaken my own passion and I hope to see more in the future! I hope to stop by again - Ryan
Thanks, Sarah, for a very interesting video. You brought up an excellent point that troubleshooting becomes a lot more intense when customers are out of service. I enjoyed the wax string lacing demonstration. I have always been impressed how great a job the Western Electric installers did when lacing up the back plane wiring no matter the type of switch.
I used to work A-7s, it had a lot of relays and relay cards. The TA-7C almost exclusivity had relays and cards made in Iran. You couldn't get new after the Fall of the Shah. Please note, electromechanical relays do have a usable life, and those are teaching the end of theirs. The tantalum pads erode, corrode and Arc Over fusing the relay shut, the springs and leaves fracture and break leaving the circuit open. Everything ends.... everything.
There's a CO three blocks from my apartment that was built in the 1920's and still operates today. I can only dream of what kind of treasures are in that building (probably more internet stuff than POTS nowadays, so the cool stuff has long been silent), especially since South Central Bell bricked up all the windows sometime before I was born. I don't really know the story of how you guys got a CO and equipment to start a museum, but I would love to do something like that way down here in Alabama. Less than an hour away from my home in Birmingham is where the first ever 911 call was placed.
Yeah, funnily enough the same zip tie issue from the old times translates into current times. I just so happened to slice my whole hand open on goddamn cableties in a data center
Fascinating. The largest relay I've had trouble with was a six-pole double-throw in a National NCX-5 transceiver. The engineers really pushed way past the specs, and they were switching 280 volts DC with contacts rated at 24 VDC. To make it worse they were switching 280 volts off with the break contacts and turning ground on with the make contacts, so a few milliseconds of time they were shorting. They didn't last more than a year or two at best. Ma Bell probably didn't push things that far. I wonder if you could have used a current-clamp type of meter to find the short? Or do you only use period-correct test equipment?
Total respect for your attention to nicely lacing the wiring back together at the end. I'm curious what kind of string are you using? Thank you for the great video!
Ma Bell had a lacing waxed nylon cord, it was single ply and it was used for for that purpose. It is very strong, it could not be broken with you hands.
Wow, brings back loads of memories. The sound of the panel office…I’ve heard that spinning selector sound whenever something major happened in NYC (where I worked) An other sense that brings it all back is the smell….any time I’m near a box of crayons it smells like panel, because the wires and numerous other items were coated in paraffin. Thanks for the memories. 😊
It would have been good to actually busy out the E decoder connector and then show that the call sim could work normally with just one bad component busied out. A lot of work lacing wires that don't actually go anywhere just to make it look good. Kudos!
That was an AWESOME troubleshooting and fixing video ! This is not only very interesting to watch but can help others in the future as a troubleshooting aid for the museum and for troubleshooting in general. The museum is SO much better with you there, Sarah ! Thank you !
Great troubleshooting video! It also illustrates the relationship between schematics and wiring diagrams, and the immense amount of labor required to construct these electromechanical switches.
Awesome troubleshooting Sarah, and it your lacing was extremely impressive. I had learned how to lace back in the stone age, but have forgotten how to do it.
It’s nice to see you guys take care of this old equipment. Here in Norway the entire landline network is retired. The last subscribers were shut down last year. Tho the cables remain to supply other forms of communication for the time being (a few years). An era is over and it’s sad to see it go.
It will be some time until the US pulls the plug on copper telecom circuits. Even systems at major airport, when our company was having to move circuits to a new equipment office, still required copper / Mux fed T1's over the new equipment path.
@@poormanselectronicsbench2021 yes, for many local applications I’m sure it will be kept up here too. But the law in Norway used to say that everyone has the right to have a phone where they lived, back when we had one government owned phone company. This made them build a lot of infrastructure that would never economically exist if it was a private company. We are only about 5 million people in a huge country with lot of mountains and fjords. It resulted in an extensive network based on copper and microwave radio links. On the whole tho, the network is being scaled/taken down now that there are optional ways of providing a reliable communication option to all of us. If this is as secure and reliable is a huge concern and question. But that’s the road we are heading.
I assume that landlines still exist, but they don't use the landline network. They'll use VOIP over the Internet now. And I hope the government is now declaring that everyone has the right to have an Internet line where they live.
@@thewhitefalcon8539 Up until mid 2021, I handled both the equipment, as well as part of the copper loops for one large Telecom for my area in the Chicago suburbs and outlying areas. They (In Norway) most likely still employ a "copper loop" for some services, but I am betting that analog "POTS" (Plain Old Telephone Service) has been changed to VOIP over a ADSL/VDSL service. Replacing all the copper with a fiber overlay would be great, but most countries still haven't gotten that far, and end users still are using old SONET circuits like T1's and have not as of yet made plans to update, so copper loops will still be around for awhile.
@@poormanselectronicsbench2021 Yeah that's pretty much what I'm accustomed to in NZ, about 10-15 years ago when a similar transition was made in cities. DSL is a fine way to reuse old lines. Not every customer needs fiber speeds yet so if it ain't broke don't fix it. I'll mention that DSL is not reliable on every loop (experienced this a lot on MY loop!) so it's also good for the telephone company to keep the option to leave the landline alone - they can plug it into a VOIP adapter in the telephone office instead of the user premises.
Sarah, It is a shame you only have 1 Decoder.. That would really pin point it. Stop blowing fuses HeHe. Sure, if the Sender did not release the connector there goes the Decoder. Good technique. Oh god memories back to 1970.
Sarah, since you know one end of the offending wire is one side of the SG magnet, why not just pick a working one, connect one side of the SG magnet to the meter, and just scan for continuity on the SF contacts? That would give you one side and observation of the SF contacts should show you the other side.
I don't use meters that much, and I didn't visualize the problem the same way you did. Your solution would have worked, but introducing a meter into the troubleshooting would have made things much worse for me.
@@ConnectionsMuseum No criticism intended, I'm just boggled that you keep so much high complexity in your head, much less keep it straight. I can get tangled following the sequence of a step switch :). I'm glad you folks are keeping this stuff alive :)
@@ConnectionsMuseum Coming at this from another perspective, instead of wiring in a meter, would it have been possible to use a split-core current transformer to tag the grounded wire on either a working or the troubled SG relay? You'd check for blips in the transformer output (induced voltage) when the relay changed state.
Wow what an intense debugging job! Great technique to chase down the problem though. I had not realized how intense and concentrated the wiring behind the frame is, ouch!
Amazing old machines... but I couldn't help but think when you said that magnet was about 30-35 degrees C... imagine the old guys that worked in telco back in the day would have hurled some verbal abuse at anyone heard using such non-patriotic units of measure!😂
Wow. That system is quite a maze to work your way through. I think you’re one of the few people left in the world that understands it so well. I’m always impressed by how you figure things out. A great analyst!
I tie cables like that in the heavy machinery I service, it makes it so much nicer to work on in the future. Funny enough, I picked it up from old telco stuff.
17:25 feels like a far too crude way to search for connections between 48 volts and ground. Discharge machining the contacts is often not a good way to make things last. I would have a suitable resistor, a small bridge rectifier, and an LED. Would light up visibly when in circuit, but not cause needless sparks. (A suitable resistor would be around 10K ohm or so.) edit: 24:30 is something I thought were a joke. That is a ridiculously large iron for the job. Heating up surrounding wires is a wonderful way to degrade the insulation and cause future shorts as the degraded isolation fails. I understand trying to be authentic to the period and such. But when it comes to maintaining the machines, it is better to use working practices that helps preserve the machine. 26:08 yes, those are improperly installed cable ties. They should be flush cut such that they won't cut someone's arm open in the future. (I know of companies that has fired people for not doing this properly. It is a liability for the company to ship products that can inflict such preventable injuries.) 26:22 "Does anybody ever do that?" Well, I do. To the point that I fix those that aren't done properly when I see them. 28:50 is a classic. Stiff wires getting pulled about over the years usually dislodges terminal blocks and other stuff over time. Now these days the issue is often cracked solder joints leading to intermittent connections, rather than bent metal terminals shorting out. But the root cause is the same, stiff wires getting forced about through years of service and eventually pulling something along with them..
Reminds me of the wall of relays at my last job.. A 1950s nurse call system in a hospital. The coils had a light wired across them so you would know which relay had power. Chasing down glitches could be a nightmare.. That looks to be orders of magnitude more complicated than the nurse call system.
Well done Sarah! As an electronics engineer, I can truly applaud your debugging. Had the short been a metallic insect, you would have clearly found and eradicated the BUG ;o)
Same here. Though my experience with common control systems was limited to Western Electric 7A, 7A1 and 7A2 rotary systems then later ESS and digital PBX systems.
@@paulwarner5395 the only element of common control in the UK GPO's strowger system was the directors on the larger cities, (and later register translators for subscriber trunk dialling)... they'd set up the call, then somehow drop out of the connection, something to do with marking a trunk, then a cheaper and less complex selector hunts for it, siezes it.. and the translator/director drops out to be free for another call. Witchcraft, like RF antenna design! LOL
What would you say to a video answering a fuse alarm, from a major and/or minor initiation to possibly showing the alarm bay to zeroing in by way of aisle lights and panel lights Possibly other types of alarms. This small but very effective tool would demonstrate the conviction to service from the Bell System.
i think we should all go back to lacing, it's really not that hard honestly, and i agree. Cable ties are just horrible, one-use trash. (Except re-usable ones, but they always leave a long tail if you choose not to clip them for that reason)
I wonder if a thermal camera could have been useful, following the one wire carrying current for way longer than normal? The newer higher resolution thermal imagers are getting cheaper and cheaper these days, and have enough detail to be pretty useful.
Is Sarah only trying to use techniques and materials that were available when the panel switch was built 100 years ago? No thermal cameras, zip-ties, signal tracers, Velcro, etc.
Love the analysis. There are so many tips in your videos. I’m building a step by step switch and this video just taught me how to lace properly. Thank you .. J
We would drive past an office like this when I was a kid. I peered in as best I could as we drove past-And I would see what looked like a warehouse with rack after rack of stuff like this. Who knew what marvels went on inside!
It's so cool to see a mechanical switching office still working. I worked in a DMS100 office for about 10 years but we recently got moved to our call centre office which is a standard cubicle farm so don't get to see the equipment anymore. I just monitor stuff so don't really work on it but it was fun to be in that building and see things get added, changed etc over time.
This is my first communications museum video that just happened across my suggestions. I see people calling you Sarah… are you the same Sarah I remember from Cablevision in Shelton? If not you’ve got an East Coast doppelgänger.
23:45 c'mon, you've got over 13k subscribers! They're just not hooked on your exchange... yet :D Nice troubleshooting video. Reverse-engineering an exchange makes electronics rev-eng bleak in comparison. "The cake is a lie."
Really interesting to watch. and think about when they were used in larger settings to be a tech and get that work order. i don't think they had a few hours to solve the problem. zip ties are great but when they are cut to short you get that minefield of sharp edges. so nice to see that cable stitching being done and very important to do.
Great vid for FIRST TIME viewer! Now I gotta learn MORE antique electronics, thanks MA BELL! Just learned how to brew a tetrode from scratch and how to make ic's from sand... Okay, last was not real. Just saying...
As a Telcom engineer calling e1 and b channel/sigtran signaling ancient the marvel of mechanical switches is quite awesome! Then again I also very much like the old skool pinball machines :)
BTW, we would never put all that back together till we verified we had completed the repair by testing. Yes, I know you are confident you found a problem, still there could, for example, be more to the problem.
Well that was today's lesson in humility. I started as a hobbyist in electronics at age 3.5, I am now in my late 30s. I spent +13 years working as a research engineer. I had no idea a special cutter existed for cable ties until now. 1. Thank you. 2. How is this tool for such a common use so obscure? I don't even know the name of it.
The switch sounded like it was full of angry bees at the beginning when it couldn't find a sender. Nice video! (I am an old Nortel guy who cut my teeth on a SP-1)
I missed the step where you prove that the system no longer runs out of resources when you run the call generator. (In the spirit of bug tracking & fixing as described in "The pragmatic programmer".)
I have a wonder for you @Connections Museum. When i search for foults in our relayboxes for railway, we have two ways to do. With a volt meter och a lamp with a magnet for the ground and the test lead. Is it not possible to do someting sumular in your applications?
I used to be able to make that happen at my local CO back in the days of switched systems! I would take the CO down for a day or two before they could fix the problem.
Zip-ties are the devil, but when I have to use them, i've found that toenail clippers are a good inexpensive substitute for the special zip tie flush cutter!
Nice to see someone who knows how to lace cable. But, I am one of the people who uses cable ties.. but I do have the proper tool and use it every day :-)
A thermal camera might have helped locate the fault, if the short was causing the wiring and components to heat up -- but you'd probably consider that to be cheating!
I have scars from reaching into server racks with zip-ties... One cut almost needed stitches. I like my Velcro straps, but eventually I would love to learn wax lacing.
Great job as usual! It makes me happy seeing people take pride in a hobby, especially the electro-mechanical types of hobbies.
I learned how to lace in 1998 when I was installing muxes... and then when I started to work in resi POTS a few years later I completely forgot how to do it!
I've not done lacing, but one thing that I think is important is the wire ties or lacing can put too much pressure on a bundle and cause the insulation on the wires to deform. It's not a problem with these wires because they're cotton covered. But the plain pvc insulation could be deformed. This isn't a problem with velcro strips plus it's really great to be able to remove a strip and find the wire and then put the same strip back on without having to replace it with a new one.
I've learned wax lacing in the 90s. Never used it in ma professional career. And I'm glad about that. I don't really need the blisters on my fingers again.
@@MarcoTedaldi Amen to that
That's why you use flush cut snips. I've had cuts as well wiring in locomotive cabinets. I've only used Velcro for important Radio antenna cabling..
A housekeeping hint for all the cut cabling in the back of the panel. Lace them back into their own form then you can move them as a bundle.
All the wires look to be color coded, but it appears that all the wires in the "Buss" are coded the same, or nearly. That is troubleshooting heck. The wires are bundled at the old Western Electric plant to aid in placing the laced loom in the frame. There was a wire board operator who spent his entire day running wire around a bunch of nails according his instructions. Then he would lace them all up and cut the wires. That whole assembly went to the operator that soldered all the terminals, and he would check his work with a "Beeper". Labor intensive but "Made in America".
17:15 Spark!
Thanks Sarah for showing a real time trouble shooting sequence really brought back gobs of old memories from the 1970's. I can't count the number of faults exactly like that, shorted and or bent terminals, or a solder ball sitting on the banjo that I have cleared in my 31 years with just a quick visual inspection. Many a time I came in at 4pm and was handed a trouble that they had been working on the day shift with no resolution and I cleared it in a few minutes as I always did that visual inspection prior to getting out the books and tools. Bent contacts in connectors was another quick fix that everyone had overlooked. On trunk troubles I would sort them out as to trouble types and work on only one type at a time and there were only a few causes to that type and I would clear bunches at a time because I was lazy and made it easy on myself. I was actually mediocre at best but got a good reputation by being sneaky and fixing the easy stuff first then had plenty of time to spend on the head scratchers. My eyesight was always lousy so I was crap on the backplane, especially the newer wire-wrapped backplanes. Looking at all those close wire-wrapped terminals gave me virago.
We had guns that tightened up and clipped off the cable ties at a specific pressure so as to not damage the wires. I still use one they are not that expensive.
I always am very pleased to see younger folks that both know HOW older technologies are supised to work, and care enough to do things correctly.
If you're worried about blowing a fuse, grab one of the 24v halogen clippin lights and use it as a fuse. Instead of consuming fuses you'll just turn the light on 😊
Yeah I know that trick. I also have a fuse with a lamp wired in place of the fuse element. I should have grabbed that, instead of being lazy.
Another is to get a automotive trouble shooting probe light, and put a -48 volt in it.
@@steveschulte8696 That's what the headset is for. It has a resistor built in so that when I touch things with the probe, it will click in my ear, but not blow the fuse. But if the probe touches ground *and* the thing I'm testing at the same time (because I was careless with my probe) then the resistor won't be in the circuit, and I will short to ground through the metal part of the probe.
I could have put a resistance lamp in place of the fuse, but I use this probe so much, and I can't be bothered to do that every single time I use it.
lamp turn on :)
Hey Sarah, thanks for that! I enjoyed every minute, the complexity of central offices from back then never ceases to amaze me. Sheer engineering marvels and I wish I'd learned about this stuff 40 years ago. Cheers from the UK.
Thanks Sarah. What a great video. You really laid out how to think and work like a veteran trouble shooter. I got a call one day from a guy who had been using an IBM PC XT for close to 20 years (yeah, I know). The PC had recently started acting like someone had tuned the big red power switch (remember those on IBM PCs?) off and quickly back on. The power supply appeared to have been undisturbed and the cover was installed with security Torx screws. With a little gentle rocking I heard something. I got my security Torx driver, took the cover off and found an extra cover screw lying between a heat sink and an inductor lead which had just a little solder tinning exposed above the circuit board. That screw had apparently been there since the power supply was manufactured and only recently made firm enough contact against the inductor lead to cause trouble.
It is interesting how a latent short can sit around for years, in a panel frame, or PC power supply, and just the right temperature shift, vibration, or whatever, will finally push the short together… ensuring job security for a repair person.
Hey, in my most recent work as a radio Chief Engineer before fully retiring, we had a transmitter with redundant control power supplies. Dead, dead, dead. Both Polyfuses (one immediately physically above the other) burnt up. Replaced them and one of the two redundant supplies worked. Troubleshot the other and eventually found a factory wiring error, two wires swapped in a connector, shorting the supply to ground. How it got through factory final inspection and out the door like that rather a mystery. But after 20 years of opening to protect against the permanent short, the poor polyfuse burnt up and took out the one above it. Not the first time I found things like that over the years!
@@jumbie6 This one isn’t a short circuit, but a strange repair request. I got a call from a neuroscience lab where a hydraulic micro manipulator was being used to position electrodes. The control box had suddenly quit responding. I opened it up and found a green Radio Shack alligator clip had fallen off of two pieces of wire that it had been bridging. It was apparently a low volume piece of equipment and it had been sent from the factory without remembering to make a temporary engineering change permanent.
Panel machines amaze me. I can still remember the very first time I walked into the PArkway central office in Seattle for a T-Carrier class and was met by the odd smell of hot grease and burnt cork. Smelled nothing like the Step-by-Step machine I worked. 😉Spent some time just watching the the machine work. Fascinating.
Great video! Do you never use a tone sender/tracer when trying to find a wire run? Or was that not possible because of the wire connected to ground?
That false ground would have caused an issue to use a signal tracer, it might have bled through a bit, but it could have just been a time waster if tried.
If someone were to place a call when this fault was happening, what would happen to their call? Hangs up? Some sort of busy signal? Stuck?
Also why not trace that abandoned wire bundle and eliminate stuff like that? Is there a concern that moving it around too much may cause other faults?
I choose whether or not to eliminate extra wires depending on what they are, and how they look. If they were original to the switch, I leave them. If they were added after, in a haphazard way, I remove them. These were original.
A call during this failure would just get stuck.
You know. In a hundred years time, in the year of 2123, there will be more working 200 year old windup grammophones around than working CRT-televisions, AM/FM radios, videorecorders and analog telephones from the second half of the 1900's.
Thinking of you guys keeping generations of telephone centrals alive and working is therefore of utmost importance for our common technological history due to their incredible rarity.
Great job of troubleshooting! @ 31:36 just before you said "send it around the back" , you needed to take that short tag end and tie a half hitch with the long lacing end to lock it in. I've done a good amount of lacing as an AT&T tech, and that half hitch will lock that loop in very well.
Thanks for the tip!
It's amazing how similar the debugging process is to what I do today in web development. Tracing through what works, what doesn't, how far the signal gets, where it doesn't, or where it gets corrupted, until you find a single parenthesis out of place. xD
I love watching you at work. Great fun!
Lovely work lacing up the looms (probably not how you say it in US) :)
Rigidity and neatness, nobody wants to work with a so-called Hay Baler when it comes to wire racks.
There are two main faults in electrical engineering: either there is a contact where it should not be, or there is no contact where it should be)
You couldn't find a bigger soldering iron?! 🤣 Thankyou Sarah for another amazing video ♥
One of those new TS80S USB-C irons would probably be awesome in this setup
You should check out the one they have for doing soldering work on telephone lines. But yes, she could've found a bigger one x)
@ 24:15 That is a standard "frame iron" , its plugged into an outlet that connects to AC power on a sliding track connection along with that ladder. It's powered on with a switch at the end of the aisle, along with any other iron plugged into an outlet on that aisle. They are more commonly used to solder the connections from the Vertical / Cable pair, to the horizontal / equipment connections, and are left on all day in a working central office.
You don't need a special tool to cut zip ties properly, small cutters as you were using to remove them will work just fine. My personal pet peeve is when they cut them on an angle leaving a sharp point just waiting to rip the back of my hand open.
I manage a team of maintenance technicians in manufacturing. You would be amazed at how frequently we run into similar situations with lack of documentation and needing to troubleshoot hardware the way you had to in the video. Good job!
May fav part of this video was the cable lacing. Those zip ties were really bugging me. Cable lacing really is a lost art and I would love to see it used more often.
Look up "Chicago stitch lacing" and "Kansas City Stitch lacing" on YT, there's a few videos if you want to see more
its neat how you can see problems happening in real time on these old mechanical switches. it almost throws a tantrum haha.
I just wanted to let you know that I've been trying to find the time to offer my self to volunteer but due to mt own obligations, job and personal issues I've not ben able to do so Sarah. But seeing you and your passion in maintaining these systems bring me great joy and awaken my own passion and I hope to see more in the future! I hope to stop by again - Ryan
Thanks, Sarah, for a very interesting video. You brought up an excellent point that troubleshooting becomes a lot more intense when customers are out of service. I enjoyed the wax string lacing demonstration. I have always been impressed how great a job the Western Electric installers did when lacing up the back plane wiring no matter the type of switch.
Happy late birthday panel.
Also. A fellow trans girl being ideological about safety precautions and protocols will never not be cute and awesome.
I used to work A-7s, it had a lot of relays and relay cards. The TA-7C almost exclusivity had relays and cards made in Iran. You couldn't get new after the Fall of the Shah. Please note, electromechanical relays do have a usable life, and those are teaching the end of theirs. The tantalum pads erode, corrode and Arc Over fusing the relay shut, the springs and leaves fracture and break leaving the circuit open. Everything ends.... everything.
There's a CO three blocks from my apartment that was built in the 1920's and still operates today. I can only dream of what kind of treasures are in that building (probably more internet stuff than POTS nowadays, so the cool stuff has long been silent), especially since South Central Bell bricked up all the windows sometime before I was born. I don't really know the story of how you guys got a CO and equipment to start a museum, but I would love to do something like that way down here in Alabama. Less than an hour away from my home in Birmingham is where the first ever 911 call was placed.
They got the equipment because the museum is a CO that ran it until the 70s and upgraded to a single floor ESS
Yeah, funnily enough the same zip tie issue from the old times translates into current times. I just so happened to slice my whole hand open on goddamn cableties in a data center
I remember the old timers they took good care of those switches and you didn't dare touch THEIR switches !
Fascinating. The largest relay I've had trouble with was a six-pole double-throw in a National NCX-5 transceiver. The engineers really pushed way past the specs, and they were switching 280 volts DC with contacts rated at 24 VDC. To make it worse they were switching 280 volts off with the break contacts and turning ground on with the make contacts, so a few milliseconds of time they were shorting. They didn't last more than a year or two at best. Ma Bell probably didn't push things that far.
I wonder if you could have used a current-clamp type of meter to find the short? Or do you only use period-correct test equipment?
Total respect for your attention to nicely lacing the wiring back together at the end. I'm curious what kind of string are you using? Thank you for the great video!
As a WECO installer, we used rolls of "12 cord" which was 12 ply waxed cotton cord. Nowadays, they use 8 & 9 ply waxed cord instead.
Ma Bell had a lacing waxed nylon cord, it was single ply and it was used for for that purpose. It is very strong, it could not be broken with you hands.
You can still get 12 ply. Specialized has it.
The cake is a lie.. 😋
All this fun stuff is making me want to buy another electro-mech pinball machine to restore. Another awesome video!
I can _smell_ that blueprint.
A strange mix of ammonia and ozone for the "diazo" process, lol
Wow, brings back loads of memories. The sound of the panel office…I’ve heard that spinning selector sound whenever something major happened in NYC (where I worked) An other sense that brings it all back is the smell….any time I’m near a box of crayons it smells like panel, because the wires and numerous other items were coated in paraffin. Thanks for the memories. 😊
It would have been good to actually busy out the E decoder connector and then show that the call sim could work normally with just one bad component busied out.
A lot of work lacing wires that don't actually go anywhere just to make it look good. Kudos!
12:50 Wow, a schematic printed in cyanotype? A literal blueprint for the wiring
That was an AWESOME troubleshooting and fixing video ! This is not only very interesting to watch but can help others in the future as a troubleshooting aid for the museum and for troubleshooting in general. The museum is SO much better with you there, Sarah ! Thank you !
Great troubleshooting video! It also illustrates the relationship between schematics and wiring diagrams, and the immense amount of labor required to construct these electromechanical switches.
That is why it is necessary to wrap the place of soldering in an insulating tube so that it does not close with adjacent contacts.
Awesome troubleshooting Sarah, and it your lacing was extremely impressive. I had learned how to lace back in the stone age, but have forgotten how to do it.
It’s nice to see you guys take care of this old equipment. Here in Norway the entire landline network is retired. The last subscribers were shut down last year. Tho the cables remain to supply other forms of communication for the time being (a few years). An era is over and it’s sad to see it go.
It will be some time until the US pulls the plug on copper telecom circuits. Even systems at major airport, when our company was having to move circuits to a new equipment office, still required copper / Mux fed T1's over the new equipment path.
@@poormanselectronicsbench2021 yes, for many local applications I’m sure it will be kept up here too. But the law in Norway used to say that everyone has the right to have a phone where they lived, back when we had one government owned phone company. This made them build a lot of infrastructure that would never economically exist if it was a private company. We are only about 5 million people in a huge country with lot of mountains and fjords. It resulted in an extensive network based on copper and microwave radio links. On the whole tho, the network is being scaled/taken down now that there are optional ways of providing a reliable communication option to all of us. If this is as secure and reliable is a huge concern and question. But that’s the road we are heading.
I assume that landlines still exist, but they don't use the landline network. They'll use VOIP over the Internet now. And I hope the government is now declaring that everyone has the right to have an Internet line where they live.
@@thewhitefalcon8539 Up until mid 2021, I handled both the equipment, as well as part of the copper loops for one large Telecom for my area in the Chicago suburbs and outlying areas. They (In Norway) most likely still employ a "copper loop" for some services, but I am betting that analog "POTS" (Plain Old Telephone Service) has been changed to VOIP over a ADSL/VDSL service. Replacing all the copper with a fiber overlay would be great, but most countries still haven't gotten that far, and end users still are using old SONET circuits like T1's and have not as of yet made plans to update, so copper loops will still be around for awhile.
@@poormanselectronicsbench2021 Yeah that's pretty much what I'm accustomed to in NZ, about 10-15 years ago when a similar transition was made in cities. DSL is a fine way to reuse old lines. Not every customer needs fiber speeds yet so if it ain't broke don't fix it. I'll mention that DSL is not reliable on every loop (experienced this a lot on MY loop!) so it's also good for the telephone company to keep the option to leave the landline alone - they can plug it into a VOIP adapter in the telephone office instead of the user premises.
Sarah, It is a shame you only have 1 Decoder.. That would really pin point it. Stop blowing fuses HeHe. Sure, if the Sender did not release the connector there goes the Decoder. Good technique. Oh god memories back to 1970.
Sarah, since you know one end of the offending wire is one side of the SG magnet, why not just pick a working one, connect one side of the SG magnet to the meter, and just scan for continuity on the SF contacts? That would give you one side and observation of the SF contacts should show you the other side.
I don't use meters that much, and I didn't visualize the problem the same way you did. Your solution would have worked, but introducing a meter into the troubleshooting would have made things much worse for me.
@@ConnectionsMuseum No criticism intended, I'm just boggled that you keep so much high complexity in your head, much less keep it straight. I can get tangled following the sequence of a step switch :). I'm glad you folks are keeping this stuff alive :)
@@ConnectionsMuseum Coming at this from another perspective, instead of wiring in a meter, would it have been possible to use a split-core current transformer to tag the grounded wire on either a working or the troubled SG relay? You'd check for blips in the transformer output (induced voltage) when the relay changed state.
Wow what an intense debugging job! Great technique to chase down the problem though. I had not realized how intense and concentrated the wiring behind the frame is, ouch!
Amazing old machines... but I couldn't help but think when you said that magnet was about 30-35 degrees C... imagine the old guys that worked in telco back in the day would have hurled some verbal abuse at anyone heard using such non-patriotic units of measure!😂
Wow. That system is quite a maze to work your way through. I think you’re one of the few people left in the world that understands it so well. I’m always impressed by how you figure things out. A great analyst!
I tie cables like that in the heavy machinery I service, it makes it so much nicer to work on in the future. Funny enough, I picked it up from old telco stuff.
17:25 feels like a far too crude way to search for connections between 48 volts and ground.
Discharge machining the contacts is often not a good way to make things last.
I would have a suitable resistor, a small bridge rectifier, and an LED. Would light up visibly when in circuit, but not cause needless sparks.
(A suitable resistor would be around 10K ohm or so.)
edit:
24:30 is something I thought were a joke. That is a ridiculously large iron for the job. Heating up surrounding wires is a wonderful way to degrade the insulation and cause future shorts as the degraded isolation fails.
I understand trying to be authentic to the period and such. But when it comes to maintaining the machines, it is better to use working practices that helps preserve the machine.
26:08 yes, those are improperly installed cable ties. They should be flush cut such that they won't cut someone's arm open in the future. (I know of companies that has fired people for not doing this properly. It is a liability for the company to ship products that can inflict such preventable injuries.)
26:22 "Does anybody ever do that?" Well, I do. To the point that I fix those that aren't done properly when I see them.
28:50 is a classic. Stiff wires getting pulled about over the years usually dislodges terminal blocks and other stuff over time. Now these days the issue is often cracked solder joints leading to intermittent connections, rather than bent metal terminals shorting out. But the root cause is the same, stiff wires getting forced about through years of service and eventually pulling something along with them..
Reminds me of the wall of relays at my last job.. A 1950s nurse call system in a hospital. The coils had a light wired across them so you would know which relay had power.
Chasing down glitches could be a nightmare..
That looks to be orders of magnitude more complicated than the nurse call system.
Well done Sarah! As an electronics engineer, I can truly applaud your debugging. Had the short been a metallic insect, you would have clearly found and eradicated the BUG ;o)
So much of this reminds me of working on pipe organs. So much of the technology is the same, down to the giant soldering iron you used.
I totally understand strowger, but the panel system and crossbar, are so..... beyond my ken
Same here. Though my experience with common control systems was limited to Western Electric 7A, 7A1 and 7A2 rotary systems then later ESS and digital PBX systems.
@@paulwarner5395 the only element of common control in the UK GPO's strowger system was the directors on the larger cities, (and later register translators for subscriber trunk dialling)... they'd set up the call, then somehow drop out of the connection, something to do with marking a trunk, then a cheaper and less complex selector hunts for it, siezes it.. and the translator/director drops out to be free for another call. Witchcraft, like RF antenna design! LOL
Very interesting Trouble and great troubleshooting, Sarah!
Woah... I've always got the probes out, but I work on 5V digital stuff... I don't get sparks... I'm not sure if I'm jealous or glad. ;)
What would you say to a video answering a fuse alarm, from a major and/or minor initiation to possibly showing the alarm bay to zeroing in by way of aisle lights and panel lights Possibly other types of alarms. This small but very effective tool would demonstrate the conviction to service from the Bell System.
i think we should all go back to lacing, it's really not that hard honestly, and i agree. Cable ties are just horrible, one-use trash. (Except re-usable ones, but they always leave a long tail if you choose not to clip them for that reason)
I do wonder, why keep the extra wires? Do they serve a use just being there? Purely for the look?
+pokes ground to random contacts+ 2 sparks - hey you found the battery :)
Is this what they had in the USA? Were there none of those Strowger switches?
Somebody in your tour bumped their butt up against the rack. That is the initial source of the problem.
I always cut my cable ty's at a 45% angle lol. I'm a retired tech. 41 years of doing that. People will be bleeding for decades. lol
I love this stuff. We lost the character of the phone system when we went digital. I wish this weren't on the other side of the country.
Man I was your place like this existed around where I live
You need a thermal camera. That'll show you where current is going when it shouldn't be
I wonder if a thermal camera could have been useful, following the one wire carrying current for way longer than normal? The newer higher resolution thermal imagers are getting cheaper and cheaper these days, and have enough detail to be pretty useful.
Is Sarah only trying to use techniques and materials that were available when the panel switch was built 100 years ago? No thermal cameras, zip-ties, signal tracers, Velcro, etc.
@@GusFernCa I'm assuming part of the fun is trying to do it as accurately (for the time period) as possible.
Theses panels, Can you actually make a call with this system?
Love the analysis. There are so many tips in your videos. I’m building a step by step switch and this video just taught me how to lace properly. Thank you .. J
Haven’t they used a line ringer to find the wires?
We would drive past an office like this when I was a kid. I peered in as best I could as we drove past-And I would see what looked like a warehouse with rack after rack of stuff like this. Who knew what marvels went on inside!
Busy plug has what component(s) in it? How does it make the piece of equipment unavailable?
The busy plug, will open a contact in the panel or place a ground through a contact via a lead which makes the unit show busy to any additional calls.
MEGA huge soldering iron, I love it
It's so cool to see a mechanical switching office still working. I worked in a DMS100 office for about 10 years but we recently got moved to our call centre office which is a standard cubicle farm so don't get to see the equipment anymore. I just monitor stuff so don't really work on it but it was fun to be in that building and see things get added, changed etc over time.
Awesome you answered some of my questions.
This is my first communications museum video that just happened across my suggestions. I see people calling you Sarah… are you the same Sarah I remember from Cablevision in Shelton? If not you’ve got an East Coast doppelgänger.
Yep, it’s me, Bryce.
Now that you have discovered the terminals that provide that ground, do you go back and note that on the schematic?
Ah, thanks for the reminder!
23:45 c'mon, you've got over 13k subscribers! They're just not hooked on your exchange... yet :D
Nice troubleshooting video. Reverse-engineering an exchange makes electronics rev-eng bleak in comparison.
"The cake is a lie."
Do you have one of those RF wire trace and probe devices? They could make it possible to trace wires without cutting the bundles.
A "ground" would kill the tracing signal, so it wouldn't have gone well. That tech knew that also.
Really interesting to watch.
and think about when they were used in larger settings to be a tech and get that work order.
i don't think they had a few hours to solve the problem.
zip ties are great but when they are cut to short you get that minefield of sharp edges.
so nice to see that cable stitching being done and very important to do.
Great vid for FIRST TIME viewer! Now I gotta learn MORE antique electronics, thanks MA BELL! Just learned how to brew a tetrode from scratch and how to make ic's from sand... Okay, last was not real. Just saying...
As a Telcom engineer calling e1 and b channel/sigtran signaling ancient the marvel of mechanical switches is quite awesome! Then again I also very much like the old skool pinball machines :)
I love the in-depth videos like this, really shows a lot about how things were done and troubleshooted when these switches were still in use.
Great demonstration of lacing. Thanks for showing that. We sold spools of the waxed lacing tape at Vetco but I never got to see how it was done.
BTW, we would never put all that back together till we verified we had completed the repair by testing. Yes, I know you are confident you found a problem, still there could, for example, be more to the problem.
Here for the sewing back up face 😊👍🪡💚
Well that was today's lesson in humility. I started as a hobbyist in electronics at age 3.5, I am now in my late 30s. I spent +13 years working as a research engineer. I had no idea a special cutter existed for cable ties until now.
1. Thank you.
2. How is this tool for such a common use so obscure? I don't even know the name of it.
Is there a advantage (other than cost) of stitching over velcro-type ties?
Easy removal and adding more wires
The switch sounded like it was full of angry bees at the beginning when it couldn't find a sender. Nice video! (I am an old Nortel guy who cut my teeth on a SP-1)
I missed the step where you prove that the system no longer runs out of resources when you run the call generator. (In the spirit of bug tracking & fixing as described in "The pragmatic programmer".)
I have a wonder for you @Connections Museum.
When i search for foults in our relayboxes for railway, we have two ways to do.
With a volt meter och a lamp with a magnet for the ground and the test lead.
Is it not possible to do someting sumular in your applications?
25:53 AGREE
I used to be able to make that happen at my local CO back in the days of switched systems! I would take the CO down for a day or two before they could fix the problem.
Zip-ties are the devil, but when I have to use them, i've found that toenail clippers are a good inexpensive substitute for the special zip tie flush cutter!
If this system is running what specifically is the purpose? Is it actually being used? A bit confused here.
Like
lol
In wiring pipe organ relays I have a saying, "Neatness Counts". It counts in telephone relays as well. Lacing in fun.
Nice to see someone who knows how to lace cable. But, I am one of the people who uses cable ties.. but I do have the proper tool and use it every day :-)
Nice troubleshooting, and good eye. (Also nice to see someone who still knows how to lace properly!)
Great stuff. I remember looking at those banks of cables as a trainee thinking wtf. And the noise!