This is SO COOL! The engineers who made these analog electromechanical switches were really brilliant. Amazing that you are an expert on keeping this equipment running. Thank you for making this video!
An excellent video thank you. The complexity of the Trouble Recorder is amazing. I have enormous respect for the minds that created the concepts and yet, individual design features are in essence very simple. It exemplifies the distinction between complex and complicated! 1,200 error lines earthing from deep within the system and then being displayed in a meaningful form on a card is an enormous feat of design, engineering and wiring. Thank you for explaining the entire machine with such simple clarity. This has been one of my favourite presentations. The cinematography also continues to improve. 😍😍
As a software developer, I have extreme envy for the level of documentation these industrial systems have. That documentation allowed you to go from "I was afraid to break the machine" to "I fixed the machine" through careful study. Nowadays, our electrical "machines" are lucky to come with a thrown together internal Wiki that isn't even correct. You also introduced me to the Ellsworth museum, which is driving distance to me, so thank you so much for that.
For new cards: Have a vector trace of the existing cards made. There is new printing technology which can print and laser cut the cards to shape with notches, etc. so you can order as many as you want.
the cheaper way if you need a lot of cards is to have a professional printer make a stencil so they can print and punch out the cards, the way they're supposed to be made - and if you have a few thousand made it is by far the cheapest method. Most printers require you to order at least a thousand (some places more) before they'll accept an order that require a stencil, but it can be done. Laser cut is fine if you only need a dozen or two cards, but it quickly becomes very expensive (and thus unfeasible for a museum run by volunteers) when you're into hundreds or thousands of cards, and it is very time consuming compared to a proper printing press. Plus laser cut card stock doesn't have the perfect smooth edges you need for these old machines, so they would likely be useless. And by professional printer I mean an actual printer with rotary presses, not one of those shops that set up an oversized laser printer and claim to be a printer
The type of paper and thickness of the paper are critical for making them work in the machine. No company makes that kind of paper anymore so it would have to be special ordered just for this purpose. Punch card paper is special!
@@binaryguru I didn't know that! I remember the paper being a particularly stiff and thin card stock, but never considered it was no longer manufactured. Do you have any reference to a group or something that discusses this? I like research projects like this and have a sneaking suspicion that someone, somewhere in the world could make it, but don't know how I would find the people who still need it for various projects if I could get it organized. is there a board, group, or something for people with this equipment?
I remember when computers were installed into the system. A person would pickup the receiver and instead of getting a dial tone, you'd hear conversations, dial tones, dialing tones all kinds of stuff. One thing, I think was a big problem was programs didn't recognize that old poles lines had a lot of cross talk especially after it rained. So programs had to be altered to deal with these situations.
Sarah 😊... Got a idea, instead of the nuisance of printing expense of replacement trouble cards, make a transparency to overlay a punched yet unprinted card. Thats all.. ;p
Decades ago, we were getting prank phone calls that went on at all hours. Our phone number was on a #5 crossbar switch. Pacific Telephone put a "trap" on our phone line that essentially traced the call, just like you showed on here. The person doing the prank calls seemed to know he was caught, as he calmly said, "Impressive. You got me. Your ringtone sounded funky on the first ring. It dropped a card on me, I just know it." We had two separate phone lines at the time, and I later called from one phone number to the other and listened to see what he was talking about, and he was right. There was weird timber to the first ringtone (heard on the callers side), and then the rest sounded normal. Anyone know why this would have been noticed by the calling party? I'm sure it had to do with the trouble recorder being used to trace the call, but never could figure out why that would happen. Turned out the prank caller was quite sophisticated and according to telco security was "tandem stacking" to avoid getting caught. They only were able to trace it to the local tandem switch and that's where it was lost. Oddly, he never called again once he knew the trap was on there. That was circa 1982.
I'd have to test this to be sure, but my scientific wild ass guess would be: The marker drops the card as the last thing it does before releasing. This means that the marker would connect both halves of the linkage, set up the ringing switch in the trunk, then drop the card and release. It could be possible that the caller was hearing ringback while the marker was still attached to the trunk and generating the card, and something about that sounded different. This might be a little hard to test, because we don't know exactly which trunk circuits the office was using, and what type of markers they were using.
It's been 35 years but I found myself quickly reading the cards you displayed. The missing 2 out of 5 punch was easy since the marker obviously didn't get very far and another one failed at the JXPA/LXPA/SL phase. You didn't show the dreaded turnover card. That usually meant that someone dropped a wire scrap in a number group or translator. Where I worked (Bell Canada) we had only one technician that did routine maintenance on trouble recorders, no one else touched them. While he was working it was stressful for me as a troubleman. The recorder was the only thing that told me how "my" switch was feeling that day. Amazing, at 61 years old I can't remember what I had for breakfast today but I can still remember how to read a crossbar card.
I look at this process of diagnosis and repair, and makes me think of what would have been required to design, build and operate something like Charles Babbage's Difference Engine (had it ever been built). Given this is 1922 technology, you can see things would need to be monitored, errors collected and adjustments made.
This made me wonder just how many (hundreds of?) thousands of trouble cards were generated by just a handful of phone phreaks exploring the network in the 70s and 80s.
I used to drop a bunch of cards intentionally, then a day or so later, go to the C/O and fish them out of the rubbish. Some of the BSPs I had taught me a lot about the card machine.
You may want to show one of the techniques to trace harassing calls. The recipient of harassing calls would be instructed to keep a log of the date and time of all incoming calls (including the harassing ones). The switch is then set up in such a way that every call to the recipient's line drops a card. Once a harassing call was received the log from the recipient is then compared to the cards dropped. The harassing call's card is then used to trace the call. Compare this to a step-by-step office where the call trace would have to be initiated by busying out the recipient's line before the call ended, then traced back through the switch from the connector, through the selectors, all the way to the line finder.
Sara, of all the museum curators in the world, you are my favorite. Yes, I think I understand the concepts you're explaining but . . . when I consider the fraction of a second mechanical timing and logic and other analog intricacies of what may be the pinnacle of 20th century information technology. . . I get a little lost in it all. Human brilliance is more clearly demonstrated here than with computers. The engineers of the 'Greatest Generation' made neat stuff, didn't they?
That was an awesome video! I was playing along with my own trouble recorder card at home. Mine has the “turn over” box punched and not so many holes in it, I wonder what kind of trouble it was? Looking forward to more videos!
Awesomely explained Sarah, thanks for sharing and all the lovely work you do. It's lovely to be able to follow all of the Museum's endeavors from the other side of the world :-D
Thank you for your dedicated attention to the health of this extremely intricate and irreplaceable cross bar switch system. Your video was well presented and clearly explained. Well done, Sarah!
How brave of you to fix the broken trouble recorder. It certainly paid off. That recorder is also very mechanical like the phone system. What blows my mind is that the phone exchange knows every piece of equipment a call is sent to and that information is then punched onto cards if a problem is encountered. How complex of a mechanical machine this is. I guess this sort of information would be useful for tracing numbers of someone making bomb threats or other threatening calls. You really need to have a broad understanding of the whole system to piece this together. Thanks again for another great educational video. Super fantastic.
I love the tester with the nixie tubes. I have built a few clocks with them in it. It just amazes me how everything was done mechanically. I would think a good print shop could make the cards but you would probably need to devise some sort of punch jig to put the proper holes in them. Is there any history on how bell made the cards or if it was farmed out to other companies that might still have the equipment? I would guess in its hay day they had to be able to produce massive amounts of them all over the world.
I worked for Bell Canada on a 4AXB Toll Switch and became very good at fixing and maintaining our trouble recorder . (Nobody else wanted the job as it was very tedious and time consuming and they didn't have the patience for it)
You make some truly fantastic videos! I have always wondered how the billing of calls happened with these systems, have you ever made a video about how that works? If not, could you make a video about how calls are recorded and billed?
Excellent video. I worked SxS cama in the late 70's and early 80's. Our trouble recorder had the yellow roll of paper type. I once had a trunk that would drop a trouble ticket with no apparent reason for years. After I used a automatic test set, (the one with the pegs to program telco no's), I noticed the cama inc. Trunk was having issues with call abandonment. After utilizing the test frame and reading the sequence drawing for the trunk and cama sender, I found a wiring error. Those are the days I miss--- the good ole Bell System. Wished I could have worked panel and crossbar. Anyways, thank you so much. Very interesting.
Is this what happened in the 2000 presidential election with hanging chads? Kidding aside this electromechanical recorder is incredible now if I watch your well done video a few more times I’ll start to understand.
I love making new parts and refurbishing old ones for mechanisms like this. Fabulous to see the inner workings of the exchange equipment I used to work with back in the 70s. Splendid video
I hired on with Southern Bell in 1962. I worked step, number 5 crossbar and electronic toll in 38 years. I liked and enjoyed working number 5 crossbar the best. Are there and 5X bar offices left in the USA. Today I taught my Great grand daughter age 16 how to use a dial phone. 1919 brass candle stick phone.
I repaired a few Northern Electric #5 Crossbar trouble recorders when I worked in those offices over 40 years ago now. They were exactly the same as the Western Electric one you have there.
I worked a a switchman on #5 xbar in the 1970's for about 4 years....thank gawd I never any problems on the trouble recorder.....all the trouble cards punched were taken to the recycle joint and paid for our coffee fund....
Do you know why the choice was made to make the debug bus parallel? Why wasn't an approach similar to JTAG or Boundary Scan used where each relay has shift-in and shift-out ports, relays are wired in daisy-chain fashion, and the debug status is shifted out one relay at a time? Did the designers really not mind huge 1000+ bundles of wires snaking all over the frame? That seems like it would be much more susceptible to installation error. Assuming a different type of card puncher, this also could have greatly simplified the relays and circuitry required to punch a card (e.g. no latching or less latching required)?
Data flow in electromechanical switches is generally parallel. Relays don't move fast enough to pump 1200 bits serially without taking a long time. In addition, the extra logic required to do a serial shift-thing would take many more relays, which are expensive, but the parallel approach means that we can just pick up signals where we need them and carry them across the room. It seems like a lot from a modern perspective, but in a telephone switch, 1000+ wires is not that big of a deal. It’s just how things were done. In a normal central office like this, it would be commonplace to have hundreds of thousands or millions of individual wires. Their installation forces were highly skilled workers and before the job was done, a complete set of acceptance tests had to be performed, which ensured that each individual function of the machine was working to specification. It’s all part of what makes these machines so impressive! These designers were extremely good at designing insanely complex relay circuits and making them ultra reliable. Systems like this are where the concept of 99.999% uptime originated.
On the first Trouble Recorder with the bad shaft,You should be able to remove with the gear and have the shaft made by a machinist or to a good auto rebuilder if it can be repaired. Those are pretty complicated for the day. I grew up with classic cars and power equipment which was easy for me to understand.
My Dad worked in the industry when these were still in use. As a child I fondly remember when my dad would bring home defective panels and components for me to play with and disassemble. I had disassembled some part he had gave me and thousands of the little paper punches fell out and my dad told me that the small paper dots were called "paper chads". Don't know why, but I always remembered that. Thanks for your videos. They bring me back to my childhood and give me a very warm feeling.
I am always in awe that people designed, built, and maintained these amazing complex machines. Also amazed at the amount of work you put in to keep them running or get them running for a museum! Thanks for your commitment to preserving this bit of history.
Would it be possible for you guys to have an outside line connect to the Crossbar like sxsphil does with his Step system? Dial it to get dial tone and then have a few numbers available going to busy signals/an announcing machine or some other interesting sounding intercepts? Maybe even have one that goes to an actual phone in the museum so if any of the volunteers or eventual visitors are in the mood can pick up and talk to random people? I'm in Europe so that coupled with the pandemic means I won't be able to come in person any time soon and I'm sure many other people would really enjoy being able to use the electromechanical switches remotely. Kind of like a remote museum visit :)
Yeah, you can call in via CNET or NPSTN to either of our crossbars. See their directories for our DISA numbers. Visit witch.spmh.us/disa for a list of fun numbers to call.
There a really cool app for the iPhone called Phonal Tonal that will give you all the MF tones, ring tones, dial tones, etc. I miss the days of phone freaking and loop lines.
Love it! But you didnt show us the chad hopper where all the punched bits go. ;-) I restored a asr33 teletype once and it was FULL of paper dots all gunked into every cranny. Thanks for the great vids!
Great video! I always loved hearing the dial tone from a #5 crossbar CO. I was once in the central office on Mother’s day in 1972. Lots of switching sounds!
Sarah, you did a great job of explaining all the complexities of the system. The graphics you provided along with the short video clips, showing the equipment in action worked very well together. Thank you for your dedication to learning how it all works and being able to explain it so very well. Hope you are doing well. The level of engineering involved to make the trouble recorder work and the cards to be printed properly is amazing. A lot of challenges had to be overcome; not the least of which was the sheer volume of connections required. Plus, if I'm not mistaken, all of those relays had to have contacts dedicated to indicating their status for the trouble recorder. The clever engineering of the recorder itself, moving the individual bars over so that the rotating shaft would strike them, and then the punch, reminds me of the similar tactic used for the crossbar switch and the skinny wire. Clever and creative engineering.
I wonder, if old telephone exchange machinery like this could be the most complex electromechanical devices ever built? 🙈 It is absolutely mindblowing, how all of this works and how people solved technical problems back then! And also it is awsome that you were able to fix this card punching device, Sarah! Keep up the great work of showing people all of this technology! Greetings! 😊
I went to a school for a week to repair and adjust the trouble recorder. We had a very heavy duty adjustable steel table with a a spare recorder. If the operating one failed, it could be unplugged and the spare rolled in and exchanged quickly. Despite its complexity and frequent use it never failed on us.
We're so used to digital electronics that it's shocking to see someone servicing telephone equipment like it's a steam locomotive. It's amazing how the telephone network operated for so long in the pre-digital era. Hat's off to the pioneers!
Great video with a lot of detail. I remember intentionally dropping cards by flashing the switchhook 11 times for the last digit and this would punch 3 out of 5 (as a digit 11). Dialing A, B, C or D with a 16 button touchtone keypad would do the same thing. Another way was to dial something that caused "no route" trouble such as 0+ringback number. Dial administration didn't consider that someone might try to call the ringback number through TSPS (which would be impossible anyway). To get more blank cards, you might consult with a printing company. I know they can get blank card stock, cut it to size and laser print on it but the hard part is the little notches and holes which would require a custom punch. The cost of setting this up would be high but you could order in a large quantity to justify the cost. Something else I thought of is a "virtual machine". A computer that would interface with the card machine and display a virtual image of the card, or just the raw date. Not as fun though.
Thanks for the suggestions! Yeah, we are looking into getting some cards printed, and it seems like the tooling setup would be the majority of the cost. The difficulty with the tooling is that the notches and cutout dimensions are critical, so it would have to be pretty darn close. The 5XB also has a computerized card function built right in. In the late 1970s, as offices were staffed less and less, they wanted to centralize all of the maintenance facilities, so they built an adjunct to the trouble recorder called the MDT/PROCON. It sends trouble indications to a centralized PDP-11 with a bunch of modems attached to it. Ours worked, sort of, but then it died, and we don't have the maintenance tools for it.
@@ConnectionsMuseum I was thinking the same thing, the tooling setup for the notches and cutouts would be critical. Even variations in card stock could cause problems. I can remember when they did more centralized maintenance. In the East Bay area of California, they had one C/O that took care of maintenance and trouble indications from surrounding C/Os would be reported to the central one. I could tell when the trouble recorder was taken offline because when it would drop a card, it would normally freeze a line for about 7 seconds (3.5 seconds per trial). When offline, the line was frozen for only about a second or two before getting reorder. You can hear a little "tick" in between the first and second trial. I also remember some mini computers being used on #5 crossbar in the early 1980s. One had a prompt "MONITOR RUNNING STATE REQUEST" which I think let you look at battery voltages and a few other things. CAROT was another mini computer for automated trunk transmission testing. If the adjunct to the trouble recorder used a serial port (would be serial if modems were used), then it would be easy to interface that to USB port on a laptop and write a program to handle the data.
It would be work, but you could take a picture or scan of a card to see where the holes are and interpret them. Then it wouldn't matter how they were printed. Simpler would be to have a microprocessor record the states of the relays that drive the punches. My third comment tonight. Can you tell I'm having trouble falling asleep?
with todays 3d printers and lathes you could make new parts for the recorder. you maybe able to call up a ticket factory (a factory that makes event tickets, lunch tickets for schools and arcade tickets for arcades that use tickets that are used as points for buying prizes at the arcade's shop) and ask them to make you new cards. or if the card punches are not pre scored you can just feed blank card stock say cut from food boxes and make your own punch cards.
For the cards I'd just look at the nearest printshop. They will at least be able to point you to the right direction. It shouldn't be to hard to get something like that printed, and as for paper properties they will likely be able to send you some stock of paper that's simmilar to the one in use.
Lightning has struck my brain and generated a weird question: Does running a test on the MTF seize the customer endpoint equipment, or does it limit on either end to just telco side, or is that an option you can set on the MTF panel?
I've never used them but smartpress has what they call digital laser cutting. They may be able to accommodate your cuts. Another option would be to contact b something like Boise cascade corrugated or Volm Corporation.
For BC/Volm there may be substantial fees to create the dies...I then again you might get surprised and find out who made them originally and they've still got them in a warehouse....I don't hardly know where to start but there might be clues on or in those boxes you've got as to whom had the original cutting dies
Very cool and amazing machinery! I really enjoy your videos about this old phone equipment. I can only imagine the challenges with having a "diagnostic" system with this many moving parts. It would be somewhat hard to know if the diagnostic system its self was having problems, or if it was indicating a system problem! I have a great desire to save and restore other vintage industrial and refrigeration equipment, which seems to be a very niche interest. My real job is commissioning the current version of what I love to restore. I would expect that your day-job is related to this industry? Regardless, please keep up the videos.
Can anybody comment? I was an ex crossbar 5 phreaker from many decades ago. In my collection I have both single-sided and double-sided trouble ticket cards, are the different cards dependent on the type of setup in a particular office, ie, the single-sided cards used in smaller, less complex crossbar 5 offices, and the double-sided card in larger offices with many Centrex cx's? Or, otherwise, is the single-sided card always the incorrect card, because you want to be using the double-sided card, or were they from different offices.?
There were several generations of cards. The first ones were all single-sided. Later they added double-sided to more complex offices. One of the main factors that contributed to an office using double-sided cards was if it had LAMA capability--that is, local billing for long distance calls. If the billing hardware was in the tandem office, the 5XB did not need the extra data points provided by two-sided cards.
@@ConnectionsMuseum : Thank you for the speedy reply. I find your videos educational, informative and very entertaining. I continue to watch the "troubleshooting problems in #1 crossbar office" over and over again. I would enjoy seeing more crossbar 5 videos, if possible. Because that's the type that was in use in the Maryland area where I grew up, when I did my phreaking as a teenager. I realize that doing a video about a number 5 crossbar marker would be quite boring, as it is basically a huge collection of multi contact relays, but that stuff really interests me. ROCK ON, SARAH!!!! Keep up the great work 💪!
Yes, definitely. That's actually on my list, but I really want to get billing working in the 5XB in order to show it in the video. It's really cool, but its going to take a few more months of work before its ready.
Its fun to watch what I read about in the 1980s to know I have knowledge that only I can be glad I know . This was truly the age of hands on equipment. Today's systems are just not as fascinating to even watch cause there isn't anything to watch. Semi conductors are nothing like where we started with. This is a classic time of ALL COPPER WIRE USE THAT WILL NEVER BE THE SAME
All i can say is... wow. I would think building a 3ess/5ess wouldve been on an order of magnitude easier than all the mechanical components. Kudos to those who engineered AND built these.. AND those who continue to keep them running, Sara.. wow.. just wow, the level of your understanding is epic. My hat's off.....
It looks like the rear recorder--the one not reparable--had some bad mechanical parts. There are so many knowlegeable machinists (professional and amateur) just on YT alone, that it would not be impossible to locate someone who would volunteer to duplicate worn or damaged parts. Perhaps it would be worth doing an episode on exactly what you need to get the thing going again. If it has been possible for computer fanboys to reverse engineer and reproduce bits to get authentic Apollo capsule gear working again, why not a trouble recorder?
This is SO COOL! The engineers who made these analog electromechanical switches were really brilliant. Amazing that you are an expert on keeping this equipment running. Thank you for making this video!
An excellent video thank you.
The complexity of the Trouble Recorder is amazing. I have enormous respect for the minds that created the concepts and yet, individual design features are in essence very simple. It exemplifies the distinction between complex and complicated!
1,200 error lines earthing from deep within the system and then being displayed in a meaningful form on a card is an enormous feat of design, engineering and wiring.
Thank you for explaining the entire machine with such simple clarity. This has been one of my favourite presentations. The cinematography also continues to improve.
😍😍
As a software developer, I have extreme envy for the level of documentation these industrial systems have. That documentation allowed you to go from "I was afraid to break the machine" to "I fixed the machine" through careful study. Nowadays, our electrical "machines" are lucky to come with a thrown together internal Wiki that isn't even correct.
You also introduced me to the Ellsworth museum, which is driving distance to me, so thank you so much for that.
If you go to Ellsworth, ask for Martin and let him know that Sarah sent you :)
For new cards: Have a vector trace of the existing cards made. There is new printing technology which can print and laser cut the cards to shape with notches, etc. so you can order as many as you want.
the cheaper way if you need a lot of cards is to have a professional printer make a stencil so they can print and punch out the cards, the way they're supposed to be made - and if you have a few thousand made it is by far the cheapest method. Most printers require you to order at least a thousand (some places more) before they'll accept an order that require a stencil, but it can be done. Laser cut is fine if you only need a dozen or two cards, but it quickly becomes very expensive (and thus unfeasible for a museum run by volunteers) when you're into hundreds or thousands of cards, and it is very time consuming compared to a proper printing press. Plus laser cut card stock doesn't have the perfect smooth edges you need for these old machines, so they would likely be useless. And by professional printer I mean an actual printer with rotary presses, not one of those shops that set up an oversized laser printer and claim to be a printer
The type of paper and thickness of the paper are critical for making them work in the machine. No company makes that kind of paper anymore so it would have to be special ordered just for this purpose. Punch card paper is special!
@@binaryguru I didn't know that! I remember the paper being a particularly stiff and thin card stock, but never considered it was no longer manufactured. Do you have any reference to a group or something that discusses this? I like research projects like this and have a sneaking suspicion that someone, somewhere in the world could make it, but don't know how I would find the people who still need it for various projects if I could get it organized. is there a board, group, or something for people with this equipment?
I absolutely loved the "Oh, hi!" intro. It made me literally laugh out loud. Well done!
I just discovered this channel. This is soo cool!
I remember when computers were installed into the system. A person would pickup the receiver and instead of getting a dial tone, you'd hear conversations, dial tones, dialing tones all kinds of stuff. One thing, I think was a big problem was programs didn't recognize that old poles lines had a lot of cross talk especially after it rained. So programs had to be altered to deal with these situations.
Sarah 😊... Got a idea, instead of the nuisance of printing expense of replacement trouble cards, make a transparency to overlay a punched yet unprinted card.
Thats all.. ;p
Now I feel old for listening to our neighbors calls on our party line in da' day
Decades ago, we were getting prank phone calls that went on at all hours. Our phone number was on a #5 crossbar switch. Pacific Telephone put a "trap" on our phone line that essentially traced the call, just like you showed on here. The person doing the prank calls seemed to know he was caught, as he calmly said, "Impressive. You got me. Your ringtone sounded funky on the first ring. It dropped a card on me, I just know it." We had two separate phone lines at the time, and I later called from one phone number to the other and listened to see what he was talking about, and he was right. There was weird timber to the first ringtone (heard on the callers side), and then the rest sounded normal. Anyone know why this would have been noticed by the calling party? I'm sure it had to do with the trouble recorder being used to trace the call, but never could figure out why that would happen. Turned out the prank caller was quite sophisticated and according to telco security was "tandem stacking" to avoid getting caught. They only were able to trace it to the local tandem switch and that's where it was lost. Oddly, he never called again once he knew the trap was on there. That was circa 1982.
I'd have to test this to be sure, but my scientific wild ass guess would be:
The marker drops the card as the last thing it does before releasing. This means that the marker would connect both halves of the linkage, set up the ringing switch in the trunk, then drop the card and release. It could be possible that the caller was hearing ringback while the marker was still attached to the trunk and generating the card, and something about that sounded different. This might be a little hard to test, because we don't know exactly which trunk circuits the office was using, and what type of markers they were using.
@@ConnectionsMuseum Awesome! Thank you so much for that information. I've waited 39-years for that, LOL!
It's been 35 years but I found myself quickly reading the cards you displayed. The missing 2 out of 5 punch was easy since the marker obviously didn't get very far and another one failed at the JXPA/LXPA/SL phase. You didn't show the dreaded turnover card. That usually meant that someone dropped a wire scrap in a number group or translator.
Where I worked (Bell Canada) we had only one technician that did routine maintenance on trouble recorders, no one else touched them. While he was working it was stressful for me as a troubleman. The recorder was the only thing that told me how "my" switch was feeling that day.
Amazing, at 61 years old I can't remember what I had for breakfast today but I can still remember how to read a crossbar card.
@@michaelrees1376wow your amazing! Thanks for the comment loved reading it knowing that in those CO's good minds were at work!
Yes,!!!
It’s crazy how much thoughts went into designing this mechanism driven purely by relay logic. That’s some really impressive engineering.
I look at this process of diagnosis and repair, and makes me think of what would have been required to design, build and operate something like Charles Babbage's Difference Engine (had it ever been built). Given this is 1922 technology, you can see things would need to be monitored, errors collected and adjustments made.
This made me wonder just how many (hundreds of?) thousands of trouble cards were generated by just a handful of phone phreaks exploring the network in the 70s and 80s.
I used to drop a bunch of cards intentionally, then a day or so later, go to the C/O and fish them out of the rubbish. Some of the BSPs I had taught me a lot about the card machine.
And how many kids playing with the hook switch and dialing random numbers?
That would be me
You may want to show one of the techniques to trace harassing calls. The recipient of harassing calls would be instructed to keep a log of the date and time of all incoming calls (including the harassing ones). The switch is then set up in such a way that every call to the recipient's line drops a card. Once a harassing call was received the log from the recipient is then compared to the cards dropped. The harassing call's card is then used to trace the call. Compare this to a step-by-step office where the call trace would have to be initiated by busying out the recipient's line before the call ended, then traced back through the switch from the connector, through the selectors, all the way to the line finder.
Sara, of all the museum curators in the world, you are my favorite. Yes, I think I understand the concepts you're explaining but . . . when I consider the fraction of a second mechanical timing and logic and other analog intricacies of what may be the pinnacle of 20th century information technology. . . I get a little lost in it all. Human brilliance is more clearly demonstrated here than with computers. The engineers of the 'Greatest Generation' made neat stuff, didn't they?
That was an awesome video! I was playing along with my own trouble recorder card at home. Mine has the “turn over” box punched and not so many holes in it, I wonder what kind of trouble it was? Looking forward to more videos!
Thanks for the tour! I'd love visit your museum someday. This was a great explanation of how this all worked.
Glad you enjoyed it!
Yours looks like a fun job, and you seem to really enjoy it! I 'd love to have a job like that.
Awesomely explained Sarah, thanks for sharing and all the lovely work you do. It's lovely to be able to follow all of the Museum's endeavors from the other side of the world :-D
Ditto, Sarah! Nicely done vid and explanation. Bloody amazing tech!
Thank you for your dedicated attention to the health of this extremely intricate and irreplaceable cross bar switch system. Your video was well presented and clearly explained. Well done, Sarah!
How brave of you to fix the broken trouble recorder. It certainly paid off. That recorder is also very mechanical like the phone system. What blows my mind is that the phone exchange knows every piece of equipment a call is sent to and that information is then punched onto cards if a problem is encountered. How complex of a mechanical machine this is. I guess this sort of information would be useful for tracing numbers of someone making bomb threats or other threatening calls. You really need to have a broad understanding of the whole system to piece this together.
Thanks again for another great educational video. Super fantastic.
I love the tester with the nixie tubes. I have built a few clocks with them in it. It just amazes me how everything was done mechanically. I would think a good print shop could make the cards but you would probably need to devise some sort of punch jig to put the proper holes in them. Is there any history on how bell made the cards or if it was farmed out to other companies that might still have the equipment? I would guess in its hay day they had to be able to produce massive amounts of them all over the world.
I worked for Bell Canada on a 4AXB Toll Switch and became very good at fixing and maintaining our trouble recorder . (Nobody else wanted the job as it was very tedious and time consuming and they didn't have the patience for it)
Who would have known the Trouble Recorder can write in "loose script" notes in the left/right side margins ;)
You make some truly fantastic videos! I have always wondered how the billing of calls happened with these systems, have you ever made a video about how that works? If not, could you make a video about how calls are recorded and billed?
A billing video is on our list for sure. Just want to get the billing mechanism working in the 5XB first, so that we can talk about it.
What a well-produced video. I especially loved the slo-mo parts. Kudos! Looking forward to the next time I can visit Seattle.
Excellent video. I worked SxS cama in the late 70's and early 80's. Our trouble recorder had the yellow roll of paper type. I once had a trunk that would drop a trouble ticket with no apparent reason for years. After I used a automatic test set, (the one with the pegs to program telco no's), I noticed the cama inc. Trunk was having issues with call abandonment. After utilizing the test frame and reading the sequence drawing for the trunk and cama sender, I found a wiring error. Those are the days I miss--- the good ole Bell System. Wished I could have worked panel and crossbar. Anyways, thank you so much. Very interesting.
We just got our paper tape trouble ticketer working in the No.1 Crossbar. In our machine it handles trouble tickets for ANI/CAMA.
@@ConnectionsMuseum excellent. Hope you will make a video on this.
It was sooo kool to see an old Panel trouble ticket, made my day!
Is this what happened in the 2000 presidential election with hanging chads?
Kidding aside this electromechanical recorder is incredible now if I watch your well done video a few more times I’ll start to understand.
I have a question. What if a subscriber wants to change his number? How is that done?
I love your video’s. Excellence !
Sarah GOOD video. Where do you get blank cards these days? Those cards always reminded me of IBM punch cards. ()()()
Just watch the whole video (or at least the last 1,5 minutes of it) and you will figure it out :)
@@rpriesol Thanks.
I love making new parts and refurbishing old ones for mechanisms like this. Fabulous to see the inner workings of the exchange equipment I used to work with back in the 70s. Splendid video
I hired on with Southern Bell in 1962. I worked step, number 5 crossbar and electronic toll in 38 years. I liked and enjoyed working number 5 crossbar the best. Are there and 5X bar offices left in the USA. Today I taught my Great grand daughter age 16 how to use a dial phone. 1919 brass candle stick phone.
I repaired a few Northern Electric #5 Crossbar trouble recorders when I worked in those offices over 40 years ago now. They were exactly the same as the Western Electric one you have there.
Those where the good days on the seventh floor
@@verybusy8728 And Elbow Park, Huntingdon Hills, Bonavista, and even for a short while in Peace River.
Love your videos! I miss my #5 XBAR days.
I worked a a switchman on #5 xbar in the 1970's for about 4 years....thank gawd I never any problems on the trouble recorder.....all the trouble cards punched were taken to the recycle joint and paid for our coffee fund....
I'm in total amazement of the Engineering abilities of Bell Labs and the manufacturing capabilities of Western Electric. 👍😎👍
I literally can't get enough of this. Great commentary too, good job hon.
2:15 is that keypad in a sense a Silver box? Basically, all the standard DTMF tones plus A, B, C, and D?
Do you know why the choice was made to make the debug bus parallel? Why wasn't an approach similar to JTAG or Boundary Scan used where each relay has shift-in and shift-out ports, relays are wired in daisy-chain fashion, and the debug status is shifted out one relay at a time? Did the designers really not mind huge 1000+ bundles of wires snaking all over the frame? That seems like it would be much more susceptible to installation error. Assuming a different type of card puncher, this also could have greatly simplified the relays and circuitry required to punch a card (e.g. no latching or less latching required)?
Data flow in electromechanical switches is generally parallel. Relays don't move fast enough to pump 1200 bits serially without taking a long time. In addition, the extra logic required to do a serial shift-thing would take many more relays, which are expensive, but the parallel approach means that we can just pick up signals where we need them and carry them across the room.
It seems like a lot from a modern perspective, but in a telephone switch, 1000+ wires is not that big of a deal. It’s just how things were done. In a normal central office like this, it would be commonplace to have hundreds of thousands or millions of individual wires. Their installation forces were highly skilled workers and before the job was done, a complete set of acceptance tests had to be performed, which ensured that each individual function of the machine was working to specification. It’s all part of what makes these machines so impressive!
These designers were extremely good at designing insanely complex relay circuits and making them ultra reliable. Systems like this are where the concept of 99.999% uptime originated.
@@ConnectionsMuseum Thank you for the insight!
On the first Trouble Recorder with the bad shaft,You should be able to remove with the gear and have the shaft made by a machinist or to a good auto rebuilder if it can be repaired. Those are pretty complicated for the day. I grew up with classic cars and power equipment which was easy for me to understand.
The secret life of telephone exchanges!
I'd be very surprise3d if Tim Hunkin isn't a personal hero to you!
Heck yeah, I love The Secret Life of Machines. That and "Connections" by James Burke, are awesome to watch :)
What about phone phreakers and the countermeasures? QT
My Dad worked in the industry when these were still in use. As a child I fondly remember when my dad would bring home defective panels and components for me to play with and disassemble. I had disassembled some part he had gave me and thousands of the little paper punches fell out and my dad told me that the small paper dots were called "paper chads". Don't know why, but I always remembered that. Thanks for your videos. They bring me back to my childhood and give me a very warm feeling.
Dotcels seething over paperchads
I am always in awe that people designed, built, and maintained these amazing complex machines. Also amazed at the amount of work you put in to keep them running or get them running for a museum! Thanks for your commitment to preserving this bit of history.
Would it be possible for you guys to have an outside line connect to the Crossbar like sxsphil does with his Step system?
Dial it to get dial tone and then have a few numbers available going to busy signals/an announcing machine or some other interesting sounding intercepts? Maybe even have one that goes to an actual phone in the museum so if any of the volunteers or eventual visitors are in the mood can pick up and talk to random people?
I'm in Europe so that coupled with the pandemic means I won't be able to come in person any time soon and I'm sure many other people would really enjoy being able to use the electromechanical switches remotely. Kind of like a remote museum visit :)
Yeah, you can call in via CNET or NPSTN to either of our crossbars. See their directories for our DISA numbers. Visit witch.spmh.us/disa for a list of fun numbers to call.
There a really cool app for the iPhone called Phonal Tonal that will give you all the MF tones, ring tones, dial tones, etc. I miss the days of phone freaking and loop lines.
This is fascinating. You explain so well.
Love it! But you didnt show us the chad hopper where all the punched bits go. ;-) I restored a asr33 teletype once and it was FULL of paper dots all gunked into every cranny. Thanks for the great vids!
...imagine you would have to regularly 'change the oil' on todays network/voip switch equipment... o-m-g ^_^
Great video! I always loved hearing the dial tone from a #5 crossbar CO. I was once in the central office on Mother’s day in 1972. Lots of switching sounds!
Sarah, you did a great job of explaining all the complexities of the system. The graphics you provided along with the short video clips, showing the equipment in action worked very well together. Thank you for your dedication to learning how it all works and being able to explain it so very well. Hope you are doing well.
The level of engineering involved to make the trouble recorder work and the cards to be printed properly is amazing. A lot of challenges had to be overcome; not the least of which was the sheer volume of connections required. Plus, if I'm not mistaken, all of those relays had to have contacts dedicated to indicating their status for the trouble recorder.
The clever engineering of the recorder itself, moving the individual bars over so that the rotating shaft would strike them, and then the punch, reminds me of the similar tactic used for the crossbar switch and the skinny wire. Clever and creative engineering.
I wonder, if old telephone exchange machinery like this could be the most complex electromechanical devices ever built? 🙈 It is absolutely mindblowing, how all of this works and how people solved technical problems back then! And also it is awsome that you were able to fix this card punching device, Sarah! Keep up the great work of showing people all of this technology! Greetings! 😊
The IBM Accounting machines (402, 403, 407...) were relay logic. Not as many relays as the switch in this video though.
I went to a school for a week to repair and adjust the trouble recorder. We had a very heavy duty adjustable steel table with a a spare recorder. If the operating one failed, it could be unplugged and the spare rolled in and exchanged quickly.
Despite its complexity and frequent use it never failed on us.
Given the complexity of it, I assume there is a Trouble Recorder for the Trouble Recorder 😁
awesome! Operation of the marker has always been a “gap” in explanations of old phone equipment so this is great!!!
We're so used to digital electronics that it's shocking to see someone servicing telephone equipment like it's a steam locomotive. It's amazing how the telephone network operated for so long in the pre-digital era. Hat's off to the pioneers!
I never know that tech support term of "tickets" originated from literal ticket printed out by trouble recorders that print out tickets.
We went from this, to not even be able to swap our phone batteries anymore... I swear the world is evolving backwards again...
This is an exceptionally great video.Beautifully filmed, edited and explained. Well done, Sarah!
Fascinating! I imagine these were not infrequently used to aid "debugging" in the most literal sense - critters in relays, etc!
Did you ever get any luck on having more cards printed? I'm sure there is a company that does custom paper form creation
Great video with a lot of detail. I remember intentionally dropping cards by flashing the switchhook 11 times for the last digit and this would punch 3 out of 5 (as a digit 11). Dialing A, B, C or D with a 16 button touchtone keypad would do the same thing. Another way was to dial something that caused "no route" trouble such as 0+ringback number. Dial administration didn't consider that someone might try to call the ringback number through TSPS (which would be impossible anyway).
To get more blank cards, you might consult with a printing company. I know they can get blank card stock, cut it to size and laser print on it but the hard part is the little notches and holes which would require a custom punch. The cost of setting this up would be high but you could order in a large quantity to justify the cost.
Something else I thought of is a "virtual machine". A computer that would interface with the card machine and display a virtual image of the card, or just the raw date. Not as fun though.
Thanks for the suggestions! Yeah, we are looking into getting some cards printed, and it seems like the tooling setup would be the majority of the cost. The difficulty with the tooling is that the notches and cutout dimensions are critical, so it would have to be pretty darn close.
The 5XB also has a computerized card function built right in. In the late 1970s, as offices were staffed less and less, they wanted to centralize all of the maintenance facilities, so they built an adjunct to the trouble recorder called the MDT/PROCON. It sends trouble indications to a centralized PDP-11 with a bunch of modems attached to it. Ours worked, sort of, but then it died, and we don't have the maintenance tools for it.
@@ConnectionsMuseum I was thinking the same thing, the tooling setup for the notches and cutouts would be critical. Even variations in card stock could cause problems.
I can remember when they did more centralized maintenance. In the East Bay area of California, they had one C/O that took care of maintenance and trouble indications from surrounding C/Os would be reported to the central one. I could tell when the trouble recorder was taken offline because when it would drop a card, it would normally freeze a line for about 7 seconds (3.5 seconds per trial). When offline, the line was frozen for only about a second or two before getting reorder. You can hear a little "tick" in between the first and second trial. I also remember some mini computers being used on #5 crossbar in the early 1980s. One had a prompt "MONITOR RUNNING STATE REQUEST" which I think let you look at battery voltages and a few other things. CAROT was another mini computer for automated trunk transmission testing.
If the adjunct to the trouble recorder used a serial port (would be serial if modems were used), then it would be easy to interface that to USB port on a laptop and write a program to handle the data.
It would be work, but you could take a picture or scan of a card to see where the holes are and interpret them. Then it wouldn't matter how they were printed.
Simpler would be to have a microprocessor record the states of the relays that drive the punches.
My third comment tonight. Can you tell I'm having trouble falling asleep?
with todays 3d printers and lathes you could make new parts for the recorder.
you maybe able to call up a ticket factory (a factory that makes event tickets, lunch tickets for schools and arcade tickets for arcades that use tickets that are used as points for buying prizes at the arcade's shop) and ask them to make you new cards.
or if the card punches are not pre scored you can just feed blank card stock say cut from food boxes and make your own punch cards.
Digits always end in a 7 you say? That explains one of Evan Doorbell's observations in his phreaking narration...
If you haven't fixed that shaft 1 year after posting this video that looks like a job Abom79 could sort for you!
Well explained. Good job.
You must have been a CO doctor in a past life. You don't have near enough gray hair to know all that.
do you use all this for your actual calls, or do you use something like a VoIP System? also you have a tiktok?
Nice job getting this machine working!
Holy cow, it's like describing the depths of the Krell technology. 🤯
Can you still place a phone call utilizing these marvelous mechanical monstrosities?
You mentioned that decoders have "translations". Can you explain how that works? I am genuinely curious.
Absolutely mind blowing. Insanely awesome.
For the cards I'd just look at the nearest printshop. They will at least be able to point you to the right direction. It shouldn't be to hard to get something like that printed, and as for paper properties they will likely be able to send you some stock of paper that's simmilar to the one in use.
Excellent video. Thanks!
Nice restoration project thanks for sharing. Regards Chris
Thanks Sarah
Lightning has struck my brain and generated a weird question: Does running a test on the MTF seize the customer endpoint equipment, or does it limit on either end to just telco side, or is that an option you can set on the MTF panel?
You can choose via a key on the test panel.
Excellent explanation of a very complicated machine! I love the work you do! A+ 👍👍
Worked 4A Card Translator, some of the systems mentioned are similar.
In CANTV Venezuela, loss call loos call
And here I am trying to figure out the relays on my sxs connector switch
How many calls per oil change do you get? :-‘P
You are the smartest person on Earth! Awesome video
Awesome explanations !
The 5XB is almost a computer.
I've never used them but smartpress has what they call digital laser cutting. They may be able to accommodate your cuts. Another option would be to contact b something like Boise cascade corrugated or Volm Corporation.
For BC/Volm there may be substantial fees to create the dies...I then again you might get surprised and find out who made them originally and they've still got them in a warehouse....I don't hardly know where to start but there might be clues on or in those boxes you've got as to whom had the original cutting dies
Very cool and amazing machinery! I really enjoy your videos about this old phone equipment. I can only imagine the challenges with having a "diagnostic" system with this many moving parts. It would be somewhat hard to know if the diagnostic system its self was having problems, or if it was indicating a system problem!
I have a great desire to save and restore other vintage industrial and refrigeration equipment, which seems to be a very niche interest. My real job is commissioning the current version of what I love to restore. I would expect that your day-job is related to this industry? Regardless, please keep up the videos.
yayyyy! new post!
Can anybody comment? I was an ex crossbar 5 phreaker from many decades ago. In my collection I have both single-sided and double-sided trouble ticket cards, are the different cards dependent on the type of setup in a particular office, ie, the single-sided cards used in smaller, less complex crossbar 5 offices, and the double-sided card in larger offices with many Centrex cx's? Or, otherwise, is the single-sided card always the incorrect card, because you want to be using the double-sided card, or were they from different offices.?
There were several generations of cards. The first ones were all single-sided. Later they added double-sided to more complex offices. One of the main factors that contributed to an office using double-sided cards was if it had LAMA capability--that is, local billing for long distance calls.
If the billing hardware was in the tandem office, the 5XB did not need the extra data points provided by two-sided cards.
@@ConnectionsMuseum : Thank you for the speedy reply. I find your videos educational, informative and very entertaining. I continue to watch the "troubleshooting problems in #1 crossbar office" over and over again. I would enjoy seeing more crossbar 5 videos, if possible. Because that's the type that was in use in the Maryland area where I grew up, when I did my phreaking as a teenager. I realize that doing a video about a number 5 crossbar marker would be quite boring, as it is basically a huge collection of multi contact relays, but that stuff really interests me.
ROCK ON, SARAH!!!! Keep up the great work 💪!
Nice video. Similar to 4A CT
Thanks once again, enjoyed this video
😶🌫️ Punch & Judy
Oh my, this is way above me
Could you please do a video on how billing was handled? How were the calls recorded and charged to customers?
Yes, definitely. That's actually on my list, but I really want to get billing working in the 5XB in order to show it in the video. It's really cool, but its going to take a few more months of work before its ready.
@@ConnectionsMuseum awesome, thank you. These videos are so interesting
Its fun to watch what I read about in the 1980s to know I have knowledge that only I can be glad I know . This was truly the age of hands on equipment. Today's systems are just not as fascinating to even watch cause there isn't anything to watch. Semi conductors are nothing like where we started with. This is a classic time of ALL COPPER WIRE USE THAT WILL NEVER BE THE SAME
All i can say is... wow. I would think building a 3ess/5ess wouldve been on an order of magnitude easier than all the mechanical components. Kudos to those who engineered AND built these.. AND those who continue to keep them running, Sara.. wow.. just wow, the level of your understanding is epic. My hat's off.....
It looks like the rear recorder--the one not reparable--had some bad mechanical parts. There are so many knowlegeable machinists (professional and amateur) just on YT alone, that it would not be impossible to locate someone who would volunteer to duplicate worn or damaged parts. Perhaps it would be worth doing an episode on exactly what you need to get the thing going again. If it has been possible for computer fanboys to reverse engineer and reproduce bits to get authentic Apollo capsule gear working again, why not a trouble recorder?
Amazing!
A nuw suscriber
Absolutely love seeing these complex elctro mechanical systems and its great seeing these being kept alive.