I've done a lot of testing on aircraft avionics. The beeping is a necessity. Please don't turn it off. Don't change your technique to suit some random viewers.
Don't rely on the beeping as evidence that wires can carry current, however, as not all meters will beep at the same resistance threshold. If you really want to be sure, re-measure with the ohms setting and not the diode/beep setting.
@@Stoney3K nobody says it's the full test, just use when you are trying to find shorts you don't want to be working blind because you have to look at the meter.
I have hyper-sensitivity to noise due to tinnitus, and that beeping is painful - common with older people -, that's why I tell repairs to tape over the buzzer in DMMs, ie, keep it working, just make it quieter.
@@SidneyCritic As a fellow "older person" the volume knob can't be that far away. I mean it's right under the video. I don't have tinnitus but i do have issues with very high pitched noises. I just turn it down. That seems to work.
YES! I made a reply that's NOT so nice, that sound MEANS continuity!! Come on people, if you're sad hearing that sound, I'm sorry but uh, you obviously don't use a multimeter. That's fine, but to all of us who DO use one, we KNOW and NEED that sound!!
Suspicion: This board NEVER worked. It was built, didn't work, got shelved for QA, someone eventually got hold of it, swapped a bunch of chips trying to find the fault(s) and never did. Adrian is just a troubleshooting beast. And folks, when you submit a board to be fabbed by PCBWay, JLPCB, etc. and they tell you that they need to check it for clearances between traces and pads, this is why.
Fun experiment would be to swap everything over to the brand new board. If that one is produced okay then it would be a lot more reliable than this one. After 2 shorts due to too not enough clearance between tracks I would not trust it anymore. I would not be surprised is there is still another short...
It is entirely possible that environmental conditions like slight corrosion from humidity, slight deterioration of solder mask or something could have turned a temperamental marginal board into a nightmare of various shorts. I also wonder if the boards were designed and tested with a higher quality PCB manufacturer that had better turnaround times and the production boards were made by a much lower cost manufacturer that had lower quality and caused all the issues.
What!? I love the assuring sound of the multimeter beeping! That's the best part of the multimeter! :D Yes, this was definitely a rollercoaster of emotions. I'm amazed of your knowledge of these systems. I can kinda follow but if I were attempting to fix a machine like this I'd be stumped, I wouldn't even know where to start. Great job on finding those faults! I hope you can fully resurrect this board and apply that color-killer circuit on the blank parts of the motherboard.
Thanks! Hey and before this I barely knew the apple II, so this was a lot of studying of that book to grasp how it all worked. This work has been thrilling and exciting for me too, and very very time consuming. (But it's been amazing, no complaints)
I totally agree about the beep. For finding the exact location of the fault, a good milliohms reading is very important but for basic conductivity tests, you don't want to have to look at the display all time. I even judge the quality of a conductivity tester by the beep latency, some take some 100ms until they beep. Can't use them :)
Someone could make an audio filter as the beep has a very specific frequency. Similar to like a colleague made a Vuvuzela filter for VLC back during the Brazil soccer world championship.
Tip for you - if you short the meter probes, then hit the REL button on your meter, it will zero it, and subsequent measurements will not include your probe resistance.
This is the greatest saga of all time. I'm almost glad (sorry Adrian) that there will be a third part. You gotta have three parts, you know, like The Godfather.
Maybe it's the Star Trek II / III / IV trilogy? At the end of II the board is dead. At the end of III it's alive, but its status is seriously in question. At the end of IV it will be working 100%. So I must ask, illegal Apple II clone, "How do you feel?"
My vote definitely goes to tin whiskers or some other kind of corrosion gradually creating bridges between those traces over time, rather than a board that was always non-functional. The fact that lightly dragging your blade between the traces was sufficient to break the connection lends support to that theory, as any such bridges would be quite thin/fragile. Incredible methodical troubleshooting work though! I certainly would have given up myself already, but you've almost got the board to a fully working state. Your perseverance is admirable.
What you are seeing with that board is almost certainly the tin whisker phenomenon. It’s more of a problem in spacecraft applications, but it can happen on any board, particularly if it uses lead-free solder.
Sometimes over the years traces can migrate and expand on PC boards. The metal flattens out and takes up more room. That's why power supply PC boards have a big empty space between the high voltage and low voltage sections.
It's only more of a problem in spacecraft (and aircraft) systems only because of the much more extensive testing and more strictly defined acceptance requirements for those applications. Also, it has nothing whatsoever to do with lead free solder. Even if it did this board predates any wide use of lead free solder in consumer electronics applications. Tin whiskers and etching faults have in common that they are likely to first show up where traces are very close together.
@@BassByTheTonne : Fix tin whisker? Maybe, but don't count on it. And if it does get fixed that way, don't be surprised if it comes back faster the second time- it's unlikely that you'll be able to get the whisker to properly retract, and more likely that it'll just break into smaller pieces that have an easier time rejoining. At some point, you just have to replace the entire board.
It don't have to be lead free. A lead content over 20% reduces the effect greatly, but addition like traces of copper, iron or zinc can expedite the process. Cheap solder made from "dirty" sources are often more affected from Tin creep / Tin whiskers.
This did reminding me of a couple of co-workers in the early 80s. They were building clones and complained at length about the quality/lack of of the boards.
Those little shorts on the board remind me of issues that companies had with leadless solder. I found a paper called "Formation of Intermetallic Crystals in Lead-free Wave Soldering". Basically making it lead free causes these "solder spikes" to grow and cause shorts between components...
Holy cursed motherboards. I'm astonished that board didn't end up in a dumpster somewhere many years ago, after frustrating those who dealt with it prior to it landing in your hands. Amazing job and perseverance.
It’s amazing how he found the bad screen mask in the motherboard. I wonder and don’t understand if that bad trace would have cause the damage to the other components? Did it cause some areas to experience heavy current drain?
Man! This is perhaps my "New Favorite Episode of Adrian's Digital Basement(TM)!" Love that low level work. I totally felt the satisfaction when you found that short. SO good!
A long video? *grabs popcorn* I'm ready! :D EDIT: Awesome troubleshooting! Most people I know only check for connectivity using the beeps on their multimeter, but it helps if you actually use the Ohms readout so you can more accurately pinpoint where the short could be.
Could it be tin fingers from the tin content of the solder? If so, it'll probably continue to develop new shorts. Maybe the chemistry was not quite right on the bulk solder they used for wave soldering, or the solder became contaminated at some later time with something that promotes finger growth. Might be interesting to look at some of the solder pads with medium power microscope, see if they look fuzzy.
I like that there were unusual problems as it allowed us viewers to see how you approached each of them. Also quite happy that it's mostly working now!
I am not the target audience for these videos. I have the most basic understanding of computer architecture and how circuit boards work. Almost 90% of what Adrian talks about and does is over my head. But I grew up on early computers (Apple II being my first at around 10 years old) and love retro tech, even if I'm mystified by how it works. I find these videos fascinating. I spent 2 hours watching someone resurrect an old machine and love every minute of it. Thanks, Adrian, for doing what you do even if I have no clue how you do it (although I do know why - most of the time).
As for the missing colors, I seem to remember reading something years and years ago about early Apple II models not having all the colors that slightly later revisions did. There was supposed to be a simple mod you could do to upgrade the early II boards to have all the colors. However....this is me remembering something I read 35 or more years ago. I might be wrong.
Your memory is correct; there was an article published in Byte magazine showing the modification necessary to add 2 more colors in hi-res mode, and that became a standard part of the Apple ]['s made later on. However, that is not the problem here, exactly, since after the title screen, you can indeed see all 6 colors on screen.
OMG those shorts! I have a feeling that that computer has been incredibly unreliable all its life! If you listen closely you could hear its previous user's frustration :D Thank for the deep dive, very entertaining! And yes, I can very much understand how many hours you spent on that desk so thank you!
BEEP AWAY!!! As a computer geek, that is music to my ears! Not constantly, mind you, but it does represent a valid data point, and that is the point to debugging, as you do, often, here.
Nice work! My hypothesis is that the manufacturer of the PCB didn't have their process properly controlled. Not surprising for a Chinese board and the time period, I suppose. If the masking is inaccurate or the chemistry isn't well controlled, you can get a lot of marginal traces. A bit of corrosion or thermal cycling will do the rest over time.
I appreciate these long-form explorative videos, where you explain the thought processes you're going through - and where you don't shy away from explaining the hows & the whys of fault finding, especially along withe do's and dont's!
With my experience with the Apple ] [ from 1979 to now, and I have 4 of them including this exact clone, all I can say is that Adrian is the best retro debugger of all time...awsome if not the best EVER! congratulations 🎉 😊
That are the videos that make me to enjoy your channel. And I'm someone who learned 30 years ago to layout and etch PCBs , solder it, wind the transformer and make a case for it and finally program it with some Z80 assembler. Never give up! Never surrender!
Dendritic growths. Given it's been kept in damp conditions its quite likely that the fine shorts were dendrites. Not uncommon on low cost vintage PCBs.
You never cease to amaze me with your troubleshooting skills. And the suspense was there that I actually cheered with you when it worked. I wish there was a weighted like button to give this video a higher value. Good work!
As you probably know, if identifying a short - the most efficient way to locate it is to inject 1V/2A current limited etc. into the short and monitor the board on a thermal camera. This will directly lead you to the short location. Maybe someone have an old one you could get donated - it will save a ton of time debugging.
Other options I've seen mentioned are IPA on a freshly _refrigerated_ board (might not work here, you have to watch for where evaporates first), or a current tracer (seem to be in short supply, HP made them back in the day, and there seems to be one manufacturer selling a large apparatus today).
Would the board have to be depopulated first, at least for any traces with continuity to the shorted ones? I imagine this works mainly when the cross section of conductor in the short is tiny in comparison to the rest of the circuit? But I guess if it's a "large" short like a big blob of solder it would be easier to spot visually in the first place.
David Jones demonstrates a preaty sweet way of locating exactly where a short is on a pcb in his EEVBlog #1061. Maybe something like that could save some hours of effort on the next one.
I have a similar illegal Apple II+ clone. Its ROMs are all exact copies of the Apple II+ ROMs, with the exception that "APPLE ][" is replaced with "COMPUTER". I concur with an earlier commenter that the mismatched heritage of the ROMs could be an issue... but it could be a red herring if all those ROMs have the same checksum (something I have not checked). I'd like to point my finger at the RAM that tested bad in the chip tester for the issues you had in Fat City, but logic chips could still be to blame. My clone had issues with showing no colors at all until I replaced the 74LS-series chips that tested bad in my T48, and I could totally see a bad logic chip being to blame here, too. If there is anyone I know who will persevere to get to "It Freakin' Works!" on this Apple II+ clone, it is you - keep up the great work!
One bit of leverage-- the short is likely much narrower than the trace widths, so you can exploit this by running a few hundred milliamps across the shorted lines. That should burn out the tiny shorted area.
Dude - identifying that short in the expansion connector through logical reasoning was slick. I had an Apple II clone for a while as a kid and it required you to open the lid and push the chips back into the sockets any time you moved it. Maybe it wasn’t the sockets but it had the same shorts problem you are seeing?
Absolutely fantastic. I had an Apple ][+ when I was a teen in the eighties and seeing someone going through an explanation of how it actually works has been mesmerizing. I almost pursued studying electronics but ultimately had a career in IT. I'm very much looking forward to Part III. This reminds me that I have a drawer full of "working pulls" expansion cards (Gibson Light Pen, etc.) that I bought in garage sales as well as at least three of the computers, one of which is the clone known as the "Classie" which I got through a high school friend's father who worked for a certain company in the eighties (cough, cough, Martin).
Remember to use a CRT if you want to play with the light pen- LCDs don't provide the correct visual behavior (a moving reflector system could also work, but I don't know of any such monitors...)
@@scottlarson1548 : Not a problem at the time, RAM was still often faster than the processors. Unfortunately, if you want to do that today, you may as well just be making a RAM disk from DDR for something from the 80s.
@@absalomdraconis That wasn't what I was seeing at the time. While the 6502 in the Apple II ran at a slow 1 MHz, I was running the 6502 in my Ohio Scientific C1P at 2 MHz because they were finally producing RAM chips with 450 ns access times. Of course you had to keep the 6502 at half speed in the Apple II because the video system depended on it.
It's almost like tin whiskers but not really since there's gold plating and almost certainly leaded solder... weird but exciting episode! Did you look at the other board to see if it has these deformed pads as well?
Excellent, one of the best diagnostic / repair videos I've watched, fantastic edition, keep then coming. Also I have absolutely no problem with continuity testing beeps, a necessary aid. Thanks for this upload. All the best to you
You make diagnosing and repairing stuff seem so natural and fun. It'd take me many more years to get to your level. Love watching your videos as inspiration.
Apart from all the expansion slots, that board really reminds me of my original Galaxian board. Just a billion discrete logic chips, a Z80 and a handful of ROMs. It's a work of art. Just need a cabinet to mount it in! 😂
Steve Wozniak's video system was ingenious, but really hard to figure out. Compare how few chips it took for the Apple 2 video the entire large board the monochrome graphics adapter on the original IBM PC.
After clearing the bridged connection. Perhaps some fingernail polish/ lacquer.. To act as a solder mask. That connection looked really close. It was very fun watching you figure this out. Looked impossible.
Two of the most difficult things to troubleshoot are: 1) A circuit/board/module/component that's not known to have ever worked. 2) Something that wasn't repaired by someone else that may have created new problems inadvertently and/or tried substitute parts that weren't actually equivalent. I think you got both. Most likely someone bought the board unpopulated and gradually assembled it when they had more parts and more time. It didn't work because of one or more reasons that could have been using some substitute transistors where 2N3904's would have worked -OR- there were etching shorts in more than one place -OR- improperly handled static sensitive chips -OR- salvaged chips of unknown status -OR- didn't carefully check their soldering as they proceeded -OR- badly cloned ROM's (and many other possibilities). Then, they or another person tried to repair it and maybe found and fixed several things, but gave up. This may have been as late as 1989, when some of the chips were already hard to find. Then after possibly passing through a number of hands (maybe tossed out and scavenged from a recycler), you got it without the advantage of knowing any of the history.
The solder mask may be misaligned exposing the edge of some traces/pads and excess solder could have bridged the gap. Alternatively copper ‘whiskers’ can remain from the etching process and the bridges could have always been there. In this case there could be more. Were they on top of the solder mask? If below they have to be whiskers; on top and solder is the likely cause.
Enjoy your videos. Been thinking about offering this suggestion for quite some time. Years ago I needed to repair a large number of Z80 based boards. I soon discovered one could go crazy trying to do it dynamically. I built a Z80 "simulator" in a small minibox with a bunch of SPDT toggle switches and an LED to set and show the state of every data, address and control line. The simulator replaced the processor and by flipping switches was used to manually step through the instructions the processor would go through at boot up. Obviously it didn't work with dynamic RAM but for any peripheral that was static I could trace the state of any line using only a LED probe or observing the LEDs on the simulator. It usually only took one or two instructions to find a bad IC or open trace. All with a simple LED indicator.
Assuming the system will try to run, you'll usually be better off with a diagnostic ROM (e.g. an IBM compatible's POST code system). Even better though would be a "high reliability" simulator ("high reliability", because approximately the same technique gets used for some high-reliability hardware) that both runs the code like normal, _and_ runs it with partially (or entirely) simulated hardware so that it can detect when _results_ are different between the two running instances (you partially simulate mostly to deal with e.g. unpredictable keyboard input) despite running at full speed.
Adrian! Nice find and repair sir! I'm guessing that this computer never worked correctly from DAY ONE! I'd clean all the sockets, and replace those tested bad memory ICs, and then see if it functions better.
This has been one of my favourite videos, thank you. I worked on a Apricot computer from the mid/late 80s and the tracks had similar problems, they were just too close and then random dirt or tiny corrosion would bridge them.
This is fascinating. Clearly this motherboard was exposed to a lot of hostile conditions with that water and rust. So it could be this happened after it was no longer used anymore.
I wonder if ultrasonic cleaning of the board would have knocked off those weird shorts? Such a weird fault case. Who knows if you'd ever run in to something similar again. Like a nightmare of someone using steel wool near a PCB. Great work on the repair marathon. So many things learned.
Hi Adrian, an other option. I bought the same board when I was young. (around 1982 - 1983) Installed (soldered) al new sockets, (The good golden round type) placed all the chips and powered the board up. And it gave me the well known Apple Beep. The only problem I had: The character Rom was swapped. For some reason the manufacturer had swapped the data lines on the character ROM. What I'm pointing out. Maybe this board was also hand made. And the poor user never had the luck I had. I mean, I bought all the IC's from an local supplier. And they where all fine. I wrote an assembler program on my business computer (CBM 8032) to swap the character ROM info. After programming it to an new Eprom the APPLE II clone worked fine. And by the the way it still does in 2023
When you scraped off those two shorts it would have been a good idea to put solder-resist on the scratches because when you resolder the sockets on some solder can flow in to the scratch and recreate the short. That would be a bit frustrating!
I wonder if this is a case of "tin whiskers." Basically they are small filaments or crystals of metal that sometimes spontaneously grow off of the tin in solder over time. These whiskers can eventually grow long enough to bridge adjacent connections. This is especially problematic on pads/traces that are very close to each other, such as the two shorts that you found on this board. Those things that you pointed out sure do look like it to me. Yeah there is solder mask that is supposed to prevent this from happening, but maybe there was a small break in the solder mask that the tin whisker just happened by pure coincidence to grow into? That's the only thing I could think of that might be what is going on here. Very weird problem for sure. Glad you were able to get the board to boot at least!
I believe that solder mask only helps while soldering, not afterwards. The only known solutions to tin whiskers seem to be in solder chemistry, rather than board manufacturing.
Adrian, could these shorts between the vias of the sockets and traces be caused by a phenomenon called "tin whiskers"? Have a read up about it. Probably caused by both the environment it was stored in
Fantastic job in tracking down the shorts. Some board manufacturers produce great boards others produce lesser quality boards and might push it down on a test jig just enough to get it pass then down the line it goes. I personally would have cleared the short under the expansion slot and if it still messed up would have somehow ended up embedded in my wall.
I wonder if the code in those EPROMS are mismatched enough to cause compatibility issues? Perhaps try with a compete set of originals or at least from the same source? Also we need to get Christopher Nolan or Denis Villeneuve to direct the cinema edit of this repair series. It's freaking Amazing! 😉
Adrian, this was a fantastic video. Every problem you hit and knocked down was so satisfying to watch. I know its been a challenge but the victory will be sweet!
After realising the manufacturing fault first with the expansion slot I thought it would be strange if no other similar issues elsewhere on the board. I'd poke around and search for more silly shorts for the last minor errors left. Fascinating brain you have.
A few people are suggesting tin whiskers, but this board is from pre lead free manufacturing, wasn't it? Leaded solder was whisker free. Could be more like poor storage conditions and then operating with damp/mold and electrolyticly forming bridges? Anyway, excellent fault finding skills, great video.
Leaded solder wasn't _universally_ whisker-free. The percentage _did_ actually matter, and if you pushed too far off the sweet spot they _would_ show up again, albeit slower.
Looks like the PCB could be delaminating, and this is causing the foil of the traces to raise off the FR4 and bridge. This might happen especially if it has been exposed to excessive humidity and heat. So this well could happen again.
Suggestion - the hi-res pages are between $2000 - $5FFF. It appeared that one hi-res page was working and the second one not. Fat City would likely have used page-flipping for animation. While the Siemens RAM chips tested fine with the Apple-cillin diagnostic, you indicated some failed in the chip-tester. Try swapping with known good RAM. Also, Apple provided a diagnostic disk, "Apple II Plus Dealer Diagnostics", which I've used for years on my ][ Plus hardware, which may have more rigorous test routines than Apple-cillin.
Nice work! It was like watching Sherlock Holmes :) I think someone was "correcting" cold solder joints and added too much solder. I had it many times, that too much solder went through the connection hole and made short in neighbour circuit. Additionally, in this case, PCB was in bad quality that made this shorts invisible.
I feel your frustration and happiness once finished fixing because I have your case for several times when I was fixing ZX Spectrum computers in late 80s and early 90s ... It seems that solder mask process is the problem ..
Correct, or just Siemens as they were already called at that time. While Siemens still exists today, their semiconductor business hast been outsourced long ago and is now called Infineon.
ProTip: The "shimmering" can be caused by the CPU processing instructions but because they're the wrong instructions, it's shuffling data around randomly in RAM, including in the screen RAM. If it gets stuck in a loop, it may wind up continually shuffling video RAM contents which leads to the "shimmering" effect, or it may get stuck into such a tight loop that it doesn't do any data shuffling at all in which case the screen display will be stable but garbage.
I've done a lot of testing on aircraft avionics. The beeping is a necessity. Please don't turn it off. Don't change your technique to suit some random viewers.
Don't rely on the beeping as evidence that wires can carry current, however, as not all meters will beep at the same resistance threshold. If you really want to be sure, re-measure with the ohms setting and not the diode/beep setting.
@@Stoney3K nobody says it's the full test, just use when you are trying to find shorts you don't want to be working blind because you have to look at the meter.
I have hyper-sensitivity to noise due to tinnitus, and that beeping is painful - common with older people -, that's why I tell repairs to tape over the buzzer in DMMs, ie, keep it working, just make it quieter.
@@SidneyCritic As a fellow "older person" the volume knob can't be that far away. I mean it's right under the video. I don't have tinnitus but i do have issues with very high pitched noises. I just turn it down. That seems to work.
YES! I made a reply that's NOT so nice, that sound MEANS continuity!! Come on people, if you're sad hearing that sound, I'm sorry but uh, you obviously don't use a multimeter. That's fine, but to all of us who DO use one, we KNOW and NEED that sound!!
Suspicion: This board NEVER worked. It was built, didn't work, got shelved for QA, someone eventually got hold of it, swapped a bunch of chips trying to find the fault(s) and never did. Adrian is just a troubleshooting beast.
And folks, when you submit a board to be fabbed by PCBWay, JLPCB, etc. and they tell you that they need to check it for clearances between traces and pads, this is why.
Fun experiment would be to swap everything over to the brand new board. If that one is produced okay then it would be a lot more reliable than this one. After 2 shorts due to too not enough clearance between tracks I would not trust it anymore. I would not be surprised is there is still another short...
Definitely. The (lack of) clearances on that board surprise me only that there aren't _more_ faults. That design was doomed.
It is entirely possible that environmental conditions like slight corrosion from humidity, slight deterioration of solder mask or something could have turned a temperamental marginal board into a nightmare of various shorts. I also wonder if the boards were designed and tested with a higher quality PCB manufacturer that had better turnaround times and the production boards were made by a much lower cost manufacturer that had lower quality and caused all the issues.
What!? I love the assuring sound of the multimeter beeping! That's the best part of the multimeter! :D
Yes, this was definitely a rollercoaster of emotions. I'm amazed of your knowledge of these systems. I can kinda follow but if I were attempting to fix a machine like this I'd be stumped, I wouldn't even know where to start. Great job on finding those faults! I hope you can fully resurrect this board and apply that color-killer circuit on the blank parts of the motherboard.
The beep is the song of my people!
Thanks! Hey and before this I barely knew the apple II, so this was a lot of studying of that book to grasp how it all worked. This work has been thrilling and exciting for me too, and very very time consuming. (But it's been amazing, no complaints)
The beep makes it that much easier to follow what's going on, who the heck is complaining about it?!
I totally agree about the beep. For finding the exact location of the fault, a good milliohms reading is very important but for basic conductivity tests, you don't want to have to look at the display all time. I even judge the quality of a conductivity tester by the beep latency, some take some 100ms until they beep. Can't use them :)
Someone could make an audio filter as the beep has a very specific frequency. Similar to like a colleague made a Vuvuzela filter for VLC back during the Brazil soccer world championship.
Tip for you - if you short the meter probes, then hit the REL button on your meter, it will zero it, and subsequent measurements will not include your probe resistance.
I never knew this.. Thanks!
Steve !
Thank you for your tips! Learned a new thing!
@@gryzman Gregg! How’s things?
I never knew this
This is the greatest saga of all time. I'm almost glad (sorry Adrian) that there will be a third part. You gotta have three parts, you know, like The Godfather.
Haha hard to compare my stuff to that series, but hey I'll take the compliment!
@rommix0Still better than Endgame!
Maybe it's the Star Trek II / III / IV trilogy? At the end of II the board is dead. At the end of III it's alive, but its status is seriously in question. At the end of IV it will be working 100%. So I must ask, illegal Apple II clone, "How do you feel?"
My vote definitely goes to tin whiskers or some other kind of corrosion gradually creating bridges between those traces over time, rather than a board that was always non-functional. The fact that lightly dragging your blade between the traces was sufficient to break the connection lends support to that theory, as any such bridges would be quite thin/fragile.
Incredible methodical troubleshooting work though! I certainly would have given up myself already, but you've almost got the board to a fully working state. Your perseverance is admirable.
This was an *outstanding* feat of troubleshooting.
What you are seeing with that board is almost certainly the tin whisker phenomenon. It’s more of a problem in spacecraft applications, but it can happen on any board, particularly if it uses lead-free solder.
And it will have taken a long time, with voltages present to grow those whiskers.
Sometimes over the years traces can migrate and expand on PC boards. The metal flattens out and takes up more room. That's why power supply PC boards have a big empty space between the high voltage and low voltage sections.
It's only more of a problem in spacecraft (and aircraft) systems only because of the much more extensive testing and more strictly defined acceptance requirements for those applications.
Also, it has nothing whatsoever to do with lead free solder. Even if it did this board predates any wide use of lead free solder in consumer electronics applications.
Tin whiskers and etching faults have in common that they are likely to first show up where traces are very close together.
@@BassByTheTonne : Fix tin whisker? Maybe, but don't count on it. And if it does get fixed that way, don't be surprised if it comes back faster the second time- it's unlikely that you'll be able to get the whisker to properly retract, and more likely that it'll just break into smaller pieces that have an easier time rejoining. At some point, you just have to replace the entire board.
It don't have to be lead free. A lead content over 20% reduces the effect greatly, but addition like traces of copper, iron or zinc can expedite the process. Cheap solder made from "dirty" sources are often more affected from Tin creep / Tin whiskers.
This did reminding me of a couple of co-workers in the early 80s. They were building clones and complained at length about the quality/lack of of the boards.
Those little shorts on the board remind me of issues that companies had with leadless solder. I found a paper called "Formation of Intermetallic Crystals in Lead-free Wave Soldering". Basically making it lead free causes these "solder spikes" to grow and cause shorts between components...
Holy cursed motherboards. I'm astonished that board didn't end up in a dumpster somewhere many years ago, after frustrating those who dealt with it prior to it landing in your hands. Amazing job and perseverance.
I used to call it the African Queen Syndrome when I was in support - some people seemed to get attached to their barely working pile of crap
It’s amazing how he found the bad screen mask in the motherboard.
I wonder and don’t understand if that bad trace would have cause the damage to the other components? Did it cause some areas to experience heavy current drain?
There are likely thousands that did end up in a dumpster and this one got missed.
@@cll1out maybe this came from a dumpster dive.
It is amazing that some chips even function when traces are shorted.
They certainly don't like it and it's not good for them, but it shows how resilient this stuff all is.
AND THE CROWD GOES WILD!!! That was quite the ride. A satisfying end to a great series.
Man! This is perhaps my "New Favorite Episode of Adrian's Digital Basement(TM)!" Love that low level work. I totally felt the satisfaction when you found that short. SO good!
Thanks!! And the saga continues next week! :-)
A long video? *grabs popcorn*
I'm ready! :D
EDIT: Awesome troubleshooting! Most people I know only check for connectivity using the beeps on their multimeter, but it helps if you actually use the Ohms readout so you can more accurately pinpoint where the short could be.
Could it be tin fingers from the tin content of the solder? If so, it'll probably continue to develop new shorts. Maybe the chemistry was not quite right on the bulk solder they used for wave soldering, or the solder became contaminated at some later time with something that promotes finger growth. Might be interesting to look at some of the solder pads with medium power microscope, see if they look fuzzy.
Outstanding Job Adrian! I'm glad I could help somehow!
I like that there were unusual problems as it allowed us viewers to see how you approached each of them. Also quite happy that it's mostly working now!
This time, I was seriously worried for your sanity, Adrian. Glad you figured out most of the faults.
@martinr1834 Well, because with each new issue he found, he sounded more desperate?
I am not the target audience for these videos. I have the most basic understanding of computer architecture and how circuit boards work. Almost 90% of what Adrian talks about and does is over my head. But I grew up on early computers (Apple II being my first at around 10 years old) and love retro tech, even if I'm mystified by how it works. I find these videos fascinating. I spent 2 hours watching someone resurrect an old machine and love every minute of it. Thanks, Adrian, for doing what you do even if I have no clue how you do it (although I do know why - most of the time).
As for the missing colors, I seem to remember reading something years and years ago about early Apple II models not having all the colors that slightly later revisions did. There was supposed to be a simple mod you could do to upgrade the early II boards to have all the colors. However....this is me remembering something I read 35 or more years ago. I might be wrong.
Your memory is correct; there was an article published in Byte magazine showing the modification necessary to add 2 more colors in hi-res mode, and that became a standard part of the Apple ]['s made later on. However, that is not the problem here, exactly, since after the title screen, you can indeed see all 6 colors on screen.
OMG those shorts! I have a feeling that that computer has been incredibly unreliable all its life! If you listen closely you could hear its previous user's frustration :D Thank for the deep dive, very entertaining! And yes, I can very much understand how many hours you spent on that desk so thank you!
BEEP AWAY!!! As a computer geek, that is music to my ears! Not constantly, mind you, but it does represent a valid data point, and that is the point to debugging, as you do, often, here.
This was great! I can see how frustrating it was for you but it was great to watch. Can't wait for part 3!
Nice work! My hypothesis is that the manufacturer of the PCB didn't have their process properly controlled. Not surprising for a Chinese board and the time period, I suppose. If the masking is inaccurate or the chemistry isn't well controlled, you can get a lot of marginal traces. A bit of corrosion or thermal cycling will do the rest over time.
I wonder if the other bare PCB that Adrian has shown before will have the same problems?
I'd have figured Taiwan or Japan for fabbing Appler II clones in that timeframe.
I don't think commie china were capable to made the board during that era.
Most of apple ii clone were made in taiwan.
@@rog2224 Your correct, mine, an empty PCB was made in Taiwan
I appreciate these long-form explorative videos, where you explain the thought processes you're going through - and where you don't shy away from explaining the hows & the whys of fault finding, especially along withe do's and dont's!
With my experience with the Apple ] [ from 1979 to now, and I have 4 of them including this exact clone, all I can say is that Adrian is the best retro debugger of all time...awsome if not the best EVER! congratulations 🎉 😊
That are the videos that make me to enjoy your channel. And I'm someone who learned 30 years ago to layout and etch PCBs , solder it, wind the transformer and make a case for it and finally program it with some Z80 assembler. Never give up! Never surrender!
Dendritic growths. Given it's been kept in damp conditions its quite likely that the fine shorts were dendrites. Not uncommon on low cost vintage PCBs.
You never cease to amaze me with your troubleshooting skills. And the suspense was there that I actually cheered with you when it worked. I wish there was a weighted like button to give this video a higher value. Good work!
This mini series was interesting. Your diag abilities always impress
As you probably know, if identifying a short - the most efficient way to locate it is to inject 1V/2A current limited etc. into the short and monitor the board on a thermal camera. This will directly lead you to the short location. Maybe someone have an old one you could get donated - it will save a ton of time debugging.
Other options I've seen mentioned are IPA on a freshly _refrigerated_ board (might not work here, you have to watch for where evaporates first), or a current tracer (seem to be in short supply, HP made them back in the day, and there seems to be one manufacturer selling a large apparatus today).
Would the board have to be depopulated first, at least for any traces with continuity to the shorted ones?
I imagine this works mainly when the cross section of conductor in the short is tiny in comparison to the rest of the circuit? But I guess if it's a "large" short like a big blob of solder it would be easier to spot visually in the first place.
David Jones demonstrates a preaty sweet way of locating exactly where a short is on a pcb in his EEVBlog #1061. Maybe something like that could save some hours of effort on the next one.
I have a similar illegal Apple II+ clone. Its ROMs are all exact copies of the Apple II+ ROMs, with the exception that "APPLE ][" is replaced with "COMPUTER". I concur with an earlier commenter that the mismatched heritage of the ROMs could be an issue... but it could be a red herring if all those ROMs have the same checksum (something I have not checked).
I'd like to point my finger at the RAM that tested bad in the chip tester for the issues you had in Fat City, but logic chips could still be to blame. My clone had issues with showing no colors at all until I replaced the 74LS-series chips that tested bad in my T48, and I could totally see a bad logic chip being to blame here, too.
If there is anyone I know who will persevere to get to "It Freakin' Works!" on this Apple II+ clone, it is you - keep up the great work!
One bit of leverage-- the short is likely much narrower than the trace widths, so you can exploit this by running a few hundred milliamps across the shorted lines. That should burn out the tiny shorted area.
Your tenacity makes an excellent troubleshooting video.
FYI, early revisions of the Apple II only had four colors in hi-res mode. Later revisions increased it to six colors.
Amazing. The Power Button + workaround for the S key was brilliant.
Chasing shorts on electronics is like finding stray compilable code that fails on runtime.
I'm in awe of the knowledge, skill and patience.
He’s working on this machine like it holds the secret to life.
Wow, a long video, totally worth it. Never seen this sort of fault and you've done an outstanding job diagnosing and explaining what you do!
Dude - identifying that short in the expansion connector through logical reasoning was slick. I had an Apple II clone for a while as a kid and it required you to open the lid and push the chips back into the sockets any time you moved it. Maybe it wasn’t the sockets but it had the same shorts problem you are seeing?
Absolutely fantastic. I had an Apple ][+ when I was a teen in the eighties and seeing someone going through an explanation of how it actually works has been mesmerizing. I almost pursued studying electronics but ultimately had a career in IT. I'm very much looking forward to Part III. This reminds me that I have a drawer full of "working pulls" expansion cards (Gibson Light Pen, etc.) that I bought in garage sales as well as at least three of the computers, one of which is the clone known as the "Classie" which I got through a high school friend's father who worked for a certain company in the eighties (cough, cough, Martin).
Remember to use a CRT if you want to play with the light pen- LCDs don't provide the correct visual behavior (a moving reflector system could also work, but I don't know of any such monitors...)
That address/data mux combination is pretty smart -- the video generator basically does DMA in the moments that the CPU isn't talking to the RAM.
You need is fast expensive RAM and keep the CPU slow.
@@scottlarson1548 : Not a problem at the time, RAM was still often faster than the processors. Unfortunately, if you want to do that today, you may as well just be making a RAM disk from DDR for something from the 80s.
@@absalomdraconis That wasn't what I was seeing at the time. While the 6502 in the Apple II ran at a slow 1 MHz, I was running the 6502 in my Ohio Scientific C1P at 2 MHz because they were finally producing RAM chips with 450 ns access times. Of course you had to keep the 6502 at half speed in the Apple II because the video system depended on it.
Keep the beeps. It’s part of the diagnosis process. And that’s why we are here.
It's almost like tin whiskers but not really since there's gold plating and almost certainly leaded solder... weird but exciting episode! Did you look at the other board to see if it has these deformed pads as well?
Excellent, one of the best diagnostic / repair videos I've watched, fantastic edition, keep then coming. Also I have absolutely no problem with continuity testing beeps, a necessary aid. Thanks for this upload. All the best to you
You are the master. Very well done 😊
You make diagnosing and repairing stuff seem so natural and fun. It'd take me many more years to get to your level. Love watching your videos as inspiration.
I feel so happy because you repaired do this. You are really technician.
Excellent detective work. And perseverance.
man! that's crazy! I would have quit that repair job a looong time ago, it really was a bundle of broken parts, but it turned beautiful thanks to you
Apart from all the expansion slots, that board really reminds me of my original Galaxian board. Just a billion discrete logic chips, a Z80 and a handful of ROMs.
It's a work of art.
Just need a cabinet to mount it in! 😂
Great video. When you really take a deep dive in things like this is the best!
Steve Wozniak's video system was ingenious, but really hard to figure out. Compare how few chips it took for the Apple 2 video the entire large board the monochrome graphics adapter on the original IBM PC.
After clearing the bridged connection. Perhaps some fingernail polish/ lacquer..
To act as a solder mask.
That connection looked really close. It was very fun watching you figure this out. Looked impossible.
Two of the most difficult things to troubleshoot are:
1) A circuit/board/module/component that's not known to have ever worked.
2) Something that wasn't repaired by someone else that may have created new problems inadvertently and/or tried substitute parts that weren't actually equivalent.
I think you got both. Most likely someone bought the board unpopulated and gradually assembled it when they had more parts and more time. It didn't work because of one or more reasons that could have been using some substitute transistors where 2N3904's would have worked -OR- there were etching shorts in more than one place -OR- improperly handled static sensitive chips -OR- salvaged chips of unknown status -OR- didn't carefully check their soldering as they proceeded -OR- badly cloned ROM's (and many other possibilities). Then, they or another person tried to repair it and maybe found and fixed several things, but gave up. This may have been as late as 1989, when some of the chips were already hard to find. Then after possibly passing through a number of hands (maybe tossed out and scavenged from a recycler), you got it without the advantage of knowing any of the history.
I know this is a lot of work, but I think this is my favorite video you made to date! Great detective work!
The solder mask may be misaligned exposing the edge of some traces/pads and excess solder could have bridged the gap. Alternatively copper ‘whiskers’ can remain from the etching process and the bridges could have always been there. In this case there could be more. Were they on top of the solder mask? If below they have to be whiskers; on top and solder is the likely cause.
Enjoy your videos. Been thinking about offering this suggestion for quite some time. Years ago I needed to repair a large number of Z80 based boards. I soon discovered one could go crazy trying to do it dynamically. I built a Z80 "simulator" in a small minibox with a bunch of SPDT toggle switches and an LED to set and show the state of every data, address and control line. The simulator replaced the processor and by flipping switches was used to manually step through the instructions the processor would go through at boot up. Obviously it didn't work with dynamic RAM but for any peripheral that was static I could trace the state of any line using only a LED probe or observing the LEDs on the simulator. It usually only took one or two instructions to find a bad IC or open trace. All with a simple LED indicator.
Assuming the system will try to run, you'll usually be better off with a diagnostic ROM (e.g. an IBM compatible's POST code system).
Even better though would be a "high reliability" simulator ("high reliability", because approximately the same technique gets used for some high-reliability hardware) that both runs the code like normal, _and_ runs it with partially (or entirely) simulated hardware so that it can detect when _results_ are different between the two running instances (you partially simulate mostly to deal with e.g. unpredictable keyboard input) despite running at full speed.
What a rollercoaster ride! Glad you finally got it. Not sure I could take any more....
Wasn't quite done with the video when I posted. Are you going to continue in the future?
How have you not chucked the board as far as you can, by now. I think I would have given up a long time ago .
You have great patience Adrian 👍
Excellent work i realy like this fault finding episodes. Keep up good work. Regards
Adrian! Nice find and repair sir! I'm guessing that this computer never worked correctly from DAY ONE!
I'd clean all the sockets, and replace those tested bad memory ICs, and then see if it functions better.
This has been one of my favourite videos, thank you. I worked on a Apricot computer from the mid/late 80s and the tracks had similar problems, they were just too close and then random dirt or tiny corrosion would bridge them.
This is fascinating. Clearly this motherboard was exposed to a lot of hostile conditions with that water and rust. So it could be this happened after it was no longer used anymore.
I think the board is drilled and/or plated ever-so-slightly off, like someone knocked on the machine a tiniest bit. So close yet so far!
Outstanding work on your part, but worth it, so far, good troubleshooting. Just needs more. All good wishes.
I wonder if ultrasonic cleaning of the board would have knocked off those weird shorts? Such a weird fault case. Who knows if you'd ever run in to something similar again.
Like a nightmare of someone using steel wool near a PCB.
Great work on the repair marathon. So many things learned.
The SH logo on the RAM chips is from the german brand Siemens. I remember seeing them when I was repairing old computer boards back in the nineties.
Adrian just needs to admit he loves 'em long (the videos! 😝). That was quite a fascinating troubleshooting adventure!
Hi Adrian, an other option. I bought the same board when I was young. (around 1982 - 1983) Installed (soldered) al new sockets, (The good golden round type) placed all the chips and powered the board up. And it gave me the well known Apple Beep. The only problem I had: The character Rom was swapped. For some reason the manufacturer had swapped the data lines on the character ROM.
What I'm pointing out. Maybe this board was also hand made. And the poor user never had the luck I had. I mean, I bought all the IC's from an local supplier. And they where all fine.
I wrote an assembler program on my business computer (CBM 8032) to swap the character ROM info. After programming it to an new Eprom the APPLE II clone worked fine.
And by the the way it still does in 2023
When you scraped off those two shorts it would have been a good idea to put solder-resist on the scratches because when you resolder the sockets on some solder can flow in to the scratch and recreate the short. That would be a bit frustrating!
What a place to find a short on the board. I'm wondering if this board ever worked.
I like the way you hunted down the shorted traces
I wonder if this is a case of "tin whiskers." Basically they are small filaments or crystals of metal that sometimes spontaneously grow off of the tin in solder over time. These whiskers can eventually grow long enough to bridge adjacent connections. This is especially problematic on pads/traces that are very close to each other, such as the two shorts that you found on this board. Those things that you pointed out sure do look like it to me. Yeah there is solder mask that is supposed to prevent this from happening, but maybe there was a small break in the solder mask that the tin whisker just happened by pure coincidence to grow into? That's the only thing I could think of that might be what is going on here. Very weird problem for sure. Glad you were able to get the board to boot at least!
I believe that solder mask only helps while soldering, not afterwards. The only known solutions to tin whiskers seem to be in solder chemistry, rather than board manufacturing.
Adrian, could these shorts between the vias of the sockets and traces be caused by a phenomenon called "tin whiskers"? Have a read up about it. Probably caused by both the environment it was stored in
Usual joy to watch, thank you for taking us on that rollercoaster journey and look forward to part 3!
Fantastic job in tracking down the shorts. Some board manufacturers produce great boards others produce lesser quality boards and might push it down on a test jig just enough to get it pass then down the line it goes. I personally would have cleared the short under the expansion slot and if it still messed up would have somehow ended up embedded in my wall.
Oh boy, after hearing your introduction there I'm looking forward to this!
Brilliant job Adrian! Your reaction when you'd fixed it was brilliant
I wonder if the code in those EPROMS are mismatched enough to cause compatibility issues? Perhaps try with a compete set of originals or at least from the same source? Also we need to get Christopher Nolan or Denis Villeneuve to direct the cinema edit of this repair series. It's freaking Amazing! 😉
The ROMs seem to not cause any issues. It's an odd mismatch but who knew it doesn't really matter it seems. I won't talk about how I know :-)
@@adriansdigitalbasement Part 3 will reveal all perhaps? 😁
Oh what a cliffhanger! Adrian's repair work is the best drama for tech people.
@@jwhite5008 Indeed. I think I'll ask Christopher Nolan to edit the together into a three hour IMAX epic 😁
Adrian, this was a fantastic video. Every problem you hit and knocked down was so satisfying to watch. I know its been a challenge but the victory will be sweet!
After realising the manufacturing fault first with the expansion slot I thought it would be strange if no other similar issues elsewhere on the board. I'd poke around and search for more silly shorts for the last minor errors left. Fascinating brain you have.
Man, your persistence is legend!!!!
Brilliant bit of diagnostic work!
brilliant - just brilliant, your patience and perserverance are amazing!
Dang it, now I'm super invested in this... you GOTTA get that ornery thing working now!
Back in the mid-eighties, I spent quite some time on several AppleII clones. Ran CP/M with Wordstar and Visicalc.
A few people are suggesting tin whiskers, but this board is from pre lead free manufacturing, wasn't it? Leaded solder was whisker free. Could be more like poor storage conditions and then operating with damp/mold and electrolyticly forming bridges? Anyway, excellent fault finding skills, great video.
Leaded solder wasn't _universally_ whisker-free. The percentage _did_ actually matter, and if you pushed too far off the sweet spot they _would_ show up again, albeit slower.
Looks like the PCB could be delaminating, and this is causing the foil of the traces to raise off the FR4 and bridge. This might happen especially if it has been exposed to excessive humidity and heat. So this well could happen again.
Suggestion - the hi-res pages are between $2000 - $5FFF. It appeared that one hi-res page was working and the second one not. Fat City would likely have used page-flipping for animation. While the Siemens RAM chips tested fine with the Apple-cillin diagnostic, you indicated some failed in the chip-tester. Try swapping with known good RAM. Also, Apple provided a diagnostic disk, "Apple II Plus Dealer Diagnostics", which I've used for years on my ][ Plus hardware, which may have more rigorous test routines than Apple-cillin.
Nice work! It was like watching Sherlock Holmes :)
I think someone was "correcting" cold solder joints and added too much solder. I had it many times, that too much solder went through the connection hole and made short in neighbour circuit. Additionally, in this case, PCB was in bad quality that made this shorts invisible.
Awesome repair series!!
The RAM-Chip at 10:10 is a german Siemens & Halske brand (SH), normally Siemens manufactured high quality chips at that days.
What a journey! Very nice work diagnosing so thoroughly such an obscure problem.
Troubleshooting is an art.
I feel your frustration and happiness once finished fixing because I have your case for several times when I was fixing ZX Spectrum computers in late 80s and early 90s ... It seems that solder mask process is the problem ..
The RAM brand logo at 10:10 are a connected S & H for Siemens & Halske (en.wikipedia.org/wiki/Siemens_%26_Halske) 🙂
Correct, or just Siemens as they were already called at that time.
While Siemens still exists today, their semiconductor business hast been outsourced long ago and is now called Infineon.
Epic repair video. Thank you so much. I cannot believe that you didn't give up!
ProTip: The "shimmering" can be caused by the CPU processing instructions but because they're the wrong instructions, it's shuffling data around randomly in RAM, including in the screen RAM. If it gets stuck in a loop, it may wind up continually shuffling video RAM contents which leads to the "shimmering" effect, or it may get stuck into such a tight loop that it doesn't do any data shuffling at all in which case the screen display will be stable but garbage.