Dave didn’t go into this specifically,, but for those who remember “single density” vs. “double density” floppy drives, single density used FM encoding while double density used MFM encoding.
@@GeneCash I know my first day on a new job they were rolling out a 6ft long drum disk from the Air Force facility I spent the next 8 years working on everything Dave talked about from 7 track tapes to 40/300mbyte spinning multi platter disks to the first 8 inch floppies and finally to sealed media disks.
@@paulcohen1555 I remember those. Also the ST-277R was a selected ST251 that could handle RLL reliably to get 60 MB instead of 40. Funny how we now have multi-terabyte drives.
Although you didn’t mention it, this is all to do with clock recovery: the receive side clock must synchronise with the incoming data so uses these transitions to do so. Furthermore, for some signalling, sometimes there’s a physics need to maintain DC balance, so over time the encoding leads to equal time spent in each state. Coding such as FM, MFM and Manchester achieve this. Manchester encoding is still used extensively, for example in USB C power delivery signalling over the CC pins of a USB C connector on the physical layer, and 4B5B (like GCR) on top of that at the packet layer. NRZI is used on USB D+/D-, plus a zero bit is stuffed after six ones to avoid too long a period without a transition, so is a semi-variable length coding scheme. So this is all still very relevant.
Yep, found out this NRZI for USB the hard way. Once I was tasked to implement a specific USB camera for some CNC measurement machine. Since it was extremely important to get the images RAW from the sensor and also implement some latching to record the positions when/where the image was taken, it was a homebrew solution using FPGA for the data handling and some other fancy stuff. However since we were running very close to the USB2.0 bandwidth for bulk transfers, it really matters when you send out a nearly completely black image or with lots of white. And since it was a CNC measurement machine, both were likely scenarios. With lots of zeroes in a row, you would loose data. Later I found out it was due to this NRZI. So my quick fix was to XOR all bytes with 0xA5 (10100101) before sending and on the receiving PC XOR it back to get the correct data again. Since it was more likely to have long sequences of 1's or 0's than to have long sequences of 0xA5 or 0x5A, this was a perfectly simple fix to stay within the available USB bandwidth.
Great video. Many moons ago I used to change the interleave on my mfm hdd to get better performance on my 20Mb hard disk. Your wealth of knowledge regarding PCs is astounding. Your articulate presentations are very informative and easy to understand. Well done.
I lived through the transition from MFM to RLL, but never had a good understanding of the differences between the two formats until your lucid explanation. Thank you for that. Also your new book helped me understand my son, who was diagnosed as ADHD but seems to also have some aspects of the spectrum.
Dave you are a godsend! I share your videos with as many people as I can. You've helped me better connect with my autistic son and realize that I'm on the spectrum too.
Thank you SOO much for this video. I just completed a program for restoring old C64 disk images encoded with GCR, but I've always wondered what the difference was between GCR and MFM. Now i know!! I started with zero knowledge of GCR or the 1541 and ended up with a program that could successfully reproduce even the most difficult protection schemes, thus preserving these images for eternity in their original glory
You are absolutely correct that you could use an RLL controller on an MFM drive. In fact, Perstor sold "aftemarket" RLL controllers specifically to use with MFM drives. I used one on my 71MB Maxtor XT-1085 to get over 100MB of storage. I also swapped out one of my floppies to add in my old ST-223 for a 30 MB boot drive. Good times...
Some information that might help some better understand the Manchester encoding explanation: If we look at the example square wave at 4:33 you might reasonably ask yourself (as I did) "how does the read head know where the boundaries between bits are placed?" The corollary of this question being "what if the read head is misaligned by half a bit?" The answer to both questions is that, if the bit boundaries are misaligned by half a bit, there will now no longer always be a transition within each bit for all the data, so the reader knows to shift over the boundaries by half a bit.
Ugh. SCSI. I had to work with that for about 10 years. I thought the rules for terminators were simple, but apparently nobody else did. I had a colleague that could debug SCSI issues at the hardware/firmware level.
@@GeneCash I loved SCSI drives and accessories! It wasn't until SATA or SATA II came along that you could burn a disk and watch a movie at the same time without making a frisbee! SCSI could do it from day 1.
@@GeneCash I've done that. My motivation was: coming in every 18 hours to swap a tape over Xmas break, because the autoloader couldn't talk to the backup software. To the previous sysadmins they were getting O/T pay for it so they didn't care. But I was lazier than that lot:)
DAT, DCC, DVI, HDMI and USB 3.0 (among others) use 8b/10b encoding which sounds like a variation on GCR; for every 8 bits of data, there's a 10 bit pattern which is used to avoid long strings of 0 / 1 at a time. This helps to ensure that the receiving end can extract clock / timing data from the data stream, without needing a separate clock channel (parallel printer ports usually had 8 data lines and a clock line, at a bare minimum). There are also 64b/66b and 128b/130b variations used on other, newer systems. Same basic idea but pushing a greater percentage of the total signal as payload.
There were a lot of conversations on BBSs of the time about putting your MFM disk on an RLL controller and getting more space available. The common thread was YMMV, sometimes it worked and you got more space/performance, sometimes it failed. I recall it being highly dependent on the drive and controller combination, but being so long ago, hard to remember any specifics. What I do remember was, at least for performance, it was better to get a 1:1 MFM controller (WD 1006 card) and stick with the same encoding rather than chance doing an RLL conversion and possibly losing files. If you did a low-level format on an MFM drive with RLL, the drive could end up very iffy...
Yes! I remember my 20mb RLL drive on an Atari 800XL using an ICD MIO. Bonus, with the MIO you could connect a DSSD 3.5in drive for a whopping 320kb of storage. I remember putting every single scrap of data onto the hard drive. Games, programs, I mean everything. Then wondering, "what am I going to do with the 16mb leftover on the hard drive?"
@@8BitNaptime I know man, i had every gizmo known to man for an 800XL, it was quite impressive, had a meg ram, realtime 8 cart, sparta dos x cart, etc. And some of those RGB and s-video monitors are big bucks nowadays. No, I didn't keep any of it. :(
Been a subscriber for years, but this is probably my favorite video on this channel so far. I do need to watch it again to absorb all the knowledge, but I surely know more now than I did 20 minutes ago. I have a rule to never click like on any videos where the creator asks for a like, but I gave a like on this one anyway because it deserves it.
Definitely worth tring a MFM drive on a RLL controller. Especially when they started using plated media. Very reliable and another bonus: 50% more storage per track also meant faster read/write data transfers as the drive was rotating the same speed but the data was transferring that much faster. Made for noticeably faster operation.. Win-win! Ran dozens of different drives that were MFM originally with RLL controllers for years. My old multi-line BBS ran with 3 old Priam(?) drives on an RRL controller. Never lost any data, rand for years... Great video! Love these deep dives into stuff like this. Thanks! Blessings, Stu
When I worked in the HSBC data centre in London (formerly Midland Bank) back in the early 80's, I remember using PE coded tape reels. A couple a couple of years later transitioning to GCR tapes. Also used 80 column punch cards. Thanks Dave, you brought back some old memories.
I am old enough that I have worked on tapes and disk drives that used all of the mentioned recording techniques. The greatest improvement was when the CRC code was written at then of each block so that the data read could be verified against the CRC code and if it was wrong then the hardware would try 3 times to retry reading the data and getting the correct CRC code but if the CRC still indicated an error in the data then the software would use the CRC code to do a mathematical calculation to produce the correct data. I have changed many read/ write heads on head per track disk drives and the old removable disk pack drives requiring special alignment packs and oscilloscope. A few years before I retired all the physical magnetic tapes had been replaced by virtual tapes stored on arrays of hard drives. All the hard drives for fast storage had been replaced with arrays of SSD drives. Original head per track drives that I worked on held 10 MB of data driven by a 1 HP synchronized AC motor, 5 drives per cabinet and 50 MB capacity.
You are absolutely correct that hard drives using the old "ST506" interface could get an instant 50% capacity upgrade by replacing the MFM controller card with an RLL card and then performing a low-level format. Case in point, the venerable ST-225 (20 MB) drive and the ST-238R (30 MB) were essentially the same drive, but the 238R had an RLL controller, while the 225 had an MFM controller. Drive manufacturers warned you to not to use an unsupported controller like this. They said that the surface and timing circuitry of an MFM drive could not reliably handle RLL data, and if you did this, the result would be unreliable and you would void the warranty. Maybe this was technically true, but lots of people did it anyway and it almost always worked. Of course, once RLL drives started shipping, nobody went back to MFM. I think all drives today use some kind of RLL encoding. Also of interest is the GCR encoding used by the Apple II floppy drives. Apple actually used two different kinds of GCR over the years. The original one (DOS 3.2 and older) created disks with 13 sectors (of 256 bytes) per track. With 35 tracks and one head, that produced a capacity of 113.75 KB per disk. But when Apple released DOS 3.3 for the Apple II, they changed to a different GCR encoding that was able to pack more bits on a track. DOS 3.3 disks had 16 sectors (of 256 bytes) per track, for a total capacity of 140 KB per disk. Interestingly, had Apple chosen to format and use all 40 tracks that the drive could (I assume) have formatted, the DOS 3.3 encoding would have produced 160 KB per side - very close to the single-sided capacity of an IBM PC's original floppy drive (which was 9 sectors of 512 bytes per track, for 180 KB on single-sided media or 360 KB on double-sided media). But Apple never went there. I assume there was a technical reason why they limited their 5.25" floppies to only 35 tracks. Also interestingly, Apple stuck with GCR encoding for their original double-density 3.5" drives. But they use a variable-speed spindle motor in order to slow down the rotation and pack more bits (and therefore more sectors) on the outer tracks, where there was more linear space to write them. With an 80 track drive, this produced a single-sided capacity of 400 KB and a double-sided capacity of 800 KB (vs the PC-standard MFM 3.5" disks that were 360 KB single-sided and 720 KB double-sided, using a fixed-speed motor and the same number of sectors per track for all tracks). Apple eventually relented and used industry-standard MFM for their high-density 3.5" floppies, producing the same 1.44 MB that PCs have. And this is why it is nearly impossible for anyone without an old Mac to read an Apple 400K or 800K disk. The GCR encoding and variable number of sectors per tracks makes it impossible for a standard PC floppy drive (with a fixed count of sectors and MFM encoding) to read anything meaningful.
Same reason Commodore also limited their drives to 35 tracks per side: there were a lot of dodgy drive mechanisms and disks in circulation at the time which had trouble reading and writing later tracks reliably. Limiting the format to 35 tracks was an ugly, but effective, solution to this problem.
I often find myself being interested in some of these niche topics. I also watched usagi's video. After I watch a video I like to do research on the topic to get a better understanding. But most of the time the research pulls me out of the happy mood I was in because the topic or idea is hard to wrap my head around. Thank you Dave for making supplemental material that is both interesting and easy to understand. Thank you Dave.
Loved this. I spent time back in the day using my Amiga 500 and writing a 68000 MFM decoder to read files from floppies without using the Amiga ROM routines. Thanks.
Ah ok, the last bit about MFM explains a lot. I remember back in about 1988 being told I could buy a 30MB Seagate harddrive and format it at 40MB under DOS 3.3, or some numbers along those lines. The question back then was "how is that possible?". Nobody knew. Even the people at the computer store didn't know, they just said we could do it even if they didn't know why it worked. Well, it did work, and very well. 10MB was a major win back then.
Umm... dos 3.3 had a 32MB size limit, so no way were you formatting it at 40mb. And by 1988, pretty much all drives were IDE drives. It is *possible* that you had DOS 3.31, and actually had a 40MB drive - people would have been formatting 32MB partitions in 3.3 on a 40MB drive, and then you were "magically" able to format greater than that on 3.31. If it was an MFM/RLL change, you would have got 45MB out of your 30MB hard drive.
When we were involved in PCs in the 80s and early 90s, we encountered literally thousands of RLL controllers running MFM drivers without issue. It only phased out as IDE drives became the norm. Thanks for the lookback!
As someone who was growing up just as MFM/RLL drives were being phased out for IDE I found this quite interesting.... I did format drives MFM/RLL with their controllers but never really understood the technology. Thanks Dave! And yes you could MFM/RLL swaps with the right controllers and drives.
I did the MFM to RLL swap many times on various hard drives, and it worked reliably about 50% of the time. It would go from 17 to 26 sectors per track. Then I found the Perstor PS series controllers, and with those you could get 31 SPT (if I recall correctly), almost doubling the MFM capacity of your hard disk and it was much more reliable than just doing RLL on a MFM drive. I fondly remember my old Atasi 3046 MFM drive - I got about 72 megs out of it. Oh, the memories. :D
Great video, Dave; it was a good review of the early years of data storage. Next, you have to cover the different RAID levels on disk storage and what you get with each level.
When I looked at the title, I just though " I Know all of them but GCR". but when you start explaining how GCR is working, it came back to me. When I worked on mainframe we had to know all the details.
One shop I worked back in the 20/30MB days, we would screen the MFM drives to work as RLL. The yield got better over time to the point they would all work as RLLs. It saved a bit of money.
Of all your great videos, this may be my favorite. I was fortunate to be at IBM in the 80s, working on magnetic and optical storage. Then spent the next three decades designing PSTN and LMR data solutions. Your presentation was excellent!
Great presentation Dave, thanks for that :-) I remember getting more capacity on the same hard drive when upgrading OS. I can't remember the details, but it certainly happened, and the explanation given at the time was "different encoding" - which you confirm here. The first CP/M computers I worked on (1 MHz Z80, later 2 & 4 MHz) had two single sided, single density 8" floppies with 256 KB on each. Later came double sided, single density (512 KB) and in the end double sided, double density (1 MB). Imagine - TWO MEGABYTES total disk space! A HUGE step up from the Apple II, which was my very first computer, with its two 72 KB 5.25" drives for a total of 144 K! That limitation was what launched me on a career in computing, but that's a VERY long story that I'll spare you ;-)
I was a computer operator in the 70's. I heard the letters NZRI. Never knew what it meant at the time. As I was listening to this video, about errors and tape stretching and all the things that could have gone wrong, i must say that aside from the odd data check, the working correctly percentage would have to have been 99.9%. I think the vacuum tubes on the tape drives were there to stop the stretching. Can't remember the tape hardware numbers. Then we had tapes that did not have the vacuum tubes, those desk like ones. I am always amazed at the sheer mental grunt that has gone into computing over the decades. As I was doing my job, there were genuises beavering away all over the world. Like Dave. He can sit there and just rattle off the evolution of magnetic tape storage. I feel so small when I think of the effort and brilliance of human beings in all manner of endeavours, that have made this world what it is today.
you explained this so elegantly that i was surprised when you said you had struggled to understand GCR encoding, as I had understood it as the words left your mouth
Yes, you could format an MFM drive with RLL - but only if the drive was good for it. Early MFM drives could not do RLL as they were not precise enough for RLL, but as the technology grew more mature they could handle the more precise timings of RLL just fine. Modern day hard drives are all RLL encoded internally.
I've been using some of these acronyms all my life and never knew how they worked. Thanks! And, oh yeah, you could plug an MFM PC drive into an RLL controller and get more space. But some drives would work and some would be damaged by this.
The controller created/decoded the analog signal of each track. The small cable carried these signals to and from the head selected over the large digital cable. The disk was a blank magnetic media until the low level format ( remember debug G=C800:5 ) laid down the framework of tracks and sectors. This is how the magic 50% could be created. I forget the details now but there must have been an increase in sectors per track. I never trusted RLL for longer then 6 months even on drives sold as RLL compatible. If I got the drive back before it really corrupted Spinrite would fix it for another 6 months by rewriting the entire track, LL format and data of all tracks. A wonderful explanation of the long forgotten. Huge thanks big Dave.
The problem was that the R/W heads assembly became slightly magnetized and after passing over the stored data track enough times without the track being re-written the magnetized track would be slightly erased. I worked at a large data center with large mainframes and there were a few databases that were read only, only re-written when soft errors began to be detected. A HDA (head disk assembly) would only last about 30 days on one of these read only databases, Engineers came from the disk vendor's facility in California to try to determine the reason for this datacenter was using so many HDAs. The HDAs had to be replaced when they started to have too many errors as the errors were not data errors but servo timing track errors and formatting the HDA did not recreate the servo timing tracks, the servo timing tracks had to be created in the manufacturing plant with a special machine. By using an oscilloscope, I had determine the that the amplitude on the servo timing track had gradually deteriorated over time until errors started to occur, reported the problem to the Engineering section of the vendor and they said it was impossible!! To make a long story short the vendor sent an Engineer to the data center, I escorted him into the data center where the disk drives were located and supplied him with an oscilloscope, he did some scope probing on the backplane of the disk drive then said "it is impossible for any computer to do that many I/Os per second", I replied it is not even a busy time of the transaction day yet. The only solution was to do 4 way mirroring on the disk drives that were used for that read only database. Months later the Engineering Lab from the vendor had discovered the root cause of the degaussing of the servo timing track but by that time the customer was doing a tech turn on their hard drives and another vendor was chosen.
Great video! It brought back memories of working with carrier transmission equipment in the early 90's that used HDB3 and CMI line codes to transmit signals up to 140mb/s. The codes were not only used to embed clocking info, they also had to invert in order to stop capacitance in the coaxial cables from building up and flattening the AC signal.
MFM drive, no internet just some magazine specials about hard drives witch explained how to low level format my new 20 MB extra hard drive. Love your video's. Learn so much mode about the things what keeps me bussy in the 90's
These are very similar problems to those encountered in the telecoms industry when trying to get streams of 1s and 0s to travel along non fibre-optic, copper cables. Transmission lines rounded the square wave transmissions making transitions between low and high states difficult and unreliable to detect. Another issue was the need to have the net voltage on the line around the midpoint between high and low over a period of time regardless of the number of consecutive 1s or 0s transmitted in a sequence. Also, as the telecoms networks became digital, synchronisation and timing information needed to be recovered reliably from the messy signal arriving to ensure network stability and propagation down the synchronisation hierarchy was well managed. All good fun stuff made so much easier by fibre optic cables.
Like a lot of folks, I used to double the capacity of 5-1/4" floppies for my Commodore 64 using a notch cutter so that I could flip the disk over and write on the other side. Gee, that was a long time ago. I had no idea how the data was written. Thanks, Dave..
You're right. In fact I always heard that the Seagate st-225 (20mb mfm) and st-235 (30mb rll) were the same drive. I loved those old systems I had a 60mb mfm drive formatted via rll to 90mb. One of the platters had a bad track 0. I was able to tell the low level format utility to ignore that platter. Lost some space there. My grandfather worked for IBM back in the day, often bragged about being the inventor of RLL. Wish I had some confirmation of it.
Maintaining sync is often cited as being the reason why encoding schemes are required for storing long groups of zeros. This is only a minor reason, the major reason that is rarely ever mentioned is due to magnetic fields only being able to induce a current if they are changing. To read a changing magnetic field the signal is differentiated using an op-amp. This is a mathematical, calculus differentiation where the output voltage is directly proportional to the input voltage’s rate-of-change with respect to time. A second op-amp configured as a comparator can then be used to detect the zero crossings, with its output sent to a one shot multivibrator used in the next stage to create the digital pulses. The flux transitions in the signal are like a continuous, 3D gaussian curve where it can go up and down. Ideally, the head attempting to read the signal needs to be precisely positioned over the peeks and troughs of the gaussian curve. Due to tolerances between different drives, or even hysteresis in the same drive, this can be hard to do and the head can end up on the shoulder of the gaussian curve. To help with this the signal is first amplified by an auto-gain amplifier. A flux transition is used to represent a 1 and the absence of a transition is used to represent a 0. But now there is a problem if you need to store many zeros between the flux transitions. For example, when moving from one transition going up and the other going down the distance between the peeks dictates how many zeros are stored between the two ones. But the more zeros inserted, the more the slope between the gaussian peeks becomes shallower and shallower. The more shallow it becomes the higher the auto gain amplifier ramps up. If there are too many zeros then the slope can become so shallow it almost becomes a constant straight line. Induction stops in the presence of a constant field so this is bad. Mathematically, differentiating a constant is zero. The derivative of a shallow sloping, almost constant signal can be close to zero. So with the first amp with its auto gain maxed and the second differentiating close to zero, any noise can easily cause a random incorrect zero crossing to be sent through to the one shot and be incorrectly detected as a one. Some floppy disk copy protection schemes deliberately did this on purpose. When reading a section of the disk with a slowly changing magnetic field or a section of the disk that was manufactured with no magnetic material multiple times, the code expects the data to be random. If the data is always the same each time this section of the disk is read then the code knows it is a copy.
I remember writing assembly code on my Apple 2+ to read and write raw sectors to crack various copy protection schemes. The hardware controller was pretty dumb and most of the drive characteristics (bit timing, track spacing, etc.) were controlled by software. Great presentation of the low level encodings. A few of this (Manchester and group codes) got heavily used in various network protocols like Ethernet and FDDI, too.
Hi Dave, Dave here. I'm glad to see again that cute old sign-off segment with the tiny furniture and welcoming narration. I've been lucky enough to live through all of the technologies you've talked about being used. I *think* I've seen a drum drive, has the chance to make the heads of an IBM 1620 disc drive move using assembly language. I still have a CPM machine with a Tarbell Tape Drive board in it. Now I took an the proud owner of boxes of my old hard drives from XTs to the current computers. Most haven't been spun since their host computers were recycled. And there is the rub, what am I going to use on my 40 plus year old Seagate? I'm sure the technology is out there, but I'm not that motivated to relearn the old ways of forgotten hardware and software. Sigh.
I don't know about being "lucky" enough to live through all that, but I did enjoy the days of wrangling a bunch of CompuPro boxes on Arcnet with some 40MB drives that were the size of a modern desktop. I helped transition the UCF undergrad lab from Arcnet to Ethernet. Right now my 3D printer is creating a thing to help me clean my motorcycle chains. An hour ago that design was just a page of OpenSCAD code. That is absolute magic.
6:43 fun fact: the USB Power Delivery protocol also uses this strategy of encoding 4 bits into 5. However is also uses some of the left over 5 bit codes for special uses.
I remember having two 20MB seagate drives on my BBS and I bought a brand new on the market RLL controller. I copied all the BBS software and scripts to floppies as well as the second hard disk, reformatted the first Hard drive to RLL and copied it all back - Viola ! 30MB. Re-Loaded (copied) all the software back to that drive after re-installing DOS and did the same thing to the second drive and now I had two 30MB drives. It so seemed like magic at the time. Some time after that, I got an ESDI drive, then a SCSI drive and then an IDE drive. It was an incredible journey.
@@DavesGarage - LOL. Memories 1 - I got 2 bigger capacity drives in the days where they were expensive (in Oz) and ended up only paying for a controller which was less than half the price of one drive. The Diagnostic BBS needed space urgently and that did it nicely. Good memories.
I worked for Hard Drives International back in the MFM/RLL/ESDI/SCSI days (and early IDE/xt-ide). Yes, you could take a Seagate st225 MFM drive, put it on an RLL controller and if you were lucky you'd get the same capacity of a ST-238r. Basically, a 50% increase in storage. If you weren't lucky, it would work but you would end up with data corruption.
About the MFM vs RLL encoding scheme.. I do remember finally getting enough money together to get a 20 MB hard drive for my IBM PC clone. It was MFM encoded. A year later or so later a friend of mine said he got a 30 MB hard drive for his IBM clone. He said the drive mechanism was identical to mine except it used RLL encoding. So I do remember those discussions about RLL being able to store 50% more then MFM. I wouldn’t have had the confidence to try swapping just the controller on my drive to see if it worked. Especially typically I built my PC’s at the time by just buying all new technology anyway. My next hard drive was in a whole new computer, everything was much faster, stored more, faster ram ,etc.
Yes. I do remember taking an MFM drive, putting it onto an RLL controller and having it work. Granted, the MFM drive required low-level formatting to work on the RLL controller.
I remember that when the first RLL drives came out (30MB), they were essentially MFM (20MB) drives hooked into an RLL controller. However, that resulted in those drives being much more finicky and data loss reliability on those first 30MB drives was a problem. It always seemed to me that RLL drives throughout the 1980s were less reliable. The term "RLL" still gives me chills.
PRML (Partial Response/Maximal Likelihood) was a key advancement in the read-channel that significantly boosted bit density. Disk drives can get rather nasty at the inner cylinders where the bit-density is higher and bits interfere with eachother. Funny fact: When I graduated in 1985, at one of my interviews the sys admin was very proud to show-off their brand new 1GB disk drive. It was the size (and noise-level) of a residential dishwasher. Today, my dashcam has a 256GB microSD that's smaller than my fingernail.....what will we see 40 years from now ????
As well as regular improvements in data recording devices, there was a corresponding improvement in the recording media. Greater data densities required denser magnetic particles which needed stronger magnetic forces to record on. A similar situation occurred with audio cassettes. Just as a recorder designed to work with basic ferric oxide tapes were unable to successfully record on higher quality Chrome tapes, 40 track single density drives couldn’t write to media designed for use on 80 track double density drives even though they could usually read the higher quality media if the data on it had been written to it at the lower specification by a drive capable of writing at the higher densities.
When you mentioned frequent modulation I thought you were going to go into how some early consumer computers stored data on audio cassettes. The second computer I built (from a kit in Australia) used this method. I replaced the original 8080 CPU with a Z80, doubled the clock speed, and then used a few spare gates on the board to software select the original clock speed when reading or writing tape as the system ROM routines relied on the system clock for generating and reading the audio signals.
Yes, you could use a MFM drive with a RLL controller with the caveat that the drive might not be of high enough quality to support it well. It's a matter of manufacturing tolerances and qualities of the magnetic media. I'm sure some drive manufacturers sold their RLL drives that didn't pass quality checks as MFM drives. Quite often the the difference between an MFM and RLL drive was just the label having a different suffix letter on the model number.
Yes, you could take an RLL controller onto a MFM drive and get more space. My buddy and I did that as well, AND.... if you you really got crazy, you could change the clock speed on the RLL controller, and you could cram even more on the same disk. There was a limit though because too fast and as you'd expect data errors really started to creep in. What a great time to be playing and learning 'cutting edge' computers and their hardware. I do miss setting interrupts with jumper pins though!
I definitely remember getting an RLL controller to replace my MFM controller and low-level reformatting my hard disk to get something like 50% more space. This was in the early 1990s, when everyone was switching to IDE, so old MFM and RLL controllers and drives were easy to find second-hand, and cheap too. Especially Western Digital controllers. And here's why: There had been an open source hardware project in a German magazine that allowed you to connect an ISA MFM or RLL controller and hard disks to an Amiga, and they said it didn't work with Western Digital controllers and didn't know why. So it was almost impossible to find other controllers because everyone was buying those, but Western Digital controllers were a dime a dozen. I actually built one of those projects with a Western Digital controller, and reverse-engineered the software (with a little help of the hard disk BIOS listings in the IBM PC/XT Technical Reference manual). Turns out the only reason why it didn't work with Western Digital controllers was that the software didn't hold the Reset line long enough. That was easy to fix, and I used a 70MB hard disk on my Amiga as a 105MB hard disk for years. And that was a lot of space back then!
On the subject of using the same drives with both MFM and RLL controllers, I had the opposite experience. I had a 30mb RLL drive/controller combination that was proving very un-reliable. I was constantly having issues and re-formatting the drive. Eventually I conceded defeat and connected the drive to an MFM controller, re-formatted and ended up with a very reliable 20mb drive. Lost 50% capacity but at the time it was worth it for rock solid reliability. I guess the head alignment was slightly off and struggled with RLL but was perfectly happy with MFM. Good times!
I remember formatting a 20MB MFM hdd with RLL , and getting 35MB from it. There were more chances of reports of bad sectors, but even then the 5.25” hard disk I was using was somewhat antiquated!
On PE, part of why it was popular was you could do faster encoding without worrying about syncing up to your clock signal which I believe often meant you could do faster encoding those putting more data per inch of tape.
PE is used for the data tracks on credit cards. You can record it as simply as pulsing in two polarities. Reading it back it shows up as pulses either up or down on a scope and is EZ to decode using a comparater + AND gate.
I closely followed the research on the GMR effect in Switzerland. The GMR effect was discovered by Albert Fert and Peter Grünberg, for which they were awarded the Nobel Prize later in 2007. It had a significant impact on hard drive technology, as the read heads utilized this effect. The first hard drives featuring this technology were released by IBM with the Deskstar 16GP in 1997 and were not even that expensive. I bought two 14GP models from IBM at the time.
Yes, can remember swapping out MFM controllers and installing RLL controllers. Worked a treat. You could improve a 10MB drive to about 15MB. A big deal in the day…
Back in 80's I worked for Micro Technology (tunbridge wells) and then Plus 5 Engineering (crowborough), who USP was producing external hard disk and tape units for PC''s that matched the PC cabinet design, so different containers for IBM PC, Compaq, Sirrus, etc etc. I wrote all the disk and tape device drivers for CP/M, IBM Dos, MSDOS etc, and eventually Xenix. These units were really expensive but gave a PC 10MB of storage, yes a whole 10MB. My watch today has 32GB of user storage, what progression. I even wrote a shared disk system between several PC's connected by SCSI flat cables and abritrating access using the SCSI id's. Happy days.
Yep it was my first job working for a importer distributor in new zealand selling hard drives, controllers to pc dealers over the whole country. we had Miniscribe 8425 5 1/4 " 20mb drives that used MFM ISA controller where as a 8435 5 /14" 30mb drive used a 27RLL ISA controler which was the same as the 20mb drive, but was rated as being more reliable to handle the extra 50% increase in data density onto the platters. we also had and also 3425's 3.5" 20mb drives later on. And yes there was the case where miniscribe shipped bricks on ships... they were failing at producing enough products that were capable of keeping up with performace improved western digital drives....
analog computing is fascinating, it naturally forces more creative solutions to practical problems, when you can't even assume that data is already encoded into logical binary frames
Flashback to 1990 when I decided to implement low level floppy I/O in my Amiga games with help from the blitter, same way as the system. The blitter made the encoding and decoding much faster and simpler. In retrospect, it was pure chutzpah to use my own floppy I/O.
You're reminding me of when I did a report in college on how the CD recorded data. Dave, this David says you will find that very interesting. I still remember parts of it all these years later. Like the 8 to 14-bit transformation where there may be no more than 10 zeros in a row and no fewer than two zeros in a row or was it three? And like some of your encoding techniques, it was the change that was the one, not the surface. They used to change through time it, and they use the laser reader to align it. I also had a lot of checksums in the blocks of data. Do one of these videos on how a CD is encoded. For real, do one because they're interesting.
Ooohhhh, Thank you for answering a many decade old question from the back of my mind. I was in grade school at the time but I definitely remember a day where my dad brought the family 286 back from the shop and our previously fairly full 40Mb HDD was now reading as 60Mb and seemed much roomier and even a bit faster. My dad had told me that it wasn't a replacement drive but a replacement drive controller (I thought it was MFM to RLL but that might just be confirmation bias after seeing the video.) I always wondered how a controller change could change the capacity of a drive by such a large amount and now I know!
My first hard drive was an MFM 20MB ST-225, and the controller that I got with it was a single port controller. When I got my second drive, it was a 30MB ST-238R, and came with a dual port controller card. At some point, I ended up reformatting the ST-225 to RLL on the dual port card, giving me 60MB of storage in one XT machine. Those drives were with me until I went to upgrade Windows 3.0 to 3.1, and 3.1 no longer supported the 8 bit controller. At that point, I picked up a 50MB Quantum Fireball IDE, and the old drives got used to make a gaming computer for the wife and kids ( think 286 CPU and monochrome graphics, great for hangman and board games. )
I don’t remember how i did it, but back in the day on the Amiga i think i used DMA to read the MFM data in the middle of the disk. Decoded the MFM with the blitter (also DMA) and analyzed the BAM to proceed to read the meaningfull data of the disk. This could read and decoded the data without using cpu power. The goal was to use the cpu to send the data over the parallel cable so we could copy disks from Amiga to Amiga at incredible speed at copy parties 😇 I tested, but never finished the “networking” part, a bit of a shame because the idea was great.
Amiga had a funky timing thing too. Instead of x sectors per track with gaps for timing signals, the sectors abutted one-another with a 'special' sync mark that couldn't ordinarily be written with MFM -- 3389 -- that's why AmigaDOS disks were able to pack 880K instead of the conventional 720K.
Very interesting. Some of the signal processing methods were invented by my FIL in a backyard hammock in Raleigh NC (for IBM). I still find it interesting to compare some of the patents of his methods to the implementations used for these devices.
A number of serial line codes were developed to eliminate any sort of DC bias present in the signalling method as this caused problems for transmitters and in some cases, radio equipment due to the bandwidth required. 8b/10b maps 8 bits to a 10 bit string. 64b/66b achieves the same on a longer code space. Unrelated, it's possible to configure a modern UART to run 9600, N, 8, 2 (two stop bits), which means the whole string is 10 bits long and will show up on the oscilloscope as a nicely divided number. For instance, AX.25 or X.25 uses a polynomial scrambler in an attempt to eliminate DC bias as well. The feedback taps on the shift register were well known to produce that polynomial and the FCC allowed only three sets tap configurations.
In the 80z my 48k 8 bit sinclair spectrum had a tape drive but was using sound to store the data on a tape recorder. It sounded like a modem when it was loading or saving a program and was fun mostly.
I enjoyed it Dave. It’s relevant to me as I’m altering the way my homebrew computer stores and reads data from tape. Yes, SD cards exist, and no, I’m not going that way just because. I may add floppy though.
My first PC (386SX) in 1991 had a full height 5.25" 40MB hard drive. I remember looking through the manual for the hard drive controller and there was a mention of the default encoding being MFM but after changing a jumper it could be RLL. I didn't really know what this was... but I changed it, did a low level format and my 40MB drive turned into a 63MB drive! It was magical 🙂
Great video Dave! I remember taking my 20MB HDD and attaching it to an RLL controller and it formatted out to 30MB. I wasn't sure it would work, but to my surprise it worked great! Was cool to brag about it back in the day. 🤠
Adrian's Digital Basement channel had an episode of recovering data on some old MFM and RLL drives. He also went through explaining the encodings used. It was very interesting. My first HDD was an MFM 40MB drive. I think I paid $500 for it at the time. Had to low-level format it before you could then partition it then high-level formatting the drive so the OS can store any files on it. Ah, those were the days.
13:45 - Absolutely have memories of this and all through the video I was hoping for an explanation of how and why this worked. That "trick" combined with using a magical software called "stacker" was amazing in the early 90's on XT clones!
Dave didn’t go into this specifically,, but for those who remember “single density” vs. “double density” floppy drives, single density used FM encoding while double density used MFM encoding.
Cool! I did not know that!
Man, I feel ancient for remembering that... [creak]
@@GeneCash I know my first day on a new job they were rolling out a 6ft long drum disk from the Air Force facility I spent the next 8 years working on everything Dave talked about from 7 track tapes to 40/300mbyte spinning multi platter disks to the first 8 inch floppies and finally to sealed media disks.
Sure, done that some times.
ST238 was a selected ST225. That 20MB MFM drive on a RLL controller gave 30MB.
@@paulcohen1555 I remember those. Also the ST-277R was a selected ST251 that could handle RLL reliably to get 60 MB instead of 40. Funny how we now have multi-terabyte drives.
I understood about 40% of this but loved every minute of it.
Is that before or after the bit lookup table? 😆
You have great pacing and cadence when you explain concepts. Very clear.
Although you didn’t mention it, this is all to do with clock recovery: the receive side clock must synchronise with the incoming data so uses these transitions to do so.
Furthermore, for some signalling, sometimes there’s a physics need to maintain DC balance, so over time the encoding leads to equal time spent in each state. Coding such as FM, MFM and Manchester achieve this.
Manchester encoding is still used extensively, for example in USB C power delivery signalling over the CC pins of a USB C connector on the physical layer, and 4B5B (like GCR) on top of that at the packet layer.
NRZI is used on USB D+/D-, plus a zero bit is stuffed after six ones to avoid too long a period without a transition, so is a semi-variable length coding scheme.
So this is all still very relevant.
Thanks for the explanation 👍
Yep, found out this NRZI for USB the hard way. Once I was tasked to implement a specific USB camera for some CNC measurement machine.
Since it was extremely important to get the images RAW from the sensor and also implement some latching to record the positions when/where the image was taken, it was a homebrew solution using FPGA for the data handling and some other fancy stuff.
However since we were running very close to the USB2.0 bandwidth for bulk transfers, it really matters when you send out a nearly completely black image or with lots of white. And since it was a CNC measurement machine, both were likely scenarios.
With lots of zeroes in a row, you would loose data. Later I found out it was due to this NRZI.
So my quick fix was to XOR all bytes with 0xA5 (10100101) before sending and on the receiving PC XOR it back to get the correct data again.
Since it was more likely to have long sequences of 1's or 0's than to have long sequences of 0xA5 or 0x5A, this was a perfectly simple fix to stay within the available USB bandwidth.
Great video. Many moons ago I used to change the interleave on my mfm hdd to get better performance on my 20Mb hard disk. Your wealth of knowledge regarding PCs is astounding. Your articulate presentations are very informative and easy to understand. Well done.
I lived through the transition from MFM to RLL, but never had a good understanding of the differences between the two formats until your lucid explanation. Thank you for that. Also your new book helped me understand my son, who was diagnosed as ADHD but seems to also have some aspects of the spectrum.
Dave you are a godsend! I share your videos with as many people as I can. You've helped me better connect with my autistic son and realize that I'm on the spectrum too.
Thank you SOO much for this video. I just completed a program for restoring old C64 disk images encoded with GCR, but I've always wondered what the difference was between GCR and MFM. Now i know!! I started with zero knowledge of GCR or the 1541 and ended up with a program that could successfully reproduce even the most difficult protection schemes, thus preserving these images for eternity in their original glory
You are absolutely correct that you could use an RLL controller on an MFM drive. In fact, Perstor sold "aftemarket" RLL controllers specifically to use with MFM drives. I used one on my 71MB Maxtor XT-1085 to get over 100MB of storage. I also swapped out one of my floppies to add in my old ST-223 for a 30 MB boot drive. Good times...
I remember having a Seagate MFM 40MB hard drive that I reformatted to 67 MB RLL with a controller I bought off of Computer Shopper.
@@ronjenkins6674 ST251-1? Had one of those... didn't convert it to RLL though...
Right I was wracking my Brain.... I kept coming up with Plextor card but was Perstor Plextor was a HDD brand I think.... god Im old...
I had a Miniscribe 3053 that would not format to RLL. It would only run on MFM.
I remember RLL'ing ST-225's to get 30MB.
Some information that might help some better understand the Manchester encoding explanation:
If we look at the example square wave at 4:33 you might reasonably ask yourself (as I did) "how does the read head know where the boundaries between bits are placed?" The corollary of this question being "what if the read head is misaligned by half a bit?" The answer to both questions is that, if the bit boundaries are misaligned by half a bit, there will now no longer always be a transition within each bit for all the data, so the reader knows to shift over the boundaries by half a bit.
Having dedicated the better part of two decades to study and invention in data compression, I found this to be a really enjoyable watch. Thank Dave!
man, I bet you could make a pretty neat presentation with the different compression formats that used to be big back in the 1200 baud BBS days...
I wrote mainframe tape analysis software in the 70s. Thank you for the walk down memory lane!
Waiting for the electrical interface vid! SCSI, ESDI, SATA, IDE for hard drives and maybe the various floppy drive interfaces.
Ugh. SCSI. I had to work with that for about 10 years. I thought the rules for terminators were simple, but apparently nobody else did. I had a colleague that could debug SCSI issues at the hardware/firmware level.
@@GeneCash It was the king of enterprise interfaces for quite awhile, but yeah not too user friendly for Joe Blow.
@@GeneCash I loved SCSI drives and accessories! It wasn't until SATA or SATA II came along that you could burn a disk and watch a movie at the same time without making a frisbee! SCSI could do it from day 1.
@@GeneCash I've done that. My motivation was: coming in every 18 hours to swap a tape over Xmas break, because the autoloader couldn't talk to the backup software. To the previous sysadmins they were getting O/T pay for it so they didn't care. But I was lazier than that lot:)
Me too!
DAT, DCC, DVI, HDMI and USB 3.0 (among others) use 8b/10b encoding which sounds like a variation on GCR; for every 8 bits of data, there's a 10 bit pattern which is used to avoid long strings of 0 / 1 at a time. This helps to ensure that the receiving end can extract clock / timing data from the data stream, without needing a separate clock channel (parallel printer ports usually had 8 data lines and a clock line, at a bare minimum).
There are also 64b/66b and 128b/130b variations used on other, newer systems. Same basic idea but pushing a greater percentage of the total signal as payload.
And CDs use EFM: Eight-to-Fourteen-Modulation for encoding data.
Ethernet as well. 10Mbps Ethernet used Manchester encoding. 100/Gbit use 8b/10b encoding
Confirmed: DCC uses eight-to-ten encoding and CD uses eight-to-fourteen encoding. Both invented by Kees Schouwhamer Immink at Philips.
@@JacGoudsmitI still have my DCC recorder from when I worked at NAP in the 90's.
There were a lot of conversations on BBSs of the time about putting your MFM disk on an RLL controller and getting more space available. The common thread was YMMV, sometimes it worked and you got more space/performance, sometimes it failed. I recall it being highly dependent on the drive and controller combination, but being so long ago, hard to remember any specifics. What I do remember was, at least for performance, it was better to get a 1:1 MFM controller (WD 1006 card) and stick with the same encoding rather than chance doing an RLL conversion and possibly losing files. If you did a low-level format on an MFM drive with RLL, the drive could end up very iffy...
Hello 1990! I remember that very well!
Yes! I remember my 20mb RLL drive on an Atari 800XL using an ICD MIO. Bonus, with the MIO you could connect a DSSD 3.5in drive for a whopping 320kb of storage. I remember putting every single scrap of data onto the hard drive. Games, programs, I mean everything. Then wondering, "what am I going to do with the 16mb leftover on the hard drive?"
Hope you kept that thing, probably very valuable now.
@@8BitNaptime I know man, i had every gizmo known to man for an 800XL, it was quite impressive, had a meg ram, realtime 8 cart, sparta dos x cart, etc. And some of those RGB and s-video monitors are big bucks nowadays. No, I didn't keep any of it. :(
A question to answer a prompt to your question: would the 16 mb be cached for delivery on the next cycle?
@@msromike123 I felt a slight stabbing pain in my nerd heart reading that.
That was awesome!
These techniques/concepts also apply to communications cables (Ethernet, HDMI, USB, SATA) and to optical media (CDs, DVDs)
Love how you broke down the features and made it easy to understand. Super helpful
Been a subscriber for years, but this is probably my favorite video on this channel so far. I do need to watch it again to absorb all the knowledge, but I surely know more now than I did 20 minutes ago. I have a rule to never click like on any videos where the creator asks for a like, but I gave a like on this one anyway because it deserves it.
Definitely worth tring a MFM drive on a RLL controller. Especially when they started using plated media. Very reliable and another bonus: 50% more storage per track also meant faster read/write data transfers as the drive was rotating the same speed but the data was transferring that much faster. Made for noticeably faster operation.. Win-win! Ran dozens of different drives that were MFM originally with RLL controllers for years. My old multi-line BBS ran with 3 old Priam(?) drives on an RRL controller. Never lost any data, rand for years... Great video! Love these deep dives into stuff like this. Thanks! Blessings, Stu
When I worked in the HSBC data centre in London (formerly Midland Bank) back in the early 80's, I remember using PE coded tape reels. A couple a couple of years later transitioning to GCR tapes. Also used 80 column punch cards.
Thanks Dave, you brought back some old memories.
I am old enough that I have worked on tapes and disk drives that used all of the mentioned recording techniques. The greatest improvement was when the CRC code was written at then of each block so that the data read could be verified against the CRC code and if it was wrong then the hardware would try 3 times to retry reading the data and getting the correct CRC code but if the CRC still indicated an error in the data then the software would use the CRC code to do a mathematical calculation to produce the correct data. I have changed many read/ write heads on head per track disk drives and the old removable disk pack drives requiring special alignment packs and oscilloscope. A few years before I retired all the physical magnetic tapes had been replaced by virtual tapes stored on arrays of hard drives. All the hard drives for fast storage had been replaced with arrays of SSD drives. Original head per track drives that I worked on held 10 MB of data driven by a 1 HP synchronized AC motor, 5 drives per cabinet and 50 MB capacity.
You are absolutely correct that hard drives using the old "ST506" interface could get an instant 50% capacity upgrade by replacing the MFM controller card with an RLL card and then performing a low-level format.
Case in point, the venerable ST-225 (20 MB) drive and the ST-238R (30 MB) were essentially the same drive, but the 238R had an RLL controller, while the 225 had an MFM controller.
Drive manufacturers warned you to not to use an unsupported controller like this. They said that the surface and timing circuitry of an MFM drive could not reliably handle RLL data, and if you did this, the result would be unreliable and you would void the warranty. Maybe this was technically true, but lots of people did it anyway and it almost always worked.
Of course, once RLL drives started shipping, nobody went back to MFM. I think all drives today use some kind of RLL encoding.
Also of interest is the GCR encoding used by the Apple II floppy drives. Apple actually used two different kinds of GCR over the years. The original one (DOS 3.2 and older) created disks with 13 sectors (of 256 bytes) per track. With 35 tracks and one head, that produced a capacity of 113.75 KB per disk. But when Apple released DOS 3.3 for the Apple II, they changed to a different GCR encoding that was able to pack more bits on a track. DOS 3.3 disks had 16 sectors (of 256 bytes) per track, for a total capacity of 140 KB per disk.
Interestingly, had Apple chosen to format and use all 40 tracks that the drive could (I assume) have formatted, the DOS 3.3 encoding would have produced 160 KB per side - very close to the single-sided capacity of an IBM PC's original floppy drive (which was 9 sectors of 512 bytes per track, for 180 KB on single-sided media or 360 KB on double-sided media). But Apple never went there. I assume there was a technical reason why they limited their 5.25" floppies to only 35 tracks.
Also interestingly, Apple stuck with GCR encoding for their original double-density 3.5" drives. But they use a variable-speed spindle motor in order to slow down the rotation and pack more bits (and therefore more sectors) on the outer tracks, where there was more linear space to write them. With an 80 track drive, this produced a single-sided capacity of 400 KB and a double-sided capacity of 800 KB (vs the PC-standard MFM 3.5" disks that were 360 KB single-sided and 720 KB double-sided, using a fixed-speed motor and the same number of sectors per track for all tracks). Apple eventually relented and used industry-standard MFM for their high-density 3.5" floppies, producing the same 1.44 MB that PCs have.
And this is why it is nearly impossible for anyone without an old Mac to read an Apple 400K or 800K disk. The GCR encoding and variable number of sectors per tracks makes it impossible for a standard PC floppy drive (with a fixed count of sectors and MFM encoding) to read anything meaningful.
Same reason Commodore also limited their drives to 35 tracks per side: there were a lot of dodgy drive mechanisms and disks in circulation at the time which had trouble reading and writing later tracks reliably. Limiting the format to 35 tracks was an ugly, but effective, solution to this problem.
I often find myself being interested in some of these niche topics. I also watched usagi's video.
After I watch a video I like to do research on the topic to get a better understanding. But most of the time the research pulls me out of the happy mood I was in because the topic or idea is hard to wrap my head around.
Thank you Dave for making supplemental material that is both interesting and easy to understand.
Thank you Dave.
Loved this. I spent time back in the day using my Amiga 500 and writing a 68000 MFM decoder to read files from floppies without using the Amiga ROM routines. Thanks.
Just as I did in the games “Dino Wars” and “Bill ‘n’ Ted’s Excellent Adventure” because hardware direct was the one true way!
Your data output and format is magically synced to my brains input circuitry...
Ah ok, the last bit about MFM explains a lot. I remember back in about 1988 being told I could buy a 30MB Seagate harddrive and format it at 40MB under DOS 3.3, or some numbers along those lines. The question back then was "how is that possible?". Nobody knew. Even the people at the computer store didn't know, they just said we could do it even if they didn't know why it worked. Well, it did work, and very well. 10MB was a major win back then.
That might have been a case of cramming more sectors per track, either/both with shorter gaps or using more of the slack space after the last sector.
Umm... dos 3.3 had a 32MB size limit, so no way were you formatting it at 40mb. And by 1988, pretty much all drives were IDE drives. It is *possible* that you had DOS 3.31, and actually had a 40MB drive - people would have been formatting 32MB partitions in 3.3 on a 40MB drive, and then you were "magically" able to format greater than that on 3.31. If it was an MFM/RLL change, you would have got 45MB out of your 30MB hard drive.
@@Rybagz Only if he was hacking the hardware... since the computer store didn't know, I very much doubt they were doing that.
Great from the ground up explanation of encoding for magnetic media that extends to almost all other storage, RF and optical applications.
When we were involved in PCs in the 80s and early 90s, we encountered literally thousands of RLL controllers running MFM drivers without issue. It only phased out as IDE drives became the norm. Thanks for the lookback!
As someone who was growing up just as MFM/RLL drives were being phased out for IDE I found this quite interesting.... I did format drives MFM/RLL with their controllers but never really understood the technology. Thanks Dave! And yes you could MFM/RLL swaps with the right controllers and drives.
Extremely cool delivey of the computer science & application. I absolutely love your channel! My favorite out of all content creators. Thank you Dave!
I did the MFM to RLL swap many times on various hard drives, and it worked reliably about 50% of the time. It would go from 17 to 26 sectors per track. Then I found the Perstor PS series controllers, and with those you could get 31 SPT (if I recall correctly), almost doubling the MFM capacity of your hard disk and it was much more reliable than just doing RLL on a MFM drive. I fondly remember my old Atasi 3046 MFM drive - I got about 72 megs out of it. Oh, the memories. :D
Great video, Dave; it was a good review of the early years of data storage.
Next, you have to cover the different RAID levels on disk storage and what you get with each level.
Yes, RAID would be a good candidate for a topic in another video - imo
When I looked at the title, I just though " I Know all of them but GCR". but when you start explaining how GCR is working, it came back to me. When I worked on mainframe we had to know all the details.
Best overview of these techniques I recall ever seeing.
One shop I worked back in the 20/30MB days, we would screen the MFM drives to work as RLL. The yield got better over time to the point they would all work as RLLs. It saved a bit of money.
Of all your great videos, this may be my favorite. I was fortunate to be at IBM in the 80s, working on magnetic and optical storage. Then spent the next three decades designing PSTN and LMR data solutions. Your presentation was excellent!
Thanks! It's a pretty niche topic, but a few folks enjoyed it!
Great presentation Dave, thanks for that :-) I remember getting more capacity on the same hard drive when upgrading OS. I can't remember the details, but it certainly happened, and the explanation given at the time was "different encoding" - which you confirm here. The first CP/M computers I worked on (1 MHz Z80, later 2 & 4 MHz) had two single sided, single density 8" floppies with 256 KB on each. Later came double sided, single density (512 KB) and in the end double sided, double density (1 MB). Imagine - TWO MEGABYTES total disk space! A HUGE step up from the Apple II, which was my very first computer, with its two 72 KB 5.25" drives for a total of 144 K! That limitation was what launched me on a career in computing, but that's a VERY long story that I'll spare you ;-)
I was a computer operator in the 70's. I heard the letters NZRI. Never knew what it meant at the time. As I was listening to this video, about errors and tape stretching and all the things that could have gone wrong, i must say that aside from the odd data check, the working correctly percentage would have to have been 99.9%. I think the vacuum tubes on the tape drives were there to stop the stretching. Can't remember the tape hardware numbers. Then we had tapes that did not have the vacuum tubes, those desk like ones. I am always amazed at the sheer mental grunt that has gone into computing over the decades. As I was doing my job, there were genuises beavering away all over the world. Like Dave. He can sit there and just rattle off the evolution of magnetic tape storage. I feel so small when I think of the effort and brilliance of human beings in all manner of endeavours, that have made this world what it is today.
you explained this so elegantly that i was surprised when you said you had struggled to understand GCR encoding, as I had understood it as the words left your mouth
Yes the MFM on an "RLL" drive did actually work that way, at least on certain SCSI controllers.
Yes, you could format an MFM drive with RLL - but only if the drive was good for it. Early MFM drives could not do RLL as they were not precise enough for RLL, but as the technology grew more mature they could handle the more precise timings of RLL just fine. Modern day hard drives are all RLL encoded internally.
I've been using some of these acronyms all my life and never knew how they worked. Thanks!
And, oh yeah, you could plug an MFM PC drive into an RLL controller and get more space. But some drives would work and some would be damaged by this.
The controller created/decoded the analog signal of each track. The small cable carried these signals to and from the head selected over the large digital cable. The disk was a blank magnetic media until the low level format ( remember debug G=C800:5 ) laid down the framework of tracks and sectors. This is how the magic 50% could be created. I forget the details now but there must have been an increase in sectors per track. I never trusted RLL for longer then 6 months even on drives sold as RLL compatible. If I got the drive back before it really corrupted Spinrite would fix it for another 6 months by rewriting the entire track, LL format and data of all tracks. A wonderful explanation of the long forgotten. Huge thanks big Dave.
The problem was that the R/W heads assembly became slightly magnetized and after passing over the stored data track enough times without the track being re-written the magnetized track would be slightly erased. I worked at a large data center with large mainframes and there were a few databases that were read only, only re-written when soft errors began to be detected.
A HDA (head disk assembly) would only last about 30 days on one of these read only databases, Engineers came from the disk vendor's facility in California to try to determine the reason for this datacenter was using so many HDAs. The HDAs had to be replaced when they started to have too many errors as the errors were not data errors but servo timing track errors and formatting the HDA did not recreate the servo timing tracks, the servo timing tracks had to be created in the manufacturing plant with a special machine. By using an oscilloscope, I had determine the that the amplitude on the servo timing track had gradually deteriorated over time until errors started to occur, reported the problem to the Engineering section of the vendor and they said it was impossible!!
To make a long story short the vendor sent an Engineer to the data center, I escorted him into the data center where the disk drives were located and supplied him with an oscilloscope, he did some scope probing on the backplane of the disk drive then said "it is impossible for any computer to do that many I/Os per second", I replied it is not even a busy time of the transaction day yet. The only solution was to do 4 way mirroring on the disk drives that were used for that read only database. Months later the Engineering Lab from the vendor had discovered the root cause of the degaussing of the servo timing track but by that time the customer was doing a tech turn on their hard drives and another vendor was chosen.
Yey! The outro is back!
Its what charmed me into staying between the technical talk.
I just found Usagi Electric a few weeks before this video.
Great video! It brought back memories of working with carrier transmission equipment in the early 90's that used HDB3 and CMI line codes to transmit signals up to 140mb/s. The codes were not only used to embed clocking info, they also had to invert in order to stop capacitance in the coaxial cables from building up and flattening the AC signal.
Your’re correct, I used MFM drives on RLL controllers all the time to double my hard drive space. Worked great on most MFM drives.
MFM drive, no internet just some magazine specials about hard drives witch explained how to low level format my new 20 MB extra hard drive. Love your video's. Learn so much mode about the things what keeps me bussy in the 90's
I loved your reference to disk drives as "spinning rust"! Hadn't heard that one before. 😀
These are very similar problems to those encountered in the telecoms industry when trying to get streams of 1s and 0s to travel along non fibre-optic, copper cables. Transmission lines rounded the square wave transmissions making transitions between low and high states difficult and unreliable to detect. Another issue was the need to have the net voltage on the line around the midpoint between high and low over a period of time regardless of the number of consecutive 1s or 0s transmitted in a sequence. Also, as the telecoms networks became digital, synchronisation and timing information needed to be recovered reliably from the messy signal arriving to ensure network stability and propagation down the synchronisation hierarchy was well managed. All good fun stuff made so much easier by fibre optic cables.
Like a lot of folks, I used to double the capacity of 5-1/4" floppies for my Commodore 64 using a notch cutter so that I could flip the disk over and write on the other side. Gee, that was a long time ago. I had no idea how the data was written. Thanks, Dave..
Friendly Giant FTW! I'm also a Canadian of a certain age who has spent decades in the US, but who can forget the shows of one's childhood?
You're right. In fact I always heard that the Seagate st-225 (20mb mfm) and st-235 (30mb rll) were the same drive. I loved those old systems I had a 60mb mfm drive formatted via rll to 90mb. One of the platters had a bad track 0. I was able to tell the low level format utility to ignore that platter. Lost some space there. My grandfather worked for IBM back in the day, often bragged about being the inventor of RLL. Wish I had some confirmation of it.
Maintaining sync is often cited as being the reason why encoding schemes are required for storing long groups of zeros. This is only a minor reason, the major reason that is rarely ever mentioned is due to magnetic fields only being able to induce a current if they are changing. To read a changing magnetic field the signal is differentiated using an op-amp. This is a mathematical, calculus differentiation where the output voltage is directly proportional to the input voltage’s rate-of-change with respect to time. A second op-amp configured as a comparator can then be used to detect the zero crossings, with its output sent to a one shot multivibrator used in the next stage to create the digital pulses.
The flux transitions in the signal are like a continuous, 3D gaussian curve where it can go up and down. Ideally, the head attempting to read the signal needs to be precisely positioned over the peeks and troughs of the gaussian curve. Due to tolerances between different drives, or even hysteresis in the same drive, this can be hard to do and the head can end up on the shoulder of the gaussian curve. To help with this the signal is first amplified by an auto-gain amplifier.
A flux transition is used to represent a 1 and the absence of a transition is used to represent a 0. But now there is a problem if you need to store many zeros between the flux transitions. For example, when moving from one transition going up and the other going down the distance between the peeks dictates how many zeros are stored between the two ones. But the more zeros inserted, the more the slope between the gaussian peeks becomes shallower and shallower. The more shallow it becomes the higher the auto gain amplifier ramps up. If there are too many zeros then the slope can become so shallow it almost becomes a constant straight line. Induction stops in the presence of a constant field so this is bad. Mathematically, differentiating a constant is zero. The derivative of a shallow sloping, almost constant signal can be close to zero. So with the first amp with its auto gain maxed and the second differentiating close to zero, any noise can easily cause a random incorrect zero crossing to be sent through to the one shot and be incorrectly detected as a one.
Some floppy disk copy protection schemes deliberately did this on purpose. When reading a section of the disk with a slowly changing magnetic field or a section of the disk that was manufactured with no magnetic material multiple times, the code expects the data to be random. If the data is always the same each time this section of the disk is read then the code knows it is a copy.
I remember writing assembly code on my Apple 2+ to read and write raw sectors to crack various copy protection schemes. The hardware controller was pretty dumb and most of the drive characteristics (bit timing, track spacing, etc.) were controlled by software. Great presentation of the low level encodings. A few of this (Manchester and group codes) got heavily used in various network protocols like Ethernet and FDDI, too.
Hi Dave, Dave here. I'm glad to see again that cute old sign-off segment with the tiny furniture and welcoming narration.
I've been lucky enough to live through all of the technologies you've talked about being used. I *think* I've seen a drum drive, has the chance to make the heads of an IBM 1620 disc drive move using assembly language. I still have a CPM machine with a Tarbell Tape Drive board in it. Now I took an the proud owner of boxes of my old hard drives from XTs to the current computers. Most haven't been spun since their host computers were recycled. And there is the rub, what am I going to use on my 40 plus year old Seagate? I'm sure the technology is out there, but I'm not that motivated to relearn the old ways of forgotten hardware and software. Sigh.
I don't know about being "lucky" enough to live through all that, but I did enjoy the days of wrangling a bunch of CompuPro boxes on Arcnet with some 40MB drives that were the size of a modern desktop. I helped transition the UCF undergrad lab from Arcnet to Ethernet. Right now my 3D printer is creating a thing to help me clean my motorcycle chains. An hour ago that design was just a page of OpenSCAD code. That is absolute magic.
Mesmerising, I don't understand any of this but it's still mesmerising. Thanks Dave.
6:43 fun fact: the USB Power Delivery protocol also uses this strategy of encoding 4 bits into 5. However is also uses some of the left over 5 bit codes for special uses.
I remember having two 20MB seagate drives on my BBS and I bought a brand new on the market RLL controller.
I copied all the BBS software and scripts to floppies as well as the second hard disk, reformatted the first Hard drive to RLL and copied it all back - Viola ! 30MB.
Re-Loaded (copied) all the software back to that drive after re-installing DOS and did the same thing to the second drive and now I had two 30MB drives.
It so seemed like magic at the time.
Some time after that, I got an ESDI drive, then a SCSI drive and then an IDE drive. It was an incredible journey.
I'm literally formatting my ST-251 as I write this!
@@chrismadog8004 Computers were certainly more fun in those days.
@@DavesGarage - LOL. Memories 1 - I got 2 bigger capacity drives in the days where they were expensive (in Oz) and ended up only paying for a controller which was less than half the price of one drive.
The Diagnostic BBS needed space urgently and that did it nicely. Good memories.
I worked for Hard Drives International back in the MFM/RLL/ESDI/SCSI days (and early IDE/xt-ide). Yes, you could take a Seagate st225 MFM drive, put it on an RLL controller and if you were lucky you'd get the same capacity of a ST-238r. Basically, a 50% increase in storage. If you weren't lucky, it would work but you would end up with data corruption.
About the MFM vs RLL encoding scheme.. I do remember finally getting enough money together to get a 20 MB hard drive for my IBM PC clone. It was MFM encoded. A year later or so later a friend of mine said he got a 30 MB hard drive for his IBM clone. He said the drive mechanism was identical to mine except it used RLL encoding. So I do remember those discussions about RLL being able to store 50% more then MFM. I wouldn’t have had the confidence to try swapping just the controller on my drive to see if it worked. Especially typically I built my PC’s at the time by just buying all new technology anyway. My next hard drive was in a whole new computer, everything was much faster, stored more, faster ram ,etc.
Yes. I do remember taking an MFM drive, putting it onto an RLL controller and having it work. Granted, the MFM drive required low-level formatting to work on the RLL controller.
Great video Dave and your videos are always educational.
I remember that when the first RLL drives came out (30MB), they were essentially MFM (20MB) drives hooked into an RLL controller. However, that resulted in those drives being much more finicky and data loss reliability on those first 30MB drives was a problem. It always seemed to me that RLL drives throughout the 1980s were less reliable. The term "RLL" still gives me chills.
PRML (Partial Response/Maximal Likelihood) was a key advancement in the read-channel that significantly boosted bit density. Disk drives can get rather nasty at the inner cylinders where the bit-density is higher and bits interfere with eachother.
Funny fact: When I graduated in 1985, at one of my interviews the sys admin was very proud to show-off their brand new 1GB disk drive. It was the size (and noise-level) of a residential dishwasher. Today, my dashcam has a 256GB microSD that's smaller than my fingernail.....what will we see 40 years from now ????
As well as regular improvements in data recording devices, there was a corresponding improvement in the recording media. Greater data densities required denser magnetic particles which needed stronger magnetic forces to record on. A similar situation occurred with audio cassettes. Just as a recorder designed to work with basic ferric oxide tapes were unable to successfully record on higher quality Chrome tapes, 40 track single density drives couldn’t write to media designed for use on 80 track double density drives even though they could usually read the higher quality media if the data on it had been written to it at the lower specification by a drive capable of writing at the higher densities.
When you mentioned frequent modulation I thought you were going to go into how some early consumer computers stored data on audio cassettes. The second computer I built (from a kit in Australia) used this method. I replaced the original 8080 CPU with a Z80, doubled the clock speed, and then used a few spare gates on the board to software select the original clock speed when reading or writing tape as the system ROM routines relied on the system clock for generating and reading the audio signals.
Yes, you could use a MFM drive with a RLL controller with the caveat that the drive might not be of high enough quality to support it well. It's a matter of manufacturing tolerances and qualities of the magnetic media. I'm sure some drive manufacturers sold their RLL drives that didn't pass quality checks as MFM drives. Quite often the the difference between an MFM and RLL drive was just the label having a different suffix letter on the model number.
Yes, you could take an RLL controller onto a MFM drive and get more space. My buddy and I did that as well, AND.... if you you really got crazy, you could change the clock speed on the RLL controller, and you could cram even more on the same disk. There was a limit though because too fast and as you'd expect data errors really started to creep in.
What a great time to be playing and learning 'cutting edge' computers and their hardware. I do miss setting interrupts with jumper pins though!
I definitely remember getting an RLL controller to replace my MFM controller and low-level reformatting my hard disk to get something like 50% more space. This was in the early 1990s, when everyone was switching to IDE, so old MFM and RLL controllers and drives were easy to find second-hand, and cheap too. Especially Western Digital controllers. And here's why:
There had been an open source hardware project in a German magazine that allowed you to connect an ISA MFM or RLL controller and hard disks to an Amiga, and they said it didn't work with Western Digital controllers and didn't know why. So it was almost impossible to find other controllers because everyone was buying those, but Western Digital controllers were a dime a dozen.
I actually built one of those projects with a Western Digital controller, and reverse-engineered the software (with a little help of the hard disk BIOS listings in the IBM PC/XT Technical Reference manual). Turns out the only reason why it didn't work with Western Digital controllers was that the software didn't hold the Reset line long enough. That was easy to fix, and I used a 70MB hard disk on my Amiga as a 105MB hard disk for years. And that was a lot of space back then!
On the subject of using the same drives with both MFM and RLL controllers, I had the opposite experience. I had a 30mb RLL drive/controller combination that was proving very un-reliable. I was constantly having issues and re-formatting the drive. Eventually I conceded defeat and connected the drive to an MFM controller, re-formatted and ended up with a very reliable 20mb drive. Lost 50% capacity but at the time it was worth it for rock solid reliability. I guess the head alignment was slightly off and struggled with RLL but was perfectly happy with MFM. Good times!
I remember formatting a 20MB MFM hdd with RLL , and getting 35MB from it. There were more chances of reports of bad sectors, but even then the 5.25” hard disk I was using was somewhat antiquated!
Great video Dave ! MFM is the one I heard of .
On PE, part of why it was popular was you could do faster encoding without worrying about syncing up to your clock signal which I believe often meant you could do faster encoding those putting more data per inch of tape.
PE is used for the data tracks on credit cards. You can record it as simply as pulsing in two polarities. Reading it back it shows up as pulses either up or down on a scope and is EZ to decode using a comparater + AND gate.
I really like these computer technology history explanations!
I closely followed the research on the GMR effect in Switzerland. The GMR effect was discovered by Albert Fert and Peter Grünberg, for which they were awarded the Nobel Prize later in 2007. It had a significant impact on hard drive technology, as the read heads utilized this effect. The first hard drives featuring this technology were released by IBM with the Deskstar 16GP in 1997 and were not even that expensive. I bought two 14GP models from IBM at the time.
Yes, can remember swapping out MFM controllers and installing RLL controllers. Worked a treat. You could improve a 10MB drive to about 15MB. A big deal in the day…
“Closet full of spinning rust”…. Made me lol.
Back in 80's I worked for Micro Technology (tunbridge wells) and then Plus 5 Engineering (crowborough), who USP was producing external hard disk and tape units for PC''s that matched the PC cabinet design, so different containers for IBM PC, Compaq, Sirrus, etc etc. I wrote all the disk and tape device drivers for CP/M, IBM Dos, MSDOS etc, and eventually Xenix. These units were really expensive but gave a PC 10MB of storage, yes a whole 10MB. My watch today has 32GB of user storage, what progression. I even wrote a shared disk system between several PC's connected by SCSI flat cables and abritrating access using the SCSI id's.
Happy days.
I remember some of that. It's been a while! Thanks for the memories!
Yep it was my first job working for a importer distributor in new zealand selling hard drives, controllers to pc dealers over the whole country. we had Miniscribe 8425 5 1/4 " 20mb drives that used MFM ISA controller where as a 8435 5 /14" 30mb drive used a 27RLL ISA controler which was the same as the 20mb drive, but was rated as being more reliable to handle the extra 50% increase in data density onto the platters. we also had and also 3425's 3.5" 20mb drives later on. And yes there was the case where miniscribe shipped bricks on ships... they were failing at producing enough products that were capable of keeping up with performace improved western digital drives....
Fondly remember RLLing my Seagate MFM drives to get more storage. I recall there was also an extended RLL eRLL which stretched that even further.
GO:C800;1 do you remember the embedded HDD formatting on the ISA controller
Yup, executed that command many times in the debugger to get the controller chip to low level format. Good times!
analog computing is fascinating, it naturally forces more creative solutions to practical problems, when you can't even assume that data is already encoded into logical binary frames
compression algorithm that scared me 💀
it gives extra complexity
Flashback to 1990 when I decided to implement low level floppy I/O in my Amiga games with help from the blitter, same way as the system. The blitter made the encoding and decoding much faster and simpler. In retrospect, it was pure chutzpah to use my own floppy I/O.
You're reminding me of when I did a report in college on how the CD recorded data. Dave, this David says you will find that very interesting. I still remember parts of it all these years later. Like the 8 to 14-bit transformation where there may be no more than 10 zeros in a row and no fewer than two zeros in a row or was it three? And like some of your encoding techniques, it was the change that was the one, not the surface. They used to change through time it, and they use the laser reader to align it. I also had a lot of checksums in the blocks of data. Do one of these videos on how a CD is encoded. For real, do one because they're interesting.
Ooohhhh, Thank you for answering a many decade old question from the back of my mind. I was in grade school at the time but I definitely remember a day where my dad brought the family 286 back from the shop and our previously fairly full 40Mb HDD was now reading as 60Mb and seemed much roomier and even a bit faster. My dad had told me that it wasn't a replacement drive but a replacement drive controller (I thought it was MFM to RLL but that might just be confirmation bias after seeing the video.) I always wondered how a controller change could change the capacity of a drive by such a large amount and now I know!
My first hard drive was an MFM 20MB ST-225, and the controller that I got with it was a single port controller. When I got my second drive, it was a 30MB ST-238R, and came with a dual port controller card. At some point, I ended up reformatting the ST-225 to RLL on the dual port card, giving me 60MB of storage in one XT machine. Those drives were with me until I went to upgrade Windows 3.0 to 3.1, and 3.1 no longer supported the 8 bit controller. At that point, I picked up a 50MB Quantum Fireball IDE, and the old drives got used to make a gaming computer for the wife and kids ( think 286 CPU and monochrome graphics, great for hangman and board games. )
Still have my ST-225. And the 5150 I used it it.
I don’t remember how i did it, but back in the day on the Amiga i think i used DMA to read the MFM data in the middle of the disk. Decoded the MFM with the blitter (also DMA) and analyzed the BAM to proceed to read the meaningfull data of the disk. This could read and decoded the data without using cpu power. The goal was to use the cpu to send the data over the parallel cable so we could copy disks from Amiga to Amiga at incredible speed at copy parties 😇 I tested, but never finished the “networking” part, a bit of a shame because the idea was great.
Amiga had a funky timing thing too. Instead of x sectors per track with gaps for timing signals, the sectors abutted one-another with a 'special' sync mark that couldn't ordinarily be written with MFM -- 3389 -- that's why AmigaDOS disks were able to pack 880K instead of the conventional 720K.
Yay! for Gibson and SpinRite! Also for Core hard drives, especially those 15Mhz ESDI versions.
Very interesting. Some of the signal processing methods were invented by my FIL in a backyard hammock in Raleigh NC (for IBM). I still find it interesting to compare some of the patents of his methods to the implementations used for these devices.
A number of serial line codes were developed to eliminate any sort of DC bias present in the signalling method as this caused problems for transmitters and in some cases, radio equipment due to the bandwidth required. 8b/10b maps 8 bits to a 10 bit string. 64b/66b achieves the same on a longer code space. Unrelated, it's possible to configure a modern UART to run 9600, N, 8, 2 (two stop bits), which means the whole string is 10 bits long and will show up on the oscilloscope as a nicely divided number.
For instance, AX.25 or X.25 uses a polynomial scrambler in an attempt to eliminate DC bias as well. The feedback taps on the shift register were well known to produce that polynomial and the FCC allowed only three sets tap configurations.
In the 80z my 48k 8 bit sinclair spectrum had a tape drive but was using sound to store the data on a tape recorder. It sounded like a modem when it was loading or saving a program and was fun mostly.
I enjoyed it Dave. It’s relevant to me as I’m altering the way my homebrew computer stores and reads data from tape. Yes, SD cards exist, and no, I’m not going that way just because. I may add floppy though.
My first PC (386SX) in 1991 had a full height 5.25" 40MB hard drive. I remember looking through the manual for the hard drive controller and there was a mention of the default encoding being MFM but after changing a jumper it could be RLL. I didn't really know what this was... but I changed it, did a low level format and my 40MB drive turned into a 63MB drive! It was magical 🙂
Great video Dave! I remember taking my 20MB HDD and attaching it to an RLL controller and it formatted out to 30MB. I wasn't sure it would work, but to my surprise it worked great! Was cool to brag about it back in the day. 🤠
@Daves.Garage01 I'm guessing this is not you Dave.
Adrian's Digital Basement channel had an episode of recovering data on some old MFM and RLL drives. He also went through explaining the encodings used. It was very interesting.
My first HDD was an MFM 40MB drive. I think I paid $500 for it at the time. Had to low-level format it before you could then partition it then high-level formatting the drive so the OS can store any files on it. Ah, those were the days.
And when the IDE drives came out and were told NOT to low-level format them... I was very confused.
13:45 - Absolutely have memories of this and all through the video I was hoping for an explanation of how and why this worked. That "trick" combined with using a magical software called "stacker" was amazing in the early 90's on XT clones!