As a radiotherapy physicist, Fritz Hager is something of a personal hero. He didn’t just spot the error - he spotted it by sitting through a weekend with a therapist until he got to the bottom of it. He refused AECs explanations and refused to back down. He’s the paragon of a medical physicist!
the counter overflow problem is so basic the programmer must have had no experience at all. but sometimes people just do the very least they can get by with including consciously pushing a problem into the future instead of addressing it, which that' could also be an example of
Android now disables all apps, even emergency alert ones, if they're not used often enough. You cannot know they are disabled, the tiny notification - a list of disabled apps - doesn't always work. You just suddenly stop getting all emergency alerts...unless you manually check each apps's settings, every day. FEMA, Red Cross, severe weather alerts, wildfire Watch Duty - even the built-in-phone alerts from the government. Not strictly a bug, but, unintended. And has probably killed already. If not, it will soon.
@@randomdude1053 B737MAX crashes were not caused by a software bug. MCAS functioned exactly as it was designed to do, given the inputs it received. The problem was the design was poor (there was practically no limitations on how much and when MCAS would activate) and the input was incorrect (MCAS was fed data only from a single sensor, instead of 2 that exist on the plane, and these sensors are known to have issues). Additionally, Boeing's insistence about not disclosing the existence of the system meant that pilots had only a vague idea of what was happening and what to do whenever unintended MCAS activations occurred.
Having your brain fried by a radiation machine and slowly realizing your own death is imminent... is definitely the stuff of nightmares =[ RIP to those who passed
@@civotamuaz5781 If it makes everyone feel better, the sizzling sound was most likely caused by the saturation of the ionisation chamber that measures radiation below his head in the bottom part of the machine. There was no "real" heat applied. It was just a reaction to the radiation to the nerves/receptors in the skin.
Redundancy in ANYTHING that has the possibility of taking a life if it fails, is an important necessity that gets overlooked a lot of the time due to cost or human stupidity. A device should only really be considered safe for human operation if a drunk, sleep deprived, brain damaged capuchin monkey could operate it safely.
the worst part here, is that the guy writing the software didn't write it to be used without a mechanical interlock. Yeah, he wasn't even a professional programmer. which is why the UI for the software was so bad. Dangerously bad tBh... numbered errors the techs can't interpret? SAFETY HAZARD!!!! but as originally designed, the software worked safely.... then the company redesigned the hardware in a way that made it unsafe.
@@marhawkman303 yeah, it's on the company, rather than the programmer. Arguably you should have someone else doing the UI. It's a programmers job to make the program work, it's a designers job to make the UI user friendly. But yeah, mechanical fail-safes are very important, both as emergency stops/safety overrides AND for mechanical backup operation.
@@kiritotheabridgedgod4178 Yeah, I think the company was trying too hard to push the idea of the treatment being "safe" that... they were creating the ILLUSION of safety. The previous models didn't have this safety issue, not because it was an amazing design, but because it had a mechanical interlock. then someone came up with the idea of touting a lack of mechanical interlock as a "SAFETY" feature?!?!!?!!? the UI being terrible certainly hurt. the people using it had no idea what the errors meant. and... the software had no idiot proofing. But I get the impression that the company wanted to HIDE the idea the machine could even in theory hurt someone. but took this so far they didn't write into the technical documentation that there were dangers. Thus the techs using it didn't KNOW that some of the errors were actually fatal... to the patient. and that's the REAL issue. the company making the machine was so focused on touting it as safe that they had compromised safety by simply not discussing dangers.
It's like ignoring your check engine light. And just like when you're dealing with the a mechanical item, there can be irreparable damage caused to human tissue resulting in death. In the case of your car you could blow your engine up.
@@Smedley1947 Most if not all check engine warning lights are emissions related and will not cause the engine to blow up, hence why the light is yellow and not red. A better analogy would be to ignore the oil pressure warning light.
The worst part is that the professionals and doctors around them didn't take them seriously so they were suffering alone until it was undeniable that something horrible has gone on.
The machines were in use constantly and treated hundreds of patients. One person says I felt like I got burned and there’s no mark what should the operator think? Unfortunately, with a lot of infrequent issues of this nature. People just think it’s a fluke until there’s enough incidence to show that it isn’t.
My radiation doctor is Strange. When he asks if I'm in pain and I say yes, he immediately says it's not from the tumor. Yet that is where the pain is, plus where they screwed up putting chest tubes in me. But some doctors just really don't listen to their patients.
Yikes the case of the one guy who heard a sizzling sound and died of radiation burns on his brain is so scary makes you wonder how many unreported accidents and deaths happened because of the machine
Medical incompetence and mistakes are more common than the general public would like to believe. Try to sue anybody who works for the NHS in the UK and you have absolutely no chance of any form of compensation.
@@memyself1566 nowadays if you show up for a headache to a us big hospital, you will be given and charged for a CT scan in case it is a rare disease and the doctors don't want to get sued, again
Well not really how they think about it, eventually someone would’ve figured it out. The big fish don’t really care about the rest of us, if that were the case they would’ve released a finished product.
This was decades ago. The concept of a finders fee for software didn’t exist. His concern was is this very expensive. Very useful machine. Going to be a problem at my hospital.
@@coreyjohnson2205 Safety systems are not supposed to be trial and error. They’re supposed to be carefully engineered and carefully tested. But kudos to his well constructed reality test. An example to others doing software testing
I'll never forget the patient who recieved radiation burns on his BRAIN. And the description of him hearing the sizzling of his own brain burning..... rest in peace man. Its so tragic that he suffered until he died, as well as the other patients who recieved burns from the machine.
Copy-pasting another comment from above: Interestingly the "Sizzling sound" turned out to be the tungsten target being destroyed due to the beam firing at an intensity many times greater than it was intended to receive still bad for the patient but it did help them eventually figure out what was going on.
@@z9297you'd think that they could have test fired this POS on some hog carcasses first..I'd rather see it fired on the chain of people that signed off on that knowing damn well it just burned someone to death. Start with the tech that said it's fixed... Ok put your leg in and I'll listen for the sizzle. Oh would you prefer the hog carcass instead?😮
For those who care what the actual bug was: Once you entered all data and reached the end of the input process, a "data entry complete" routine was triggered, which would call a sub-routine multiple times to get everything on the machine arranged. While that happened, the software still allowed changes to be made to the data, so you could manipulate data while this data was read at the same time to setup the machine. Of course that can lead to an inconsistent setup and the programmer was aware of that. So while the setup was running in the background, a flag has been set and if you made any input change while that flag has been set, the whole setup process started all over again to be sure the setup used a consistent set of data. The problem was, that the first time the sub-routine returned, the flag was cleared at the end (about 8 seconds after data entry finished got triggered, that's the actual bug here!) and any change you made thereafter went unnoticed by the software, yet the changed values were still used in the repeated sub-routine calls yet to follow (the setup routine was called up to 8 times in a row). So when you switched between electron and x-ray mode 8 seconds after you first triggered "data entry complete", the beam setup and the turntable position could go out of sync. In x-ray mode the beam intensity must be high (as the beam never hits you directly, it hits metal that will then produce x-rays that hit you), in electron mode it must be low (as the beam hits you directly), so if the beam was in x-ray mode but the turntable was set for electron treatment, the beam that hits you would be way too intense as there was no metal in-between as should have been. And Error 54 meant that the dosage you just shot at the patient was way too high ("Overdose Warning") and thus the device stopped the treatment; overriding that error was thus fatal as you just continued to shoot way too high dosages at the patient. As software developer I see three fundamental flaws here. The first flaw is that critical errors must not be overridable . Error 54 should have stopped the treatment completely with no way to continue at all and the software should have informed the operator that this was due to an overdose in clear words ("WARNING!!! OVERDOSE!!! (Error 54)"). The second flaw is that even if the flag bug didn't exist, the thing was broken by design. Either the user interface should not even have allowed any changes while the setup was in progress ("Setting up. Please wait ..." and no interaction is possible other than "Hit C to Cancel") and if you cancel, the whole process starts all over. Or it would allow changes, however data input and setup would not use the same data set. Instead "Data Entry Finished" would once copy all entered data and the setup would only work with that copy, so when the user changes the data, that has no direct effect whatsoever. Only after all setup was done, the code would check once more if the copy it used for setup is still in sync with the data entered and if not, a new copy is made and the setup process repeats, followed by another check and so on. That way no such flag is even required. Both perfectly prevent such issues, the last one allows changes at any time but requires more memory (a data set copy must be made), the first one requires no extra memory but you cannot make changes while setup is in progress without canceling setup first. And the third flaw is that there was no final sanity check. The software should have been able to query the currently set beam intensity and the current position of the turntable via sensors and before starting the treatment, it should have re-verified that those two are for sure in sync and that the beam is never set to strong unless the metal shield is in place. If those two are out of sync, which should have never happened in the first place, the machine should lock itself up ("FATAL ERROR!!! CALL TECHNICIAN!!! (Error ...)") and not allow any treatment at all until unlocked by a technician of the vendor who must find out how this could even happen.
As a fellow software engineer, this was the best explanation of the error that I've ever seen. Nice going! One thing I thought of is, if they had a semaphore on the write function that disallowed writing while the data was being read (AKA when the system is setting up), it would have prevented it as well. But no clue if those computers of the time had things like that...
@@n00byscrazycorner43 You can write a semaphore without any extra hardware support, as long at the memory model of the CPU is strongly consistent. Only if the memory model is not, you require a special CPU instruction that guarantees strongly consistent ordering. However, even the 8086 and 8088 (16 bit CPUs, launched 1978) already had a LOCK instruction for exactly that purpose as the memory model of x86 is not strongly consistent. Yet if he had obtained the semaphore when the setup process started and released it only once it's done, this would have lead the "Please wait..." solution I suggested. Also my other solution, the one with the copy, would normally require a semaphore as well, since you can only get a consistent copy if you ensure that no data is altered while you are copying the data (copying the data is not atomic, after all). Yet both these solutions don't require a semaphore as the software was in control when keyboard input is processed and when not, so it could always disable keyboard input, then perform whatever operations it want, before re-enabling it again. What it did instead was never disabling it but every time a keyboard event was triggered, checking if that special flag was set to know if it must re-start the setup process at the end (if there was a key event for data input and the flag was set, the state of the state machine would have been reset to an earlier value, triggering a re-setup in the end). This would have worked correctly if the flag had only been cleared at the very end of the setup procedure and not every time the sub-routine has been called (the flag was cleared at the very end of the sub-routine, instead it should have been cleared after the loop finished that called that sub-routine multiple times). Still, only relying on such a simple flag looks way too fragile IMHO. I even dare to take a guess how this bug came into being in the first place: Initially the setup process was probably simpler and the sub-routine was called only once and that's why the flag has been cleared there. Then things got more complicated and the programmer thought: No big deal, I just call that sub-routine in a loop and then it can make adjustments one by one, overseeing that the flag was still cleared on first call. During testing he never altered any data later than 8 seconds after the process started, as why would he consider waiting that long? He just tested starting the process, making a data change immediately and as expected, the flag was considered and everything worked smoothly. That's why every unit test framework today allows you to randomize timings and this should always been used when simulating asynchronous events as when those events happen can in deed make a huge difference.
And their is present day naiveté and normalcy bias. "Of course the government isn't corrupt, yes they experimented on US citizens and gave black people syphilis, but that was in the 70s
What's scary about the Therac-25 was the amount of radiation patients had been exposed to, the radiation exposure was on par with Cecil Kelly after the criticality accident, it's frightening that these poor people were exposed to that much radiation, and it was sickening that AECL was too arrogant to believe their machine was a killer.
Anything that shoots radiation at a person is a killer, period. This entire industry is criminal!! I’ve watched a friend from school who was only 30, then my mom & my best friends mom along with another childhood close friends mom, all back to back, found out they had cancer but all looked fine upon diagnosis….post “treatment” all 4 of them slowly withered away until they died…all 4 died *EXCRUCIATING DEATHS!!* All of them had chemo & radiation. I don’t know how a single oncologist wakes up in the morning & is eager to start their day….how many patients have to die slow agonizing miserable & EXPENSIVE deaths before they realize their entire industry is bullshit??? 🤷🏼♀️ On top of that, my grandmother was told she was dying from breast cancer & only had months to live, desperate to survive she went through a double mastectomy & radiation that burned her so badly she had to have her arm amputated at the shoulder- yet she was misdiagnosed & never even had terminal cancer, she sued the shit out of whoever did that to her & lived another 10yrs but I’d imagine she would have been happier without money if it meant keeping her body parts & avoiding the trauma of believing she was facing death….
@@EphemeralProductions a bit like “guns don’t kill people, people kill people” so if we got rid of the defective people and just kept the guns, all would be fine. In reality we need to get rid of the guns AND the people both. This radiation machine needed to be looked at the first time not as is normal after a long time and many more deaths or injury. People are so full of themselves they just cannot see that there could be an error.
That error counter reminds me of a part of the Jurassic Park novel. The company has a computer system that scans the number of dinosaurs in the park which is set at a specific number (say 250). The problem is, nobody ever sets the number higher to see if there are more dinosaurs.
@Patriarcus Rex one unknown programmer. Who could have been just a hobbiest. I can't imagine being that guy really. Imagine hearing about all these software errors that KILLED people.
Copy-pasting another comment from above: Interestingly the "Sizzling sound" turned out to be the tungsten target being destroyed due to the beam firing at an intensity many times greater than it was intended to receive still bad for the patient but it did help them eventually figure out what was going on.
@@z9297 Stop trying to sugarcoat it because we all know it was that poor man's brain frying in his skull from the massive dose of radiation that was fired into his head that caused the sizzling sound and it ended up killing him!
The fact that this device is cited as a cautionary tale in both mechanical engineering and programming disciplines as a "What not to do" is why I first learned about this. Hearing about the horrors the patients suffered though...
I work in IT and honestly the over-dependence on software is a PROBLEM, to the point where people sometimes fail to use their heads for problem solving. Like…software is built by humans. It’s subject to human error.
Both my parents were excellent at driving, and had a great sense of direction. If my father had previously been somewhere he could remember the road layout really well, for example. Once he got a Sat Nav, he couldn't remember the 20 minute route to my home without it!
I looked for a comment like this so I wouldn't have to write it. You would think there would be no trivial errors when operating a machine dispensing radiation treatments.
@Alex your comment reminded me of the MCAS aviation software that malfunctioned, causing a fatal crash and many lives lost. I love watching Air Disasters and I think too many pilots rely on software rather than their pilot training. It seems that the more that is done for us, the duller we become.
@@Nalla762 excellent point to remember that MCAS issue @ Boeing! You're exactly right regarding pilots flying now. Basically they are present in the aircraft to take off and land...that's pretty much it. Scary, huh?
Interestingly the "Sizzling sound" turned out to be the tungsten target being destroyed due to the beam firing at an intensity many times greater than it was intended to receive still bad for the patient but it did help them eventually figure out what was going on.
I mean,even with the number of accidents,they were still a rarity. But definitely way to high. Aside of that, as this channel shows, something bad having happened most of the time means itwill happen way less or not anymore. So the fact something didnt work right in the past, and we're still using it, means that its safer than ever. Finally, I hope youve overcome your cancer. Best of wishes and luck for a good recovery mate
As a radiation physicist that works on cancer treatment devices, this incident is such a big case study while in grad school. My coworkers and I even discussed it about a month ago because it is such a notable incident in our field. From my experience in a cancer clinic for 3 years, I can assure you that when it comes to treating patients, it is done as quickly as possible. If there is a shortcut discovered, it will almost always be used. This becomes a frustrating point as most of the time if the operator slowed down, did not rush, and pushed the buttons in the correct order, then faults, collision, and accidents would not occur as frequently.
Thanks for sharing that info. If I ever get the dreaded cancer diagnosis, I'll probably pass on the radiation treatment if such is proffered as a method of treatment to me. In this day & age, it actually seems archaic anyway. I could be wrong, but I have never heard of a patient's life being saved from radiation treatment. Please correct me if I am wrong, though.
@@kimmuckenfuss2284 Radiotherapy is one of the safest fields in medicine, things have come a long way since the 1980s. It is used to treat over half of cancers and depending on the type of tumour you have, can often cure you. If you're working somewhere where therapists are rushing and taking shortcuts you should be reporting it to the relevant safety authority. I'm a therapist working in radiation safety, I promise it's not the wild west out there!
@@kimmuckenfuss2284 Not to frighten you. It is an incredibly safe treatment method when done correctly. There is about 8-12 weeks of testing every single safety aspect of the system before it is released to a customer for use. With that, there is still a human element to all radiation treatment. Patients not aligned correctly, incorrect accessory use, accidental collisions, and other user induced faults occasionally occur. These can be more prevalent in a rushed environment.
I am a medial physicist. The kind that works with radiation therapy machines. I just want to say you did a great job covering this topic! You explained it excellently!
I’m an RT medical physicist as well. Excellent video! These incidents were one of the first things we covered in school showing just how important checking and rechecking everything is in this field. I think about these and other incidents almost daily.
I like how you couldn't resist bragging about yourself and your qualifications, and tried as hard as you could to disguise it as a compliment to the channel 😂😂
Thanks for your comment. I'm sure you worked hard for those degrees and unlike some others actually have the right to comment about the content. Sadly this happened and I'm glad to hear they're teaching about it because we know something similar will happen again with another machine of some sort.
@@Adelicows bro? They were listing their qualifications as someone who can speak on the subject, confirming that it was correct That’s helpful to the channels reputation on top of being a compliment
People think consultants in any industry don't do anything, but this is one case where hiring someone from outside to review the code could have saved lives. It's the same as any kind of writing--copy, correspondence, etc. Putting another set of eyes on it often reveals an error the author missed during their own composition and editing process.
In my IT career I cannot remember the number of times mission-critical software was released full of bugs. For example, payment processing systems that could not accept payments, simply ridiculous the number of errors that get by. It is pretty much 1 hour of testing required per 1 hour of programming. No, the developer CANNOT test their own code, they already thought of everything they could and guaranteed those things wouldn't happen. Still, there are a million things they did not think of, did not implement code to prevent, could not conceive of occurring that will happen on the first day of release to the public. Professional software testing is critical and expensive. You want good, fast, and cheap? Pick any two. That's why I take all these assurances that A.I. cannot and will not be a problem...with a grain of salt. Or a bag full.
My wife received lung cancer treatment using a Novalis photon beam machine. One day they called and told us the machine had not passed its self test. A beam forming shutter actuator was not operating within its specified time limit. A service technician replaced the failed part. She received her treatment later that evening. The level of engineering related to safety has made huge progress.
@@n00bfest32Bro just shared a story of an experience with updated safety checks on his wife’s cancer treatment and this was your chosen response? If you couldn’t infer that the tech clearly explained the issue to him via the phone call or later appointment, I have serious questions about your literacy skills.
I can't imagine being already vulnerable and fighting for your health and life then being severely injured and/or killed by the means in which should be treating you. How awful.
Hey, thanks for pronouncing Yakima correctly! Most people dont even attempt to pronounce it correctly when Ive watched other videos about the Therac (Plainly Difficult, Im looking at you). -someone who grew up there I remember my dad talking about this when I was a kid in the 80s. He knew people who worked in the Oncology dept at the hospital when both the Yakima accidents happened. Apparently, when the first lady was burned (she survived her injuries), radiotherapists were suspicious the machine somehow overradiated her, even though the company AECL basically blew off the concerns and said there was "no way" it could have been the machine. If they had taken the reports seriously, the man that died in Jan 1987 (also in Yakima) wouldnt have been killed. But they kept being assured at first there was no issue, then "oh we fixed it". AECL was more concerned with covering their own rears than making sure people werent hurt.
As someone who is currently battling cancer and having gone through radiation here in Georgia, this is terrifying. I know technology isn't perfect, but when it's life or death, it really takes things to a whole new level. I'm so glad these errors were found and fixed. RIP to those affected by this ❤
This incident was the literal textbook example during my software testing course on importance and the potential problems of testing (because bug-less programming is a pipedream).
It's also used to teach human factors/ergonomics engineers (i.e. the people who design the user interfaces on kit like this to make sure it's simple to use and that it isn't possible to accidentally kill anyone) - whilst it's horrific, hopefully the fact that it's so widely taught will go some way to making sure this never happens again
I can't speak for this particular radiation treatment, but it's certainly true radiotherapy is even today a double edged sword. My uncle had an inoperable brain tumour to which he was offered a very new intense therapy in the hope it would extend his life. All I can say is for all he was terminal, the "new improved" radiotherapy shortened his life drastically which makes me believe these treatments are far from "tried and tested". Tested, maybe...on my uncle.
@@jessikatkins1173so true! I had radiation in 2012 to treat breast cancer and I had a bad burn that got infected. Even tidy years later I still have trouble with the lung the radiation was beamed on.
Maybe I’ve seen too many horror films, but radiation therapy machines scare the crap out of me. This is exactly the kind of nightmare scenario my brain always plays. How horrifying for the patients who suffered from this malfunction/bad programming!
If it makes you feel any better, this is very much a case of multiple things going wrong all at once. I have reason to suspect a modern radiotherapy machine would NOT have the problems the Therac-25 did, given it's now a literal textbook example of what can go wrong.
@@shadowldrago I actually started looking into this after watching this documentary, and you're right - it looks like it's used in computer programming degrees and also medical ethics in computer programming. Good!
@@JustMeUpNorth Agreed. It's terrible that this went so wrong in the first place, but it's good that those involved learned from this scenario and took steps to make sure it would never happen again.
Whelp! Time to add ‘radiation burns to the brain’ to my list of irrational fears. Just the idea of needing any surgery puts me on edge, but hearing a sizzling sound during treatment is a whole other level!
Great job as always! My ex-husband and I both work in the tech industry, specifically around software coding and testing. He told me about an interview he had early in his career (mid-90s) where he was asked if he could do a job that if he made a mistake, it could result in someone's death (it was a job writing software for medical devices). He didn't take that job but I always thought about it and realized I would never want that level of responsibility. I wonder how that software developer felt when he heard his software caused these deaths.
Not so bad. You can go for big bucks, and be sober, cautious. If you think about it, taxi driving, or even giving a ride to anyone in your car is the same thing. Just a mind trip one can overcome.
@@KathrynsWorldWildfireTracking if you crashed and killed your passenger(s), that’s only 1 to 5 max people killed or injured. If technicians aren’t picking up on a software problem that’s killing people in a way that doesn’t obviously point to the software (in theory, it is possible), then that could potentially lead to a lot of people dying. A lot more than would be involved in a typical car crash. And if you’re so bad at driving you’re regularly having accidents, it’s unlikely you’ll keep your license for long.
My husband is currently undergoing radiotherapy - for some cancers, it is the best treatment & can lead to a full recovery. This is what we hope for. However, I knew of the Therac-25 accidents & this was the first thing my mind went to when he began treatment. Sometimes I wish I knew less, but forewarned is forearmed. The likelihood of another accident of this nature is astronomically low these days because of what was learnt here.
I wanted to say that even though I can't watch this vid (I'm dealing with cancer and might need radiation, so brain started screaming at this lol), I really appreciate your steady uploads and fascinating stories of things I'd never heard about. Thank you so much!!!
As unfortunate as these events were, radiation therapy has been made much safer thanks to them. Events like these are no longer something you have to worry about when getting radiation treatment, we learned well from our past and the Therac-25 is frequently used as an example in computer/programming ethics to ensure we don't make the same mistake again. Know that you will be safe and your concerns will be taken seriously. I wish you all the best in your fight, stay strong, you got this.
My understanding of the original bug is that a race condition existed where if you got good at navigating the menus you could move too fast for the software and produce invalid conditions. Also, although the software made note of errors, it didn't really properly treat them all as interlocks. This was definitely the result of treating the software as you would in any other industry, by hiring someone to produce a working program and then fixing it as problems arose. The fact that the machine routinely threw out vague errors is basically the same thing as training operators to ignore safety in favor of getting the job done.
what i dont get is why errors where so easy to disable. an error should be a huge deal and cause for a specialist to come check and manually verify the machine
@@jonathandpg6115 Because just like anything else, some errors are minor in summer important. In this particular case, the documentation didn’t tell you what the errors meant. The screen certainly didn’t. And as he pointed out lots of errors that you have to constantly cancel trains you to ignore errors.
Okay, as a Gen Xer, I remember the days when when computers were dumb, cumbersome, and really limited and you could out-race them. How I hated when you'd get UTTERLY USELESS messages along the lines of "Error 5" but with no further explanation of the problem. Making the error message basically a bunch of useless nothingsauce. A bunch of 1s and 0s would've been just as helpful. Oh, how the early days of personal computing honed my hatred of bad software interface design. Having some sort of explantion with those Therac 25 error messages might have saved a life. Really, how hard would "Error 54 Danger, turntable in wrong position" have been? And even back in the days of 128 and 640k computers, it would've been possible to have a message like that. You just had to be willing to write it instead of assuming that everybody always had the manual to hand (because those never go missing at large institutions), or had memorized a table of error codes.
Woof! This brought back memories of getting radiation treatment for breast cancer. My machine worked correctly and I still had burns and lots of pain. I don’t want to imagine what those other folks went through. Horrible! On a nicer note, love your videos. They are always interesting, thoughtfully written, and your narration is so smooth.
As a bioengineer, as soon as you started explaining the modes on the turntable, I recognized this case. It was a case study used in one of my grad school classes. It only went over the first patient in that presentation, and for some reason, only mentioned the issue with the turntable, never the overconfidence in the code! And it is so chilling to learn just how much worse this series of events actually was. As someone who dabbles with code myself, the second I saw an ambiguous error and heard “programmed by one guy and never debugged by external sources”, my heart dropped. That is so horrifyingly irresponsible for a medical device using RADIATION!?! And don’t give me “It was the 80’s computers were new”. Hell the NES was a home console and had more thorough debugging than that! People had comodore computers and shit! With advanced music and games! A high end medical device company that was literally putting peoples lives on the line should have had way more thotough ethical coding practices than that. And also MECHANICAL BACKUPS!!!?! Why the HELL would you do away with that!?! It literally would have added what… pennies? Maybe a few dollars? To these likely million dollar builds? In exchange for the priceless cost of actual human lives? Absolutely shameful. And that overflow error. If you maximize a bit, do NOT overflow!?! Lock it out at like *half of that* just to be safe! Its literally an “if” check and a “stop” statement. Algorithm would be like: “code to add to error counter”, “if errorcount >= 100”, “STOP”. Do Not wait until the end of setup!?! Sometimes your programmer needs to be taken out back and shot. I’m so thankful for Fritz Hager sitting down with the techs, seeing how they use the machine, and manually debugging that shitty code that was literally causing that “state of the art” death trap to kill people, and ultimately get it taken off market. Sorry for the wall of text. The full context of this just really upset me as someone in the field that makes such things. Patient safety should always be the absolute top priority for anything you would even think about bringing to market. Regulations are written in blood, especially in this industry, and that pisses me off.
@@nojuanatall3281 I can recommend some noticeably better ones if you like? The sort of people who tend to work for an entire year or more on making just one single documentary the absolute best it can possibly be.
Same here with the huge amount of overlap between Fascinating Horror and Well There's Your Problem, who did this a couple of months ago, I think? Though it's short and concise here, and with WTYP it's 2-3 hours of telling the story and half making dumb jokes and doing recurring bits 😄
Plainly difficult has waaay too many factual errors in his videos. Make it a habit to read the comment section for fact checks if you watch his content.
I wondered when you'd get to this series of events, which make up *the* textbook case study of unintended consequences in the discipline of software engineering and continue to be a sobering warning to others down through the decades.
Yes, all software people should study this case, since it’s a textbook case of how not to write safety software. These days you’re supposed to do a much better job to get your equipment certified
I'm studying to work as a rad tech in radiation therapy and we have discussed this case before! It definitely was an horrible tragedy that should have never happened. Here in Italy now there are daily check ups made to the machines to prevent these kinds of accidents from happening, and during the planning of the treatments there are a lot factors considered for the safety of the patient. Thanks again for the informative video!
This is my absolute favourite story of safety failures in software. If anyone is interested in reading more, check out the Cobalt 60 machines used in South America, some awful stories of similar events. The hospitals were in poorer areas and had no support from the manufacturer, no logbooks etc. Please also look into the Toyota unintended acceleration case! Lots of settled cases and deaths.
Yikes! I live in Atlanta where the first accidents took place and never heard about it. I had radiation for breast cancer 15 years ago, so glad I didn't know about it.
Used to be an X-ray/radioactive materials inspector for the state of PA…you wouldn’t believe how many accidents happen during these treatments but they make so much money doing them 🙃
That's terrifying! I work in radiation safety in Europe and I can tell you serious incidents are few and far between here. Safety culture in Radiotherapy is huge.
This is incredible! Operators just clean the error message and carry on. Wow! These poor people! They were already suffering with cancer and then this happens!! The machine had to many flaws worsened by bad operation decisions. RIP to all.
I have a friend who does software testing for an enterprise solution type company. Almost daily, he's telling me about incidents where one of the engineers insisted that his code was perfect and didn't need testing only to find out that the code was not perfect, did need testing, and in the cases where the engineer was able to push the untested code through, then caused serious problems. The level of logical disconnect where someone can be so smart and yet not see this really obvious pattern boggles my mind.
The book "Humble Pi" by Matt Parker mentions both the Therac-25 errors in separate chapters; I did not know from reading that book that so many people died as a result.
In 2015, my daughter was in this insane device. She had what they call “Cyberknife surgery” for a brain AVM. It was scary! She has made a full recovery, and was deemed “cured” in 2017. In the back of my mind, I am fully aware that she may likely end up with brain cancer later on in life. Here we are in May of 2023, and I sincerely hope and pray that she has a very long life full of love, joy, happiness, and good health.❤
Im sad to hear that. They should have found a replacement and retired this machine long time ago. I pray your daughter lives a long and healthy life :)
I’m very sorry to hear your daughter had cancer, I’m glad that she recovered well! CyberKnife Surgery is different from the Therac-25, as the machine in this video was banned in the late 80s. CyberKnife is a very helpful machine, usually used to treat brain tumors without open surgery. I will say I am not an expert and just started school for radiation therapy but I wanted to share this as it is thankfully not the tragic machine described in this video (but of course all radiation treatments can seem unnerving at first some to patients even when safe!). Once again I’m glad your daughter’s treatment went well and I too wish her good health!! :)
I remember this case study from an ethics class for software developers that I took nearly 20 years ago. It stuck with me all these years as a reminder of the impact engineering decisions can have on peoples lives.
My bf is a software developer not in the medical field, and he's shocked and disheartened every time he finds out of code being put into production without testing. If I tell him about medical software not being tested... especially since we both know people who got radiation therapy
This happened to my great grandmother in Illinois. Uterine cancer and radiation after the hysterectomy. My dad said you could see through to her bones, pelvis most likely, before she died.
Given how it always reset at 256, that means it was only a 16-bit counter… seems like it wasn’t quite the time for software operated medical equipment yet.
00:22 A state-of-the art machine and what looks like an oil lamp converted into electric light. The 1980's were a trippy time for tech vintage mixing :D
I'm quite sure the majority of people needing treatment with such a machine,had suffered greatly before laying down on that table,but due to the negligence and lack of oversight,one treatment caused suffering beyond measure and then death.. Just 💔 my whole heart 😞
As a radiology technologist this is my worst nightmare and I’ve seen issues with machines and poor programming. Nothing this severe but when I was in school…I can’t remember what kind of equipment a tech happened to be using but she went to do a knee on the wall Bucky and it made the exposure like it was in the table resulting in overexposure of the patient. With other machines I have seen usually it would prevent you from making the exposure if you were aligned with the wall and it thought you were taking an exposure on the table or you hadn’t manually switched it over. Same thing with how the equipment should prevent you from exposing if the door is open but I’ve seen that happen too. These are really important safety measures that were somehow overlooked in the design of this equipment.
i can always appreciate how with stories like this, or with rollercoasters, you take the time to reassure the audience that these incidents were corrected and modern technology is MUCH safer than it was. you try to mitigate fearmongering, which is refreshing
I remember this as part of our classes on radiotherapy in undergraduate medicine in my country, even though they were pretty recent (I graduated in 1989). To that day, the hospitals in my country were still using the previous models and there were no accidents that I recall. To the best of my knowledge, the Therac-25 was never installed in my country. Very interesting video, touching something in my line of work. Thanks a lot!
Smart man Andrew. I was in the hospital for 30 days in 2021 due to COVID. I almost died four times, not because of COVID but because of human error by the staff. I always say if the disease doesn't kill you, the hospital will. Stay out of them if at all possible.
9:00 "did not believe that software could malfuntion" And the software did not malfunction. The problem was that the design of the system was such that the software had no way of knowing the position of the mechanial parts. it simply assumed that when it started a new program it would always have all the motors in the starting position. So when somebody did something faster than the motors could return things to their starting position, the software simply commanded the motors to move from wherever they happened to be, not from the starting position. This is a typical "rookie mistake" that many many developers, designers and testers make; they blinbly assume that things will allways go the way they designed it and forget to handle the failure modes. Just look at the number of websites that have forms where the field for your name is too short because the designer never thought about people called "Octavianna Methafordia Withersham of Bletherobe". or the infuriating opposite, where your firstname has to be three letters or more, and nobody is allowed to be called "Ed". The Therac would have been completely fine if they had implemented a delay between ficing an error and starting the program. ofcourse, more sane people would add double or tripple safety where the software has access to positional sensors for the mechanicals and a sepaate controller would co-read the sensors to give an alarm if the main software put the motors in an ansurd position. But the safer a machine is, the more expensive it is... Companie like to make money, patiensts don't want to spend money...
As a cancer survivor, having gone through a long course of radiation treatment I appreciate your careful presentation. Thanks for pronouncing’Yakima’ correctly - outside of the Pacific Northwest USA most people turn the letter i into an e.
I was just looking through radiation incident reports last night and saw one that happened in April, 2023, where a patient recieved 3x the intended dose. In the report the authorities stated that it was not enough to cause any issues. But it's scary these things still happen!!
I’m cervical cancer survivor. My treatment included 6 chemotherapies, 28 external radiations and 5 brachytherapies ( internal radiation) . Watching this now, 8 years cancer-free, still makes me grateful for advanced technology. Although I experienced side effects and burns, nothing as severe as these people. 😢
Yeah, I'm a Firmware Engineer, and I used to work in the medical industry and I am glad I don't anymore. There is a lot of pressure knowing "If I screw this up, someone could very easily die." Software does what the programmer tells it to do, but when we make mistakes, it still does what we tell it to do. As soon as you started talking about the counter, I suspected an overflow of the byte, and that is just bad. Now we are definitely taught that this kind of thing can happen, and there is a lot of emphasis on system critical code, but things still aren't perfect. Engineers make mistakes all the time, as any professional can. Hopefully we can learn from these kinds of things, and build in better testing before things go out the door. This is also why code review is so important.
Sadly, I know a family who went through a traumatic thing similar. Except it wasn't the machine that made an error but the doctor or tech doing the radiation work. They had apparently, for a prolonged period, radiated a completely wrong part of their child's body and sadly, they succumbed to the damage done and the cancer still in their body on the opposite side they were radiating. I have deep trust issues with doctors ever since.
" I have deep trust issues with doctors ever since." That's understandable, just don't let this experience make you stay away from doctors, I've sadly seen people keep their kids away from doctors for this exact reason, and they got to bury their kids too.
@@vinny142 oh goodness no man. Human error ocurrs in life, it's all a matter of finding the right doctor and doing our best. I fear doctors but we gotta do what we gotta
Such a sad story especially for all that passed away due to this treatment. I can’t imagine how scary it would be to hear a sizzling sound in you head.
Giving the code to write for such a critical machine to a single developer without supervision is just utter madness. Developer here. Given what i saw during my career, it could have been much, much worse. The logic of a dev is not the one of a normal person ;)
While I can't possibly know all the details (I was five years old at the time) please consider that an enormous amount of this software development was in its infancy in 1985. This was a time when my dad would be hunched over a workbench soldiering chips onto ISA boards himself. Expansion cards came as kits you had to build. You could go to Radio Shack to buy spare parts for your expansion cards. It was kind of wild. It was absolutely _nothing_ like development today. Also: Cocaine.
As mention this case is covered in some software and UI design courses as an example of how not to run a software project. The fact that the software was developed by one person with very little documentation and testing was a recipe for disaster. When the programmer left AECL in 1986, the company was left with a system they did not understand, that was not tested (except in hospitals on patients I guess), and could not really maintain or fix. This is a perfect example of how *not* to build software and design user interfaces, in particular the timing of events and the order in which they occur is especially important to get right and can be tricky to test exhaustively.
I feel bad for the patients not only because of the pain and suffering they went thru but because of the indifference all the hospital staff must have shown, making life-altering mistakes and not even having the balls to own up to it or apologize. It’s scary how often people probably get hurt in hospitals and then gaslit by the staff so much they question what’s even real anymore. RIP
This was a case study in my computer science curriculum. There was a similar incident at the Cancer Institute in Panama with a Theratron 780-C Cobalt 60 teletherapy system (manufactured by the same company as the Therac-25) which overexposed 28 patients and resulted in ~9 deaths. I don't have the final outcome of the court cases, but the the courts attempted to hold both the operators and programmers liable.
The Theratron system worked correctly in all respects. The overdoses were caused by incorrect treatment planning. The treatment planning software (Multidata) was completely separate from the Theratron machine, and was developed by a different company. In my opinion the operators and the developers are both responsible. The operators knowingly used the software outside its intended use, and did not verify the results by measuring actual radiation doses. The developers used data entered by the operators, without verifying that the data conformed to design expectations.
I remember discussing this in a computer ethics class in university. I now do IT work for a pharmaceutical firm and i think often if the statement "your regulations are written in blood."
"A further complicating factor was that the Therac-25 often produced error messages. [...] Since most errors were minor and not safety critical operators were in the habit of clearing error messages and continuing with treatment whenever they occurred." Seriously? This is medical equipment and people's lives are on the line; regular error messages _should not be a normal thing_ no matter how minor.
I have seen the plainly difficult videos about accidents concerning these Therac devices. Those videos were NOT about accidents while in use though. Now I get why these machines turned up in less developed countries. Most of the videos ive seen about this machine are about people getting hold of them by either having a contract to clear a building or basically breaking in to abandoned buildings and stripping scrap metal. I personally would be very very dubious about pulling apart machinery that gets bought out of a medical centre that treats cancer or uses nuclear medicines or centres that provide xray photos to the public for the reason of hospital treatment. It just goes to show how sloppy the registration and maintenance of these machines are handled . Now I realise how these machines made their way to these poorer more deprived areas of the world.
Um, wow. 1. There are other radiation therapy devices besides the Therac-25. Not all radiotherapy orphan source incidents are related to this particular machine. It's like you watched a bunch of videos about random car accidents and decided they were all Ford Pintos because you also heard that Ford Pintos had a poor design that led to an unusually high frequency of fuel tank ingnition after crashing. 2. If Plainly Difficult did a video specifically on the Therac-25, it was not an orphan source video. Don't know if they did. I've definitely seen at least one other RUclipsr in the "nuclear disaster" or "general engineering accident" genre cover the incidents, though.
Goiânia incident in the '80s in Brazil (I think it had been covered on this channel before) -- but that wasn't a Therac-25, but a different radiotherapy machine. (Looked it up and it was a Cesapam F-3000)
@@Amanda-C. Yeah, from what I can gather, the Therac 25 did not have a radioactive radiation source in it. It produced an electron beam. A high power, high voltage one when used with the target to produce high energy x-rays.
the washing machine I'm using only shows numbers when error occurs, and I never bother to check manual when that happens. how ignorant those operators were for treating cancer patients worse then my laundry!
I've actually heard of this machine before in the "Set Phasers on Stun" by Steven Casey. The first chapter covers the story of how this error was found and covers many instances where products bring up the question if there's design error or human error in their operation.
The absurdity that led to this tragic series of injuries reminds me a little of that time NASA lost a craft because one team was working in km and another in nautical miles. Nobody was injured, but it was a huge loss.
I work in the field of medical device commercialisation and a HUGE part of that is ensuring regulatory compliance for both hardware and software medical devices. The fault with the Therac appears to be a design flaw that was not picked up due to inadequate verification processes during design. Some proper risk management during the design process might have stopped this happening because it would have been noticed that using a single software programmer is inherently risky if there are no checks and verification steps.
As a software engineer I can tell this code had been written by an amateur or mid-level software developer at best. Not thinking about overflows when counting errors via incrementing a variable, is like building a water container and thinking it can never overflow with water. Such a problem is so mundane in embedded systems engineering, that you are likely to even encounter it in a recruitment test.
To anyone watching this who still wants to know more, I would suggest Kyle Hill here on RUclips. This is really just a dip in the pool of the Therac-25. Amounts of radiation given could have been up to 100 times the intended amount. Please, look into it more
Damn, this reminds me of the horrific scene in Final Destination 5, where the LASIK eye surgery machine malfunctions. It’s awful to think that something similar really happened to several people, but I’m glad that lessons have been learned, and that mechanical safety features and independent software review have been added for modern cancer treatment machines.
God that scene made me cry. It was horrific. Still can’t watch it again to this day. Trying to relate it to these poor living people doesn’t even comprehend
As an aspiring mechanical engineer, its really unfortunate and surprising that this happened. Your comment about AECL almost not understanding that there could be communication errors between the software and hardware feels correct, as crazy as it sounds. How could they put so much faith in the software otherwise? Really bizarre, but maybe it does come back to greedy and rushed decisions (as it usually does).
Not necessarily greed, often incompetence and a failure to understand how software works. The leads to inadequate testing because possible failure modes are not tested because they are assumed to be impossible. Also, error messages should be understandable to the operators, e.g. error 54 is meaningless to most.
Thank you for all the content you have produced. A light you have shined on various topics most don't know that are important. It's crazy to think that it's been 4 years since the first vids 🥰 awesome work!
As a radiotherapy physicist, Fritz Hager is something of a personal hero. He didn’t just spot the error - he spotted it by sitting through a weekend with a therapist until he got to the bottom of it. He refused AECs explanations and refused to back down. He’s the paragon of a medical physicist!
Today this same person would have their medical license revoked and be blacklisted from ever practicing again.
@@knurlgnar24 shut up
He's a man who thought for himself, and was confident and determined enough to go after the truth. Absolute inspiration!
@@knurlgnar24 A sadly-accurate take from you.
the counter overflow problem is so basic the programmer must have had no experience at all. but sometimes people just do the very least they can get by with including consciously pushing a problem into the future instead of addressing it, which that' could also be an example of
No software bug has killed more than Therac-25, and this accident is still the #1 case study in software quality assurance over 30 years later.
How about the Boeing 737 MAX?
Android now disables all apps, even emergency alert ones, if they're not used often enough. You cannot know they are disabled, the tiny notification - a list of disabled apps - doesn't always work. You just suddenly stop getting all emergency alerts...unless you manually check each apps's settings, every day. FEMA, Red Cross, severe weather alerts, wildfire Watch Duty - even the built-in-phone alerts from the government. Not strictly a bug, but, unintended. And has probably killed already. If not, it will soon.
@@randomdude1053 A big part of that was a documentation/training issue. Poor documentation is often even worse than bugs.
Yeah...... Alerts are an Android setting, not an app. The system disabling apps has absolutely nothing to do with the alert system.
@@randomdude1053 B737MAX crashes were not caused by a software bug. MCAS functioned exactly as it was designed to do, given the inputs it received. The problem was the design was poor (there was practically no limitations on how much and when MCAS would activate) and the input was incorrect (MCAS was fed data only from a single sensor, instead of 2 that exist on the plane, and these sensors are known to have issues). Additionally, Boeing's insistence about not disclosing the existence of the system meant that pilots had only a vague idea of what was happening and what to do whenever unintended MCAS activations occurred.
Having your brain fried by a radiation machine and slowly realizing your own death is imminent... is definitely the stuff of nightmares =[ RIP to those who passed
Hearing your brain sizzle is in the top 10 worst things imaginable
yeah, that made me feel sick...
All our death are imminent tho, relatively speaking
@@civotamuaz5781 - Top 10?
I'd put it in the top 1.....but that's just me. 😉
@@civotamuaz5781 If it makes everyone feel better, the sizzling sound was most likely caused by the saturation of the ionisation chamber that measures radiation below his head in the bottom part of the machine.
There was no "real" heat applied. It was just a reaction to the radiation to the nerves/receptors in the skin.
Redundancy in medical devices is an often overlooked necessity. Kudos to the physicist that investigated error messages on his own.
Redundancy in ANYTHING that has the possibility of taking a life if it fails, is an important necessity that gets overlooked a lot of the time due to cost or human stupidity.
A device should only really be considered safe for human operation if a drunk, sleep deprived, brain damaged capuchin monkey could operate it safely.
the worst part here, is that the guy writing the software didn't write it to be used without a mechanical interlock. Yeah, he wasn't even a professional programmer. which is why the UI for the software was so bad. Dangerously bad tBh... numbered errors the techs can't interpret? SAFETY HAZARD!!!! but as originally designed, the software worked safely.... then the company redesigned the hardware in a way that made it unsafe.
@@marhawkman303 yeah, it's on the company, rather than the programmer. Arguably you should have someone else doing the UI. It's a programmers job to make the program work, it's a designers job to make the UI user friendly.
But yeah, mechanical fail-safes are very important, both as emergency stops/safety overrides AND for mechanical backup operation.
@@kiritotheabridgedgod4178 Yeah, I think the company was trying too hard to push the idea of the treatment being "safe" that... they were creating the ILLUSION of safety.
The previous models didn't have this safety issue, not because it was an amazing design, but because it had a mechanical interlock. then someone came up with the idea of touting a lack of mechanical interlock as a "SAFETY" feature?!?!!?!!?
the UI being terrible certainly hurt. the people using it had no idea what the errors meant. and... the software had no idiot proofing.
But I get the impression that the company wanted to HIDE the idea the machine could even in theory hurt someone. but took this so far they didn't write into the technical documentation that there were dangers.
Thus the techs using it didn't KNOW that some of the errors were actually fatal... to the patient. and that's the REAL issue. the company making the machine was so focused on touting it as safe that they had compromised safety by simply not discussing dangers.
Exactly this
The thought of someone just casually clearing error messages on a machine that shoots radiation scares the hell out of me 😨
It's like ignoring your check engine light. And just like when you're dealing with the a mechanical item, there can be irreparable damage caused to human tissue resulting in death. In the case of your car you could blow your engine up.
@@Smedley1947 Most if not all check engine warning lights are emissions related and will not cause the engine to blow up, hence why the light is yellow and not red. A better analogy would be to ignore the oil pressure warning light.
“If I clear the error it never happened!”
The worst part is that the professionals and doctors around them didn't take them seriously so they were suffering alone until it was undeniable that something horrible has gone on.
I hope those idiots were fired for incompetence.
The machines were in use constantly and treated hundreds of patients. One person says I felt like I got burned and there’s no mark what should the operator think?
Unfortunately, with a lot of infrequent issues of this nature. People just think it’s a fluke until there’s enough incidence to show that it isn’t.
No the worst part was radiation poisoning.
Rare are doctors who take people seriously in situations beyond the norm, unfortunately.
My radiation doctor is Strange. When he asks if I'm in pain and I say yes, he immediately says it's not from the tumor. Yet that is where the pain is, plus where they screwed up putting chest tubes in me. But some doctors just really don't listen to their patients.
Yikes the case of the one guy who heard a sizzling sound and died of radiation burns on his brain is so scary makes you wonder how many unreported accidents and deaths happened because of the machine
I was thinking along the same lines… a radiation burn to the brain sounds like an absolutely awful way to go…
Medical incompetence and mistakes are more common than the general public would like to believe. Try to sue anybody who works for the NHS in the UK and you have absolutely no chance of any form of compensation.
@@memyself1566 nowadays if you show up for a headache to a us big hospital, you will be given and charged for a CT scan in case it is a rare disease and the doctors don't want to get sued, again
@Me Myself as someone who is from the uk myself i know the nhs isn't fit for purpose
Man, this is the stuff of modern day nightmares.
The guy who found the problem should be paid a finders fee. Think of the number of lives he saved.
Well not really how they think about it, eventually someone would’ve figured it out. The big fish don’t really care about the rest of us, if that were the case they would’ve released a finished product.
This was decades ago. The concept of a finders fee for software didn’t exist. His concern was is this very expensive. Very useful machine. Going to be a problem at my hospital.
trial and error is the way of life. unfortunately people are not blessed with making something perfect the first time.
@@coreyjohnson2205
Safety systems are not supposed to be trial and error. They’re supposed to be carefully engineered and carefully tested.
But kudos to his well constructed reality test.
An example to others doing software testing
For a second, I thought you were going to say the guy who found the problem is a "hero".
I'll never forget the patient who recieved radiation burns on his BRAIN. And the description of him hearing the sizzling of his own brain burning..... rest in peace man. Its so tragic that he suffered until he died, as well as the other patients who recieved burns from the machine.
Copy-pasting another comment from above:
Interestingly the "Sizzling sound" turned out to be the tungsten target being destroyed due to the beam firing at an intensity many times greater than it was intended to receive still bad for the patient but it did help them eventually figure out what was going on.
@@z9297you'd think that they could have test fired this POS on some hog carcasses first..I'd rather see it fired on the chain of people that signed off on that knowing damn well it just burned someone to death. Start with the tech that said it's fixed... Ok put your leg in and I'll listen for the sizzle. Oh would you prefer the hog carcass instead?😮
For those who care what the actual bug was: Once you entered all data and reached the end of the input process, a "data entry complete" routine was triggered, which would call a sub-routine multiple times to get everything on the machine arranged. While that happened, the software still allowed changes to be made to the data, so you could manipulate data while this data was read at the same time to setup the machine. Of course that can lead to an inconsistent setup and the programmer was aware of that. So while the setup was running in the background, a flag has been set and if you made any input change while that flag has been set, the whole setup process started all over again to be sure the setup used a consistent set of data. The problem was, that the first time the sub-routine returned, the flag was cleared at the end (about 8 seconds after data entry finished got triggered, that's the actual bug here!) and any change you made thereafter went unnoticed by the software, yet the changed values were still used in the repeated sub-routine calls yet to follow (the setup routine was called up to 8 times in a row).
So when you switched between electron and x-ray mode 8 seconds after you first triggered "data entry complete", the beam setup and the turntable position could go out of sync. In x-ray mode the beam intensity must be high (as the beam never hits you directly, it hits metal that will then produce x-rays that hit you), in electron mode it must be low (as the beam hits you directly), so if the beam was in x-ray mode but the turntable was set for electron treatment, the beam that hits you would be way too intense as there was no metal in-between as should have been. And Error 54 meant that the dosage you just shot at the patient was way too high ("Overdose Warning") and thus the device stopped the treatment; overriding that error was thus fatal as you just continued to shoot way too high dosages at the patient.
As software developer I see three fundamental flaws here. The first flaw is that critical errors must not be overridable . Error 54 should have stopped the treatment completely with no way to continue at all and the software should have informed the operator that this was due to an overdose in clear words ("WARNING!!! OVERDOSE!!! (Error 54)").
The second flaw is that even if the flag bug didn't exist, the thing was broken by design. Either the user interface should not even have allowed any changes while the setup was in progress ("Setting up. Please wait ..." and no interaction is possible other than "Hit C to Cancel") and if you cancel, the whole process starts all over. Or it would allow changes, however data input and setup would not use the same data set. Instead "Data Entry Finished" would once copy all entered data and the setup would only work with that copy, so when the user changes the data, that has no direct effect whatsoever. Only after all setup was done, the code would check once more if the copy it used for setup is still in sync with the data entered and if not, a new copy is made and the setup process repeats, followed by another check and so on. That way no such flag is even required. Both perfectly prevent such issues, the last one allows changes at any time but requires more memory (a data set copy must be made), the first one requires no extra memory but you cannot make changes while setup is in progress without canceling setup first.
And the third flaw is that there was no final sanity check. The software should have been able to query the currently set beam intensity and the current position of the turntable via sensors and before starting the treatment, it should have re-verified that those two are for sure in sync and that the beam is never set to strong unless the metal shield is in place. If those two are out of sync, which should have never happened in the first place, the machine should lock itself up ("FATAL ERROR!!! CALL TECHNICIAN!!! (Error ...)") and not allow any treatment at all until unlocked by a technician of the vendor who must find out how this could even happen.
As a fellow software engineer, this was the best explanation of the error that I've ever seen. Nice going!
One thing I thought of is, if they had a semaphore on the write function that disallowed writing while the data was being read (AKA when the system is setting up), it would have prevented it as well. But no clue if those computers of the time had things like that...
@@n00byscrazycorner43 You can write a semaphore without any extra hardware support, as long at the memory model of the CPU is strongly consistent. Only if the memory model is not, you require a special CPU instruction that guarantees strongly consistent ordering. However, even the 8086 and 8088 (16 bit CPUs, launched 1978) already had a LOCK instruction for exactly that purpose as the memory model of x86 is not strongly consistent.
Yet if he had obtained the semaphore when the setup process started and released it only once it's done, this would have lead the "Please wait..." solution I suggested. Also my other solution, the one with the copy, would normally require a semaphore as well, since you can only get a consistent copy if you ensure that no data is altered while you are copying the data (copying the data is not atomic, after all). Yet both these solutions don't require a semaphore as the software was in control when keyboard input is processed and when not, so it could always disable keyboard input, then perform whatever operations it want, before re-enabling it again.
What it did instead was never disabling it but every time a keyboard event was triggered, checking if that special flag was set to know if it must re-start the setup process at the end (if there was a key event for data input and the flag was set, the state of the state machine would have been reset to an earlier value, triggering a re-setup in the end). This would have worked correctly if the flag had only been cleared at the very end of the setup procedure and not every time the sub-routine has been called (the flag was cleared at the very end of the sub-routine, instead it should have been cleared after the loop finished that called that sub-routine multiple times). Still, only relying on such a simple flag looks way too fragile IMHO.
I even dare to take a guess how this bug came into being in the first place: Initially the setup process was probably simpler and the sub-routine was called only once and that's why the flag has been cleared there. Then things got more complicated and the programmer thought: No big deal, I just call that sub-routine in a loop and then it can make adjustments one by one, overseeing that the flag was still cleared on first call. During testing he never altered any data later than 8 seconds after the process started, as why would he consider waiting that long? He just tested starting the process, making a data change immediately and as expected, the flag was considered and everything worked smoothly. That's why every unit test framework today allows you to randomize timings and this should always been used when simulating asynchronous events as when those events happen can in deed make a huge difference.
That's a long over detailed list of reasons, while meanwhile it could be boiled down to *corporate corruption.*
"
And their is present day naiveté and normalcy bias. "Of course the government isn't corrupt, yes they experimented on US citizens and gave black people syphilis, but that was in the 70s
If the regulators stepped in, and did their job it would be cool but sadly that isn't strictly trustworthy anymore:
It's always incredibly sad when a machine designed to save you ends up being a machine that harms you.
When it comes to medicine, sadly Help & Harm are two sides of the same coin, separated only by a hair.
Like experiment mRNA 'vaccines' ?
Pretty much everything we make kills us either over the life time like cancer or quick like a car crash
yep, imagine if the remedy cause more d34ths than the illness... oops.
Every robot uprising ever
What's scary about the Therac-25 was the amount of radiation patients had been exposed to, the radiation exposure was on par with Cecil Kelly after the criticality accident, it's frightening that these poor people were exposed to that much radiation, and it was sickening that AECL was too arrogant to believe their machine was a killer.
They should have been shut down forever far as I’m concerned
Machine wasn't a killer.
It was made by humans after all.
Why not blame them?!
Anything that shoots radiation at a person is a killer, period. This entire industry is criminal!! I’ve watched a friend from school who was only 30, then my mom & my best friends mom along with another childhood close friends mom, all back to back, found out they had cancer but all looked fine upon diagnosis….post “treatment” all 4 of them slowly withered away until they died…all 4 died *EXCRUCIATING DEATHS!!* All of them had chemo & radiation. I don’t know how a single oncologist wakes up in the morning & is eager to start their day….how many patients have to die slow agonizing miserable & EXPENSIVE deaths before they realize their entire industry is bullshit??? 🤷🏼♀️
On top of that, my grandmother was told she was dying from breast cancer & only had months to live, desperate to survive she went through a double mastectomy & radiation that burned her so badly she had to have her arm amputated at the shoulder- yet she was misdiagnosed & never even had terminal cancer, she sued the shit out of whoever did that to her & lived another 10yrs but I’d imagine she would have been happier without money if it meant keeping her body parts & avoiding the trauma of believing she was facing death….
@@EphemeralProductions a bit like “guns don’t kill people, people kill people” so if we got rid of the defective people and just kept the guns, all would be fine. In reality we need to get rid of the guns AND the people both.
This radiation machine needed to be looked at the first time not as is normal after a long time and many more deaths or injury.
People are so full of themselves they just cannot see that there could be an error.
@@Whocares158They fed cows the animal entrails through feed. When cows got sick, they cleverly said that cows are mad. Don't look at us.
Radiation is a terrible killer. When you realise there is a problem you are often dead already.
"Everyone else is ok. Except me." - Louis Slotin
And then you often have to deal with that error until you are dead, could be weeks even if you received a fatal dose
"Not only will this kill you, it will hurt the whole time you're dying."
So it is the opposite of a "terrible" killer.
The worst part of it is that there is so much propaganda about it, that not believing it happened as a default state is perfectly understandable.
That error counter reminds me of a part of the Jurassic Park novel. The company has a computer system that scans the number of dinosaurs in the park which is set at a specific number (say 250). The problem is, nobody ever sets the number higher to see if there are more dinosaurs.
That is one of my favourite moments in literature. Such a chilling example of "you only find what you are looking for".
"the hell is that?"
"We've picked up another compy"
"From where??"
Such a good book
It has nothing to do with that though. This is a buffer overflow.
The therac company also made the same mistake by putting all their trust in one glorified programmer.
@Patriarcus Rex one unknown programmer. Who could have been just a hobbiest. I can't imagine being that guy really. Imagine hearing about all these software errors that KILLED people.
I can’t even imagine being on that table and not only feeling but hearing your brain sizzle? Definitely nightmare material.
Copy-pasting another comment from above:
Interestingly the "Sizzling sound" turned out to be the tungsten target being destroyed due to the beam firing at an intensity many times greater than it was intended to receive still bad for the patient but it did help them eventually figure out what was going on.
@@z9297 Stop trying to sugarcoat it because we all know it was that poor man's brain frying in his skull from the massive dose of radiation that was fired into his head that caused the sizzling sound and it ended up killing him!
The fact that this device is cited as a cautionary tale in both mechanical engineering and programming disciplines as a "What not to do" is why I first learned about this. Hearing about the horrors the patients suffered though...
Yep. Im a computer software engineer and that is how i heard this. I also worked on medical software 😳
Yep. You NEVER increment a variable to set a true state (or decrement for a false).
The weird thing is that some hospitals still use them.
I work in IT and honestly the over-dependence on software is a PROBLEM, to the point where people sometimes fail to use their heads for problem solving.
Like…software is built by humans. It’s subject to human error.
Exactly so, and it makes me wonder what's going to happen to the human race once AGI takes over.
A creation is only as good as its creator.
Both my parents were excellent at driving, and had a great sense of direction. If my father had previously been somewhere he could remember the road layout really well, for example. Once he got a Sat Nav, he couldn't remember the 20 minute route to my home without it!
I looked for a comment like this so I wouldn't have to write it. You would think there would be no trivial errors when operating a machine dispensing radiation treatments.
@Alex your comment reminded me of the MCAS aviation software that malfunctioned, causing a fatal crash and many lives lost. I love watching Air Disasters and I think too many pilots rely on software rather than their pilot training. It seems that the more that is done for us, the duller we become.
@@Nalla762 excellent point to remember that MCAS issue @ Boeing! You're exactly right regarding pilots flying now. Basically they are present in the aircraft to take off and land...that's pretty much it. Scary, huh?
Another thorough episode. However, I’m glad I didn’t see this before undergoing radiation treatment last year. “Sizzling sound”
Interestingly the "Sizzling sound" turned out to be the tungsten target being destroyed due to the beam firing at an intensity many times greater than it was intended to receive still bad for the patient but it did help them eventually figure out what was going on.
@@lesdmark "Bad for the patient" is a contender for understatement of the year.
His brain slowly melted after.
I mean,even with the number of accidents,they were still a rarity.
But definitely way to high.
Aside of that, as this channel shows, something bad having happened most of the time means itwill happen way less or not anymore.
So the fact something didnt work right in the past, and we're still using it, means that its safer than ever.
Finally, I hope youve overcome your cancer.
Best of wishes and luck for a good recovery mate
Hope you’re doing better now
It's like air travel. Keeps getting safer.
As a radiation physicist that works on cancer treatment devices, this incident is such a big case study while in grad school. My coworkers and I even discussed it about a month ago because it is such a notable incident in our field. From my experience in a cancer clinic for 3 years, I can assure you that when it comes to treating patients, it is done as quickly as possible. If there is a shortcut discovered, it will almost always be used. This becomes a frustrating point as most of the time if the operator slowed down, did not rush, and pushed the buttons in the correct order, then faults, collision, and accidents would not occur as frequently.
Thanks for sharing that info. If I ever get the dreaded cancer diagnosis, I'll probably pass on the radiation treatment if such is proffered as a method of treatment to me. In this day & age, it actually seems archaic anyway. I could be wrong, but I have never heard of a patient's life being saved from radiation treatment. Please correct me if I am wrong, though.
@@kimmuckenfuss2284 Radiotherapy is one of the safest fields in medicine, things have come a long way since the 1980s. It is used to treat over half of cancers and depending on the type of tumour you have, can often cure you.
If you're working somewhere where therapists are rushing and taking shortcuts you should be reporting it to the relevant safety authority. I'm a therapist working in radiation safety, I promise it's not the wild west out there!
@@SillyGoose-d5z thanks for the info!
@@kimmuckenfuss2284 Not to frighten you. It is an incredibly safe treatment method when done correctly. There is about 8-12 weeks of testing every single safety aspect of the system before it is released to a customer for use. With that, there is still a human element to all radiation treatment. Patients not aligned correctly, incorrect accessory use, accidental collisions, and other user induced faults occasionally occur. These can be more prevalent in a rushed environment.
@@SillyGoose-d5z After covid I don't trust any of you. Sorry but actually I'm not sorry.
I am a medial physicist. The kind that works with radiation therapy machines. I just want to say you did a great job covering this topic! You explained it excellently!
I’m an RT medical physicist as well. Excellent video!
These incidents were one of the first things we covered in school showing just how important checking and rechecking everything is in this field. I think about these and other incidents almost daily.
I like how you couldn't resist bragging about yourself and your qualifications, and tried as hard as you could to disguise it as a compliment to the channel 😂😂
Thanks for your comment. I'm sure you worked hard for those degrees and unlike some others actually have the right to comment about the content. Sadly this happened and I'm glad to hear they're teaching about it because we know something similar will happen again with another machine of some sort.
chill out ??
@@Adelicows bro? They were listing their qualifications as someone who can speak on the subject, confirming that it was correct
That’s helpful to the channels reputation on top of being a compliment
Imagine suffering through cancer and a radiation burn, only to get taken out by a car crash 💀 What a plot twist
Right? She didn’t need that. Universe needs to chill and cut this lady some slack.
Sounds more like a suicide after getting screwed by the hospital and justice system
Wow, life, never know what's around the corner!!!🙏🤔❣️
@@__________________________Fred really? I immediately thought of Corporate hit to silence someone.
@@alastor8091 me to
It is surprising they left software on this level of importance up to a single individual without ANYONE reviewing the work.
You expected better from the Canadian government?
@@The20thHijacker lmao
Ikr!?
People think consultants in any industry don't do anything, but this is one case where hiring someone from outside to review the code could have saved lives. It's the same as any kind of writing--copy, correspondence, etc. Putting another set of eyes on it often reveals an error the author missed during their own composition and editing process.
In my IT career I cannot remember the number of times mission-critical software was released full of bugs. For example, payment processing systems that could not accept payments, simply ridiculous the number of errors that get by. It is pretty much 1 hour of testing required per 1 hour of programming. No, the developer CANNOT test their own code, they already thought of everything they could and guaranteed those things wouldn't happen. Still, there are a million things they did not think of, did not implement code to prevent, could not conceive of occurring that will happen on the first day of release to the public. Professional software testing is critical and expensive. You want good, fast, and cheap? Pick any two. That's why I take all these assurances that A.I. cannot and will not be a problem...with a grain of salt. Or a bag full.
My wife received lung cancer treatment using a Novalis photon beam machine. One day they called and told us the machine had not passed its self test. A beam forming shutter actuator was not operating within its specified time limit. A service technician replaced the failed part. She received her treatment later that evening. The level of engineering related to safety has made huge progress.
This reads like an AI-generated story. Did you take the owner’s manual home with you or something? Lol
@@n00bfest32Bro just shared a story of an experience with updated safety checks on his wife’s cancer treatment and this was your chosen response? If you couldn’t infer that the tech clearly explained the issue to him via the phone call or later appointment, I have serious questions about your literacy skills.
@@n00bfest32What is wrong with you???
As a cancer patient that did chemo and radiation, this vid is in the top 5 scariest you've done!!! Thank u. Bless the patients and their advocate
Bless you too.
I’m going thru chemo now and def feeling some shivers from this video 😅some horrors are the ones you find in everyday life huh
Breathe easy. It's 💯 better now. You're stronger than you know. 💓💞💕
I used to be a radiation therapy tech and this is one of the cases we covered in class. It was terrifying
Can X-rays cause radiation sickness?
@blazee3895 Yes if given at very high doses, and to the whole body.
I can't imagine being already vulnerable and fighting for your health and life then being severely injured and/or killed by the means in which should be treating you. How awful.
Psyche meds, painkillers, vaccinations... this is a common result coming from "modern medicine".
@@nojuanatall3281how about you go spew your conspiracy theories elsewhere xoxo
Hey, thanks for pronouncing Yakima correctly! Most people dont even attempt to pronounce it correctly when Ive watched other videos about the Therac (Plainly Difficult, Im looking at you).
-someone who grew up there
I remember my dad talking about this when I was a kid in the 80s. He knew people who worked in the Oncology dept at the hospital when both the Yakima accidents happened. Apparently, when the first lady was burned (she survived her injuries), radiotherapists were suspicious the machine somehow overradiated her, even though the company AECL basically blew off the concerns and said there was "no way" it could have been the machine. If they had taken the reports seriously, the man that died in Jan 1987 (also in Yakima) wouldnt have been killed. But they kept being assured at first there was no issue, then "oh we fixed it".
AECL was more concerned with covering their own rears than making sure people werent hurt.
Corporate greed knows no bounds.
I grew up in Yakima too and just cringe when people mispronounce the name.
@@mklaebel That is still a problem no one seems to be able to fix!
Its said yak-i-mahh correct? That’s how I’ve always heard it said but I’ve also heard yak-i-muh.
@EphemeralProductions
Both those work. Locals say it as a weird cross between the two. But as long as you dont say ya-KEEma no one will get pissed 😆
My deceased father was a radiation therapist, but never talked about his job much. Thank you for the enlightenment.
you know this channel really delivers on its premise. completely fascinating. completely horrific.
As someone who is currently battling cancer and having gone through radiation here in Georgia, this is terrifying. I know technology isn't perfect, but when it's life or death, it really takes things to a whole new level. I'm so glad these errors were found and fixed. RIP to those affected by this ❤
I think that in this particular case, the machine itself was the error. Fixing it was ultimately ceasing use.
Good luck.
Congratulations. I've seen a couple of reports / videos on the Therac-25, and this one is easily the most comprehensive.
Kyle Hill's video is pretty good
Painly Difficult's is also pretty good.
Well There's Your Problem's three hour-long podcast does a good job of it, too... if you've got the time commitment.
Radiation is incredibly useful but ridiculously dangerous especially if the machine administering the radiation was poorly programmed.
true, it is a double-edged sword
This incident was the literal textbook example during my software testing course on importance and the potential problems of testing (because bug-less programming is a pipedream).
It's also used to teach human factors/ergonomics engineers (i.e. the people who design the user interfaces on kit like this to make sure it's simple to use and that it isn't possible to accidentally kill anyone) - whilst it's horrific, hopefully the fact that it's so widely taught will go some way to making sure this never happens again
I can't speak for this particular radiation treatment, but it's certainly true radiotherapy is even today a double edged sword. My uncle had an inoperable brain tumour to which he was offered a very new intense therapy in the hope it would extend his life. All I can say is for all he was terminal, the "new improved" radiotherapy shortened his life drastically which makes me believe these treatments are far from "tried and tested". Tested, maybe...on my uncle.
@@jessikatkins1173so true! I had radiation in 2012 to treat breast cancer and I had a bad burn that got infected. Even tidy years later I still have trouble with the lung the radiation was beamed on.
Maybe I’ve seen too many horror films, but radiation therapy machines scare the crap out of me. This is exactly the kind of nightmare scenario my brain always plays. How horrifying for the patients who suffered from this malfunction/bad programming!
You hope you don’t get cancer then because many people have no choice with this treatment. Sucks being human
@@ShinmegamiPersona Seriously, I'd panic just being in the same room. Here's hoping!
If it makes you feel any better, this is very much a case of multiple things going wrong all at once. I have reason to suspect a modern radiotherapy machine would NOT have the problems the Therac-25 did, given it's now a literal textbook example of what can go wrong.
@@shadowldrago I actually started looking into this after watching this documentary, and you're right - it looks like it's used in computer programming degrees and also medical ethics in computer programming. Good!
@@JustMeUpNorth Agreed. It's terrible that this went so wrong in the first place, but it's good that those involved learned from this scenario and took steps to make sure it would never happen again.
Whelp! Time to add ‘radiation burns to the brain’ to my list of irrational fears. Just the idea of needing any surgery puts me on edge, but hearing a sizzling sound during treatment is a whole other level!
Yes, that was sad and tragic. I hope that man is in a better place now
ill take the cancer thanks
Yeah, this was SO VERY HELPFUL FOR MY ANXIETY, THANK YOU. 😭
The sizzling sound wasn't the flesh melting, but the tungsten target, wich it's still terrible, as tungsten shouldn't be sizzling in the first place
🥓Sizzling fried brain🧠
🤭
Great job as always! My ex-husband and I both work in the tech industry, specifically around software coding and testing. He told me about an interview he had early in his career (mid-90s) where he was asked if he could do a job that if he made a mistake, it could result in someone's death (it was a job writing software for medical devices). He didn't take that job but I always thought about it and realized I would never want that level of responsibility. I wonder how that software developer felt when he heard his software caused these deaths.
Not so bad. You can go for big bucks, and be sober, cautious. If you think about it, taxi driving, or even giving a ride to anyone in your car is the same thing. Just a mind trip one can overcome.
@@KathrynsWorldWildfireTracking if you crashed and killed your passenger(s), that’s only 1 to 5 max people killed or injured. If technicians aren’t picking up on a software problem that’s killing people in a way that doesn’t obviously point to the software (in theory, it is possible), then that could potentially lead to a lot of people dying. A lot more than would be involved in a typical car crash. And if you’re so bad at driving you’re regularly having accidents, it’s unlikely you’ll keep your license for long.
My husband is currently undergoing radiotherapy - for some cancers, it is the best treatment & can lead to a full recovery. This is what we hope for. However, I knew of the Therac-25 accidents & this was the first thing my mind went to when he began treatment. Sometimes I wish I knew less, but forewarned is forearmed. The likelihood of another accident of this nature is astronomically low these days because of what was learnt here.
Good luck mate, and I think it's better to know these things so you can keep an eye out.
Best wishes to you and your husband!
Best treatment is fasting
@@showmethemonny5796 Don't spread such stupid conspiracies 🤦🏼♀️
I wanted to say that even though I can't watch this vid (I'm dealing with cancer and might need radiation, so brain started screaming at this lol), I really appreciate your steady uploads and fascinating stories of things I'd never heard about. Thank you so much!!!
I hope you’re doing well & recover soon ❤
💚🤍🧡🤞🤞🤞
As unfortunate as these events were, radiation therapy has been made much safer thanks to them. Events like these are no longer something you have to worry about when getting radiation treatment, we learned well from our past and the Therac-25 is frequently used as an example in computer/programming ethics to ensure we don't make the same mistake again. Know that you will be safe and your concerns will be taken seriously. I wish you all the best in your fight, stay strong, you got this.
I hope you heal quickly! Sending positive vibes!
Yes, cancer treatment/therapy is quite the slog. Chin up & I wish you the very best results.
My understanding of the original bug is that a race condition existed where if you got good at navigating the menus you could move too fast for the software and produce invalid conditions. Also, although the software made note of errors, it didn't really properly treat them all as interlocks. This was definitely the result of treating the software as you would in any other industry, by hiring someone to produce a working program and then fixing it as problems arose. The fact that the machine routinely threw out vague errors is basically the same thing as training operators to ignore safety in favor of getting the job done.
The race condition was only one of the issues with the software. It had a lot of issues.
what i dont get is why errors where so easy to disable. an error should be a huge deal and cause for a specialist to come check and manually verify the machine
@@jonathandpg6115
Because just like anything else, some errors are minor in summer important.
In this particular case, the documentation didn’t tell you what the errors meant. The screen certainly didn’t. And as he pointed out lots of errors that you have to constantly cancel trains you to ignore errors.
The problem was that it was a warning but should have been an error, programmatically speaking.
Okay, as a Gen Xer, I remember the days when when computers were dumb, cumbersome, and really limited and you could out-race them. How I hated when you'd get UTTERLY USELESS messages along the lines of "Error 5" but with no further explanation of the problem. Making the error message basically a bunch of useless nothingsauce. A bunch of 1s and 0s would've been just as helpful. Oh, how the early days of personal computing honed my hatred of bad software interface design.
Having some sort of explantion with those Therac 25 error messages might have saved a life. Really, how hard would "Error 54 Danger, turntable in wrong position" have been? And even back in the days of 128 and 640k computers, it would've been possible to have a message like that. You just had to be willing to write it instead of assuming that everybody always had the manual to hand (because those never go missing at large institutions), or had memorized a table of error codes.
Woof! This brought back memories of getting radiation treatment for breast cancer. My machine worked correctly and I still had burns and lots of pain. I don’t want to imagine what those other folks went through. Horrible!
On a nicer note, love your videos. They are always interesting, thoughtfully written, and your narration is so smooth.
As a bioengineer, as soon as you started explaining the modes on the turntable, I recognized this case. It was a case study used in one of my grad school classes.
It only went over the first patient in that presentation, and for some reason, only mentioned the issue with the turntable, never the overconfidence in the code! And it is so chilling to learn just how much worse this series of events actually was.
As someone who dabbles with code myself, the second I saw an ambiguous error and heard “programmed by one guy and never debugged by external sources”, my heart dropped. That is so horrifyingly irresponsible for a medical device using RADIATION!?!
And don’t give me “It was the 80’s computers were new”. Hell the NES was a home console and had more thorough debugging than that! People had comodore computers and shit! With advanced music and games! A high end medical device company that was literally putting peoples lives on the line should have had way more thotough ethical coding practices than that. And also MECHANICAL BACKUPS!!!?! Why the HELL would you do away with that!?! It literally would have added what… pennies? Maybe a few dollars? To these likely million dollar builds? In exchange for the priceless cost of actual human lives?
Absolutely shameful. And that overflow error. If you maximize a bit, do NOT overflow!?! Lock it out at like *half of that* just to be safe! Its literally an “if” check and a “stop” statement. Algorithm would be like: “code to add to error counter”, “if errorcount >= 100”, “STOP”. Do Not wait until the end of setup!?!
Sometimes your programmer needs to be taken out back and shot. I’m so thankful for Fritz Hager sitting down with the techs, seeing how they use the machine, and manually debugging that shitty code that was literally causing that “state of the art” death trap to kill people, and ultimately get it taken off market.
Sorry for the wall of text. The full context of this just really upset me as someone in the field that makes such things. Patient safety should always be the absolute top priority for anything you would even think about bringing to market. Regulations are written in blood, especially in this industry, and that pisses me off.
One of the best channels on YT, hands down
It's decent for sure, but there's way too many impeccably strong contenders for that title.
Absolutely agree!!!🙏👌🦉❣️
The documentary style presentation is very high quality here. I do enjoy Kira's documentary style videos as well.
@@nojuanatall3281
I can recommend some noticeably better ones if you like? The sort of people who tend to work for an entire year or more on making just one single documentary the absolute best it can possibly be.
Have I seen this covered by Plainly Difficult? Yes
Have I seen this covered by Kyle Hill? Yes.
Will I still watch this video, anyway?
Damn straight 💪
Same here with the huge amount of overlap between Fascinating Horror and Well There's Your Problem, who did this a couple of months ago, I think? Though it's short and concise here, and with WTYP it's 2-3 hours of telling the story and half making dumb jokes and doing recurring bits 😄
This and Plainly Difficult are two of my favourite content creators
Same here! 👍
but im gay
Plainly difficult has waaay too many factual errors in his videos. Make it a habit to read the comment section for fact checks if you watch his content.
I wondered when you'd get to this series of events, which make up *the* textbook case study of unintended consequences in the discipline of software engineering and continue to be a sobering warning to others down through the decades.
Yes, all software people should study this case, since it’s a textbook case of how not to write safety software.
These days you’re supposed to do a much better job to get your equipment certified
I'm studying to work as a rad tech in radiation therapy and we have discussed this case before! It definitely was an horrible tragedy that should have never happened. Here in Italy now there are daily check ups made to the machines to prevent these kinds of accidents from happening, and during the planning of the treatments there are a lot factors considered for the safety of the patient. Thanks again for the informative video!
Imagine a machine like that throwing an error message and it being common procedure to just ignore it!
This is my absolute favourite story of safety failures in software. If anyone is interested in reading more, check out the Cobalt 60 machines used in South America, some awful stories of similar events. The hospitals were in poorer areas and had no support from the manufacturer, no logbooks etc.
Please also look into the Toyota unintended acceleration case! Lots of settled cases and deaths.
I looked into it and **holy crap**. I'd never trust Toyota after something like that!
Philippines also had that Toyota moment, but on Mitsubishi cars and fortunately no deaths.
The Max was a monumental software failure. Not sure any other software failure killed 200+ in an instant within 6 months of one another.
@@bradsanders407 “in an instant within 6 months” bro come on
The Toyota case isn’t so clear-cut. Apparently their software sucked. But a lot of people panicked and didn’t do what they needed to stop the car.
Yikes! I live in Atlanta where the first accidents took place and never heard about it. I had radiation for breast cancer 15 years ago, so glad I didn't know about it.
How,d it go you recover good?
@@chasechildress2720 yep, doing well!
That's wonderful news!
@@sharonsmith583 that's amazing.
Used to be an X-ray/radioactive materials inspector for the state of PA…you wouldn’t believe how many accidents happen during these treatments but they make so much money doing them 🙃
remind me not to ever get medical treatment in Pennsylvania
Great, thanks.
Will have to wait until fatal accidents for them to make a second thoughts on using them
Yes, I use accident's'
Murica
That's terrifying! I work in radiation safety in Europe and I can tell you serious incidents are few and far between here. Safety culture in Radiotherapy is huge.
This is incredible! Operators just clean the error message and carry on. Wow! These poor people! They were already suffering with cancer and then this happens!! The machine had to many flaws worsened by bad operation decisions. RIP to all.
I have a friend who does software testing for an enterprise solution type company. Almost daily, he's telling me about incidents where one of the engineers insisted that his code was perfect and didn't need testing only to find out that the code was not perfect, did need testing, and in the cases where the engineer was able to push the untested code through, then caused serious problems. The level of logical disconnect where someone can be so smart and yet not see this really obvious pattern boggles my mind.
Sadly there are lots of narcissists around,and some of these are operating these machines 😳
The book "Humble Pi" by Matt Parker mentions both the Therac-25 errors in separate chapters; I did not know from reading that book that so many people died as a result.
This is a very timely video as AI dependence and reliance is a hot topic here in the US (and probably other countries, too).
Don't worry, AI isn't the only issue, the software still is utter crap :) (I work as medical device software engineer)
This seemed a lot more like humans using technology erroneously to me
@@scp-682-cu6 Which is the fear with humans utilising AI.
In 2015, my daughter was in this insane device. She had what they call “Cyberknife surgery” for a brain AVM. It was scary! She has made a full recovery, and was deemed “cured” in 2017. In the back of my mind, I am fully aware that she may likely end up with brain cancer later on in life. Here we are in May of 2023, and I sincerely hope and pray that she has a very long life full of love, joy, happiness, and good health.❤
Im sad to hear that. They should have found a replacement and retired this machine long time ago. I pray your daughter lives a long and healthy life :)
I’m very sorry to hear your daughter had cancer, I’m glad that she recovered well!
CyberKnife Surgery is different from the Therac-25, as the machine in this video was banned in the late 80s. CyberKnife is a very helpful machine, usually used to treat brain tumors without open surgery. I will say I am not an expert and just started school for radiation therapy but I wanted to share this as it is thankfully not the tragic machine described in this video (but of course all radiation treatments can seem unnerving at first some to patients even when safe!).
Once again I’m glad your daughter’s treatment went well and I too wish her good health!! :)
It was retired in late 1980s if I remember right, and gamma knife is definitely a major upgrade
I remember this case study from an ethics class for software developers that I took nearly 20 years ago. It stuck with me all these years as a reminder of the impact engineering decisions can have on peoples lives.
My bf is a software developer not in the medical field, and he's shocked and disheartened every time he finds out of code being put into production without testing. If I tell him about medical software not being tested... especially since we both know people who got radiation therapy
Humans are dumb. Think of how many pharmaceuticals are put on the market after only 8 weeks of testing? Then humans use them for years. Scary stuff.
And that's when normally us, QA come into Light! Good our profession got more and more important in IT!
@@MiaMizuno what’s qa? Quality assurance?
@@lillidaisyASMR yes, it is.
These stories lately have been terrifying. Please continue ❤
This happened to my great grandmother in Illinois. Uterine cancer and radiation after the hysterectomy. My dad said you could see through to her bones, pelvis most likely, before she died.
Of course they denied any wrong doing even though it was clearly their fault its always the way isn't it blatant arrogance really
Yep. It's the "corporate way." Thank you capitalism.
thank you so so much for always having captions!!
Thanks
Its some dark comedy that it had too many errors for the error checker to keep track and just went ahead anyways.
Given how it always reset at 256, that means it was only a 16-bit counter… seems like it wasn’t quite the time for software operated medical equipment yet.
@@BlighterProductions If it goes back to 0 when you add 1 to 255, it is *8-bit* and not 16-bit. The 16-bit maximum value is 65535
00:22 A state-of-the art machine and what looks like an oil lamp converted into electric light. The 1980's were a trippy time for tech vintage mixing :D
I'm quite sure the majority of people needing treatment with such a machine,had suffered greatly before laying down on that table,but due to the negligence and lack of oversight,one treatment caused suffering beyond measure and then death.. Just 💔 my whole heart 😞
As a radiology technologist this is my worst nightmare and I’ve seen issues with machines and poor programming. Nothing this severe but when I was in school…I can’t remember what kind of equipment a tech happened to be using but she went to do a knee on the wall Bucky and it made the exposure like it was in the table resulting in overexposure of the patient. With other machines I have seen usually it would prevent you from making the exposure if you were aligned with the wall and it thought you were taking an exposure on the table or you hadn’t manually switched it over. Same thing with how the equipment should prevent you from exposing if the door is open but I’ve seen that happen too. These are really important safety measures that were somehow overlooked in the design of this equipment.
i can always appreciate how with stories like this, or with rollercoasters, you take the time to reassure the audience that these incidents were corrected and modern technology is MUCH safer than it was. you try to mitigate fearmongering, which is refreshing
I remember this as part of our classes on radiotherapy in undergraduate medicine in my country, even though they were pretty recent (I graduated in 1989). To that day, the hospitals in my country were still using the previous models and there were no accidents that I recall. To the best of my knowledge, the Therac-25 was never installed in my country.
Very interesting video, touching something in my line of work. Thanks a lot!
Yea, I refuse to go to a hospital unless my life is on the line because of things like this.
Smart man Andrew. I was in the hospital for 30 days in 2021 due to COVID. I almost died four times, not because of COVID but because of human error by the staff. I always say if the disease doesn't kill you, the hospital will. Stay out of them if at all possible.
@@irafair3015 Were you vaccinated, Ira?
Congratulations on hitting 1 million subscribers! 🎉🎉
9:00 "did not believe that software could malfuntion"
And the software did not malfunction. The problem was that the design of the system was such that the software had no way of knowing the position of the mechanial parts. it simply assumed that when it started a new program it would always have all the motors in the starting position. So when somebody did something faster than the motors could return things to their starting position, the software simply commanded the motors to move from wherever they happened to be, not from the starting position.
This is a typical "rookie mistake" that many many developers, designers and testers make; they blinbly assume that things will allways go the way they designed it and forget to handle the failure modes.
Just look at the number of websites that have forms where the field for your name is too short because the designer never thought about people called "Octavianna Methafordia Withersham of Bletherobe". or the infuriating opposite, where your firstname has to be three letters or more, and nobody is allowed to be called "Ed".
The Therac would have been completely fine if they had implemented a delay between ficing an error and starting the program. ofcourse, more sane people would add double or tripple safety where the software has access to positional sensors for the mechanicals and a sepaate controller would co-read the sensors to give an alarm if the main software put the motors in an ansurd position.
But the safer a machine is, the more expensive it is... Companie like to make money, patiensts don't want to spend money...
As a cancer survivor, having gone through a long course of radiation treatment I appreciate your careful presentation. Thanks for pronouncing’Yakima’ correctly - outside of the Pacific Northwest USA most people turn the letter i into an e.
I was just looking through radiation incident reports last night and saw one that happened in April, 2023, where a patient recieved 3x the intended dose. In the report the authorities stated that it was not enough to cause any issues. But it's scary these things still happen!!
My mother’s chest and shoulder were destroyed by a hospital radiation machine in the 1960s. I doubt it was ever recorded as an accident
I’m cervical cancer survivor. My treatment included 6 chemotherapies, 28 external radiations and 5 brachytherapies ( internal radiation) . Watching this now, 8 years cancer-free, still makes me grateful for advanced technology. Although I experienced side effects and burns, nothing as severe as these people. 😢
I forgot my lunch today, so I'll just load up on a giant helping of existential dread.
The first case really scared me because my Grandma was going to Kennestone Hospital for cancer treatment at the exact same time!
Yeah, I'm a Firmware Engineer, and I used to work in the medical industry and I am glad I don't anymore. There is a lot of pressure knowing "If I screw this up, someone could very easily die." Software does what the programmer tells it to do, but when we make mistakes, it still does what we tell it to do. As soon as you started talking about the counter, I suspected an overflow of the byte, and that is just bad. Now we are definitely taught that this kind of thing can happen, and there is a lot of emphasis on system critical code, but things still aren't perfect. Engineers make mistakes all the time, as any professional can. Hopefully we can learn from these kinds of things, and build in better testing before things go out the door. This is also why code review is so important.
Sadly, I know a family who went through a traumatic thing similar. Except it wasn't the machine that made an error but the doctor or tech doing the radiation work. They had apparently, for a prolonged period, radiated a completely wrong part of their child's body and sadly, they succumbed to the damage done and the cancer still in their body on the opposite side they were radiating. I have deep trust issues with doctors ever since.
" I have deep trust issues with doctors ever since."
That's understandable, just don't let this experience make you stay away from doctors, I've sadly seen people keep their kids away from doctors for this exact reason, and they got to bury their kids too.
@@vinny142 oh goodness no man. Human error ocurrs in life, it's all a matter of finding the right doctor and doing our best. I fear doctors but we gotta do what we gotta
Such a sad story especially for all that passed away due to this treatment. I can’t imagine how scary it would be to hear a sizzling sound in you head.
It sounds like sheer arrogance caused AECL to do a less than thorough investigation into the machines. RIP all of their victims.
Giving the code to write for such a critical machine to a single developer without supervision is just utter madness. Developer here. Given what i saw during my career, it could have been much, much worse. The logic of a dev is not the one of a normal person ;)
@@davidbocquelet-dbodesign thank you.
While I can't possibly know all the details (I was five years old at the time) please consider that an enormous amount of this software development was in its infancy in 1985. This was a time when my dad would be hunched over a workbench soldiering chips onto ISA boards himself. Expansion cards came as kits you had to build. You could go to Radio Shack to buy spare parts for your expansion cards. It was kind of wild. It was absolutely _nothing_ like development today.
Also: Cocaine.
As mention this case is covered in some software and UI design courses as an example of how not to run a software project. The fact that the software was developed by one person with very little documentation and testing was a recipe for disaster. When the programmer left AECL in 1986, the company was left with a system they did not understand, that was not tested (except in hospitals on patients I guess), and could not really maintain or fix. This is a perfect example of how *not* to build software and design user interfaces, in particular the timing of events and the order in which they occur is especially important to get right and can be tricky to test exhaustively.
I feel bad for the patients not only because of the pain and suffering they went thru but because of the indifference all the hospital staff must have shown, making life-altering mistakes and not even having the balls to own up to it or apologize. It’s scary how often people probably get hurt in hospitals and then gaslit by the staff so much they question what’s even real anymore. RIP
This was a case study in my computer science curriculum. There was a similar incident at the Cancer Institute in Panama with a Theratron 780-C Cobalt 60 teletherapy system (manufactured by the same company as the Therac-25) which overexposed 28 patients and resulted in ~9 deaths. I don't have the final outcome of the court cases, but the the courts attempted to hold both the operators and programmers liable.
The Theratron system worked correctly in all respects.
The overdoses were caused by incorrect treatment planning.
The treatment planning software (Multidata) was completely separate from the Theratron machine, and was developed by a different company.
In my opinion the operators and the developers are both responsible. The operators knowingly used the software outside its intended use, and did not verify the results by measuring actual radiation doses. The developers used data entered by the operators, without verifying that the data conformed to design expectations.
Happens sadly... I respect you for the manner that you share these appalling cases and God bless all those involved and their loving families. 💙👍☘️
I remember discussing this in a computer ethics class in university. I now do IT work for a pharmaceutical firm and i think often if the statement "your regulations are written in blood."
"Maybe we should put the mechanical interlocks back in?"
"Nah, the software will be fine!"
"You did check for other issues, right?"
"..."
"RIGHT?"
"A further complicating factor was that the Therac-25 often produced error messages. [...] Since most errors were minor and not safety critical operators were in the habit of clearing error messages and continuing with treatment whenever they occurred." Seriously? This is medical equipment and people's lives are on the line; regular error messages _should not be a normal thing_ no matter how minor.
"... and the dangers of putting too much faith in software" *stares intensely at people's growing reliance on chatgpt*
I'm wondering what's going to happen to the human race with AGI.
Yeah let's not let ChatGPT program medical devices, thanks 😂
This is also something that should be a case study for why organizations shouldn't be allowed to investigate themselves
I have seen the plainly difficult videos about accidents concerning these Therac devices. Those videos were NOT about accidents while in use though. Now I get why these machines turned up in less developed countries. Most of the videos ive seen about this machine are about people getting hold of them by either having a contract to clear a building or basically breaking in to abandoned buildings and stripping scrap metal. I personally would be very very dubious about pulling apart machinery that gets bought out of a medical centre that treats cancer or uses nuclear medicines or centres that provide xray photos to the public for the reason of hospital treatment. It just goes to show how sloppy the registration and maintenance of these machines are handled . Now I realise how these machines made their way to these poorer more deprived areas of the world.
Wow, that's so true and unfortunate
I'm sure they didnt know what they were taking apart. You would be concerned because you know the purpose and what's inside.
Um, wow.
1. There are other radiation therapy devices besides the Therac-25. Not all radiotherapy orphan source incidents are related to this particular machine. It's like you watched a bunch of videos about random car accidents and decided they were all Ford Pintos because you also heard that Ford Pintos had a poor design that led to an unusually high frequency of fuel tank ingnition after crashing.
2. If Plainly Difficult did a video specifically on the Therac-25, it was not an orphan source video. Don't know if they did. I've definitely seen at least one other RUclipsr in the "nuclear disaster" or "general engineering accident" genre cover the incidents, though.
Goiânia incident in the '80s in Brazil (I think it had been covered on this channel before) -- but that wasn't a Therac-25, but a different radiotherapy machine. (Looked it up and it was a Cesapam F-3000)
@@Amanda-C. Yeah, from what I can gather, the Therac 25 did not have a radioactive radiation source in it. It produced an electron beam. A high power, high voltage one when used with the target to produce high energy x-rays.
My husband went through a horrible bought of cancer some years ago...I'm extremely grateful he never had to potentially face anything like this
the washing machine I'm using only shows numbers when error occurs, and I never bother to check manual when that happens. how ignorant those operators were for treating cancer patients worse then my laundry!
You explained a complicated technical subject very well. Good illustrations too.
I've actually heard of this machine before in the "Set Phasers on Stun" by Steven Casey. The first chapter covers the story of how this error was found and covers many instances where products bring up the question if there's design error or human error in their operation.
The absurdity that led to this tragic series of injuries reminds me a little of that time NASA lost a craft because one team was working in km and another in nautical miles. Nobody was injured, but it was a huge loss.
I work in the field of medical device commercialisation and a HUGE part of that is ensuring regulatory compliance for both hardware and software medical devices. The fault with the Therac appears to be a design flaw that was not picked up due to inadequate verification processes during design. Some proper risk management during the design process might have stopped this happening because it would have been noticed that using a single software programmer is inherently risky if there are no checks and verification steps.
As a software engineer I can tell this code had been written by an amateur or mid-level software developer at best. Not thinking about overflows when counting errors via incrementing a variable, is like building a water container and thinking it can never overflow with water. Such a problem is so mundane in embedded systems engineering, that you are likely to even encounter it in a recruitment test.
To anyone watching this who still wants to know more, I would suggest Kyle Hill here on RUclips. This is really just a dip in the pool of the Therac-25. Amounts of radiation given could have been up to 100 times the intended amount. Please, look into it more
Damn, this reminds me of the horrific scene in Final Destination 5, where the LASIK eye surgery machine malfunctions. It’s awful to think that something similar really happened to several people, but I’m glad that lessons have been learned, and that mechanical safety features and independent software review have been added for modern cancer treatment machines.
God that scene made me cry. It was horrific. Still can’t watch it again to this day. Trying to relate it to these poor living people doesn’t even comprehend
The guy who invented lasik no longer recommends anyone get it. It’s wild
I did it and I'm totally fine
@@bellezanegra0206Oh, why?
great, now i'm scared to have my eyes done
As an aspiring mechanical engineer, its really unfortunate and surprising that this happened. Your comment about AECL almost not understanding that there could be communication errors between the software and hardware feels correct, as crazy as it sounds. How could they put so much faith in the software otherwise? Really bizarre, but maybe it does come back to greedy and rushed decisions (as it usually does).
Not necessarily greed, often incompetence and a failure to understand how software works. The leads to inadequate testing because possible failure modes are not tested because they are assumed to be impossible.
Also, error messages should be understandable to the operators, e.g. error 54 is meaningless to most.
@@washingtonradio super agree!
Greed is a strong word, especially when I’m speculating on intent entirely.
Incompetence is much better thank you!
Ack, this had me saying "wtf?" out loud, several times throughout the video. The astonishing carelessness and hubris!
Thank you for all the content you have produced. A light you have shined on various topics most don't know that are important. It's crazy to think that it's been 4 years since the first vids 🥰 awesome work!