BS First only Enterprise PCs and Servers were affected unless someone bought Crowdstrike. Second, only a fraction of devices with Crowdstrike were affected. I work for a major nonprofit Hospital and about 25% of our servers were affected. I spent 11 hours manually remediating servers because in a large enterprise environment, that can be a large number. But get the freaking facts right. The reason this was so impactful is that it was large corporations affected. And the fix was a very manual process since Microsoft had a "feature" that put everything into "recovery mode" after two failed boots.
well at least if youre self employed (so you can demand a fuck this is my weekend surcharge)or live in any country with workers protections (aka not the usa or like india) you gonna get at leas weekend pays and overtime pay
Imagine having "I broke the planet" as a hold my beer anecdote whenever you and your colleagues start trying to one-up each other on times you screwed up at work.
Ha! You may have been behind the 2024 Crowdstrike update that made Windows machines boot loop, but I was behind the 2027 Windows update that bricked all online Windows 11 computers permanently! Try to one up that! *They then get one upped by the 2030 guy.
Beats my "oops" on a Linux box, "rm *.* -r" Worst part, I tested it on a DOS box first, made the shell script work, translated it to Linux command line, and... Ran it in the wrong location, as an admin (because we all ran under Admin credentials.) Thank God it was a test environment! 🤦♀️🤯
@@chinesesparrows we would be in 2142 with probably Nuclear Fusion, a Mars and Moon colony, new space stations, BFR, multiple cures for cancer, synthetic lifeforms, probably alien contact, and we would still be running 120+ years old windows code 😕
Do you mean the windows code that loads drivers? What do you think the catch block should when it reads a file full of empty bytes? ignoring the file would mean booting up without the security features the system was supposed to have, the fault is mostly at crowdStrike
@@omri9325 i agree with that, just that pretty much any 3rd party driver can cause BSOD bootloop is a massive vulnerability. I dont know maybe MS could add a group policy that adds a standard flow where windows creates "driver backup point" before any new driver updates, if after driver update repeated BSOD revert to driver backup point and contain the problem driver with alert and log.
@@chinesesparrows No? It's a driver with kernel access. This makes the driver very powerful but also makes crashes easier, due to the fact that the OS cannot babysit it. I'm pretty sure you can cause a kernel panic on linux by writing a bad driver as well
YES IT "WAS SABOTAGE" YOU ARE SPOT ON. PEOPLE ARE SO EASILY FOOLED, IT'S INCREDIBLE. I KNEW IT IN MY GUT. NO ONE CAN CONVINCE ME THAT IT WASN'T SABOTAGE. OH, EXCUSE MY CAPS PLEASE. I CAN'T SEE THE SMALL LETTERS WELL ENOUGH TO USE THEM. GOD BLESS YOU FOR THE TRUTH
@@benheidemann3836you can google "red hat crowdstrike", it states that the driver works in kernel mode and user mode
Месяц назад+97
They failed to do a smoketest of their agent after build but before deploying it worldwide. Sounds like their software and update development process is just really not up to professional software engineering standards. At Meta, we had to have other engineers, sometimes multiple, review diffs before they would be accepted. And then there were multiple layers of CI/CD testing before exponential deployment with canary testing. You don't just push new code to all the machines all at once, because it's way too dangerous.
things go wrong even after full in-house testing. that's why you have canary testing and phased roll outs. the ceo comes across as totally disingenuous and is ultimately the reason for the company's poor practices.
Crowdstrike is ransomware, they just have a different payment plan. You pay up front for the privilege of being ransomwared at some unknown point in the future. Turns out the unknown point in the future was today! Surprise!
The other problem is not just bitlocker, but if your company locked the microsoft account so you can only sign in from company devices to get the recovery key, but all your company devices are hosed. I can sign into the account from my personal linux box, but it just says it's restricted, and I can't do anything, or get the recovery key. I was on hold with IT support for hours today, and at one point, the phone system hung up on everyone, and calling back the number went to a busy signal for the next hour. I spent 3 more hours on hold with IT, and they either answered when I went to get something to eat, or hung up on me again. What a cluster F
And don’t forget that Crowdstrike was responsible for the 2016 story that Russia hacked the DNC in order to get dirt on Hillary and favor Trump. Which was a lie.
The problem with the bootable USB thing is that a lot of corporate devices block booting from USB by default, which means the IT Team would have to tell the end user the BIOS password to get into the BIOS to change the boot order to enable booting from USB. Its a total nightmare!
Yeah, as a former IT support bod in logistics, taking a user through the steps will be painful and in some cases practically impossible. Can you get into safe mode please, power on the computer, wait eight seconds, hold power to turn off. Repeat again. Err, it's not coming on again. Oh, it's is now, oh god blue screen again. We need to start again. Warehouse bod, eff this you'll have to come over and do it. That's before you even get to bit locker and talking though command line. Lots of journeys will be happening this weekend.
Crowdstrike says "hey, they didnt respond like this during covid/bitcoin. Which was also highjacked, but nobody said anything and everyone joined in for the fun. Lmao It may not have even been on purpose
Apparently an idiot dereferencing a null pointer and another senior idiot pushing it to production past code review is now a “zero day” No it’s just a really obvious bug. There was no hacker here unless the bad code was intentionally put there by the employees in order to sabotage the company. So if there’s any hack, it’s internal.
@@Jimothy-723 what , criminal negligence , last week there was that event and the person in command got no punishment nor lose their job for that absurd amount of negligence.
the fact that one company can take everything down like this is scary, one bad actor and this could've been a mass malware attack instead of a simple driver error
@@blakenolingberg1556 that was the mistake. Don't trust anything running on the ring0 that's third party, except drivers made by the vendor. No toy software or rootkit is allowed.
Capitalism market innovation... this is freedom of choice... The truth is corporate American hates competition, and capitalism always produces monopoly.
I literally spent my entire friday manually fixing computers and explaining to people at remote locations how to fix their computers. Our entire IT department became helpdesk because of this update. You don't know the pain of explaining to a non-tech person over the phone - how to make a bootable USB, boot to it, and then enter their bitlocker recovery key so they can delete a file via command prompt until you've done it personally. I got to do that dozens of times on Friday and theres going to be lots more of this for the foreseeable future... I cannot express how much this sucks to fix even though its a relatively simple fix. It just can't be automated and it's horrible for that reason.
I mean, Windows might be the least secure how most people use it, but there's another huge facet to why it's the target of randsomware: it's absolutely dominates the end-user/workstation market, especially when you are wagering the victim can't just restore from a backup and ignore you.
It's purposely written like swiss cheese, full of back doors and vulnerabilities so they and intelligence agencies can access your computer any time they want. Linux is not. So, Linux users don't have to worry about viruses or malware and don't have to put middle-man software between them and their machines to protect them from their malware/spyware OS. The only thing I use on my Linux servers is a firewall and Fail2Ban to prevent brute force pwd cracking.
@@Texas3Percenter This specific issue is not exclusive to Windows. This is an issue of operating systems architecture and how drivers are able to run in Kernel mode. Falcon have Kernel driver for Linux, Windows and macOS. The IT departments of companies are asking the OS to load Falcon driver an allow it to run in Kernel mode, allowing it to watch for user data but also enabling it get OTA updates and not able to choose of get or not the update. Same bad driver could be for Linux or macOS.
@@unaquetzadillaalso, macOS still needs antivirus/endpoint protection. It just works a bit different and the audience of people who get MacBooks at work is probably more technical
Just what I was thinking. The reason Windows is targeted more is because it’s sitting on 73% of desktops. The PC Security channel showed that malware vendors absolutely have Linux versions of their tools. It is not immune.
@@unaquetzadillaWhat you said is partly true. In my previous role, I deployed crowdstrike for a major broadcaster and one common misconception in all of this, is that crowdstrike can push updates to customer endpoints without their knowledge or consent. It's simply not true. Endpoint management is handled centrally by IT admin and we can choose if we want to use the latest Falcon sensor version or not. You can of course configure crowdstrike to auto update the sensors but that would be ludacris.
Their asses are already opened up to lawsuits big time! 3 billion in cash $ watch it evaporate! Their customers are now PUBLIC DATA which is a huge liability for their future cash flows too.
I literally just turned down an offer from Crowdstrike two weeks ago in favor of another job offer…it was a tough decision to make at the time but now it’s definitely looking like I made the right decision! 😬
Where in the hell is the testing cluster they should have deployed to first? CrowdStrike should deploy their Falcon updates to all their own machines and if they don’t BSOD after a week THEN release to the entire galactic empire.
Lets accept for a second that they did that, because they probably did, they've been doing this for a while. They probably did NOT send out a null file. So somewhere between them releasing it and the end users getting it, it got nulled out. That's where the problem was. Azure went down right before all this happened...
This is tough because as an antivirus you want updates deployed as quickly as possible. If some new exploit comes out you wouldn't want your customers to be vulnerable to it for over a week.
If they tested, then they either didn’t test what they were deploying or they didn’t deploy what they tested. This is a basic control problem. I have had to point out to management numerous times that whatever cockamamie plan they have for maintaining a claim that the product was tested, that they weren’t testing what they were deploying. That internal process needs to be seriously scrutinized.
@@piquat1 i cannot believe they were smart enough to do a phased roll out, but then neglect a simple hash check of the deploy-able. the evidence suggests they did neither.
@@petargolubovic5300 Agreed Mac booted Anti-,virus software from the kernel after creating an endpoint security API for them to use and Linux has ebpf hooks and bpf programs to screen for AV to screen for potentially malicious activity with a guarantee that they can't crash the kernel (bpf programs have strict security/stability guarantees while being non-turing complete)
Honestly, when I was an i.t. call center tech, the older people could often be counted on to listen to my instructions and not go off script. It depended on the person of course, but I was often able to wrangle an older person to listen. 😅
Crowdstrike is the company that shuts down your computer when you're hacked, and they wont allow it to turn on until they check manually how bad is the hack...
We don’t shut it down, we hit a button labelled “network containment” which only allows it to communicate with the admin dashboard for forensic analysis. Your MSSP should be ringing you immediately if they have to use network containment
I was dealing with this today as an IT tech. Oh boy, What a joke. Took around 20 minutes on average per person affected. Since hundreds of devices were affected it made for a long day... :(
Yeah, I'm one of those tech guys. I'm in charge of our enterprise's cloud infrastructure (which is all our servers). I was up till 2am restoring a couple servers affected on our European side, thinking it was some weird Win update that took things down. I went to bed and was woken up 3 hours later by my boss freaking out. I spent all morning force shutting down systems, detaching and attaching drives to working systems to remove this .sys file and all. What a HUGE pain. I finally got everything working after like 5 hours of doing this crap nonstop. The poor helpdesk was stuck doing bitlocker based safe mode fixes for end users. I don't envy them...
Why updates are not tested in quarantine environment by businesses themselves? You should have mini cloud for that. Seems like everyone blindly relies on Microsoft and their partners.
Absolutely insane that one file can cause that much havoc. The fragility of all aspects of life as we know it... Mind-blowing. Hope you get some rest 💙
@@egria We do test them a day before on some servers we don't care much about and can restore in minutes from backups. We're also a tiny shop of 2 IT guys and a contractor, and I'm the only one qualified to do the cloud admin stuff. The Win update thought did cross my mind, but it was 2am and I was burned out. I just restored them and called them a night. Little did I know it was a global shitstorm from our endpoint solution provider.
My favourite part of the disappearing air traffic example is that while they will occasionally get crippling downtime from their infrastructure, Southwest still running primarily Windows 3.1 with a sprinkling of Windows 95 here and there rather isolated them from the CrowdStrike issue.
Our company installed CS on thousands of Windows clients recently. A few weeks ago they uninstalled it because it was causing massive system performance issues. Giant bullet dodged. On the Windows side, it's just nuts that any driver causing continual stop errors is not auto disabled /quarantined by the OS.
The driver affects the boot process which is why it wouldn’t fail until a restart. Cant auto detect a failing driver before that driver gets used by the system.
Most drivers do not, but generally AVs mark theirs as a boot driver, so the system cannot boot if it is failing. So it's not just "any driver" as you state.
"Windows is the only OS that is insecure enough to have problems like this" Let me tell you why that's bullshit - Crowdstrike did this to our production Linux fleet back on April 19th.
Linux versions of Falcon also hook into the kernel. Talking about how this fuck is somehow because it is Windows is disingenuous. Why can't we just blame Crowdstrike and just Crowdstrike instead of bringing up and blaming Microsoft?
i wholly agree with you, this guy just wanted to hate just because its windows, i also don't like windows myself but its pretty disingenuous to blame Windows when its not their fault.
@@Abaddon231 falcon also exists on Linux and Mac. Is Falcon there because of Windows? Falcon exists because there are malicious actors, not because Windows exists. Then he mentions kernel drivers without acknowledging that this also happens on Linux and the fact that if you want a solution like Falcon, they should hook into the kernel, because that is how they can most effectively do their job. Or why bother having and EDR at all. He would have been able to effectively convey his message instead he went all mainstream media ranting about Windows. This shit isn't his forte, but he still has a take on it. Low Level Learning has a much better and nuanced take on this.
Falcon on Linux uses ebpf hooks which can't crash the kernel (they have extraordinarily strict guarantees, restrictions, and limitations that would prevent an ebpf program from doing that, they aren't even turing complete) Apple on the other hand has an API for AVs to do their job and doesn't permit them from installing a driver.
@@brandonn.1275 Falcon on Linux in User Mode uses eBPF. There are still many systems that use Kernel mode. Although Falcon is transitioning to User Mode, Kernel Mode is still officially the default mode for Falcon on Linux. Per official CrowdStrike docs. Many of my Linux systems still use Kernel Mode.
Root cause analysis - 1) Using Windows for mission critical work in 2024 2) Terrible code that somehow made it past code review 3) Build system that corrupts files 4) No validation checks prior to rollouts 5) Rollouts to the entire install base rather than a staged rollout
It's WILD to me that so many big companies have built critical business infrastructure around Windows. I do it for my little piddly business but I'm aware of the shortcomings!
@@Jimothy-723 1. DEI isn't a word, it's an acronym. 2. These kind of issues occurred long before 'DEI' was a thing, so how do you explain those issues if this is obviously because of DEI? I'll wait.
@@user-uo5eu6yk1b DEI is just an acronym for affirmative action. It's always been a thing in the internet age. Sorry that you're wrong. It's more prevalent today since there are few qualified white males willing to work for crap wages.
Thats why they call IT. The amount of people who do not know how to log in to their work email on their phones so I dont have to read out their 69000 digit recovery key is incredible... and then hope to god that they wrote it down correctly or heard you correctly... and then hope to god that they enter it correctly so that 75 year old Dorothy doesn't have to be talked through booting into safe mode again BEFORE you have to talk her through deleting a file.
So, there was this episode of _The Munsters_ that followed the same idea... They wound up blowing up a very expensive antique box to find a video cassette (and not a heap of treasure).
There was a post on reddit about a huge org that has the bitlocker keys on a box with bitlocker and they don't know where the key for the centralized box is because the documentation is also behind bitlocker
I did that by mistake once, I stored by password database on the bit locker drive, the bit locker password was on the password database. Lucky I have a offline backup of the password database.
@@tatumsh9 Or maybe the company shouldn't limit the microsoft account to only be accessed from company devices, when the only company device is hosed! Sure, don't allow access to everything, but they could allow access to the damn recovery key. Also if I have the company email setup on the phone, then it creates a huge nightmare in outlook making me authenticate on the phone every time to use outlook on the pc - I discovered by accident (when the company "mistakenly" removed email from everyone's phone) that not having it on the phone solved all the auth issues on the PC too, so I just left it off the phone. I don't need company email on my phone 24/7 - company email is only for during work time on the PC
its shockingly inept. the fact they did a rollout with no percentage based rollout and metrics on how the rollout was performing and no rollback plan is literally insane and then on top of that the fact their deploy pipeline distributed this with tests either not run or failed.... makes it seem like theres more to this story than just oh we just pushed some bad code. There are safeguards in code and policy that should prevent this.
also the fact it's literally *in* the kernel and not something like ebpf where it can have hooks in the kernel and have saftey guarantees about it not crashing the computer... thats microsoft fault for writing a steaming pile of bitrot masquerading as an operating system.
3:40 Not true. Falcon sensor is available for Mac and for Linux too. The real reason that it has happened on Windows only, because 1.: CrowdStrike seemingly made a mistake only in the Windows driver, 2.: Windows is waaaay more popular in businneses (both server and desktop side) than anything else.
There was an issue with CrowdStrike on Debian several months back, that caused the OS to not boot… This isn’t the first time CrowdStrike has massively broken an operating system
One point to make, (I am a Mac user so don’t come at me 😂) Microsoft has the biggest market share for desktop by a large margin so make sense for hackers to focus on them, not sayings it shouldn’t be more secure but also theywill get bigger focus from hackers just due to market share
Linux runs most of the world's servers though. If Linux is much more vulnerable than Windows it would make more sense to focus on Linux since you can have more control. The thing is Windows simply is a lot less secure and more prone to crashes like this.
@@haroldcruz8550 you know, it is much harder to trick user to install malware on server, than it is to make user to launch it on own PC. Which is, coincidentally, predominantly Windows.
@@haroldcruz8550Linux flaws are invisible to the general public and end-user Linux enthusiasts. That said, there are many well known attacks on servers.
its insane that crowdstrike didnt integration test the update, and even more insane that mission-critical infrastructure is OK with automatic OTA patches.
One thing though, using a kernel mode driver is not exclusive to crowdstrike. Many other AV/EDR systems use drivers as well. In fact a very similar thing happened with Symantec a while ago.
At this point windows is going to need to boot Anti- virus software out of the kernel and provide an API for AVs to do their job instead of having them insert a driver into the kernel. This is what Mac did when they booted AV vendors from the kernel after publishing an endpoint security API for them to use. Linux has something similar in the form of ebpf hooks and bpf programs that can run in Kernelspace while being guaranteed to be unable to crash the kernel.
iirc even linux security software sometimes needs kernel mode drivers. stuff like cloudstrike can avoid needing a driver on linux bc they can just use ebpf tho. mac still needs a kernel driver.
Yeah the affected CS update just happened to target a Windows named pipe vulnerability, so in this case only Windows was updated. So to just blanket blaming using Windows like OP did is a bit lazy. Their Apple and Linux customers just got lucky.
@@JSmith73 macbrained webdevs can't see the world as it actually is. They're used to rewriting their entire app (which is just plumbing between AWS, databases and APIs) in the latest JS framework every 3 months. They don't realize that most legacy vendor software which is what the world actually runs on is a piece of crap and was written by a team of 10 contractors over a few months for a specific pentium windows xp machine in 2005 and has become a mountain of hacks and patchwork because management doesn't want to spend money on improving it.
@@dead-claudia Mac doesn't allow AVs into the kernel anymore instead they published an API for them to use and won't allow AVs to install kernel extensions anymore. In fact kernel extensions have been deprecated for a while now and only userspace system extensions can be installed.
this seems like one of those applications where you'd expect every pull requests to go through a "committee" such that you don't have some one-guy write a bug into the code..
Or malware. Seriously, if this could happen due to some stupid mistake, imagine how bad it would have been if an actual bad actor had social engineered their way into position to abuse the hell out of it.
“Pretty much every PC in the world”… What? No. Nobody’s home computer would have CrowdStrike software. No Mac or Unix or Linux systems would be affected. The number of business computers running Windows with this software loaded is absolutely mind-boggling, though. I mean, we’re in seriously WTF territory. I hope people can get things straightened out quickly, especially for the more critical areas.
If you search hard enough, you'll find that there were two Linux Distros that got a bad update from CrowdStrike that resulted in the same issue. I think I read that it was cases of kernel panics in this case.
After using Linux for 10 years and Arch for 5 years I will say... use Mac instead. If you need something linux or windows specific you can spin up a vm with Parallels. And overall, mac laptops lasts long and have overall great performance & build quality. As for servers - linux is the only choice.
@@dyto2287Until your display cable is too short and rips when you open it and apple won't cover it under warranty... Not to mention the throttling that occurs because they don't put fans in their laptops. Hope you didn't need performance...
Wow, fortune 50 company I used to work for had all the users on windows, of course, the back end for the most critical things ran on RHEL. Wonder how they're doing now. lol
I don't quite get the bit about most antivirus not using drivers. Drivers have been an important part of AV and other security software going back to the Windows 95 days, I worked for a company doing security back in those days and we had a file system driver that was the core component for the file system security portion of the product, and a keyboard driver that was part of the system that ensured commands given to our system were coming from the interactive user and not from some script or application. Drivers are so important as they are outside of Windows. A normal application, even if running as a privileged user, cannot kill or modify the driver, nor can it bypass the driver when talking to the devices controlled by the driver.
Not sure what is scarier, that CrowdStroke released a bad version or that so many companies just blindly went with it without testing and staging it first.
Airlines, banks etc. without testing updates in isolated environments is absurd yet reality. And most systems if done properly don't even need antivirus because they suppose to be nit connected to public networks. So this would be some management's push to order that software. Important systemd should be setup in a way that assumes that something goes wrong that means having staging environment. This incident shows massive tech incompetence either by itself or with push if higher management wanting to reduce cost or just fall on lies of vendors of how great everything would be if company trust them blindly.
Friday morning at the office I was jokingly asked if I caused it. The day before in the company Town Hall I announced that I cancelled the CrowdStike contract and we have mostly removed Falcon from all devices. Only two left over machines had issues.
@@Micloren basic Falcon didn’t do much and got better data from the Checkpoint and Fortigate UTMs. Unless you shell out for the full SIEM, felt limited. Decided to up the network security instead and pay for a SOC/NOC service.
OMFG, Theo thank you I had no idea how terrible this was!! The encrypted bitlocker problem is absolutely horrendous! Oh my god, I could maybe get this sorted but most people definitely can't, this is BAD!
WOW. I'm a developer currently on vacation, and I'll be back to my job on Monday. My computer has been off for the past two weeks so I guess I'm lucky? Assuming right now they are no longer shipping the bad update and I can safely turn on my PC, but will definitely make sure next week!
At first glance this is actually hilarious to see, but I feel so bad for the patients at hospitals affected by this. That's the worst part. Sometime like this will literally cost lives. Crazy
And Hammond gets a shoutout here as well 😎. Also, great video of course, really enjoyed the take on how to and how not to communicate in such a situation.
Every single time something really shitty like this happens.. almost without fail. EVERY SINGLE TIME: Look at the education of the CEO. George Kurtz: Degree in Accounting. (no formal science, tech, engineering, or computer education of any kind... I can see he claims he can program but as far as I can tell, he's never actually worked a job involving programming or science of any kind.) The fact that this company pushed a driver rollout to hundreds of millions of people SIMULTANEOUSLY without checking it worked first tells you EVERYTHING you need to know about how this dude runs this company. If he had any actual knowledge of computer systems he wouldn't have allowed that to happen. And yet he did. When will the world wake up and start to realize that shit like this always happens when you put business people in charge of technology they don't comprehend. Now I'm not saying George Kurtz knows nothing about programming, its completely fair to be self-taught. But as far as I can tell he never did anything science-related at his job in any capacity, and his claim to fame is that he co-wrote a book about computer hacking with some actual computer scientists back in the 90s. Ever since then everyone has treated him as though he himself is a computer scientist, even though he's not actually. And after a #$#@ up this extraordinarily bad, it seems it was wrong to believe he knew what he's doing. There's NO WAY a mistake this bad could have happened without his express knowledge and instruction that this is how they operate. Stop trusting bean counters with important technology.
3:34 - People always think hackers target Windows because it is insecure but that's actually not true. Microsoft Windows is the most used at 72.22%, followed by Apple's macOS at 14.73%, desktop Linux at 3.88%. Just by looking at this, one can easily deduce what operating system a sensible hacker would target if they wanted to create malware. So, its not that the other Operating Systems are secure but a matter of ROI. If you spend a month creating malware for windows you get 72% possible targets while on the other hand spending a month creating macOS malware will give you only 14.73% possible targets.
It's purposely written like swiss cheese, full of back doors and vulnerabilities so they and intelligence agencies can access your computer any time they want. Linux is not. So Linux users don't have to worry about viruses or malware and don't have to put middle-man software between them and their machines to protect them from their malware/spyware OS. The only thing I use on my Linux servers is a firewall and Fail2Ban to prevent brute force pwd cracking.
@t3dotgg Your description of the Windows / Active Directory / BitLocker login process is inaccurate. The bitlocker key is not retrieved from AD or other remote DB when you auth, but rather from the device’s local TPM.
Every time I hear about this it starts saying (pretty much every PC in the world was affected). I never heard about CrowdStrike before. Every person I know is unaffected and I didn't hear about any company or service affected where I live. Anyway, good luck guys.
I think it primarily affected enterprise clients, and that's what he should have led with. But I guess he couldn't resist a little Microsoft slander 😂😂
Ooooh yeah I love the voice that tech influencers have when they weren't expecting to go recording and their humanness comes out more??? Like, idk what it is about it, it sounds like you just woke up (compliment)
@NormCantoral that makes sense, I just thought that interpretation was so unreasonable that it was more likely he just hasn't ever seen a software bug of this scale and severity that was caused by something so easily preventable. made sense to me bc he runs a tech channel.
Saying the fix has been deployed without that being technically possible is such a stupid move that likely created a lot of additional stress. If a non technical boss of a company reads this, he might blame the IT team like "crowedstrike deployed the fix 5 hours ago, why can ours hundreds of employees still not work?"
I don't understand why everyone is so caught up over this "kernel level driver" thing - this is not built for consumer PC's. EDR solutions REQUIRE kernel level access to even be effective at catching as much malicious software as possible - it gives you such an upper hand and allows you to check and scan EVERYTHING on the machine. For a consumer non-business user device, this is super undesirable and would not be a good solution - but for a business that requires intense security to protect their data? At least for the moment, there is no other way. Those saying that Windows is the only OS unsecure enough to need this and jabbing at windows... Yeah. Just Windows things. I mean it's not like Linux and MacOS are perfectly secure, but the general consensus is those OS's are more resilient to viruses than windows. (Though it's not a bad thing to point out, that most malware is written for windows, because of how ubiquitous it is in industry.)
No, they do not require kernel level access. In fact they are not allowed to have kernel access on Macs and even Linux is moving over to ePBF where a bad driver can't crash the system.
@@uzlonewolfkernel drivers are allowed in macs. and linux kernel drivers are limited in a number of ways. notably, a kernel module is needed to monitor syscalls for processes you didn't spawn. and seccomp filters don't let you count anything, only filter.
@@uzlonewolf Because of the way that Windows is built and the way that threats/malware currently operate, the only way you can hope to catch everything, at least right now with current technology, ideas, and software, is with kernel level access. Hopefully this changes, but as of right now it is what we have. I am only applying this to Windows however because, obviously, we've found alternate solutions for Mac and Linux and have not needed to do this.
Hah! That USB boot idea would be fine, if it weren't booting you into safe mode after unlocking your bitlocker. That's prime attack territory for planting a rootkit while "helping" you clean up that crowdstrike BSOD. You'd have to audit everything about how that USB key came to be and all the software on there, from extremely trustworthy sources.
Companies that run something like CrowdStrike often use BitLocker AND have take measures to block USB devices. You have to lock a Windows down way more to be ‘compliant’ for auditing
Why does a badly written driver stop the machine from booting? Shouldn’t that driver just be skipped and whatever device it targets not work? Seems like a terrible design within Windows.
It is not Windows design issue. It is driver being written as kernel-mode, to gain extensive privileges over system in attempt to prevent malware activity. Meaning it becomes required to boot before even Windows takes over. And if it crashes, well, Windows hadn't booted yet. And it cannot exclude this driver from list as it is listed as mandatory. Loop repeats. And no revovery window appears because system crashes before it even reaches said state. Someone even said that CrowdStrike already did similar stuff to Linux systems at 19-th April. Linux and Macs versions of CrowdStrike are also using kernel mode drivers, albeit with some nuances (like there being restricted version for Linux). And similar case could've also caused bootloop (on Linux BSOD is more known as Kernel Panic).
@@DimkaTsv This is a Windows design issue. Try to load exact same file as a Linux module. It will not crash, but You will have a simple warning and that's it.
the prob is it had to be done as a driver to get kernel mode. MS tried to develop and license an interface that would allow that without having to load potentially sketchy kernel-mode drivers before boot, but then the EU stepped in and forbade it, because, they figured, it would give MS an unfair advantage on the market, if they were to choose who would get costly privileged kernel access. So you can blame the EU as well. Or you can blame Crowdstrike for knowing they are doing very, very sensitive stuff, but OTA-pushing a bad, untested update regardless.
The fact their stock price barely took a dent is insane. This is the kind of mistake I'd expect from a solo startup with 30 users, not a multi billion dollar corporation!
1. it takes more than a day for such issues to result in stock price drops. 2. cloudstrike fixed it same-day. it's just the damage done was so severe, it's taking customers a disproportionate amount of time to fix.
That sucks! I have to close on 2 real estate deals Monday. They won't happen til this is fixed. Thanks, banks, for using crowdstrike! If you were using Linux instead of Windows, you wouldn't need to.
Microsoft outsourced some of their customer support overseas. In the Philippines, they train their agents about BitLocker *NOT* an MS thing, but rather a 3rd party like a MOBO. That baffles me since it doesn't exists or required in Linux or Mac systems.
yep, especially since they're publicly traded. privately owned companies can get away with more, but only bc there's a lot fewer people who could have standing to sue over that.
This situation can happen again if we do not fix this broken system: - Stop using Windows only, if you buy multiple OS types it can be more work but it is the only way, even if it is not Windows it is better. - We cannot have deploys without lots of testing pipelines, I am sure that they did not test it, if they test the tests are really bad. - We cannot buy a PC that has forced updates, even if it is not fully secure for some hours the users need more control over it. Even if we keep using force updates they need to have some stages to pass first, for example, let's test a small number of users first, then scale up in a controlled way.
While I hate Microsoft about as much as Apple nowadays, your take on Windows in this specific case is totally wrong and uncalled for. Only Windows was affected because only that update was affected. Windows doesn't have anything worse than Mac or Linux here. Also Mac and Linux could've been just as much affected. That whole section is very cringe. Edit: I mean the section starting at 3:35 The rest of the video is a-ok
Yeah, this could’ve happen to any of the OSs especially in a corporate setting. That said, Apple actively discourages kernel extensions and have built alternatives in user mode.
In this particular case I *suspect* Linux would handle a faulty kernel module bettter, but I'm not sure. This is crowdstrikes fault for sure, but I wonder if the way Windows handles these kernel level drivers could be better though.
@@fallingintime But was it as catastrophic as a unbootable BSOD ? Edit: Although it would probably be hard to compare unless it was at least a similar mistake.
@@jonnyso1 kernel panic that required a kernel fix I believe. Probably wasn't widespread as it only affected a specific kernel version and kernel updates are not ota
The take on windows being the least secure OS is a bit biased. The real reason why most cyber attacks happen through windows is because 1. It is the most used OS by a huge margin. 2. Since it is so widely used, hackers focus on finding vulnerabilities in windows instead of other OS like Linux.
It's purposely written like swiss cheese, full of back doors and vulnerabilities so they and intelligence agencies and access you computer any time they want. Linux is not. So Linux users don't have to worry about viruses or malware and don't have to put middle-man software between them and their machines to protect them from their malware/spyware OS. The only thing I use on my Linux servers is a firewall and Fail2Ban to prevent brute force pwd cracking.
My rage at everyone downplaying this for CrowdStrike is immeasurable. This is a billion dollar company, with a B, trusted by critical government, public, and private services and they shafted each and everyone. The lack of outrage from our authorities is absolutely disgusting. Speaks a lot to the state of cybersecurity and tech in general
i personally think that while the emergency is occurring i don't want the CEO wasting time crafting apologies. Their goal should be dispensing accurate actionable information to help people recover. apologies can be crafted after.
This is why a mega level cyberattack is so scary. Too many companies rely on the same systems to function (ABS, Google, Crowdstrike) so to see something like this happen it just shows how vulnerable our networks are.
I don't get it - does this incredibly impactful product not roll out to testing environments/deployments?? aside from just unintentionally shipping a file of null and not apologizing, what if a rogue employee pushed some badware into the product? apparently you would only know when it's already deployed 'in production' in the most important enterprises in the world at a kernel level.
I can't imagine the infrastructure that could let this happen. I have worked at a lot of companies, and even ones that haven't always had the best practices. But every single one of them had measures in place that would have caught this problem in at least 5 places before a customer got it.
Yea, being locked out by a faulty update of a security thingy dodat! JUST one question: who will pay for all the hours of work for this faulty update? Are lawyers already drafting suits? 😦🤑
So, I tried to fix this on my wife's work laptop myself. I got into safe mode, entered the bitlocker key and... I needed an admin account to access the folder and delete the driver. 😮💨
You really lost me on this one theo. There isn't anything here that couldn't happen on Linux, in fact, it did -- about a month ago. It's not guarding your bootup and preventing you from booting in an unsafe environment, its just a broken kernel mode driver. I got bluescreened. You know what happened? Windows dumped it's ram, rebooted into last know good config, i shrugged, and moved on. I can't believe this overshadowed Trump being shot to the point i hadnt heard about the latter until an hour ago
@@mwahlert "omg Windows"... No... Omg the market share! Bet you didn't know Windows ran in all *these* places :3. "Omg Windows needs this?" Up until 2019 CS didn't even have any offensive antimalware tech, it purely was marketed as a distributed, crowd sourced IDS for setting a baseline and then detecting anamolies (whether abnormal for your business, or just plain university unusual activity) "Omg a driver?" Yeah .. if you want network inspection (NDIS) or any kind of endpoint DLP, and unless you want your employees ripping it out, changing it's permissions, etc. As someone inconvenience by it daily since i run executables classified as "hacking tools", CS does an amazing job of preventing bad stuff from running, no matter who you are. First CS issue I've ever faced the wasn't as simple as requesting an exception for a given piece of software.
I mean this happened last night and trump got shot 6 days ago.. living under a rock sounds like a choice also windows is the hodgepodge of configurations and programs and apis he said it is. Everything is an inconsistent mess and it seems reasonable to think that would leave more security issues..
I wouldn't say this overshadowed the Trump shooting, that's just a reflection of the content you interact with, there's been tons of coverage over the last week regarding Trump
Crowdstrike, now probably the most hated company by end users and IT folks globally. Customers will start to make plans to exit from their offerings and consider other options.
I am a support partner for Microsoft. To resolve this issue, boot into safe mode or recovery environment and then go to the C:\Windows\System32\drivers\CrowdStrike directory and delete the “C-00000291*.sys.” file. Then restart the system in normal mode. That should fix the issue.
@@malvoliosf even if you han encryption enabled, if you have the recovery key. The recovery key can be obtained from your Microsoft account. Enter it to get into the recovery console. Once you do that, get into safe mode and delete the crowdstrke.sys file.
I’m happy to say that none, zero, zilch of the servers I look after were impacted by this. Because last year when there was a corporate push to install CrowdStrike on them, I pushed back. I was and remain highly resistant to giving any 3rd party company the ability to install software on my systems at their whim/discretion, for exactly the reason we saw this past week. There are other methods of monitoring your systems that don’t require that you give away the keys to your infrastructure.
It's literally the FBI, rofl. They are the same company that led the DNC server and email investigations from the last 2 elections and helped conclude it was Russians working for Trump (server) and clean up all those pesky emails proving the DNC aided Hillary in harassing Bernie out of the election. You liberals, man.
It was a memory safety issue. It looks to me like a developer in India did compile using an old version of c++ and deployed to prod before testing . The compiler didn’t identify the null pointer dereference.
DUDE!!! I managed to use a combination of your gudance and a "shortcut" and I fixed my workstation. You're awesome!!!! My company doesn't allow access to the folder so I had to shortcut my email to my personal machine to get temp admin access. Other than that, followed the instructions, and viola, I'm in. Thanks.
Sorry about the frame rate issues, CrowdStrike took down my main recording rig and I had to do this on my Mac :(
TO BE CLEAR THIS IS A JOKE. My recording rig does run windows though… 🙃🙃🙃
BS First only Enterprise PCs and Servers were affected unless someone bought Crowdstrike. Second, only a fraction of devices with Crowdstrike were affected. I work for a major nonprofit Hospital and about 25% of our servers were affected. I spent 11 hours manually remediating servers because in a large enterprise environment, that can be a large number. But get the freaking facts right. The reason this was so impactful is that it was large corporations affected. And the fix was a very manual process since Microsoft had a "feature" that put everything into "recovery mode" after two failed boots.
@@timk9847What part of “to be clear this is a joke” did you not get? 💀
The whole thing
@@timk9847 everyone who installed this spyware willingly had it coming
This is the best named company in history. This is the exact same outcome as if the entire crowd went on strike.
The entire _IT Crowd_ ...?
There was that episode where they borrowed (then dropped) *the internet.*
@@morsemurraidh1314 Bet the CrowdStrike CEO found some "irregularities in the pension fund" today
Would make a good movie title too.
Exactly! The company name sounds like the name of an exploit!
automatic update is just stupid
To everyone cleaning up this mess: my condolences, may your weekend rest in peace.
Their weekends will probably be full of people complaining and PC boot looping
it's log4j all over again except it's actually crashing everything instead of hypothetically being possible to turn into a crash
😂😂😂😂
well at least if youre self employed (so you can demand a fuck this is my weekend surcharge)or live in any country with workers protections (aka not the usa or like india) you gonna get at leas weekend pays and overtime pay
Thank god that thing never happens to me because Thanks by my PC never get windows update.
When I got multiple calls at 2AM I knew this was going to go down as one of the worst days in recent IT history.
Aouchh
Thank god that thing never happens to me because Thanks by my PC never get windows update.
@@C.A._OldYes, keep sending more dumb replies, because in the end, you don't know anything.
@@C.A._Oldthis never would have happened to you because there's absolutely no reason for you to have crowdstrike
@@goaliedude32 doesn’t windows as a company have crowdstrike?
Imagine having "I broke the planet" as a hold my beer anecdote whenever you and your colleagues start trying to one-up each other on times you screwed up at work.
Ha! You may have been behind the 2024 Crowdstrike update that made Windows machines boot loop, but I was behind the 2027 Windows update that bricked all online Windows 11 computers permanently! Try to one up that!
*They then get one upped by the 2030 guy.
Beats my "oops" on a Linux box, "rm *.* -r"
Worst part, I tested it on a DOS box first, made the shell script work, translated it to Linux command line, and... Ran it in the wrong location, as an admin (because we all ran under Admin credentials.)
Thank God it was a test environment! 🤦♀️🤯
The largest disruption in human history caused by a missing try/catch block
Seriously windows acts like it has nothing to do with them but any driver issue can result in BSOD bootloop is ridiculous
@@chinesesparrows we would be in 2142 with probably Nuclear Fusion, a Mars and Moon colony, new space stations, BFR, multiple cures for cancer, synthetic lifeforms, probably alien contact, and we would still be running 120+ years old windows code 😕
Do you mean the windows code that loads drivers? What do you think the catch block should when it reads a file full of empty bytes? ignoring the file would mean booting up without the security features the system was supposed to have, the fault is mostly at crowdStrike
@@omri9325 i agree with that, just that pretty much any 3rd party driver can cause BSOD bootloop is a massive vulnerability. I dont know maybe MS could add a group policy that adds a standard flow where windows creates "driver backup point" before any new driver updates, if after driver update repeated BSOD revert to driver backup point and contain the problem driver with alert and log.
@@chinesesparrows No? It's a driver with kernel access. This makes the driver very powerful but also makes crashes easier, due to the fact that the OS cannot babysit it. I'm pretty sure you can cause a kernel panic on linux by writing a bad driver as well
Sending out a sys file filled with nulls looks to me like sabotage
or a failure of automation
sending out nulls is something that 100% should've been detected before it got sent out, tho.
The file wasn’t full of nulls.
YES IT "WAS SABOTAGE" YOU ARE SPOT ON. PEOPLE ARE SO EASILY FOOLED, IT'S INCREDIBLE. I KNEW IT IN MY GUT. NO ONE CAN CONVINCE ME THAT IT WASN'T SABOTAGE. OH, EXCUSE MY CAPS PLEASE. I CAN'T SEE THE SMALL LETTERS WELL ENOUGH TO USE THEM. GOD BLESS YOU FOR THE TRUTH
I mean, all the malware also targets windows because that's the big user facing desktop OS.
That's the biggest reason. I bet he can't even name 3 reasons why Windows is much more vulnerable than the other OS
Linux runs most of the world's servers though. Cyber criminals alwyas go for the path of least resistance and that's what Microsoft provides.
crowdstrike did the same thing to mac and Linux as well
@@torquetheprisoner when was this? Can you link to news articles?
@@benheidemann3836you can google "red hat crowdstrike", it states that the driver works in kernel mode and user mode
They failed to do a smoketest of their agent after build but before deploying it worldwide. Sounds like their software and update development process is just really not up to professional software engineering standards. At Meta, we had to have other engineers, sometimes multiple, review diffs before they would be accepted. And then there were multiple layers of CI/CD testing before exponential deployment with canary testing. You don't just push new code to all the machines all at once, because it's way too dangerous.
And deployment of system files, much less kernel level files should have a hash/checksum too, no?
And still they fucked up all their systems for a DNS error
things go wrong even after full in-house testing. that's why you have canary testing and phased roll outs. the ceo comes across as totally disingenuous and is ultimately the reason for the company's poor practices.
It's weird how they got to around 80 billion $ market cap with this incompetence!
@@grastant6819 this wasn't a kernel level driver/file. The .sys is misleading.
Crowdstrike is ransomware, they just have a different payment plan. You pay up front for the privilege of being ransomwared at some unknown point in the future. Turns out the unknown point in the future was today! Surprise!
The other problem is not just bitlocker, but if your company locked the microsoft account so you can only sign in from company devices to get the recovery key, but all your company devices are hosed. I can sign into the account from my personal linux box, but it just says it's restricted, and I can't do anything, or get the recovery key. I was on hold with IT support for hours today, and at one point, the phone system hung up on everyone, and calling back the number went to a busy signal for the next hour. I spent 3 more hours on hold with IT, and they either answered when I went to get something to eat, or hung up on me again. What a cluster F
It's how McAfee works on all the PCs of retired folks I know, who installed this "shiny, free antivirus software." 🤦
LOL call it reversalware
This is facts.
And don’t forget that Crowdstrike was responsible for the 2016 story that Russia hacked the DNC in order to get dirt on Hillary and favor Trump. Which was a lie.
The problem with the bootable USB thing is that a lot of corporate devices block booting from USB by default, which means the IT Team would have to tell the end user the BIOS password to get into the BIOS to change the boot order to enable booting from USB. Its a total nightmare!
Would the block not be a gpo ? So won't apply to a local admin or admin profile
Most companies I’ve been around weren’t secure enough to lock the BIOS.
@@user-zc5lf9xb2gBIOS loads before the Windows OS.
Yeah, as a former IT support bod in logistics, taking a user through the steps will be painful and in some cases practically impossible.
Can you get into safe mode please, power on the computer, wait eight seconds, hold power to turn off. Repeat again.
Err, it's not coming on again. Oh, it's is now, oh god blue screen again.
We need to start again.
Warehouse bod, eff this you'll have to come over and do it.
That's before you even get to bit locker and talking though command line. Lots of journeys will be happening this weekend.
Just use PXE.
Definitely a "zero" day problem.
The only things saving CrowdStrike from a class action is most law firms are Windows users too :)
Crowdstrike says "hey, they didnt respond like this during covid/bitcoin. Which was also highjacked, but nobody said anything and everyone joined in for the fun. Lmao
It may not have even been on purpose
@@DanielSmith-lv5eddoesnt matter. this level of negligence is actualy criminal. somone will go to prison over this.
😂😂
Apparently an idiot dereferencing a null pointer and another senior idiot pushing it to production past code review is now a “zero day”
No it’s just a really obvious bug. There was no hacker here unless the bad code was intentionally put there by the employees in order to sabotage the company. So if there’s any hack, it’s internal.
@@Jimothy-723 what , criminal negligence , last week there was that event and the person in command got no punishment nor lose their job for that absurd amount of negligence.
the fact that one company can take everything down like this is scary, one bad actor and this could've been a mass malware attack instead of a simple driver error
Nope. Crowdstrike had this access because they were trusted. Malware doesn't get to waltz this close to the kernel as easily.
@@blakenolingberg1556 that was the mistake.
Don't trust anything running on the ring0 that's third party, except drivers made by the vendor.
No toy software or rootkit is allowed.
Capitalism market innovation... this is freedom of choice... The truth is corporate American hates competition, and capitalism always produces monopoly.
@@monad_tcplotta people run this due to legal obligation, and crowdstrike has historically been far better than their competition (let that sink in)
No - Microsoft have a security feature if bad data updates it's designed to crash. Falcon - it's been busy - on this direction for a while I'd guess
I literally spent my entire friday manually fixing computers and explaining to people at remote locations how to fix their computers. Our entire IT department became helpdesk because of this update. You don't know the pain of explaining to a non-tech person over the phone - how to make a bootable USB, boot to it, and then enter their bitlocker recovery key so they can delete a file via command prompt until you've done it personally. I got to do that dozens of times on Friday and theres going to be lots more of this for the foreseeable future... I cannot express how much this sucks to fix even though its a relatively simple fix. It just can't be automated and it's horrible for that reason.
I mean, Windows might be the least secure how most people use it, but there's another huge facet to why it's the target of randsomware: it's absolutely dominates the end-user/workstation market, especially when you are wagering the victim can't just restore from a backup and ignore you.
It's purposely written like swiss cheese, full of back doors and vulnerabilities so they and intelligence agencies can access your computer any time they want. Linux is not. So, Linux users don't have to worry about viruses or malware and don't have to put middle-man software between them and their machines to protect them from their malware/spyware OS. The only thing I use on my Linux servers is a firewall and Fail2Ban to prevent brute force pwd cracking.
@@Texas3Percenter This specific issue is not exclusive to Windows. This is an issue of operating systems architecture and how drivers are able to run in Kernel mode. Falcon have Kernel driver for Linux, Windows and macOS. The IT departments of companies are asking the OS to load Falcon driver an allow it to run in Kernel mode, allowing it to watch for user data but also enabling it get OTA updates and not able to choose of get or not the update. Same bad driver could be for Linux or macOS.
@@unaquetzadillaalso, macOS still needs antivirus/endpoint protection. It just works a bit different and the audience of people who get MacBooks at work is probably more technical
Just what I was thinking. The reason Windows is targeted more is because it’s sitting on 73% of desktops. The PC Security channel showed that malware vendors absolutely have Linux versions of their tools. It is not immune.
@@unaquetzadillaWhat you said is partly true. In my previous role, I deployed crowdstrike for a major broadcaster and one common misconception in all of this, is that crowdstrike can push updates to customer endpoints without their knowledge or consent. It's simply not true. Endpoint management is handled centrally by IT admin and we can choose if we want to use the latest Falcon sensor version or not. You can of course configure crowdstrike to auto update the sensors but that would be ludacris.
They don't want to apologize cuz they don't want to admit fault and open them up to lawsuits.
Their asses are already opened up to lawsuits big time! 3 billion in cash $ watch it evaporate! Their customers are now PUBLIC DATA which is a huge liability for their future cash flows too.
too late for that! this company shoudnt exist in the future!
THEY'D NOT BE ABLE TO "PAY FOR ALL THE QUATRILLIONS OF DOLLARS LOST😂
I literally just turned down an offer from Crowdstrike two weeks ago in favor of another job offer…it was a tough decision to make at the time but now it’s definitely looking like I made the right decision! 😬
😂
dodged a bullet
Where in the hell is the testing cluster they should have deployed to first? CrowdStrike should deploy their Falcon updates to all their own machines and if they don’t BSOD after a week THEN release to the entire galactic empire.
Yeah, I was gonna say don't they do gradual deployment? To like 1% of machines first, then 10%, 25% etc. or at least some A/B testing damn
Lets accept for a second that they did that, because they probably did, they've been doing this for a while. They probably did NOT send out a null file. So somewhere between them releasing it and the end users getting it, it got nulled out. That's where the problem was. Azure went down right before all this happened...
This is tough because as an antivirus you want updates deployed as quickly as possible. If some new exploit comes out you wouldn't want your customers to be vulnerable to it for over a week.
If they tested, then they either didn’t test what they were deploying or they didn’t deploy what they tested. This is a basic control problem. I have had to point out to management numerous times that whatever cockamamie plan they have for maintaining a claim that the product was tested, that they weren’t testing what they were deploying. That internal process needs to be seriously scrutinized.
@@piquat1 i cannot believe they were smart enough to do a phased roll out, but then neglect a simple hash check of the deploy-able. the evidence suggests they did neither.
uh, no. Crowdstrike on mac is just as deep, and slows down my work mac just as much.
Shhh, man. Macs are suposed to be fancy.
@@ivocass4332 it's a very slow, hot, but pretty piece of aluminum after corporate IT gets to it. XD
Oh that suuuuuuuuucks
But it difference is that is doesn't run on kernel level. Mac and Linux have fixed this particular problem long ago
@@petargolubovic5300 Agreed Mac booted Anti-,virus software from the kernel after creating an endpoint security API for them to use and Linux has ebpf hooks and bpf programs to screen for AV to screen for potentially malicious activity with a guarantee that they can't crash the kernel (bpf programs have strict security/stability guarantees while being non-turing complete)
hey grandma, all you have to do is start up in safe mode, grandma? Grandma?
Honestly, when I was an i.t. call center tech, the older people could often be counted on to listen to my instructions and not go off script. It depended on the person of course, but I was often able to wrangle an older person to listen. 😅
Grandma just rebooted into safe mode
grandmas on mint
grandma:beeeeeeeseeeseeeeeeeeeeeee
Please be patient: Grandma is busy beta testing in production. 😅👵🏾
loved the title "The day the world went blue"
Video with that title is delisted
@@LagowiecDev yeah i know i just loved that title this one doesnt flow off the tounge like that one did
I loved that title and will likely change back to it. This one's performing way better though :(
@@t3dotgg :)
just make it ur description or a pinned comment
Crowdstrike is the company that shuts down your computer when you're hacked, and they wont allow it to turn on until they check manually how bad is the hack...
We don’t shut it down, we hit a button labelled “network containment” which only allows it to communicate with the admin dashboard for forensic analysis. Your MSSP should be ringing you immediately if they have to use network containment
Yep, and since it is using a heuristic, there may not be a hack at all. It's arbitrary DOS
I was dealing with this today as an IT tech. Oh boy, What a joke. Took around 20 minutes on average per person affected. Since hundreds of devices were affected it made for a long day... :(
😮😮😮😮
20 minutes? You could've set up a few rubber ducky USBs to automatically run those commands.
@@theairaccumulator7144 He gets paid by the hour
Yeah, I'm one of those tech guys. I'm in charge of our enterprise's cloud infrastructure (which is all our servers). I was up till 2am restoring a couple servers affected on our European side, thinking it was some weird Win update that took things down. I went to bed and was woken up 3 hours later by my boss freaking out. I spent all morning force shutting down systems, detaching and attaching drives to working systems to remove this .sys file and all. What a HUGE pain. I finally got everything working after like 5 hours of doing this crap nonstop. The poor helpdesk was stuck doing bitlocker based safe mode fixes for end users. I don't envy them...
Sorry to hear that man. I hope you get to rest up over the weekend.
I hate bitlocker....
Why updates are not tested in quarantine environment by businesses themselves? You should have mini cloud for that. Seems like everyone blindly relies on Microsoft and their partners.
Absolutely insane that one file can cause that much havoc. The fragility of all aspects of life as we know it... Mind-blowing.
Hope you get some rest 💙
@@egria We do test them a day before on some servers we don't care much about and can restore in minutes from backups. We're also a tiny shop of 2 IT guys and a contractor, and I'm the only one qualified to do the cloud admin stuff. The Win update thought did cross my mind, but it was 2am and I was burned out. I just restored them and called them a night. Little did I know it was a global shitstorm from our endpoint solution provider.
My favourite part of the disappearing air traffic example is that while they will occasionally get crippling downtime from their infrastructure, Southwest still running primarily Windows 3.1 with a sprinkling of Windows 95 here and there rather isolated them from the CrowdStrike issue.
surely that is a joke… surely…??
@@bullpup1337 of course. No way southwest is running something as modern as windows 3.1 :p
@@bullpup1337🙂
Once again Southwest comin in for the win
@@Ignisami Actually, they are
Our company installed CS on thousands of Windows clients recently. A few weeks ago they uninstalled it because it was causing massive system performance issues. Giant bullet dodged.
On the Windows side, it's just nuts that any driver causing continual stop errors is not auto disabled /quarantined by the OS.
are you sure you aren't neo? bc that's some wicked bullet dodging ability
The driver affects the boot process which is why it wouldn’t fail until a restart. Cant auto detect a failing driver before that driver gets used by the system.
Kernel drivers will cause severe problems on any OS, not just Windows. Just search RHEL CrowdStrike if you don't believe it.
Most drivers do not, but generally AVs mark theirs as a boot driver, so the system cannot boot if it is failing. So it's not just "any driver" as you state.
The fact it has to be implemented as a driver at all is nuts.
"Windows is the only OS that is insecure enough to have problems like this"
Let me tell you why that's bullshit - Crowdstrike did this to our production Linux fleet back on April 19th.
Don't say something like that that's unpresedented
Biased video for sure, but any opportunity to bash Windows they take it!
@@theoryianabsolute8777 I do not believe that word means what you think it means. This has as of now also been confirmed by other news sources.
Accessing a CIFS share did this to a few of our linux boxes last month. Cause was a broken kernel "security update".
Linux versions of Falcon also hook into the kernel. Talking about how this fuck is somehow because it is Windows is disingenuous. Why can't we just blame Crowdstrike and just Crowdstrike instead of bringing up and blaming Microsoft?
i wholly agree with you, this guy just wanted to hate just because its windows, i also don't like windows myself but its pretty disingenuous to blame Windows when its not their fault.
He didn't blame windows , he said they are the most insecure OS on the market , and that's why software like Falcon exists
@@Abaddon231 falcon also exists on Linux and Mac. Is Falcon there because of Windows? Falcon exists because there are malicious actors, not because Windows exists. Then he mentions kernel drivers without acknowledging that this also happens on Linux and the fact that if you want a solution like Falcon, they should hook into the kernel, because that is how they can most effectively do their job. Or why bother having and EDR at all. He would have been able to effectively convey his message instead he went all mainstream media ranting about Windows. This shit isn't his forte, but he still has a take on it. Low Level Learning has a much better and nuanced take on this.
Falcon on Linux uses ebpf hooks which can't crash the kernel (they have extraordinarily strict guarantees, restrictions, and limitations that would prevent an ebpf program from doing that, they aren't even turing complete)
Apple on the other hand has an API for AVs to do their job and doesn't permit them from installing a driver.
@@brandonn.1275 Falcon on Linux in User Mode uses eBPF. There are still many systems that use Kernel mode. Although Falcon is transitioning to User Mode, Kernel Mode is still officially the default mode for Falcon on Linux. Per official CrowdStrike docs. Many of my Linux systems still use Kernel Mode.
Root cause analysis - 1) Using Windows for mission critical work in 2024 2) Terrible code that somehow made it past code review 3) Build system that corrupts files 4) No validation checks prior to rollouts 5) Rollouts to the entire install base rather than a staged rollout
I've never anywhere seen a build system that produces a file of the right size, but filled with all 0's - how does that even happen?
It's WILD to me that so many big companies have built critical business infrastructure around Windows. I do it for my little piddly business but I'm aware of the shortcomings!
@@Jimothy-723 1. DEI isn't a word, it's an acronym. 2. These kind of issues occurred long before 'DEI' was a thing, so how do you explain those issues if this is obviously because of DEI? I'll wait.
1-4 are BS, but 5 is on point
@@user-uo5eu6yk1b DEI is just an acronym for affirmative action. It's always been a thing in the internet age. Sorry that you're wrong. It's more prevalent today since there are few qualified white males willing to work for crap wages.
"Pretty much every PC in the world just BSOD" - Incorrect, only PC's that ran CrowdStrike.
Let's not forget all the people who probably put their very important bit locker passwords... inside of their bit lockers.
Thats why they call IT. The amount of people who do not know how to log in to their work email on their phones so I dont have to read out their 69000 digit recovery key is incredible... and then hope to god that they wrote it down correctly or heard you correctly... and then hope to god that they enter it correctly so that 75 year old Dorothy doesn't have to be talked through booting into safe mode again BEFORE you have to talk her through deleting a file.
So, there was this episode of _The Munsters_ that followed the same idea...
They wound up blowing up a very expensive antique box to find a video cassette (and not a heap of treasure).
There was a post on reddit about a huge org that has the bitlocker keys on a box with bitlocker and they don't know where the key for the centralized box is because the documentation is also behind bitlocker
I did that by mistake once, I stored by password database on the bit locker drive, the bit locker password was on the password database.
Lucky I have a offline backup of the password database.
@@tatumsh9 Or maybe the company shouldn't limit the microsoft account to only be accessed from company devices, when the only company device is hosed! Sure, don't allow access to everything, but they could allow access to the damn recovery key. Also if I have the company email setup on the phone, then it creates a huge nightmare in outlook making me authenticate on the phone every time to use outlook on the pc - I discovered by accident (when the company "mistakenly" removed email from everyone's phone) that not having it on the phone solved all the auth issues on the PC too, so I just left it off the phone. I don't need company email on my phone 24/7 - company email is only for during work time on the PC
Crowdstrike painted the town BLUE !
This is going to be a LONG weekend for some folks in tech
Weeks sir... weeks
No one goes home til this is fixed!
I'm a glass half full type of guy. At least now companies know how important their IT are
i guess this takes time because fix need to be done manually on every computer
@@cabpacedilla yeap
its shockingly inept. the fact they did a rollout with no percentage based rollout and metrics on how the rollout was performing and no rollback plan is literally insane and then on top of that the fact their deploy pipeline distributed this with tests either not run or failed.... makes it seem like theres more to this story than just oh we just pushed some bad code.
There are safeguards in code and policy that should prevent this.
also the fact it's literally *in* the kernel and not something like ebpf where it can have hooks in the kernel and have saftey guarantees about it not crashing the computer... thats microsoft fault for writing a steaming pile of bitrot masquerading as an operating system.
once you remotely brick a pc, the rollback becomes difficult...
@@jeronimo196 thats why its also done as a percentage rollout.
3:40 Not true. Falcon sensor is available for Mac and for Linux too. The real reason that it has happened on Windows only, because 1.: CrowdStrike seemingly made a mistake only in the Windows driver, 2.: Windows is waaaay more popular in businneses (both server and desktop side) than anything else.
more like clown strike haha gottem
There was an issue with CrowdStrike on Debian several months back, that caused the OS to not boot… This isn’t the first time CrowdStrike has massively broken an operating system
Their reaction to that issue suggested they really did not care. It should have been a red flag.
Running bit-locker feels more like a "Hurt Locker"...
One point to make, (I am a Mac user so don’t come at me 😂) Microsoft has the biggest market share for desktop by a large margin so make sense for hackers to focus on them, not sayings it shouldn’t be more secure but also theywill get bigger focus from hackers just due to market share
Linux runs most of the world's servers though. If Linux is much more vulnerable than Windows it would make more sense to focus on Linux since you can have more control. The thing is Windows simply is a lot less secure and more prone to crashes like this.
@@haroldcruz8550 you know, it is much harder to trick user to install malware on server, than it is to make user to launch it on own PC.
Which is, coincidentally, predominantly Windows.
@@haroldcruz8550Linux flaws are invisible to the general public and end-user Linux enthusiasts. That said, there are many well known attacks on servers.
its insane that crowdstrike didnt integration test the update, and even more insane that mission-critical infrastructure is OK with automatic OTA patches.
How in the WORLD do you not test the system BEFORE you send it out. CRAZY!
Doesn't Microsoft force updates on their Windows users too?
One thing though, using a kernel mode driver is not exclusive to crowdstrike. Many other AV/EDR systems use drivers as well.
In fact a very similar thing happened with Symantec a while ago.
At this point windows is going to need to boot Anti- virus software out of the kernel and provide an API for AVs to do their job instead of having them insert a driver into the kernel.
This is what Mac did when they booted AV vendors from the kernel after publishing an endpoint security API for them to use.
Linux has something similar in the form of ebpf hooks and bpf programs that can run in Kernelspace while being guaranteed to be unable to crash the kernel.
iirc even linux security software sometimes needs kernel mode drivers.
stuff like cloudstrike can avoid needing a driver on linux bc they can just use ebpf tho. mac still needs a kernel driver.
Yeah the affected CS update just happened to target a Windows named pipe vulnerability, so in this case only Windows was updated.
So to just blanket blaming using Windows like OP did is a bit lazy. Their Apple and Linux customers just got lucky.
@@JSmith73 macbrained webdevs can't see the world as it actually is. They're used to rewriting their entire app (which is just plumbing between AWS, databases and APIs) in the latest JS framework every 3 months. They don't realize that most legacy vendor software which is what the world actually runs on is a piece of crap and was written by a team of 10 contractors over a few months for a specific pentium windows xp machine in 2005 and has become a mountain of hacks and patchwork because management doesn't want to spend money on improving it.
@@dead-claudia Mac doesn't allow AVs into the kernel anymore instead they published an API for them to use and won't allow AVs to install kernel extensions anymore. In fact kernel extensions have been deprecated for a while now and only userspace system extensions can be installed.
this seems like one of those applications where you'd expect every pull requests to go through a "committee" such that you don't have some one-guy write a bug into the code..
Or malware. Seriously, if this could happen due to some stupid mistake, imagine how bad it would have been if an actual bad actor had social engineered their way into position to abuse the hell out of it.
Or there is a nice vulnrability sitting there for malware to poke at. I'm sure Falcon is going to be put through some fuzzing by the bad black hats.
“Pretty much every PC in the world”… What? No. Nobody’s home computer would have CrowdStrike software. No Mac or Unix or Linux systems would be affected. The number of business computers running Windows with this software loaded is absolutely mind-boggling, though. I mean, we’re in seriously WTF territory. I hope people can get things straightened out quickly, especially for the more critical areas.
If you search hard enough, you'll find that there were two Linux Distros that got a bad update from CrowdStrike that resulted in the same issue. I think I read that it was cases of kernel panics in this case.
RIP to the intern😢
If an intern did the coding or deployment, then their supervisor needs to be in big trouble.
Lesson learned: use Linux (I use arch btw(I use arch btw))
After using Linux for 10 years and Arch for 5 years I will say... use Mac instead. If you need something linux or windows specific you can spin up a vm with Parallels. And overall, mac laptops lasts long and have overall great performance & build quality. As for servers - linux is the only choice.
What if I want to play Elden Ring @@dyto2287
@@dyto2287 That or, if you do want to use Linux, just use ubuntu.
looks like you use lisp btw too
@@dyto2287Until your display cable is too short and rips when you open it and apple won't cover it under warranty... Not to mention the throttling that occurs because they don't put fans in their laptops. Hope you didn't need performance...
Do note Crowdstrike has a kmode driver for Linux as well. And that also broke RHEL recently... OOPS
Yup, this isn't just an accident. It's a pattern.
Wow, fortune 50 company I used to work for had all the users on windows, of course, the back end for the most critical things ran on RHEL. Wonder how they're doing now. lol
I don't quite get the bit about most antivirus not using drivers. Drivers have been an important part of AV and other security software going back to the Windows 95 days, I worked for a company doing security back in those days and we had a file system driver that was the core component for the file system security portion of the product, and a keyboard driver that was part of the system that ensured commands given to our system were coming from the interactive user and not from some script or application. Drivers are so important as they are outside of Windows. A normal application, even if running as a privileged user, cannot kill or modify the driver, nor can it bypass the driver when talking to the devices controlled by the driver.
Not sure what is scarier, that CrowdStroke released a bad version or that so many companies just blindly went with it without testing and staging it first.
this was supposed to be more like a config or signature update. this is like pushing a bad signature file to windows defender and causing it to crash.
Airlines, banks etc. without testing updates in isolated environments is absurd yet reality. And most systems if done properly don't even need antivirus because they suppose to be nit connected to public networks. So this would be some management's push to order that software. Important systemd should be setup in a way that assumes that something goes wrong that means having staging environment. This incident shows massive tech incompetence either by itself or with push if higher management wanting to reduce cost or just fall on lies of vendors of how great everything would be if company trust them blindly.
The update was automatic I think D:
Friday morning at the office I was jokingly asked if I caused it. The day before in the company Town Hall I announced that I cancelled the CrowdStike contract and we have mostly removed Falcon from all devices. Only two left over machines had issues.
Curious, what was your reason for cancelling? Was it affecting productivity?
@@Micloren basic Falcon didn’t do much and got better data from the Checkpoint and Fortigate UTMs. Unless you shell out for the full SIEM, felt limited. Decided to up the network security instead and pay for a SOC/NOC service.
I’m shocked how much windows there is in infrastructure
OMFG, Theo thank you I had no idea how terrible this was!! The encrypted bitlocker problem is absolutely horrendous! Oh my god, I could maybe get this sorted but most people definitely can't, this is BAD!
WOW. I'm a developer currently on vacation, and I'll be back to my job on Monday. My computer has been off for the past two weeks so I guess I'm lucky? Assuming right now they are no longer shipping the bad update and I can safely turn on my PC, but will definitely make sure next week!
Lol, you dodged a bullet, brother!
Disconnect from internet and then boot up.
They fixed it. I just booted my computer no problem after not using it at all for the past 2 days.
Urrrgh must check mine grrrr
Unplug it from the internet when you boot, and then turn off the updates.
At first glance this is actually hilarious to see, but I feel so bad for the patients at hospitals affected by this. That's the worst part. Sometime like this will literally cost lives. Crazy
And Hammond gets a shoutout here as well 😎. Also, great video of course, really enjoyed the take on how to and how not to communicate in such a situation.
Absolutely! If I didn't shout him out in the video directly that was an absolutely L on my part
@@t3dotgg Nono, you did, 12:40 👍Hammond collab when? 😉
Southwest Airlines unaffected. Use older version of Windows. Brilliant. Latest/Greatest not always good.
Could also be that they simply not using Falcon...
It's not Microsoft bug/error. it's a Crowdstrike bug/error
CrowdStrike engineers: “Update is ready, let’s deploy it to the world Friday morning, and let’s test on production”
Every single time something really shitty like this happens.. almost without fail. EVERY SINGLE TIME: Look at the education of the CEO.
George Kurtz: Degree in Accounting. (no formal science, tech, engineering, or computer education of any kind... I can see he claims he can program but as far as I can tell, he's never actually worked a job involving programming or science of any kind.)
The fact that this company pushed a driver rollout to hundreds of millions of people SIMULTANEOUSLY without checking it worked first tells you EVERYTHING you need to know about how this dude runs this company. If he had any actual knowledge of computer systems he wouldn't have allowed that to happen. And yet he did.
When will the world wake up and start to realize that shit like this always happens when you put business people in charge of technology they don't comprehend. Now I'm not saying George Kurtz knows nothing about programming, its completely fair to be self-taught. But as far as I can tell he never did anything science-related at his job in any capacity, and his claim to fame is that he co-wrote a book about computer hacking with some actual computer scientists back in the 90s. Ever since then everyone has treated him as though he himself is a computer scientist, even though he's not actually. And after a #$#@ up this extraordinarily bad, it seems it was wrong to believe he knew what he's doing. There's NO WAY a mistake this bad could have happened without his express knowledge and instruction that this is how they operate.
Stop trusting bean counters with important technology.
3:34 - People always think hackers target Windows because it is insecure but that's actually not true. Microsoft Windows is the most used at 72.22%, followed by Apple's macOS at 14.73%, desktop Linux at 3.88%. Just by looking at this, one can easily deduce what operating system a sensible hacker would target if they wanted to create malware. So, its not that the other Operating Systems are secure but a matter of ROI. If you spend a month creating malware for windows you get 72% possible targets while on the other hand spending a month creating macOS malware will give you only 14.73% possible targets.
It's purposely written like swiss cheese, full of back doors and vulnerabilities so they and intelligence agencies can access your computer any time they want. Linux is not. So Linux users don't have to worry about viruses or malware and don't have to put middle-man software between them and their machines to protect them from their malware/spyware OS. The only thing I use on my Linux servers is a firewall and Fail2Ban to prevent brute force pwd cracking.
@@Texas3Percenter bruh you repeat this borderline insane conspiracy shit on every single comment
@@nalstudio_official You're just not knowledgeable of these things, bruh. Educate yourself before you go talking shit.
Problem with bootable USB device is that, a lot of corporate systems also disable USB for security reasons, this gets more interesting 😂
Somebody changed one life of code on a Friday afternoon, pushed the pipeline, and now we get this
@t3dotgg Your description of the Windows / Active Directory / BitLocker login process is inaccurate. The bitlocker key is not retrieved from AD or other remote DB when you auth, but rather from the device’s local TPM.
I believe he is talking about the recovery key, which *is* retrieved from AD or another DB.
Every time I hear about this it starts saying (pretty much every PC in the world was affected). I never heard about CrowdStrike before. Every person I know is unaffected and I didn't hear about any company or service affected where I live. Anyway, good luck guys.
Lol. So Annoying. His own PC was not even affected so I'm wondering how he came about "every PC in the world"
As soon as he said that I stopped listening.I wonder what else he is going to get wrong. So much misinformation going around.
Yep, this video is just full of misinformation.
I think it primarily affected enterprise clients, and that's what he should have led with. But I guess he couldn't resist a little Microsoft slander 😂😂
Ooooh yeah I love the voice that tech influencers have when they weren't expecting to go recording and their humanness comes out more??? Like, idk what it is about it, it sounds like you just woke up (compliment)
@@NormCantoralthose weren't tech issues
@NormCantoral that makes sense, I just thought that interpretation was so unreasonable that it was more likely he just hasn't ever seen a software bug of this scale and severity that was caused by something so easily preventable. made sense to me bc he runs a tech channel.
Saying the fix has been deployed without that being technically possible is such a stupid move that likely created a lot of additional stress.
If a non technical boss of a company reads this, he might blame the IT team like "crowedstrike deployed the fix 5 hours ago, why can ours hundreds of employees still not work?"
I don't understand why everyone is so caught up over this "kernel level driver" thing - this is not built for consumer PC's. EDR solutions REQUIRE kernel level access to even be effective at catching as much malicious software as possible - it gives you such an upper hand and allows you to check and scan EVERYTHING on the machine. For a consumer non-business user device, this is super undesirable and would not be a good solution - but for a business that requires intense security to protect their data? At least for the moment, there is no other way.
Those saying that Windows is the only OS unsecure enough to need this and jabbing at windows... Yeah. Just Windows things. I mean it's not like Linux and MacOS are perfectly secure, but the general consensus is those OS's are more resilient to viruses than windows. (Though it's not a bad thing to point out, that most malware is written for windows, because of how ubiquitous it is in industry.)
No, they do not require kernel level access. In fact they are not allowed to have kernel access on Macs and even Linux is moving over to ePBF where a bad driver can't crash the system.
@@uzlonewolfkernel drivers are allowed in macs. and linux kernel drivers are limited in a number of ways.
notably, a kernel module is needed to monitor syscalls for processes you didn't spawn. and seccomp filters don't let you count anything, only filter.
@@uzlonewolf Because of the way that Windows is built and the way that threats/malware currently operate, the only way you can hope to catch everything, at least right now with current technology, ideas, and software, is with kernel level access. Hopefully this changes, but as of right now it is what we have. I am only applying this to Windows however because, obviously, we've found alternate solutions for Mac and Linux and have not needed to do this.
The file full of zeros look suspicious. Could it be supply-chain attack?
CS biggest investors/owner...Blackrock
@@Wayoutthere blackrock hacked crowdstrike?
Hah! That USB boot idea would be fine, if it weren't booting you into safe mode after unlocking your bitlocker. That's prime attack territory for planting a rootkit while "helping" you clean up that crowdstrike BSOD. You'd have to audit everything about how that USB key came to be and all the software on there, from extremely trustworthy sources.
alot of hospital computers are down too which is very dangerous
This being an accident is more terrifying that if it was done on purpose.
Maybe this was the canary test for that intent. Integration has teeth
I mean there's no way a company with that many skilled people rolls out a zero bytes file does anyone think this was deliberate? Theres more to this
Companies that run something like CrowdStrike often use BitLocker AND have take measures to block USB devices. You have to lock a Windows down way more to be ‘compliant’ for auditing
Why does a badly written driver stop the machine from booting? Shouldn’t that driver just be skipped and whatever device it targets not work?
Seems like a terrible design within Windows.
It is not Windows design issue.
It is driver being written as kernel-mode, to gain extensive privileges over system in attempt to prevent malware activity. Meaning it becomes required to boot before even Windows takes over.
And if it crashes, well, Windows hadn't booted yet. And it cannot exclude this driver from list as it is listed as mandatory. Loop repeats. And no revovery window appears because system crashes before it even reaches said state.
Someone even said that CrowdStrike already did similar stuff to Linux systems at 19-th April.
Linux and Macs versions of CrowdStrike are also using kernel mode drivers, albeit with some nuances (like there being restricted version for Linux). And similar case could've also caused bootloop (on Linux BSOD is more known as Kernel Panic).
@@DimkaTsv This is a Windows design issue. Try to load exact same file as a Linux module. It will not crash, but You will have a simple warning and that's it.
the prob is it had to be done as a driver to get kernel mode. MS tried to develop and license an interface that would allow that without having to load potentially sketchy kernel-mode drivers before boot, but then the EU stepped in and forbade it, because, they figured, it would give MS an unfair advantage on the market, if they were to choose who would get costly privileged kernel access. So you can blame the EU as well.
Or you can blame Crowdstrike for knowing they are doing very, very sensitive stuff, but OTA-pushing a bad, untested update regardless.
The fact their stock price barely took a dent is insane. This is the kind of mistake I'd expect from a solo startup with 30 users, not a multi billion dollar corporation!
Stock market PCs are probably down. Next week though …
1. it takes more than a day for such issues to result in stock price drops.
2. cloudstrike fixed it same-day. it's just the damage done was so severe, it's taking customers a disproportionate amount of time to fix.
This had to happen during my vacation week
Lol
oof
This had to happen during my school's online subject selection
That sucks! I have to close on 2 real estate deals Monday. They won't happen til this is fixed. Thanks, banks, for using crowdstrike! If you were using Linux instead of Windows, you wouldn't need to.
suck it up lol
Microsoft outsourced some of their customer support overseas. In the Philippines, they train their agents about BitLocker *NOT* an MS thing, but rather a 3rd party like a MOBO.
That baffles me since it doesn't exists or required in Linux or Mac systems.
CEOs can't apologise to that extent for legal reasons. That proposed comment would bankrupt the company.
yep, especially since they're publicly traded.
privately owned companies can get away with more, but only bc there's a lot fewer people who could have standing to sue over that.
This situation can happen again if we do not fix this broken system:
- Stop using Windows only, if you buy multiple OS types it can be more work but it is the only way, even if it is not Windows it is better.
- We cannot have deploys without lots of testing pipelines, I am sure that they did not test it, if they test the tests are really bad.
- We cannot buy a PC that has forced updates, even if it is not fully secure for some hours the users need more control over it. Even if we keep using force updates they need to have some stages to pass first, for example, let's test a small number of users first, then scale up in a controlled way.
While I hate Microsoft about as much as Apple nowadays, your take on Windows in this specific case is totally wrong and uncalled for. Only Windows was affected because only that update was affected. Windows doesn't have anything worse than Mac or Linux here. Also Mac and Linux could've been just as much affected. That whole section is very cringe.
Edit: I mean the section starting at 3:35
The rest of the video is a-ok
Yeah, this could’ve happen to any of the OSs especially in a corporate setting. That said, Apple actively discourages kernel extensions and have built alternatives in user mode.
In this particular case I *suspect* Linux would handle a faulty kernel module bettter, but I'm not sure. This is crowdstrikes fault for sure, but I wonder if the way Windows handles these kernel level drivers could be better though.
I believe cloud strike has uploaded a faulty module not a while a go there was a red hat incident posted for it. But I guess it was not as widespread.
@@fallingintime But was it as catastrophic as a unbootable BSOD ?
Edit: Although it would probably be hard to compare unless it was at least a similar mistake.
@@jonnyso1 kernel panic that required a kernel fix I believe. Probably wasn't widespread as it only affected a specific kernel version and kernel updates are not ota
The problem is the monopoly crowdstike has on critical infrastructure
the problem is critical infrastructures shouldn't use an OS designed for gaming
The take on windows being the least secure OS is a bit biased. The real reason why most cyber attacks happen through windows is because 1. It is the most used OS by a huge margin. 2. Since it is so widely used, hackers focus on finding vulnerabilities in windows instead of other OS like Linux.
It's purposely written like swiss cheese, full of back doors and vulnerabilities so they and intelligence agencies and access you computer any time they want. Linux is not. So Linux users don't have to worry about viruses or malware and don't have to put middle-man software between them and their machines to protect them from their malware/spyware OS. The only thing I use on my Linux servers is a firewall and Fail2Ban to prevent brute force pwd cracking.
Complete fabrication. Linux is the most widely used OS. Get your facts straight
@@insu_nareally? How so? I thought it was windows.
@@insu_na In which context did you mean Linux is the most widely used OS?
@@Texas3Percenter Oh so your Linux box is insanely easy to get into. Interesting that you let us know this....
My rage at everyone downplaying this for CrowdStrike is immeasurable. This is a billion dollar company, with a B, trusted by critical government, public, and private services and they shafted each and everyone. The lack of outrage from our authorities is absolutely disgusting. Speaks a lot to the state of cybersecurity and tech in general
i personally think that while the emergency is occurring i don't want the CEO wasting time crafting apologies. Their goal should be dispensing accurate actionable information to help people recover. apologies can be crafted after.
This is why a mega level cyberattack is so scary. Too many companies rely on the same systems to function (ABS, Google, Crowdstrike) so to see something like this happen it just shows how vulnerable our networks are.
I don't get it - does this incredibly impactful product not roll out to testing environments/deployments??
aside from just unintentionally shipping a file of null and not apologizing, what if a rogue employee pushed some badware into the product? apparently you would only know when it's already deployed 'in production' in the most important enterprises in the world at a kernel level.
This begs the age old question: “who watches the watchers?”
I can't imagine the infrastructure that could let this happen. I have worked at a lot of companies, and even ones that haven't always had the best practices. But every single one of them had measures in place that would have caught this problem in at least 5 places before a customer got it.
Yea, being locked out by a faulty update of a security thingy dodat!
JUST one question: who will pay for all the hours of work for this faulty update? Are lawyers already drafting suits? 😦🤑
Every cloud strike has a silver lining 🤣
The most detail explanation I’ve seen. Thank you! Great video.
Why are there fireworks behind Theo at 7:05
On new macs the camera sees your hand gestures and makes these animations. There are few of them
So, I tried to fix this on my wife's work laptop myself. I got into safe mode, entered the bitlocker key and... I needed an admin account to access the folder and delete the driver. 😮💨
Kernel mode driver! What do you expect?
@@McZsh It's just another reason why "Kevin from accounting" won't be able to fix this. Even if he happens to understand the instructions.
You really lost me on this one theo. There isn't anything here that couldn't happen on Linux, in fact, it did -- about a month ago. It's not guarding your bootup and preventing you from booting in an unsafe environment, its just a broken kernel mode driver. I got bluescreened. You know what happened? Windows dumped it's ram, rebooted into last know good config, i shrugged, and moved on. I can't believe this overshadowed Trump being shot to the point i hadnt heard about the latter until an hour ago
Exactly! Ignorant people acting like this cant’t happen on Linux. EDR agents on Linux are just as deeply engrained, and highly privileged.
@@mwahlert "omg Windows"... No... Omg the market share! Bet you didn't know Windows ran in all *these* places :3.
"Omg Windows needs this?" Up until 2019 CS didn't even have any offensive antimalware tech, it purely was marketed as a distributed, crowd sourced IDS for setting a baseline and then detecting anamolies (whether abnormal for your business, or just plain university unusual activity)
"Omg a driver?" Yeah .. if you want network inspection (NDIS) or any kind of endpoint DLP, and unless you want your employees ripping it out, changing it's permissions, etc. As someone inconvenience by it daily since i run executables classified as "hacking tools", CS does an amazing job of preventing bad stuff from running, no matter who you are. First CS issue I've ever faced the wasn't as simple as requesting an exception for a given piece of software.
I mean this happened last night and trump got shot 6 days ago.. living under a rock sounds like a choice
also windows is the hodgepodge of configurations and programs and apis he said it is. Everything is an inconsistent mess and it seems reasonable to think that would leave more security issues..
I wouldn't say this overshadowed the Trump shooting, that's just a reflection of the content you interact with, there's been tons of coverage over the last week regarding Trump
Crowdstrike, now probably the most hated company by end users and IT folks globally. Customers will start to make plans to exit from their offerings and consider other options.
I am a support partner for Microsoft. To resolve this issue, boot into safe mode or recovery environment and then go to the C:\Windows\System32\drivers\CrowdStrike directory and delete the “C-00000291*.sys.” file. Then restart the system in normal mode. That should fix the issue.
If you have an unencrypted drive...
@@malvoliosf even if you han encryption enabled, if you have the recovery key. The recovery key can be obtained from your Microsoft account. Enter it to get into the recovery console. Once you do that, get into safe mode and delete the crowdstrke.sys file.
@@malvoliosf There are now instructions on how to do it on encrypted drives without a known key as well.
Average person & a good chunk of IT people don’t know what safe mode is not how to navigate a command line.
It asks for uac, too. So still need IT to do it for me, even if I know how 🤷
I’m happy to say that none, zero, zilch of the servers I look after were impacted by this. Because last year when there was a corporate push to install CrowdStrike on them, I pushed back. I was and remain highly resistant to giving any 3rd party company the ability to install software on my systems at their whim/discretion, for exactly the reason we saw this past week. There are other methods of monitoring your systems that don’t require that you give away the keys to your infrastructure.
wonderful day to run mac and linux
This is why you don’t cheat in your computer science courses
Crowdstrike looks like a russian sleeper agent tbh
the one is from Russia there, i saw on wiki 😂
It's literally the FBI, rofl. They are the same company that led the DNC server and email investigations from the last 2 elections and helped conclude it was Russians working for Trump (server) and clean up all those pesky emails proving the DNC aided Hillary in harassing Bernie out of the election.
You liberals, man.
It was a memory safety issue. It looks to me like a developer in India did compile using an old version of c++ and deployed to prod before testing . The compiler didn’t identify the null pointer dereference.
@@tacorevenge87 one button for the whole world ??? 🤔 today russian heckers put down Split Airport 🙄
@@FuelforLife001 button? It doesn’t work like that. The problem happened in India
DUDE!!! I managed to use a combination of your gudance and a "shortcut" and I fixed my workstation. You're awesome!!!! My company doesn't allow access to the folder so I had to shortcut my email to my personal machine to get temp admin access. Other than that, followed the instructions, and viola, I'm in. Thanks.
She wrote the tweet with ChatGPT. 😂