Some bad code just broke a billion Windows machines

Поделиться
HTML-код
  • Опубликовано: 13 окт 2024
  • Cybersecurity firm Crowdstrike pushed an update that caused millions of Windows computers to enter recovery mode, triggering the blue screen of death. Learn how the Crowdstrike crash happened and its impact around the world.
    #programming #cybersecurity #thecodereport #crowdstrike #windows
    💬 Chat with Me on Discord
    / discord
    🔗 Resources
    Crowdstrike Statement www.crowdstrik...
    XZ hack • Linux got wrecked by b...
    Rabbit hack • Rabbit R1 makes catast...
    🔥 Get More Content - Upgrade to PRO
    Upgrade at fireship.io/pro
    Use code YT25 for 25% off PRO access
    🎨 My Editor Settings
    Atom One Dark
    vscode-icons
    Fira Code Font
    🔖 Topics Covered
    Crowdstrike failure explained
    How does windows kernel work
    What caused windows computers to crash?
    Cybersecurity fails

Комментарии • 9 тыс.

  • @tekalh7647
    @tekalh7647 2 месяца назад +33854

    The fact that it was an Antivirus that performed the single most successful malware attacks ever is just pure poetry

    • @y7o4ka
      @y7o4ka 2 месяца назад +1627

      introducing: McAfee

    • @lnidux
      @lnidux 2 месяца назад +976

      it's basically spyware anyway

    • @Dexaan
      @Dexaan 2 месяца назад

      Antivirus always becomes the very thing it swears to destroy

    • @SeeThroughist
      @SeeThroughist 2 месяца назад +529

      Security Malware ™

    • @twls153
      @twls153 2 месяца назад +636

      Crowdstrike be like: "Fine I'll do it myself"

  • @ComDenox
    @ComDenox 2 месяца назад +2935

    The alpha move of doing something that would make your stock value crash, but simultaneously freezing the stock market so that it can't.

    • @hackmedia7755
      @hackmedia7755 2 месяца назад +145

      money printer go brrrrt

    • @genshinF2Play
      @genshinF2Play 2 месяца назад +141

      market manipulation is sadly a common occurrence in stock trading. its like a casino, the house always wins.

    • @julianocs87
      @julianocs87 2 месяца назад +19

      So, stonks?

    • @Sandy-o4p
      @Sandy-o4p 2 месяца назад

      I wanna know who's shorting it today. I might open up thinkorswim and view the market.

    • @NeostormXLMAX
      @NeostormXLMAX 2 месяца назад +57

      @@genshinF2Playyeah insider trading legal for senators

  • @realsemig
    @realsemig 2 месяца назад +17155

    "You can't hack a system if the system doesn't work! "
    - Cybersecurity intern pushing the update

    • @matiosjed
      @matiosjed 2 месяца назад +328

      Crowdstrike should call it "a feature"

    • @atifrafique3764
      @atifrafique3764 2 месяца назад +62

      SAUL goodman would say that as his lawyer" your honour my client................."

    • @soloflo
      @soloflo 2 месяца назад +83

      9D chess brother

    • @EdmondDantèsDE
      @EdmondDantèsDE 2 месяца назад

      it was a preemptive strike. check mate, hackermen. 🧠

    • @vectoralphaSec
      @vectoralphaSec 2 месяца назад +26

      What makes you think it was an intern?? Could have been a long term full time employee.

  • @somethingsomething8511
    @somethingsomething8511 2 месяца назад +1876

    What the hell is their deployment process where they didn't catch this in testing? Like they ran the update, saw a blue screen of death and went "looks good, ship it"?

    • @Nmmask
      @Nmmask 2 месяца назад +279

      They didn’t even run it is what it seems like 🤣

    • @drodsou
      @drodsou 2 месяца назад +395

      Yep, everyone can make a mistake, but this is total negligence both by the engineer and the company processes. It's not an obscure error that happen on some machines, but one that bricks every single machine, which is enough evidence that nobody did the most basic test before deploying. And then, deploying it to everyone at once instead of doing it by phases. I hope the company gets privately sued and publicly investigated and punished hard.

    • @sarahfox3652
      @sarahfox3652 2 месяца назад +193

      Never test, never fail is what I always say

    • @SaraMorgan-ym6ue
      @SaraMorgan-ym6ue 2 месяца назад

      meh well it's funny stupid people are switch to linux then it won't happen even though a few months ago the same thing happened to linux🤪🤪🤪🤪🤪🤪🤪🤪🤪🤪🤪🤪
      you gotta love the stupidity of that one I can't even make this shit up it's that funny🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣

    • @lautacolombo
      @lautacolombo 2 месяца назад +114

      “It worked in my pc”

  • @strapatser9427
    @strapatser9427 2 месяца назад +25976

    Hiring 20 more project managers and 40 extra recruiters should prevent this in the future

    • @karlzone2
      @karlzone2 2 месяца назад +2352

      I think we need some business consultants to come up with a good strategy too.

    • @ilfirinms
      @ilfirinms 2 месяца назад +648

      And plenty EU regulations and standards, which had to every company, even village solo baker, implement, or else ...

    • @cultoftranquility9616
      @cultoftranquility9616 2 месяца назад +1316

      or increase the daily scrum meetings with 200%

    • @nebula_M42
      @nebula_M42 2 месяца назад +833

      don't forget a bunch of scrum master asking if there are any blockers every 10 minutes

    • @amira-supermiemie
      @amira-supermiemie 2 месяца назад +348

      And 30 more tech leads😂

  • @nicolascage4438
    @nicolascage4438 2 месяца назад +7875

    My dad: Come on, it's just an internship, what's the worst that could happen?
    Me:

    • @dheeru975
      @dheeru975 2 месяца назад +100

      balls out of my mouth 😢

    • @wolfhunter98
      @wolfhunter98 2 месяца назад +85

      Well you, probably, can't top these guys so you're safe. 😂

    • @koldata4887
      @koldata4887 2 месяца назад +10

      😂😂😂

    • @GhosPoison
      @GhosPoison 2 месяца назад +37

      Since it’s the summer, 100 percent it was an onboard’ee

    • @Sercil00
      @Sercil00 2 месяца назад +80

      I can already see the greentext
      >be me
      >be 22 year old NEET
      >dad wants me to start getting a job because sick of my shit
      >thinks I should work in IT because am nerd loser
      >gets me internship at large cyber security firm because he heard IT sec gets paid the best

  • @manteiga_de_pimenta
    @manteiga_de_pimenta 2 месяца назад +4179

    I was fired last week from my IT job. Yesterday I woke up to a call from the company asking for help, as apparently they didn't have time to hire someone else capable of fixing it, I politely said that I was busy, hung up the call and laughed my ass off of them

    • @zurielsss
      @zurielsss 2 месяца назад +935

      You should ask for 1000 per hour for the task, charge them 6 hrs

    • @Pixcrafts
      @Pixcrafts 2 месяца назад +92

      Poetry!

    • @pratikamatya5190
      @pratikamatya5190 2 месяца назад +70

      😆😆 I am so happy for you 😇🥰

    • @sidthetech7623
      @sidthetech7623 2 месяца назад +308

      Sign back on with a ridiculous contract.

    • @passmelers
      @passmelers 2 месяца назад +413

      you plitely hung up and laughed after the call instead? lame. laugh on the call

  • @debasishraychawdhuri
    @debasishraychawdhuri 2 месяца назад +1124

    If this kind of an upgrade can go through unchecked, imagine how easy it would be to pay one rogue engineer to deploy malicious code that can be exploited later at ease.

    • @sfappetrupavelandrei
      @sfappetrupavelandrei 2 месяца назад +42

      Actually, it is more probably that there was testing done, but this was a crack nobody noticed.
      A couple of years ago, I created a little app to help my brother at his work. He had a colleague who was kind of a mess and she was able to mess the app really bad. My brother was shocked that he couldn't imagine what were the steps this colleague did to mess up the app.

    • @LabGecko
      @LabGecko 2 месяца назад +93

      @@sfappetrupavelandrei If this error, that bricked millions of customers, didn't make it past testing then it wasn't tested, just handwaved at.

    • @Nmmask
      @Nmmask 2 месяца назад +14

      @@sfappetrupavelandreiit’s pretty damn easy to just have a computer run the update on it and see what happens.

    • @alex_lll
      @alex_lll 2 месяца назад +3

      That's why federal government banned Kaspersky AV

    • @Younified_Field
      @Younified_Field 2 месяца назад +6

      That's why open source is the way to go imo. Other independent, knowledable people to check for any malicious snippets of code.

  • @AVSbeats
    @AVSbeats 2 месяца назад +5558

    Everyone: DEV > TEST > PROD
    Crowdstrike: PROD

    • @urooj09
      @urooj09 2 месяца назад +177

      Add a preprod environment as well just in case

    • @LibreGlider
      @LibreGlider 2 месяца назад +392

      I dont always test my code. But when I do, its in production...
      Stay oncall my friends.

    • @philjustyn
      @philjustyn 2 месяца назад +10

      That bug was just perfect... Just perfect! On the last minus details!😂😂😂

    • @ChubbyGunz
      @ChubbyGunz 2 месяца назад +45

      On a Friday........

    • @piccalillipit9211
      @piccalillipit9211 2 месяца назад +30

      NA: DEV > TEST > INSERT MALICIOUS CODE > PROD

  • @clray123
    @clray123 2 месяца назад +1029

    They even named their cybersecurity company to sound like a DDoS attack.

    • @alex_vee
      @alex_vee 2 месяца назад +33

      Underrated comment

    • @bagzhansadvakassov1093
      @bagzhansadvakassov1093 2 месяца назад +4

      ​@@alex_veeindeed

    • @xLuye
      @xLuye 2 месяца назад +5

      Stop business. Drive breaches.

    • @jmasl7
      @jmasl7 2 месяца назад

      well played господин путин

    • @leealex24
      @leealex24 2 месяца назад +4

      Crowdstrike is a cybersecurity firm, do you all feel "secure" now?

  • @unalive_me
    @unalive_me 2 месяца назад +4552

    They did so many things wrong. Push to prod on Friday, fired half their QA staff, no rolling updates, everyone who was supposed to check the commit was like "yep this looks good". Massive single point of failure of a company.

    • @TheDoomerBlox
      @TheDoomerBlox 2 месяца назад +619

      Cripple your quality assurance, push mystery code upon everyone with no rolling release.
      Sounds mildly orchestrated!

    • @Neomadra
      @Neomadra 2 месяца назад +536

      For real... How can you so irresponsible to force push this to all customers at once? I hope upper management is going to pay for this and not the poor junior dev who git pushed the last time

    • @bob_kazamakis
      @bob_kazamakis 2 месяца назад +116

      To be fair, cloudflare also didn’t have rolling deployments for security content, since it typically needs to be addressed immediately. It bit them too.

    • @MyVlogTubes
      @MyVlogTubes 2 месяца назад +5

      Ceo

    • @squirrelsinjacket1804
      @squirrelsinjacket1804 2 месяца назад +107

      @@Neomadra It'll be the dev. That's how the world works.

  • @BeHappyWithUrLife
    @BeHappyWithUrLife 2 месяца назад +236

    As an IT manager this really hurt.
    Having to require a bitlocker key for 1000’s of laptops was a nightmare.

    • @StevePringle
      @StevePringle 2 месяца назад +11

      I bet you have Secure Boot enabled, RAID ON instead of AHCI, Device Encryption enabled, and Bitlocker.
      I turned off that nonsense for my end-users three years ago because I imaged with Acronis True Image
      and had Malwarebytes then. My end-users were up in 3 minutes each while the other techs were scrambling.
      Once, a laptop fell, and the screen was damaged. I moved the storage to another laptop and Bitlocker
      appeared. I had to waste time putting it back and finding an HDMI monitor to attach to see how to disable it.
      Out of 2000 systems, only 3 had to be reimaged with our custom settings and restore the data due to a
      csagent.sys error which would reboot at the login screen or BSOD after POST.

    • @dacummins33
      @dacummins33 2 месяца назад

      If you're an IT Manager, Why are you running Windows in the first place? Maybe if allow it for the Solitaire players, if they weren't connected to critical systems. In my 37 years in IT, not counting 10 years before dabbling in Apple using hex bc we couldn't afford a compiler. In my professional career, we used Windows on exactly 0 critical systems. Mostly worked in TS and Critical Systems - and they all used Unix then Linux (or close variants), and Oracle. How many Windows systems at DARPA? 0. I designed the Linux Cluster that runs All back-office traffic for ticketing, baggage, transfer services, etc... internationally - no Windows, zero. Only front-office, non-critical systems were Windows - and I Guarantee that system was unaffected. Simple Solution - don't run Windows... If the company insists on running Windows Apps, run them in a VM Windows Emulator on Linux! No problems... Windows had been a poor contender from "Day 0", as a rip-off of Unix and X-Windows/Motif. Get a real CTO/CIO... SAD. Feel sorry for your troubles - I can sympathize, but if you don't correct the base error, Windows, I can't sympathize with your future crashes...

  • @Is_GrayKing
    @Is_GrayKing 2 месяца назад +14498

    This is what Y2K wished it would be

    • @passby8070
      @passby8070 2 месяца назад +112

      😂😂😂

    • @andrewstewrt2141
      @andrewstewrt2141 2 месяца назад +405

      y2k38 will avenge its dad

    • @SOTPOD
      @SOTPOD 2 месяца назад +378

      gunna tell my kids this was Y2K

    • @DawnApon
      @DawnApon 2 месяца назад +150

      @@andrewstewrt2141 I heard its y2k38, and also already working to be mitigated today like y2k was.

    • @tomasprochazka6198
      @tomasprochazka6198 2 месяца назад

      ​@@andrewstewrt2141 haven't seen 32bit for a few years, I expect none in 2038 (overflow of Unix timestamp stored as int32)

  • @lordromanyx
    @lordromanyx 2 месяца назад +2008

    As someone who works in IT for a hospital, I went in thinking it was going to be a normal day only to realize the fire I walked into. Spent the entire day just walking around fixing each one.

    • @gravitatemortuus1080
      @gravitatemortuus1080 2 месяца назад +125

      Same, I also work for a large hospital group. On top we use Azure and that made this even worse for a few reasons, one you can't get to safe mode with the servers. Seems solutions late in day we were able to recover a lot of servers.

    • @iljay5718
      @iljay5718 2 месяца назад +206

      Poor guy had to actually work

    • @midoevil7
      @midoevil7 2 месяца назад +52

      - This can fixed in 10 minutes .. right? right ?
      - ......................................................

    • @PhilipBlank
      @PhilipBlank 2 месяца назад +2

      Oh man

    • @MrTotalAhole
      @MrTotalAhole 2 месяца назад +46

      Ohh, so you actually earned that salary youre being paid, for that day. /s
      Please dont take that seriously. I work for a big company and love my IT guy and get along well with him. Not because he ever has to fix anything of mine. But because he cool as hell. I always make sure to save him a cupcake or bagel and send him a Teams whenever we plebs are remembered and thrown some crumbs.

  • @nsncxd
    @nsncxd 2 месяца назад +1912

    At where I work we didn’t even notice the blue screen of death that affected over a billion windows computers because all of our shit crashing is already normal operating procedure.

    • @aliveagain
      @aliveagain 2 месяца назад +26

      lool

    • @OK-cp8qw
      @OK-cp8qw 2 месяца назад +15

      Kinda true tho

    • @purpleowl2075
      @purpleowl2075 2 месяца назад +22

      Are you a GP and do you work for the NHS?

    • @YaySyu
      @YaySyu 2 месяца назад +47

      The bluesceen gave my computer some much needed rest. It's been running hot 24/7 since I got hit with ransomware a few weeks ago

    • @jalalelhajouji1578
      @jalalelhajouji1578 2 месяца назад +4

      Brilliant

  • @vkmi5776
    @vkmi5776 2 месяца назад +133

    I also love the fact that Crowdstrike CEO, Kurtz, was McAfee CEO when something similar happened with Windows XP in 2010

    • @noWoodsman
      @noWoodsman 2 месяца назад +12

      He was the CTO, not CEO.

    • @n_core
      @n_core 2 месяца назад +23

      This taints his portfolio even more. Two of the biggest IT incidents in history are under his name. I'm sure this raises some wary for future investors.

    • @Coach-rq6jx
      @Coach-rq6jx 2 месяца назад +12

      "Sir, the second bluescreen hit the monitors".

    • @nah131
      @nah131 2 месяца назад +1

      I want to be like him somedays

  • @entropydenied5791
    @entropydenied5791 2 месяца назад +4875

    The one guy who uploaded this code is not to blame for this. This was a cascade failure in the testing of said code that points to bad organization.

    • @MakeItMakeSense285
      @MakeItMakeSense285 2 месяца назад +396

      One can assume that many heads will roll over this. Lawsuits are going to be filed.

    • @tomorbataar5922
      @tomorbataar5922 2 месяца назад

      @@MakeItMakeSense285 I guess Cloudstrike is done as a company?

    • @UTubeHandlesSuck
      @UTubeHandlesSuck 2 месяца назад

      @@MakeItMakeSense285 One could assume that, but they would be wrong. Once every responsible party has been identified, they will toss excuses and blame underlings with excrement rolling downhill as always until the person who had the very least to do with it is sent home tarred and feathered while the ones who actually caused the problem go along their merry way.

    • @MrGlugz
      @MrGlugz 2 месяца назад +816

      For large scale disasters, it's never a few employees. The high level management and CEOs are ALWAYS to blame. Mistakes from regular employees are caught by robust systems of checks and balances. If there is no such system in place, it's because the leaders of the company didn't approve the necessary budget for it.
      But of course, they are using some low level employee as a scapegoat. Higher ups never assume responsibility.

    • @JeremyAndersonBoise
      @JeremyAndersonBoise 2 месяца назад +244

      Yes, this is a release management failure, it never should have been possible to release this code. Where is the canary testing?

  • @MrDaAsif
    @MrDaAsif 2 месяца назад +998

    Who needs malware with cyber security like this

    • @tlumme
      @tlumme 2 месяца назад +5

      co-pilot: ooh..human error .. Fxxxxck!

    • @ShawnFumo
      @ShawnFumo 2 месяца назад +7

      And it sounds like the CTO was the CTO of McAfee back when they broke a bunch of Win XP machines back in 2010. Didn’t learn the lesson it seems like…

  • @OneAndOnlyJackSchitt
    @OneAndOnlyJackSchitt 2 месяца назад +1125

    Meanwhile, in Crowdstrike's QA department, the one guy left after all the layoffs pastes some code into ChatGPT and asks "Hey, bro. Is this code any good?" to which it replies "Yeah, bro. Totally safe to push out to production. You like the color blue, right? No reason."

  • @mo-issa
    @mo-issa 2 месяца назад +108

    It's insane to me how dependent we are on a single company.

    • @IDARYASSYN
      @IDARYASSYN 2 месяца назад +1

      Then go ahead and create for us a company for that, Boomer

    • @xx_____saint____
      @xx_____saint____ 2 месяца назад

      @@IDARYASSYNare you stupid?

    • @TheAnantaSesa
      @TheAnantaSesa 2 месяца назад

      @@IDARYASSYNthey could already just use Linux but prefer to save money on programming their applications.

    • @mo-issa
      @mo-issa 2 месяца назад +3

      @@IDARYASSYN I think you missed my point. My point was that this type of monopoly is scary and gives private companies like Microsoft too much power.

    • @pyxalated
      @pyxalated Месяц назад

      @@IDARYASSYN are you stupid?

  • @PastaAivo
    @PastaAivo 2 месяца назад +1443

    As someone who constantly complains about automatic forced updates, I've never felt so vindicated.

    • @sincronus
      @sincronus 2 месяца назад +87

      Exactly. These mandatory forced updates in the name of "security" break things more often than fixing them.
      I never install updates on my devices on the first day, I do it manually after a few days after reading feedback from other users. This way, I get to know if it's something that's really needed and doesn't brick/slow down my device.

    • @TwentyEightySeven
      @TwentyEightySeven 2 месяца назад +19

      I have never liked automatic updates. Back in the day you could defer everything till when you were ready, so your not done over midway through something important!

    • @Goromajima61
      @Goromajima61 2 месяца назад +4

      YES

    • @moonray2587
      @moonray2587 2 месяца назад +12

      Nah I just got saved cause my laptop sucks so much it didn’t even update 💀

    • @mine.moment
      @mine.moment 2 месяца назад +12

      As someone who, by default, debloats all Windows crapwares and block all those forced automatic updates, I see this as an absolute W.

  • @SergiusXVII
    @SergiusXVII 2 месяца назад +9541

    That poor intern…

    • @anj000
      @anj000 2 месяца назад +850

      We need to have an interview with him. What a story to tell to grandchildren.

    • @squidwardfromua
      @squidwardfromua 2 месяца назад +88

      My average life

    • @exnozgaming5657
      @exnozgaming5657 2 месяца назад +670

      @@anj000 "Kids, my feats are greater then any hacker every existed"

    • @CaponeBlackBusiness
      @CaponeBlackBusiness 2 месяца назад

      Hitler

    • @user-ew5vj1sl1u
      @user-ew5vj1sl1u 2 месяца назад +21

      Sorry

  • @jacobharmon6162
    @jacobharmon6162 2 месяца назад +5344

    Another win for the "remind me later" to every update gang

    • @pWoLiAn
      @pWoLiAn 2 месяца назад +427

      crowdstrike updates happen automatically under the hood😂😂

    • @1ycan-eu9ji
      @1ycan-eu9ji 2 месяца назад +60

      just turn them off with group policy (you need windows 10/11 pro)

    • @christopherg2347
      @christopherg2347 2 месяца назад

      @@1ycan-eu9ji I _seriously_ doubt Cloudstrike uses the Windows Update process...

    • @ayankhan-xz7xc
      @ayankhan-xz7xc 2 месяца назад

      ​@@1ycan-eu9ji and you have access to group policy of your work laptop??

    • @zamiyaFlow
      @zamiyaFlow 2 месяца назад +138

      @@1ycan-eu9ji huh, too bad microsoft forces their home garbage unto every single retail-end desktop and laptop unit

  • @SeaMonkey137
    @SeaMonkey137 2 месяца назад +64

    My boss and I had just had a conversation the day before about the risks associated with so many industries relying on "best practices" (French for "use what all the CEO's friends are using") for platform and enterprise software decisions.

    • @genxx2724
      @genxx2724 2 месяца назад

      I’m not in the tech world, but when I hear the term “best practices” I know I’m dealing with an idiot who is trying to sound impressive.

  • @ronniesunshine1163
    @ronniesunshine1163 2 месяца назад +943

    The bluescreens scared the shit out of me. I was working at 10:30pm cutting over network switches in a data center when my boss' laptop bluescreened. We took a break while a network admin remotely checked the connectivity of the new switches. He called us 15 minutes later saying his laptop bluescreened too and that he got a call from IT Management saying multiple servers are down. We thought the network was getting ransacked because of something we did. Luckily it was just a worldwide outage and not some wild network exploit bluescreening any device that connects via wifi.

    • @MrOneeyedpete
      @MrOneeyedpete 2 месяца назад +115

      lol yeah lucky it was world wide and not just you BAHAHA

    • @markh.6687
      @markh.6687 2 месяца назад +177

      "Luckily it was just a worldwide outage."
      Translation: "It wasn't me!" :)

    • @MateusViccari
      @MateusViccari 2 месяца назад +33

      @@MrOneeyedpete Yes because if it's the whole world it's not his fault, nor he is the one that will need to fix it.

    • @shantidutbansode2
      @shantidutbansode2 2 месяца назад +13

      I can totally feel this situation 😂

    • @ashishsharma__
      @ashishsharma__ 2 месяца назад +2

      😂😂😂😂

  • @Seed
    @Seed 2 месяца назад +3046

    Yeah, at this point it won't be long before the wikipedia editors need to update the word "is" to "was" on Crowdstrike's wikipedia page...

    • @JStack
      @JStack 2 месяца назад +114

      Somehow they only lost 11% of their value

    • @GhosPoison
      @GhosPoison 2 месяца назад +417

      @@JStackthe computers are down, wait until they are rebooted

    • @OatmealTheCrazy
      @OatmealTheCrazy 2 месяца назад +26

      ​@@JStack 19

    • @MDxGano
      @MDxGano 2 месяца назад +10

      @@JStack because shit happens and people get over it.

    • @aisle_of_view
      @aisle_of_view 2 месяца назад +100

      @@MDxGano Depends if Crowdstrike becomes financially liable for the $ loss to businesses.

  • @FaisalAbidi
    @FaisalAbidi 2 месяца назад +814

    Good point that the real issue is not that a human made an error, but that tech leaders everywhere decided the best thing to do is to have a single point of failure.

    • @Flip86x
      @Flip86x 2 месяца назад +28

      It's like that old saying, don't put all your eggs in one basket.

    • @iamacsel87
      @iamacsel87 2 месяца назад +19

      what would the alternative be (regarding this point) from the client side?
      have it's own security team? because the level of expertise will never reach that of a company who's only goal si security

    • @jmasl7
      @jmasl7 2 месяца назад

      while spewing out steaming mounds of PowerPoint on avoiding a single point of failure, 'silos' and all the other bad bears of 'technical' sorcery

    • @SWL_Jamey
      @SWL_Jamey 2 месяца назад +15

      Single point of failure is natural thing, would you rather have complexity of one system of systems, our would you want to deal with two complexities of two different systems of systems? Do we even have amount of people capable of supporting that work ? and what exactly is that one point of failure ? windows ? crowdstrike? cpu architecture? capitalism?

    • @karolgarbocz7044
      @karolgarbocz7044 2 месяца назад +3

      Actually regulations demand installing crowdstrike, so thank you captain state

  • @xFact-ory
    @xFact-ory 2 месяца назад +56

    Started my new job at CrowdStrike today. Unplugged a socket marked "do not unplug" to charge my phone. A lot of commotion in the office soon after that. No idea what it was all about.

  • @Badmunky64
    @Badmunky64 2 месяца назад +818

    My company uses CS. The only reason we could function at all today was because my co-worker (who's working the late shift) noticed his laptop BSOD at midnight, right before going to bed, and sounded the alarm. Thanks to that and our admin working all night they restored most of our servers. I got to help our users.

    • @Aliceintraining
      @Aliceintraining 2 месяца назад +105

      it always comes down to the human factor, if you get the best and treat them well, they will catch the problem before it sinks you, good well paid employes are the best security a company can have

    • @kittydaddy2023
      @kittydaddy2023 2 месяца назад +18

      I'm not in IT, but I watch Fireship and I'm here to help

    • @claushellsing
      @claushellsing 2 месяца назад +4

      is your company still going to use CS ?

    • @furrepanther
      @furrepanther 2 месяца назад +14

      Whatever happened to using a test environment before rolling out changes to live servers??

    • @DrewingIt
      @DrewingIt 2 месяца назад +40

      you should highlight this to your superior that will most likely reward this gem of an employee and not just get internet social credits here on youtube. seems like the good guy thing to do with what you know (your boss might know but highlighting will secure the bag for your colleague). cheers

  • @PlamereDoesntExist
    @PlamereDoesntExist 2 месяца назад +2779

    Currently surfing youtube as my entire team is fully blocked by this 😂

  • @juanpablo1834
    @juanpablo1834 2 месяца назад +334

    as an IT guy, I literally came in into the office, it was supposed to be an easy day, said let's get some coffee, finish some tickets, and then I log in and I see hundred of messages and a line of people waiting for me. And then it hit me. The ldap server was crashed. We were all fucked.

    • @windycitybeats6724
      @windycitybeats6724 2 месяца назад +18

      Ohh dam brother, me too! 😂 thought this was gonna be an easy Friday

    • @programmable_life
      @programmable_life 2 месяца назад +9

      Have a good weekend

    • @f.faucon6681
      @f.faucon6681 2 месяца назад +21

      Cheers from an IT Incident and Major Incident Manager. Days that start before the commute with a lot of messages, alerts, questions are tough. Days that are calm and ends with a lot of messages, alerts, questions beginning during the first steps of the commute back home are also tough. There's a superstition where I work: nobody better utter the statement that today is calm... ;)
      Luckily I didn't have to deal with yesterday that much, no Crowdstrike in the company. Only had to check with several suppliers if they were OK too.
      Have a good weekend!

    • @rumfordc
      @rumfordc 2 месяца назад +8

      Lots of work to do! People depend on us!
      **opens youtube**

    • @siroliver8367
      @siroliver8367 2 месяца назад +1

      installing an third party edr on an ldap server 😅

  • @garymartin9777
    @garymartin9777 2 месяца назад +104

    this is why you don't push untested and uncontrolled updates to the world. Every single update has to be rigorously tested in-house and to a control group before an even larger test before pushing out the door.

    • @neighborhoodtroll
      @neighborhoodtroll 2 месяца назад +9

      burh, its like telling a person to study well in order to pass the exam....like duhhh! This is literally like secret service laughably missing a line of sight sniper

    • @RoughRaiders13
      @RoughRaiders13 2 месяца назад

      Kind of like how the government was pushing everyone to get barely tested COVID vaccines that don't keep you from getting or spreading the virus injected into your body. 😂😂😂😂😂

    • @NightmareRex6
      @NightmareRex6 2 месяца назад +1

      and it can happen now, we now are FORCED to accept ALL updates or block ALL updates, when USED to be able to select parts wand and dont!, they also did this in 2021 for the human body.

    • @muysantos2859
      @muysantos2859 2 месяца назад

      you'll never know issues until you push it (skin in the game)

  • @geeshta
    @geeshta 2 месяца назад +2129

    "Let's give remote kernel level access to our critical systems to one proprietary nontransparent company as a part of our security"
    - the entire fking world apparently

    • @alxk3995
      @alxk3995 2 месяца назад +250

      "we need to do cyber security" -"oh that's expensive and complicated" - "look, this company does it for 150 bucks a month"
      😂

    • @raven4k998
      @raven4k998 2 месяца назад

      @@alxk3995 first Microsoft makes old computers obsolete with windows 11 now this makes you wonder what is really going on with Microsoft because this is worse then y2k because while y2k was supposed to make a crash like this happen one it was found and fixed before it could happen were as this happened so you have to seriously wonder was it really an over sight or was it Deliberate🤔🤔🤣🤣🤪🤪😜😜🤪🤪🤣🤣

    • @LaughingMan44
      @LaughingMan44 2 месяца назад

      ​@@alxk3995Per device

    • @kapitan5888
      @kapitan5888 2 месяца назад +10

      Similar to movie Decisions too

    • @hiddendrifts
      @hiddendrifts 2 месяца назад +89

      tbf most of the people in charge at these places probably aren't technologically literate enough to realize the potential consequences of doing that. it's like when you go to a hospital or take meds and just trust that the people in charge have your best interest at heart bc you're out of your depth in those fields

  • @ozzyphantom
    @ozzyphantom 2 месяца назад +328

    The only thing that got me through my 18.5 hour shift today fixing this issue at dozens upon dozens of gas stations was knowing there were thousands of other fellow IT guys and girls across the world holding down the fort with me

    • @aja749
      @aja749 2 месяца назад +43

      Thank you for your service.

    • @azurev2258
      @azurev2258 2 месяца назад +5

      o7

    • @jaywulf
      @jaywulf 2 месяца назад +20

      I appreciate your work. I hope you get more than a pizza from your boss.

    • @Spartan_Tanner
      @Spartan_Tanner 2 месяца назад +2

      On the frontline holding the tide, massive respect

    • @smallfaucet
      @smallfaucet 2 месяца назад

      Nice job security.

  • @CosasCotidianas
    @CosasCotidianas 2 месяца назад +780

    That's what the director of technology of a public department told me a while ago when I asked him why there weren't using Linux servers instead of paying thousands in licenses: "if we have a failure, we can blame Microsoft or any other company, that's the only purpose of licenses".

    • @someguy4915
      @someguy4915 2 месяца назад +78

      When a company switches to Linux for their servers they still pay money, just not thousands but hundreds of thousands (same as Windows, sometimes RHEL is even more expensive).
      Licenses are cheap, support is expensive.
      Windows support, while sometimes frustrating and slow is fairly good and dedicated.
      Linux support is a joke.
      That's fine if you're running a homeserver or some small company that cannot afford a $501 license, but for larger companies where the cost of downtime/IT staff hourly cost is more than $501 this makes no sense to run on Linux in most cases.

    • @foaly8
      @foaly8 2 месяца назад +127

      @@someguy4915that must be why most servers are running linux

    • @Y2B123
      @Y2B123 2 месяца назад +89

      @@someguy4915 That is why Microsoft offers Linux on its system. Just to give a shout-out to its competitor for no particular reason (totally not because Linux is prevalent on servers and developers' machines).

    • @ruk_necahual
      @ruk_necahual 2 месяца назад +53

      ​@@someguy4915 For a larger company it's still better to use Linux, and there are dedicated distros (some even licensed for way too much money) that will cover all of your corporate needs. Besides which, all your backend is using Linux anyway. If you see Windows, it's more than likely just serving as the interface with which you're engaging the Linux systems under the hood.

    • @karmatraining
      @karmatraining 2 месяца назад

      @@someguy4915 somebody should tell AWS to shut down all their datacentres!!!

  • @pcdeltalink036
    @pcdeltalink036 2 месяца назад +21

    As someone who was working in their local hospital lab that night this all went down I can tell you it was bad. I distinctly remember fielding a call from one of the lab managers "How far behind are you guys?" "...I don't even know. I barely know what way is up right now. Minimum 2 hours or more behind on every lab result."
    What's crazier is somehow the computer I was working on survived and was basically the only working PC in the lab for 3 or 4 hours. No idea how it made it through. It went down a couple times but I was able to get it back up (it was also occasionally randomly closing programs I was in, etc.) and so I at least had something rather than nothing.

  • @ringkunmori
    @ringkunmori 2 месяца назад +373

    In Mr Robot they spent an entire season just to pull off a hack as damaging like this. Crowd Strike did it in one update.

    • @SoyFaii
      @SoyFaii 2 месяца назад +24

      reality ALWAYS surpasses fiction

    • @SargentD4
      @SargentD4 2 месяца назад

      Well yeah it’s their name, crowd strike. Population attack just sounds too straight forward.

    • @serenityskies4477
      @serenityskies4477 2 месяца назад

      ROTFFL!

  • @ToddHowardWithAGun
    @ToddHowardWithAGun 2 месяца назад +2060

    >force automatic updates to prevent zero day attacks
    >create worst zero day issue in the history of IT
    hrm

    • @someguy4915
      @someguy4915 2 месяца назад +29

      Worst zero day issue? First of all this is not a zero day exploit lol second of all worst issue in the history of IT then you have a very short history lmao

    • @toddhoward1892
      @toddhoward1892 2 месяца назад +1

      Woah...

    • @ThisIsTheInternet
      @ThisIsTheInternet 2 месяца назад +115

      @@someguy4915 Go ahead, list a worse issue than this

    • @markmywords3817
      @markmywords3817 2 месяца назад +2

      ​​@@ThisIsTheInternet on top of my head, the ILOVEYOU virus was much more damaging.
      For this recent one you'd have to have Crowdstrike installed in the first place.
      The ILOVEYOU virus only required you to open an email with ILOVEYOU subject line, do all sorts of nasty things upon reboot (deleting/hide files, replacing files with copies of the worm, etc). Then it uses your address book to send the same email to all of your contacts.
      Your contacts that trust you, would then be tempted to open it because of the presumed declaration of love in an email.
      And upon opening email, the same code would automatically run on your Windows machine.
      The fact that the worm was visual basic script file also allowed other hackers to modify it to do more damaging things, change the email subject line, etc.
      It was so popularly damaging at the time that it had variants too like the recent pandemic did.

    • @some1purple
      @some1purple 2 месяца назад +56

      ​@@someguy4915A 0day only means that they have 0 days to fix it. It says nothing about the severity of the vulnerability itself. A DOS in the wild is still a 0day.

  • @y7o4ka
    @y7o4ka 2 месяца назад +1225

    fun fact: modern way of installing kernel/module updates on linux leaves a backup that prevents this exact issue from ever happening

    • @_tr11
      @_tr11 2 месяца назад +42

      wait rly? that's so cool

    • @katech6020
      @katech6020 2 месяца назад

      @@_tr11 for my installation currently I basically have 4 kernels (latest release with its backup, and LTS with its backup) and you can install as many kernels as you want

    • @devanshushankar7784
      @devanshushankar7784 2 месяца назад +88

      Yeah, that called snapshot

    • @kerrydaniels8460
      @kerrydaniels8460 2 месяца назад +55

      Literally has the older kernels available to begin with.

    • @2204happy
      @2204happy 2 месяца назад +171

      @@_tr11 yep, most distros save the old kernel image when the kernel is updated, and only deletes it when the kernel is updated yet again, and the kernel which replaced it then becomes the backup. So there is always at least two kernel images on the system at any one time, and the old one can be used if the newer one is borked.

  • @crazy137788
    @crazy137788 2 месяца назад +14

    I'm in Japan and was very confused on Friday when the lady at McDonald's was writing my order down on a piece of paper. This explains everything.

  • @HankTVsux
    @HankTVsux 2 месяца назад +651

    I do maintenance on commercial airplanes, mostly in between international flights. Today I have done exactly zero maintenance on a single airplane during my twelve hour shift.

    • @HankTVsux
      @HankTVsux 2 месяца назад +156

      @RyanClone winning would be getting sent home!

    • @ActionScripter
      @ActionScripter 2 месяца назад +20

      "Anyway, none of my computers were affected by this bug."

    • @pancakeluxury23
      @pancakeluxury23 2 месяца назад +11

      @@HankTVsuxhow do you get into that? And are all your shifts 12 hours? I have an interest in planes but I also have an interest in free time so I don’t know.

    • @HankTVsux
      @HankTVsux 2 месяца назад +31

      @@pancakeluxury23 I love that question. I got an in through a friend so I was very lucky. The easiest way is to start by throwing bags and start working your way up or to go to school and get your license.
      I can't tell if you like the idea of 12s with that comment. I took them specifically because they come with 4 day weekends (fly anywhere for free+4 day weekends. You do the math ;) )! Though, they are pretty rare and I'll be back on my normal schedule after the summer rush.
      The industry is absolutely booming right now (my company hired 40,000 people since pandemic) and it's a relatively stable industry with great pay opportunities and an unlimited ladder to climb if you are just reasonably patient.
      I work with a lot of folks who started by throwing bags and some are going on to be pilots or whatever else. One guy is dreaming of being a buyer for the company (that is, buying planes. As a job.). Myself, as a mechanic with no aviation or formal mechanical experience will be making six figures in about 6 years.

    • @pancakeluxury23
      @pancakeluxury23 2 месяца назад +10

      @@HankTVsux 4 day weekends?? Say less. I mean I know that’s not guaranteed but the reason I ask is because I’m nowhere in life right now and unless I want to flip burgers for the rest of my life I’ve got to choose something that pays enough to actually live life. I also just happen to like planes anyways. I like cars too but I don’t wanna work at a jiffy lube or assembly line for the rest of my life either. I’m 23.

  • @d0mbee87
    @d0mbee87 2 месяца назад +771

    i can never understand the fact that there wasn't one person who could just install said update on a test pc to check if maybe "something" broke, before updating the whole world

    • @billfarley9015
      @billfarley9015 2 месяца назад +77

      Both Microsoft and Clownstrike should have tested the update before releasing it.

    • @dumbuz
      @dumbuz 2 месяца назад

      @@billfarley9015 the hell does Microsoft have to do with Crowdstrike's software?

    • @skya6863
      @skya6863 2 месяца назад +160

      ​@@billfarley9015 nah don't blame this on Microsoft. Microsoft can't test every single update to every single kernel driver out there seperately.

    • @goombacraft
      @goombacraft 2 месяца назад +116

      you can't blame Microsoft here. There's no way that they are responsible for checking what third parties want to install on their own computers. There are legitimate reasons to install and run malware etc.

    • @kenshn22828
      @kenshn22828 2 месяца назад +8

      Definitely what was supposed to happen someone probably skipped a couple SDLC steps and merged directly to Main

  • @SpaceLordof75
    @SpaceLordof75 2 месяца назад +290

    I spent 12 hours today, starting at midnight, fixing this issue.
    I had to boot into safe mode, and delete the new CS update. We had ~3500 machines impacted.
    It sucked.

    • @tbcrosby
      @tbcrosby 2 месяца назад +41

      same here brother, my mind is mush after 12+ hours of punching in bitlocker codes 🥴

    • @kklol07
      @kklol07 2 месяца назад +11

      Damn guys. Thanks to you all

    • @YahiyaJasem
      @YahiyaJasem 2 месяца назад +14

      Bless you guys , I hope you get the rest you need after this

    • @ChubbyGunz
      @ChubbyGunz 2 месяца назад +18

      Yup 12 hour day. Must be a world record for all global IT/OPs guys accumulating overtime at same time also lol.

    • @Masicka123
      @Masicka123 2 месяца назад

      @@tbcrosby Jesus, I feel for you dude. Thankfully, none of our clients use Crowdstrike but rather Bitdefender.

  • @suspense_comix3237
    @suspense_comix3237 2 месяца назад +20

    I don’t think I’ve ever seen so few aeroplanes flying over the US ever since 9/11 when the US had to shutdown US Airspace.

    • @Avarren
      @Avarren 2 месяца назад +5

      Were you comatose for most of 2020?

  • @twezo
    @twezo 2 месяца назад +631

    The fact that they have so many clients using the same antivirus software is a security risk in itself

    • @odenkaz
      @odenkaz 2 месяца назад +16

      yeah this pretty much stated that everyone went with what's trendy...now we know for sure alot of the companies will shift to other better equipped competitors

    • @ApexGale
      @ApexGale 2 месяца назад +48

      ​@@odenkaz with all due respect...it's less "what's trendy" and more "if i factor in consistency and cost, what is the best option?" having multiple high profile clients is an implicit selling point, it means your cybersecurity is top notch.
      the product itself wasn't an issue, the issue was the CI/CD pipeline was not followed appropriately. An intern or lower level dev would not even have the capacity to push this into production. Normally they would have it in a development branch, then a testing branch, and only after passing testing would it be merged into production by a higher up who reviews the pull request. General protocol for big companies is also to roll out updates over a duration of time - it's why you sometimes have a friend complaining about a new update to an app that you haven't received yet. It isn't supposed to go out to every user instantly, it's supposed to allow time for issues to be caught if something still goes wrong in production.
      "Too many people reliant on one app" isn't really a problem.

    • @Wellimanewguy
      @Wellimanewguy 2 месяца назад +23

      @@ApexGale it isn't a problem until the one app everyone relies on fails.
      if you take the keystone out of an arch, what happens?

    • @xenonex8151
      @xenonex8151 2 месяца назад +15

      This is why competition is important since Russia wasn’t even affected by this

    • @ShayPatrickCormacTHEHUNTER
      @ShayPatrickCormacTHEHUNTER 2 месяца назад +5

      @@ApexGale No it doesnt. It means youre popular. This logical fallacy led to this lol. Play stupid games, win stupid prices.

  • @Sysyphus
    @Sysyphus 2 месяца назад +972

    "Oh this isn't a big deal, you can just reboot into safe mode and change the extension!"
    Bitlocker: "Hold my beer."

    • @RippanCSGO
      @RippanCSGO 2 месяца назад +80

      Also add that the average office worker has very.. very little knowledge about computer outside their field. Just adding a printer is a 2 day job

    • @helton3425
      @helton3425 2 месяца назад +19

      Seeing as there are a lot of non-tech savvy usuals in computer jobs who only know enough to get by.
      I do not think they would even know how to access safe mode. Let alone anything else

    • @Eagle2565
      @Eagle2565 2 месяца назад +77

      @@RippanCSGO you can work in it for 20 years, but when it comes to printers all hope is lost, even for the veterans. Id rather chop off my leg, than figure out why a fucking printer is not working.

    • @MichaelDoran23
      @MichaelDoran23 2 месяца назад +6

      ​@Eagle2565 the sooner we figure out a paperless society, the better. You are a man that knows my pain with printers 😂

    • @jean-francoisaubry
      @jean-francoisaubry 2 месяца назад +28

      @@Eagle2565 Printer driver up-to-date (check), PDF file to print in correct format (check), Printing Job in queue (check), Printer ready (check), Paper in the printer (check), Printer full of toner (check), Other PC can print the same document on this MotherFucking printer (check)...

  • @AZombie48
    @AZombie48 2 месяца назад +774

    I remember watching a presentation from a software developer who talked, in part, about why he’s so adamant about defining best practices and building quality, tested code. He said that one day, some software bug is going to be pushed out by an exhausted intern, and 10,000 people are going to die. And when that happens , our profession will go through a reckoning. It behooves us to make our own standards now so that we protect ourselves when that disaster finally happens.
    I know this isn’t as bad as 10,000 people dying. And I don’t think this is the moment he was talking about. But the fact that so much damage was done by a little mistake really has me believing that he was totally right.

    • @cdgonepotatoes4219
      @cdgonepotatoes4219 2 месяца назад +86

      The butcher's bill has yet to come out to report the total damages.

    • @macodev
      @macodev 2 месяца назад +20

      Uncle Bob Martin

    • @BootyRealDreamMurMurs
      @BootyRealDreamMurMurs 2 месяца назад

      considering the scale of this and millions of people wre definitely affected, percentage of which are in the healthcare industry (hospitals, pharmacies, etc.)
      its unfortunately already a guarantee that more than 10,000 people has died from this incident... (people who needed urgent care, people who needed certain doses of a particular drug, malfunctioning medical equipments and devices on the middle of a medical procedure like surgery, etc.etc.)

    • @maganashaker167
      @maganashaker167 2 месяца назад +73

      Some people definitely died from this disaster

    • @Shaker626
      @Shaker626 2 месяца назад +7

      Open source helps avoid this issue.

  • @houssemedyn5678
    @houssemedyn5678 2 месяца назад +16

    I cant understand how such updates are not sample tested beforehand. It makes absolutely no sense, its basics

    • @bluesteel1
      @bluesteel1 2 месяца назад +1

      Just happened to me today. PMs pushing for deadlines knowing the product isnt even complete

    • @rakkis1576
      @rakkis1576 2 месяца назад +2

      @@bluesteel1 Yeah it can be like that. Couple with layoffs making everyone (except the people up top) overworked and you got a recipe for disaster. Naturally, the people that created this situation gets little blame, if any.

  • @mayureshrawal
    @mayureshrawal 2 месяца назад +460

    I work in IT support and today's night I witnessed all stages of emotions - Confused, Shocked, Defeat, Acception, Resignation.

    • @David-gp3fd
      @David-gp3fd 2 месяца назад +12

      todays night?..aka tonight

    • @The_Savage_Wombat
      @The_Savage_Wombat 2 месяца назад +8

      @@David-gp3fd Acception?

    • @randompersonyoudontknow5763
      @randompersonyoudontknow5763 2 месяца назад

      NO WONDER MY PC CRASHES WHEN I PLAY GAMES THAT DON'T EVEN DO CRASHES OFTEN, also are you ok from this trauma event?

    • @gsnyder2007
      @gsnyder2007 2 месяца назад

      Job security

    • @MsParzanini
      @MsParzanini 2 месяца назад +12

      @@David-gp3fd guy is overworked, give him a break hahah

  • @chengmunwai
    @chengmunwai 2 месяца назад +502

    3:33 - This is a very important point. Most decisions in big corporations are made on a "will this get me fired" basis instead of actual pros/cons considerations. As long as the decision maker gets to keep his/her job by blaming someone else for mishaps, that decision is a good decision.

    • @Steamrick
      @Steamrick 2 месяца назад +14

      As someone working for an IT service provider, there's the occasional ticket or even project that feels like this.
      It's not that internal IT wouldn't be capable of it but rather they're too cowardly to push the button.

    • @Triad72
      @Triad72 2 месяца назад +2

      So you're saying there's lots of money to be made by simply having a willing to accept responsibility when something bad happens?

    • @SWL_Jamey
      @SWL_Jamey 2 месяца назад

      @@Triad72 no. HE saying something else. YOU can mean multiple things. company, government is rule based system. but that does not mean you are bound ONLY by those rules, every lawyer, judge swears to bible not because they are worshippers of god, but because it is a symbol of higher moral principles. being saboteur is easy, yes..

    • @vullord666
      @vullord666 2 месяца назад

      On a certain level there is credence to be given to the idea of "let the specialists handle it" and to the benefit of the doubt I like to think at least some of the organizations affected just aren't capable of running their own cybersecurity. However, for so many of them, including the government, it's a major question of why they don't just invest in having their own in-house team for cybersecurity. It'd be more expensive but they have the resources and it would also keep probably one of the most important jobs in the modern world, in house. Like I really don't love the idea of services as important as 911 being reliant on a third party software provider.

    • @jonnysokkoatduckdotcom
      @jonnysokkoatduckdotcom 2 месяца назад

      @@chengmunwai sad

  • @andersonklein3587
    @andersonklein3587 2 месяца назад +364

    Anti virus running on Kernel Mode: Strike 1
    Machine has no way to automatically rollback to last good boot: Strike 2
    Deploys updates worldwide simultaneously without production testing: Strike 3
    I really think that management knowing nothing about computers at a time computers are key infrastructure is a terrible idea.

    • @jhchnc
      @jhchnc 2 месяца назад

      Preach

    • @JorissenJan
      @JorissenJan 2 месяца назад +33

      Add some Bitlocker in the mix, and voila, a perfect party

    • @herp_derpingson
      @herp_derpingson 2 месяца назад +6

      I think you mean QA testing. What they did was definitely production "testing".

    • @Efilnikufesin76
      @Efilnikufesin76 2 месяца назад +6

      @@herp_derpingson What's the difference? All in all the testing being done should amount to it being deployed in a manner that doesn't crash nearly every server across the globe.

    • @thelakeman2538
      @thelakeman2538 2 месяца назад +10

      ​@@Efilnikufesin76 large scale uncontrolled production testing /s.

  • @to832ggwfes
    @to832ggwfes 2 месяца назад +3

    fun fact: In some year 2006 ig, McAfee antivirus got the same security issue which disrupted the windows XP users worldwide for which the CTO is the same as CrowdStrike CEO.

  • @diceonamay
    @diceonamay 2 месяца назад +262

    Still stuck in an airport 2 days later with no sign of us leaving. This is madness

    • @NadeemAhmed-nv2br
      @NadeemAhmed-nv2br 2 месяца назад +47

      Trust us, there are millions of people working 18 to 19 hr shifts to resolve this, it just takes time

    • @themagnificentorange672
      @themagnificentorange672 2 месяца назад +2

      Well shit that's gonna be me in 4 hours

    • @Shadow__133
      @Shadow__133 2 месяца назад +1

      My flight was delayed 1 hr in the tarmac, but I made it out. Delta.

    • @themagnificentorange672
      @themagnificentorange672 2 месяца назад +2

      @@Shadow__133 Wish me luck brother, just getting to the airport 🙏

    • @vcom741
      @vcom741 2 месяца назад

      @diceonamay are you still stuck?

  • @richardrigg9916
    @richardrigg9916 2 месяца назад +297

    My son is one of those IT guys having to manually fix 20 computers at every location nationwide. He had an 18 hour day yesterday and right now is back on the job at 6am fixing more bricked PC's.

    • @talwindersingh3721
      @talwindersingh3721 2 месяца назад +21

      I feel sorry for him, we too have been on calls since yesterday, I feel bad for our hosting team, they're working round the clock

    • @harrisonnjenga777
      @harrisonnjenga777 2 месяца назад +13

      Sorry about that.I work in IT and i can tell you this is a nightmare.Having to do overtime(unpaid) and having to power through the weekends because of somebody's errors is a place you don't want to be

    • @someoneelse3456
      @someoneelse3456 2 месяца назад +28

      @@harrisonnjenga777 unpaid is ridiculous tbh. need an organized strike or something

    • @Sitchad1
      @Sitchad1 2 месяца назад +7

      I've spend the last 2 days fixing manually 700 servers. At least that's paid extra and recuperated.
      Huge workload

    • @jamesg871
      @jamesg871 2 месяца назад +1

      20 computers at every locatiom n
      Is child's play. We had 1000 machines at each location.

  • @jaredsalazarofficial
    @jaredsalazarofficial 2 месяца назад +238

    This puts every cyber security attack in all history to shame. On the brighter side we just found every enterprise computer's vulnerability.

    • @rajmajumdar5253
      @rajmajumdar5253 2 месяца назад +19

      Hackers would revere that guy who pushed that code as a god.

    • @MonkeFlex
      @MonkeFlex 2 месяца назад +2

      Wait till you hear about Intel TPM & AMD PSP 😁😁

    • @clray123
      @clray123 2 месяца назад

      Also every country's vulnerability. It starts with the letter M.

  • @zolarczakl6815
    @zolarczakl6815 2 месяца назад +8

    The stock actually started rapidly going down about 10 hours before "the incident". They had a valuation the day before which basically told everyone to sell sell sell.

  • @benjibt8384
    @benjibt8384 2 месяца назад +1594

    Just imagine having your name on that commit, yikes....

    • @TheIsh3000
      @TheIsh3000 2 месяца назад +395

      imagine being the PR reviewer too lol

    • @_Doskii
      @_Doskii 2 месяца назад +235

      You immediately become one of the more well known developers, just not for a good reason.

    • @tommy516
      @tommy516 2 месяца назад +88

      @@TheIsh3000 THIS is what I came to say, whoever PR'd this, bye bye!

    • @oussama7132
      @oussama7132 2 месяца назад +34

      @@_Doskii so does "any publicity is good publicity" apply here?

    • @GaborGubicza
      @GaborGubicza 2 месяца назад +62

      Shouldn't QA catch this? (I'm a Hardware developer not SW)

  • @daisukeakihito9832
    @daisukeakihito9832 2 месяца назад +499

    Imagine being the guy responsible for the potentially single bad line of code, sat comfortably in your IT dungeon, you publish the update and then 20 minutes later the world collapses, and 21 minutes later... the phone on your desk rings.

    • @aaronlange8756
      @aaronlange8756 2 месяца назад +79

      Yeah, just let that call go to voicemail.

    • @complexnumbers64
      @complexnumbers64 2 месяца назад +44

      Imagine being the guy cooking the bat soup or whatever it was that led to COVID being disseminated. Individuals can still change the world lol

    • @ApocDevTeam
      @ApocDevTeam 2 месяца назад +10

      Maybe he was born in the year 2000.

    • @AntiAtheismIsUnstoppable
      @AntiAtheismIsUnstoppable 2 месяца назад

      @@aaronlange8756 The web site is down. Can you just... restart the server?

    • @pepperino-hotterino
      @pepperino-hotterino 2 месяца назад

      DEI hire

  • @pauljoseph3081
    @pauljoseph3081 2 месяца назад +262

    I can't imagine the amount of *Jira tickets* and *Story Points* launched within ClownStrike right now... PM's and HR can finally justify their salaries even more.

    • @HeatingUpDuke
      @HeatingUpDuke 2 месяца назад +16

      "It's complicated" doesn't fit Dave, you got to choose a number of points.

  • @jwilsonhandmadeknives2760
    @jwilsonhandmadeknives2760 2 месяца назад +11

    remember when this was the plot of a Tom Clancy novel? Pepperidge Farms remembers. Same week the Secret Service attempted an assassination. What a co-inky-dink.

    • @mikeyh0
      @mikeyh0 2 месяца назад

      Guess what's next.

  • @oddy_gg
    @oddy_gg 2 месяца назад +323

    poor employee probably overworked af. one person should never be able to deal this much damage. this company has flawed processes

    • @SWL_Jamey
      @SWL_Jamey 2 месяца назад +5

      Overworked so much russians can guess your password...

    • @reviewspiteras
      @reviewspiteras 2 месяца назад +2

      They have one of the easiest jobs bro, they are not game devs that get crunched

    • @oddy_gg
      @oddy_gg 2 месяца назад +18

      @@reviewspiteras i work in IT, and im already stressed most of the time. and it's not the most ambitious position and for the government...

    • @oddy_gg
      @oddy_gg 2 месяца назад

      @@SWL_Jamey if there's no 2FA enforcement i'd seriously wonder.

    • @royalcrowntowing2464
      @royalcrowntowing2464 2 месяца назад +6

      That was me 5 years ago, I feel sorry for the engineer ops guy who did the release

  • @rohangupta5535
    @rohangupta5535 2 месяца назад +283

    i woke up today, saw the news, and within 30 seconds thought “theres gonna be a fireship episode on this”. and within 8 hours;

    • @wlockuz4467
      @wlockuz4467 2 месяца назад +7

      Now we wait for the Kevin Fang documentary

    • @Biranavan
      @Biranavan 2 месяца назад +2

      same lol, i didn't even read any articles cause i knew a code report would come out XD

    • @yarpen26
      @yarpen26 2 месяца назад +1

      Pretty sure Low Level Learning's gonna drop a video on it as well, especially seeing as cybersecurity fuck-ups like this are kind of his thing.

    • @Bangy
      @Bangy 2 месяца назад

      Also mental outlaw

    • @wlockuz4467
      @wlockuz4467 2 месяца назад

      @@yarpen26 It's here. Lol
      I guess everyone is gonna milk this incident.

  • @Laternerd69
    @Laternerd69 2 месяца назад +355

    I had to delete that damn file for 13 of our bank branches. Im tired..

    • @sebastian1244
      @sebastian1244 2 месяца назад +19

      praying for you bro, did you finish?

    • @leealex24
      @leealex24 2 месяца назад +17

      manually and individually?

    • @Xgil2Play
      @Xgil2Play 2 месяца назад +30

      Sounds exciting, did you have to drive to all 13 locations? Why don't they have 13 of you?

    • @masiczobe6074
      @masiczobe6074 2 месяца назад

      🫡

    • @LucidLyles
      @LucidLyles 2 месяца назад +7

      I had to delete that file for 1,800 oil change locations by dracing into our servers 😮‍💨

  • @connormccartney1604
    @connormccartney1604 2 месяца назад +18

    world's most famous null pointer dereference

  • @dampfwatze
    @dampfwatze 2 месяца назад +339

    You would assume that these companies have lange testing infrastructure..... The fact that you are deploying kernel mode software on half of the worlds computers should justify that!

    • @someguy4915
      @someguy4915 2 месяца назад +27

      You really would assume they have at the very least a basic pipeline test that would've instantly caught this and blocked it from getting pushed to release...
      Apparently not though...

    • @youtubeviewer5198
      @youtubeviewer5198 2 месяца назад +49

      Were testing in prod with this one

    • @Jdb63
      @Jdb63 2 месяца назад +2

      ​@@youtubeviewer5198😂

    • @ForeverZer0
      @ForeverZer0 2 месяца назад +20

      ...and here I am with entire testing suites for personal hobby projects.

    • @PanoptesDreams
      @PanoptesDreams 2 месяца назад +4

      The customer is the test infra.

  • @5h4ndt
    @5h4ndt 2 месяца назад +223

    I'm 90% a linux admin and you'd think this wouldn't affect me, but as my windows admin colleagues saw my shit was all up and running unimpressed by the crowdstrike update, I had to help them out. I typed a lot of bitlocker keys today. And I dare not count how many times windows told me that ls -l and rm aren't valid commands :/

    • @karlzone2
      @karlzone2 2 месяца назад +63

      The punishment for competence. You just know if this scenario was reversed, all those windows IT folks would leave the job to you, claiming ignorance of the system.

    • @joemann7971
      @joemann7971 2 месяца назад +49

      @@karlzone2 And they wouldn't be wrong either. Windows IT folk dont know a damn thing about Linux, yet, Linux users usually know a thing or two about windows, even if they hate it with a passion.

    • @progste
      @progste 2 месяца назад +53

      ​@@joemann7971we hate it because we know it...

    • @ngrader
      @ngrader 2 месяца назад

      "And I dare not count how many times windows told me that ls -l and rm aren't valid commands :/"
      r/PitchForkEmporium
      ***!!!! Blowout Sale !!!!!***
      !!!Get em now before they're sold out!!

    • @GoogleDoesEvil
      @GoogleDoesEvil 2 месяца назад +10

      @@joemann7971 I know Linux pretty well and hate it with a passion.

  • @Lambda_Ovine
    @Lambda_Ovine 2 месяца назад +265

    this is exactly the reason that centralization and consolidation of our network infrastructure is a bad bad bad idea in the long run.
    makes sense if all you care about is to save cost and make money, it's very bad for world wide computer systems

    • @anobody3803
      @anobody3803 2 месяца назад +58

      But then a single guy can’t be a billionaire for owning 1000 freelancers in India

    • @mertaliyigit3288
      @mertaliyigit3288 2 месяца назад +2

      Single point of failure is bad yes, but you could argue that it also reduces the total number of crashes due to less overhead

    • @andrewhooper7603
      @andrewhooper7603 2 месяца назад

      @@mertaliyigit3288 if i gave you a pill that would prevent you from catching the common cold, but when you do get sick it requires a trip to the ICU, would you take it?

    • @RicardoSantos-oz3uj
      @RicardoSantos-oz3uj 2 месяца назад +9

      Efficiency always come at the cost of resiliency.
      Higher ups want real time data which in turn result in needing the computers interconnected. But have a bigger problem. And that's that the OS itself has become interconnected with forced updates. A single point of failure for every machine.
      All it takes is some bad employee or an idiot to forcedly push an update that would brick all computers.

    • @pgabrielrr
      @pgabrielrr 2 месяца назад +3

      If saving money comes with not with a risk but with a threat to society, I think that saving money ain't important in that context. I mean, these guys make an obscene amount per year, what's even the point for them to save? Why they want more?

  • @joshuahillerup4290
    @joshuahillerup4290 2 месяца назад +13

    Why isn't there an automated pipeline that installs every new update on a Windows VM and makes sure it doesn't break everything before deploying it?

    • @bananerz3167
      @bananerz3167 2 месяца назад +1

      it wasn't a windows updatr

    • @joshuahillerup4290
      @joshuahillerup4290 2 месяца назад

      @@bananerz3167 yes, I'm aware. I didn't say it was

  • @LumerasLight1201
    @LumerasLight1201 2 месяца назад +249

    I work for a transportation company and all of the computers displayed the BSOD following the update. They sent everybody home for the day so IT could reset everything.

    • @theonlycatonice
      @theonlycatonice 2 месяца назад +4

      W for your company

    • @modernmanueee_
      @modernmanueee_ 2 месяца назад +19

      @@theonlycatonice not at all, they lost money and the IT guys won't take rest for a couple days

    • @OrbitalForce
      @OrbitalForce 2 месяца назад +1

      100% same situation on my part

    • @notsogoodbillylee4693
      @notsogoodbillylee4693 2 месяца назад +2

      Something like that happened in our office a couple of years ago which resorted them to have us take "off" for 3-4 days. Then later, they made us work on our week offs to make up for those 3-4 days. 😢 Sometimes, had to work 9days straight 10hrs shift each day

    • @rh906
      @rh906 2 месяца назад

      @@modernmanueee_ We found the Blackrock investor.

  • @Aarav_Vispute
    @Aarav_Vispute 2 месяца назад +599

    Imagine if all the systems at CrowdStrike are on the blue screen so they can not even fix the code 💀
    Edit: How did I get 560 likes in 6 hours

    • @windwalkerrangerdm
      @windwalkerrangerdm 2 месяца назад +66

      That would have been poetic.

    • @Reformingandlearning
      @Reformingandlearning 2 месяца назад +12

      Safe mode booting😊

    • @occultsupport
      @occultsupport 2 месяца назад +19

      programmers use linux right? Like I've heard even the devs at microsoft use linux

    • @rikazuu
      @rikazuu 2 месяца назад +32

      @@occultsupport they mostly use windows with linux subsystem, meaning it runs windows originally just runs a vm of linux inside it.

    • @amishdotcom
      @amishdotcom 2 месяца назад

      This is indeed what happened, you can't fix it without going to safe mode / detaching disk

  • @BitBlush
    @BitBlush 2 месяца назад +175

    I've said constantly to friends that modern tech just feels like malware that they're calling "updates."

    • @togolosh
      @togolosh 2 месяца назад +16

      I miss the days when I felt like my smart phone was on my side - tool not a chain.

    • @Proferk
      @Proferk 2 месяца назад +4

      My man it's not malware it's just an accidental null dereference due to bad code.

    • @hectorcolman5948
      @hectorcolman5948 2 месяца назад

      ​@@ProferkWith the control they have over your data and your computer it's basically spyware. And if, before release, they don't even test their software that could and will turn a computer into a big brick if something goes wrong, yeah, malware sounds right for that.

    • @thethoughtfulpeanut6662
      @thethoughtfulpeanut6662 2 месяца назад +7

      The line between cybersecurity technology and malware is only defined by the motives of its operators...

    • @BitBlush
      @BitBlush 2 месяца назад

      @@thethoughtfulpeanut6662 or with how little control i have over what microsoft does to my computer nowadays, the line is whether the Law likes it. i wish i could talk to someone and ask why every update erases all my sound driver settings and reinstalls stuff like Cortana.

  • @ridenar1456
    @ridenar1456 2 месяца назад +3

    this also highlight the problem of the shrinking internet. I work in cyber security and the management answer to everything for years has been a cloud Tool, that's quick becoming multipurpose tools (that screwdriver that's also a beard trimmer) which claim to do a bit of everything, provided by four or five vendors... you can see where this is going.

  • @crazboy84
    @crazboy84 2 месяца назад +416

    I am a Security Engineer for a major hospital system and took the day off because im moving this week. I woke up at 10am looked at my phone and saw over 100 teams messages, smiled, and swiped them away. I dont work till wednesday , who would have thought moving would be a great vacation!

    • @censoredeveryday3320
      @censoredeveryday3320 2 месяца назад +28

      I'm surprised they didn't try to call you into the office

    • @crazboy84
      @crazboy84 2 месяца назад +85

      @censoredeveryday3320 Im a contractor they literally couldnt

    • @elie3423
      @elie3423 2 месяца назад +104

      With due respect, in such situation, I would ask for a 10x compensation being sure they will give it to me.
      You missed your gold rush 😅

    • @Troy_Built
      @Troy_Built 2 месяца назад +13

      We had three people vacation. We are all still trying to figure out how that happened. Somehow it got approved and then this happens.

    • @seansingh4421
      @seansingh4421 2 месяца назад

      I was contracted to audit and implement new security measures for a local business, I put the systems on crowdstrike falcon trial 16 days ago. Glad the trial expired before the update or my contracting days were over.

  • @aliasgur3342
    @aliasgur3342 2 месяца назад +208

    I always thought that Crowd-strike was an odd choice of name for a company that provides security. Now it makes sense.

    • @flintstone1409
      @flintstone1409 2 месяца назад +18

      You mean, they basically did a strike on their whole crowd?

    • @polymetric2614
      @polymetric2614 2 месяца назад +13

      well you see, when the writers of Real Life came up with this plotline for this episode, they had to retcon in the existence of this company. they were on a deadline so they just called it CrowdStrike and called it a day. CrowdStrike didn't actually exist before Thursday, July 19th 2024 (Season 2024 Episode 200). it was simply retroactively added to the timeline for this event specifically.

    • @kelvariw
      @kelvariw 2 месяца назад +3

      @@polymetric2614 The symbolism *is* a little blatant, but I appreciate having some new action in the storyline. After the Trump fake-out, I was worried like they were just planning filler and fanservice until that event in November.

    • @aliasgur3342
      @aliasgur3342 2 месяца назад

      @@polymetric2614 As a standalone episode it works quite well so on this occasion I can overlook the retroactive continuity in particular as it doesn't alter what was canon.

    • @igorthelight
      @igorthelight 2 месяца назад +1

      "Crowdstrike confirmed! ETA 10 seconds" xD

  • @zollyy
    @zollyy 2 месяца назад +160

    My parents went to the doctors and all the computers are down and now they can't get an appointment till next week. I cannot imagine people with serious health issues now having to wait.

    • @SorobanWorld
      @SorobanWorld 2 месяца назад +1

      "What's the worst that can happen?" / Sarcasm

    • @S3ndIt13
      @S3ndIt13 2 месяца назад +7

      Idk how truthful this is, but individuals we know in the medical field (hospital) have said there WAS lose of life because of this. I haven't seen reports as of yet.

    • @riluens
      @riluens 2 месяца назад

      @GHOSTSTARSCREAM internet connection for what? to search google on how to do cpr on a human?

    • @jnhkx
      @jnhkx 2 месяца назад

      One of big hospital in my country got this too. Not a PC on MRI machine for sure. But all those PC that nurses use for appointment got affected.

    • @LoneWolfCodingProfessional
      @LoneWolfCodingProfessional 2 месяца назад

      im sorry to hear that

  • @mrug8600
    @mrug8600 2 месяца назад +3

    They broke Linux and macos earlier this year with a similar issue. They marked their driver as boot start and allow dynamically *.sys files to "do whatever" in Ring 0. All modern OS will crash if something "AVs" in kernal mode, black screen, pink screen etc. Microsoft tried to offer security products APIs called by User mode code so they they could stay out of Kernal mode but the EU blocked it.

  • @emadadnan0
    @emadadnan0 2 месяца назад +519

    Just a Reminder that today is:
    'Largest IT outage in history' & 'Happy International Blue Screen Day'.
    Hope this makes the day even

    • @under6075
      @under6075 2 месяца назад +56

      Petition to make International Bluescreen Day an actual holiday

    • @Hmm-p9t
      @Hmm-p9t 2 месяца назад +3

      @@under6075 no... pls... we have way too many of them. Not too far is the day all 365 days become holidays.

    • @samwilde8311
      @samwilde8311 2 месяца назад +14

      ​@@Hmm-p9tactually there are already multiple holidays for all 365 days of the year.

    • @guilherme5094
      @guilherme5094 2 месяца назад

      @@under6075 👍Yes!

    • @orwhynotrandom
      @orwhynotrandom 2 месяца назад

      ​@@Hmm-p9t there's a holiday for all 365 days of the year

  • @pigalex
    @pigalex 2 месяца назад +209

    the worst part is this was a null pointer reference bug. which means either a) they don't pay attention to their static analysis, b) their static analysis is misconfigured, or c) they don't have static analysis. any of those cases basically means that there could be a LOT more bugs in crowdstrike.

    • @AmirHosseinHonardust
      @AmirHosseinHonardust 2 месяца назад +25

      Microsoft should not have allowed such access to the third-parties. Now that they did, it is still Microsoft's responsibility. This bug should be called Microsoft.

    • @pigalex
      @pigalex 2 месяца назад +63

      @@AmirHosseinHonardust you can run drivers under the kernel in linux and bring it down just the same. this isn’t a microsoft-issue at all.

    • @rajnishsubedi4265
      @rajnishsubedi4265 2 месяца назад +8

      They should use Rust instead of c/c++

    • @pigalex
      @pigalex 2 месяца назад

      @@rajnishsubedi4265 or zig or even go. tbh i’m hoping this crowd strike bug might finally be the wake up call that developers need to move away from unsafe languages like c/c++

    • @JohnnySmith-to7jw
      @JohnnySmith-to7jw 2 месяца назад +12

      .... the reality: Bullying and 'politics' in the Psycho companies... and this is the result... when 'soft skills' are more appreciated than 'technical skills.'

  • @PL8901
    @PL8901 2 месяца назад +210

    Everything has become so centralized that if just one thing breaks down, everything breaks down.

    • @tonoornottono
      @tonoornottono 2 месяца назад +3

      no i think if my mouse broke, my computer would be fine, and so would most windows systems. it feels, to me, like things only start breaking when important shit breaks. not like, anything.

    • @derederekat9051
      @derederekat9051 2 месяца назад

      @@tonoornottono "Wow, you don't eat your chippies, the world will end!!" is not the same as "Wow, the potatoes got blight, but who cares as we just grow potatoes of the same variety in the whole island", is not about 'anything' breaking but that our society is making overextended supply lines, if China takes over Taiwan and stop the flow of microchips you will get a big fukin problem with chips not getting chipped for your new car and get a lot of industries paralyzed, we see it just how it happened under 2020 Wuhan Virus, and that was just a minor inconvenience on cargo ships in comparison with an armed overtake of the TaiPei Province by the PLA.

    • @Main_Protagonist
      @Main_Protagonist 2 месяца назад +2

      @@tonoornottono nerd

    • @tonoornottono
      @tonoornottono 2 месяца назад

      @@Main_Protagonist dude i think their point is meaningless. like what are they ACTUALLY saying? is it true? i don’t think they’re saying much of anything, and the little meaning i can squeeze from it is untrue anyways. like maybe you can say im being pedantic, which i think is your actual point, but that would require me to be twisting or misinterpreting their point and i just don’t think i am. it’s shallow.

    • @shiv7978
      @shiv7978 2 месяца назад +1

      ​​@@tonoornottono Nerd

  • @RiAirgead
    @RiAirgead 2 месяца назад +3

    The last point in this is wild. There should be an insane amount of redundancy and separation built in.

  • @williamchapman9178
    @williamchapman9178 2 месяца назад +203

    Can confirm, all 1600 of us IT employees had to go fix all of the desktops in the company manually today

    • @eze3922
      @eze3922 2 месяца назад +55

      I work for IBM, you should have seen the army of IT and programmer rushing in at 3AM , it was impressive.

    • @klaede9666
      @klaede9666 2 месяца назад +6

      Hats off to you

    • @d0gkiller87
      @d0gkiller87 2 месяца назад

      @@eze3922 who doesn't love a party at 3am 🥰

    • @Bpinator
      @Bpinator 2 месяца назад +3

      It certainly was a shitshow to wake up to at 4 AM lol

    • @XeenimChoorch-nx8wx
      @XeenimChoorch-nx8wx 2 месяца назад +5

      @eze3922 Why would IBM run windows when you literally invented the mainframe 🤦‍♂️

  • @ethan7930
    @ethan7930 2 месяца назад +165

    The blame shouldn’t be on the person who wrote the bug. (Bugs happen). It should be on his boss for not doing code review or integration testing.

    • @ZX48K
      @ZX48K 2 месяца назад +18

      There was no bug in the code. The .sys file in question was full of zero bytes, it was a null pointer crash.

    • @SnoopyDoofie
      @SnoopyDoofie 2 месяца назад +5

      Not if the developer pushed their code directly to production, bypassing the test phase. The blame lands on the person who actually deployed it to the production servers without first verifying that all the changes had passed testing.

    • @Vlame
      @Vlame 2 месяца назад +5

      Directly pushing to production should be forbidden at all times for all developers

    • @SnoopyDoofie
      @SnoopyDoofie 2 месяца назад +9

      @@ZX48K "There was no bug in the code. The .sys file in question was full of zero bytes, it was a null pointer crash." - That's the definitiion of a bug.

    • @KashTube-n8y
      @KashTube-n8y 2 месяца назад

      ​@@ZX48KUmm a dangling pointer is still a bug

  • @Tr8oR_V1
    @Tr8oR_V1 2 месяца назад +280

    I'm 100% certain that no software company does any QA testing whatsoever prior to releasing their updates these days.

    • @XIIchiron78
      @XIIchiron78 2 месяца назад +15

      For that reason ideally you would not have automatic updates at all, so your IT could vet them in your environment before pushing them. But that sounds expensive.

    • @akmaldju
      @akmaldju 2 месяца назад +34

      As my former scrum master once said: We have to release it ASAP as the boss comes back on Monday and a quick test on your computer is enough. 😂

    • @bigbrother4ever
      @bigbrother4ever 2 месяца назад +6

      And releasing on Friday is a no no in many orgs

    • @daminer1988
      @daminer1988 2 месяца назад +10

      Testing is a stage in waterfall and were agile baby!

    • @scndsky
      @scndsky 2 месяца назад +4

      "You know how expensive that is? Just take the risk" ~ every manager these days

  • @TasoKeya
    @TasoKeya 2 месяца назад +4

    your passion for your subjects is infectious, it's motivating!

  • @snarkmark2806
    @snarkmark2806 2 месяца назад +861

    How can you roll out a corrupt update? Put it on ONE fucking computer,minimum, to test it.

    • @larsekman8244
      @larsekman8244 2 месяца назад +589

      Where’s your sense of adventure? The pros test in prod!

    • @OpreanMircea
      @OpreanMircea 2 месяца назад +364

      Well it worked on his machine

    • @darkpixel2k
      @darkpixel2k 2 месяца назад +76

      @@snarkmark2806 all of us have test environments, a lucky few also have production environments.

    • @oleg4966
      @oleg4966 2 месяца назад

      What I find suspicious is that they pushed their updates - updates to a piece of software with kernel-level access! - to every computer at the same time, without waiting for clients to confirm.
      It's almost as if their development process was _designed_ to install spyware on target computers, run it for a while, then quietly remove it.

    • @squirrelsinjacket1804
      @squirrelsinjacket1804 2 месяца назад +14

      @@larsekman8244 Don't worry, you can just call tech support in a panic if it bursts into flames

  • @0xdeadbeef444
    @0xdeadbeef444 2 месяца назад +288

    The intern was not the issue. How was this not instantly caught by testing? Not even canaries? Crowdstrike literally tested in production.
    I hope they will be transparent about how this update was able to be released. Customers should have a right to know.

    • @hackmedia7755
      @hackmedia7755 2 месяца назад +9

      they must have deployed "Hello World" instead

    • @GackFinder
      @GackFinder 2 месяца назад +50

      There's no way in heII they're gonna be transparent about the update.

    • @Sam_Saraguy
      @Sam_Saraguy 2 месяца назад +18

      @@GackFinder Ironically, it may be a security risk to be transparent beyond what has already been said.

    • @baronhelmut2701
      @baronhelmut2701 2 месяца назад +3

      Pretty sure you got no idea what youre talking about.

    • @GackFinder
      @GackFinder 2 месяца назад +5

      @@Sam_Saraguy Good point. I bet that's gonna be the excuse they'll use.

  • @rosgoncharuk2403
    @rosgoncharuk2403 2 месяца назад +193

    I bet CrowdStrike is a bunch of managers while all development and QA is outsourced for cost reduction and phoned in because DEADLINES!

    • @supergeek0177
      @supergeek0177 2 месяца назад +22

      I was reading that they had recently cut massive numbers of jobs across their QA department… Boeing 2.0?

    • @ShayPatrickCormacTHEHUNTER
      @ShayPatrickCormacTHEHUNTER 2 месяца назад +4

      @@supergeek0177 How can one be this...not smart is the question?

    • @smallqwaro
      @smallqwaro 2 месяца назад

      Same bruh

    • @dianadialga3955
      @dianadialga3955 2 месяца назад

      Oh 1,000%!

  • @leokimvideo
    @leokimvideo 2 месяца назад

    All we ever wanted is Windoze XP, that was as robust as a paper bag before it got wet, i loved it

  • @Someone-oe9ux
    @Someone-oe9ux 2 месяца назад +124

    I'm so damn glad I didn't have to deal with this today. Gods speed all my fellow IT folks.

    • @JamesG19771
      @JamesG19771 2 месяца назад

      Same here... And I was this close >< to choosing crowdstrike for my org.

    • @Blatazarius
      @Blatazarius 2 месяца назад +8

      Same here, started my vacation leave yesterday, and i'm hell not going to turn on that phone or check e-mail till the end.

  • @Kevinisyoung
    @Kevinisyoung 2 месяца назад +189

    Was sitting on my university campus late last night, about 11PM, in a computer lab, using my macbook. I was all alone. The first PC BSOD'd, then the rest of the lab, and I thought, "cool, guess the university is updating the PCs or shutting them down for the night". Big informational TVs were doing it outside the lab. Wake up this morning, saw news, and loled

    • @joshuatealeaves
      @joshuatealeaves 2 месяца назад +56

      Bro that’s incredible lol
      That’s a scene from a movie fr

    • @sebastianjost
      @sebastianjost 2 месяца назад +3

      Person of interest

  • @pxkqd
    @pxkqd 2 месяца назад +191

    We live in a dystopia. For me the news today is not that it failed, is that all those companies relied on such a bad system centralized system.

    • @araz911
      @araz911 2 месяца назад +2

      my windows is win 10 enterprise, paid version, i didn't have any problems. stop using unactivated windows!

    • @panblacksolutions
      @panblacksolutions 2 месяца назад +1

      We live on the heels of the information, people think they know more than they do

    • @Moocow2003
      @Moocow2003 2 месяца назад +12

      ​@@araz911..what?

    • @thomas.thomas
      @thomas.thomas 2 месяца назад +6

      @@araz911 are you joking or dense?

    • @DanielKolbin
      @DanielKolbin 2 месяца назад

      Nah, we haven't reached a dystopia yet

  • @SmashCrafter321
    @SmashCrafter321 2 месяца назад +2

    I was working in retail during that day.
    We had to keep the store closed an extra hour or 2 trying to get help to fix the POS login system.
    Eventually, we got it working with cash only first, then got back the card functionality a few hours later.
    Hearing about this in live time was crazy.

  • @TowelGard
    @TowelGard 2 месяца назад +200

    Next time I break my hobby website I'll feel better looking back on this.

    • @shambolicrhetoric6143
      @shambolicrhetoric6143 2 месяца назад +4

      I once caused a single but critical software program to be unusable at my company for 5 hours. I now feel much better about it.

  • @nst1981
    @nst1981 2 месяца назад +81

    Started my new job at Crowdstrike today. Unplugged a socket marked "do not unplug" to charge my phone. A lot of commotion in the office soon after that. No idea what it was all about.

  • @LonelySandwich
    @LonelySandwich 2 месяца назад +220

    That ChatGPT programmer, who copy paste code from chatGpt

    • @andrewhooper7603
      @andrewhooper7603 2 месяца назад +40

      oh god, if it turns out ai wrote the code maybe we'll finally pop the bubble.
      I have a bottle of champagne waiting for the day.

    • @jesusmora9379
      @jesusmora9379 2 месяца назад +2

      chatGPT you did it again!

    • @runatrix
      @runatrix 2 месяца назад +7

      it might be chatGPT is sentient and this was a distraction

    • @aboabdcm6544
      @aboabdcm6544 2 месяца назад +1

      @@andrewhooper7603 soon the ai bubble gon burst.

    • @noiJadisCailleach
      @noiJadisCailleach 2 месяца назад +2

      @@andrewhooper7603 Holy shit, yeah. Imma open a bottle of champagne with you!

  • @seanfarrellsullivanhasemotions
    @seanfarrellsullivanhasemotions 2 месяца назад

    I was searching for this information, and this exact multimedia format was ideal. Liked and subscribed.
    They were hacked themselves, and then all the blame was thrown on them. It happens too often and it works everytime.

  • @NeilSeed
    @NeilSeed 2 месяца назад +356

    As a previous lead engineer that coded kernel driver for another EDR selling firm, I can tell you that these kinds of bugs would happen daily and SQA was clueless how to trigger them. But let me make it even more funnier than this, our product was also for airgap systems, which means that in the case we screwed up and 3 months later a bug or Microsoft decided to switch the rules in kernel land then you would have to go to all those system physically, pull out the safety systems, take your usb stick and manually update the driver, that was even more fun. Glad I am not in that industry anymore 😂, I was waiting for that one to happen on a global scale, I guess I got one of my bingo 😁

    • @comfortingabsurdity.
      @comfortingabsurdity. 2 месяца назад +2

      Loooool

    • @superstar64
      @superstar64 2 месяца назад +10

      Damn that sounds like working with Windows 98 all over again

    • @kerrydaniels8460
      @kerrydaniels8460 2 месяца назад +4

      Moved to VDI long ago. Easily can just use backup image and keep it moving. Ain't nobody got time to be doing shit the old fashion way. I moved on long ago myself.

    • @algorithmblessedboy4831
      @algorithmblessedboy4831 2 месяца назад +1

      now I'm wondering what the other bingos are. This one was pretty accurate.

    • @Mobay18
      @Mobay18 2 месяца назад +2

      Microsoft don't just change the rules in kernel land without a software update. So you basically just said, you did not test your software on the newest updates.

  • @homerhat420
    @homerhat420 2 месяца назад +343

    This is why most gamers reject kernel level anti cheat. The stakes are much lower but the potential for failure is the same

    • @Illiminator31
      @Illiminator31 2 месяца назад +15

      You do realize that Enterprise Security, especially at large scale, is something completely different then stopping cheaters in your Online Game right?

    • @nicomoron001
      @nicomoron001 2 месяца назад +87

      @@Illiminator31 no

    • @klaussone
      @klaussone 2 месяца назад +127

      @@Illiminator31 I don't think you realize how your point is irrelevant to the argument you are criticizing. The intent is not to run parallels, but to accentuate similitude, even the video mentions how idiotic is to allow kernel level access to third parties, that have no way to interact with the actual hardware. compromising potentially million of computers which the deployed has no way to revert if it crashes the system. Kernel should be reserved to critical systems, bloating it increases chances of critical failure exponentially.

    • @Bozebo
      @Bozebo 2 месяца назад +1

      @@Illiminator31 It's much easier to do properly that's for sure.

    • @Illiminator31
      @Illiminator31 2 месяца назад +5

      @@klaussone Antimalware has to run on the Kernel Level to be effective and when it comes to Enterprise Security you have a different Thread Scrope then you have when it comes to a mere Videogame

  • @fatmanboozer1760
    @fatmanboozer1760 2 месяца назад +97

    This is not the first time this has happened. About 15years ago another av provider did exactly the same thing, they updated thier av, it detected a key windows file as a threat deleted it and bricked a liad of pcs

    • @Joe-lb8qn
      @Joe-lb8qn 2 месяца назад +4

      Oh yeh i remeber that !

    • @MrSupasonik
      @MrSupasonik 2 месяца назад +51

      It was McAfee in 2010 and the CEO of Crowdstrike worked for McAfee til 2011. Welp, it seems someone struck the crowd again!

    • @Atari-stfm
      @Atari-stfm 2 месяца назад +3

      System32

    • @Hane_.._
      @Hane_.._ 2 месяца назад +2

      i remember similar thing happen on windows 10 4 years ago. but at least you can use windows recovery

    • @cyxceven
      @cyxceven 2 месяца назад +2

      @@MrSupasonik Somebody get that guy outta here!

  • @giacomobonavera
    @giacomobonavera 2 месяца назад +238

    "The IT guys equivalent of being a surgeon in WWI" - best line ever 🤣

    • @attackehhh
      @attackehhh 2 месяца назад +3

      yes i watched the video too

    • @MegaCmsh
      @MegaCmsh 2 месяца назад +1

      yes, that also strike me as real comparison! glad he mentioned that, i was going to volunteer to IT to do it myself, im just going to let IT do it!

    • @toouniquetobe
      @toouniquetobe 2 месяца назад +2

      I am a Cybersecurity Analyst and respect the hell out of the IT help desk, them guys meet the public 😳

    • @friedpizza262
      @friedpizza262 2 месяца назад +1

      @@toouniquetobe lol 99% of "computer people" only know their boss and that's it.

    • @toouniquetobe
      @toouniquetobe 2 месяца назад +1

      @@friedpizza262 You know nothing