@@Danforth2730 No worries mate. I will tell you to press the button for 3 story points, Bob here will second my opinion for 2 story points, and then you can press the button for 5 story points. We might need to add a 1 or 2 pointer to swipe the keycard and biometrics to enter the server room too, so let's not forget about that.
In my company, that would be too many hours which would be outside of the project budget. We'd have to ask for upper-management go-ahead to extend the hours. We'd have to come up with a strong-convincing case. Nevermind that all of this takes way more hours than to just do the thing.
Well, what you do is, you go in steps, so first you restart all of the 4000 computers in safe mode, because it will take some time to boot up, then you go on to the next and do the same, while the previous is booting up. When you're in safe in all computers, from the last compujter you can start working your way backwards doing the deletion of the 291 file and so on on each computer, then turn it off, when you're again at the first copmputer, turn that on, then the next and so on. With 4000 computers, that will still take for one man around 222 hours by quick calculations
The Lawyers long names are the business names. They create their agency name by putting together all the founding partner's last names. So when when the agency is four names long, that's four people.
Personal biases should not shadow for the reality which is that microsoft isn't really involved in it. You could say, that why hasn't microsoft banned anyone from entering into kernel mode this way, but this is seen as monopolistic behavior, because then it is microsoft controlling what you can do with your own computer. I still detaste microsoft like nothing else, but being real, bringing them into the law suit, Delta will get nothing from that. So I'd like to see MS down, but on what they're actually guilty of, not on this. Imagine that MS will actually get convicted, it will not only affect MS, it will affect anyone else in the future too, which is not guilty.
Delta deserves the pain for using Windows Server. This court case is the fight between 2 incompetent organisations trying to determine who was the most incompetent.
The irony being that they have to both reset 40,000 servers and they also lose half a billion in revenue. It's a real lose-lose scenario. They would have probably gladly paid the half billion if they could have had their infrastructure back up again immediately.
Linux in non-software organization is usually anything but free. Redhat ... damn, I can't say. Not everyone uses a free version in corporations (I've been around), but when they do they run CentOS - "what's next?" is a a good question since CentOS is not what it was.
@@vilian9185 The cost added to software development will far out weigh any cost savings. Unless your software is just a network service and doesn't depend on the OS, developing for Linux is a real pain. Usermode Linux has the approach of breaking things and expecting all apps to just change their code and re-compile. Display a GUI? Oh fun, X Windows vs Wayland, different window managers, different GUI toolkits. Play audio, we've got alsa, jack, pulseaudio. Just gotta change your code and recompile all the time when the OS changes!
@@vilian9185 More secure? Debatable. Perhaps security through obscurity, which everyone knows doesn't have anything to do with security. There are plenty of hacked Linux servers that would like to claim otherwise.
Probably meant "servers and workstations"; I'm sure that the "terminals" used by the folks at the various desks (baggage, ticketing, etc.) were also affected. If not, yeah 40K Windows servers... that's too many.
I thought the reason we had to put up with the security risk of the intel management engine is so this kind of problem doesn't happen. There's a separate computer and software running in your machine that can completely control it outside of the OS. You don't have to be physically there. Why wasn't this used?
reason why diskless client booting is a better option for office work related stuff you only need to modify few device to do the removal process save and reboot all clients
@@lifelover69even working at ~30k employee international tech company, the number of servers we have is not close to 40k. if we count VMs, then sure it's easy to hit.
For the more experienced here if all my servers boot thru network from a central OS installation in a NFS this could be resolved just deleting the problematic file in that installation and wait for the restart triggered by the kernel panic?? enumerate all the reasons this is a bad idea!!! 😂
You're wrong about Economy+ I just took a 17 hour flight from LA -> Sydney in Economy+ (business was full) and it is remarkably better than cattle class. Its basically a smaller version of business.
![Noob To Max With DRAGON REWORK In Blox Fruits [FULL MOVIE]](http://i.ytimg.com/vi/LnBMOinoOvA/mqdefault.jpg)
Delta job openings:
Computer safe mode booter
Kernel operator
Computer restart operator
🤣🤣🤣
Kernel operator? Just make sure you don't use the operator SDK.. that thing's a PITA
Restarting in safe mode? That'll be at least 12 story points
soo, next sprint?
It's over 8, means you gotta break that up into two 5 pointers
@@Danforth2730 No worries mate. I will tell you to press the button for 3 story points, Bob here will second my opinion for 2 story points, and then you can press the button for 5 story points. We might need to add a 1 or 2 pointer to swipe the keycard and biometrics to enter the server room too, so let's not forget about that.
In my company, that would be too many hours which would be outside of the project budget. We'd have to ask for upper-management go-ahead to extend the hours. We'd have to come up with a strong-convincing case. Nevermind that all of this takes way more hours than to just do the thing.
Well, what you do is, you go in steps, so first you restart all of the 4000 computers in safe mode, because it will take some time to boot up, then you go on to the next and do the same, while the previous is booting up. When you're in safe in all computers, from the last compujter you can start working your way backwards doing the deletion of the 291 file and so on on each computer, then turn it off, when you're again at the first copmputer, turn that on, then the next and so on. With 4000 computers, that will still take for one man around 222 hours by quick calculations
The Lawyers long names are the business names. They create their agency name by putting together all the founding partner's last names. So when when the agency is four names long, that's four people.
It's a good idea including Microsoft in the lawsuit, they'll happily push CrowdStrike under the bus for you.
Personal biases should not shadow for the reality which is that microsoft isn't really involved in it. You could say, that why hasn't microsoft banned anyone from entering into kernel mode this way, but this is seen as monopolistic behavior, because then it is microsoft controlling what you can do with your own computer. I still detaste microsoft like nothing else, but being real, bringing them into the law suit, Delta will get nothing from that. So I'd like to see MS down, but on what they're actually guilty of, not on this. Imagine that MS will actually get convicted, it will not only affect MS, it will affect anyone else in the future too, which is not guilty.
Delta deserves the pain for using Windows Server. This court case is the fight between 2 incompetent organisations trying to determine who was the most incompetent.
The Alien vs. Predator of technical prowess
My work couldn't reboot into safe mode because the only person with the boot loader key was on vacation 😂
The pain of manually resetting 40,000 servers.
Ahh id rather lose the half billion $
The irony being that they have to both reset 40,000 servers and they also lose half a billion in revenue. It's a real lose-lose scenario. They would have probably gladly paid the half billion if they could have had their infrastructure back up again immediately.
Linux in non-software organization is usually anything but free. Redhat ... damn, I can't say. Not everyone uses a free version in corporations (I've been around), but when they do they run CentOS - "what's next?" is a a good question since CentOS is not what it was.
paying for a red hat license and paying a sysadmin is cheaper than that amount of windows server and crowdstrike license, and more secure too
@@vilian9185 The cost added to software development will far out weigh any cost savings. Unless your software is just a network service and doesn't depend on the OS, developing for Linux is a real pain. Usermode Linux has the approach of breaking things and expecting all apps to just change their code and re-compile. Display a GUI? Oh fun, X Windows vs Wayland, different window managers, different GUI toolkits. Play audio, we've got alsa, jack, pulseaudio. Just gotta change your code and recompile all the time when the OS changes!
@@vilian9185 More secure? Debatable. Perhaps security through obscurity, which everyone knows doesn't have anything to do with security. There are plenty of hacked Linux servers that would like to claim otherwise.
This is a task that can be massively sped up with rubber duckies.
3:37 The theoretically safest computer possible, is one that has halted.
If Delta airlines has 40,000 servers they are doing something very very wrong.
Probably meant "servers and workstations"; I'm sure that the "terminals" used by the folks at the various desks (baggage, ticketing, etc.) were also affected. If not, yeah 40K Windows servers... that's too many.
And that’s 40,000 Windows servers! Have they not heard of Linux?
@@steves9250 they have 40000 of windows machines, of course not, if they had a competent sysadmin there this wouldn't have happened
I thought the reason we had to put up with the security risk of the intel management engine is so this kind of problem doesn't happen. There's a separate computer and software running in your machine that can completely control it outside of the OS. You don't have to be physically there. Why wasn't this used?
If I was a Delta shareholder I would be asking why they have 40,000 Windows servers
Seriously, what the f were they thinking?
chatgippity said 3-10 minutes per windows server to boot into safe mode.
reason why diskless client booting is a better option for office work related stuff you only need to modify few device to do the removal process save and reboot all clients
... Why does delta have 40,000 servers?
every time a new delta employee is onboarded, they give him a complimentary windows server? idk
@@lifelover69even working at ~30k employee international tech company, the number of servers we have is not close to 40k. if we count VMs, then sure it's easy to hit.
Forget about following Netflix on microservices, I bet delta re-invented microservices. Every function call needs its own service!
60 man-days?
That's like 200 woman-days!
For the more experienced here if all my servers boot thru network from a central OS installation in a NFS this could be resolved just deleting the problematic file in that installation and wait for the restart triggered by the kernel panic?? enumerate all the reasons this is a bad idea!!! 😂
Crowdstrike is downright demonic
4:21 - F-word?!?!?!?!?!😮😂😂😂
to be fair, using windows with automatic updates enabled for important things and not having a competent sysadmin team is asking for trouble
It wasn't windows update. It was croudstrike update. If it was windows update, this would be easier to manage.
@@username7763 the "automatic updates" was including the crowdstrike, they had windows and crowdstrike both with automatic updates
welcome to the law offices of BOYS CHILL FLEX lol
Lol. I would blame Delta too 😅
American air companies sound so terrible. We are so lucky our airlines are not that terrible
just summon 10k ppl out of thin air teleport them to each server and you are done in less then 10 mins easy
Its easy. If the server doesn't boot, it is save from ransomware.
Clearly we need an AI Robot to do that manually... ahahahah not
You're wrong about Economy+
I just took a 17 hour flight from LA -> Sydney in Economy+ (business was full) and it is remarkably better than cattle class. Its basically a smaller version of business.
Airlines suck, but Microsoft is way worse. I hope Delta wins the litigation.
Microsoft is bad in many ways, but I don't see it here. Croudstrike's driver is what broke. Don't blame the OS for a vendor's bad code.