Response To Engineers Should Be Held Reliable
HTML-код
- Опубликовано: 8 сен 2024
- Recorded live on twitch, GET IN
Article
yieldcode.blog...
By: Dmitry Kudryavtsev | x.com/skwee357
My Stream
/ theprimeagen
Best Way To Support Me
Become a backend engineer. Its my favorite site
boot.dev/?prom...
This is also the best way to support me is to support yourself becoming a better backend engineer.
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Kinesis Advantage 360: bit.ly/Prime-K...
Get production ready SQLite with Turso: turso.tech/dee...
The only reason C suites exist is to be in charge of a team, therefore responsible for them. If engineers will be responsible for all they produce can we just fire all C suites?
What is a c suite?
@@halalmon executive-level managers, like CEO, CFO, etc.
@@halalmonTop management: CEO, CFO, CTO, etc.
Exactly and their increased liability & accountability is often used to justify their inflated salaries
@@halalmon the Chieftains
It’s not like having a plumber. It’s like having 10 plumbers, 2 leads and a manager that order the plumbers around when working on your bathroom, and then a plumber is ordered to use let’s say half the sealant quantity so they can save some money. And then you try and sue the plumber. If that 1 plumber would refuse, he’d just get replaced with someone who wouldn’t refuse.
"if I didnt do it some else would" that wasnt good enough during the Nuremberg trials, it aint gonna work here chap.
@@octomancerELDEN RING MENTIONED !!!
@@octomancer what is wrong with godwins law? We were supposed to learn from the past. Many important lessons learned from then. Like not being a sheep. You need to hold yourself and your own code to higher standards. Dont care what other people might do. It always starts with 1 individual to lead change. Refuse to do something you know is not good. Have pride in your work.
@octomancer and ironically, this programmer following orders effected 8million computers. Some were part of critical hospital infrastructure. He had the potential to kill 8 million people with a regex. The situations are very similar.
@@pluto8404you've never been in such a situation and you probably never will be, but sure if it was you you would definitely do the right thing regardless of the consequences!
Company makes record profit because engineer did a good job --> ceo and c-suite get bonuses.
Company crashes whole world computers because they didn't hire testers , outsourced all the work ---> ceo and c-suite still get bonuses, engineers are held responsible.
In fairness, if it isn't my company, and I didn't make the mess, but I know it's there and you are hiring me at least partially to deal with it, I'm going to demand higher pay to account for the stress and the BS of dealing with a pre-existing dumpster fire. I don't think that's what's happening in most of tech, but generally speaking, there are CEOs with specialties -- scaling a proven product, dealing with a mature and declining business, dealing with major internal issues, etc.
Maybe we need more of that in tech and fewer founders and internal people who aren't professionally detached enough to recognize that things are f-ed. I don't really know.
As a civil engineer, I would like to point out that yes its more complex to build bigger house than smaller house. lmao. The reason we have models and actual buildings afterwards. As a software engineer, what I've noticed is because its easy to cut corners in software, managment will force you to do so and hence increasing the chances of enshitification. Its like civil engineers delivering the model houses as final projects.
And in civil engineering you have a guy who has credentials to put his signature on "thats okey". And those guys are paid reaaaaaaaaly well. I would love to have that paper in software engineering. Do you imagine how much i would cost? :D And how much it would skyrocket minor apps cost? xD Forget about open source and free aps. Not going to happen :P Nobody would put jail time for some open source small time project ;)
and in civil engineering there are codes and standards
Ok build a 100sq atom house and build a 100sq foot house and let me know which was harder. With your bigoted blanket statement.
yeah... soooo my hot take. This comparison is why i hate the term "software engineer". Every other engineering discipline has the person who underwrites the design or calculations. The building example is great because the projects design and calculations are checked over and over on multiple levels. There are standards, safeguards, safety factors. A bunch of people and authorities have to check the design and the calculations. If you are the person who said "this is good to go". Your work is still scrutinised by authorities. Also you need to have a license to practice, and that has to be renewed or you have to take courses every year. I know the swe field is pretty new but for critical infrastructure i think these kind of standards and safeguards will be future.
I think he was saying that it can be less complex to build a bigger house than bigger code because you can lean on math and reality when it comes to building whereas in coding as things get bigger it become harder to make sense of the code.
I think people don't realize how little control any single engineer has
Especially in large corporations you have to get things tested, team reviewed, approved by your lead/business partner etc etc
No single person is responsible for such a event of anyone is responsible it is the CEO/CTO for not having better control systems
If there's no control they aren't really engineers are they, just useless titles to inflate egos, lower standards, and stagnate wages.
The issue at crowd strike may have been that single engineers did have more control than they should have... but that's still managements fault
I think people have a right to not care. That's why we pay money to someone else to care for us, because we simply cannot handle the problem. If you ALSO cannot handle the problem, why did I pay you? Not only that, but you made things WORSE. Once again, why should I bear ANY responsibility or care as a client that requested this to be done? This is just incompetency in any other craft.
@@misbegotten3508 why did you watch this video if you don’t care? Why do you write so many angry btchy comments if you don’t care Karen?
In other industries, things are organized to give people sane amounts of responsibility and control. And engineers get paid to tell management no. That's part of the job. And it's expected. If you don't hold people to some standard internally, managers game the compensation system to manipulate their numbers at the long term expense of the company. That's why engineers, accountants, and lawyers all have professional ethics and get to tell people "no" -- it's in *everyone's* interest.
But for some reason, software guys don't get that level of respect and seem to be ignorant of even the existence of this alternative world.
Honestly if Developers can get held liable for stuff, that means I can make demands as a developer. I’m not working on a garbage app if I can be held liable for bad outcomes.
"This code is still easily portable to windows. I will not sign off on it."
And this is how it should be. Because software is so relied on nowadays to run critical systems I think devs, at least developing in certain industries, should be subject to being targets in an investigation if things go bad.
My wife for example is a nurse. If someone dies and it was found that she was grossly negligent, she could lose her license and held criminally liable.
With that said, the entire chain of command should also be subject to the same scrutiny in an investigation. Managers, the C’s, heck even HR.
This would be great because software development would slow waaaay down. This has many benefits. Job longevity, safer deployments, company accountability and better working conditions.
All orgs that can’t function without computers and networks have long ago become software companies without intellectual realization of that, or transfer of wealth and power to devs that should of come with the responsibility. See even now you are denying I’m correct.
When I would get pounded on by C-Suite “when is this project going to be done we demand A DATE!” I like to reply, when is your business ‘going to be done?’. That question maybe applied when each software element was a tool in a department like a sophisticated customer calculator or something. But when the entire network, all data, and every interface is on the NET/WAN/LAN, this isn’t ‘tool time with tim’ any more. It’s literally your business rendered in code only a handful of people know how to even read let alone write.
@@Horsewithnoname88 Potential liability when it's blatant individual negligence...but not when it's a confluence of bunches of engineers doing different things.
@@Horsewithnoname88HR wouldn't know anything about how this update was generally made, at best the most they would know about this is a reference check
No liability without ownership
This.
I mean, except for brazen actions, like internationally sabotaging the product
Unless you're a plumber, an electrician, a gas fitter, etc? Right? Curious.
@@michelians1148 Those are not responsible either if they are just employees. Unless they are self employed. Companies are also responsible to provide guidelines and training for its employees. Employee is only liable if he bypasses those on his own, does neglience or sabotages the work purposedly.
@@michelians1148In the US at least, most plumbers, electricians etc. are sole proprietors who have to buy their own liability insurance.
Charlie Munger used to say "reverse it" and see if it still works. Instead of saying we're going to hold engineers financeable liable for mistakes, we reverse it and say, we're going to make liable and compensate engineers, above their salary and percentage based, for all successes. Yeah, they'd never do that in the same scope.
If you believe software engineers are well paid then you should see what will happen to their salaries once they begin to be held responsible for the consequences their software, it will fucking skyrocket
X company would spend more money on turnover and lose all their profit.
People don’t realize how fast some of these companies need to work. I don’t care how talented your group of programmers are, working faster is going to produce errors. The key is to fail in a safe way. Which is where crowdstrike freaked up
@@ttred7621 fail in a safe way and have plan b, c, d in place. there's a reason why air-gapped systems exists, there's 3 major desktop OS that can be used, with how windows updates can fuck up setups time and time again i still baffled by the fact that everyone update at the same time and fail together.
most likely u gonna have to hire some "certified developer" that gets paid a shit ton to do nothing but to sign all ur code, while everyone else earns nothing.
Reliable? you mean liable? or responsible?
Reliable engineer is a new title in SF for those that didn’t move interstate
your pic 🤣
Reliable means you show up.
Yes
Maybe "responsible" + "liable" = "reliable"
If a bridge collapse, do you sue the manual labor workers or the construction company?
The construction company also you can sue the architects, principal engineers, material suppliers, etc anyone with actual power in making the bridge.
It depends on the results of the investigation that follows the collapse, but I'm sure you didn't bother to imagine there'd be an investigation at all in your half-baked analogy.
Depends on who signed off on it. If an engineer won’t sign off, it doesn’t get built, and if an engineer signed it under duress to meet “deadlines”, the company should be sued.
@@misbegotten3508every single comment you've made in these replies is so idiotic, yet you somehow feel confident enough to trash talk the OP in each one lmao
Actually it's not just this video apparently, every comment I can see from this guy is full of rage for no reason
Fundamentally the bug was no different from any other programming bug. The difference was at the business/product level where the impact was much larger because of the business CrowdStrike ran. That is not the responsibility of the individual programmer.
Anesthesiologists and Structural Engineers have the final say in how shit gets done. We software developers have to eat shit from management and compromise daily. I'm not sticking out my neck under those circumstances.
Probably 'cause you're not an engineer.
@@misbegotten3508 OK. I mean we have 50 million users and our services scale to meet their needs seemlessly but I mean, yeah, I ain't no engineer. I'm just a guy who types shit on a keyboard for money.
Exactly.
@@misbegotten3508doesn't matter if you have a PhD in software engineering or not, management still doesn't listen and still doesn't give a fuck about code quality.
We can write reliable software, you just dont want to pay for it.
By the way, I would never do that for any kind of money.
Haha. Yes. Always more money.
"If you just PAID me more all these bad things go away!"
Psychopath.
@@misbegotten3508 its pretty telling that you immediately interpret that as greed, rather than them saying that the whole project would take 3 times as long to complete, thus costing 3 times more money
@@beer_4781 I'm not convinced this is true. In *other* disciplines, quality improvement initiatives usually save *massive* amounts of money. If you had to put a dollar figure on technical debt, it's be pretty massive.
And the types of stuff that need to be done to prevent basic issues like memory safety errors more-or-less amounts to having machine checked documentation that always stays in sync with the code. And they automatically give you tests that are machine checked to ensure 100% flawless coverage.
If you replaced all the tests and comments you had to write and all the half-measures with QA and the rest with writing JML or ACSL annotations, would you *really* be increasing costs? What if enough people used it to justify making tools that automated much of that work and made it less tedious and repetitive in the same way we have tools for what is done now?
I'm just fundamentally not convinced that this does cost more or at least that it *has* to cost more.
Yep, anyone who has doubts look up what goes into critical airline software certification and how reliable that software is.
Choose 1.5 of 3: fast delivery, safe, feature rich.
If you want safe software slow down your development iterations and vastly reduce your feature expectations. After all, the safest code does nothing.
Oh so as SE you can follow an order, push a bug to prod and get liable, or refuse to follow the order and get fired.
C'mon they are just following the company process.
So they are just following orders
But a civil engineer is held liable if they're ordered to stamp an unsafe and untested building design.
There exists software too important to be pushed untested for safety.
If an employer punishes a professional engineer for not stamping a design, the employer should be fined.
It's ultimately on PEs to ensure safety, regardless of management. And it's on management to respect that.
@@connorskudlarek8598you can't compare those jobs just because they have engineer in them. This is not how development works
@@connorskudlarek8598 This is the problem of shallow analogies, the world isn't that simple. You can't compare both, they are totally different areas with different requirements and process. The similarity stops on the name.
@@FirstYokai this is how SOME development SHOULD work.
Do you think Boeing should be allowed to push planes out with unsafe mechanical systems? I imagine not. So why should they be allowed to push out unsafe software systems?
Nobody cares if your no user website has certified engineers behind it. But an insulin pump better not just infinite loop inject insulin.
the react comparison is wrong imo, that's like saying the BOEING 747 MAX MCAS software killed more than 300 people because it was in C++ or they used whatever design pattern, which obviously isn't true. Also, many many things in structural engineering/construction isn't purely defined by math since the construction isn't built by math with math, it's built by real materials by real people and usually there are confidence intervals *everywhere*. What is true is how the software behaves, and that behavior is 100% measurable and testable.
Many bridges collapsed because at some point winds picked up just to reach the bridge's material frequency which completely fucks up the bridge and we can measure what weather conditions will exist on the exact spot the bridge will be built. Pretty much the same concept applies to software: testing is possible and not enough testing should be a crime whenever it leads to real consequences like people not being able to go to the hospital.
Also, anesthesiologists don't know exactly how much anesthesia should be used, that's just plainly wrong. They exist *because* we don't know exactly how much anesthesia someone will need because literally everyone's body processes it differently. We use math to give them a baseline, but they are there because we don't know how to simulate each body that receives anesthesia. We know 50 kilograms of anesthesia is probably too much, and 0.0000005 micrograms is too little. Anything in between, which is where it gets hard to predict mathematically, sounds like software a little maybe?
Confidence intervals do not mean thaat it is not built using math. It obviously is, there is just room for material imperfection, construction error, etc. And all of that is measured in by creating the building to hold much bigger weight than it was designed for. You can not do that in Software. Your analogy with testing does not work. You can not make sure that something works and is completely bug free. Because you can easily have the same exact issue inside your test that you have in your implementation. Testing something does not guarantee bug free code.
@@martintvrdik1655 en.wikipedia.org/wiki/Formal_methods
Anesthesiologists are actually a really good example of what changing a profession to introduce quality control and standards looks like -- they went from having some of the highest malpractice insurance to some of the lowest because of the effort of a lot of people across the entire industry who wanted to look at the problem end-to-end and actually solve things so that errors only ever happened once and then things got changed to ensure that they would never happen again.
@@martintvrdik1655 There is math that lets you put bounds on software reliability. And just like there are modeling tools in engineering, there are tools in software that can eliminate entire categories of problems systematically people generally don't use them.
People quote that 70% of Microsoft's bugs are memory safety related. But they ignore that Microsoft's own tooling people say that 80% of the bugs can be caught by their own tools, it's just that software devs and managers turn that stuff off or subvert it instead of making use of it.
It's bad culture and bad incentives end-to-end.
i think also an issue here is how quickly junior developers are given access to mission critical codebases and infrastructure. If we want to talk about the difference between "real" engineers and software engineers, you'd never see a junior civil engineer making critical decisions when it comes to say... designing load bearing elements of a building.
Right now, in this moment, there are people wanting some policy put in place so this is prevented in the future. Among these people are also some that are raging against government regulations that they are slowing things down and costing too much. We didn't start out as a nation of regulations. Those are written in blood, we always wait until the Bad Thing happens, then we create rules to prevent it.... then 20 years later we complain about the rules.
"then 20 years later we complain about the rules" because maybe the rules don't solve the bad thing at all
Speed and price are chosen over quality at every step. This was kinda inevitable.
The reason it’s so hard to make software engineers “accountable” is that it’s hard to figure out what exactly any one person working on it is “accountable” for. How many complaints have we heard about regarding vague KPIs or obtuse management or unreasonable demands? What measurements do we start and end with? I’m sure there’s a lot of consensus we can form about general best practices and all that, but who do we trust to codify that? Who do we trust to enforce that? And how can we argue for and against those rules and regulations? I don’t think it’s an impossible task, but it’s definitely nowhere near a straightforward one. From what I can tell, there’s no easy one-to-one comparison to other professions.
Base pay of SE in US is 105k. Anesthesiologist is 348k. Prime talks out his ass sometimes.
If i get 348k i can affort the 50k or 100k a year insurance that is needed then. Simple.
"A government that creates absurd regulations by senile people whose familiarity with technology is limited to their TV remote."
The government: "The Internet is a series of tubes..."
He wasn't wrong...per se....
Re VW:
1) I'm of the view that the regulations, as worded, only pertained to passing the actual emission test. The conditions of the test were far different than real driving conditions. Arguably VW did nothing wrong.
2) The only VW employee that was jailed was some new manager put on the project, it seems he new nothing of what went on in that emissions regulating software and certainly did not instigate it of create it. He was a scapegoat.
So, looking at it the VW saga is a very poor argument for nailing a hapless engineer at random when something goes bad.
3) I worked for Lucas back in the early 1990's. One team there was writing engine management code for Volvo. Over lunch one of that teams engineers proudly explained to me how they had implemented code that detected when the engine was undergoing a fuel consumption test and optimised appropriately, otherwise it operated normally. That suggests that the entire industry has been dodging tests for decades.
Usually these kinds of things happen because either 1. Management put unrealistic deadlines on their engineering team because they're totally clueless about how the job is done, and they refuse to listen to any pushback. Most competent engineers want to do good work, but there are only so many hours in a day. Get things done on time or get a different job. Or, 2. Management outsourced the majority of the work to people who are not remotely qualified to do the job.
Either way, seems pretty obvious who's at fault.
it is absolutely delusional to want software engineers to take responsibility for bugs and outages, obviously some will have to!
but imagine this, imagine you are an open source developer, making free pieces of software or games and someone sues you for a bug LOL
why should that person be held liable, it makes no sense at all, the software was free, no one forced people to use their software nor paid them
I think people will jump on that, because it allows to shift the blame.
Then the government will ban open source software, because it is dangerous.
That depends on the licence. The MIT licence, GPLv3, 3-Clause BSD and Apache v2 all limit the liability of the authors regarding the software.
Because the licenses would disclaim liability.
Meanwhile if some scanning machines code nukes someone a TOS doesn't get them out of that
If you're an engineer, you deserve responsibility. That's literally the whole downside of being an engineer.
I don’t know if you know this, but “engineer” is a protected term in other countries, and in the US is referred to as a professional engineer. Software developers are not engineers, because they do not have a professional engineer certification.
This is important because if a professional engineer gives their stamp, they are held liable for the project. But this goes both ways. That stamp is also a protection and is more important to have than their actual job, as they will NOT give their stamp if they know the project is fucked, so management cannot threaten their job because the consequence of losing their professional engineer license is worse than getting fired.
Software developers don’t have this. Their jobs can be threatened and replaced. This doesn’t happen to engineers.
It’ll also help if you stop mixing these terms together
Hot taker should take a intro to business or contract law course. What do they think incorporation is for? It doesn't just sound fancy.
REEEEEEliable as supposed to "liable"?
supposed?
@@AnEntityBrowsingYTREEEposed
its spelled suppository. They are suppository reliable.
No, the managers and C-suites should be held liable and all engineers should be protected under some kind of whistleblower law, so they should be allowed to discuss their current workplace engineering standards (or lack of) without any consequences, then definitely the software of the world would be way different!
I agree that the dev who messed up the code should have consequences.
But that's minor. The management that allowed this entire process to get to this point where NOTHING is 'tested' are the ones to blame.
AND the companies that allow 'patches' to root-kit your employee computers are also to blame.
How do you know it was a dev that messed up the code? In my company, QA packages the release and they don't know code.
Making the SWE liable for their code would radically inflate the already astronomical cost of SWE, and mostly serve to feed lawyer's children and malpractice insurance policy salesmen. Who do you want to work for? Insurance? Lawyers? End Users? Take your pick.
@@JimAllen-Persona Then your company is trash for not having written interaction logs? How do you keep track of anything lmao
Making devs liable for bugs would immidiately kill the field on the spot. People here talk about how CrowdStrike and other software failures killed people. Yet noone mentions how many more people were saved because of efficiency increase and better metrics that new software provided. Software that would not have existed otherwise because noone would pay the cost of development.
@@JimAllen-Personayou think a place that doesn't run tests had QA release the update? Buddy, QA doesn't exist in this picture.
To the point about constuction/engineering being defined by math and software not, theoretically we can prove software using logic (i.e. math). However in practice it takes much longer kinda like construction.
This should be how we create important systems (where people's safety is at stake). We can even verify proofs using computers, so we theoretically should be able to create correct programs reliably, though slowly, and it reduces to the structural engineer
Yes, but this is worsened significantly by the industry's obsession with imperative unsafe languages. A decent program in haskell can be formally verified quite quickly, but we for some reason just throw up our hands like there is no better way and continue mutating our singleton global array of unmanaged pointers
Modern T.V. remotes have way too many buttons where there is no uniformity in button placement between them. Its nonsense. Keyboards have a standard layout for a reason.
24:40 My mom is a dentist, and she has to give more local anesthetics to readheads. Because they are, for some reason, resistant to the stuff. I assume there are similarly groups of people who are resistant to general anesthesia.
Interesting...
23:30 [Calculable Numbers] No joke it is very much a skill-issue, it's a built in bias soft-dev doesn't have such numbers because soft-dev is not legally required to even bother to think about it meanwhile business interests wouldn't spend the money on it .. but we wax poetic about perf all day long or all tests green etc etc ..
and thus no research for implementing the standards or formulas to calculate those numbers as a discipline ever happens.
i don't think a programmer can be held responsible for their code since there are so many people involved. who do you blame the guy who directly coded the bug? the person who did not make proper error handling? the person that structured the initial code that allowed the error? the person who set up the validation policies? or maybe if a you are using a package that had a weird quark? who do you blame? one person just does not have the control needed to be held responsible.
This assumes the person enforcing the law knows anything about software engineering and isn't just looking for a scapegoat.
Cosmic ray flips a bit in your computer and causes my software to error? How do I prove I shouldn't be held liable if software engineers are liable for their mistakes?
Bit of an extreme example but another one for thought, Intel's recent CPUs have been comparatively unstable, if my program crashes as a result of that can I be sued before such issue is known? Do I then have to be able to afford to sue Intel for damages? What if the PSU is fautly or other hardware component?
If a programmer is especially negligent on their own accord I would say that could be reasonable, but always being able to sue for damages? I'd leave the profession.
Probably depends on the contract.. If it specifically requires the software to be fault-tolerant (against bit flips, etc.) then yes. If not, I'd say it's reasonable to expect that the SW will run on a reliable HW.
Maximizing profit for shareholders is NOT fiduciary duty, it's a doctrine and it is not imposed by any law
tbf, i think the law do require company to take actions in their sharholders' favors. just that some take different paths toward that goal (making good products and ensure the longevity of the company or maximizing stock price at all cost for example)
Yup money goes up, quality goes down.
@TayVal-cx8fy it depends on what the contract that goes with shares says. Most are about maximizing money in some way but some shares come with the disclaimer about specific values they will uphold over money.
It's also a non-answer. When an analyst asks why a company did something and the C-suite says "maximizing shareholder value" or some other corporate speak, we all known it's nonsense. Good financial reporters regularly call them out on this too.
But for some reason, while the general public is good at recognizing when politicians deflect and don't answer questions, they aren't very good at recognizing when management does this.
"maximize value", "increased monetization", and the rest don't actually explain why a decision is in the long run in the best interest of the company or how it is supposed to increase the company's ability to add economic value.
So generally, we take it as a red flag that something is off. Generally people don't deflect and refuse to answer questions when giving a truthful answer would be good for them.
I think there's case law behind it now
Lawyer here. Incorrect understanding about corporate veil, indemnification, etc.
elaboration?
With respect to the anesthesiologist analogy, if they make a mistake, isn't it the hospital that gets sued? (unless criminal intent can be proven on the part of the individual)
It really depends. If there’s malpractice that’s a lawsuit immediately, even without criminal intent. So for example a drunk anaesthesiologist administering a lethal dose by accident is malpractice and will rightly be sued. They also have insurance though.
@3:40 hell yeah man I want to see more positive messaging around marriage and kids online. Thanks Prime.
Every profession we are comparing to Software are licensed, credentialed professions. Software is not.
In the US, most engineers are not government licensed. Membership in the IEEE is completely voluntary. But whether you are a member or not, companies still treat you with a level of respect and trust that software guys don't get.
@@MaxHaydenChiz They shouldnt get it. There arent any credentials certifying you are competent
@@AaronBrooks0321 Other than a 4 year degree and your publications, and all of your professional association stuff ? And then there are industry specific stands certifications, also private. How much value does a government piece of paper add on top of everything else? In the US, the conclusion has always been "nothing". I don't see why you think there are "no credentials".
A board certified anesthesiologist absolutely makes more than the average software dev. Maybe if you only work at a high level at FAANG or something you have a higher average. Depending on which numbers you look at you are making 3x as much.
Someone who says employees should be liable for anything that isn't just completely outrageous negligence/malicious and criminal behavior doesn't understand at least part of what companies are.
They're structures to shield induvidual from consequences that come about from actions they don't fully control.
And it scales. If you're self-employed your company goes under if you do a crowdstrike. You lose tons of clients at a minimum. Maybe the business isn't sustainable, and what money was in the company may be lost.
If you're an employee of the actual crowdstrike and you do this they're supposed to be responsible and check your work so they can confidently release it. This went through what's a normal path for rapidly deployed AV updates. It's not the employee. In a just world this employee isn't even fired. They do that to save face, which might work for the public but us devs should at least know better. I hope some less customer facing company hires him quickly. Having the guy that dos'd millions of machines must be a good brag for some.
The problem is that there is a culture of feature-creep pushed by greed which creates instability in software. Have you ever heard the phrase, if it's not broken don't fix it? Well these businesses are like salesmen at your door wanting to convince you that everything in your home is broken, everything needs constant fixing constant replacing. If you let them in eventually they are going to trash your house.
No responsibility of reliability without required revenue, and restricting management from ignoring risk; i.e if you program it you have veto power/sign off, etc.
Crowdstrike is a large event but is nothing in comparison to whichever future-event, or realization through journalism, that finally kicks off legislature for software to become an actual discipline.
It took events like a 100+bodies in the Hyatt Regency walkway collapse to force engineering to grow up faster.
Meanwhile all of softwares dead bodies are obscured, spaced out and hard to quantify; there is no morbidity table for 'software' afaik.
Phrases such as "software failure" do a LOT of work in abstracting away serious avoidable problems as innocuous oopsies that are unavoidable , protecting the software industry from much needed consequences.
I think IP Is to blame for this problem. Someone pointed out that management in software pushes developers to cut corners, and I think it's definitely true. The goal with software is to get it done to achieve some utility as fast as possible, to get the money as fast as possible, and responsibility is properly diffuse because the impacts are so vast that no one can reasonably be accountable for everything that can't foresee.
More responsibility needs to be taken closer to the point of use, but those at point of use can't take responsibility because the software is frequently a black box, reverse engineering is not allowed due to licensing agreements that IP enforces.
It's unrealistic to imagine a world where no problems occur. But one that is more decentralized and diffuse is imaginable, even in software, provided all software is either source available or can be safely reverse engineered, and if you can't understand what it's doing you are either not going to use it or you are going to demand responsibility by the relatively smaller niche player on the market.
Isn't this why companies pay for support, so they have Service Level Agreements (SLAs) or other contracts in place so that if something goes wrong the company is in the hook for fixing it to their best effort? It doesn't fall to a specific engineer or person to be on the hook for the issue.
The average salary for an anesthesiologist in the US in 2024 is 434k/year. If you pay me that much too, I am down for responsibility.
To blame the developer in anyway is stupid.
This effectively negates all responsibility of ownership.
A company could tell a developer to send out a nuclear bomb or threaten to fire it they don’t.
Then the developer is held responsible for the outcome.
How stupid.
Not to mention it’s not the developers job to make sure their code doesn’t cause a catastrophe.
I think people fail to realise just how much of the entire world would seize without all of the necessary precautions other companies are employing.
In the particular case, it’s clear corners were cut.
My hotel could not take credit cards for 5 days. There was a non zero chance we could have potentially lost payment for 100s of rooms with no recourse. Didn't happen but easily could have. How many people got stranded at airports and literally couldn't get rooms because why tf would we sell a room there's a chance of making no money on and having to manage everything like it's the 1960s...
imo you SHOULD manage everything like it's 1960s. tech is just a tool and if the tool is unusable then use another tool. what's stopping you from using services like paypal/ stripe/ other payment processor to accept payment?
To me, even an emergency config change should be tested. Because a bug that causes it to *skip* reading attack definitions would've been horrible as well. So you'd want a pool of VMs to download the update and simulate a known attack to ensure the file's being read. Use the first, last, and a handful of entries from the middle, and the process could still complete in a second or two, making it feasible to run even in the most panicked emergency. If any of the test systems don't report catching its simulated attack (or, as it would've in this case, don't report back at all because the system crashed), then there's a major bug *somewhere* that makes the emergency update broken, and likely unable to catch what it's being rushed out for anyway!
So, the obvious design for a minimal-overhead integration/smoke test, that isn't even testing for crashes, should've been enough to catch this error, and there's definitely a process issue.
To me it depends on who is responsible for reviewing and signing off on the work. In the event of an anesthesiologist messing up they are directly interacting with their patient and the hospital has no opportunity to review the work before it causes problem.
In crowdstrike’s case they were the one saying the work was good and had full opportunity to test the update before it got pushed. If crowdstrike had been upfront that they don’t do any testing before they push updates and that their devs have direct interaction with the clients I could see an argument being made that the dev that pushed should get sued. But the fault here is that they sold this product with the assumption that they would be doing any testing at all before pushing updates. They had every opportunity to stop this but their negligence allowed this update to get pushed without testing.
yea, I disagree that the difference is that "we don't have numbers", we do, and better ones than anesthesiologists for example, our target systems are way more predictable, much more standard; the difference is that as society we accepted the unreliability in favor to speed and convenience; in embedded systems that tradeoff is different and that's why errors are way less common, we have NASA taking reliability way more seriously and not allowing code to do certain things; we could have much more reliable web stripping 90% of JS, and in most cases even 100%, but we prefer the pretty, the animation, etc. We, as a "tech society" could have replaced JS or made it more reliable, but that had a tradeoff we don't want to pay; even with Rust (not a fan), but when you threat to slow dev speed and put a steeper learning curve (even disregarding Rust async), just the requirement to learn to use borrow checker is a no-go for many, so "we" chose the unreliable path of the tradeoff. That's not good or bad, just how it is. But is not a inherent limitation of the fields non-web is way more reliable, take a look on the requirements for code to be accepted in the Linux kernel, en industry embedded, etc. Is not inherent of the field. The problem is that the unreliability of webdev can permeate to other domains, and we start to think as a "webdev" things that MUST be reliable, when "not being down" is not enough, you also have to be correct, were not only money and image is lost, but lives are met with "just good enough" mentality, push fast and often and push through errors like they are irrelevant, no personal accountability because you''ll probably will be working somewhere else in a year and you code will probably disappear in a couple of years, probably become no more than an annoyance later. Well, when you "move fast and break things", guess what?, things will break, and not always is just a "like button" not registering or a misaligned form.
Now I can't stop thinking about Pelosi. Board we need a motion of no confidence and 3 anesthesiology asap.
The industry needs liability, and it could be done in a similar manner to civil engineering. A certified engineer would need to sign off on code prior to use in production. It doesn't add as much development time as you'd think and most respectable organisations already have processes in place.
My company has that, a member of the group of the most senior engineers (CTO, chief architect, director of engineering) has to sign off on certain major changes after it passes testing
Chances of a 3rd party understanding the code, what it does, what it is supposed to do and not do is a stretch. Also it would take them several days to weeks if they had to read all the code to get the proper context for the changes in most cases.
Imagine having millions lines of code and then having one engineer sign it up
@@nikos4677 doesn't just have to be one engineer, and large applications are typically modular. I'd also argue that if one doesn't completely understand the code, then it shouldn't go to production.
Construction complexity _is_ exponential. The same elevator that serves a 2 or 3 story building will take 20 minutes to reach the top of a sky scraper. The HVAC systems _cannot_ operate at low pressures or half the ground floor will be duct. The pressure on a simple water standpipe would be immense (in practice, leading to no water pressure above the 60 or 120 foot mark).
TBF, the crowdstrike outage probably also caused some people to MEET the love of their life. They would have left Dallas without ever meeting the charming young man they met outside the local Applebees during their extended layover.
There should be a shared blamed company wide on multiple levels. No scapegoating which corporate culture will always try to do. Sometimes you just gotta take an L
It's like when Americans incarcerated people that cause the car crash but don't fix the design of shitty dangerous stroads
That one company from heavily regulated industry: Hold my beer.. And see my disappearance trick! Twice!
My take is (a bit of a devils advocate for arguments sake, but not really )that if "Software can kill people", then don't use outsourced software on your products where if it fails, could kill people. Or if you do, you have a clause in your contract about this exact situation that "if it fails yadda yadda" and have proper procedures and systems in place to recover immediately from a software bug or bad update.
Let alone have procedures in place to thoroughly TEST every single update before it goes live to your systems. what an idea, huh?
CrowdStrike is not responsible for their customers shoddy practices or bad decisions when choosing software provider for systems that could kill people if CrowdStrike messes up.
That on companies putting too much trust, that they didn't pay or contract for, into a third party software that they didn't test before putting it online.
This is not a failure of engineers this a failure of processes and management to keep engineers in check.
The whole point Uncle Bob makes about clean code, software architecture and development process is exactly this: if this profession doesn't develop a craftmanship, something to profess, the state will step in and enforce it. And he used the same example, the emission scandal, where the manager pointed to the developer, he did it. Here, the stock market does his role. It's a warning.
I am not sure what happened with the CrowdStrike thing but the staging environment must be faulty if it didn't raise the issue.
Honest question from someone who does not do web dev: if React is bad, what should people use instead? Htmx seems nice, but it hits a limit with high-interaction websites, as far as I know.
React isn't bad, Prime is just a hater
I'm trying to understand.
This guy wants the government to regulate how software update rollouts should go?
3:59 but that flight was destined to crash, so crowdstrike actually saved the lives of 235 people
Uncle Bob has predicted this very thing:
1. Some crazy thing happens. Outage, explosion, etc.
2. Software engineers get blamed, perhaps rightly so.
3. The cause can arguably be as a result of inconsistent practice in software engineering discipline, such as testing, review, etc.
4. We start to get a whole bunch of regulation up in our business.
5. Life as a programmer becomes burdened and crappy due to the regulation that is applied onto us, because we lack self-discipline to do it (as an industry).
But is the plumber really liable? If the plumber works for company A then company A gets sued and not the plumber
Software realiability and safety IS formally measurable just like for the bridge.
There are formal mathematical methods for proving that a piece of code works as expected (or not) but they are not used for 99% of software because 99% of software (incl. 99.999% of web services) most of the time is not "mission critical" and doesn't usually result in injury or death of major financial loss in case of failure.
We often feel that "people fail upwards" partly because of the whole "People rise to their level of incompetence". This doesn't mean more incompetent people go up, it means that good people get promoted and keep getting promoted until they are not longer good enough to warrant a promotion, typically due to incompetence. Like a junior dev, becomes senior, becomes principal, becomes lead, but here they now need leadership skills, and they don't really have them, they are incompetent at that. So now that's where they are. They were good at something else and got promoted until that thing they are good at is no longer what they do.
4:48 Or for example, of an Engineer hides something from management and that bug goes out would be the engineer. But if the engineer is doing what they were hired to do and management gave them instructions that lead to the bug, that's on management.
Good reminder that all the software and computerized systems in planes, ics, medical equipment, etc doesnt have to be in there. Everything is the digital option by default, low tech or no tech is still and option
If software engineering fs up enough politics will notice and there will be regulations. The industry needs to step up their game or the government will get involved.
Flip! Let this stay
Flip: takes it out
I can't imagine anyone seriously trying to go after the devs as long as they followed routine company procedures (not what is written down, but what actually happens on a weekly basis with management's tacit approval).
FWIW, industry standards absolutely should be a thing. i.e. spydevs writing making the next "Uber but for taxis" probably don't need regulation.
But it is entirely possible to create a standards using a consortium model for safety critical software.
soydevs*
Most of those other jobs have a process that is being followed. In our field everything is optional. Some companys don't test at all, some do unit tests, some integration tests, some TDD, some model the software and have fancy graphics, some do pair programming, some do code reviews, some use linters, some do this, some do that. But there is no this is the way to do it right. Beside that our job is creative and although there might be similarities at the end each program is unique and was never done that way before else we would just copy & paste it. And even when using dependencies etc. we don't stick standardized parts into each other. We have parts that don't fit together and creating those "adapters".
And beside that how often have you heard developers complaining that they don't get time to fix bugs or refactoring things they know will explode. But that doesn't bring in money. You can't charge someone. There are so many things wrong in our codebase. Some which lead to the poorest performance you can imagine, some which will blow up if you miscofigure something, some which will break if you change something else and a lot which makes adding features hard and onboarding new people. Those issues been addressed, including possible solutions etc. What does the boss man say? That's too much work and nobody is going to pay for that. Do you want to pay for that? So I'm ordered to let the problems there. And those things are causing outages on every update too. But as those are customer specific projects it's not like we kill 8,5 million pcs.
Another thing to think about what is if someone is creating a module/function whatsoever which is working fine with the inputs/calls that exist in a given system and that system gets new features which call this module/function with different parameters and this parameter combination is causing a bug that wasn't able to happen before. Whose fault is it? Caller or callee? Especially in large highly configurable systems there is a limit how much you can really test of the real thing. I'm often working on small changes which are things that take a few minutes but setting up everything so I could test the real thing would literally take several hours or even days and that is just for one happy path. You can't charge your customer that much money for a really small change. So at the end we are pretty much testing in production. But at the end, even if we would test it in dev, we wouldn't catch everything because of the configuration and prod data.
If I would be held reliable for anything I do on a daily basis I wouldn't touch a single line in our codebase. It doesn't feel much different as if you would put me in front of a patient in a hospital with a opened skull and expect me to do neurosurgery on him. You want me to do a one inch cut on the left side? Yes I guess I can do it. I understand what an inch is and how to use a knife to cut. But there is no way I understand what this god guy implemented in those guy's skull and what will happen if I do this cut. Yes I can test this on a pig (mock) before. Well pig is still alive and hasn't complained, so I call that 100% test coverage. Well turns out this guy's brain has another configuration..
P/E 500? That has to be an error, or Crowdstrike should lose at least 90% of its valuation out of P/E correction alone.
A great company with close to triple digit growth commands a P/E of 50-60.
P.S: I have just checked, and no, it is correct. At the moment it has a P/E of 466.
Forward P/E is 56 which is still extremely high, but more reasonable. Still, unless they 10x their Earnings in a year, it is going to drop significantly out of correction.
Sure, engineer can be held responsible... but which one? there's hundred of them working on a software like crowdstrike, if not thousand. Some of them no longer work at the company, most of them have to deal with the repercussion of decision made in the past. They have to deal with the decision of management and director.
Shall we also add Microsoft engineers in the mix? Afterall crowdstrike work on windows...
I think which person the buck stops at depends on the particular failure claim. Programmers can often be thought of as making simple parts, that other programmers assembles into machines. I do see a case for some issues to blow right past the person who assembled it and land on the part maker. The function is named SafeSort() and it was documented to perform a sort that will never write to unintended memory, but it looks like there is backdoor that....
The distinction between an engineer and a developer needs widening. Put in place charters and tests that people have to undergo to prove their quality and the same thing for companies. There needs to be governance in place to assure that things are being held to high quality
Won't make any difference unless you do the same thing with management, no one cares about tech debt or security untill something breaks
It should be the managers responsibility to allow developers the discretion to make changes as needed to avoid things like this, but many managers are worried about feature development and new code will always be a source of production failures. It could be an old piece of code that had a bug that would never be triggered, but then a new feature could expose the bug, but it’s more likely that it would make it to production before being found. Management needs to refer to the developers for all estimates and they should be able to sign off on the code, too many companies have managers as all of this and no control is at the developer level.
look like square enix had crowstrike issue last week end . It was not ddos
There should be more regulation on safety critical software, mandatory testing and certification. Especially for updates, which frequently break a system that was working before
I work in hospitality and we had an outage due to this issue, and it was due to AWS running it on their systems and bringing us down with it.
Thing about the people who would have met the love of their life on the airplane or whatever, is that it is just as likely that they could have met the love of their life because the flight was delayed
There's already tons of regulation in software engineering! Some business specific, like PCI, AML, anti-fraud in fintechs. Some more general, like audit or PII data protection. No amount of regulation will fully prevent negligence, malicious intent or a culture of cutting corners. And this applies to all kinds of engineering - unfortunately hotel walkways still fell, data leaks happened and kernel drivers crashed.
6:28 I left my job in Law and decided to become a software developer because here you have QA and the responsibility doesn't fall on your head if something goes wrong. There's no QA in Law. If I made a mistake, most of the time, at least in my situation, there was no second person to catch it, and if large amounts of money were involved... how can I put it... I didn't sleep so well. And Law is vast. It's like an OS for the whole society. Very easy to miss something and find out about it when it's too late. You're basically testing in production.
Right but in software engineering, you don't have the authority to decide testing and release policy.
You can't make developers responsible for their bugs and still have them beholden to non technical managers trying to hit kpis
Triple our salaries and project duration and maybe we can negotiate.
MY MAN WANTS 300K A YEAR FOR NOT DOING A SHIT JOB. DAMN. IM IN THE WRONG BUSINESS.
EDIT: AND PROJECT DURATION. WOOOOOOOOOOOOOOOO. SOFTWARE'S ALREADY DEVELOPING AT A CRAWL, BUT WE WANT MORE TIME.
Why not the fucking moon while you're at it, desk-jockey.
Development is not responsible, theres a testing pipeline. QA teams and other testing.
Most companies don't have a dedicated QA. It's just the devs.
@@connorskudlarek8598 That's the problem of the company.
@@connorskudlarek8598 Com'on, my small company has a QA team of 2 people. One of their jobs is to package releaes. Even they would've caught this.
@@connorskudlarek8598 companies as big as crowd strike?
@@SageHeru ah, I see. My mistake. I thought you were speaking in general.
Although loads of teams at companies as large as Crowdstrike push code to prod without running it by QA, and instead rely on devs to be QA.
In fact, they laid off dozens of QA workers in 2023. Entirely possible that this was deployed on the assumption QA wasn't needed.
"fail to the top" just means "Peter's principal"
What if the dev who released the bug was insider trading?!
Everything goes straight to medical/aerospace industry.
There are software proofing math too, so technically software engineers could have that math too, just that would make software development magnitudes more expensive - though in some environment that would make sense.
There was a lot of talk about developer/certifcation a long time ago.
If you require developer qualifications then you also need to give them the power and regulations to apply that discipline in the face of incompetent management.
We have regulated software for aerospace. Its all written by the industry and then approved as an appropriate method to fulfill federal code. It would have to be something like that. Do-178 or iso for automotive already exists. Its just massively hard to do. Move fast and break things is effective at innovation just sometimes you break everything including your company.
17:16 "We had to destroy the village in order to save it."
There's a dissolution of responsibility
Imagine if everyone in the company took responsibility
You can't outlaw evil or stupidity
I think Prime's wrong on assuming that software is unique from other professions. I can't speak to doctors, but engineering problems can be quite difficult to model and make guarantees about: It involves a lot of testing on site, a lot of trial and error, and a lot of mathematics and modeling. It's not like you can definitively say whether or not a certain foundation is suitable for a high rise before going out into the field to test the soil, and even after testing you may still run into surprises after construction has started. And yet the impression I get from Prime's video is that he thinks that you can make absolute guarantees about engineering and not software.
Prime says you can't prove React wrong or right, but you also can't prove what type of French drain to install in a farm, you can only weigh the pros and cons for the particular environment you're in and provide an educated guess. This is exactly the same as weighing the pros and cons for React. The only difference is that traditional engineering fields have had hundreds of years of failures to learn from and codify while software has not.
This is exactly what makes engineering engineering. That there isn't a perfect solution and things can always break under the right circumstances. There is no such thing as a perfect software project without bugs. Not only does SE not have the same time as traditional engineering fields but the problem solving space is much wider all while the incentive to make the systems as reliable as possible is secondary to a lot of business decisions. Just goes to show what a clueless yapper Prime really is for not even understanding the essence of engineering
Obviously the company has to be liable, but as for individuals just the managers who put in place protocols that made this possible, no canary testing, no mandatory tests befor deploy, etc.
And the stock sinks lower.
Tech is not one of those industries of redemption. In tech, you either make a dynamite product (in a good way), you fail, or the gov't bails you out. Crowdstrike now has to elevator pitch again, why companies should keep their contract with crowdstrike. You can literally go to another company (or kid) who can program the same thing or less or equal the bid. Crowdstrike is going to have to prove themselves via a hardened audit. But maybe I don't know; they might have some leveraging blackmailing information on somebody and they're going nowhere but up.🤷♂