Basically they did, back to the Board of Directors, they make the choices and the CEO is a puppet winging his little head in a positive manner, it's not surprise they all seem stupid while their CV says another story.
How dare you? Your curelty it's beyond human! [Sarcasm starts] How come are they gonna pay teir 5th mega yate and the 10th mansion with a peasant salary?[Sarcasm ends]
I worked for a $2 billion company in the US. It was outsourcing software development to an Indian company. I designed a new command for managing a hardware device. The essence of my design was to gather information about configuration from a database. It was generic - it wasn't dependent on the hardware model. The team in India wanted to hard-code for each hardware model. More money to make if a new model were introduced. My design was eventually used, after significant pushback. The software was still in use 20 years later. I should point out that it isn't just outsourcing to other countries. Outsourcing to big domestic consulting firms regularly results in catastrophes.
I will give consulting firms one, small, bone. They're often brought in by companies which have some horribly outdated process or another and asked to deal with a problem for the least amount of money. When often the problem is no one wants to actually fix the real process problem. As an internal engineer, you're used to the problems and know what the real requirements are. Meanwhile, the consulting firm is given a set of crappy requirements that are very specific and are not forward thinking at all. Plus, sub-contractors are used to working with insane clients. For example, on one job the customer demanded we use their ticketing system. To create a ticket for a feature or defect, I had to fill out a PDF and E-mail it to someone. That person would then click the "new" button I wasn't allowed to and manually assign the issue a number. The ticketing system assigned a number and prominently displayed it, but they did not like it, so used their own number in all the paperwork.
So based on the difference between what thlse HCL employee resumes claimed vs reality the 737 Max flgiht crashes and deaths can be chalked up to lndia's penchant for being cheats and Iiars. Got it. Also stupid white peopIe being so trusting and thinking that aII cuItures are as good as their own...because they're aII equal (i.e. the same). They don't aII have Christendom. They don't aII have the enIightenment. They don't aII have the renaissance. They are not your equaIs.
As a remote Dev that lives in Africa and works for an American company I do agree actually, the big outsourcing firms have a problem, I'm an independent person I just apply independently for remote jobes. I do try to make sure I do thinks right, follow best practices, write things that are maintainable and scalable. But then again I would never work for $9 an hour so it is what it is
Boeing used to be run by engineers. Now, it's run by financial types whose prioritize profits over everything else, including safety. Back in the 90's I wrote software for air-traffic control systems. Everything was designed, reviewed, and thoroughly tested, and had redundancy built into they system
Oh please stop this accountant bull crap. Dennis Muilenberg (former Boeing CEO, just in case you forgot) is an engineer, and look where Boeing went under his watch. It’s simply greedy a-holes that put greed above anything, not just because someone is a “beancounter” or not.
That is very true. Boeing CEO has been a marketing guy, economist and stuff like that. The times when CEO was a real engineer, are long gone. I made another comment here to tell how MCAS works fine, but the economy-people wanted to get 19000 USD for a software flag that could say "AoA disagree"-alert. Lets see if that comment will be visible, and how it will be at least shadow banned. And yes, there lies the problems of the Boeing Commercial Aircraft Company. The CEO is not an engineer, in a company that is 90% engineering and creating NEW stuff and items. Now Boeing has changed their CEO many times, and still, none of the new ones are engineers. As that is the root cause, including that the current new management board is bunch of children straight out from Harvard (etc), they have zero understanding of anything in airplanes and aviation. The good board members all resigned and stayed in Seattle area, and these new clowns are the 100% culprits of those really bad design principles. For economist, marketing etc person, it must be hard to understand that engineers need to be there, in a huge engineering company, but again, they still haven't tried that. And that is why Boeing will not recover, until they hire from their knowledgeable engineering ranks, and stop murdering whistle blowers.
Since they acquired the DC Douglas folk, they also inherited the toxic management culture which contributed to their current woes. The solution is really simple for Boeing - if you are an engineering / aeronautics company, make sure to put engineers at the helm, not the ones with cushy MBA pencil pushers with their greedy business models.
Companies are turning into financial business instead of serving the purpose of their particular existence. Product quality, customer satisfaction, and worker happiness are all going out the window for "maximize profit".
This is probably why so many large IT projects go wrong. It is not that programmers in 3rd world programmers are not smart, it is that they are often managed by non-technical “managers” who are thousands of miles away while also not scoping out the project correctly. Failure is inevitable.
They are smart but their priorities aren’t the same. They aren’t concerned about safety. If it breaks, you have to deal with it. They only care about maximising the amount of work they get from you. They don’t care about readability or reusability etc.
@@OneIdeaTooManyyep. This is an issue with outsourcing overall. It can be cheaper but the outsourcing company has different priorities. I see this all the time.
@@OneIdeaTooManyI'm an engineer in a third country and it is not like us, the sw engineers, are not concerned about safety. It is more like we are understaffed for the speed they want us to deliver and management prefers if we deliver something 90% tested if it takes two weeks less. Just as Boeing management is trying to bank as much money as they can, our management is doing the same. Fuck capitalism and shareholders that is
Yeah they really are just fiends for work. Understandable why as their country of residence would pay them in peanuts compared to salary from a first world country, but the work this mentality creates harms the entire project.
Indeed the reason of outsourcing now is very simple. CEO and C-level cut cost, get bonuses and leave with gold parachute. Company can die after after that, but it is no interesting for C-level anymore, they don't try to build something, just max money for minimum time. And even after failure in one company, they will be willingly hired in the next one.
Couldn't agree more with this. Also telling that there have been multiple studies that have failed to find a statistically significant relationship between CEOs who move between companies and the performance of those companies, yet the companies feel they have to pay ridiculous money to get the "Best" CEO.
Agree. Some of the best work I've done was in collaboration with an on-site Subject Matter Expert (SME). If you look at the companies which are successful at using foreign labor, you'll see close working collaboration and at least one person who knows both the company and the foreign culture.
@@username7763 Most managers nowadays simply have no domain knowledge. Forget about the engineer that did the jobs well and decided to take that lead in his/her company. Does not happen anymore.
Hi Dee, if I remember correctly about the MCAS system it was the lack of redundancy in the sensor that reports the aircraft angle of attack was the root cause and rests squarely with Boeing itself. The sensor would erroneously report a stall condition and then the MCAS system would nose the plane down. There was a stacking of issues starting with having a critical a flying metric having having a single point of failure whereas several sensors are needed. There were also issues of FAA collusion in ignoring the Boeing safety failures as well as Boeing hiding the MCAS system so that no new pilot training (expensive) would be required. The insane level of disregard for passenger safety cannot be explained by cost cutting measures designed to save money. It is absolutely puzzling to me.
The outsourcing of development was a huge problem, you're right, but I think it deserves more context. This was all part of the arbitrary top-down approach to driving business, instead of understanding the problem and the product. As a response to a popular aircraft development from Airbus, Boeing decided to fast track development of a new plane by basing it off of an existing one. To make the plane more powerful, they had to install larger engines, which necessitated mounting them differently. Mounting them differently changes the physics of how the plane flies, and they tried to compensate for that inherently less stable design with software. In addition, where there should have been two sensors for the MCAS system, they saved cost by only including one and leaned on software. To shave off additional time and cost, they did the least amount of testing possible and claimed that the plane was essentially similar, so the only additional training required for pilots of the existing 737 was a 1 hour iPad training. The software was just one part of the confluence of shaving off every penny even when it wasn't appropriate. It really seems like companies today are no longer interested in producing a good product but instead the minimum viable product.
Slight correction, 737 max engines are not "more powerful" they're more efficient because they have larger fans and larger air bypass volume. Efficiency of fuel consumption is king in airline industry. Agreed on other points
@@thecompanioncube4211 That's a fantastic correction. Thanks a lot. It's cool to know that making them larger can have the ability to make them have similar power output at a lower energy input.
Another thing if they were even told of the system pilots were not explained how to override it if it was wrong. So even if the pilots knew what was wrong it could be possible they didint know how to stop it.
I will suggest that MCAS software worked exactly as designed, but the MCAS design was flawed in at least two ways: first because it depended on a single point of failure by using only one angle of attack sensor (though the 737 has redundant AoA sensors available), and second because it didn’t stop trying to trim the aircraft when the pilots were trying to override it. The third fatal mistake was not even informing many pilots that the system existed, so they didn’t know there were switches to stop it located in the center console. None of which is really a software development issue in my opinion. Respectfully submitted.
MCAS evolved overtime. It started out as a minor feature on the airplane that didn’t matter. But over time it became flight critical. It was never handled as a flight critical device so there was no failure. Analysis done. I don’t know if the systems engineers knew that this had happened. Or didn’t want to bring it up because it would push the schedule back.
The maneuvering characteristics augmentation system was implemented by Boeing for the 737 MAX because the aircraft was being built with different engines that, due to airport gate design, must be fitted low and therefore can cause differences in handling. To avoid the costs of retraining pilots and making them get new type ratings, the MCAS was designed to make the MAX handle like previous 737 models. Putting costs ahead of safety was unwise in my opinion. As long as corporate executives are more concerned about the price of their stock options, perverse incentives and shortsighted decisions will continue to plague us. Respectfully submitted.
@@ADHJkvsNgsMBbTQe So you’re saying when they added MCAS they knew it was unsafe? It wasn’t the implementation was unsafe. So the trade-off you claim was never made. If implemented properly, it could’ve achieved the same type rating. But since it wasn’t implemented properly, it was decided to add the training anyway. The issue was trying to get the aircraft, certified cheaply and quickly.
I remember when Siemens Medical laid off people with decades of experience to contract with an Indian company for dev ops. The new guys took 5 times as long to get things done and often had to reach out to others to figure out how to do their job tasks. Brass did this on purpose to jack up company stock before selling to Cerner to get the price up for stock holders. It's a shame that ethics, quality, and standards are all just things you can cut for capital.
I interviewed a lady that worked in a company that outsourced software development to an Indian company. She told me it took more time to explain what was to be done than do it herself. However, management thought otherwise. I personally dealt with three distinct cases where the outsourced Indian/Pakistani developers did not deliver: a restaurant application, a car spare part application and a ERP system. Spaghetti code full of bugs. Do I imply Indians are not capable? Certainly not. It's just that those that are good developers are not cheaper.
You're totally right! Gave a simple task to two developers, they spent more than 100 hours, plus my time on daily meetings. Their process takes hours to produce results after any changes. I wrote the same process for 3 hours, mine takes 35 seconds to produce results.
Indians themselves are not incapable. They have created really good companies and services from India. But when it comes to IT outsourcing, my god it's a sh*tshow. If the system itself rewards incompetence and boot licking, you shouldn't expect anything good from those companies. You don't manage 200k employee company just like that
a friend of mine is working as a freelance consultant, most of his gigs are about fixing spaghetti code that other people make, and they pay him big bucks, turns out its 95% indians, might be anectodal but it is what it is
I am a retired software engineer who worked in software for my entire adult working life. I was appalled when I heard about the design defects of the MCAS software in the 737 MAX, and how the people at the top of both Boeing and the FAA seemed oblivious to the problem. The Crux of the problem was that the software would ignore incompatible readings from the AoA sensors, and make automatic trim adjustments using only one (possibly defective) sensor. That is an egregious design error. The only reasonable thing to do in response to disagreement of the AoA sensors is to alert the pilots and refrain from. aking any automatic flight control adjjstments depending on those sensors. Outsourcing software explains it. You have the programming being done by people who are not specialists in Boeing flight control software, and you may have poor communication between the Boeing engineers who (presumably) know how things should work, and the people doing the programming, who may know very little about airplanes.
They used to teach ethics in college and the idea was not to make questionable decisions that impacted safety and other things like reliability. How in the hell did we forget how to make software and/or manage companies. Is higher-ed part of the problem?
One of the other issues in outsourcing to cheap staff is staff turnover. As soon as the programmers can they will change jobs to get more money. So at 9 dollars an hour it is very likely that you are dealing with trainees. As we know a good software engineer builds in things which are not specified. No spec is complete. We test for conditions that are not specced or raise them to get a decision. Trying to do that by phone, through time zone differences is challenging so it’s just easier to do what is in the spec and consider it done. Bad decision Boeing. Google you will not get the innovative thinking that has been the hallmark of your company. Not because there are not creative people in these low cost countries- they just won’t be working for you at 9 dollars per hour. They are doing their own start ups.
@@brightonbabe2139 Just one thing - if you are developing avionic software it's normally done in accordance with DO-178 (or the equivalent European spec ED-12) - and that specifically prohibits implementing any behaviour in the code other than that which is specified. For example, in the case of MCAS if someone had decided to implement a consistency check between the AOA sensors then that code would have failed review because you couldn't point to the part of the spec that requires it.
Once you've written specs so detailed that a $9/hr developer can understand, it doesn't take much more time or effort to just code it yourself. I just don't understand this approach.
@@arthurdefreitaseprecht2648 I heard from the management many times that you can teach anyone to write code well, but it has never been true in my experiences. First of all, *most* of low-cost developers do not care much about learning. They think they can just follow instructions without real understanding of stated goals. Moreover, even the minority who care about learning something new often are very slow learners... Take chess as an example. Can you teach anyone to play chess? Sure, but can you teach anyone to play at the GM level, I am not so sure. Even it was possible, can you imagine how much hours of teaching would it require, especially if one is incapable to learn anything on their own?
So based on the difference between what thlse HCL employee resumes claimed vs reality the 737 Max flgiht crashes and deaths can be chalked up to lndia's penchant for being cheats and Iiars. Got it. Also stupid white peopIe being so trusting and thinking that aII cuItures are as good as their own...because they're aII equal (i.e. the same). They don't aII have Christendom. They don't aII have the enIightenment. They don't aII have the renaissance. They are not your equaIs.
A complicated, or large, application, especially if it is "real time", is itself too difficult to code by people whose focus is algorithms. The algorithms require algorithm experts, and the code requires code experts. So, systems engineers write the algorithms, and the test procedures, and the software people write the code.
The MCAS software did not malfunction. The software did exactly what the designers intended it to do, but the designers had forgotten to include the possibility of bad data getting into the system. From the fix to the software, limiting the amount of nose down trim that could be applied by the MCAS to strictly less than what the pilot could counteract by pulling on the control column, we can determine that the MCAS would keep applying nose down trim, if the angle of attack did not decrease. This meant that the MCAS was unknowingly programmed to crash the aircraft, if the angle of attack sensor provided bad data that was persistently high. This is an example of code being a correct implementation of a flawed design.
Well said. This is why the business context understanding is so important. It is hard to add related use cases/acceptance criteria to things because it makes the feature too big for one sprint.
The system also lacked redundancy, as the AoA data was provided from a single pitot tube in most cases - I think *some* 737s are specced with with redundancy of AoA sensors, but it was/is a costed option. I don't remember whether MCAS ever took results from the redundant sensor (if it was available) anyway?
Cheapskate accountants chopping software engineers is nothing new. Years ago, I worked for a risk management company in the UK and was sent over to Mumbai for a few months to beef up and get the team there up to speed. Whilst I was away, a top-notch fast and accurate software engineer (that I had recruited), worth 4 average engineers was ditched by the accountants because "he was expensive...". Output from the UK dropped noticeably and I quit not so long after, tired with trying to get cheap, inexperienced guys to work accurately. The stories I heard from other departments where the users of the software internally (before selling or upgrading to clients) would berate the engineers because of stupid bugs where the engineers would blithely reply, "Oh, yeah, we know about that, you do x as a work around" was driving us crazy, and I had enough of the fire-fighting and the constant battles with "cheap" engineers to bring them up to spec. The company eventually folded.
Yep. I too have been "downsized" as I was the most expensive. CEO seems to have forgotten that not six weeks before I pulled an all nighter (unpaid) to get enough systems working that we could participate (sucessfully) in an industry wide demo of our tech. Without that, our company would have been a laughingstock. But I was expensive.
So based on the difference between what thlse HCL employee resumes claimed vs reality the 737 Max flgiht crashes and deaths can be chalked up to lndia's penchant for being cheats and Iiars. Got it. Also stupid white peopIe being so trusting and thinking that aII cuItures are as good as their own...because they're aII equal (i.e. the same). They don't aII have Christendom. They don't aII have the enIightenment. They don't aII have the renaissance. They are not your equaIs.
Same thing is happening to Enbridge, a pipeline operations company. I know that one day there will be another major oil spill because of the sheer incompetency coming from the people that safety-critical roles have been outsourced to, Tata Consulting Services. I got so frustrated and tired of teaching and coaching them that I left a month ago, after seeing how many great engineers and colleagues have been let go.
Great video. Outsourcing isn't just a recent trend. It's been going on for decades in tech. I first noticed it in the 90's. It was and is getting more prevalent. After that I never worked for a company that didn't depend on cheap contractors. It was always a mess trying to get good or even correct code - cheap labor is cheap labor. Combine that with time lag, language issues and lack of product knowledge or ownership and you get what you get. :)
I have worked with some great software engineers in India ... but the vast majority of them end up in the US. The best engineers don't stay with contracting companies. This is also a symptom of an even greater issue that is not specific to India or contracting companies. For years, we've pushed people to enter this field with little regard to aptitude. I taught myself to write my first program on my TRS-80 in 1982 using the manual that came with the computer and books from the public library. Started with BASIC (of course), moved on to 6809 assembly and Lattic C on a PC XT in high school. I'm now a senior director of software architecture ... and still write software daily. Why? It's just something (frankly the only thing) I seem to have a natural aptitude for as well as a genuine interest. The same could be said for most any technical field. For some reason, we've gone so far over the top in encouraging everyone to "learn to code". CODING is one thing ... being a software ENGINEER is quite another. I can (and have) installed my own crown molding - but I am certainly NOT a finish carpenter.
Oh man! Back in the day, the software product was expected to be as perfect as humanly possible. Of course the software environment was quite a bit simpler. Meaning you didn't have the WEB with all its kooky frameworks and languages and design patterns. It was classic waterfall without the need for things like Scrum. Times have changed and not for the better.
When I heard that Boeing put a single monitor on the outside of the aircraft that if it malfunctioned, would drop the nose of the craft and that pilots would tire within fifteen seconds while trying to regain control, unless they properly reset the system after reading an update to a 500 page manual, I thought... who the hell doesn't have multiple points of redundancy? Who the hell doesn't train pilots on new features? Oh, they fired all of the software engineers whose job it is to think. I could make that determination in my company back when I was getting paid like $70k a year for a $50 million dollar company as a junior software engineer. Boeing, a $50 BILLION dollar company really has NO software engineers that are to that level of competency?
They 737Max was design to compete with the Airbus A320 Neo which uses the new Leap engines which are larger but uses less fuel. Fitting this to the 737 design made the aircraft have a different centre of gravity so MCAS was to make it behave like an old 737. So no retraining or recertification for the pilots were needed since they claimed it worked like the old one
@@tibbydudeza Yep, that was one of the big things. They deliberately didn't train the pilots! Since doing so would have made the 737 Max cost more to operate.
This is not the true. They did safety evaluation of the mcas with some requirements and they end up that risk is minimal so mcas may not have redundancy. And that was all fine. Then someone changed requirements to be more aggressive but there was no second evaluation that could end up with system redesign. And this was not a fault of 10$ / hour devs.
Exactly. If they'd had more experienced internal engineers, they'd have realised the flaws and raised hell about them. This is also what management wanted though. If you read between the lines, you can tell that "engineer says no" was seen as A PROBLEM at Boeing, not a strength. All management could see was "this guy who makes WAY less than me is causing me problems", and their solution was "hire a cheap contractor who WILL DO WHAT I DAMN WELL SAY". The basic problem is that management were making engineering decisions that they weren't qualified to make. They thought they had every right to make those decisions because... aren't they in charge? They didn't realise that reality gets in the way. IMO, this was all 100% a management problem. It's 100% about managers being UNQUALIFIED to manage the things they're managing. There should be no such thing as a manager, at every level, right up to the CEO, who doesn't have at least a decade of actual engineering experience (NOT in management) before being promoted to manage even one person. The "non-technical manager" is the modern scourge of all technical companies, and the source of an overwhelming number of problems.
Dude, working for defense and common fortune 500 companies, a software engineer who tries to discuss tasks with managers or innovate is now called "insubordinate" and "not fit for development teams". All management wants now is go go go, must make the sprint points and deadlines!!! No thinking, just code!
I worked with the HCL guys in India while in the finance sector, back in the early-ish days of HCL. They were very sharp guys all, even the cheap ones. The issues that did come up, stemned from poor communication from our end as both sides had major technical, social, cultural blind spots towards another, many of them unmentionable. Looking back it was shocking the result was as good as it was despite those issues.
I want to clarify something extremely important which is misrepresented in this video. MCAS was NOT a software error. The software performed its function exactly as designed. This issue was with the system design: the designers deliberately designed it the way they did to skirt regulations, and failed to account for what real world implications that might have. I think it's extremely unfortunate that this nuance is lost, and it's simply reported as a "bug", since categorising this as a software issue downplays the disaster to a simple mistake, and not the deliberate and wilful choice to prioritise profit at all costs over safety that it was. Unfortunately a lot of people are now using the troubles at Boeing to push a union focused agenda of "this would never have happened at the 'Old Boeing' when everything was done in house", and while this is probably true in many cases, there are also plenty of situations (like the article you're reading here), which just seem to be a pretty transparent and blatant attempt to mislead people into thinking that Washington Airplane engineers are somehow some superhuman mutants who never made mistakes of any sort, period, and that land of milk and honey will be returned if only they are all rehired again.
I agree. The main problem was using one sensor for MCAS and no backup or second sensor or a routine to check for sensor errors and what to do at that point. It was a system design failure and not just some bugs in the code. Thanks for pointing this out .
@@nine7295Think of how many engineers at Boeing must have signed off on the one AOA sensor. I wonder if they were pressured to sign off or just didn’t bother thinking about it.
@@mshepard2264Once again, Upton Sinclair: ‘It is difficult for a man to understand something when his salary depends on not understanding it”. One of the most insidious things is that Boeing charged extra for 2 AOA sensors and the software that alerted if they were not in agreement.
One more trend in Indian consulting firms is that they would have one experienced dev and rest junior devs. Where you would end up with just garbage code due to tight deadlines, experienced devs burning out and quitting, also junior devs don’t get mentored and grow.
Western consulting companies have the same strategy. Quite sad. Experienced devs fight tooth and nail to escape these terrible places. But slowly and surely the governments make sure this is the only future ahead.
The way it works is that the outsourcing firm submits the resumes of their "star" developers and when they start the project, they pull those top-notch developers and leave you with those 9 dollar an hour characters. I've seen that happen so many times. It's completely ridiculous.
From the little I learned about the system, the main issues had nothing whatsoever to do with the code. They were design problems that were based on decisions almost certainly NOT made by programmers or 'overseas' personnel. The two main issues I am aware of were that they deliberately tried to hide the system from oversight and made some decisions for that reason and there were insufficient safeguards if a sensor was giving incorrect readings. The aircraft has two sensors and even I, as someone who has not worked on aircraft software (although I have built a customer reservation system for a chartered aircraft), I know that if you have two sensors then you should read both sensors and if there is a discrepancy you should do something about it whether that be figuring out which is correct and relying on that or informing the pilots that there is a problem etc. But instead they simply use only one sensor and ignore the other making the whole redundancy aspect totally useless. That was NOT a programmers decision. That was a management decision.
Issue is: without the experience and maturity you DO get from well paid, market rate software developers who DO see the total design even though they are on code, you wont have people who state a simple and obvious intelligent point about redundancy, design and integration of the code theyvre written. MOST good software devs will absolutely just tell you when the glaring implementation fault is there. And yeah just make sure you get decent people working on an integrated solution in this scenario, its peoples lives. Boeing should be ashamed.
@@johns5558 In my own job I often point out glaring design flaws or things that should be done, but the reality is that in corporates that doesn't always result in action. The fault lies squarely with management in this case. If I was the programmer, I would have definitely suggested improvements to the system, but if management said no, just stick to the design or we get someone else then I would have stuck to the design and got paid. Outsourcing does have its problems as the employees feel less attached to the company as a whole and are less likely to 'do the right thing'. But lets be clear that Boeing outsources almost everything (and has suffered as a result) and that has nothing to do with whether or not the outsourced company was in India, Italy (recent case) or the US. I once worked for Vodacom (a telco in South Africa) and two thirds of the people at work were 'contractors' including myself. It worked, but it had its problems because there was no job security. Outsourcing has its pros and cons. A contractor is under higher pressure to impress than an employee, but they are less invested in the long term health of the company. I remember a recent case that was reported on where a bank fired a large number of employees because they had installed software that emulates moving the mouse and keyboard. They did this because management was monitoring them and evaluating them with software that monitors mouse and keyboard activity. The employees were fired when the right move would have been to fire the management. Management is hard and evaluating employee performance is hard and when you get it wrong you get the wrong results. Boeing values cost savings over safety and that is what they get.
This is a trend that started decades ago and made famous by Jack Welch as the CEO of GE. When an organization gets hyper-focused on the numbers it starts to get hollowed out with cost-efficiencies. Give it long enough and like GE, it loses its intrinsic value and becomes a hollow shell with significantly reduced value.
Wrong. Insurance industry puts values on lives. Issue I have is every family got a least a million a person for wrongful death but if you try to insure your spouse for that they want to put you in jail first and then ask all the questions after “why do you want that much money”? Shoot even 250 to 500K on a spouse for “lost wages” if they die isn’t cheap to buy. Totally a money grab by the dead families. Less than 1 million dollars US for a life. I’d say actuaries peg it at 250K
This sort of poor management is a problem at Boeing, and MANY other companies and organizations. The senior management no longer has any real world engineering, production, leadership, and basic business skills or experience. They listen to marketing/sales first, cost accountants next, and finally quarterly stock market returns. Senior managements are the winners of decades of internal power politics, in some companies a low grade war. This is all a long way from building great airplanes.
Sorry, but it wasn't a software problem. It was a system engineering problem. A sensor with SIL4 (Safety Integrity Level) responsibility (highest safety level) was electrically connected like a sensor without safety responsibility. This was not a software error but a system error.
My understanding is that the aircraft had 2 AoA sensors, but they made the decision to only take input from ONE in the MCAS. It IS a system error but potentially one that could have been spotted and flagged had the devs had more domain knowledge.
@@ralphprice7365 I genuinely don't understand what point you are trying to make. Not saying you're wrong, just asking for clarification. My understanding is that the problem was that it only used one sensor and so if that sensor developed a fault that caused it to produce bad data, without any error conditions, the system would behave badly. The system error was not identifying the single point of failure and using both sensors to detect bad data. I'm not sure what relevance handshaking with hardware has in this case. Did they discover bad handshaking? I missed that, if it was announced.
@@ralphprice7365 Boeing was trying to "hide" the fact the 737 max tend to pitch up when pilot throttle up ( due to the new layout of the bigger engine ) . Boeing is using MCAS as a "mask" of the real problem of a badly designed engine layout. The worst part is that a running MCAS does not give pilot the authority to manually correct pitch. The pilot has to turn off MCAS .
I agree with your assertion that chasing low cost/lower skill engineering talent can often be more costly in the long run, but I believe it is incorrect to blame software engineers for the issues with the Max. The 737 is a design from the 1960's. The original airframe was not designed for the large diameter, high bypass turbofan engines used by the Max to improve fuel efficiency. To provide adequate ground clearance, these bigger engines had to be installed further forward and upward on the wing than the engines on prior 737 models. This moved the center of gravity of the airplane, which in turn changed its performance characteristics at high angles of attack. Boeing needed the airplane to fly just like previous models of 737s so their airline customers could avoid costly aircrew retraining. The MCAS system was added to help pilots recover from nose-high attitudes and make the airplane appear to fly just like earlier 737 models. Unfortunately, the MCAS system was only fed by a single angle of attack sensor. If that sensor malfunctioned, MCAS would aggressively push the nose down with more force than the pilots could overcome. MCAS was a band aid created to squeeze more life out of an outdated airframe. It enabled Boeing to offer a product to compete with the Airbus neo series without the massive expense of developing an entirely new airframe. The entire idea was flawed. The decision to control MCAS from a single, non-redundant sensor was crazy, and the failure to inform or provide training about it to pilots was inexcusable. These actions caused the Max crashes, not the low cost software engineers. The MCAS software performed as designed. It was just a horrible design.
From what I gather, there was nothing wrong with the software, the problem that a single sensor was used, which was Boeing's decision. I work in the oil & gas industry and all our safety systems use 3 sensors and 2 out of 3 voting on what signal to use. Additionally, measuring devices are able to indicate when they have a fault. Again, Boeing's choice not to do this. Boeing made the AOA disagree alert an optional feature and it looks like they never considered what would happen if there is a fault on the sensor, which by all accounts is quite regular.
I did some work for a Boeing division (off and on) over the years. When all the stuff going on with the Max happened and the MCAS was identified as the main culprit I hear from a lot of engineers that the real problem was as you suggested with the placement of the engines causing a "hack" to be implemented with the MCAS. Boeing got caught with their pants down with the 737 vs. Airbus A320 Neo. They couldn't take the time to redesign the airframe and landing gear area to gain more clearance for the bigger fuel efficient engines. Basically they would have to go through flight testing and certification again, which would have costed them 5-8 years of work, to be able to compete with the Neo. It would have seriously crippled them in the market as the 737 was their most popular plane.
Yes, But, it was not the retraining but the very costly type specification which prevents changing major parts - like the distance from the wings to the ground of this type. A new type specification costs billions.
@@smmasongt Now you're just blaming the engineers 😆 Their concern is not cutting corners with bad design to compete financially with Airbus, that's on the executives. My guess is most of the engineers looking at the plans thought "WTF are you doing?!" In fact, many of them complained to management and were ignored or told it would be fine. I mean, technically it would work, it was just precarious.
Great Video Dee!! Yes, Quality is the key here. We also don’t know who is coding under such cheap contracts. Many resumes I have seen, when this takes place, are somewhat hard to validate and many candidates cannot back it up. This is even evident back here in the states, because the model of these outsource firms is being stablished here, undercutting experience developers with very inexperience ones, yet they are listed in the contract as senior developers. I have seen this first hand, since I would have to support and teach many of these so called “senior” developers.
I've been telling my tech compatriots for a long time now: You get what you pay for. When you pay $9/hr for a software engineers you get someone unfocused who is doing multiple contracts at the same time who will give you fast-food quality if you're lucky and toxic quality if you're not.
There are other "hidden" costs to this outsourcing to the cheapest bidder. The people left behind to manage the outsourcing are demotivated because their job is reduced to a high impedance clerical job where nothing they can do will improve things. Either they resign themselves to it and productivity reduces further or they resign completely and the job goes to someone with less understanding of the actual requirements and zero interest in fixing things.
You are right on, yet it applies to local (high paid) employees, too: when I do external consulting (at high hourly rates), I usually solve tasks in a fraction of the time it takes the employed staff, making me a bargain. I have no interest in stretching time as I have enough other things to do and want to be creative and not bored. Work for me is part of life, not a separated balance. So outsourcing seems just a part in a more complex game that‘s about „competition“. Whenever there is honest competition and true evaluation of individual service vs. cost, things work out. Whenever general „standards“ and „procedures“ dilute individual performance, efficiency goes down. Outsourcing often feels like the helpless controllers’ answer to problems that started locally: when qualified local engineers become uninspired and start taking advantage of anonymity and job security, it ruins local productivity and then triggers unreflected outsourcing. Obviously, outsourcing is just trying to shift responsibility but that most often doesn’t work and won’t solve the original problem. The answer may just be to focus again more on *individual* performance, regardless of where the engineer is located, and pay for what you get.
Hey Dee, this is a very refreshing channel for me as a "long term > 30y" Computer Scientist (MSc.). Working in the automotive industry for the past 10+ years, I can confirm that outsourcing is a huge issue due to the cost pressure in the industry. But cutting costs will eventually cost lives!
You missed the entire reason these CEOs do this. It all comes down to how they all pay themselves with lots of stock to gain more control and if they do it well, a board seat. That one dollar salary steve jobs took with stock was the biggest grift of our lifetime. It is all about stock. By paying themselves in stock, they get all the voting rights and get to act as an owner. To get around the fact that they can't sell the stock without losing voting rights, they take loans against the stock instead. The loans are tax free and once rich execs figured that out, they all started taking large amounts of stock as compensation. This means when they pay themselves in stock, they get both the voting rights and up to the entire value of the stock in cash tax free. They can loan against smaller amounts at a time so they benefit even more from their own stock price manipulation in the future. The voting rights are used to run the company based on stock price. The goal is to try to push the stock price up at all costs so they are not forced to sell stock to cover the interest payments or aren't forced to sell stock if it drops too far. The only outcome they want if there is a decline is a quick bankruptcy because the banks eat the losses and the CEO is not forced to pay any loan money back. This is why they also created the golden parachute. They know bankruptcy is almost a guarantee over time, so they plan for it. If they bankrupt the company, they get paid huge cash payments as a reward. This is why the modern CEO will cut from anything and will outsource everything if they have to. Natural growth of the company value is pegged to inflation. That level of growth does not satisfy their scheme of paying themselves lots of stock bonuses. They negotiate bonuses based on the stock price going up, juice the stock, get paid lots of free stock, and become an owner with a board seat to ramp up the juicing even more. Look at how every large company has billions in debt and most CEOs are also on the board. These companies are faking profit increases and being loaded up with debt. When the ability to raise cash runs out, the company will either be sold to avoid the scrutiny of a bankruptcy judge or go into bankruptcy. Either way is fine, the golden parachute will be paid out in some fashion.
Was that the method the new MD leaders took over the existing Boeing leaders, after the merger with McDonnell Douglass? It was the MD leaders that changed the company’s focus to short term profit and caused all the damage.
I enjoyed this video and will watch for others from you. But I'm surprized you didn't also mention the space capsule "Boeing Starliner" with its failure to reach the Int'l Space Station in Dec 2019, and with the helium leakage problem which prevented the 2-astronaut crew from returning with the capsule last month. You could also have mentioned that Boeing's merger with McDonald Douglas in 1997 has been blamed by some for a change in corporate culture which shifted the emphasis away from engineering excellence to cost-cutting measures.
I'm from the other side of the planet, but all you say here is like listening to my own thoughts. One thing I would like to add, though: 10 years ago I used to work in global corporations with software development sites in the cheapest and largest countries. We had all the classic problems that come with hundreds of beginner coders working with little to no context and it was so painful. At some point I managed to mentor a couple of the most promising ones to a decent working proficiency in the tech and domain we were working on and I was so relieved: finally we could get some work done. But, alas, next week those people had switched jobs. And I don't blame them. They probably got 100% pay raise from some other place and had literally no reason whatsoever to remain with their exploitative and cheap current job. There's just no way you get anything but what you pay for in this world and imagining otherwise is just fooling yourself. Any executive, director or manager falling for this trap should be immediately fired for incompetence.
MCAS was a fairly robust system done a while ago for the military. Digital augmentation of airframe characteristics isn't a new thing in military aviation with fly-by-wire inherently unstable fighters, planes having asymmetric load-outs when they fire externally mounted missiles or receive battle damage etc. The issue was implementing it on the MAX while making the angle of attack sensor non-redundant.
Right, but if you disconnect the software developers from the whole implementation then there's the issue. that they are paid $9.00 an hour only tells you that there's so little experience that not only would they not know about redundancy in the total system but at $9.00 they wont speak up about it either. There are few good ( / well paid ) software developers who won't at some point tell you if there is a fundamental design flaw even though they arent the architect or designer. And they will know. This is the issue and this is Boeings penny pinching fault.
@@johns5558That’s not how software development for safety-certified systems (which includes avionics) works. The coders are the least important part of the process, and they have no influence whatsoever on the design of the software. That is done by different software engineers whose job it is to write specifications.
@@tookitogo , John, you're both almost there, but are missing the particular MCAS context. The *entire point* for that system was to be non-flight-critical & non-redundant, as per Boeing's own requirements - not $9/hr programmers' inexperience or indifference or the savings to be had in hiring them. We're talking about many orders of magnitude more money, elsewhere in the business, as the driver. Please allow me to explain. Let me back-track a little. The 737 sits quite low to the ground, which had played a big role in its success over the decades, as that not only reduced the landing gear weight & complexity & allowed it to be tucked inside the wing instead of the body, it significantly reduced the size and complexity of all the requited airport machinery for service / loading & unloading / de-icing etc. Personnel, procedures, didn't need to be heights-rated. A devastating competitive advantage. As material science and fabrication advances allowed greater engine fuel economy through larger turboprop bypass ratios thanks to giant front fan blades becoming viable, the engine diameters naturally grew. Eventually to a point when compelling new engines would fit under the Airbus A320's wing but not under the 737's wing. Boeing engineers pulled a design rabbit out of a hat, flattening engine shroud on the bottom, shortening its pylon and putting the fattest part of the engine way in front of the wing (taking up the wing thickness' worth of space). With all these measures combined, the new engines fit on the old airframe, which ( *this bit is key* ) qualified the 737MAX, as far as the FAA et al were concerned, for the simplified deployment, without extensive pilots' retraining incl simulator hours on what woukd otherwise have been a whole new type. This new type retraining requirement is worth many millions *per airline* in lost profits and risks - that manufacturers typically pony up. It was *key* that the 737MAX not require that retraining, because the A320NEO didn't, b/c even if the Boeing had paid up for the sims etc it'd still take pilots offline, giving airbus oprators a market share in over their brethen running boeings. With me so far? Good. There was only one small issue with Boeing's engineering miracle: the altered engine placement, in certain very small areas of a flight envelope, made the plane suddenly pull up. Boeing has had just the solution, digital flight control augmentation - a little autopilot of sorts that specializes in patching up / normalizing such small irregularities, making the plane behave seamlessly from a pilot's perspective. Many advanced mil airframes (fighters designed for an inherently unstable flight to maximize maneuverability) aren't even flyable by hand without such a system. The only pickle was, while just about everything of the sort is doubly or triply redundant on a military plane, adding a redundant system on a commercial jet automatically flags it as potentially flight safety critical in the FAA review (large chunks of which were delegated by the FAA to Boeing to self-certify). That would in turn require pilot training, defeating the purpose of the entire effort & putting Boeing back to square one. Boeing's plan - as per its lesdership and bean counters NOT programmers - was to sail under the radar of this review, by specifically NOT having MCAS redundant. The rest is history: the gamble didn't pan out, and whistleblowers stripped any shread of plausible deniability, which is how I'm able to explain to you all this without any rational fear.
@@AlexKarasev I am not sure if that reply was meant to come off as condescending or not, but to me it was a little. I already knew everything you said there. There’s no evidence of the $9/h programmers being used on MCAS. (The video quotes an article about flight display systems), and nonetheless, the coders don’t design the software. The redundancy “flagging” is a real issue, in that that is a _horrible_ reason to not make it redundant, obviously. As for if being safety-critical: any system that has direct control of flight controls _is_ safety-critical, whether it’s been categorized as such or not. I haven’t looked into details of this, but I’d be very surprised if the MCAS code was not designed using most aspects of safety-critical software design practices. (It obviously did not follow all of them, but that seems to be more deliberate than accidental.)
@@tookitogo certainly wasn't meant to be condescending - apologies if to you it read as such, same as for my part I had no way of knowing that all of this was known to you. I only had the senior engi eers make the specs bit to go on - whereas here we've the context that it was a business call even though in the end the senior engineers probably did write the specs. Although you're tagged in the reply, it's really for everyone reading this incl 10yrs from now. Who knows, maybe some young engineer or leader or investor will read it and understand how we humans sleepwalk into such disasters and humankind gains another percentage point towards some life not being lost somewhere in an avoudable disaster. This story needs to be told (I wish, taught) and not just in aerospace engineering / FAA / OSHA etc but everywhere decisions are made. There's a pattern to bad decisions that's very different from "those guys were dumb; we're smart".
India consulting companies have some very good engineers, however not many. They tend to give you the good engineers at the start of contract then swap them for the lower quality engineers after a couple of months. I used a company that over 2000 developers, one of my team would find the error for them, tell them the change the 1 PM, 1 manager and 3 engineers would make the change. When I found out that my engineer was basically doing all the work, they where gone. We weren't outsourcing because of cost, just lack of specialist skills (particular telecoms protocol) their good engineers where probably one of the best in the world for that skill. however they were in project for 3 months then gone, to another project.
Exactly this, another branch of engineering but the same story. We have local engineers who have moved here from India and they are just excellent. We send work to a pool of engineers overseas and it's multiple cycle time that finishes with local rework anyway.
Just some Indian context to keep in mind : India is also the country that runs ISRO, with its cheap, well planned and successful Moon and Mars missions, as well as highly successful commercial space launch industry. And ISRO is cheap, and a lot of ISRO engineers get paid at par with 10USD per hour, which still comes out to a lot of money here (= 10 x 84 x 22 x 8 ~= 150K INR / month) In my mind, outsourcing to a company without aerospace engineering expertise seems to be the actual issue. Practically half a dozen aerospace giants "outsource" to India, but they do so in the form of running their own branches in India. These engineers are paid anywhere between 200k to 1000k per month, which is 2500 - 12000 USD per month. They're the cream of the aerospace industry. HCL Technologies is not one of these. It is a body shop. I felt the need to say this only because a good friend works in a leading American aerospace company and he is getting awarded patents and certificates in aerospace systems. I would also like to add that cheap Indian developers are pathetic. I suffer them in office every day, being an Indian living in India.
Most outsourcing plans are entirely to cut costs, not for expertise. There are great Indian engineers but you don't get them when your goal is cheap labor. I shudder any time I hear a manager talk about number of resources. Its not about the number of bodies, its about the professional knowledge, ability and the team.
Coding for ISRO certainly is on another level. Once ISRO starts to put people in space, the quality of the code will need to be better still in terms of redundancy, and failsafe designs to keep crew alive. That's the standard that Boeing must always be at.
This is a little misleading. The code didn't have any problems , it was doing exactly what it was meant to do: take control of the plane when a single sensor failed, and tricked the avionics into activating MCAS. This was designed by the engineers at a higher level, point the nose down over and over if one sensor output data that said the plane's nose was pointed up.
They didn't outsource the MCAS, that was all Boeing genius. I'm unclear exactly what they did outsource to India, but they did outsource some 737 Max code.
True that. The MCAS software checks the ONE active AoA sensor (Angle of Attack, eq how high the nose is compared to the airflow around it). If the AoA gets too funky, the MCAS will start pushing the nose down by trimming the horizontal stabilizer to push the nose down. Everything is fine and dandy, as long as the ONE SELECTED AoA sensor is working well. Those senors are of course outside, they have air/sand/garbage flying on them all the time, so they get quite easily broken. Now the broken AoA sensor gives out some funky AoA readings. There is the problem. There is another AoA sensor on the other side of the nose/fuselage. That ought to be ok, but Boeing Chicago management clows (all economics people, kids, straight from universities, knowing nothing at all), decided that checking on AoA sensors, is not needed, and it is a "optional advanced AoA software package, for 19000 USD per plane". That software would then tell the the pilots (and MCAS) that "the left AoA and right AoA are saying very different results", eg AoA DISAGREE. Only Delta ever bought that software, but other airline pilots (not Lion/Ethipioan) also knew what is a runaway-trim and how to fix it. Those two airlines did not know and did not care. Pilots had to select left or right AoA sensor to be used, but as that optional software was not installed, they could not have "Auto, using both". Now when the active AoA sensor fails, the MCAS can freak out, and start helping the pilots by pushing the nose down. Again: Stop the runaway trim by pressing Auto-Trim: CUTOUT right next to the flap lever) AND/OR Stop the running trim wheel by hand (can be done easily). Continue the flight to the destination as normal, with of course now manual trimming. Easy. But again, not easy when pilots are such bad bus drivers, from the lowest tree branch, and Boeing board thinks that all of this is "act of god, nobody knows why the MCAS freaks out...." . Plus of course a lot of corruption, blackmail and threats, and murdering of the whistle blowers, which we are all sure will produce fine Boeing aircrafts in the future..... sick and sad.
I'm a controls Software Engineer and was born and raised in SA. I have immigrated to get a decent rate. Man, I do miss SA a lot!! Oddly enough, my situation reminds me of that Mexican fisherman Parable. Sidenote. Great video. Keep up the good content.
Ugh! This is depressing. I'm already ousted because of my age and it just frustrates me to see it the whole industry trending downward. I wasted money getting a degree.
You wasted a lot of money and most likely were misled about all the opportunities, good luck recouping your money. Sounds a lot like false advertising.
One additional issue is that when outsourcing work literally half-way around the world (e.g. USA to India) makes it very slow and awkward to communicate. Mid-day in one location is midnight in the other and with weekends being skewed you end up with an effective three day work week.
From a software point of view this video is spot on, though I happen to personally know that Boeing has far far far far more troubles than software. If it’s Boeing, I’m not going. MCAS has a shortage of air speed sensors such that even if the software was written by a genius it couldn’t safely do its job.
The C-Suites are universally staffed by managers that rarely understand the tech involved. In nearly everything. They are focused on the quarterly results, their personal stock options, and their personal advancement path. Boeing's previous merger caused many experience Boeing engineers to leave the company. The new management structure gave the world the 737-MAX. Oddly enough, one does wonder what those stock options are worth today.
I have worked for an outsourcing company. I fondly recall them designing and building an overengineered system to achieve something that could have been done by a few tweaks of existing code. A common theme with these companies was a lack of understanding regarding testing.
You brought up loyalty, which is a very good point since it goes both ways. Who is more likely to leak information or put something suspicious in the code base, your local salaried employee or a faceless €9-an-hour entity in a hive of 2000 drones?🤷🏼♂️ _But hey, it’s not like they would put that software in anything critical, like an aeropl… right._ 🤦🏼♂️
$9 per hour for a software developer is bad enough. But there is a bigger question: what about testing the software? Did Boeing test this software to see if it complied with the safety assessments or FAA regulations? Or did they recycle the tests from older designs that weren't as software dependent?
The bigger thing that a lot of airplane safety guys were screaming about right after the investigation came out was lack of redundancy. The software depended on a single sensor for its input data, and when that sensor failed, the software that relied on it went haywire. As you will hear time and time again in _any_ class related to industrial safety, the minimum is two sources for safety (when a system is allowed to fail completely) and three sources for fault tolerance (when it is not allowed to fail completely, usually for safety reasons). The software actually performed perfectly -- assuming the input data wasn't wrong and that the pilot would never need to disengage it quickly and intuitively (the process to disengage it was quite involved and rarely mentioned in training material, if at all).
@@mage3690 that was the first and main issue one of my neighbours mentioned as soon as details about the crashes had been made public. He was an engineer with Airbus. He said that such a design just wouldn't have received approval by European authorities which is why Airbus wouldn't even think of doing something like that. Failed approval procedures are just way too expensive.
Display software is not code that read and cross check the sensors...Who wrote that unsafe original code of reading from only 1 sensor when aviation is required to have a back up sensor. ie . there are 2 sensors to read and compare?
I went to business school, worked in business. I've also worked as a coder as well as system design overseeing developers. Knowing what and why you're working on something makes all of the difference to what you're doing and how you do it. Overseeing coders who don't know is much more effort. One idea in management is that people with the same job title are es interchangeable as different products with the same name. Neither is true. Modern managers are often administrators without any concept in the different qualitative factors that don't show in reports. That's why they focus on cost so much, it has a defined result. Investing in qualitative factors does not have an easily discerned result, so nothing direct to show. Managers without intimate knowledge of the product will neglect qualitative factors. One premise in management theory is that you can manage a company by changing the amounts in the balance sheet, incomes statement and cash flow statement.
02:18. Oh yes. I was raised a software engineer. Am now a projects manager. Management doesn't understand software. Full stop. Make box do thing must be easy right?
I spent 25 years in airline ops for various airlines. Over the years technology has pretty much taken away the common sense of engineers and other professionals. To give some perspective. I was involved with weight and balance since the early 80s. We were taught the theory as well as how to correct issues if the computer system crashed (very often in those days). We were also taught on how to use various airlines computer systems to perform our duties. Those days, we had to learn the computer entries and type them in. These days, it is just point, click, drag, drop. or even tell the system to "auto load". Sure, there are error checks, but if you do not have the basic knowledge of the principles, you are pretty much screwed if the error persists or if the system goes down. I use to work with avionics tech folks as well. These days, there is no thaught process involved. You just tell the "AI" what you need and poof - out comes aa sh*t load of spagetti code. Then you outsource because of "costs". Penny wise pound foolish. I digress.
Very good video and interesting topic! Please, keep in mind General Electric company and Jack Welch decisions… I think that there is something deeper here. Regards!
Thank you so much for bringing to light what IT people from Western countries have known for and experienced for quite a long time. Time when this whole outsourcing business model came up. The only thing upper to higher level IT management people care about is 'making an impact' to justify their jobs. Cutting costs is always a sure fire way to provide that visibility! As these people are mostly fixed term employees; aka contractors, all they care about is the bonus payout they will receive at the end of their term. Short term, short sighted outcomes will get them their KPI's. When these big outsource companies make their bid presentations, things look good on paper, structurally. But we in the IT industry know how easy it is in these 3rd world countries to procure the right credentials. They promise big but almost always under deliver or the quality is very poor. Not the $9 workers' fault as they're mostly working on 3 or 4 projects at the same time. It is modern day slave labour.
A friend worked for an engineering company when they outsourced the shop floor. The failure rate shot up because the in house engineers knew 20% of the drawings were wrong and how to compensate . The obvious solutions of either correcting the drawings or warning the new workforce was never going to happen if they were being laid off. Same with software, there is nothing like experience with the same company and no college or knowledge transfer will compensate for this.
As a 30+ year software engineer and coder, I dislike a lot of Indian outsource companies, but you need to do much more research here. As far as I can read about it, HCLTech coded the software correctly, and correctly to the Specification. The Boeing SPEC was wrong/bad! It does not appear the Boeing tested the system completely, or well. This will go into CompSci classes as an example of bad outcomes even when the SUBsystem SPEC is correct. A key aspect in the MCAS failure is that once the pilot overrode the system, it would wait 30seconds and then return to its bad behavior. Another aspect is using 1 sensor that is known to fail, which is why there are 2 sensors on the aircraft. BUT if they had used 2 sensors, they triggered the FAA DO-178C spec testing phase which is REALLY expensive.
The Boeing 737 MAX failure due to the MCAS system was a known issue within Boeing, and they deliberately tried to remove all mentions of MCAS in the training manuals to minimize pilot training time as much as possible. There is no possibility that HCL Technologies had any involvement with the MCAS system. Boeing has clarified that critical flight control systems, including MCAS, were handled by its own engineers, while HCL's work was limited to more routine, less critical tasks related to the 737 MAX.
Heh, it's not because of outsourcing or coding errors. It's because Boeing decided to bring in McDonnell Douglas execs into their own executive structure when they merged in 1997. Heck the CEO of McD became the CEO of Boeing. Every company including Airbus, Dassault, Bombardier and Embraer outsource coding. But they review and test the code before putting it in a plane. Before the merger Boeing was an engineers company and the ones who made the planes called the shots. Even the CFO didn't care about impressing bankers. After the merger, Boeing got a chairman who didn't have an aviation background. The passion changed from making planes to making money. The saying in wall street was that McD bought Boeing with Boeing's own money. To understand the basic problem today with the 737 max, you have to understand that the original 737 was designed in the mid 60s. He engines then we're much smaller in diameter so that it was easier to reach by ladder for maintenance. The engine fans then we're about 42 to 49 inches in diameter. The Leap1 engine of current 737 Max is about 69 inches. Obviously you couldn't compensate for that kind of difference in ground clearance by adjusting the height of the landing gear. They should have gone for a fresh plane design. Instead to cut costs, they pushed the engine way forward on the wing which raised a whole bunch of issues. They tried to take care of it in software, but there is only so much you can do in software. This is the root cause of all 737 max issues.
A thing not covered in the video is how when it comes to outsourcing, it is the big hr company that gains most from the relationship more than the client and the engineers combined. Engineers are paid on the bargain.
I agree there were rookie mistakes made with Boeing's MCAS. Boeing fired many of its veteran flight control engineers a year earlier, saying that their flight controls were already proven. The problems was MCAS was new. Boeing decided to go with one MCAS sensor. One way the sensor can fail is via a bird strike (at least one of the Boeing crashes, is tied to a bird strike on that sensor). An experienced engineer, in the U.S. Knowing full well they can be sued for any failure or omission during the design process, would have pointed out the necessity to install a redundant sensor. And, to test the system against a variety of mechanical failures. And do their best to detect those kinds of failure also in the software (which really would require some push back from the engineers). But, a 3rd party company, with inexperienced engineers, it might never occur to them, this is a bad design (one MCAS sensor). And, if an engineer did realize this, would they be stifled by inexperienced managers that they report this problem to? The retail cost of the 2nd sensor was only about 120k installed (that's what Boeing charged). The actual device was a tiny fraction of that cost. So there is blame to be had by Boeing sales and design at the U.S. level for not thinking about this. But then, if they just fired all of their experienced U.S. engineers, the U.S. sales and design team might have only the bean counters left, who are trying to make sales, anyway possible. Boeing sacked their U.S. engineers, they outsourced the job to a 3rd party, that then went with an inexperienced remote company. That remote company will never pay the real cost for this disaster (which is billions), no one in the remote team can be jailed for gross negligence, which this was. In short, they outsourced because "No Accountability" is way cheaper than "Responsibility". And this isn't a communication problem, because there wasn't anyone at Boeing HQ failing to communicate the requirements. It was a failure of Boeing management to recognize the necessity to keep experienced flight control engineers on payroll for a few more years. Also, there is responsibility at Rockwell and the 3rd party company responsible for developing the software. But, you'll never get them to admit (except in court, which will never happen with the Indian engineers). That they either should have realized the implications of having just one sensor and/or software too inferior to sense it was a mechanical anomaly (which is possible to do, although difficult, and requires testing and more sophisticated software). A few thousand dollars saved, and priceless lives lost. And Boeing deserves it's un Air Worthy reputation, until they fix the organizational problem from the top down.
As a software engineer who works in the aerospace/aviation industry, I worked on a project that included computer cockpit modules for Boeings aircraft. Another reason these companies outsource the software is because it is typically much quicker to get a product turn around. In the US, mission critical software goes through an insane amount of testing and it is very expensive. So not only is the outsourced labor is cheaper, but they can also cut costs on some of the testing for its software. I dont work at Boeing myself so this is a bit of speculation but just a thought.
Great presentation. I have seen companies outsource developer jobs and really the saying “you get what you pay for”. Many executives seem to look at only at the costs and truly believe that more robust developers who understand the business and have good communication skills are not required. I am sure if these executives were hit with direct consequences (jail time or personal financial penalties) they would make better decisions.
My dad was an Intel casualty. Monday was his last day after 25 years. They're floundering. The only thing holding them together is the US government because it's the only fab manufacturer left in the US. He LITERALLY told me to get an AMD CPU for my gaming rig build 😭 These corporations don't think long term. A lot of Indian developers on H-1b visas are leaving to go back home. They've truly pooped the bed on this issue. For myself, I am a hobbyist but would love to work as a dev full time eventually. I'm planning to go to Europe...and I'm from California. I grew UP in Silicon Valley more or less and I'm leaving. 😂 It's already biting them in the ass.
I work on ed tech software and i sometimes wake at night worrying about bugs or some other thing I might have overlooked. If I worked on planes, I would just live in a constant state of worry and testing
Genuinely amazed it's allowed. If you look at the documentation and verification systems required to maintain planes, it's amazing the same isn't required for Fly-by-wire aircraft. Maybe it's a historical thing ; when the control columns is connected to the tail surfaces directly or by hydraulic links if the navigation system crashes it's a PITA but you can still fly and land the thing. This doesn't apply with A320 type aircraft.
When big trees fall, there is more light for saplings to grow. And big trees fall for a multitude of reasons acting in concert. Boeing looks a bit creaky at the moment and the winds of change are blowing hard.
Problem, Boeing is "too big to fail." It's more like this mutant tree grew to giant size and gobbled up or killed most of the forest. Yet instead of splitting it up, we're actually allowing it to continue to subsume more trees. Alternate example, we built a neighborhood around the monster trees. The only space for saplings to take root is the empty space left by houses the tree destroys when it falls down. Turns out gutting anti-trust, and allowing monopolies to flourish for 30+ years has consequences. Who would have guessed?
That Loyalty bit in tech hit home Dee.. So much for quantity over quality... This doesn't even just apply to tech but we see it the hospitality business.. Fast food chains cutting cost changing the quality of their foods (cough) Chick-fila (cough). "The shitification of everything" -According to Nicole
I think around 12:15 you talked about billable hours. The time and materials projects are long gone, at least here in the US. For years now the projects are "fixed fee". The consulting firms underbid to get the project, then they get cheap labor mostly from India and work those programmers to exhaustion, 60 to 80 hour weeks, for months, with no weekends or time off. The clients are also at fault by trying to increase the scope, for free of course... I've been in the software consulting industry since 1990 and I've witnessed this trend first hand.
nice video.. This has been issue for a long time (at least 20 years). I think that the biggest issue in these companies that make big products is that even tho they have millions of lines of code in their system they still think that they are automation company or something like that and they do not understand the value good software. It looks nice on Quarterly report to have lower costs, but nobody knows how much more that costs in long term. More experienced sw developers change role and get more money when they manage these outsourced developers and do less coding. So these things just pile up.
Aircraft systems have some pretty strict restrictions on what code semantics can be used. I have more than forty years of experience in C/C++ and I recognize that it would take me some time to learn to code well within those restrictions. I would never consider giving this sort of projects to a junior engineer. But I don't want to kill anyone. But for bean counters that do a cost benefit analysis of plane crashes versus profits, with the option of killing people as an acceptable loss , it might make financial sense.
Outsourcing is not cheap they still end up paying over 250k per engineer. The money saving comes from once the project is ended they can’t just fire a full time employee at Boeing, but they can easily terminate the outsourcing team contracts.
Once I was talking with guy, which said he was working on a first MCAS. He said they did "everything as it should be" and 100% as demanded from Boeing. Crazy words if You ask me.
It's called a tombstone reaction. When people die or the project eats dirt, something starts happening. But not until. Been hearing this topic over & over for many years. Your particular rant is a good one though. Cheers.
I've worked with outsourcing devs, from certain country well known for outsourcing. Their approach is absolutely horrendous. I remember back in the days they even refuse to use git because they can copy paste the folder and add the date after the folder name. They refused to unescape the json string before sending them to the API. Things like that
Its not actually a programmer problem its a management problem where an MBA with no knowledge in the domain or the tech stack ends up messing everything up
Hello, the focus is not cutting costs but increasing profits in order to pay the top shareholders a higher dividend which includes yes cutting costs but not only, could also include inferior materials and cheaper manufacturing with a diminished quality control....
I worked on a large government project in the '80s which outsourced a large part of the development to an overseas supplier. They supplied dozens of fresh graduates with zero experience and billed by the line. We had the most extensively documented code I've ever seen as comments were counted as lines. We analysed the code and worked out there was something like 3.4 comment lines per code line in the source - and this was Cobol (which, for you youngsters, is pretty much like writing in english to start with). The quality of the code was OK, not brilliant, but OK. The biggest problem was the 'consultants' would never admit they had made any mistakes. It took days to get even small errors fixed as they would just say, "t's working as spec'd" even if it clearly wasn't. The culprits? A large American accounting firm with a computing consultancy division.
I have experience in similar safety oriented industry and there are some glaring points to consider: 1) There are some stringent coding standards (coding standards) which should be maintained. Doing so ensures easy debugging and less possibility of errors. 2) The Verification and Validation team should be able easily pick up flaws after thorough testing using simulators and checking project data/parameters. Seems like coding standards were ignored and poor validation was done. This means that the problem is systemic and completely overlooks/skips all systems of checks and balances.
Boeing didn't even let pilots know that MCAS was a part of the plane's avionics system. They were trying to avoid having to retrain aircrews by preserving type commonality between 737 models.
Basic question. Was the code developed in India tested thoroughly by the MCAS group in Boeing here in USA? Did they find out if the code did exactly the requirements specified?
I have worked in developement process of safety critical software components and I too noticed that management is blindly relying on the arbitrary safety standards and documentation which are all completely pointless when carried out by bad programmers
I worked on an airline reservation system, if we messed up you might miss your flight. If an aircraft programmer messes up you can loose aircraft. I know HCL, they’re the last people that should be programming Aircraft. My role got outsourced, it cost the company over $300k to build a replacement system because they couldn’t find people to do my job.
You hit the nail on the head, you get what you pay for... even engineering companies which services top 500 customers have used "low cost" centers and have to redo the work many times. The people employed by these low-cost centers are capable but in engineering school is only the start to do the work requires mentoring and educating in industry specific practices. Which are not taught by universitas, and master or PhD digress only teach more theory not industry specific practices required to do the work. An example is in the US is in hazardous electrical areas, OSHA requires the components to be listed by a testing lab acceptable to the jurisdiction having authority. ATEX components are typically tested by the manufacture and not an independent testing lab making them unusable in hazardous areas in the US.... More and more Sates in the US by law required engineering work has to be done under the supervision, reviewed by and signed for by a licensed engineer. In all states, a bachelor's degree with a minimum of four years of experience and in most cases passing an eight-hour exam in the area of practice (note there are some exceptions to the minimum requires which varies by states). If you have a master or a PhD, it only reduces the years of experience to three years. Why? because schooling only prepares you for practicing, it takes mentoring and experience to do the work.
Maybe they should outsource the Director, VP, and CEO tasks.
Many of which could be replaced very easily by a room full of chatbots.
Possible, eventually we'll have 1 person billion dollar companies
Basically they did, back to the Board of Directors, they make the choices and the CEO is a puppet winging his little head in a positive manner, it's not surprise they all seem stupid while their CV says another story.
How do you outsource saying word salads in long meetings 😂
How dare you? Your curelty it's beyond human! [Sarcasm starts] How come are they gonna pay teir 5th mega yate and the 10th mansion with a peasant salary?[Sarcasm ends]
I worked for a $2 billion company in the US. It was outsourcing software development to an Indian company. I designed a new command for managing a hardware device. The essence of my design was to gather information about configuration from a database. It was generic - it wasn't dependent on the hardware model. The team in India wanted to hard-code for each hardware model. More money to make if a new model were introduced. My design was eventually used, after significant pushback. The software was still in use 20 years later.
I should point out that it isn't just outsourcing to other countries. Outsourcing to big domestic consulting firms regularly results in catastrophes.
I will give consulting firms one, small, bone. They're often brought in by companies which have some horribly outdated process or another and asked to deal with a problem for the least amount of money. When often the problem is no one wants to actually fix the real process problem. As an internal engineer, you're used to the problems and know what the real requirements are. Meanwhile, the consulting firm is given a set of crappy requirements that are very specific and are not forward thinking at all.
Plus, sub-contractors are used to working with insane clients. For example, on one job the customer demanded we use their ticketing system. To create a ticket for a feature or defect, I had to fill out a PDF and E-mail it to someone. That person would then click the "new" button I wasn't allowed to and manually assign the issue a number. The ticketing system assigned a number and prominently displayed it, but they did not like it, so used their own number in all the paperwork.
@@arthurmoore9488 I've experienced all of that, and what the OP said also.
I'd forgotten the "our ticket ids don't map to your ticket ids" experience.
The Indians are good people but they have a big problem with business ethics. In my case, it happened the same thing
So based on the difference between what thlse HCL employee resumes claimed vs reality the 737 Max flgiht crashes and deaths can be chalked up to lndia's penchant for being cheats and Iiars. Got it.
Also stupid white peopIe being so trusting and thinking that aII cuItures are as good as their own...because they're aII equal (i.e. the same).
They don't aII have Christendom.
They don't aII have the enIightenment.
They don't aII have the renaissance.
They are not your equaIs.
As a remote Dev that lives in Africa and works for an American company I do agree actually, the big outsourcing firms have a problem, I'm an independent person I just apply independently for remote jobes. I do try to make sure I do thinks right, follow best practices, write things that are maintainable and scalable. But then again I would never work for $9 an hour so it is what it is
Boeing used to be run by engineers. Now, it's run by financial types whose prioritize profits over everything else, including safety. Back in the 90's I wrote software for air-traffic control systems. Everything was designed, reviewed, and thoroughly tested, and had redundancy built into they system
Oh please stop this accountant bull crap. Dennis Muilenberg (former Boeing CEO, just in case you forgot) is an engineer, and look where Boeing went under his watch. It’s simply greedy a-holes that put greed above anything, not just because someone is a “beancounter” or not.
That is very true. Boeing CEO has been a marketing guy, economist and stuff like that. The times when CEO was a real engineer, are long gone. I made another comment here to tell how MCAS works fine, but the economy-people wanted to get 19000 USD for a software flag that could say "AoA disagree"-alert. Lets see if that comment will be visible, and how it will be at least shadow banned.
And yes, there lies the problems of the Boeing Commercial Aircraft Company. The CEO is not an engineer, in a company that is 90% engineering and creating NEW stuff and items.
Now Boeing has changed their CEO many times, and still, none of the new ones are engineers. As that is the root cause, including that the current new management board is bunch of children straight out from Harvard (etc), they have zero understanding of anything in airplanes and aviation. The good board members all resigned and stayed in Seattle area, and these new clowns are the 100% culprits of those really bad design principles.
For economist, marketing etc person, it must be hard to understand that engineers need to be there, in a huge engineering company, but again, they still haven't tried that. And that is why Boeing will not recover, until they hire from their knowledgeable engineering ranks, and stop murdering whistle blowers.
Bean counters, lawyers, and bad leadership went to the forefront when they brought in McDonnel Diuglas.
Since they acquired the DC Douglas folk, they also inherited the toxic management culture which contributed to their current woes. The solution is really simple for Boeing - if you are an engineering / aeronautics company, make sure to put engineers at the helm, not the ones with cushy MBA pencil pushers with their greedy business models.
Companies are turning into financial business instead of serving the purpose of their particular existence. Product quality, customer satisfaction, and worker happiness are all going out the window for "maximize profit".
This is probably why so many large IT projects go wrong. It is not that programmers in 3rd world programmers are not smart, it is that they are often managed by non-technical “managers” who are thousands of miles away while also not scoping out the project correctly. Failure is inevitable.
Aaaand, the companies western companies hire are full of unqualified people. The qualified ones are getting PAID.
They are smart but their priorities aren’t the same. They aren’t concerned about safety. If it breaks, you have to deal with it. They only care about maximising the amount of work they get from you. They don’t care about readability or reusability etc.
@@OneIdeaTooManyyep. This is an issue with outsourcing overall. It can be cheaper but the outsourcing company has different priorities. I see this all the time.
@@OneIdeaTooManyI'm an engineer in a third country and it is not like us, the sw engineers, are not concerned about safety. It is more like we are understaffed for the speed they want us to deliver and management prefers if we deliver something 90% tested if it takes two weeks less. Just as Boeing management is trying to bank as much money as they can, our management is doing the same.
Fuck capitalism and shareholders that is
Yeah they really are just fiends for work. Understandable why as their country of residence would pay them in peanuts compared to salary from a first world country, but the work this mentality creates harms the entire project.
Indeed the reason of outsourcing now is very simple. CEO and C-level cut cost, get bonuses and leave with gold parachute. Company can die after after that, but it is no interesting for C-level anymore, they don't try to build something, just max money for minimum time. And even after failure in one company, they will be willingly hired in the next one.
Couldn't agree more with this. Also telling that there have been multiple studies that have failed to find a statistically significant relationship between CEOs who move between companies and the performance of those companies, yet the companies feel they have to pay ridiculous money to get the "Best" CEO.
The complete C suite explained in a few words! LOL
It is the fault of the board of directors, who are elected to represent the interests of investors and negotiate the CEO's contract.
Yup. It's unbelievable how Unity so enthusiastically hired the notorious former CEO of EA.
True, prison time would resolve that issue. But this is pointless, once you reach oligarch level wealth, you're untouchable
With over 30 yrs in IT, not having a thorough understanding of a business and its processes, was the biggest impediment to successful outsourcing.
Agree. Some of the best work I've done was in collaboration with an on-site Subject Matter Expert (SME). If you look at the companies which are successful at using foreign labor, you'll see close working collaboration and at least one person who knows both the company and the foreign culture.
That's the actual issue here, not the cost which might be a considerable wage in some economies.
I found that it cost more time to write a spec for outsourcing than to implement in house.
It amazes me when engineers know more about business than the business majors they report to. I see this happen quite frequently.
@@username7763 Most managers nowadays simply have no domain knowledge. Forget about the engineer that did the jobs well and decided to take that lead in his/her company. Does not happen anymore.
just wait until they start "outsourcing" to ChatGPT. That cuts costs even more, right?
According to my AI software resourcing app, ChatGPT would be a good choice as a software developer.
@@GurungyNoHamuster LOL because of course it would.
...and is likely to rise once OpenAI's VCs start asking for money.
There's no such thing as coding without AI anymore. Hasn't been for a while.
@@jtjames79 ya you can't keep up unless your using it too. also its using you to learn to code better it won't be long till it outshines us imo.
My father often said, when you buy cheap, you buy twice. In a way, this also holds for software engineers.
Your father is/was a smart man. It holds true with everything!
Hi Dee, if I remember correctly about the MCAS system it was the lack of redundancy in the sensor that reports the aircraft angle of attack was the root cause and rests squarely with Boeing itself. The sensor would erroneously report a stall condition and then the MCAS system would nose the plane down. There was a stacking of issues starting with having a critical a flying metric having having a single point of failure whereas several sensors are needed. There were also issues of FAA collusion in ignoring the Boeing safety failures as well as Boeing hiding the MCAS system so that no new pilot training (expensive) would be required. The insane level of disregard for passenger safety cannot be explained by cost cutting measures designed to save money. It is absolutely puzzling to me.
The outsourcing of development was a huge problem, you're right, but I think it deserves more context. This was all part of the arbitrary top-down approach to driving business, instead of understanding the problem and the product. As a response to a popular aircraft development from Airbus, Boeing decided to fast track development of a new plane by basing it off of an existing one. To make the plane more powerful, they had to install larger engines, which necessitated mounting them differently. Mounting them differently changes the physics of how the plane flies, and they tried to compensate for that inherently less stable design with software. In addition, where there should have been two sensors for the MCAS system, they saved cost by only including one and leaned on software. To shave off additional time and cost, they did the least amount of testing possible and claimed that the plane was essentially similar, so the only additional training required for pilots of the existing 737 was a 1 hour iPad training. The software was just one part of the confluence of shaving off every penny even when it wasn't appropriate.
It really seems like companies today are no longer interested in producing a good product but instead the minimum viable product.
Well, the product "Boeing 737 Max" is not even viable.
Slight correction, 737 max engines are not "more powerful" they're more efficient because they have larger fans and larger air bypass volume. Efficiency of fuel consumption is king in airline industry. Agreed on other points
@@thecompanioncube4211 That's a fantastic correction. Thanks a lot. It's cool to know that making them larger can have the ability to make them have similar power output at a lower energy input.
Another thing if they were even told of the system pilots were not explained how to override it if it was wrong. So even if the pilots knew what was wrong it could be possible they didint know how to stop it.
I will suggest that MCAS software worked exactly as designed, but the MCAS design was flawed in at least two ways: first because it depended on a single point of failure by using only one angle of attack sensor (though the 737 has redundant AoA sensors available), and second because it didn’t stop trying to trim the aircraft when the pilots were trying to override it.
The third fatal mistake was not even informing many pilots that the system existed, so they didn’t know there were switches to stop it located in the center console.
None of which is really a software development issue in my opinion. Respectfully submitted.
MCAS evolved overtime. It started out as a minor feature on the airplane that didn’t matter. But over time it became flight critical. It was never handled as a flight critical device so there was no failure. Analysis done. I don’t know if the systems engineers knew that this had happened. Or didn’t want to bring it up because it would push the schedule back.
The maneuvering characteristics augmentation system was implemented by Boeing for the 737 MAX because the aircraft was being built with different engines that, due to airport gate design, must be fitted low and therefore can cause differences in handling. To avoid the costs of retraining pilots and making them get new type ratings, the MCAS was designed to make the MAX handle like previous 737 models. Putting costs ahead of safety was unwise in my opinion. As long as corporate executives are more concerned about the price of their stock options, perverse incentives and shortsighted decisions will continue to plague us. Respectfully submitted.
@@ADHJkvsNgsMBbTQe
So you’re saying when they added MCAS they knew it was unsafe?
It wasn’t the implementation was unsafe. So the trade-off you claim was never made. If implemented properly, it could’ve achieved the same type rating. But since it wasn’t implemented properly, it was decided to add the training anyway.
The issue was trying to get the aircraft, certified cheaply and quickly.
I remember when Siemens Medical laid off people with decades of experience to contract with an Indian company for dev ops.
The new guys took 5 times as long to get things done and often had to reach out to others to figure out how to do their job tasks.
Brass did this on purpose to jack up company stock before selling to Cerner to get the price up for stock holders.
It's a shame that ethics, quality, and standards are all just things you can cut for capital.
I interviewed a lady that worked in a company that outsourced software development to an Indian company. She told me it took more time to explain what was to be done than do it herself. However, management thought otherwise. I personally dealt with three distinct cases where the outsourced Indian/Pakistani developers did not deliver: a restaurant application, a car spare part application and a ERP system. Spaghetti code full of bugs. Do I imply Indians are not capable? Certainly not. It's just that those that are good developers are not cheaper.
You're totally right! Gave a simple task to two developers, they spent more than 100 hours, plus my time on daily meetings. Their process takes hours to produce results after any changes.
I wrote the same process for 3 hours, mine takes 35 seconds to produce results.
just say it, they are not good.
Indians themselves are not incapable. They have created really good companies and services from India. But when it comes to IT outsourcing, my god it's a sh*tshow. If the system itself rewards incompetence and boot licking, you shouldn't expect anything good from those companies. You don't manage 200k employee company just like that
a friend of mine is working as a freelance consultant, most of his gigs are about fixing spaghetti code that other people make, and they pay him big bucks, turns out its 95% indians, might be anectodal but it is what it is
No need to be politically correct ... Even those top management comprised of Indians are mediocre, at best
I am a retired software engineer who worked in software for my entire adult working life. I was appalled when I heard about the design defects of the MCAS software in the 737 MAX, and how the people at the top of both Boeing and the FAA seemed oblivious to the problem. The Crux of the problem was that the software would ignore incompatible readings from the AoA sensors, and make automatic trim adjustments using only one (possibly defective) sensor. That is an egregious design error. The only reasonable thing to do in response to disagreement of the AoA sensors is to alert the pilots and refrain from. aking any automatic flight control adjjstments depending on those sensors.
Outsourcing software explains it. You have the programming being done by people who are not specialists in Boeing flight control software, and you may have poor communication between the Boeing engineers who (presumably) know how things should work, and the people doing the programming, who may know very little about airplanes.
They used to teach ethics in college and the idea was not to make questionable decisions that impacted safety and other things like reliability. How in the hell did we forget how to make software and/or manage companies. Is higher-ed part of the problem?
@@billymania11we didn't forget anything, greedy people in top position fuck it all up for everyone else and get rewarded for it
Absolutely correct. I think Aerospace engineers with flight control systems should have developed the software.
One of the other issues in outsourcing to cheap staff is staff turnover. As soon as the programmers can they will change jobs to get more money. So at 9 dollars an hour it is very likely that you are dealing with trainees. As we know a good software engineer builds in things which are not specified. No spec is complete. We test for conditions that are not specced or raise them to get a decision. Trying to do that by phone, through time zone differences is challenging so it’s just easier to do what is in the spec and consider it done. Bad decision Boeing. Google you will not get the innovative thinking that has been the hallmark of your company. Not because there are not creative people in these low cost countries- they just won’t be working for you at 9 dollars per hour. They are doing their own start ups.
@@brightonbabe2139 Just one thing - if you are developing avionic software it's normally done in accordance with DO-178 (or the equivalent European spec ED-12) - and that specifically prohibits implementing any behaviour in the code other than that which is specified. For example, in the case of MCAS if someone had decided to implement a consistency check between the AOA sensors then that code would have failed review because you couldn't point to the part of the spec that requires it.
Once you've written specs so detailed that a $9/hr developer can understand, it doesn't take much more time or effort to just code it yourself. I just don't understand this approach.
Definitely, this is what a sane person sees, unfortunately not what the executives see 😢
@@arthurdefreitaseprecht2648 I heard from the management many times that you can teach anyone to write code well, but it has never been true in my experiences. First of all, *most* of low-cost developers do not care much about learning. They think they can just follow instructions without real understanding of stated goals. Moreover, even the minority who care about learning something new often are very slow learners...
Take chess as an example. Can you teach anyone to play chess? Sure, but can you teach anyone to play at the GM level, I am not so sure. Even it was possible, can you imagine how much hours of teaching would it require, especially if one is incapable to learn anything on their own?
So based on the difference between what thlse HCL employee resumes claimed vs reality the 737 Max flgiht crashes and deaths can be chalked up to lndia's penchant for being cheats and Iiars. Got it.
Also stupid white peopIe being so trusting and thinking that aII cuItures are as good as their own...because they're aII equal (i.e. the same).
They don't aII have Christendom.
They don't aII have the enIightenment.
They don't aII have the renaissance.
They are not your equaIs.
A complicated, or large, application, especially if it is "real time", is itself too difficult to code by people whose focus is algorithms.
The algorithms require algorithm experts, and the code requires code experts. So, systems engineers write the algorithms, and the test procedures, and the software people write the code.
It's management-think.
The MCAS software did not malfunction. The software did exactly what the designers intended it to do, but the designers had forgotten to include the possibility of bad data getting into the system. From the fix to the software, limiting the amount of nose down trim that could be applied by the MCAS to strictly less than what the pilot could counteract by pulling on the control column, we can determine that the MCAS would keep applying nose down trim, if the angle of attack did not decrease. This meant that the MCAS was unknowingly programmed to crash the aircraft, if the angle of attack sensor provided bad data that was persistently high. This is an example of code being a correct implementation of a flawed design.
Yea, totally! The code is as good as the specs and integration contract by Boeing.
It's ultimately comes right back at Boeing management.
Well said. This is why the business context understanding is so important. It is hard to add related use cases/acceptance criteria to things because it makes the feature too big for one sprint.
I hope the person who made this video reads your comment and understands it.
The system also lacked redundancy, as the AoA data was provided from a single pitot tube in most cases - I think *some* 737s are specced with with redundancy of AoA sensors, but it was/is a costed option.
I don't remember whether MCAS ever took results from the redundant sensor (if it was available) anyway?
@@rao521 I hope you understand that she made exactly that point.
Cheapskate accountants chopping software engineers is nothing new. Years ago, I worked for a risk management company in the UK and was sent over to Mumbai for a few months to beef up and get the team there up to speed. Whilst I was away, a top-notch fast and accurate software engineer (that I had recruited), worth 4 average engineers was ditched by the accountants because "he was expensive...". Output from the UK dropped noticeably and I quit not so long after, tired with trying to get cheap, inexperienced guys to work accurately. The stories I heard from other departments where the users of the software internally (before selling or upgrading to clients) would berate the engineers because of stupid bugs where the engineers would blithely reply, "Oh, yeah, we know about that, you do x as a work around" was driving us crazy, and I had enough of the fire-fighting and the constant battles with "cheap" engineers to bring them up to spec. The company eventually folded.
Yep. I too have been "downsized" as I was the most expensive.
CEO seems to have forgotten that not six weeks before I pulled an all nighter (unpaid) to get enough systems working that we could participate (sucessfully) in an industry wide demo of our tech. Without that, our company would have been a laughingstock.
But I was expensive.
So based on the difference between what thlse HCL employee resumes claimed vs reality the 737 Max flgiht crashes and deaths can be chalked up to lndia's penchant for being cheats and Iiars. Got it.
Also stupid white peopIe being so trusting and thinking that aII cuItures are as good as their own...because they're aII equal (i.e. the same).
They don't aII have Christendom.
They don't aII have the enIightenment.
They don't aII have the renaissance.
They are not your equaIs.
@@erictheis6093 I wonder who else is expensive and worthless in the company... Hmm..
Same thing is happening to Enbridge, a pipeline operations company. I know that one day there will be another major oil spill because of the sheer incompetency coming from the people that safety-critical roles have been outsourced to, Tata Consulting Services.
I got so frustrated and tired of teaching and coaching them that I left a month ago, after seeing how many great engineers and colleagues have been let go.
Great video. Outsourcing isn't just a recent trend. It's been going on for decades in tech. I first noticed it in the 90's. It was and is getting more prevalent. After that I never worked for a company that didn't depend on cheap contractors. It was always a mess trying to get good or even correct code - cheap labor is cheap labor. Combine that with time lag, language issues and lack of product knowledge or ownership and you get what you get. :)
I have worked with some great software engineers in India ... but the vast majority of them end up in the US. The best engineers don't stay with contracting companies. This is also a symptom of an even greater issue that is not specific to India or contracting companies. For years, we've pushed people to enter this field with little regard to aptitude. I taught myself to write my first program on my TRS-80 in 1982 using the manual that came with the computer and books from the public library. Started with BASIC (of course), moved on to 6809 assembly and Lattic C on a PC XT in high school. I'm now a senior director of software architecture ... and still write software daily. Why? It's just something (frankly the only thing) I seem to have a natural aptitude for as well as a genuine interest. The same could be said for most any technical field. For some reason, we've gone so far over the top in encouraging everyone to "learn to code". CODING is one thing ... being a software ENGINEER is quite another. I can (and have) installed my own crown molding - but I am certainly NOT a finish carpenter.
Oh man! Back in the day, the software product was expected to be as perfect as humanly possible. Of course the software environment was quite a bit simpler. Meaning you didn't have the WEB with all its kooky frameworks and languages and design patterns. It was classic waterfall without the need for things like Scrum. Times have changed and not for the better.
No such thing as “natural aptitude”…
You got exposure at a young age due to passion and curiosity. Same for myself. The earlier you start the better.
When I heard that Boeing put a single monitor on the outside of the aircraft that if it malfunctioned, would drop the nose of the craft and that pilots would tire within fifteen seconds while trying to regain control, unless they properly reset the system after reading an update to a 500 page manual, I thought... who the hell doesn't have multiple points of redundancy? Who the hell doesn't train pilots on new features? Oh, they fired all of the software engineers whose job it is to think. I could make that determination in my company back when I was getting paid like $70k a year for a $50 million dollar company as a junior software engineer. Boeing, a $50 BILLION dollar company really has NO software engineers that are to that level of competency?
They 737Max was design to compete with the Airbus A320 Neo which uses the new Leap engines which are larger but uses less fuel.
Fitting this to the 737 design made the aircraft have a different centre of gravity so MCAS was to make it behave like an old 737.
So no retraining or recertification for the pilots were needed since they claimed it worked like the old one
@@tibbydudeza Yep, that was one of the big things. They deliberately didn't train the pilots! Since doing so would have made the 737 Max cost more to operate.
This is not the true. They did safety evaluation of the mcas with some requirements and they end up that risk is minimal so mcas may not have redundancy. And that was all fine. Then someone changed requirements to be more aggressive but there was no second evaluation that could end up with system redesign. And this was not a fault of 10$ / hour devs.
Exactly. If they'd had more experienced internal engineers, they'd have realised the flaws and raised hell about them.
This is also what management wanted though. If you read between the lines, you can tell that "engineer says no" was seen as A PROBLEM at Boeing, not a strength. All management could see was "this guy who makes WAY less than me is causing me problems", and their solution was "hire a cheap contractor who WILL DO WHAT I DAMN WELL SAY".
The basic problem is that management were making engineering decisions that they weren't qualified to make. They thought they had every right to make those decisions because... aren't they in charge? They didn't realise that reality gets in the way.
IMO, this was all 100% a management problem. It's 100% about managers being UNQUALIFIED to manage the things they're managing. There should be no such thing as a manager, at every level, right up to the CEO, who doesn't have at least a decade of actual engineering experience (NOT in management) before being promoted to manage even one person. The "non-technical manager" is the modern scourge of all technical companies, and the source of an overwhelming number of problems.
Dude, working for defense and common fortune 500 companies, a software engineer who tries to discuss tasks with managers or innovate is now called "insubordinate" and "not fit for development teams". All management wants now is go go go, must make the sprint points and deadlines!!! No thinking, just code!
I worked with the HCL guys in India while in the finance sector, back in the early-ish days of HCL. They were very sharp guys all, even the cheap ones. The issues that did come up, stemned from poor communication from our end as both sides had major technical, social, cultural blind spots towards another, many of them unmentionable. Looking back it was shocking the result was as good as it was despite those issues.
I want to clarify something extremely important which is misrepresented in this video. MCAS was NOT a software error. The software performed its function exactly as designed. This issue was with the system design: the designers deliberately designed it the way they did to skirt regulations, and failed to account for what real world implications that might have. I think it's extremely unfortunate that this nuance is lost, and it's simply reported as a "bug", since categorising this as a software issue downplays the disaster to a simple mistake, and not the deliberate and wilful choice to prioritise profit at all costs over safety that it was.
Unfortunately a lot of people are now using the troubles at Boeing to push a union focused agenda of "this would never have happened at the 'Old Boeing' when everything was done in house", and while this is probably true in many cases, there are also plenty of situations (like the article you're reading here), which just seem to be a pretty transparent and blatant attempt to mislead people into thinking that Washington Airplane engineers are somehow some superhuman mutants who never made mistakes of any sort, period, and that land of milk and honey will be returned if only they are all rehired again.
I agree. The main problem was using one sensor for MCAS and no backup or second sensor or a routine to check for sensor errors and what to do at that point. It was a system design failure and not just some bugs in the code. Thanks for pointing this out .
@@nine7295 That is my understanding as well.
A union focused agenda? Is this the CEO? What's your problem with unions bro?
@@nine7295Think of how many engineers at Boeing must have signed off on the one AOA sensor. I wonder if they were pressured to sign off or just didn’t bother thinking about it.
@@mshepard2264Once again, Upton Sinclair: ‘It is difficult for a man to understand something when his salary depends on not understanding it”. One of the most insidious things is that Boeing charged extra for 2 AOA sensors and the software that alerted if they were not in agreement.
One more trend in Indian consulting firms is that they would have one experienced dev and rest junior devs. Where you would end up with just garbage code due to tight deadlines, experienced devs burning out and quitting, also junior devs don’t get mentored and grow.
Western consulting companies have the same strategy. Quite sad. Experienced devs fight tooth and nail to escape these terrible places. But slowly and surely the governments make sure this is the only future ahead.
The way it works is that the outsourcing firm submits the resumes of their "star" developers and when they start the project, they pull those top-notch developers and leave you with those 9 dollar an hour characters. I've seen that happen so many times. It's completely ridiculous.
From the little I learned about the system, the main issues had nothing whatsoever to do with the code. They were design problems that were based on decisions almost certainly NOT made by programmers or 'overseas' personnel. The two main issues I am aware of were that they deliberately tried to hide the system from oversight and made some decisions for that reason and there were insufficient safeguards if a sensor was giving incorrect readings. The aircraft has two sensors and even I, as someone who has not worked on aircraft software (although I have built a customer reservation system for a chartered aircraft), I know that if you have two sensors then you should read both sensors and if there is a discrepancy you should do something about it whether that be figuring out which is correct and relying on that or informing the pilots that there is a problem etc. But instead they simply use only one sensor and ignore the other making the whole redundancy aspect totally useless. That was NOT a programmers decision. That was a management decision.
Issue is: without the experience and maturity you DO get from well paid, market rate software developers who DO see the total design even though they are on code, you wont have people who state a simple and obvious intelligent point about redundancy, design and integration of the code theyvre written. MOST good software devs will absolutely just tell you when the glaring implementation fault is there. And yeah just make sure you get decent people working on an integrated solution in this scenario, its peoples lives. Boeing should be ashamed.
@@johns5558 In my own job I often point out glaring design flaws or things that should be done, but the reality is that in corporates that doesn't always result in action. The fault lies squarely with management in this case. If I was the programmer, I would have definitely suggested improvements to the system, but if management said no, just stick to the design or we get someone else then I would have stuck to the design and got paid. Outsourcing does have its problems as the employees feel less attached to the company as a whole and are less likely to 'do the right thing'. But lets be clear that Boeing outsources almost everything (and has suffered as a result) and that has nothing to do with whether or not the outsourced company was in India, Italy (recent case) or the US. I once worked for Vodacom (a telco in South Africa) and two thirds of the people at work were 'contractors' including myself. It worked, but it had its problems because there was no job security. Outsourcing has its pros and cons. A contractor is under higher pressure to impress than an employee, but they are less invested in the long term health of the company.
I remember a recent case that was reported on where a bank fired a large number of employees because they had installed software that emulates moving the mouse and keyboard. They did this because management was monitoring them and evaluating them with software that monitors mouse and keyboard activity. The employees were fired when the right move would have been to fire the management. Management is hard and evaluating employee performance is hard and when you get it wrong you get the wrong results. Boeing values cost savings over safety and that is what they get.
100% this. People within Boeing raised the issue of only one sensor and were shut down. It was a deliberate action.
This is a trend that started decades ago and made famous by Jack Welch as the CEO of GE. When an organization gets hyper-focused on the numbers it starts to get hollowed out with cost-efficiencies. Give it long enough and like GE, it loses its intrinsic value and becomes a hollow shell with significantly reduced value.
What is the cost of human life. Apparently $9/hr if you ask Boeing.
that is the equation
That's too much, the figure includes commission of the HCL as well, the end engineer gets only $2-3/hr.
Wrong. Insurance industry puts values on lives. Issue I have is every family got a least a million a person for wrongful death but if you try to insure your spouse for that they want to put you in jail first and then ask all the questions after “why do you want that much money”? Shoot even 250 to 500K on a spouse for “lost wages” if they die isn’t cheap to buy. Totally a money grab by the dead families. Less than 1 million dollars US for a life. I’d say actuaries peg it at 250K
This sort of poor management is a problem at Boeing, and MANY other companies and organizations. The senior management no longer has any real world engineering, production, leadership, and basic business skills or experience. They listen to marketing/sales first, cost accountants next, and finally quarterly stock market returns. Senior managements are the winners of decades of internal power politics, in some companies a low grade war. This is all a long way from building great airplanes.
Sorry, but it wasn't a software problem. It was a system engineering problem.
A sensor with SIL4 (Safety Integrity Level) responsibility (highest safety level) was electrically connected like a sensor without safety responsibility.
This was not a software error but a system error.
My understanding is that the aircraft had 2 AoA sensors, but they made the decision to only take input from ONE in the MCAS. It IS a system error but potentially one that could have been spotted and flagged had the devs had more domain knowledge.
Upvoting this for correction
@@ralphprice7365 I genuinely don't understand what point you are trying to make. Not saying you're wrong, just asking for clarification.
My understanding is that the problem was that it only used one sensor and so if that sensor developed a fault that caused it to produce bad data, without any error conditions, the system would behave badly.
The system error was not identifying the single point of failure and using both sensors to detect bad data.
I'm not sure what relevance handshaking with hardware has in this case. Did they discover bad handshaking? I missed that, if it was announced.
@@ralphprice7365 Boeing was trying to "hide" the fact the 737 max tend to pitch up when pilot throttle up ( due to the new layout of the bigger engine ) . Boeing is using MCAS as a "mask" of the real problem of a badly designed engine layout. The worst part is that a running MCAS does not give pilot the authority to manually correct pitch. The pilot has to turn off MCAS .
@@d_lollol524 He had to switch off a system that he didn't even know existed.
I agree with your assertion that chasing low cost/lower skill engineering talent can often be more costly in the long run, but I believe it is incorrect to blame software engineers for the issues with the Max. The 737 is a design from the 1960's. The original airframe was not designed for the large diameter, high bypass turbofan engines used by the Max to improve fuel efficiency. To provide adequate ground clearance, these bigger engines had to be installed further forward and upward on the wing than the engines on prior 737 models. This moved the center of gravity of the airplane, which in turn changed its performance characteristics at high angles of attack. Boeing needed the airplane to fly just like previous models of 737s so their airline customers could avoid costly aircrew retraining. The MCAS system was added to help pilots recover from nose-high attitudes and make the airplane appear to fly just like earlier 737 models. Unfortunately, the MCAS system was only fed by a single angle of attack sensor. If that sensor malfunctioned, MCAS would aggressively push the nose down with more force than the pilots could overcome. MCAS was a band aid created to squeeze more life out of an outdated airframe. It enabled Boeing to offer a product to compete with the Airbus neo series without the massive expense of developing an entirely new airframe. The entire idea was flawed. The decision to control MCAS from a single, non-redundant sensor was crazy, and the failure to inform or provide training about it to pilots was inexcusable. These actions caused the Max crashes, not the low cost software engineers. The MCAS software performed as designed. It was just a horrible design.
From what I gather, there was nothing wrong with the software, the problem that a single sensor was used, which was Boeing's decision. I work in the oil & gas industry and all our safety systems use 3 sensors and 2 out of 3 voting on what signal to use. Additionally, measuring devices are able to indicate when they have a fault. Again, Boeing's choice not to do this. Boeing made the AOA disagree alert an optional feature and it looks like they never considered what would happen if there is a fault on the sensor, which by all accounts is quite regular.
I did some work for a Boeing division (off and on) over the years. When all the stuff going on with the Max happened and the MCAS was identified as the main culprit I hear from a lot of engineers that the real problem was as you suggested with the placement of the engines causing a "hack" to be implemented with the MCAS.
Boeing got caught with their pants down with the 737 vs. Airbus A320 Neo. They couldn't take the time to redesign the airframe and landing gear area to gain more clearance for the bigger fuel efficient engines. Basically they would have to go through flight testing and certification again, which would have costed them 5-8 years of work, to be able to compete with the Neo. It would have seriously crippled them in the market as the 737 was their most popular plane.
Yes, But, it was not the retraining but the very costly type specification which prevents changing major parts - like the distance from the wings to the ground of this type. A new type specification costs billions.
@@smmasongt Now you're just blaming the engineers 😆 Their concern is not cutting corners with bad design to compete financially with Airbus, that's on the executives. My guess is most of the engineers looking at the plans thought "WTF are you doing?!" In fact, many of them complained to management and were ignored or told it would be fine. I mean, technically it would work, it was just precarious.
Finally a sane comment , the rest are just taking out their frustration of H1B
Great Video Dee!!
Yes, Quality is the key here. We also don’t know who is coding under such cheap contracts. Many resumes I have seen, when this takes place, are somewhat hard to validate and many candidates cannot back it up. This is even evident back here in the states, because the model of these outsource firms is being stablished here, undercutting experience developers with very inexperience ones, yet they are listed in the contract as senior developers. I have seen this first hand, since I would have to support and teach many of these so called “senior” developers.
single-handedly one of the best videos on this subject. Liked and Subscribed.
I've been telling my tech compatriots for a long time now: You get what you pay for. When you pay $9/hr for a software engineers you get someone unfocused who is doing multiple contracts at the same time who will give you fast-food quality if you're lucky and toxic quality if you're not.
Engineers spend money to make money + Accountants cut costs to save money = zero.
There are other "hidden" costs to this outsourcing to the cheapest bidder. The people left behind to manage the outsourcing are demotivated because their job is reduced to a high impedance clerical job where nothing they can do will improve things. Either they resign themselves to it and productivity reduces further or they resign completely and the job goes to someone with less understanding of the actual requirements and zero interest in fixing things.
1:52 This may be before your time, but in 2002 Microsoft had an ad for SQL Server: "DOING MORE WITH LESS! DOING MORE WITH LESS!" etc etc
You are right on, yet it applies to local (high paid) employees, too: when I do external consulting (at high hourly rates), I usually solve tasks in a fraction of the time it takes the employed staff, making me a bargain. I have no interest in stretching time as I have enough other things to do and want to be creative and not bored. Work for me is part of life, not a separated balance. So outsourcing seems just a part in a more complex game that‘s about „competition“. Whenever there is honest competition and true evaluation of individual service vs. cost, things work out. Whenever general „standards“ and „procedures“ dilute individual performance, efficiency goes down. Outsourcing often feels like the helpless controllers’ answer to problems that started locally: when qualified local engineers become uninspired and start taking advantage of anonymity and job security, it ruins local productivity and then triggers unreflected outsourcing. Obviously, outsourcing is just trying to shift responsibility but that most often doesn’t work and won’t solve the original problem. The answer may just be to focus again more on *individual* performance, regardless of where the engineer is located, and pay for what you get.
Hey Dee, this is a very refreshing channel for me as a "long term > 30y" Computer Scientist (MSc.). Working in the automotive industry for the past 10+ years, I can confirm that outsourcing is a huge issue due to the cost pressure in the industry. But cutting costs will eventually cost lives!
You missed the entire reason these CEOs do this. It all comes down to how they all pay themselves with lots of stock to gain more control and if they do it well, a board seat. That one dollar salary steve jobs took with stock was the biggest grift of our lifetime. It is all about stock. By paying themselves in stock, they get all the voting rights and get to act as an owner. To get around the fact that they can't sell the stock without losing voting rights, they take loans against the stock instead. The loans are tax free and once rich execs figured that out, they all started taking large amounts of stock as compensation.
This means when they pay themselves in stock, they get both the voting rights and up to the entire value of the stock in cash tax free. They can loan against smaller amounts at a time so they benefit even more from their own stock price manipulation in the future. The voting rights are used to run the company based on stock price. The goal is to try to push the stock price up at all costs so they are not forced to sell stock to cover the interest payments or aren't forced to sell stock if it drops too far. The only outcome they want if there is a decline is a quick bankruptcy because the banks eat the losses and the CEO is not forced to pay any loan money back. This is why they also created the golden parachute. They know bankruptcy is almost a guarantee over time, so they plan for it. If they bankrupt the company, they get paid huge cash payments as a reward.
This is why the modern CEO will cut from anything and will outsource everything if they have to. Natural growth of the company value is pegged to inflation. That level of growth does not satisfy their scheme of paying themselves lots of stock bonuses. They negotiate bonuses based on the stock price going up, juice the stock, get paid lots of free stock, and become an owner with a board seat to ramp up the juicing even more. Look at how every large company has billions in debt and most CEOs are also on the board. These companies are faking profit increases and being loaded up with debt. When the ability to raise cash runs out, the company will either be sold to avoid the scrutiny of a bankruptcy judge or go into bankruptcy. Either way is fine, the golden parachute will be paid out in some fashion.
Building a legacy vs getting rich and leave?
Yes, I've also noticed that.
Was that the method the new MD leaders took over the existing Boeing leaders, after the merger with McDonnell Douglass? It was the MD leaders that changed the company’s focus to short term profit and caused all the damage.
I enjoyed this video and will watch for others from you. But I'm surprized you didn't also mention the space capsule "Boeing Starliner" with its failure to reach the Int'l Space Station in Dec 2019, and with the helium leakage problem which prevented the 2-astronaut crew from returning with the capsule last month. You could also have mentioned that Boeing's merger with McDonald Douglas in 1997 has been blamed by some for a change in corporate culture which shifted the emphasis away from engineering excellence to cost-cutting measures.
I'm from the other side of the planet, but all you say here is like listening to my own thoughts. One thing I would like to add, though: 10 years ago I used to work in global corporations with software development sites in the cheapest and largest countries. We had all the classic problems that come with hundreds of beginner coders working with little to no context and it was so painful. At some point I managed to mentor a couple of the most promising ones to a decent working proficiency in the tech and domain we were working on and I was so relieved: finally we could get some work done. But, alas, next week those people had switched jobs. And I don't blame them. They probably got 100% pay raise from some other place and had literally no reason whatsoever to remain with their exploitative and cheap current job. There's just no way you get anything but what you pay for in this world and imagining otherwise is just fooling yourself. Any executive, director or manager falling for this trap should be immediately fired for incompetence.
MCAS was a fairly robust system done a while ago for the military. Digital augmentation of airframe characteristics isn't a new thing in military aviation with fly-by-wire inherently unstable fighters, planes having asymmetric load-outs when they fire externally mounted missiles or receive battle damage etc.
The issue was implementing it on the MAX while making the angle of attack sensor non-redundant.
Right, but if you disconnect the software developers from the whole implementation then there's the issue.
that they are paid $9.00 an hour only tells you that there's so little experience that not only would they not know about redundancy in the total system but at $9.00 they wont speak up about it either. There are few good ( / well paid ) software developers who won't at some point tell you if there is a fundamental design flaw even though they arent the architect or designer. And they will know.
This is the issue and this is Boeings penny pinching fault.
@@johns5558That’s not how software development for safety-certified systems (which includes avionics) works. The coders are the least important part of the process, and they have no influence whatsoever on the design of the software. That is done by different software engineers whose job it is to write specifications.
@@tookitogo , John, you're both almost there, but are missing the particular MCAS context. The *entire point* for that system was to be non-flight-critical & non-redundant, as per Boeing's own requirements - not $9/hr programmers' inexperience or indifference or the savings to be had in hiring them. We're talking about many orders of magnitude more money, elsewhere in the business, as the driver. Please allow me to explain.
Let me back-track a little. The 737 sits quite low to the ground, which had played a big role in its success over the decades, as that not only reduced the landing gear weight & complexity & allowed it to be tucked inside the wing instead of the body, it significantly reduced the size and complexity of all the requited airport machinery for service / loading & unloading / de-icing etc. Personnel, procedures, didn't need to be heights-rated. A devastating competitive advantage.
As material science and fabrication advances allowed greater engine fuel economy through larger turboprop bypass ratios thanks to giant front fan blades becoming viable, the engine diameters naturally grew. Eventually to a point when compelling new engines would fit under the Airbus A320's wing but not under the 737's wing. Boeing engineers pulled a design rabbit out of a hat, flattening engine shroud on the bottom, shortening its pylon and putting the fattest part of the engine way in front of the wing (taking up the wing thickness' worth of space). With all these measures combined, the new engines fit on the old airframe, which ( *this bit is key* ) qualified the 737MAX, as far as the FAA et al were concerned, for the simplified deployment, without extensive pilots' retraining incl simulator hours on what woukd otherwise have been a whole new type. This new type retraining requirement is worth many millions *per airline* in lost profits and risks - that manufacturers typically pony up. It was *key* that the 737MAX not require that retraining, because the A320NEO didn't, b/c even if the Boeing had paid up for the sims etc it'd still take pilots offline, giving airbus oprators a market share in over their brethen running boeings.
With me so far? Good. There was only one small issue with Boeing's engineering miracle: the altered engine placement, in certain very small areas of a flight envelope, made the plane suddenly pull up. Boeing has had just the solution, digital flight control augmentation - a little autopilot of sorts that specializes in patching up / normalizing such small irregularities, making the plane behave seamlessly from a pilot's perspective. Many advanced mil airframes (fighters designed for an inherently unstable flight to maximize maneuverability) aren't even flyable by hand without such a system.
The only pickle was, while just about everything of the sort is doubly or triply redundant on a military plane, adding a redundant system on a commercial jet automatically flags it as potentially flight safety critical in the FAA review (large chunks of which were delegated by the FAA to Boeing to self-certify). That would in turn require pilot training, defeating the purpose of the entire effort & putting Boeing back to square one. Boeing's plan - as per its lesdership and bean counters NOT programmers - was to sail under the radar of this review, by specifically NOT having MCAS redundant. The rest is history: the gamble didn't pan out, and whistleblowers stripped any shread of plausible deniability, which is how I'm able to explain to you all this without any rational fear.
@@AlexKarasev I am not sure if that reply was meant to come off as condescending or not, but to me it was a little. I already knew everything you said there.
There’s no evidence of the $9/h programmers being used on MCAS. (The video quotes an article about flight display systems), and nonetheless, the coders don’t design the software.
The redundancy “flagging” is a real issue, in that that is a _horrible_ reason to not make it redundant, obviously.
As for if being safety-critical: any system that has direct control of flight controls _is_ safety-critical, whether it’s been categorized as such or not. I haven’t looked into details of this, but I’d be very surprised if the MCAS code was not designed using most aspects of safety-critical software design practices. (It obviously did not follow all of them, but that seems to be more deliberate than accidental.)
@@tookitogo certainly wasn't meant to be condescending - apologies if to you it read as such, same as for my part I had no way of knowing that all of this was known to you. I only had the senior engi eers make the specs bit to go on - whereas here we've the context that it was a business call even though in the end the senior engineers probably did write the specs.
Although you're tagged in the reply, it's really for everyone reading this incl 10yrs from now. Who knows, maybe some young engineer or leader or investor will read it and understand how we humans sleepwalk into such disasters and humankind gains another percentage point towards some life not being lost somewhere in an avoudable disaster. This story needs to be told (I wish, taught) and not just in aerospace engineering / FAA / OSHA etc but everywhere decisions are made. There's a pattern to bad decisions that's very different from "those guys were dumb; we're smart".
India consulting companies have some very good engineers, however not many. They tend to give you the good engineers at the start of contract then swap them for the lower quality engineers after a couple of months.
I used a company that over 2000 developers, one of my team would find the error for them, tell them the change the 1 PM, 1 manager and 3 engineers would make the change. When I found out that my engineer was basically doing all the work, they where gone.
We weren't outsourcing because of cost, just lack of specialist skills (particular telecoms protocol) their good engineers where probably one of the best in the world for that skill. however they were in project for 3 months then gone, to another project.
Experienced that also. The person I was left with (overseas) didn't understand what a function call was.
Exactly this, another branch of engineering but the same story. We have local engineers who have moved here from India and they are just excellent. We send work to a pool of engineers overseas and it's multiple cycle time that finishes with local rework anyway.
Just some Indian context to keep in mind :
India is also the country that runs ISRO, with its cheap, well planned and successful Moon and Mars missions, as well as highly successful commercial space launch industry. And ISRO is cheap, and a lot of ISRO engineers get paid at par with 10USD per hour, which still comes out to a lot of money here (= 10 x 84 x 22 x 8 ~= 150K INR / month)
In my mind, outsourcing to a company without aerospace engineering expertise seems to be the actual issue.
Practically half a dozen aerospace giants "outsource" to India, but they do so in the form of running their own branches in India.
These engineers are paid anywhere between 200k to 1000k per month, which is 2500 - 12000 USD per month.
They're the cream of the aerospace industry.
HCL Technologies is not one of these. It is a body shop.
I felt the need to say this only because a good friend works in a leading American aerospace company and he is getting awarded patents and certificates in aerospace systems.
I would also like to add that cheap Indian developers are pathetic. I suffer them in office every day, being an Indian living in India.
If you are doing job of national pride and importance, hold state position this is bit different.
They are activity looking for a new scapegoat ...
Most outsourcing plans are entirely to cut costs, not for expertise. There are great Indian engineers but you don't get them when your goal is cheap labor. I shudder any time I hear a manager talk about number of resources. Its not about the number of bodies, its about the professional knowledge, ability and the team.
Coding for ISRO certainly is on another level. Once ISRO starts to put people in space, the quality of the code will need to be better still in terms of redundancy, and failsafe designs to keep crew alive. That's the standard that Boeing must always be at.
This is a little misleading. The code didn't have any problems , it was doing exactly what it was meant to do: take control of the plane when a single sensor failed, and tricked the avionics into activating MCAS. This was designed by the engineers at a higher level, point the nose down over and over if one sensor output data that said the plane's nose was pointed up.
Yeah, that's my understanding as well. SW was doing exactly what it was designed to do.
Yes, architecs ...
They didn't outsource the MCAS, that was all Boeing genius. I'm unclear exactly what they did outsource to India, but they did outsource some 737 Max code.
True that. The MCAS software checks the ONE active AoA sensor (Angle of Attack, eq how high the nose is compared to the airflow around it). If the AoA gets too funky, the MCAS will start pushing the nose down by trimming the horizontal stabilizer to push the nose down.
Everything is fine and dandy, as long as the ONE SELECTED AoA sensor is working well. Those senors are of course outside, they have air/sand/garbage flying on them all the time, so they get quite easily broken. Now the broken AoA sensor gives out some funky AoA readings. There is the problem.
There is another AoA sensor on the other side of the nose/fuselage. That ought to be ok, but Boeing Chicago management clows (all economics people, kids, straight from universities, knowing nothing at all), decided that checking on AoA sensors, is not needed, and it is a "optional advanced AoA software package, for 19000 USD per plane". That software would then tell the the pilots (and MCAS) that "the left AoA and right AoA are saying very different results", eg AoA DISAGREE.
Only Delta ever bought that software, but other airline pilots (not Lion/Ethipioan) also knew what is a runaway-trim and how to fix it. Those two airlines did not know and did not care.
Pilots had to select left or right AoA sensor to be used, but as that optional software was not installed, they could not have "Auto, using both". Now when the active AoA sensor fails, the MCAS can freak out, and start helping the pilots by pushing the nose down.
Again: Stop the runaway trim by pressing Auto-Trim: CUTOUT right next to the flap lever) AND/OR Stop the running trim wheel by hand (can be done easily). Continue the flight to the destination as normal, with of course now manual trimming. Easy. But again, not easy when pilots are such bad bus drivers, from the lowest tree branch, and Boeing board thinks that all of this is "act of god, nobody knows why the MCAS freaks out...." . Plus of course a lot of corruption, blackmail and threats, and murdering of the whistle blowers, which we are all sure will produce fine Boeing aircrafts in the future..... sick and sad.
Sooo old school analogue instruments have something going for them?
I'm a controls Software Engineer and was born and raised in SA. I have immigrated to get a decent rate. Man, I do miss SA a lot!!
Oddly enough, my situation reminds me of that Mexican fisherman Parable.
Sidenote. Great video. Keep up the good content.
Ugh! This is depressing. I'm already ousted because of my age and it just frustrates me to see it the whole industry trending downward. I wasted money getting a degree.
You wasted a lot of money and most likely were misled about all the opportunities, good luck recouping your money. Sounds a lot like false advertising.
One additional issue is that when outsourcing work literally half-way around the world (e.g. USA to India) makes it very slow and awkward to communicate. Mid-day in one location is midnight in the other and with weekends being skewed you end up with an effective three day work week.
From a software point of view this video is spot on, though I happen to personally know that Boeing has far far far far more troubles than software. If it’s Boeing, I’m not going. MCAS has a shortage of air speed sensors such that even if the software was written by a genius it couldn’t safely do its job.
The C-Suites are universally staffed by managers that rarely understand the tech involved. In nearly everything. They are focused on the quarterly results, their personal stock options, and their personal advancement path.
Boeing's previous merger caused many experience Boeing engineers to leave the company. The new management structure gave the world the 737-MAX. Oddly enough, one does wonder what those stock options are worth today.
I have worked for an outsourcing company. I fondly recall them designing and building an overengineered system to achieve something that could have been done by a few tweaks of existing code.
A common theme with these companies was a lack of understanding regarding testing.
You brought up loyalty, which is a very good point since it goes both ways. Who is more likely to leak information or put something suspicious in the code base, your local salaried employee or a faceless €9-an-hour entity in a hive of 2000 drones?🤷🏼♂️ _But hey, it’s not like they would put that software in anything critical, like an aeropl… right._ 🤦🏼♂️
You are just a 100% right. This video is full on the spot. Have experienced all of what you said myself. Thank you for the presentation !
They have completely lost their minds.
When you get more money, but you kill or harming people's health, you're committing a crime.
That's oblivious.
$9 per hour for a software developer is bad enough. But there is a bigger question: what about testing the software? Did Boeing test this software to see if it complied with the safety assessments or FAA regulations? Or did they recycle the tests from older designs that weren't as software dependent?
The bigger thing that a lot of airplane safety guys were screaming about right after the investigation came out was lack of redundancy. The software depended on a single sensor for its input data, and when that sensor failed, the software that relied on it went haywire. As you will hear time and time again in _any_ class related to industrial safety, the minimum is two sources for safety (when a system is allowed to fail completely) and three sources for fault tolerance (when it is not allowed to fail completely, usually for safety reasons). The software actually performed perfectly -- assuming the input data wasn't wrong and that the pilot would never need to disengage it quickly and intuitively (the process to disengage it was quite involved and rarely mentioned in training material, if at all).
@@mage3690 that was the first and main issue one of my neighbours mentioned as soon as details about the crashes had been made public. He was an engineer with Airbus. He said that such a design just wouldn't have received approval by European authorities which is why Airbus wouldn't even think of doing something like that. Failed approval procedures are just way too expensive.
Boeing basically runs FAA now. FAA regs are basically "what does Boeing say?"
Display software is not code that read and cross check the sensors...Who wrote that unsafe original code of reading from only 1 sensor when aviation is required to have a back up sensor. ie . there are 2 sensors to read and compare?
I went to business school, worked in business. I've also worked as a coder as well as system design overseeing developers. Knowing what and why you're working on something makes all of the difference to what you're doing and how you do it.
Overseeing coders who don't know is much more effort.
One idea in management is that people with the same job title are es interchangeable as different products with the same name. Neither is true. Modern managers are often administrators without any concept in the different qualitative factors that don't show in reports. That's why they focus on cost so much, it has a defined result. Investing in qualitative factors does not have an easily discerned result, so nothing direct to show. Managers without intimate knowledge of the product will neglect qualitative factors.
One premise in management theory is that you can manage a company by changing the amounts in the balance sheet, incomes statement and cash flow statement.
02:18. Oh yes. I was raised a software engineer. Am now a projects manager. Management doesn't understand software. Full stop. Make box do thing must be easy right?
It’s not really “airline software”, but rather aerospace software.
Much more critical.
I spent 25 years in airline ops for various airlines. Over the years technology has pretty much taken away the common sense of engineers and other professionals. To give some perspective. I was involved with weight and balance since the early 80s. We were taught the theory as well as how to correct issues if the computer system crashed (very often in those days). We were also taught on how to use various airlines computer systems to perform our duties. Those days, we had to learn the computer entries and type them in. These days, it is just point, click, drag, drop. or even tell the system to "auto load". Sure, there are error checks, but if you do not have the basic knowledge of the principles, you are pretty much screwed if the error persists or if the system goes down.
I use to work with avionics tech folks as well. These days, there is no thaught process involved. You just tell the "AI" what you need and poof - out comes aa sh*t load of spagetti code.
Then you outsource because of "costs". Penny wise pound foolish.
I digress.
This is a sad development worldwide. Anyway I like your presentation style. Calm, informative and factual. Keep up the good work.
Very good video and interesting topic! Please, keep in mind General Electric company and Jack Welch decisions… I think that there is something deeper here.
Regards!
Thank you so much for bringing to light what IT people from Western countries have known for and experienced for quite a long time. Time when this whole outsourcing business model came up. The only thing upper to higher level IT management people care about is 'making an impact' to justify their jobs. Cutting costs is always a sure fire way to provide that visibility! As these people are mostly fixed term employees; aka contractors, all they care about is the bonus payout they will receive at the end of their term. Short term, short sighted outcomes will get them their KPI's. When these big outsource companies make their bid presentations, things look good on paper, structurally. But we in the IT industry know how easy it is in these 3rd world countries to procure the right credentials. They promise big but almost always under deliver or the quality is very poor. Not the $9 workers' fault as they're mostly working on 3 or 4 projects at the same time. It is modern day slave labour.
A friend worked for an engineering company when they outsourced the shop floor. The failure rate shot up because the in house engineers knew 20% of the drawings were wrong and how to compensate . The obvious solutions of either correcting the drawings or warning the new workforce was never going to happen if they were being laid off. Same with software, there is nothing like experience with the same company and no college or knowledge transfer will compensate for this.
I absolutely love these investigative type of videos you make!!
As a 30+ year software engineer and coder, I dislike a lot of Indian outsource companies, but you need to do much more research here. As far as I can read about it, HCLTech coded the software correctly, and correctly to the Specification. The Boeing SPEC was wrong/bad! It does not appear the Boeing tested the system completely, or well. This will go into CompSci classes as an example of bad outcomes even when the SUBsystem SPEC is correct. A key aspect in the MCAS failure is that once the pilot overrode the system, it would wait 30seconds and then return to its bad behavior. Another aspect is using 1 sensor that is known to fail, which is why there are 2 sensors on the aircraft. BUT if they had used 2 sensors, they triggered the FAA DO-178C spec testing phase which is REALLY expensive.
The Boeing 737 MAX failure due to the MCAS system was a known issue within Boeing, and they deliberately tried to remove all mentions of MCAS in the training manuals to minimize pilot training time as much as possible. There is no possibility that HCL Technologies had any involvement with the MCAS system. Boeing has clarified that critical flight control systems, including MCAS, were handled by its own engineers, while HCL's work was limited to more routine, less critical tasks related to the 737 MAX.
Heh, it's not because of outsourcing or coding errors. It's because Boeing decided to bring in McDonnell Douglas execs into their own executive structure when they merged in 1997. Heck the CEO of McD became the CEO of Boeing. Every company including Airbus, Dassault, Bombardier and Embraer outsource coding. But they review and test the code before putting it in a plane.
Before the merger Boeing was an engineers company and the ones who made the planes called the shots. Even the CFO didn't care about impressing bankers.
After the merger, Boeing got a chairman who didn't have an aviation background. The passion changed from making planes to making money. The saying in wall street was that McD bought Boeing with Boeing's own money.
To understand the basic problem today with the 737 max, you have to understand that the original 737 was designed in the mid 60s. He engines then we're much smaller in diameter so that it was easier to reach by ladder for maintenance.
The engine fans then we're about 42 to 49 inches in diameter. The Leap1 engine of current 737 Max is about 69 inches. Obviously you couldn't compensate for that kind of difference in ground clearance by adjusting the height of the landing gear. They should have gone for a fresh plane design. Instead to cut costs, they pushed the engine way forward on the wing which raised a whole bunch of issues. They tried to take care of it in software, but there is only so much you can do in software. This is the root cause of all 737 max issues.
A thing not covered in the video is how when it comes to outsourcing, it is the big hr company that gains most from the relationship more than the client and the engineers combined. Engineers are paid on the bargain.
I agree there were rookie mistakes made with Boeing's MCAS. Boeing fired many of its veteran flight control engineers a year earlier, saying that their flight controls were already proven. The problems was MCAS was new.
Boeing decided to go with one MCAS sensor. One way the sensor can fail is via a bird strike (at least one of the Boeing crashes, is tied to a bird strike on that sensor).
An experienced engineer, in the U.S. Knowing full well they can be sued for any failure or omission during the design process, would have pointed out the necessity to install a redundant sensor. And, to test the system against a variety of mechanical failures. And do their best to detect those kinds of failure also in the software (which really would require some push back from the engineers).
But, a 3rd party company, with inexperienced engineers, it might never occur to them, this is a bad design (one MCAS sensor). And, if an engineer did realize this, would they be stifled by inexperienced managers that they report this problem to?
The retail cost of the 2nd sensor was only about 120k installed (that's what Boeing charged). The actual device was a tiny fraction of that cost.
So there is blame to be had by Boeing sales and design at the U.S. level for not thinking about this. But then, if they just fired all of their experienced U.S. engineers, the U.S. sales and design team might have only the bean counters left, who are trying to make sales, anyway possible.
Boeing sacked their U.S. engineers, they outsourced the job to a 3rd party, that then went with an inexperienced remote company. That remote company will never pay the real cost for this disaster (which is billions), no one in the remote team can be jailed for gross negligence, which this was.
In short, they outsourced because "No Accountability" is way cheaper than "Responsibility".
And this isn't a communication problem, because there wasn't anyone at Boeing HQ failing to communicate the requirements. It was a failure of Boeing management to recognize the necessity to keep experienced flight control engineers on payroll for a few more years. Also, there is responsibility at Rockwell and the 3rd party company responsible for developing the software. But, you'll never get them to admit (except in court, which will never happen with the Indian engineers). That they either should have realized the implications of having just one sensor and/or software too inferior to sense it was a mechanical anomaly (which is possible to do, although difficult, and requires testing and more sophisticated software).
A few thousand dollars saved, and priceless lives lost.
And Boeing deserves it's un Air Worthy reputation, until they fix the organizational problem from the top down.
As a software engineer who works in the aerospace/aviation industry, I worked on a project that included computer cockpit modules for Boeings aircraft. Another reason these companies outsource the software is because it is typically much quicker to get a product turn around. In the US, mission critical software goes through an insane amount of testing and it is very expensive. So not only is the outsourced labor is cheaper, but they can also cut costs on some of the testing for its software. I dont work at Boeing myself so this is a bit of speculation but just a thought.
Great presentation. I have seen companies outsource developer jobs and really the saying “you get what you pay for”. Many executives seem to look at only at the costs and truly believe that more robust developers who understand the business and have good communication skills are not required. I am sure if these executives were hit with direct consequences (jail time or personal financial penalties) they would make better decisions.
It's not "business context," it's "Institutional Knowledge" that the software engineers need.
The nacelles for the Boeing 787 are made in Mexico.
Last time I looked, the wage of assembly workers in Mexicali was $3 per hour.
This was very informative and nicely shaped episode. Thank you and good luck!
My dad was an Intel casualty. Monday was his last day after 25 years. They're floundering. The only thing holding them together is the US government because it's the only fab manufacturer left in the US. He LITERALLY told me to get an AMD CPU for my gaming rig build 😭
These corporations don't think long term. A lot of Indian developers on H-1b visas are leaving to go back home. They've truly pooped the bed on this issue. For myself, I am a hobbyist but would love to work as a dev full time eventually. I'm planning to go to Europe...and I'm from California. I grew UP in Silicon Valley more or less and I'm leaving. 😂 It's already biting them in the ass.
I work on ed tech software and i sometimes wake at night worrying about bugs or some other thing I might have overlooked. If I worked on planes, I would just live in a constant state of worry and testing
My employer uses HCL as well, but they only use them for supporting roles. Developers and Quality personnel are all US based.
Another fantastic video Dee. Well done. Saving here to send to someone in the future.
Genuinely amazed it's allowed. If you look at the documentation and verification systems required to maintain planes, it's amazing the same isn't required for Fly-by-wire aircraft.
Maybe it's a historical thing ; when the control columns is connected to the tail surfaces directly or by hydraulic links if the navigation system crashes it's a PITA but you can still fly and land the thing. This doesn't apply with A320 type aircraft.
"When you said the program might crash, we didn't think you meant LITERALLY..."
When big trees fall, there is more light for saplings to grow. And big trees fall for a multitude of reasons acting in concert. Boeing looks a bit creaky at the moment and the winds of change are blowing hard.
Problem, Boeing is "too big to fail." It's more like this mutant tree grew to giant size and gobbled up or killed most of the forest. Yet instead of splitting it up, we're actually allowing it to continue to subsume more trees.
Alternate example, we built a neighborhood around the monster trees. The only space for saplings to take root is the empty space left by houses the tree destroys when it falls down.
Turns out gutting anti-trust, and allowing monopolies to flourish for 30+ years has consequences. Who would have guessed?
That Loyalty bit in tech hit home Dee.. So much for quantity over quality... This doesn't even just apply to tech but we see it the hospitality business.. Fast food chains cutting cost changing the quality of their foods (cough) Chick-fila (cough).
"The shitification of everything" -According to Nicole
Brilliant summary of what’s wrong with software development across all industries.
I think around 12:15 you talked about billable hours. The time and materials projects are long gone, at least here in the US. For years now the projects are "fixed fee". The consulting firms underbid to get the project, then they get cheap labor mostly from India and work those programmers to exhaustion, 60 to 80 hour weeks, for months, with no weekends or time off. The clients are also at fault by trying to increase the scope, for free of course... I've been in the software consulting industry since 1990 and I've witnessed this trend first hand.
nice video.. This has been issue for a long time (at least 20 years). I think that the biggest issue in these companies that make big products is that even tho they have millions of lines of code in their system they still think that they are automation company or something like that and they do not understand the value good software. It looks nice on Quarterly report to have lower costs, but nobody knows how much more that costs in long term. More experienced sw developers change role and get more money when they manage these outsourced developers and do less coding. So these things just pile up.
Aircraft systems have some pretty strict restrictions on what code semantics can be used.
I have more than forty years of experience in C/C++ and I recognize that it would take me some time to learn to code well within those restrictions.
I would never consider giving this sort of projects to a junior engineer.
But I don't want to kill anyone.
But for bean counters that do a cost benefit analysis of plane crashes versus profits, with the option of killing people as an acceptable loss , it might make financial sense.
Outsourcing is not cheap they still end up paying over 250k per engineer. The money saving comes from once the project is ended they can’t just fire a full time employee at Boeing, but they can easily terminate the outsourcing team contracts.
Once I was talking with guy, which said he was working on a first MCAS. He said they did "everything as it should be" and 100% as demanded from Boeing. Crazy words if You ask me.
It's called a tombstone reaction. When people die or the project eats dirt, something starts happening.
But not until.
Been hearing this topic over & over for many years. Your particular rant is a good one though. Cheers.
I've worked with outsourcing devs, from certain country well known for outsourcing. Their approach is absolutely horrendous. I remember back in the days they even refuse to use git because they can copy paste the folder and add the date after the folder name. They refused to unescape the json string before sending them to the API. Things like that
Its not actually a programmer problem its a management problem where an MBA with no knowledge in the domain or the tech stack ends up messing everything up
Hello, the focus is not cutting costs but increasing profits in order to pay the top shareholders a higher dividend which includes yes cutting costs but not only, could also include inferior materials and cheaper manufacturing with a diminished quality control....
My mom travelled in Nov 2019 with Ethiopian airlines safely thankfully so I get queasy every time I hear stories like this with them
I worked on a large government project in the '80s which outsourced a large part of the development to an overseas supplier. They supplied dozens of fresh graduates with zero experience and billed by the line. We had the most extensively documented code I've ever seen as comments were counted as lines. We analysed the code and worked out there was something like 3.4 comment lines per code line in the source - and this was Cobol (which, for you youngsters, is pretty much like writing in english to start with). The quality of the code was OK, not brilliant, but OK. The biggest problem was the 'consultants' would never admit they had made any mistakes. It took days to get even small errors fixed as they would just say, "t's working as spec'd" even if it clearly wasn't. The culprits? A large American accounting firm with a computing consultancy division.
I have experience in similar safety oriented industry and there are some glaring points to consider:
1) There are some stringent coding standards (coding standards) which should be maintained. Doing so ensures easy debugging and less possibility of errors.
2) The Verification and Validation team should be able easily pick up flaws after thorough testing using simulators and checking project data/parameters.
Seems like coding standards were ignored and poor validation was done. This means that the problem is systemic and completely overlooks/skips all systems of checks and balances.
Boeing didn't even let pilots know that MCAS was a part of the plane's avionics system. They were trying to avoid having to retrain aircrews by preserving type commonality between 737 models.
Basic question. Was the code developed in India tested thoroughly by the MCAS group in Boeing here in USA? Did they find out if the code did exactly the requirements specified?
I have worked in developement process of safety critical software components and I too noticed that management is blindly relying on the arbitrary safety standards and documentation which are all completely pointless when carried out by bad programmers
I worked on an airline reservation system, if we messed up you might miss your flight. If an aircraft programmer messes up you can loose aircraft.
I know HCL, they’re the last people that should be programming Aircraft.
My role got outsourced, it cost the company over $300k to build a replacement system because they couldn’t find people to do my job.
You hit the nail on the head, you get what you pay for... even engineering companies which services top 500 customers have used "low cost" centers and have to redo the work many times. The people employed by these low-cost centers are capable but in engineering school is only the start to do the work requires mentoring and educating in industry specific practices. Which are not taught by universitas, and master or PhD digress only teach more theory not industry specific practices required to do the work. An example is in the US is in hazardous electrical areas, OSHA requires the components to be listed by a testing lab acceptable to the jurisdiction having authority. ATEX components are typically tested by the manufacture and not an independent testing lab making them unusable in hazardous areas in the US.... More and more Sates in the US by law required engineering work has to be done under the supervision, reviewed by and signed for by a licensed engineer. In all states, a bachelor's degree with a minimum of four years of experience and in most cases passing an eight-hour exam in the area of practice (note there are some exceptions to the minimum requires which varies by states). If you have a master or a PhD, it only reduces the years of experience to three years. Why? because schooling only prepares you for practicing, it takes mentoring and experience to do the work.