I Accidentally Saved HALF A MILLION $ | Prime Reacts

Поделиться
HTML-код
  • Опубликовано: 4 ноя 2023
  • Recorded live on twitch, GET IN
    / theprimeagen
    Reviewed article: ludic.mataroa.blog/blog/i-acc...
    By: ludic.mataroa.blog/
    MY MAIN YT CHANNEL: Has well edited engineering videos
    / theprimeagen
    Discord
    / discord
    Have something for me to read or react to?: / theprimeagenreact
    Kinesis Advantage 360: bit.ly/Prime-Kinesis
    Hey I am sponsored by Turso, an edge database. I think they are pretty neet. Give them a try for free and if you want you can get a decent amount off (the free tier is the best (better than planetscale or any other))
    turso.tech/deeznuts
  • НаукаНаука

Комментарии • 396

  • @agarbanzo360
    @agarbanzo360 7 месяцев назад +520

    I worked at an insurance company as an engineer in our data science team. As part of exploring how to make the company more money we found that they should just fire everyone, shut down the insurance business and take the inconceivable cash reserves they were burning through and dump it into the S&P500 or a hedge fund.

    • @JGComments
      @JGComments 6 месяцев назад +51

      This is me too lol. We aren't a public company though so we have a "mission". But the real post-tax returns would be 25x what we actually make on our capital. We could just pay someone else to do what we exist to do for people :(

    • @juanshaftpatel7488
      @juanshaftpatel7488 6 месяцев назад +5

      but the the serfs wont have jobs...

    • @kakerake6018
      @kakerake6018 6 месяцев назад +16

      dear lord that's actually mental.
      how to save money 101: "do nothing"

    • @herrerkan
      @herrerkan 4 месяца назад

      10% of the global companys are "dead" companys. If interest rates wouldn't be this low and investors so many, then they would go bankrupt in seconds.

    • @allalphazerobeta8643
      @allalphazerobeta8643 4 месяца назад +4

      That company exists and it's called berkshire hathaway. Seriously, that company probably could have bought a successfully run smaller insurance company and move all the clients over.

  • @naturallyinterested7569
    @naturallyinterested7569 7 месяцев назад +1046

    I also accidentally saved half a million. Then they complained I was doing 'tax fraud'. Always the envy!

    • @jorderon
      @jorderon 7 месяцев назад +75

      This hits way too close to home. I’ve seen finance shut down engineering initiatives because it would “save too much money and we’d be audited”

    • @yegorzakharov8514
      @yegorzakharov8514 7 месяцев назад +9

      ​@jorderon what the crap, really?!

    • @baetz2
      @baetz2 7 месяцев назад +33

      ​@@jorderonI got my company audited and sued by federal tax service once. The company won and kept the money.

    • @JonathanTheZombie
      @JonathanTheZombie 7 месяцев назад +5

      How can taxes be real if our eyes aren’t real

    • @JonathanTheZombie
      @JonathanTheZombie 7 месяцев назад +2

      How can taxes be real if our eyes aren’t real

  • @neildutoit5177
    @neildutoit5177 7 месяцев назад +133

    "I asked for some compute to do the data science" hit hard. Spent a full year at a company waiting for compute. Not even a lot. We're talking a lot cheaper than my salary was. They gave me a new laptop. They hired new team members for me. And I sat there in front of my new laptop with my new team members waiting for the compute. The compute never came.

    • @scarfo
      @scarfo 3 месяца назад +9

      I can see your line manager sitting there and thinking "a new laptop! That makes perfect sense!" and then getting you a chromebook.

    • @impyrobot
      @impyrobot 22 дня назад +1

      Can't complain too hard you were being paid

  • @MsLemons12
    @MsLemons12 7 месяцев назад +182

    Big companies make me laugh. My current employer (major US pharmacy brand) accidentally disabled my credentials September 30th and they're still trying to get em re-enabled (now November 5th)...I'm paid to do literally nothing. Maybe I'm just unassigned

    • @mrrolandlawrence
      @mrrolandlawrence 7 месяцев назад +52

      i feel you. working for a uk company, my developer buddy being told to turn up but not do any work because the project could not get the funding even though HR was paying the salary. he said well you are paying me anyway so why dont i just get on with it while you sort it out. he was told he would be fired if he did so.

    • @MsLemons12
      @MsLemons12 7 месяцев назад +29

      @@mrrolandlawrence that's nutty....like they're paying him so their competitors can't hire him

    • @notapplicable7292
      @notapplicable7292 7 месяцев назад +1

      Mandatory training week!

    • @hackerhaze
      @hackerhaze 7 месяцев назад

      just sheesh... just... sheesh. Maybe they won't realise

    • @MsLemons12
      @MsLemons12 7 месяцев назад +9

      @@hackerhaze nah, they realize and have me call the IT "no support" line daily.... But they can't figure out the difference between Mac & PC and just tell me to reset my password lol

  • @MuhsinFatih
    @MuhsinFatih 7 месяцев назад +107

    that `tee` problem literally happened at my workplace a couple weeks ago and it took me 20 seconds to fix it. Turns out we've been shipping untested builds for months

    • @LudicityHackernews
      @LudicityHackernews 7 месяцев назад +16

      Ah well, I guess each of the things mentioned is fine - people make mistakes - but taken altogether it's a horrendous lasagna of badness.

    • @arthurmoore9488
      @arthurmoore9488 2 месяца назад +1

      @@LudicityHackernews I haven't seen that bad, but it's what happens when everyone is told to "use the existing tools" so builds what they need on top of the previous crappy layer.

    • @FakeDumbDummy
      @FakeDumbDummy 2 месяца назад

      set -eo pipefail
      Oh is that really you from the article​? If yes then great article btw @@LudicityHackernews

  • @germantoenglish898
    @germantoenglish898 7 месяцев назад +346

    I once worked in one company where delivery boxes were constantly being left in corridors and one manager actually said "We should start a PROJECT to deal with this problem". 🤣 After that, if a toilet wasn't flushed or someone's shoelace was untied "we should start a project" became the running gag. They had projects for everything but there were more meetings than things actually getting done.

    • @CottidaeSEA
      @CottidaeSEA 7 месяцев назад +37

      Managers doing those things is a very clear indicator that they don't have enough work to do.

    • @ChungusTheLarge
      @ChungusTheLarge 7 месяцев назад +28

      If the only tool you've got is a SCRUM, everything looks like a project

    • @tk1576
      @tk1576 7 месяцев назад +11

      I see a bright future, like 3 developers and on top 50 managers creating projects.... and then they will schedule meetings to block the 3 devs to get any shit done.

    • @sk-sm9sh
      @sk-sm9sh 6 месяцев назад +13

      Taking boxes out is no simple task. You need to research the issue to get basic understanding. Then you need to reach out any relevant people which involves cross team - maybe someone needs those boxes? You need to learn where containers are located which will involve booking a meeting with office manager who's been working from home and is visiting his parents so will only be in office back in two weeks. You need to check up with companies policy to meet the decarbonization program to see if there is any ways the boxes can be recycled or not. And finally once all the prep was done you still need to get someone to do the task and you're short staffed so until you get a new hire you cannot add any new tasks on your team's calendar.

    • @nigelnix1
      @nigelnix1 6 месяцев назад +2

      Boxes are complicated...really complicated.

  • @kamiljanowski7236
    @kamiljanowski7236 6 месяцев назад +54

    Ever since i joined the project in this multi-billion dollar company, I regularly find AWS misconfigurations that allow me to get admin access to the whole AWS account and I try to save the company from going bankrupt over high schoolers that could hack it... it would be easier if they actually cared and let me do my job instead of discussing whether we should fix it for 3 months because there are other services that actually depend on our vulnerabilities...

    • @John-lw7bz
      @John-lw7bz 5 месяцев назад +25

      "because there are other services that actually depend on our vulnerabilities" lmao

  • @transimpedance
    @transimpedance 7 месяцев назад +14

    Teej saying “what if they’re just removing sleep(10)s” was so funny to me

  • @dreadsocialistroberts
    @dreadsocialistroberts 7 месяцев назад +218

    I came up with a plan that would save my company 500k / year on AWS costs. I got laidoff instead.

    • @greyshopleskin2315
      @greyshopleskin2315 7 месяцев назад +5

      What? Why? That’s fucking insane

    • @LaughingOrange
      @LaughingOrange 7 месяцев назад

      ​@@greyshopleskin2315 Must have stepped on the toes of some manager with that one. And the terrible manager prioritized saving face over saving 500k.

    • @cmelgarejo
      @cmelgarejo 7 месяцев назад +17

      My guess: Because the company was still on aws free tier and they thought @dread was in cohoots with this other cloud so they booted him.
      Afterwards, they got a 45k bill just on RDS

    • @sutirk
      @sutirk 7 месяцев назад +65

      I just think @dread went against the "senior cloud architects" better judgement.
      A lot of companies have entitled cloud engineers and architects with no experience and a huge ego

    • @greyshopleskin2315
      @greyshopleskin2315 7 месяцев назад +3

      @@cmelgarejo makes no sense.
      First, he said he came up with a plan, not that he did something. And if you save 500k, your spending more than that. And I don’t think aws give you 500k+ for free

  • @radomane
    @radomane 7 месяцев назад +120

    Rewrite this for product managers and you have a conference talk you could live of.

    • @LudicityHackernews
      @LudicityHackernews 7 месяцев назад +15

      I have thought about this, but then I'd be rolling the dice on whether my employer friggin' guillotines me lmao.

  • @azerkahn
    @azerkahn 7 месяцев назад +123

    The project I work on is a fairly insignificant app, no huge sales, and nothing technically difficult at all. Very straightforward iOS app talking to a DB and showing some content, that's it. We have about 30-40 engineers working on this app, which is complete insanity -- three would be enough, and all that engineers actually do is writing documentation that will not be kept up to date, discussing how we can change our architecture to improve productivity (lol), or refactor some piece of code for the nth time because there's nothing else to do. Yet somehow, this app has terrible performance, it's littered with bugs, and the development hardly moves forward. I feel this dude's pain.

    • @LightninMcQuade
      @LightninMcQuade 6 месяцев назад +4

      sounds like the Dave and busters app

    • @siddharth8184
      @siddharth8184 6 месяцев назад +10

      Sounds like a dream job to moonlight on

    • @TheBcoolGuy
      @TheBcoolGuy 6 месяцев назад +6

      Guess that's what happens when you just play-pretend being a business by uncritically imitating your incestuous perception of what a business is. The best and most efficiently made projects always come from people who genuinely want to create the product and who do whatever they need to in order to make it work. Not this ritualistic child's play.

    • @scarfo
      @scarfo 3 месяца назад +1

      It's terrible BECAUSE it has 40 engineers working on it.

  • @NibbleMeTwice
    @NibbleMeTwice 7 месяцев назад +34

    These videos are some of my favorite when it comes to taking breaks from code. It gives me motivation to get back in the trenches. Thank you, Dyslexigen!

  • @TankorSmash
    @TankorSmash 7 месяцев назад +99

    The guest host was amazing on this, very charismatic. Looking forward to more!

    • @Kane0123
      @Kane0123 7 месяцев назад +34

      Do you love Microsoft Access or something?

    • @TehKarmalizer
      @TehKarmalizer 7 месяцев назад

      Agreed. I wish I knew where to find more content from teej_dv.

    • @kiikoh
      @kiikoh 7 месяцев назад +4

      He has a great mustache too!

    • @JeremyAndersonBoise
      @JeremyAndersonBoise 7 месяцев назад +2

      TJ is a an actor, he can’t code, but he’s nice. MS Access(!?) for the love of SQLing, teej, bruv..shake my head 😢😂

    • @JeremyAndersonBoise
      @JeremyAndersonBoise 7 месяцев назад +1

      @@kiikohI see what you did

  • @jasondoe2596
    @jasondoe2596 7 месяцев назад +199

    The original article is awesome, LOL.
    It's very well-written, and _you can just tell_ that the content is true, because only truth can surpass fiction so convincingly.
    edit: autocorrect fix

    • @JeremyAndersonBoise
      @JeremyAndersonBoise 7 месяцев назад +1

      Strong agree

    • @disguysn
      @disguysn 7 месяцев назад +6

      I've worked at places that had elements of this or all of this. If this isn't real for him it definitely is real somewhere.

    • @jasondoe2596
      @jasondoe2596 7 месяцев назад

      @@disguysn I've also seen such organizational absurdity.

  • @Durayne
    @Durayne 7 месяцев назад +47

    Sounds legit.
    Big Companies always have this Management overhead, where internal political "feelings" are more important than actually being productive.
    The only reason why they can survive this, is because they are big and usually have big tax advantages acting internationally.
    Thats why big companies often then to hire smaller companies for certain aspects. Because those cant afford acting like this and need to get stuff done in order to survive.

    • @VictorSamuelMosqueraArtamonov
      @VictorSamuelMosqueraArtamonov 6 месяцев назад +3

      wait a minute... are you saying that big corporations are as inefficient as goverment...

    • @Yupppi
      @Yupppi 5 месяцев назад

      This sounds a bit like a discord server I was moderating. Nobody ever did anything to progress things. If I did something to get anything done (naturally people aren't replying to that kind of messages for days where you ask about doing something) suddenly everyone would wake up to ask and criticise it (not that it was a bad move, just that someone had actually done something) and maybe then someone would do the change in the way that they wanted. If I suggested finally doing something instead of doing it myself to avoid the consequences, nothing happened.
      So basically after a while of trying to make things move, I just stopped trying and muted the server and let it descent into a tiny inside group's chat with no greater purpose for people outside the group (as was originally planned). Naming the wrong people to manage things is a quick way to kill potential. Unfortunately that server had all the people who were against any change and doing any effort in position to decide on things.

    • @lorscarbonferrite6964
      @lorscarbonferrite6964 2 месяца назад +1

      ​@@VictorSamuelMosqueraArtamonov The size of an entity is positively correlated with its similarity to a government across all domains.

  • @ambhaiji
    @ambhaiji 7 месяцев назад +34

    these 2 are funnier than any twitch streamer group I have ever watched.

    • @bearded14159
      @bearded14159 3 месяца назад

      The second one pretty noises for listening.

  • @codedusting
    @codedusting 7 месяцев назад +33

    Working for one client where everybody is just a manager and it's bad. Everything, even a version bumping of an outdated package or framework version, requires meetings and etas and God knows WTF! All the while they keep talking about low performance, deadlines, etc.

    • @CottidaeSEA
      @CottidaeSEA 7 месяцев назад +14

      But they are working with agile, fast moving teams, how could it be slow and low performance?

    • @LudicityHackernews
      @LudicityHackernews 7 месяцев назад +1

      At the workplace from the video, we once had a 3-day card which just involved me changing all our container images to say :python3.9 instead of :latest. It took like five minutes but people would REFUSE to vote for less than three days when we scoped the card.

    • @CHR1SZ7
      @CHR1SZ7 7 месяцев назад +1

      @@LudicityHackernewsoof they make you estimate on actual time??

    • @LudicityHackernews
      @LudicityHackernews 7 месяцев назад +3

      @@CHR1SZ7 Yes, but then they don't enforce the times being missed, so we just vote 3 days on everything because it makes the meeting end faster. I wish I was joking.

  • @Waitwhat469
    @Waitwhat469 6 месяцев назад +5

    Afters years of saying we need more engineers to work on some super ambitious projects and I get assigned three power point engineers.
    I understand this so much. Like hey we need to do this, we need x amount of resources and about x number of qualified engineers. That means we need to take a few years to build the planning team to build the plans to build org charts and charters to build the budgets to build the teams that will build the systems.

  • @Xe054
    @Xe054 7 месяцев назад +20

    @ThePrimeTime This article was written by the same person that mentioned PowerBi in a previous video. This person also wrote an interesting follow-up article to the first reaction video you did. It'd be interesting to hear your take on his take.

  • @pecket
    @pecket 2 месяца назад +2

    I think there are surprisingly many of us who have similar stories, that most of us aren't allowed to talk about. Love hearing them though when they're available, thanks!

  • @GRHmedia
    @GRHmedia 7 месяцев назад +9

    People tend to attach others that succeed for a number of reasons. They get upset that others are getting credit and they aren't. They worry people above them will see this person doing good and wonder what they have contributed.
    I've had people try and take credit or get in on credit for work I've done. Usually I've made documents prior to hand to ensure that no one else can do that. I made people aware of the impossibility to stealing or leech credit that caused a number of ripples with it.

  • @georgehelyar
    @georgehelyar 7 месяцев назад +30

    "actually trying it" is what agile is.
    People hate agile because they don't do agile, they do scrum or safe or waterfall and call it agile.

    • @CamembertDave
      @CamembertDave 7 месяцев назад +27

      Nothing says "self organizing" like mindlessly following the rigid structure prescribed by some document.

    • @jeremiedubuis5058
      @jeremiedubuis5058 7 месяцев назад +6

      Agile doesn't specify much except to lower the amount of intermediaries common in waterfall and enable realtime communication. Yes most people do scrum, but to be honest the goal of scrum is not efficiency it's inefficiency management. At the end of the day most large teams rely on the efficiency of a handful of high performers to make up the difference and the management strategy is just there to ensure .1 production rate rather than 0 for the rest of the team.

    • @Muskar2
      @Muskar2 7 месяцев назад

      @@jeremiedubuis5058imo, the goal of SCRUM is to improve communications... at the cost of productivity. A very appealing idea to the non-technical employees, I've found.

    • @hippocleides7105
      @hippocleides7105 7 месяцев назад

      ​@@jeremiedubuis5058Management outside of big tech is so incompetent generally that they don't realize 1 dev =/= 1 dev. You can't butts in seats programming. You need competence. Why the hell would I spend $500k on 10 $50k "devs" when I could spend the same $500k on two top 5% deva for $250k each.
      It's like, they talk all about "80-20 and pareto" but then conveniently forget it's WAY more applicable to people, too. 😊

    • @qwerty11111122
      @qwerty11111122 Месяц назад

      Faux leather bagpipes

  • @ilcorion
    @ilcorion 3 месяца назад +2

    I've been there, when everything was so red-taped, that we eventually had to have a secret chat and a secret way of doing things, like 10x-20x times faster, going under the radar

  • @MrTiteuf12
    @MrTiteuf12 4 месяца назад +2

    The 600 line of commented code and the mysterious code that nobody touches were so relatable. Entered this ancient project (they still use SVN for versioning) that has 90% of the documentation in the manager's memory and one of the biggest files in the project with almost 30K (!!) lines of code had almost 10K comments lines, just in case they need some of the old code...

  • @distinguishedmoments2277
    @distinguishedmoments2277 6 месяцев назад +11

    It seems the company i work for is actually pretty damned good, i enjoy reading these horror stories as it makes me feel better

  • @KrisRogos
    @KrisRogos 3 месяца назад +1

    28:00 onwards is so real!! My company had a 3hr meeting, including all the devs, QA, customer support, HR, the CTO and the CEO, to discuss in great detail 25 QA bugs and if it would be a good investment of resources to address them...

  • @salvatoreshiggerino6810
    @salvatoreshiggerino6810 7 месяцев назад +14

    Most of my career has been in analytics platforms. All of it has either been pointless busywork or a shameless upsell on a core product that nobody asked for.
    They're all solutions looking for problems. Instead start with the problem and like this guy says, put a couple of smart people on it instead of hiring scores to build huge complex systems with no idea of the end goal, except that it should use React, Spring and microservices.

    • @LudicityHackernews
      @LudicityHackernews 7 месяцев назад +8

      Author here. I spoke to a director recently and he posited that it is extremely difficult to approach things this way due to the incentive to "ship projects". The conversation then moved to other things, but my interpretation of that is that as an executive, you could either ship one project that works well by giving a team plenty of time to scope it properly, or you could ship four projects that barely limp over the line that the board wont understand anyway and look four times as good for your next million dollar gig.
      So... they just do that, since analytics falling over isn't a horrific bug that they'll even notice. Heck, one of our pipelines stopped working for two months and we noticed last Friday. Solution? Turn it back on and don't tell anyone, because no one noticed for two months anyway.

    • @JGComments
      @JGComments 6 месяцев назад

      That’s the name of the game! 😂 Consultants don’t solve problems, they sell solutions lol.

    • @hacktor_92
      @hacktor_92 6 месяцев назад

      sounds something that I managed to prevent at my job. every single time i was constrained by a specific tech, it was a failure, and every single time i went rogue and used whatever i felt that was needed for the project to actually be productive. and it'd have been even more productive if i'd been allowed to use whatever it fits from the start.

    • @Bramble20322
      @Bramble20322 2 месяца назад

      @@LudicityHackernews Jesus fucking christ, at that point, just stop working and lie about everything happening, you're burning mountains of money anyway.

    • @Bramble20322
      @Bramble20322 2 месяца назад +1

      @@hacktor_92 Problem with that approach is that whenever you fuck off to somewhere else, whoever's left has to figure what the fucking fuck you've done every time you went rogue. Not a good approach unless you treat your job like a hobby.

  • @StrengthOfADragon13
    @StrengthOfADragon13 3 месяца назад +2

    So, in college I was informed of a study on Agile that showed it only worked for teams of experienced programmers, which tells me that realistically "as written" agile doesn't really work well

    • @yojou3695
      @yojou3695 Месяц назад

      it's not even about experience. It's a company culture problem.

  • @jouniosmala9921
    @jouniosmala9921 6 месяцев назад +4

    The first time I saw a database it was at a month long summer job as a teen. My job was to look, in one table the age of a building, then go look on another table the location of its address and then manually color the building in a map based on age. It took me couple of weeks of trial and error and learning to write a script that did the job. (The scripting wasn't just about database but also about map projections and the coloring commands, and I had to learn everything from scratch with over thousand page manual having never seen those things before.)

  • @lvran28
    @lvran28 7 месяцев назад +14

    Jiminy Christmas, Prime. Stop doing this to me. Now I can't write a simple email without opening Neovim to do it. They hand me forms at the DMV, and I ask them, "can I fill this out with Vim motions?". I'm told that the Emacs military junta has my name on a list of undesirables, and that I have a smooth baby bottom.

  • @tomthetom_on_tour
    @tomthetom_on_tour 5 месяцев назад +2

    16:12 Yeah TJ, I second this. Access was cool at the time. Making it easy to add masks and creating a frontend on top. I guess dbase III and IV was similar but text-based.

  • @andythedishwasher1117
    @andythedishwasher1117 7 месяцев назад +6

    15:50 we're working on the PowerBI problem. Most of what it's used for can be skirted by hosting either custom or open source alternatives more cheaply on the same platform in the same networks.

    • @marceelino
      @marceelino 7 месяцев назад

      If you know what you doing, it's a great tool.

    • @andythedishwasher1117
      @andythedishwasher1117 7 месяцев назад

      @@marceelino Might be nice to use. I'd just prefer not to pay for it, which is apparently a popular sentiment based on the jobs floating around recently.

    • @Bramble20322
      @Bramble20322 2 месяца назад +1

      Problem is that if you use open source or custom alternatives, the company is stuck relying on internal manpower that may or may not know how to maintain it properly, and may or may not end up costing more for the company than just getting the damn SaaS solution.
      Plus using a industry-wide solution like powerBI facilitates hiring external talent without needing a fuckton of training and onboarding every time C-levels lays off half the company then hires new people.
      Its like using open source shit like Zabbix instead of proprietary solutions like Lansweeper on the sysadmin sphere. If you go the Zabbix role, you're just moving the costs inside your company and you dont have anyone to sue if shit goes awry.

    • @andythedishwasher1117
      @andythedishwasher1117 2 месяца назад

      @@Bramble20322 I can understand the concern there, but there's tradeoffs situationally. I think I'm probably speaking more to smaller projects and startups that assume they need the "industry standard" tools to get a job done and end up platform locking their company and sabotaging their payroll for an overpriced piece of software full of features they don't need.

  • @Yupppi
    @Yupppi 5 месяцев назад +1

    I find this story hilarious how one employee (and maybe a group of others) are actively trying to save the company in secret from the company so the company wouldn't stop them from being rational and saving the company. And the managers are trying to kill the whole secretiveness.

  • @JeremyAndersonBoise
    @JeremyAndersonBoise 7 месяцев назад +8

    I, and teams I have been a part of, have been punished for major successes in improving companies’ functionality and bottom lines so many times, I have lost count. Many companies have busted-ass toxic cultures like described here. Not sure, but I have a hunch; I may have interviewed with this company yesterday.

  • @WillDelish
    @WillDelish 6 месяцев назад

    Last year, I was told by a high up IT manager they were replacing some hardware. They were using the total number of profiles created for this hardware vs actually actively used hardware. I had done the most basic SQL, but in a few hours, was able to create a search that found online devices. Ended up saving us 7-10 million after I showed my work to him. Most of this hardware was replaced by software while the profiles were never deleted or updated.

  • @CHR1SZ7
    @CHR1SZ7 7 месяцев назад +3

    Sounds like a client i was on. Project got canned due to cloud costs getting out of control, turned out later some “consultants” set up an EMR cluster for a daily pipeline processing 200mb of data with 30 m4.xlarge instances

  • @snorman1911
    @snorman1911 5 месяцев назад +1

    The ability to recognize that your solution is becoming a Rube Goldgerg machine is a top tier skill.

  • @crackerjackmack
    @crackerjackmack 7 месяцев назад +14

    7:16 - Sounds like terraform, it will do that if you remove files. the files contain the lifecycle stanzas like prevent_delete, ignore changes. If the files aren't there, then there is no control and these errounous things are in the state so terraform goes and deletes them!

    • @LudicityHackernews
      @LudicityHackernews 7 месяцев назад +6

      You're bang on the money, minus the exact semi-niche library we use. It was in a repository using Snowflake's 'schemachange' project, which can be configured to run SQL if scripts are dropped into the right folder with the right name. It should NOT be used with a script that deletes prod but this is a temple of madness.

  • @nefrace
    @nefrace 6 месяцев назад +2

    At 14:00 in the chat: _"No better cure for impostor syndrome than reading stuff like this..."_
    For real.

  • @diogoantunes5473
    @diogoantunes5473 6 месяцев назад +3

    If one thousand of requests were spread evenly on a 40h work week that would give a request every 2.5 minutes, so that might explain the idle time, if there is indeed a start up cost.

  • @macctosh
    @macctosh 7 месяцев назад +24

    Wow, I am speechless. I actually believe this to be true. How can such incompetence exist??

    • @josephgonzalez9342
      @josephgonzalez9342 7 месяцев назад +19

      As a former contractor that got hired by my client for being a half decent software developer compared to the rest from my company, the answer is money. Companies pay for contractors to implement a project quick and cheap. That project then transfers to another contractor team to support / maintain the project in production because gotta keep the company developers free to work on other planned work. This team will not question why / how it was built like this; often saying 'it's the requirement' and scared to change it in anyway. It will be ~2 years before a software developer from the company peaks into the code base and discovers the massive technical debt created by the implementing contractors.

    • @skyhappy
      @skyhappy 7 месяцев назад

      ​@@josephgonzalez9342 The last company I worked at had a team of contractors make the frontend. They did not write a single word of documentation nor use comments. They didn't even write what component library the front end used. The only comments you found in that code base were written by employees.

    • @markm1514
      @markm1514 7 месяцев назад +9

      @@josephgonzalez9342 Same experience, different industry, its scary how close we are to outsourcing to the level of incompetence. Middle middle middle middle management disagrees

    • @josephgonzalez9342
      @josephgonzalez9342 7 месяцев назад +1

      @@markm1514 damn i did not consider that other industries would be doing this. This workers as a service trend is weird.

    • @Kane0123
      @Kane0123 7 месяцев назад +1

      Lolll this is not uncommon.

  • @daddy7860
    @daddy7860 6 месяцев назад +1

    This guy could've started his own data analytics platform as a middleman scraping percentages in that overcomplicated process and just funneled that money into his own company, and actually earned something over all that wasted time.

  • @tanchwa3740
    @tanchwa3740 7 месяцев назад +1

    What probably happened was
    Big firm: oh we've done this hundreds of times. Heres a super vanilla MVP to get you guys started.
    Stakeholders: great but also....

  • @oleg4966
    @oleg4966 7 месяцев назад +64

    9:00 It becomes more realistic if you assume that they probably didn't do it on purpose, and didn't spend any effort on making it work _reliably._
    Something this intricated and pointless must have grown organically, like a tumor.

    • @LudicityHackernews
      @LudicityHackernews 7 месяцев назад

      Can confirm that working with it does feel cancerous.

  • @Zekian
    @Zekian 7 месяцев назад +6

    This was too real and too painful.

  • @JorgetePanete
    @JorgetePanete 7 месяцев назад +3

    I'm now at a security company
    without any real security whatsoever.
    Get me out of consultingland

  • @twoolivetreesarise
    @twoolivetreesarise 7 месяцев назад +5

    great companies begin dying when they start creating presentations. When did being creative become "create a powerpoint". Let's talk about what we could do, will do, and might've done... but never do.

  • @johnathanrhoades7751
    @johnathanrhoades7751 20 часов назад

    I’m with TJ on Access. It was my first exposure to relational databases and my mind was blown. At this point I work with SQL and would never touch Access, but it was so cool to pre-programmer me.

  • @datboi449
    @datboi449 3 месяца назад

    I work at Small Firm Consulting, where we love the messes that Big Firm Consulting leaves behind which we fix.

  • @pietraderdetective8953
    @pietraderdetective8953 7 месяцев назад +3

    i love teejay's chuckle...it always makes me want to laugh with him LOL.

  • @raulmonge99
    @raulmonge99 6 месяцев назад +2

    I remember when I was new on a small company where they advertise some rural houses, the houses publish their photos to show on the web like AirBNB, they were storing the photos in like a direct hdd on a hosting where they charged them like 500€ a month, I was like bruh wtf 500€ a month for what, it was 500gb of hdd on a random bareback metal server so I changed it to an s3 bucket and next month came an invoce for like 0,03€, so you can see the incompetence and scammy tactics of some people.

  • @mangalegends
    @mangalegends 7 месяцев назад +6

    I can't even imagine what a high paced, high red tape environment would look like. Sounds way too contradictory lol

    • @HaggisMuncher-69-420
      @HaggisMuncher-69-420 4 месяца назад

      It's basically the army.
      We had a saying of "hurry up and wait"

  • @smithdoesstuff
    @smithdoesstuff 21 день назад

    Me screaming at the screen that he misread the bold 4th bullet point. “AFTER!!! It’s not a f-ing keep alive to stop it from spooking down, it forces them to constantly spool up new servers!!!!”

  • @oblivion_2852
    @oblivion_2852 7 месяцев назад +7

    A million dollars a year. Nice how about a million dollars a month? I've looked at some crazy amounts of data that was being stored but not used. "What's the business reason for this?" - *crickets*

  • @mattymattffs
    @mattymattffs 2 месяца назад

    The amount of times i declare "in just going to do it" and it's done 15-30m later is great. It makes so many people shut up

  • @aidanbrumsickle
    @aidanbrumsickle 7 месяцев назад +3

    So they used that half mil to cancel the next set of layoffs or give some engineers bonuses, right? Right?

  • @chrisstadler7111
    @chrisstadler7111 7 месяцев назад +4

    As I literally sit here writing a PowerPoint to show the people in my department that they didn’t have to rewrite core infrastructure had they done these couple things like my team did 😂

  • @jikaikas
    @jikaikas 7 месяцев назад +2

    this is amazing and everyone clearly relates

  • @Sindoku
    @Sindoku 6 месяцев назад

    I love hearing tech war stories like this.

  • @dinushkam2444
    @dinushkam2444 7 месяцев назад +3

    if the data driven and analytical comes then same issue. we had a system that cost 750K per year 1% cpu usage and nobody was willing to press the button.

  • @asdfghyter
    @asdfghyter 7 месяцев назад +6

    11:56 i’m surprised that pipefail isn’t enabled by default in ci systems. those are the lines of code you *should* blindly copy and put at the top of all your bash scripts even if you don’t understand them. (i mean it’s better to actually learn what they do, but blindly copying them is better than not having them, since they prevent footguns like that bug)

    • @Crow-EH
      @Crow-EH 7 месяцев назад +1

      I doubt someone teeing stderr into stdout would know to set pipefail first, when even putting it back into stderr would have been enough already. Or maybe using PIPESTATUS if an non 0 is required by your CI runner (most already mark failure on stderr).
      Plus pipefail will actually mark a failure when you do want something to fail silently (like a grep piped into smth as an optional operation or something, idk).
      Well, I'm saying that but it turns out I do have pipefail set in my runners config directly, and prefer explicit || true when needed.

    • @asdfghyter
      @asdfghyter 7 месяцев назад +1

      @@Crow-EH yeah, that’s why i expected it to be the default, so users wouldn’t need to be aware of it. the reverse bug caused by pipefail is generally easier to debug, since it’s an unexpected failure instead of an unexpected success. you can always unset it manually if you don’t like it. as to my second point, it’s actually kind of something i’ve done: blindly copying the `set -pipefail` etc that i’ve seen at the top of bash scripts, so it’s possible for people to accidentally solve such issues through blind copypasting of other scripts as templates

  • @ltxr9973
    @ltxr9973 3 месяца назад

    The secret group chat that runs the company? Yeah that's probably a legitimate reimplementation of the cigarette break for remote work.

  • @claudiusraphael9423
    @claudiusraphael9423 7 месяцев назад

    Made my day, thanks for sharing!

  • @pStabs
    @pStabs 2 месяца назад

    I'm on team TJ here. The entire reason I got into software development was because of Access. It was the first time I could actually see what the data was like back there behind the screen. It can do some pretty impressive things. I am going to have a hard time not calling it PowerBy now though ha! (I'm currently a Business Intelligence Engineer)

  • @DavidMorash
    @DavidMorash 2 месяца назад

    I wrote a cost accounting system for a company in Access based on the idea of linked spreadsheets. It was a nightmare but worked. Reduced month end accounting to hours versus days and allowed them to reduce their accounting staff. It was a bad idea on multiple fronts.

  • @aloe2454
    @aloe2454 2 месяца назад

    I used to maintain an old Java system. We had several random print() functions, yes empty print functions, throughout this monolithic 8,000 line main file. When I deleted them, weird things started happening, timings were off, race conditions were triggered, etc. So if I see some weird stuff on a codebase, let alone one that's older than me, I'm leaving it there because I'm sure there's a method to that madness. No methodless madness can survive for that much time.

  • @kollinpoindexter8859
    @kollinpoindexter8859 2 месяца назад

    I worked with a big firm for a media project. They told u they could match roughly 25% of impressions to sales so we needed to divide by .25 for all our reported sales. we had such low numbers that we almost lost vendors. When asked to look at it we found that of the 2 main channels we advertised through one of them was roughly a 25% match rate, but the other was only 5%. After the change to 5% we roughly 1.8x our total sales. Vendors that only did campaigns through the channel with the 5% match rate saw 5x increases in their sales. I am not sure we will be working with the consulting company again.

  • @kim15742
    @kim15742 6 месяцев назад

    This has been the best laught I‘ve had in ages

  • @thatgamingfreak
    @thatgamingfreak 2 месяца назад

    Thank god we just send our basic powerpoint slides to one of the managers who makes them very pretty for management.

  • @rujn
    @rujn 2 месяца назад

    These vids bring so me much validation to this field. Clever engineers just drown in the politics of business. Going rogue is how I have delivered my best tools. My motto is it is better to ask for forgiveness than permission. Design by committee is a sure fire way to stall a project and kill it dead.
    I believe in a balance. Have a week to crush bugs and grind on the main product. Then the next week, explore/tinker/play with ideas and try new things. I find the youngest engineers often have the best ideas due to fresh eyes and no legacy cruft in their heads.

  • @emptystuff1593
    @emptystuff1593 7 месяцев назад

    I'm using tee for exactly this reason and without any issue :-/. Though, I'm using also using a FIFO to not hide the status for the build command (because piping into tee would never fail).

    • @FryuniGamer
      @FryuniGamer 7 месяцев назад +3

      Or just using `set -eo pipefail`

  • @TreaBeardGaming
    @TreaBeardGaming 7 месяцев назад +1

    Spending $500k on Agile discovery, only to deny the $50 solution.

  • @KiterTMK
    @KiterTMK 6 месяцев назад

    This video is like the Superbowl. Commentary and commercials interrupted by the actual thing you are interested in.

  • @espen394
    @espen394 3 месяца назад

    Loved the wheel of time sidebar 😂

  • @ozkifovxvypyvp3574
    @ozkifovxvypyvp3574 7 месяцев назад +1

    Now you need the opposite of I accidentally lost half a million dollars for my company. Couple of those stories out there.

  • @liquidcode1704
    @liquidcode1704 7 месяцев назад +1

    "WOW, YOU CAN SEE WHAT'S IN THERE" 🤣😂🤣

  • @hippocleides7105
    @hippocleides7105 7 месяцев назад

    Yeah this much more common than people think at larger organizations. MBAs come in, "lower expenses" (salaries) then have to "manage" the worse human capital to get an inch of productivity.
    Processes that ensure everyone is at least 10% productive while kneecapping and ensuring nobody is more than 20% productive without breaking rules isnt a recipe for success in the modern technical world.

  • @DoctorSoulis
    @DoctorSoulis 7 месяцев назад +2

    As a PDF engineer I can confirm that I'm losing my mind.

  • @letopizdetz
    @letopizdetz 4 месяца назад

    I hope it's not the one I'm thinking of but I literally know of a company that had and idle timer set in by a dev when they were a little startup and had limited resources. Turns out nobody removed it until recently.

  • @porky1118
    @porky1118 7 месяцев назад +1

    21:30 If an intern is able to do something, I haven't been able to, I'm just happy about it. It's something I (hopefully) don't need to do anymore.
    I as the only (almost) full time programmer at my company, so maybe I should feel bad when an intern accomplishes something, I haven't been able to, maybe for weeks.

  • @albyx
    @albyx 6 месяцев назад

    6:50 I think what the letter is trying to say that Employees who were not a "fit" were given impossible tasks with the intention of letting them go. But yeah, hard to say with actual clarity.

  • @arrangemonk
    @arrangemonk 4 месяца назад

    i had an idea for increasing productivity from "do the thing takes 3 months" to "do the thing takes 2 weeks" but it would take 2 months to implement tops. it was 8 years ago i brought up the idea and its still not implemented, and we did the thing 27 times since then, so a poor soul (not me) spent 75% of the last 8 years doing the thing, instead of doing it in one

  • @pencilcheck
    @pencilcheck 7 месяцев назад

    Same in the company I left as well, I also don’t really see the product being used after years of development. The directors all were fired and moved lol

  • @zami001001
    @zami001001 6 месяцев назад

    I think we just discovered how the mythical 10x engineer is born.

  • @sproccoli
    @sproccoli 3 месяца назад

    I am with TJ on this one. As a first introduction to databases, access was in fact quite cool.

  • @JohnKerbaugh
    @JohnKerbaugh 7 месяцев назад +1

    I saved a fortune 500 company 650 direct man hours annually by fixing an Excel Macro. Lord only knows the real value.
    This was my first technical task for my second formal software job out of college. By all signs this macro had been in use and expanding for a decade.
    Some of the administrative staff were being required to come in at 6:00 a.m. to pull a report and run a macro. They would then distribute to department heads for daily processing. When it failed it may not be available for hours or until the next day.
    What was this amazing change?
    I instructed the macro to not draw to the screen while it was running. Cut the time for the macro running from 3 hours to 30 minutes. I don't know specifically how much less it crashed. Where how much it was crashing? I just know that that was a major win for the administrators coming in in the morning and the department heads actually getting their report first thing in the morning.

  • @carlynghrafnsson4221
    @carlynghrafnsson4221 6 месяцев назад

    This vid makes me want to go back to Borland Paradox. Dude stares at code... every array[0] is empty, every loop starts w 1. A thousand security checkpoints in the datapipe using way too many frameworks. When you interview, ask for the tech stack. You have to believe in it, not just work the job. Don't just get interviewed, hire your boss.

  • @adhillA97
    @adhillA97 7 месяцев назад +2

    You know who still uses COVID as an excuse? The US Embassy in London. It's _still_ fucking impossible to get appointments without getting up at the crack of dawn every day in the hopes of magically catching the one morning when they release appointments and snagging one before they go.

    • @varianbohling251
      @varianbohling251 6 месяцев назад

      They probably just need more tax dollars printed out of the ether.

  • @levizwannah
    @levizwannah 6 месяцев назад +1

    I felt the big consulting firm joke.

  • @lorscarbonferrite6964
    @lorscarbonferrite6964 2 месяца назад

    At this point I'm strongly considering shelling out the 1 grand needed to become a Certified Scrum Master™ just so I can buzzword my way into a project management role and let a dev team actually do things instead of spending most of their time on meetings and red tape.

  • @notapplicable7292
    @notapplicable7292 7 месяцев назад

    This man pulls no punches and describes most of my job is far too much detail...

  • @mattisovereighteen
    @mattisovereighteen 4 месяца назад

    How do you always select text without the leading or trailing char every time?

  • @engageintellect
    @engageintellect 7 месяцев назад +1

    Let’s set date for a date to have a pre-meeting for the meeting.

  • @nikolaosstavrou5327
    @nikolaosstavrou5327 4 месяца назад

    and imagine some decades ago astronauts went to space and back using asembly in their computers.

  • @ErazerPT
    @ErazerPT 7 месяцев назад +1

    This is painful because just about everybody has been there. Maybe not to the scale, but to the point. It's like it's either "no planning" or "no doing" with nothing in between. In a way, it's like those products where you build layer upon layer of "functionality" by adding MORE technical debt at each layer. Then, at some point, some team gets a greenfield initiative to redo the product from the ground up. Year(s) later it comes out and it's amazing, 20x performance. Then you look at the team and they have that sad look. So you dig into the docs and find out the 20x comes from just substituting a base layer that everyone knew was piss poor. It could be 100x, but it's 20x because they had to build THE EXACT same shit that was built before, because the people calling the shots were still calling the shots and they had to have what they wanted to have.Then you grep the sad looks...

    • @LudicityHackernews
      @LudicityHackernews 7 месяцев назад +1

      I am the author, and my guy, this hurts me to read because it's so true. The thing that upset me about this platform is that it actually was a green field - so it was a total own goal - but even if they did it correctly, the enterprise databases they were pulling information from are just worthless because of what you described above.

    • @ErazerPT
      @ErazerPT 7 месяцев назад

      @@LudicityHackernews Imagine a table that is basically a key-value store. Imagine it was supposed to hold configurations. Then someone said html templates are configurations. Then someone said, why not put json in there, and have configurations inside configurations. Then... you know where it's going. Sad part if that it's not even the worst part. Hell is real ;)
      Last thing i "hinted" at a team doing greenfield was "you know they will if they can, so either you put the roadblocks in NOW or they'll shot themselves in the foot sooner rather than later".

  • @Axl124124
    @Axl124124 7 месяцев назад +7

    lol I kid you not decisions by committee is how my company operates for every data transformation we do. It is a joke.

    • @markm1514
      @markm1514 7 месяцев назад +1

      I'd rather shake an 8-ball to be perfectly honest.

  • @jelliott3604
    @jelliott3604 3 месяца назад

    A lot of "success" in management in a corporate is measured by how big your team and your budget are.
    From this point of view this, halving your spend is a disaster

  • @gabecodina
    @gabecodina 5 месяцев назад

    The thing is that somebody did need to get yeeted under a bus. Whoever set the thing up, and chose or accepted a default of ten minutes. Maybe it was the service provider using some particularly dodgy default setup.

  • @snorman1911
    @snorman1911 5 месяцев назад

    "Access is actually kinda cool"
    "What was that, TJ??"

  • @sighofman
    @sighofman 2 месяца назад

    Hi five "seeing" access db. It was pretty cool in like 1997 for an Office product.