Serverless Was A Mistake | Prime Reacts
HTML-код
- Опубликовано: 7 фев 2025
- Recorded live on twitch, GET IN
/ theprimeagen
Original • Serverless was a big m...
YOU KNOW WHO THIS MAN IS!! / @fireship
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
"Self hosted vs serverless" and "monolith vs microservices" are independent axes. If you put your entire application logic into a single container that you run using App Runner, that's a serverless monolith. If you build your app as a fleet of microservice containers but you run them on your own hardware, that's self-hosted microservices.
Just goes to show how much misunderstanding/confusion is out there in the backend among even experienced devs
And there are cost calculations in which that apprunner bullshit is more expensive that IaaS. architects need to be really kicked in their balls for not considering cost calculators while outlining architecture
EC2 Cost Calculation:
Determine the cost per hour for the selected EC2 instance type. Let's assume it is $0.05 per hour.
Multiply the cost per hour by the number of hours in a month (720 hours):
$0.05/hour * 720 hours = $36
App Runner Cost Calculation:
Determine the App Runner cost per hour. Let's assume it is $0.07 per hour.
Multiply the cost per hour by the number of hours in a month (720 hours):
$0.07/hour * 720 hours = $50.40
Agreed
Exactly! Drives me nuts. The Prime Video blog shows even Prime Video devs don't know what they're talking about
He also seems to think that serverless is the only way to have managed hosting. As in, if you have dedicated servers, you have to install/manage them yourself. But there are hosts that do managed dedicated servers with 24/7 support. They can even use puppet, or similar software, so that you can have phoenix-servers without dealing with docker.
Tom is a genius, he could've scaled it much better.
he could have scaled it diagonally
He would use svn to share the pictures across the workers
who tf is tom?
@zweite Konto Tom is a genius
@@zweitekonto9654 a guy from an article on stream - I can’t recall it all but something like he made a tool called JDSL and a guy added a comment to it and it broke the whole thing
"that doesn't look like a high school project" 💀 I'm a frontend engineer at Amazon and this is so true. There is no culture of frontend development at Amazon
What's the point? If something is working and brings profit (a lot), why change it?
blockbuster circa 2002: "we're bringing in profit (a lot), why change it?"
Hey fellow FEE-I don’t think Amazon has ever had much of a front end culture though I see efforts to change it. (Like hiring people with our job title…) and if the amount of time and discussion it takes to make UI design changes I’ve seen is typical it’s going to take a while…
Then again I’m not enamored of the UI for Google either. I think you need a design-first ethos like Apple really to have the exceptional UI/UX-and even that has its pitfalls, like when Jony Ive ran unchecked for years.
That tells me that there's no FE leads at Amazon, you need someone to take the lead, to assume responsability, to go up to face people, make themselves public figures within the company to challenge standing paradigms, and it's invariably a problem if you're all just the kind of people who are content in your home office coding whatever they tell you to code.
Nothing changes this way, to change things you need people making themselves known and pushing for changes.
Admitedly in any given company, very few people (i'd even say 1 in a million) do this because it's an enormous pain in the rear in a big corporation, you'll get a lot of pushback, you'll need to join a lot of extra meetings, you'll need to take on a lot of extra accountability, all while researching and studying to make sure the next architect or designer you get in an argument with doesn't humiliate you publicly on the company chat app lol (and they will try because they will resent you if you're pushing back against their status quo)
"But why don't the architects do that?" From my experience in a medium-large (300+ devs) corporation it's because they were all drafted from the Backend development circles and don't care about FE their idea of architecture is much more focused on the links between BE microservices (which they adore and create thousands of) and various hosting and caching services, FE is always an afterthought to them
Most FE architecture guidance i ever got was that they wanted microfrontends we needed to integrate with s then they wanted webcomponents instead, then they wanted us to bundle every micro frontend in a monorepo, now they want us to switch to using Github's CI instead of Jenkins etc it's a confusing mess.
But there's always a group or taskforce that's seen as the "core", that might be the people in charge of your css framework libraries, might be people in charge of some core packages that everyone uses, a good idea if you want to change things is to try and get into those taskforces or teams, it's easier to change things from inside those teams as you'll have a bigger voice and more people will listen. But overall it's awfully political to change things in big companies and very few people are cut out to do that or even care enough to go through the hassle
I imagine because the market Amazon appeals to through AWS isn't concerned with the frontend, rather just the functionality.
Problem was not serverless and not even microservices.
Problem was bad design
someone from amazon came to twitch and suggested this moneyburning design, i'm sure
I think if you just run your micro services on Subversion, that would solve all your problems. Also, Tom is a genius.
Hey, I know you know, but not everybody knows how much of a genius Tom is
@@ThePrimeTimeagen watching you without facecam would be funny
Lmao at DHH bit
It kinda makes sense, you don't pay the internet bill if you don't transfer anything over the internet.
Pretty soon we'll all be programming in assembly again.
Already do 🤣
@@poopoo-dk4hu name checks out
We go full circle every couple of years lmao
And we will claim that it is incredible innovation and game changer
and we'll be better off for it
The key of designing system that could scale horizontally is Stateless. Monolith could scale horizontally as long as the instances are stateless. The answer was never always microservice to scale horizontally.
Amazon needs to hire UX designers and front-end developers. Amazon's website was built by a backend developer 😅
They did, they are working on QoS monitoring.
@@mennol3885 lol nice
Word!
Then it would just look like youtube but also run like shit.
not to dump on UX devs, but most of them made UX worse of basically every website I know of which had a decently big-ish rework in the last 5 years
Tom is a genius
"You got to go Because TOM IS A GENIUS!"
Prime "How do you spell gene-i-o-s"
Tom "you're fired!"
@@TheThunder005 Akshually you are wrong
TOM IS A GENIUS!
Who is tom? Serious question
@@fallenIights Tom is a genius!
@@TechBuddy_ tom is a gay
our devs recently accrued 200k spend on cloudwatch, over weekend, after Friday dev branch change because they looped lambda calling itself and their code had verbose logging. with 10mil monthly spend it is drop in a bucket but for small org that's be end of company. most dinosaurs of development don't understand cloud infra and cost pillar well enough to be doing serverless
In practice AWS will likely forgive that
@@adambickford8720 just like with cryptominers it is on account owners to pay attention to cost. it is a legit pay as you go storage usage so there is not wiggle room like with let's say ddos refund via shield advanced.
@@yuriib5483 You're vastly underestimating how willing amazon is to wiggle, but that's because you're stuck thinking officiality is real. I'm 100% certain that they would refund the company if they asked. But nobody asks, because they think like you.
@@BusinessWolf1 15 mil a month bill, shaved off another 400k by pointing out bug in RDS pricing model, we did ask and it is leging cloudwatch storage usage. THey might be lenient at legit hyper charge mistake by some tiny startup so that they don't go under. with big orgs? not so fast
“How did you not just run basic numbers here?”
Because AWS does their best to hide a lot of their costs. Their pricing calculator is a joke and AWS Sales reps don’t help you with any of that shit.
Serverless is ok depending on the use case, not for everything. I’ve done migrations and switched from a 3500 USD/mo bill using ec2, and RDS to less than 100USD/mo using nextjs running on lambdas + DynamoDB + Cloudfront
I’m using lambda and dynamo to run the backend of my game, for player profiles and stats and stuff, and I couldn’t be luckier that this stuff is available. So far I haven’t paid a dollar or spent more than an hour setting anything up. Huge improvement over 15 years ago.
so you went from relational to nosql? I would not call this a migration. this is a full replatforming. lambda + dynamodb + cloudfrom +route53 stack also assumes you have a modest usage. I is very easy to run away from your cost projections if you have bad actors hammering your app. hope you got that shield advanced
@@mattmurphy7030 lol the latency is going to be insane, lamba for a game?
@@dandogamer read it again
you are right, we rewrote all the app, and avoided a DevOps team too. You are right regarding bad actors too, we implemented WAF and api rate limits. And now identified silent errors on the client side using Sentry.
11:59 Goddamn years following Primeagem and never ever imagined that he could be a man of culture
Tom can fix serverless
Tom will JDSL the shit out of it.
The world is not ready for serverless JDSL.
3:35 that laughter is legendary
"Someone call a data scientist in here immediately." Said no one trying to get something important done...
Im one and youre right
Data engineer would be the correct role here
"You'll compute nothing and you'll be happy" - LOL
ugh aws lambdas are just ugh.
i recently joined a company that has a micro service’s architecture built with around 1000+ lambdas. each doing different things.
for each, we have asked aws to allow upto 20k concurrent requests. now we are supposed to handle over 40k as our company’s growing. and the time AWS takes to scale the lambda is over 30minutes!?? we are switching to kubernetes jobs as an alternative.
and now i’ve to rewrite code for 1k lambdas 🙂
Job security 👍?
not much to rewrite even if you're running something exotic like .net3.1, or nodejs12. most likely will just run out of the box. if lambda runs it then there is most likely container image for that too. we got 140k lambdas and some developers have died during 'Rona and no one got time to read the code, and some were edited 5 years ago, now that's some danger zone
The main point is that Amazon Prime was doing something thats inherently impossible with Serverless and thats how they save 90% by moving away from it.
I thought the main point was that they coupled two microservices to share a data store
They could have saved the 90% without even moving away from serverless.
I used to write algorithms to do something kind of similar. We would turn videos into frames and check them for big sections of similar colors and turn them into matrixes to check how close they were to solid color squares (common video encoding error artifacts) in order to validate if the movie could be sent do Netflix, Google Play, Apple TV, etc. It ATE our resources like HELL!
We had dedicated HP workstations with two quadro boards each just to do that work (and a few other validations). It was HEAVY!
How dare you question the genius of Tom. He literally built jay diesel by himself with executable comments at runtime.
JDSL definitely could’ve handled it. Tom is a genius! Could’ve deployed the whole thing in comments.
My work uses one single dedicated server (well 2, we have a mirror fall back) costing £120pm each, and it's running a collection of dockerised microservices. it's cheap, cheerful, low maintenance and scalable vertically and horizontally. Currently handling around 350k requests per hour, with file uploads, file processing including video etc. With Cloudflare caching for static files, which they do for free, it's so easy and cheap and "future proof" to go with an SOA approach. Our system on AWS or edge would cost £thousands per month... embrace the dedicated server
3:02 "And still be unable to make a wesite that doesn't look like a high school project"
I also wonder if they even have a UI designer.
amazing that AWS didn't even follow their own recommendations for serverless applications. A 1) long-running process with 2) Consistently high workloads? Nothing can go wrong here!
AWS doesn't coordinate efforts across the company incredibly well from what I can tell. They do it much better than virtually every other company out there, but it's still not that great.
Amazon also gets an AWS solutions architect to help them figure stuff out. It was sad to read this article because it's strange that they ended up here.
“It went from crazy-ass to… less crazy-ass”
It went from crazy-ass to crazy-elbow
04:02 mr vim, when you hood drop off, was like a person loosing the ears, hair, humanity... this hood is just part of you. now I understand why superman could disguise to clark kent by putting a glass
The best solution is to adopt a hybrid approach that combines serverless and monolithic architectures. Critical components benefit from serverless scalability, while less dynamic parts are consolidated into a monolith. This maximizes resource utilization, cost efficiency, and flexibility in scaling. Continuous monitoring and optimization ensure a cost-effective and scalable solution.
"for some reason it's like it's 10x cheaper to not have 10 streams."
lol
lmao, even
The funnies thing about Amazon is that it feels like a clueless booksalesman is trying to sell me a bunch of stuff when I use their site. "Oh, you bought a bed last week, would you fancy buying this another bed as well?" "I can see that you're searching for RX 7900 XT... How about these RTX 3050's instead?" It's a mystery to me how did they get so big.
LOL Never understood why people are so excited about those recommendation engine. It really sucks!
Serverless is a gamble, but Anakin memes are forever
I don’t get why everyone is up in arms about this. The promise was never that you could fragment your application into infinitely small pieces and it would magically be better. There are certain parts of an application that cannot efficiently be divided up into micro services, and there are others that are perfect for it. I don’t understand what the revelation is here.
This stream duplication business is no surprise when you see "ML" in their workflow. Probably, some data scientists planed out this mess.
Prime issue was S3 bucket read / write cost and not serverless cost.
They spent so much time asking if they could do it instead of if they should do it.
Serverless only makes sense in case when your usage pattern is extremely uneven. One example is when you have a computation heavy report to do once per month. If just you rent servers to do it they sit unused for 29/30 days. Unless you are using them for something else in the meantime (development/CI etc) then serverless is a valid solution for this. On the other hand If you just have web service in 90% of cases avoiding serverless would actually save you money.
3:09 What 1 programmer can do in 1 week, 2 programmers can do in 2 weeks. Pay a person twice the money, they'll probably take twice as long to do something.
Oh, so that's what serverless means. I thought it meant there were no server, like a normal client side only program
uh... WHAT.
Unless you're deploying your jdsl on the edge, you're an imposter.
Amazon’s websites look like shit for the same reason that the Dollar store looks like shit. I Tricks people to think that everything is cheap
3:30 Amazon prime is what makes them successful. Next day shipping for no added costs. Billions of products at good prices.
not really, cheap chinese garbage and being one of the very first on market. now practically every store has some options of free shipping
Tom is a Genius
"This is an unironic real picture of Netflix" made me feel so sorry for him.
it's a far above my hat show, but I made notes and am going to think about it in my head. I send you a message a while back about how you inspired me to change careers but something is holding me back it feels, so I resuscribed to your channel :)
I think you are wrong about what they are doing at amazon: they re-stream streams from _clients_ to qa service (or do snapshots at clients and upload, not sure). And they do it not for every client, just for some small subset. So no additional load on streaming servers.
NEVER SAY NEVER. Serverless CAN BE good. It depends on the data interdependence and flows, storage, etc. Additionally, there are many business inputs to consider: resource availability, on-premise resource utilization, etc. Yes, you can save money by self-hosting...but...the money is paid elsewhere. Reminds me of a search challenge I had 20 years ago. The query had 7 functions to transform the returned data. How to speed it up? Pre-compute the transformations, e.g. denormalize and prep asynchronously. The result: 2 min queries became 4 sec. (But hey, that was 20 years ago!)
At a certain point in scale self-hosting cloud serverices, like serverless hosting in order to reduce cloud cost, but scale out to the cloud as needed.
i love how tom keeps coming back into these videos. I bet its the same Tom from myspace :P
Netflix might have a prettier website than Amazon but come on man, Netflix's ui is so minimally useful you should know better than to throw them kinds of stones
Chat sharing a single braincell
AYAYAYA
Most people in tech just do it and don't worry about the cost. The business also know better and tell you to do it if you do ask. It is always interesting to watch the business buck pass their solution around.
One of my senior dev colleagues kept telling me that "costs are unpredictable" and "decisions shouldn't be made with costs in mind"
The next project we build now cost the company 17 cents per month in serverless stuff.
That "Ruby on Rails" programming without types is vastly superior bit. ROFL. I'd offer to give them my footgun, but it's clear they have more than I do.
Modular monolith is by far the cheapest, simplest and handles partitioning very well. There is no style which can replicate that to any degree. Service based architecture does come close with more upsides like agility and fault tolerance, but at a sacrifice of cost and simplicity.
Microservice still a unicorn architecture with many upsides but there is event driven, space based, including service based. None of them share the atrocious cost effectiveness of microservices.
I think Cloudflare Workers is a great example of lightweight and cheap “functions”.
I’m convinced Amazon makes their AWS website and documentation look terrible so that people have to pay for their support service
Our company saved something like $10,000 a year just by buying an old 2U Supermicro Xeon server and spending a couple hundred bucks a month to co-locate at an ISP. We still use some AWS stuff, but only as overflow scaling.
It depends on if you get large spikes in traffic. You'd need to buy, load balance and maintain more servers even though they're mostly not in use. That is the use case for cloud and lambda. - like you said.
Stream and S3 - how does that go together? Using S3 for realtime applications seams to me like taking aeroplane to go to shopping mall.
I diiiiiiiiiiiiied at that Amazon rant about being like oh cool I want to watch Conan. Wait I have to buy it? Screams externally and internally*
Broh, they didn't use AWS Calculator
Amazon builds sites like that so customers think they are buying cheaper stuff from those sites. It IS their success strategy.
Is it possible that AWS had a mole in Prime Video who was attempting to run up the AWS bill?
Serverless is a gamble: will you make for up the rip-off $/GFLOP of compute with your actual duty cycle? 10x the cost but 1/20th the usage is a win. Of course aws knows that by the time you hit that breakpoint you're probably locked-in :evil-pinky:
Remember, aws is renting you the same hardware in Virginia either way.
It's like the clean-codification of deployment. They wrote a load of over-encapsulated functions and tied them all together.... over a network 🙃
It wasn't even a network. It was a network drive.
6:05 The Lord's answer to all of our problems 😂
Who is Tom? We talking about Tom from MySpace?
It only costs less at the beginning, there are diminishing returns to any cloud service. It's just the nature of 'middle man' economics. Cloud services (all of them) are great if you're a startup and can't afford 100k worth of servers just to put in 2 locations that will cost another $1k a month to operate. But once you have scaled cloud services up to $1k a month, or beyond, it's time to start putting that money towards financing the ownership of those servers. Or, better yet, by the time you've scaled to $1k a month cloud services, maybe you've already made 100k???
I know of a single game server that hosts around 1k users at any given time being hosted on AWS (bare metal), and it's something like $10-20k per month. And they don't see a problem with that because they're all devs and don't want to deal with servers. Hire me for $4k a month and finance some servers! We'll both pay for ourselves!
Also, anyone familiar with the game Star Citizen will know about all the problems they've had the last few months. They are basically refusing to scale their services and just deal with poor performance. But, here's the kicker, they are 100% locked into AWS because of early sponsorships and committed discounts. They are probably spending over $100k a month (my personal estimate). They could be putting redundant servers in a new region every month for that price and then pay the $10k a month to keep 10+ locations running (plus pay 2 people $100k a year to manage them). And this is a fully funded operation with 10's of millions in cash...
Cloud is insanely expensive, costs so much more than standard VM servers.
You're just painting yourself into a proprietary corner for some sudden unpredictable scaling scenario that'll never happen.
Title should've been: "How I solved Amazon Prime Video problems in 45s".
Tom is a genius, he would've listened to Moonlight Shadow
Netflix’s architecture with 1000+ microservices reminds me of my son’s room with clothes on the floor and trash that missed the can. Craaap everywhere. A giant mess. I predict shardservices will become the next big thing. No really, need to move to macroservices. Micro is too small
Hearing J.Diesel gives me Vietnam War flashbacks, and I'm not even involved.
Prime, agree trying to use the AWS website is like geocities level sadness.
I swear it said my ES2 instance didn’t exist until connected through SSH. Kept getting these emails from AWS about my services but every time I logged in, they weren’t available to even evaluate..
AWS interface is almost as bad as Pro Tools.. that’s right Avid.. design something that looks like it was made in reality and design trends of now not the 90s.
the design of amazon is actually pretty genius in that it keeps their userbase happy and manages to completely disregard being obnoxious to the webdev and web design pros
most other companies cannot keep the galaxy-brained opinions of industry insiders on user interfaces at bay, they all eventually surrender to their siren songs
😂 Amazon streamers think that AWS costs don't exist for them. Thus, they can outprice other businesses. That's how they dominate.
Serverless is a service implementation detail.
Not a system architecture design.
OFFER SHIRTS “Tom’s a genius.” -Primatroll
The big takeaway is this, great tools with poor design will result in bad products. Serverless isn't a magic bullet.
I designed and implemented a serverless microservice that eliminated a 24-core EC2 instance. It saved us 96% of costs due to its extremely spiky workload.
But I still run ECS instead of Lambda for my highest volume APIs because Lambda isn't the right tool for it. We did the math and it costs more on Lambda.
We need to pick the right tool for the job. No blind loyalty.
What did you do?
I almost always prefer self-hosted monoliths. They’re almost always simpler to maintain and given enough traffic are cheaper to run.
Though, it’s possible I’m biased because the micro service’s architectures I’ve seen have always been inefficient piles of garbage.
I very much agree with what you said, Amazon's website has never been attractive, as well as eBay's. They seem to be websites from the 90's.
still from the 90s afaict
Serverless does not mean you have to go cloud. You can rent a $20/month VPS that already sits on a cloud. Its cheap as h311.
The cloud was meant for companies that needed a short-term fix for fast NIC, CPU, GPU, or APU or you need encryption based VMs. The cloud was meant for companies that did not want to park pizza boxes at colocations. Sometimes you have to rent a fast network for burst traffic. Netflix is a great example: they change so much and add so much content, that they need short burst of traffic and scheduling changes like every week. They need cloud for the short term stuff. But the heavy lifting is done by beefy racks in colocations. I'm over simplifying, but everything else can run on a VPS. Most VPS are already on the cloud which gives them redundancy and the ability to take snapshots.
You need VMs when you need full os encryption+isolation and to create and tear-down instances because you have a army of employees or users that need their own computer that moves to different locations. Like a hospital or government or military. It amazes me that we got here.
What the cloud did do, was jack up prices and the fools followed (which is basically middle class). A down side to the cloud is that your company and efforts never build up equity like real-estate and hard wire; instead of going cloud you could build up your own network across the land.
At the end there. That sounds like a national anthem from The Critc. Eee pii tii pii pee pee pou pii
Wait, wasn't edge computing the new on-prem? In a hybrid model?
3:41 i need a time selection for this!
The runtime executes comments like normal code!
"Beyond stupid" - Linus Torvalds about AWS engineers...
12:00 reject nightcore-chan and embrace drum and bass waifu
I come back here to watch 3:40 when i'm having a bad day
Hey prime! im curios, you use vim as editor, do you ever use a debugger? what debugger?
3:01 did no one show these degens Bootstrap?
on the edge compute, CDN will happily handle this workload, and you don't even need the scripting system to serve static Content or dynamic content with unknown refresh time!
2:46, ok I laughed so hard at this. :D
Their UI SUCKS. Be it Amazon shopping, or AWS. Really lol.
I watched Conan the Barbarian earlier this month. The dialog is awful, the editing makes it impossible to follow, but it is really hard not to love that movie. Sweaty Arnold telling Krom he can go to hell 10/10
I'm blown away by this content. I had the privilege of reading something similar, and it was truly incredible. "Mastering AWS: A Software Engineers Guide" by Nathan Vale
3:42 Prime having an episode. 😂
Tom would’ve piped all the video streams into SVN instead of using S3
I’m going to re-write all of libavcodec in typescript, Tom’s a genius
If we are referring to Amazon using their own servers, even with the stupid term server less... they never had server less because it's their OWN servers
who is this strange man posting videos on theprimagens youtube channel? he didnt tell us what his name is so ill guess well never know