Scaling Up Prime Video | Prime Reacts
HTML-код
- Опубликовано: 7 фев 2025
- Recorded live on twitch, GET IN
/ theprimeagen
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
![I.N "HALLUCINATION" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/n5B5q1Hwt_U/mqdefault.jpg)
![Seungmin "그렇게, 천천히, 우리(As we are)" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/kAzmhLHePqU/mqdefault.jpg)
Microservices are overrated, I use nano-services. Every single function and method is serverless and is elastic and beanstalky... cause Bezos needs more yachts that are... BLAZINGLY FAST! ♥
Instead of a monolithic Yacht, I use a set of microboats, tied together with cat 6 wiring. It's just as fast, and much more resilient to damage/low cost.
I use Yachtless
@@Chris-se3nc Ok, I think I'm going back to the traditional method of travelling on water.
Walking...
Blazingly fast yachts for Bezos or blazinly fast apps? Choose wisely.
@@LowIiet his yachts ... you know why? Cause they don't *rust* 😜
So you write a micro service to hold the lightbulb and I'll use step functions to rotate the house. It'll scale horizontally, only limited by your budget.
you're hired :D
The main thing I got from this is even AWS engineers making things for AWS can’t even figure out how they would be billed by AWS haha.
Ex-AWS engineer...this is wildly accurate. Not to make "us" (former) seem fully incompetent, but unless you're a senior engineer (L6), but mostly higher (L7) and above you aren't connected to the financials of a newer service. I didn't fully understand the billings of AWS internally as an L5, however I intuitively knew from an engineering design when AWS resources were being leveraged in a way that seemed excessive. People thinking you needed a database to store/retrieve data that could easily fit into MBs of memory on servers with GBs. It was a common mistake to see another entire AWS service as a wheel to not re:Invent (lol, see what I did there?), when in reality it was far cheaper and very reasonably doable within the application.
Management "We need to get view counts up."
Developer "You got it boss."
Client: "Video is broken, yo"
Server: "I DON'T REMEMBER ASKING YOU A GOD DAMN THING!"
It's worth noting that they're not doing this for VOD, it's for their Live Streams (such as sports). So it's to detect issues with the video stream coming into the Prime services and understanding the various realtime transcodes that are happening, not necessarily the individual customers UI or experience.
some how this doesn't make me feel better...
I wonder what a (former) twitch dev would have to say about Amazon's video streaming approach 🤔
@@ThePrimeTimeagen I mean netflix died trying to livestream a moderately popular show...
@@guidoderam Good point, do these teams talk to each other?
Yeah it feels that the primeagen was a little too much reactive on this one 😂
You got it wrong. They are not running this for every user, but for every live stream. It's like one inspection per live stream. Not one per user
Am I missing something? Because that wasn't at all obvious from the article
This makes much more sense
“At Prime Video, we offer thousands of live streams to our customers. To ensure that customers seamlessly receive content, Prime Video set up a tool to monitor every stream viewed by customers”
first 2 sentences of the article
yeah
this makes a lot more sense in that context
thanks @titbarros for pointing this out
I was waiting for him to realise this. Still waiting. Of course it’s not feasible to do this for every users’ stream. You could also use it for post transcoding QA.
Pime video 2030:
"We have sent a developer to every one of our viewers so they can pause the playback at every frame and report back any issues immediately. We care about the viewing quality of our services"
might be less expensive tbh
I really don't agree with the article calling that a monolith, it's still a video processing microservice. They just packed all the computation inside one server instead of the serverless madness. But since it doesn't include all the business logic for the final product it can't be called a monolith
Agreed
Yeah, there's monolithic and then there's the spaghetti monster monolith .. the one were all infra and apps are in a single CVS repo and said app infra is coupled to the deployment so updates to deploys breaks the 10 other teams committing to the repo (and branching in CSV .. kill me now) ... ah ... the early 2000's .. the cowboy days of the intra-web
I made this exact same conclusion in a similar video about this blogpost. Nothing about this is a "monolith", it's just a better optimized microservice.
In other words "serverless" != "microservices".
@@LimitedWard yeah that’s the other funny thing. Monolith vs microservice is not the same as serverful vs serverless.
I could have a “monolith” deployed to a serverless environment like lambda or workers.
Exactly! I think that article was a publicity stunt to make us talk about prime videos. Lol
Remember when streaming was done on UDP where you don't really bother to confirm that the packet made it?
imagine having one of the worlds thickest infra but they utilize it like this. i mean i respect the courage on the self report
OMG, someone took serious the how to be a -10x engineer guide.
Processing video on Amazon AWS sounds like an operation that would bankrupt any company except Amazon.
WTF, this architecture is batsh*t crazy. Somehow, I get the feeling it was merely an exercise on pushing AWS serverless infrastructure, but honestly anyone with a brain (or at least with moderate web / networking experience) could have told them what they were doing was ridiculous. Just imagine the cost to run this monstrous thing.
Someone said "AWS certified architect type solution" lmaoo
You’re my current revelation. I stumbled across your videos three days ago and I can’t get enough. I’m laughing too hard
:) ty ty
The video title is misleading it should be "Roasting Amazon Prime as a Netflix engineer."
As a millionaire 😂
really great to get a netflix engineer's perspective on this!
its crae crae to be multi-streaming video and audio...
@@ThePrimeTimeagen cræ
To me, this is not a message that serverless / microservices is dead. Frankly, this was an absolutely terrible architecture from the start -- just imagine how many senior engineers that this had to go through: "let's take insanely high throughput, realtime stream data and start a step function invocation (one of the slowest aws tools to exist) for each frame. Let's also add an unnecessary S3 IO call." AND THEY GAVE THE THUMBS UP ON IT!!!
And their v2 is hardly a monolith. It is super isolated in scope and functionality, but I guess since there's three logical components in it, it must be a monolith. That's so, so, so delusional. These people have never worked with real monoliths that are running full web applications with multiple teams merging code to the repo several times a day. What a joke.
You're the content creator I always wanted, keep it up :)
I love that you're uniquely positioned as industry experts on video streaming and reacting to this while conveniently you're a youtube personality
Succintly said!
No wonder big layoffs are happening in prime video org. This is horrible solution from what is written in the blog
"Data transfer in memory" damn, and here I thought pointers were pointless exercises wherein one attempted to segfault as quickly as possible. Now you're telling me they actually have a use case? That use case being, don't rearrange your entire hard drive every 5 seconds?
they just followed the -10x developer way!
It’s funny how AWS team found AWS too expensive 😂
Amazon is too greedy to let you watch the stream on your own, so it sends the stream back to also watch it with you. Prime video: you are never watching it alone!
Lol, Amazon is telling us not to use AWS serverless products.
prime flexing his prowess as CEO (chief edging officer) in this reaction video. Move that detection code to the edge/js/browser!
This happens when you have so much computing power AND you have nothing to do. So, you do the same thing multiple times, in parallel.
So they built the most inefficient, convoluted, overengineered system to solve a problem that basically they created themselves because they don't understand how streaming works?
You have to hand it to them for publicly confessing. Most of these things don't get written up even internally. Too much shame.
The “I work at Netflix btw” is always good
Haha prime acted exactly how i felt reading that article 😂
i just realized that prime is named after competitor of his employer
I don't think it is the matter of monolith vs microservices, they have made other optimisations and used different technologies e.g. VPS vs Lambda, VPS's typically perform better. If they replicated exactly what they have in a monolith this would be a different story. I feel the initial design was just bad design.
agreed. i still think how they are going about it is crae
@@ThePrimeTimeagen Yeah they seem to still be creating bad design. Also calling serverless microservices!? They are not the same thing.
@@ThePrimeTimeagen this is still a Microservice just in one container
Microservices never meant multiple servers or serverless don’t know where this common misconception coming from
It’s an Architectural Design Pattern
They’d till have their microservices same just all in one container
They also stopped hitting Async to Sync (oboist better for Real-Time analysis!
@@ko-Daegu Microservices are a distributed architecture e.g. multiple machines or containers lol
2:30 exactly what i thought. also funny the step away from aws lambda/step functions and say "microservices". what now, lamda or microservices, are we talking about??? sounds like the original architecture was in the "everything is a nail" phase when they discovered their shiny new hammer == lambda step functions
The analysis is on the stream coming INTO prime video from the customer streaming their event THROUGH prime video, no every individual viewer...
The article is more about serverless than microservices.
It's strange to me that the article uses the terms "microservices" and "monolith" to refer to the architecture of this service. It's clearly not a standalone service, just a component of the larger video streaming system - it IS a microservice. They designed some tightly coupled processes, decided to force a bunch of network requests and disk I/O in between, then acted surprised when there were performance problems.
"prime video" would be the perfect name for a parent company of all your channels 😅
*Bezos lawyers assemble!*
I love how the forbidden cursed word”Prime Video” wasn’t bleeped out AT FIRST.
And then after we clarified saying it basically it is as bad of a curse word as Voldemort 😂, and thus we began bleeping Prime Video.
It’s time to unleash CHADStack onto the world on the oldest Monolith in existence, the Mainframe!
Yeah, when I read the article, my first thought wasn’t that they shouldn’t have made this a microservice. I mean, the y obviously shouldn’t have, but there was a bigger issue in the over-engineering of the problem. This doesn’t sound like a solution that was created by an engineer, but like an overzealous PM os something. And I’m actually upset that they just threw it all into ECS and just said “It’s better now!” instead of thinking if there was just a more effective way to solve the problem.
and I still cannot get job at Amazon. these leecode peeps
@@voicevy3210 for real. I’ve been turned down several times by Amazon. And I know I could come up with something better than this lol.
You assume they're capable of finding a more efficient solution to this problem.
@@tokiomutex4148 I thought it would be a safe assumption with Bezos’ money and all.
This sounds like someone higher up made a high-level decision to analyze their image quality, and then the engineers scrambled to get this implemented without every thinking whether it's a good idea and pushing back. Lots of ants doing their job in their own little world without anybody ever reviewing the big picture again.
Would you risk your job to protect a rich guys quarterly bonus? What are you gonna do tell your boss his idea is stupid? No you don’t, you get paid and let your boss take responsibility for his actions.
@@PanosPitsi That's a very toxic work environment then. I wouldn't want to work at such a place.
@@anlumo1 Yes, I definitely wouldn't wanna work at a company, get paid 250k + bonuses and stock options to write and push some code with very little responsibility lol.
@@Btechdom if I hate working there, then yeah… not worth it. If I can get a job that I actually enjoy for 1/4 of that I’d probably be happy….
You are paid to do what you are told little man, if you dont like it then make your own Company or rise to manager.
I use megalith architecture and achieve peak maintainability. Basically all of my code is in a single file.
I believe they meant thousands of streaming sources instead of thousands of concurrent consumers. Anyway, looking good mr Prime
Microcervices are a problem when done by companies that cannot afford Microservices.
Amazon's Prime app on my smart TV is slow as hell - the animation is clunky, it takes a second to respond to ANY remote control input etc. In contrast the Netflix app is really faast and fluid, with smooth animations including zooms etc. It's like Amazon is using a web browser wrapper and Netflix is native.
This article was just a plot to get prime to tell them where they’re falling short
dude, on my LG TV .. each time I want to stream a trailer in prime video, 3/4 of it is delivered in 360p! WTF AMAZON?! :)
Just cause someone is smart enough to design something convoluted doesn't mean it's a smart idea to implement.
Some things have to be separated; such as video transcoding processes from the API layer. The logic within those services should be written in a modular way, so it’s possible to detach it into separate services if necessary.
what's that "atlas" at 3:14
plz someone, google didn't help
Google "texture atlas".
You mentioned not understanding why Ben did not use a 'hold' function as the layer, the problem was the hold function itself.
In other words, Ben's problem was that he was typing faster than his hand could move, his index finger and thumb would work together to hit a letter that was on a layer. But when his intentions were to hit a character that was not on a layer, his fingers would hit the next key *before* his thumb has responded and lifted from the key.
My moving this to a macro, he made it a one shot layer. His intention to hit a key not on a layer would be typed in, even though his thumb was physically pressing down the key and did not lift in time.
incredible, Wrong video.
I'm going to borrow their overengineering_skills in my job_security function
it's about live streams, not VOD, most people didn't seem to notice that
Prime Video engineer: *Slaps roof of new monolith* This bad boy can fit thousands of stream in it
so many thousands
@@ThePrimeTimeagen those thousands of streams are being streamed to millions right?
I was laughing with u there on their design 😂 now I can’t sleep coz Im still asking myself why
How is it that this video was posted 12 hours ago and the netflix techblog got moved to Medium in that time?
Excuse me, why is this not done on video ingest? 10.000 movies at 50GB each is half a petabyte (peanuts for Amazon). Transcode each to lower bitrates on ingest and it would increase storage footprint to about 5x, but there would be no need for realtime shenanigans like this. Because the way they describe it, they do it in realtime while a person is streaming.
i guess so the blog they were not talkin about video streams like movies, it was about live streams like a sports match, tv shows.
MongoDB's Atlas is actually the official cloud offering of MongoDB. Like, that's how they named it
no wonder amazon video quality is so significantly better than netflix/hulu/hbo/etc
this is still insane tho lmao
what are you doing step function?
Microservices was one of these management buzzwords and architects are some of the most useless people in IT. They don’t understand the complexity what loosely coupled means in most IT scenarios. They all say “it worked for Netflix and Amazon”. Well netflix are trivial compared for example to trading and real-time or near time applications. I told an architect that went onto the microservices idea at a customer that I was the lead engineer of when it was a startup. That it’s immensely complex to have to log and process transactions exactly in order in case of latency and errors (that increase with microservices) because we need to act on the moment, we are trading energy which means that actually bidding to supply or consume at a certain point changes the overal value of the market. It’s really-time or near time and missing that bid window because of latency pr microservice orchestration errors because service is down or busy is hard for communicate back in comparison to a monolith.
He went on anyways and the literally lost millions because and development, hosting and nd administrative burden went up and as I predicted the latency and the order of processing during failures(which always happens because also services need to be patched) had incurred more fines in a single year than all the previous 5 years of the monolithic trade service combined. So he was sacked! And I warned him he simply didn’t see the strengths or the weaknesses of microservices - and they are great for very simple not timing and order critical systems. As soon as there’s statement it’s a paradigm with incredible challenges
And with all of that Prime Video *always* lags and buffers on my [high end 2022 Sony OLED] TV, on a 600/600 FTTH connection with TV connected via Ethernet. Have not seen it happen on any other streaming service
He works at Netflix btw.
Just shows AWS engineers aint that good quality. Sounds like a rookie engineer quality to come out with such initial design.
Ok this is the video that I didn’t know I need now in my life
So does it mean amazon has ashamed itself publicly exposing their complete incompetence in streaming platform engineering in their own article where they described own mistakes in the grandiloquent beautiful speech??
In a way this post is great because it shows everyone that even big tech places like Amazon mess up their system design.
What I don't get is why they couldn't stay with server less but just shove all of the logic in a single lambda to remove the communication overhead. I'm curious how the price of that solution would compare
But why would you go serverless? This is a system getting called thousands of times a minute, why not build a server application for that specific purpose and let it eat all the petitions?
0:31 Wait till you find out who owns twitch...
Some PM saw the “stats for nerds” graph in yt, yadda, yadda, yadda, 7 months later: “we now have a scalable way to pester our engineers when a frame glitches a bit”
i really hope this is the masterpiece of a team of angry engineers
Yes was waiting for this one!
I use three layers of microserverless running redundant threads of my quatrolith connected to my NoData store. It’s amazing!
Isn't this specifically for livestreams though? I don't know much about video tech, but isn't livestreaming very different from regular videos. So they only do this to livestream videos for some reason... and the livestream videos are usually sports(?).
and this is what happens when js developers try to build the backend without doing their research.
Management: How do we blame the customer when their steam doesn't work?
Middle Management: Watch all the streams ourselves?
I have the feeling this man works for netflix
If I'd architected something like their original set up ay my work, I'd be fired 100% lol
This whole system seems like busy work for developers some middle manager thought up to justify their job
"You see, our streaming capabilities have a preset limit. Knowing this weakness, I sent wave after wave of virtualized containers at them until they reached their limit and shut down"
- Jeff Bezos or smthg, idk.
Prime- Because you're uniquely qualified to talk on video streaming architecture- think you could do a 5-10 minute breakdown of Prime Videos dumb architecture vs how it should have been designed. Maybe a couple graphics would be nice. Too lazy to Google around and figure it out myself.
Maybe I'm not familiar enough with how video streaming works, but I feel like bitrate alone is not enough to detect the quality of a video stream. You could have a stream with an insanely high bitrate, but if something's messed up when encoding the video, it could still be unwatchable, no? It seems like the goal here is to make sure the image you're sending to customers actually matches what they receive. Like you could have a 4K stream running at 60Mbps, but that's meaningless if all the customer sees is a purple image.
And I doubt they're running this for every single user. Likely just a representative subset of "canary" users to improve the chances of catching issues.
Using microservices when u don't need it is like using async when u don't need it and saying that u service works slower than its sync version (probably cause it's not a IO-bound tasks), but dumb ppl tend to use every mainstream way when it comes to choosing between mainstream ones and others
Here is the thing with AWS, at least my bill. Data is, cheap, Data out/transfer Fucken Expensive. When I read this I was also very confused.
Take a wrong approach. Make it a distributed system. Now you get a working disaster.
I am pretty sure detector here means some kind of ML jobs for some kind of content generation algorithm that probably has nothing to with bitrate.
What are you doing step functions 😳😳😳
What zero profiling and corporate hype does to a team lol
Hi, primegenrious rex, I wanted to note that they are probably doing ML detection as a process where each input probably has to be an image, hence the need to do all this, client side can't handle a vision model running even if its small (relatively). I can't think of any other way to do this process, they are probably sampling frames from stream uploading to a bucket rather than processing all of them (atleast I hope so) because I can imagine you always sample stream frames for QA and moderation purposes anyways, this would give them realtime-ish detection rate. Love your content rex man :)
Yeah, they say in the post that this is all fed to an ML model. I don't know why they need so much fuckery for that, tho.
Serverless and microservices only work to a point.
Serverless is great when you can scale to zero. If you never scale to zero, than serverless isn't nearly as useful.
And microservices can be great when you don't needlessly split them into ever tinier pieces where nothing is really able to do any work on its own.
Microservices should still be able to be pretty close to fully contained workloads, even if not a full app feature.
This seems like some machine learning engineers who don't really understand streaming devices, bit rate etc. That built an awful software solution.
I am confused, Is it not common for major platforms to have thousands if not millions of concurrent streams? I am just talking about a pure end-to-end streams.
Wait aren’t all your videos Prime videos?
I think it would be cheaper to hire a bunch of people in Asia, make them watch the video stream and hit a button when it went wrong 😂
How the hell did this pass an SDR?
Nah man serviceless computing is the future. You just type your credit card details in and they charge it. No having to deal with any code or infrastructure.
Amazon's system must have been made by a couple of interns, who's project they used as a basis for their streaming service