This was awesome one question why not cut down the complexity related to maintaining SQS, launching EC2s and downscaling them, etc by using aws lambda instead. The flow would be like this: Content uploaded to source bucket -> triggers lambda -> lambda spins up ffmpeg does the conversion and copies to destination s3 bucket -> users request videos from destination bucket. With lambda, you have to pay only for what you use, so I think it's cheap and scales very well. The only possible issue might be aws lambda can run for max 15min, so converting big video files might be problematic but even for that, we can use "AWS batch" which is similar to lambda but does not have this 15min limitation. In any case, I think it would cut down the complexity of the current architecture.
Great video - so much to learn from this than regular system design videos haha. I got some front-end questions: which player are you using? Custom built or open source like Shaka/Videojs? Also any plans to shift to HLS/DASH in future?
Damn good.! Keep this architecture series alive... I am really curious about one particular thing where you chose EC2 and not Fargate? According to some rough calculation, Fargate seems to be a better option for this scenario.
I would love to see this expanded or "split" to show handling a live stream as well. This would be great for VODs but thinking about it there would (have to) be differences in a live stream as the file would be constantly in transit (RTSP?), but then you would almost need live transcoding for your viewers sake. Finally after the stream the same process would happen as above for storing the VODs
Oh, really Nice and detailed HLD explanation of an application like YT 🔥seems like it is more complex than we think but you made it very crystal clear for a newbie in HLD like me. One video on what design patterns should a react developer (or full-stack moving to NextJS) should know to be a better developer. 👍
good video,i'm trying to gather info to understand what i need to know to make my first project which is a fake money poker site, something similar needs to happen since everytime a table is created there probably needs to be an es2 instance that runs the table or something.
Hi mehul thanks for the video amazing explanation !! Can you just make a practical video on how this works from front to back with MERN application cuz ter r no such videos out there which in terms of scalable apps , it will be much more helpful !
Can you tell. How the video resume or continue watching capture with timestamp saved with user profile and again start with what reaming left to watch...
You forgot to add the storage fees for s3 buckets. Basically in this structure you are paying twice on storage . Although it is quite low but you are paying twice, one to store raw videos on s3 bucket no 1 and another is to store processed videos...
Thanks a lot for this informative video... But I have one question... As You said that anybody can upload a .txt instead of a video and then the thing which happens is that it keeps on failing multiple times until it is shifted to DLQ... Now as fro every processing,we need an EC2 Bucket to be created... So Failing multiple times causes multiple EC2 Buckets to be created... Now to avoid this, can't we check it beforehand before uploading it in the first S3 Container.. Like what I try to convey is that, what if we try to setup an error and exception check before even uploading the video in the S3 Container... maybe that could be setup using an EC2 Server as well before the S3 Containers........
Thanks for sharing this mehul. I'm curious about one thing, you use ASG to start/stop EC2 instances for processing as you mentioned, Did you consider using docker + kube to do this task? If yes then what was the factor to go ahead with this architecture over the docker + kube one.
Hey, So I had a doubt. Why do u use the first s3 bucket. Like u could do something like have a proxy server setup and while its uploaded, The video can be sent as a passthrough stream to backend server. And When one server maxes out, it can spin up another one. And add it to the proxy via runtime api and when the processing is finished it can directly upload it to the production s3 bucket.
You don’t. Instead, on the EC2 while you’re processing the video you set a timer to keep the message in queue “invisible”. If your EC2 crashes or dies for some reason sqs will stop receiving the visibility message and would automatically make the message visible in a few minutes
what is the work of ffmpeg & ec2 there. When uploading we directly upload in s3 with suppose multer. Store the link in mongodb and when the user wants we can just send the link in front=end and show the video with a video tag. Can anyone please explain why it won't work?
It would work but raw video files are rarely something you want to ship to the user directly. For instance, a 4K60FPS video shot on iPhone would be 4GB for only 10 minutes of video. When you upload it on RUclips, youtube would process it in the same resolution with same FPS but at a much much smaller file size so that everyone could watch it
Show ads in your codedamn website videos in your platform after every 30 sec or 1 min duration this way you can make the video for free or make some courses free through ads implementation and users get the some courses for free and there be chances the user subscribe because of ads to pro membership. Profit from both side from user and from you also.
Start Learning Full Stack Development on codedamn: cdm.sh/fullstack
Do you know when the DevOps path will be available on your site pls?
I love how you explained the AWS tools. A further training on this from you would be amazing! Thank you for your wonderful teachings!
This was awesome one question why not cut down the complexity related to maintaining SQS, launching EC2s and downscaling them, etc by using aws lambda instead.
The flow would be like this:
Content uploaded to source bucket -> triggers lambda -> lambda spins up ffmpeg does the conversion and copies to destination s3 bucket -> users request videos from destination bucket.
With lambda, you have to pay only for what you use, so I think it's cheap and scales very well. The only possible issue might be aws lambda can run for max 15min, so converting big video files might be problematic but even for that, we can use "AWS batch" which is similar to lambda but does not have this 15min limitation.
In any case, I think it would cut down the complexity of the current architecture.
Great video - so much to learn from this than regular system design videos haha. I got some front-end questions: which player are you using? Custom built or open source like Shaka/Videojs? Also any plans to shift to HLS/DASH in future?
+1
This video was fantastic for understanding the entire workflow. Extremely well explained.
Thanks man. Very informative
super knowledgeable video! Hoping to see more underlying infra related video soon like this one!
This is something exquisite, love the fact that you've tried out the Whiteboard approach. Would certainly be looking forward to more videos from you.
Very nice video. Thanks. Please bring more such content.
Damn good.! Keep this architecture series alive...
I am really curious about one particular thing where you chose EC2 and not Fargate? According to some rough calculation, Fargate seems to be a better option for this scenario.
is it as cost effective as spot instances? really want to know.
I would love to see this expanded or "split" to show handling a live stream as well. This would be great for VODs but thinking about it there would (have to) be differences in a live stream as the file would be constantly in transit (RTSP?), but then you would almost need live transcoding for your viewers sake. Finally after the stream the same process would happen as above for storing the VODs
Great video, learnt allot, a code implementation means everything in action will be great to watch 😁😁😁
Learned a lot
Man man man! i really liked this video. I am building one with the same architecture. 🔥
More videos like these please!!! 😎
Can you share some resources about architecture , so we can also learn ..more on these types of topics. Like scalability , availability etc
Please make a series of video on this topic
Please make a course on this topic. It will help a lot.
Oh, really Nice and detailed HLD explanation of an application like YT 🔥seems like it is more complex than we think but you made it very crystal clear for a newbie in HLD like me. One video on what design patterns should a react developer (or full-stack moving to NextJS) should know to be a better developer. 👍
good video,i'm trying to gather info to understand what i need to know to make my first project which is a fake money poker site, something similar needs to happen since everytime a table is created there probably needs to be an es2 instance that runs the table or something.
Great video
Hi mehul thanks for the video amazing explanation !! Can you just make a practical video on how this works from front to back with MERN application cuz ter r no such videos out there which in terms of scalable apps , it will be much more helpful !
Do more videos like this about backend and architecture ... ❤️❤️
curious to know how much is your aws bill ? can you share the approximate value of running your full infra
Great video !!
What about the serverless architecture with aws ECS. Is that still relevant? which one is more preferred EC2 or ECS ? And why? Thanks
finally, youtube got some really useful videos. ty for the content. can u make one for payment gateways too?
Very helpful
can you suggest which instance type would be most suitable for this. Please.
Love it
Can you tell. How the video resume or continue watching capture with timestamp saved with user profile and again start with what reaming left to watch...
actually, 1080p is good. 4K is not required.
why not?
@@Tarunyadav976Everything is perfectly visible in 1080p. Also the render time is less, computing power required is less. Occupies less storage.
thanks.
You forgot to add the storage fees for s3 buckets. Basically in this structure you are paying twice on storage . Although it is quite low but you are paying twice, one to store raw videos on s3 bucket no 1 and another is to store processed videos...
Just wanted to understand. Why not use lambdas instead of EC2?
Thanks for such awesome videos, always waiting for your new videos, will be glad to get t-shirt giveaway
Thanks a lot for this informative video... But I have one question... As You said that anybody can upload a .txt instead of a video and then the thing which happens is that it keeps on failing multiple times until it is shifted to DLQ... Now as fro every processing,we need an EC2 Bucket to be created... So Failing multiple times causes multiple EC2 Buckets to be created... Now to avoid this, can't we check it beforehand before uploading it in the first S3 Container.. Like what I try to convey is that, what if we try to setup an error and exception check before even uploading the video in the S3 Container... maybe that could be setup using an EC2 Server as well before the S3 Containers........
What is the difference between your architecture and media convertor
cheaper
Thanks for sharing this mehul. I'm curious about one thing, you use ASG to start/stop EC2 instances for processing as you mentioned, Did you consider using docker + kube to do this task?
If yes then what was the factor to go ahead with this architecture over the docker + kube one.
For starters I didn’t know how to work with kubernetes. I’ve tried to understand kubernetes but find another simpler solution in most cases
Hey, So I had a doubt. Why do u use the first s3 bucket. Like u could do something like have a proxy server setup and while its uploaded, The video can be sent as a passthrough stream to backend server. And When one server maxes out, it can spin up another one. And add it to the proxy via runtime api and when the processing is finished it can directly upload it to the production s3 bucket.
to do that you it should be able to transcode in realtime and cannot be interrupted so you cannot use spot instances where you can save money.
Will u make this into a code
Hi, how could you adapt this to work with audio instead, or in addition to video? Thanks
For the development purposes are the aws services available on local dev machine??
grt tutorial
Damn Good
Can we use Lambda function instead of launching Ec2 machine for every video, wouldn't it be more efficient or cost effective?
Lambda has hard limits on storage and 15 minute execution time
How do you put the msg back into queue when spot instances are taken down by AWS?
You don’t. Instead, on the EC2 while you’re processing the video you set a timer to keep the message in queue “invisible”. If your EC2 crashes or dies for some reason sqs will stop receiving the visibility message and would automatically make the message visible in a few minutes
why did you prefer ec2 instead of lambdas?
Lambdas have a hard 15 minute timeout limit and very limited (512MB) disk space available
@@codedamn Thanks, clear presentation
what is the work of ffmpeg & ec2 there. When uploading we directly upload in s3 with suppose multer. Store the link in mongodb and when the user wants we can just send the link in front=end and show the video with a video tag. Can anyone please explain why it won't work?
It would work but raw video files are rarely something you want to ship to the user directly. For instance, a 4K60FPS video shot on iPhone would be 4GB for only 10 minutes of video. When you upload it on RUclips, youtube would process it in the same resolution with same FPS but at a much much smaller file size so that everyone could watch it
@@codedamn Ok got it. Thanks man. I really love your videos and recommend all my friends to your channel
Is this channel still active? Have you ever deployed this with any DRM so that users cannot download the videos?
why is cloudfront so expensive? T_T
11 9 after point
COMMENTED FOR T SHIRT GIVEAWAY!! :)
Noice vedio 😁
💚
Finally I can stream my game's to my friends and show them whose fault was.. To lose the game lol 😆
May the like be on this comment.
Can we use FIREBASE instead of AWS.... ????
🧐🧐🧐
Show ads in your codedamn website videos in your platform after every 30 sec or 1 min duration this way you can make the video for free or make some courses free through ads implementation and users get the some courses for free and there be chances the user subscribe because of ads to pro membership. Profit from both side from user and from you also.
Make some video on hacking
A programmer who uses a whiteboard to teacher, my lecturer... Nice hair style btw ❤️
random comment XD