Scaling Hotstar for 25 million concurrent viewers
HTML-код
- Опубликовано: 11 сен 2024
- Gaurav Kamboj -- Cloud Architect at Hotstar -- explains why traditional autoscaling doesn’t work for Hotstar (Disney’s OTT streaming service). Hotstar recently created a global record for live streaming to 25.3 million concurrent viewers.
Gaurav explains:
1. The challenges in scaling infrastructure for millions and how to overcome it.
2. How Hotstar runs gamedays before actual games.
Details on hasgeek.com/ro...
What bothers you more - monthly real estate rental or cloud cost bills? Plan your cloud infrastructure and optimize costs. Learn from experiences of enterprises and startups, on 26 November 2021 at hasgeek.com/rootconf/optimizing-costs-of-cloud-infrastructure/
Great !! wish he has covered the database scaling as well.
@5:05 The point about the homepage API's taking a hit as soon as Dhoni gets out is INSANE. I wouldn't have imagined that scenario. Excellent talk.
That point is absolutely on point. Never thought about it in this way. Loved the talk.
Exactly I never thought this point as developer 😳 awsome
All these are usually observations from previous occasions.
All predictions are always based on previous results.
There will be cache for home page content, why will it cause a hit, anything I am missing?
@@bhaveshssharma8826 You cannot use cache , as you need to change data of recommendation , and last watched videos , advertisements and so on!
Hence we cannot use cache. It'll be a bad user experience!
I couldn't make my younger brother understood how big an achievement this is, he's still doing his engineering. I hope one day he'll understand what level of engineering in India specifically hotstar is doing. Loved the talk
Hi ajay, can you explain a bit about what is high level about this? i am not an engineer, but would like to know. Also, how does it compare with the engineering of other global hotstar competitors?
@@deep_z concurrency is the big thing here, how to handle these many people accessing same set of services without any interruptions, while in other apps you'll see different set of data will be consumed by different set of people, thats not true here.
@@ajaykumar-xy6pw youtube , irctc is doing it all the time . isnt it
@@arjunjmenon RUclips yes, irctc I don't think so, and overall RUclips is backed up by big players like Google, which is one of the best tech companies in the world.
@@arjunjmenon irctc lol
The project I work on has 2M in a month and these guys are handling 25.3M in a day hope to work on something so big like this.
Not even 1 day, its 25.3 M concurrently. Over a day it will be multiple magnitudes higher!
the product I work on has at max 64 concurrent transactions
@@karun4663 haha
Just now of my application broke down on 100 concurrent users🤣🤣🤣🤣🤣
Ind Pak game, they reached 100M recently
I used to call myself a IAAS engineer before listening to this. Baffled me completely. Still consider myself lucky to be able to understand everything he said
Wish if I could say the same. :( I could understand only 50% of what he is saying...
Same here... Working on a mobile banking app which handled 1 billion requests every month..... Excellent talk... Very clear
@@marriagevideos308which one
And RUclips just decides this one fine morning, to recommend this video to all engineers !!
Here after 5.4 million in IND v NZ Semi-Final of 2023, crazy to think that the new record is DOUBLE than this!
I've seen the concurrent viewers hitting up to 45M in the last ipl match. Science and technology never cease to amaze me.
That's lakhs bro not millions
I too thought the same and got confused😅
And the skill of Indian engineers
Excellent level of logging, monitoring and observability. Loved the talk even though I am not a devops/ops person
Man same.i just felt this was interesting
This is not just for devops or SRE folks.This is what core engineering team does
This is amazing!
I did not move from my chair for this 46 mins!
In those tense moments of IND vs NZ, I used to wonder how these guys are keeping up with 25m users! That question has been beautifully answered after half a year!
Thankyou Gaurav! You and your team are the real heroes!
What's interesting is they're not using CPU or networking for autoscaling. They're using concurrency and request rate. I think they already have scaled infra before the big game already. Autoscaling only kicks in when that baseline is exceeded
Yes u need to scale on golden metrics ! CPU based autoscaling is not very useful ! It should always be concurrency or latency!
Yep. Proactice scaling > reactive scaling
I imagined how RUclips works perfectly with millions of calls per minute everyday
more like per second
RUclips is more distributed bcz of that u will most likely face buffering watching videos less popular youtuber outside India.
This is absolute pinnacle of software engineering. Kudos to the team. I am still a student learning cloud platform and all, the whole presentation kept me glued to it.
Wait...maybe I am not getting this...but isn't this all just configuring AWS?
@@asrajan55 hi Soundar, afai recon, the scaling of so many resources, along with the risk management and maintaining concurrency without breaking down the APIs AT THIS scale, is itself a challenge. To mention the fragile nature of the viewership graph and working on a breakdown time of 30-60 seconds, all together makes this task an absolute marvel.
They came up with their own autoscalling app - thats gangsta
TLDR; Perform Load testing to get instance limits (in terms of cpu/ram/network) and than prelaunch more instances that would serve your expected numbers of concurrent users
I think the essense was well summarised. But the actual value lied in the process of the talk.
I think now there should be part 2. Scalling Hostart for 4.7 cr concurrent viewers.
Kudos to hotstar team, handling traffic at at this level. Superb presentation to illustrate all concepts.
Wonderful talk by Gaurav and it was such a pleasure to work with him.
yeah here after ind vs pak
Same bruh
@Sir Bradulkar Where have you seen commerical content on 60 fps sir?
The way he explained 👌🔥
I have an on prem application which i build , which is based on Event driven architecture and we handle around 14M msgs/day , whereas the max spike rate we have seen is 5M/hr, but our throttling and fault tolerance isn't as effective as this , loved the details and especially the depth of the presentor.
Awesome explanation of how the Hotstar tackled the load problem to handle huge traffic. Liked it. Thanks for sharing such a good info.
Came here after India vs Pakistan match when 2 crore people watched the match without buffering.
This basic question is unanswered. 25Million concurrent viewers == 25Milllion TCP/UDP connections. are you creating the ~25 or higher TCP connections while performing the game day/load test ?
Would love to hear something like this from Netflix which handled Money heist and all very smoothly. One key difference could be getting live data vs streaming from CDN. Nevertheless great job #hotstar team. Proud of you guys 🙏🙏
Read about netflix openconnect they pre calculate the demand and preload the content at ISPs
Apart from code, its just configuring AWS, I wonder, how AWS boasts about their metrics.
Now I know why we don't get to watch the match at 4K 60fps at least. Hope we reach 4K 120fps soon with premium and 1080 60fps on regular plans.
Unfortunately hotstar management's doesn't want to spend more and let football streams be at 240p when it says Full HD.
Fyi speaker isnot from any IIT,NIT,IIIT or BITS.
Love love loved it❤...AMAAAAZING session...Every sentence is so full of substance and thought provoking...Gaurav absolutely killed it.
Wonderful presentation. Lots of insights into how you handle resiliency and load. Many thanks for uploading this.
This is insanely huge! How come this has not made news.
Because they are more concerned with Aryan khan's drug habits.
The bitrate is still pretty terrible. I mean you can straight up see pixels on screen. Of all the streaming services, hotstar is poorest. Until they improve the bitrate and the fps, I will not be impressed.
Interesting review.
About using 8 AWS regions for distributing the bandwidth for pre-match load testing, how did it play out on match day since almost 20M of the 25M hits would have come from within India and hitting only 10 edge locations (2-3 regions) if I'm not wrong. Is that load testing strategy a good match for actual production load pattern?
have the exact same question ....
if your all traffic comes from india doesn't necessarily mean your all traffic would be served from indian AZ servers only. there are always resource limitations which can loose up the performance comparatively when entire traffic is being served from indian servers vs scattered global servers. but you would not want to take the risk of serving everything from indian servers on the cost of risking the whole system as a whole. that's why a lil slower and higher network bandwidth is accepted than pushing all the traffic to nearest servers and taking the risk of system downtown. its a "Trade-off". people will be okay with a lil lower loading but not with viewership disruptions. a lot of things come into play when you are dealing with such scale. *Engineering is beautiful*
Saving this for my system desing interview
Yes 😂
Woww, wonderful explanation with right techniques along with use cases. Really liked it. Thanks Gaurav.
He's Hero of Cloud 🫡 waiting for pak vs india match case study .
Madness! The scale issues are insane! how did they even manage such spikes.
This is so amazing. No claps from the audiance? What kind of audience was there?
I've learnt soo much from this video. ❤️
3-4 hrs watching rain on a screen!! That’s dope! 😂✊
they run highlights during this time, not rain drops falling.
I want to know how they are managing the aws bills specially when he mentioned c5 9x large cpu's
All the cloud providers have corporate plans were they provide huge discounts to their potential customers.
My coding abilities are limited to printing "Hello World" on python yet this is at the top of my recommendations lmao
Its best if you start watching stuff like this,.. Makes you think a lot before writing a single piece of code.
@@godsonjoseph strongly disagree. It'll only serve to demotivate the beginner who will find all these keywords and abbreviations to be extremely complicated.
Maybe after programing for some time and reading some system design books one would be comfortable with this video.
So jist of 45 min talk is to use custom autoscaling metric like concurrent request instead of default cpu/mem metric provides by aws.
Why don't u guys open source the auto scaling tool that u developed?
Great talk Gaurav. Some especially useful and practical tips to scale on public cloud.
Hotstar: "Scaling for 25 million concurrent viewers"
RUclips:
There’s a difference between streaming live video and just doing playback of existing files
@@amanbhargava3164 even RUclips has live videos
RUclips is owned by Google. Which has it's own GCP.
@@techmad8204 but not with 25M
@@ruturajdharav4675 it's biased though. the number of viewers hotstar gets or cpus they run at any moment is positively drawfed by youtube numbers. the fact that all of those cpus are crunching the same livestream means very little.
And 1.2 Crores yesterday 🙌🏼🙌🏼🙌🏼
Nicely explained by Gaurav, there is a lot to learn from his talk
Really good production level insights, thanks for sharing...
Us humans can seemlessly serve 25 million people over the internet but when will we start making reliable projector connections???
It was indeed a great video. Thanks Gaurav for that wonderful presentation. I have one more question -> As you said for example if Dhoni is the next batsman we all know that the users are going to increase. But lets say if a batsman who is not performing well throughout the series, suddenly starts performing extremely well and again there is a spike in traffic. What is the plan to deal with such situations ?
Really awesome engineering by Gaurav Kamboj.
I think after seeing this video Amazon will make some changes on autoscaling.
Its my please to hear you
The irony when you are talking about handling load for a streaming service and the screen says no signal (from the projector),
Bro, I was desperately trying to upgrade to premium during India Vs Pakistan. That day there were more than 90 Lakh online viewers. I was unable to pay online , payment methods were not working due to high traffic.
I'm a fresher just placed. I'm happy I atleast understand what he's talking. Ssly a lot of efforts go into this.
This is the mafia level of infrastructure management!! I handled 100k users qnd felt like boss. Well now I know what boss level looks like xd
i genuinely appreciate the stability of the disney hotstar platform during intense workloads
but when you stream at 720p @25fps for 20 million vs 2160p @50fps to 2 million in uk germany,
it is not a big achievement
It's 1080p
@@kapilbhardwaj4680 did you watch it in 1080p on hotstar?
because i remember that the best quality was llimited to 720p 25fps for ind vs nz game
@@kapilbhardwaj4680 if you are a premium user only you will get 1080p! If you take hotstar mobile plan then only 720p resolution!
On TV app it was 1080p 22-30fps
@@sanjeetbisht since 2020 it is 1080p 25fps, i was specifically talking about the ind vs nz 2 day match, i am pretty sure that it was 720p 25 fps for premium users for live events
what a great talk, with details.
would like to see these more in future.
Again hotstar created another record of 1.3 billion viewer's.... Ind vs pak T20match 2021..They handled it very well...
Great talk… thinking in terms of RPS and concurrency to auto scale is out of the box… I am bit disappointed to know they need to keep watching the event to scale, hope they’ve automated this now
All we here , After india vs pak match to know its database .
This talk is amazing! Loved it.
this is amazing but i am thinking about their AWS bill 🤔
What was the AWS bill?
Whatever the bill was hotstar with this kind of performance could have already earned it back
5-6 crore each day during peak work load
probably a million on a day like this
@@shashankshekhar8970 a million dollars?
$1 Million on an Average
So inspiring to see home made apps doing so great
Thanks youtube for recommendation.
At 27:00 should not we save personal preference in client cache ? For better user performance.
Great.. Very informative.. now I'm just wondered how autoscaling works for normal telivisions in traditional way thru satellites
Autoscaling is not needed in em communication
That is called broadcasting. Any listener can get the packet and process it. If there were no viewers, in that case also satellite is going to broadcast.
But in case of internet, bandwidth is limited.
Mainly its about the configuration on AWS to scale up that they have acquired using load testing.
AWS need to be boasting about their cloud platform and services.
Watching again after 2 years.
RUclips never fails to surprise me
how much is your infra costing? Is it hosted on EC2 with ASG managed by ECS?
Good talk, but most of heavy lifting is done by AWS. Hotstar is just configuring AWS services
This is insane. I would love to join Hotstar as an Android Developer one day
Was going to go for a sleep. But then this happened. Always wondered how they scale it.
This is awesome! I would like to know the current infrastructure also, they must have come up with new strategy.
Simply superb ... Learnt a lot of things 🙏
First try to give real HD quality for live streams. Then let's see where is that 25million concurrent goes. If the server gets the loads then reducing the quality is not excellent engineering. Sorry to say!
In case you didn't hear what was said in the beginning, there isn't bandwidth available for that. Other services on AWS will be affected from being starved of bandwidth. As it stands, they used 10TB/s of bandwidth, which is 70% of India's total bandwidth. Encoding the feed to run at 4k 120fps is not exactly a challenge, as it'll only be done once, it is the network that'll take the brunt of the load as it gets hit with a few hundred terabytes of data every second.
The hero who make the Movies going from behind the curtains!,
liked the insights of ondemand scaling very much.
5:05, I'm a newbie dev but can't we just temporary cache the homepage once the user open the app, and then show the same cached files when the user will return and because most users won't be consistent with the homepage, some will leave, it won't have a huge load on the homepage.. and because users will open the app in different times, it's easy to cache the homepage at different times
You will still make API calls to return from Cache.
Depending on what level you are caching, you can shave CPU cycles but not the requests.
Every user has similar API calls like fetching the video list, categories data but there are many which are personalized and specific to a particular user such as user profile and recommended. This is not that easy to cache and needs to be sent differently. I guess this is where the backend system is most impacted. I'm not exactly sure as I also just started learning this...
In ind vs pak match. No of viewer was 1.1cr . Being a full stack developer I can relate
today our server crashed and this is what I am watching lol
about 13,500 8 CPU core 16 GB RAM load generators were used for load testing it seems, probably spot instance would be better in this case.
It was more than 70million during ipl
I feel hotstar’s Home API calls have been comparatively slower than other platforms out there. Every time I go back to home it buffers for some time always
I think they have done it intentionally
25:36 Chaos Engineering topics are nice and fantastic.
3:25
Scaling is not slow in nature since kubernets .. if u follow MS arch a pod can be up and running in about 5 seconds
This problem was solved long ago....
Not so difficult when app stream at 144p
Jio cinema handled 2.2 cr (22 million) concurrent traffic for a normal ipl 2023 league match of CSK vs RR
Wonder what will be the live numbers in IPL 2023 final
12:15 Elastic Load Balancer but not really Elastic in nature!
To be fair no LB really is. The scaling by margin of 800 servers with high capacity is harder for AWS. They don't have hardware optimization GCP has. Although I do think this is hard for GCP too from networking perspective.
This was 2 years ago! wow..
This video needs CC, the accent is a bit heavy to fully comprehend.
Where are you from?
There are captions
Amazing tech talk 🎉
Thanks, Learnt few things 👏🏻👏🏻👏🏻
This guy is a super hero :)
Tata sky and cable/satellite network providers provide contents to billion users since years concurrently without any latency and issues :) . 25M is too less for them .
Noob
They don't stream via HTTP API calls
@@raghunandan8162 - yes they use dbv protocols for streaming , they also use http btw these days . Thanks
you use 36 core machines for generating load? you're testing for huge amount of RPS maybe, but that ain't real life. Real life is TCP Connections.
Fantastic summary
And yet hotstar content moved to Hulu, Disney, ESPN in US
Disney have bought stakes in it ? do u know ?
"Dhoni is out" !! .. soldiers brace for the impact.
Scaling was a great feat but even after all these years , the UI and UX of Hotstar is absolute garbage.
zee5 has worse
I never encountered this issue.
yes