dude!!!! This is how colleges are supposed to teach. Why am I learning more on youtube than my $40k degree has ever presented to me 😭😭 You are awesome man!!!!!!
@@thisisneeraj7133 and Any professor who has real experience won't go back to teaching and become professors!!!!!!! U have to understand that teachers can only provide basic information. We as students need to digest every possible available information and filter out the ones we need to progress. then only we can succeed in life. simply saying, nobody taught me well is not acceptable. This is why Internet plays an important role in getting information from multiple sources.
I agree partially. While the content that places like RUclips provides can be as good or better than what we see in college, the major advantages that colleges have are the structured tasks and feedback that you get. I can watch this video and call it a day, but college puts us in a situation that forces to apply what we have just learned, and does so in a way that builds upon previously taught concepts. Sure I can do that on my own too, but its a lot easier after experiencing that style of learning already.
@@kumar309 infact there are many who goes back to teaching and research. any University with high focus on research will have excellent professors. Sadly quality of research and education in India is not good enough except some chosen ones like IITs.
Hey guys! This video is my attempt at designing Tinder. Points to keep in mind are: clarify requirements, design the basic system, modify per feature, use concepts learnt earlier to meet requirements. Also, if you are preparing for a system design interview, try interviewready.io 👍 Cheers!
WebSockets - create TCP connection to server, and keep is as long as needed. Server or client can easily close it. Bidirectional communication - so server and client can exchange data both directions at any time. It is very efficient if application requires frequent messages. WebSockets do have data framing that includes masking for each message sent from client to server so data is simply encrypted.
In addition to sharding what we could also do is to keep a db field called preferences which is a bit set of user interests in different activities. Using that we can then for each shard create a k nearest neighbors chart for this preferences fields which should give us users nearest to us which we could then recommend. Also not totally sure but how about using distributed ledger (nosql append type) i.e. file for storing chat history indexed by hash of both users in chat. And a message queue that is used for transporting any new message typed by any of the users? Would love to hear your thoughts on above.
Its inevitable that this channel will be successful. You got the goods man. Intelligence, eloquence, and the positive energy. Thankful for your work, will definitely pass the channel along
Engineering universities need this kind of real world problem solving exercises like your videos, I'm even surprised that I'm learning from this for free.
One thing to remember about databases are they aren't storage wise different from files. What a database provides you is multi user concurrent access without corrupting your data and acid aspects. In theory there's hardly a case where you'd put a big blob in your database. Most often just use paths to big files and keep those paths in the database table. The paths could be on an NFS or whatever distributed file system there is. Your NFS can then deal with geolocation etc. You could create read only mount points across multiple geographical locations for the same file. I'm just leaving this here for anyone to discuss these points further with me. Great video btw!
17:48 - A good use case for an image service would be when you need to send different resolutions or sizes depending on the client (e.g. mobile, desktop, print etc.)
@@man904u yup, but a good use case would be for profile picture (shown while chatting) vs profile picture shown as a card when swiping - same image but different resolutions.
I appreciate the content from all the RUclipsrs out there helping us with System Design and DS&A but I have to say, I feel like a lot of the younger ones are just doing it for monetization or popularity. Your enthusiasm really sets you apart man. Good job finding something you enjoy doing. You seem like a good dude.
if a developer can't get matches on tinder, he creates own tinder. ;) - I just want to say thank you for your hard work. Very useful content. Keep up the good work.
I have been following this playlist Daily like 2/3 videos per day. Loving the content. First System Design is Tinder, and felt interesting already. Thank you for this playlist.🙌🙌
Absolutely love the video! I don't know when will I use these concepts but just getting to learn about it is itself an experience. Hands down the best tutor!!!
Great video :) One comment - To implement recommedations, we could build composite indices on the profile table which can index on, say, (age, location) in traditional RDBMS databases such as PostgreSQL. It's slightly misleading to say we can only have indices on single columns.
Also you can have multiple single indices on a table (not referring to composite indices here), even though they wouldn't be very helpful in this case, but it is possible. Great video, nonetheless.
Apart from DB indicies, recommendation system can use quad tree kind of in memory lookup that saves userids per location. So all users near by area would be in a single leaf node of the quad tree and can easily suggest people for matching.
Once I know that I am going to have a system design interview, this is the first place I know that I must come. Not to copy the answer, but to understand how you came up with the design, the process of thinking, the required knowledge that I might miss; what should be used because of what and so on.
You made it easy! Probably some things I feel like need to be discussed: Storage and bandwidth requirements; probably authentication and authorization can be services to note; load balancers; system interface definitions eg. acceptMatch(userId1, userId2).
I wish I could have watched this video 5 years back. I have started learning in 2024 and its awesome to see this one, thanks @gkcs for this amazing learning content;
Great video. Learned a lot :) Can you please explain these topics separately: 1. p2p | web sockets and protocols | client server - when to use what? 2. managing sessions - possible solutions 3. design system that involves real time location tracking like maps or uber?
Gaurav you have the best resources on internet. So much learning in just one video. Your attention to what, why and pros and cons is simply awesome. I have learnt a lot from this channel.
Great video! One particular part that I think goes beyond what was mentioned in the video is the part about sharding the database based on geolocation of users. Let’s say we create one DB for each radius of 50 sq miles, the problem is that some people live on the edge of that 50 sq mile circle, so only a portion of that dB would be relevant for them. Also, certain 50 sq mile radius may have multitudes more people/usage than others. I myself am not sure on the ideal solution though, would love to hear people’s thoughts.
I'm a 1 year old self thought dev i love every aspect of developing apps and websites.First time watching your vids. Your explanations are in-dept and helpful. This is also my 3rd or 2nd youtube comment ever and you are fully deserving of it! Will be a regular on your channel from now on! All the Best and Happy New Year Everyone!
Great video, just a couple points I'd like to add: 1. Your gateway does not need to hit the matcher service every single time before client 1 sends out a message to client 2 2. Last I checked, Tinder uses an ELO system (popular games such as CS:GO do so as well), that influences the kind of profiles you see, based on how many people swiped left/right on you. So it's vital to store that information too.
Is storing left/right swpie information is only useful for ELO system? If we don't store that information, how do we support matches? A right swiped B, and later, B right swipes A. At this point, how do we know it's a match?
RE Recommendation Engine: you can definitely have a composite index on multiple columns. The B-tree can be searched on the first column, and each of those matches will also be structured to allow further B-tree searching on the next column, and so on.
There is a huge difference in your channel, you really shows that you really love to teach and enjoy as well, good luck Gaurav, really love your channel.
Great content! Once the basic design of segregating responsibility into multiple services are done, Could you please also talk about how to meet non-functional requirements by introducing concepts in the same design, like - Finding out single point of failures - How to scale individual services and also the databases - Introducing load balancer - Caching if required, etc.
Exactly even i was looking for something on performance, scalability, failures, and caching questions which I got in my interviews..please answer them too that will be really helpful.
Hey Gaurav, just wanted to say big thanks to you. Recently, I was asked the choice of storing images bw dfs and sql table in a design interview. Having watched your videos before, I nailed that discussion. There were lot of other stuffs too in that interview, but it was super helpful to have seen your videos before. Thanks a lot for your time and efforts.
Apart from the great stuff, I like how candid your videos are. White boarding it all live and not using presentation with animation is also what keeps it close to the real experience of an interview.
Thanks so much for your awesome systems design video. You really provide the shining part of design, which is taking customer use case into consideration and go from there, and also break down web service/application layer/db & DFS when comparing different approaches and see which one fits the best.
Add few more things: 1. Discuss about CAP while designing a system 2. Partitions should be on the basis of active users and their behaviors not with geo 3. Think about the caching (Distributed and in memory) 4. Think about static and non static resources 5. Messaging system (even for chat too) or event based approach should be considered for all offline works 6. Should discuss quite quantitatively (numbers make good impression)
omg your videos are so interesting and fun. I have 0 to do with any of this, I am in finance. Yet I've been binge-watching your videos for days. Thank you!
Thank you very much for this video.Its really amazing. I know you approached this video more from an interview perspective so you can only touch on high level points, but please think of making a series about each service so that we know how the real world implementation works and how we need to approach it at a more detailed level. I liked the BLOB vs FILE talk. Keep up the good work.
Thank you! That's a really useful video as a Product Manager I am learning a lot about how to correctly design the systems with my developers and other colleagues.
Great explanation with energy teaching providing the Awesome learning experience . This place is one of the best place of content ever found on youtube
That was one really well-thought out video. Is it possible for you to talk about a few prevalent DBS technologies(NoSQL etc) and how they are used in System Design in a future video? Many of us CS undergrads have limited knowledge in DBS; only upto RDBMS. Cheers!
Thanks for explaining the concepts so easily. I left with two questions after watching the video. Hope you might be answering someday if you see this comment. 1. How to store the users those have been liked by someone (right swipe and other like features e.g. Stars, super like etc) and the same with left swipe as well. 2. How the system calculates the distance between the users by only using the location that is stored into the database. I mean what is stored into the database for the location and how it compare this value with other locations. Thanks in advance. Cheers!!
Great one Gaurav. Keep it up. You are really helping out people like us who are beginners in this field but want to excel. Always remember people watch your videos, so keep posting amazing content.
Hey Gaurav, amazing video. Just one suggestion that the , should be probably a NoSql db so as to store multiple connections as a list or a hashSet for each userId, since this would not be possible in a RDBS if userId is a primary key. Similarly for Matcher service where each userId can have many matches to track. So something like HashMap of Lists/Sets would be better I guess. Secondly, could you please explain a bit about how to decide when to make a new service and when to continue using same service? The service partitioning is not intuitive to me yet. Thanks
Gaurav, as per me, for storing multimedia, Object storage like S3 is most cut out. 1. It gives us high scalability, which is harder to achieve in File system. 2. It is cheaper as you have various access tiers. Just put most frequently used data in Standard tier and rest in cold tiers. No such segregation in File system. 3. Object storage offers durability. File system is hard to replicate and considerable manual effort is required. What is your take on this?
As usual, great video! I started watching your videos to get ready for interviews, but I keep watching your channel just because I love how much a learn in such short videos. Thanks for sharing your awesome work :)
Nice video this is awesome! Please make more like this. Some ideas. 1. design a distributed web crawler 2. design facebook messenger 3. design a ride share service like lyft of uber 4. design a restaurant/food app like yelp
Love your video! feeling a bit dedddd at 5am, but your enthusiasm at 18:20 immediately lights me up lol .Good job man! And thank you for such good content and detailed explanation.
Amazing video Gaurav, I am now suscribed. Maybe the best channel about System Design. I would love a system design video about RUclips. It probably has a challenging system design and there are no videos in any channel about this, so can be an opportunity for you. Best wishes!
Thanks alot for such important and interesting videos. You explain critical things in simple way. I also interviewed by you at uber but that time I didn't recognize and also not cracked the problem you asked. Thanks alot for system design videos. Keep posting on new things.
Before watching this video I thought that I had cover almost all the system design part, and now I realize that its a big lie;). Learn't a lot from this video. Thanks, Gaurav!. eagerly waiting for the next one.
Hi gaurav!!! I super super love this video, I think this might be the best starter into system design ( maybe u can change the title of this video, eg [for beginner] ) :) waiting for other amazing videos!
Thanks very much. I've been learning Django through tutorials and books for couple of weeks now and i couldn't understand the structure behind apps. This makes everything much clear. i'll probably watch all your videos.
Some additional insight as to why you would want to shard/partition by location: For example: Consider all users from Mumbai are stored on one shard. In this way: a) If you have to recommend users to someone from Mumbai, you are running your query only that shard. Hence, you are querying much lesser amount of users (Users all over India vs Users only in Mumbai). Less data to process = Faster query results. b) It will rarely happen that you have to "join" data from multiple shards. Users in Mumbai would only want recommendations from Mumbai.
Recommendation of profiles is based on location(max 100miles radius). Tinder uses Elasticsearch that can have multiple data nodes -geoshard. If we can quantify the load of each geoshard (“load score”), the load score values for all the geoshards should be roughly the same. 1st Solution (not efficient/used): Divide world map to ‘n’ shard squares. But near to ocean location, big cities, load will be different. 2nd Solution: Balance geoshard using load score criteria - • Unique user count• Active user count• User’s queries count in an hour• Combination of the above. Formula used - “min(standard deviation(Loadscore1, LS2, …)”. Shard size is determining considerations - •Geoshard migration(users movement)•Querying multiple geoshards(so shards cannot be too small)•User density(based on city location). Tinder currently uses 40-100 shards based on calculations above. S2 Cell & Geosharding Algorithm - Hilbert curve. Reference : medium.com/tinder/geosharded-recommendations-part-1-sharding-approach-d5d54e0ec77a
Gaurav, I have few questions regarding recommendation service: 1.What is the advantage of horizontal partitioning over indexing using location . Isn't it the same? both essentially reduce the search space? 2.How really cassandra/nosql has an edge over sql using multiple indices? 3.Where are we storing the matches which are already shown to user? if we query based on location and filter don't we end up showing same matches again and again? I was thinking of design where we store matches like items in newsfeed and show items after the last shown item. This also improves efficiency i.e we can use sql database and pre-generate matches and show them when client opens the app. Additionally for every new user created, we can see who all users' preferences he matches( and distance) and add it to corresponding user's recommended matches.
Hey Gaurav, great content! Really learning a lot from your videos. Had a question about sharding on location. How would you handle when a user opens the app in a new city or country? Would you have to find that user, delete their record, and move it to the new location's shard?
Hi Gaurav, thank you for putting up this video. Very helpful and clear. I am into my learning phase about system designs. I need to ask why we haven't covered topics around the availability of the app across different geographies, scalability, fault tolerance, and load balancers in detail here. Love to hear your thoughts on the same.
Thanks for the video, I didn't understand the part that , (select * nightmare) @10.25, and you also say, select * is used often, can you plz elaborate a bit more ?
Imagine you want to select the age of a certain user. If you store the image along with other info about the user and you execute by mistake select * from users where username = 'whatever' this would pull the image as well thus generating useless network traffic and consuming more memory than a simple select age from users where username = 'whatever'
Hi Gaurav, excellent video. I have a couple of questions/comments:- 1) Multiple gateways behind a load balancer similar to whatsapp? or do we not see those many requests coming in? If we do have multiple gateways, the sessions service will also have to store the gateway information on which the connection is open for a particular user. Or is this implied? 2) Wouldn't it be better to have a "like" service? This covers matching as well: it is 2 people liking each other and then we notify both the users. 3) From what I understand, and it could be an overkill for a 1 hour long interview, tinder also ranks profiles that it presents to users based on matches that these profiles get etc., so we could just make a remark that a general ranking algorithm could be used in this case. 4) Not a comment on the video, but for clarification for the other viewers who were wondering why XMPP over WebScokets for chats/notifications: XMPP is an application layer protocol that you can use on top of WebSockets/TCP/HTTP connection. WebSockets is a Transport Layer protocol.
Great content! There is a place where I think there was something wrong - please correct if I have my understanding wrong. Sharding and horizontal partitioning are different. Mixing the two is wrong, sharding means partitioning the data in some way(range based, or hash based etc.) and keeping them in different nodes/servers and horizontal partititioning means also partitioning the data but keeping them in the same database, just in different tables. Apart from this, great videos and great enthusiasm, you make learning new things fun. :D
Hi Gaurav. Great video. How will the database shards get updated, if a user is navigating through different cities. userA belongs to cityA, has the user data stored in shardA. If he moves to cityB, his location has changed. Should the user data now move to shardB? Is it a good idea to keep updating user data across location based shards, as the location of the user keeps on changing?
Thank you so much Gaurva! this is very helpful to go through separated services such as profile, chat, matches and recommendation. And I am glad you mentioned p2p in chat. But how do those services interact with each other? How to handle dB fail. It's not very clear
Hey David, thanks! I'll be taking some videos on master slave architectures and API construction. Till then, have a look at the videos of "Single point of failure" and "Consistent Hashing" for ideas 😁
apparently, you don't need to pull all the recommendation in one shot, you can do incrementally, like a group of 10. joining all three and make index would be overkill in this case, i believe.
Hey Gaurav, This is really a nice video. Can you make a video on system design of online judges like codechef, codeforces. I am really curious about it. Please focus on high code submission volume, if you make it. Thanks a lot!!
Great great videos! Some quick questions: 1) When deciding between a DB or Filesystem (as to where to store images), why mention mutability as an important criteria? How do DB's and Filesystems differ in that regard? 2) You said "File systems store large objects separately. DBs can do that via vertical partioning". What did you mean by "File systems store large objects separately"? 3) What did you mean by "I'll set up a CDN around my File system"? Thanks again
What I understand was: 1) when you save information in a database you usually make some updates i.e, change numbers, dates etc, in this case, if you want to change some information about the image you should change the bits of the blob and does not seem very relevant, you usually replace the whole image and that is kind of the same as replace a file. 2) I guess he refers to a file is a single entity in the file system but in the database you can have multiple columns and one of them can be your blob, so the information is related 3) CDNs allow to get content faster for an user, suppose the app server is in Japan and you are in USA, when you make a request, this has to travel to japan process the request and send a response, for this request can reach japan it has to wait some time because the light speed (what I mean here is that your request cannot travel fastest than the light speed) and well this translate in some latency, but what happen if you put kind of a server in Cananda? then your request has to travel less distance and your latency is going to improve, this "kind of server" is a CDN.
dude!!!! This is how colleges are supposed to teach. Why am I learning more on youtube than my $40k degree has ever presented to me 😭😭 You are awesome man!!!!!!
@@thisisneeraj7133 and Any professor who has real experience won't go back to teaching and become professors!!!!!!! U have to understand that teachers can only provide basic information. We as students need to digest every possible available information and filter out the ones we need to progress. then only we can succeed in life. simply saying, nobody taught me well is not acceptable. This is why Internet plays an important role in getting information from multiple sources.
I agree partially. While the content that places like RUclips provides can be as good or better than what we see in college, the major advantages that colleges have are the structured tasks and feedback that you get. I can watch this video and call it a day, but college puts us in a situation that forces to apply what we have just learned, and does so in a way that builds upon previously taught concepts. Sure I can do that on my own too, but its a lot easier after experiencing that style of learning already.
@@kumar309 infact there are many who goes back to teaching and research. any University with high focus on research will have excellent professors. Sadly quality of research and education in India is not good enough except some chosen ones like IITs.
Hey guys!
This video is my attempt at designing Tinder. Points to keep in mind are: clarify requirements, design the basic system, modify per feature, use concepts learnt earlier to meet requirements.
Also, if you are preparing for a system design interview, try interviewready.io 👍
Cheers!
love ur videos,,thank u my friend
We could setup webRTC data channel based messaging thus avoiding the message being send via a server. And it's better for privacy.
Ladhki
WebSockets - create TCP connection to server, and keep is as long as needed. Server or client can easily close it. Bidirectional communication - so server and client can exchange data both directions at any time. It is very efficient if application requires frequent messages. WebSockets do have data framing that includes masking for each message sent from client to server so data is simply encrypted.
In addition to sharding what we could also do is to keep a db field called preferences which is a bit set of user interests in different activities. Using that we can then for each shard create a k nearest neighbors chart for this preferences fields which should give us users nearest to us which we could then recommend.
Also not totally sure but how about using distributed ledger (nosql append type) i.e. file for storing chat history indexed by hash of both users in chat. And a message queue that is used for transporting any new message typed by any of the users?
Would love to hear your thoughts on above.
Its inevitable that this channel will be successful. You got the goods man. Intelligence, eloquence, and the positive energy. Thankful for your work, will definitely pass the channel along
Thanks Victor!
No Problem Gaurav SenPai!
SenPai! 🙏
Datte Bayo!
@Peter Lustig Rather than saying the design has flaws, you should have spent time describing them. It is easy to just sit there and criticize.
Engineering universities need this kind of real world problem solving exercises like your videos, I'm even surprised that I'm learning from this for free.
One thing to remember about databases are they aren't storage wise different from files. What a database provides you is multi user concurrent access without corrupting your data and acid aspects. In theory there's hardly a case where you'd put a big blob in your database. Most often just use paths to big files and keep those paths in the database table. The paths could be on an NFS or whatever distributed file system there is. Your NFS can then deal with geolocation etc. You could create read only mount points across multiple geographical locations for the same file. I'm just leaving this here for anyone to discuss these points further with me. Great video btw!
How will NFS deal with geolocation?
I can watch your series on System Design with more interest than most of the web series coming these days. Please keep up the good work!
this is so relatable man
kudos to you gaurav sir for this wonderful playlist
17:48 - A good use case for an image service would be when you need to send different resolutions or sizes depending on the client (e.g. mobile, desktop, print etc.)
Mostly tinder is phone only. So phone only priority is considered by default. But good usecase.
@@man904u yup, but a good use case would be for profile picture (shown while chatting) vs profile picture shown as a card when swiping - same image but different resolutions.
The ear to ear grin when he talks about getting matches, makes this video more interesting
I appreciate the content from all the RUclipsrs out there helping us with System Design and DS&A but I have to say, I feel like a lot of the younger ones are just doing it for monetization or popularity. Your enthusiasm really sets you apart man. Good job finding something you enjoy doing. You seem like a good dude.
I am SO happy I found this channel! I'm watching this content like a Netflix binge watch!
you really deserve the appreciation !
i never ever saw a person discussing these complex concepts on youtube in a simple and fast way.
Thank you 😁
"if your interviewer gets too deep into cryptography [grin] best of luck" haha that was awesome lol! keep up the great videos!
Not only the content is of high quality, ur passion and energy make this even better. Really like ur videos!
I have seen the video 15+ times and everytime I think I have to watch it again lol. Kudos to you!
Gaurav, your channel is a gold mine. What an amazing engineer you are, thank you for your content!.
The way you teach the stuff it directly stores in my memory called Highly impactful you are Thanks a ton
if a developer can't get matches on tinder, he creates own tinder. ;) - I just want to say thank you for your hard work. Very useful content. Keep up the good work.
I am doing the same!
He'd still be alone in the system 🙄😞
I have been following this playlist Daily like 2/3 videos per day. Loving the content. First System Design is Tinder, and felt interesting already. Thank you for this playlist.🙌🙌
Fantastic!
Nice to see a smart young man give a smart lecture. Well done.
Thank you!
Absolutely love the video! I don't know when will I use these concepts but just getting to learn about it is itself an experience. Hands down the best tutor!!!
Thanks!
Great video :) One comment - To implement recommedations, we could build composite indices on the profile table which can index on, say, (age, location) in traditional RDBMS databases such as PostgreSQL. It's slightly misleading to say we can only have indices on single columns.
Nice tip!
Also you can have multiple single indices on a table (not referring to composite indices here), even though they wouldn't be very helpful in this case, but it is possible. Great video, nonetheless.
Apart from DB indicies, recommendation system can use quad tree kind of in memory lookup that saves userids per location. So all users near by area would be in a single leaf node of the quad tree and can easily suggest people for matching.
Once I know that I am going to have a system design interview, this is the first place I know that I must come.
Not to copy the answer, but to understand how you came up with the design, the process of thinking, the required knowledge that I might miss; what should be used because of what and so on.
You made it easy! Probably some things I feel like need to be discussed: Storage and bandwidth requirements; probably authentication and authorization can be services to note; load balancers; system interface definitions eg. acceptMatch(userId1, userId2).
I wish I could have watched this video 5 years back. I have started learning in 2024 and its awesome to see this one, thanks @gkcs for this amazing learning content;
Thank you!
Great video. Learned a lot :)
Can you please explain these topics separately:
1. p2p | web sockets and protocols | client server - when to use what?
2. managing sessions - possible solutions
3. design system that involves real time location tracking like maps or uber?
Interesting use cases Arpi! I'll try to get to them sometime 😁
@@gkcs thank you very much! Looking forward to it! :-D
@@ArpiShah1911 do u know about redis?
Gaurav you have the best resources on internet. So much learning in just one video. Your attention to what, why and pros and cons is simply awesome. I have learnt a lot from this channel.
Great video! One particular part that I think goes beyond what was mentioned in the video is the part about sharding the database based on geolocation of users. Let’s say we create one DB for each radius of 50 sq miles, the problem is that some people live on the edge of that 50 sq mile circle, so only a portion of that dB would be relevant for them. Also, certain 50 sq mile radius may have multitudes more people/usage than others. I myself am not sure on the ideal solution though, would love to hear people’s thoughts.
In that case, shard further?
at one point you have to make a merge query
and based on use case you can come up with own method to divide data
I'm a 1 year old self thought dev i love every aspect of developing apps and websites.First time watching your vids. Your explanations are in-dept and helpful. This is also my 3rd or 2nd youtube comment ever and you are fully deserving of it! Will be a regular on your channel from now on! All the Best and Happy New Year Everyone!
Thanks Kevon! Happy new year!
Great video, just a couple points I'd like to add:
1. Your gateway does not need to hit the matcher service every single time before client 1 sends out a message to client 2
2. Last I checked, Tinder uses an ELO system (popular games such as CS:GO do so as well), that influences the kind of profiles you see, based on how many people swiped left/right on you. So it's vital to store that information too.
Is storing left/right swpie information is only useful for ELO system? If we don't store that information, how do we support matches? A right swiped B, and later, B right swipes A. At this point, how do we know it's a match?
I'm system designer, but your channel helps me to grow up in domain, Great video
RE Recommendation Engine: you can definitely have a composite index on multiple columns. The B-tree can be searched on the first column, and each of those matches will also be structured to allow further B-tree searching on the next column, and so on.
There is a huge difference in your channel, you really shows that you really love to teach and enjoy as well, good luck Gaurav, really love your channel.
Great content! Once the basic design of segregating responsibility into multiple services are done, Could you please also talk about how to meet non-functional requirements by introducing concepts in the same design, like
- Finding out single point of failures
- How to scale individual services and also the databases
- Introducing load balancer
- Caching if required, etc.
Exactly even i was looking for something on performance, scalability, failures, and caching questions which I got in my interviews..please answer them too that will be really helpful.
I don't subscribe channels but the moment he starts talking I hit the subscribe button.
Hey Gaurav, just wanted to say big thanks to you. Recently, I was asked the choice of storing images bw dfs and sql table in a design interview. Having watched your videos before, I nailed that discussion. There were lot of other stuffs too in that interview, but it was super helpful to have seen your videos before. Thanks a lot for your time and efforts.
Apart from the great stuff, I like how candid your videos are. White boarding it all live and not using presentation with animation is also what keeps it close to the real experience of an interview.
Thanks!
Thanks so much for your awesome systems design video. You really provide the shining part of design, which is taking customer use case into consideration and go from there, and also break down web service/application layer/db & DFS when comparing different approaches and see which one fits the best.
😁
Add few more things:
1. Discuss about CAP while designing a system
2. Partitions should be on the basis of active users and their behaviors not with geo
3. Think about the caching (Distributed and in memory)
4. Think about static and non static resources
5. Messaging system (even for chat too) or event based approach should be considered for all offline works
6. Should discuss quite quantitatively (numbers make good impression)
"A lot of these might sound like jargon, so ... STUDY!"
Perfect! xD
omg your videos are so interesting and fun. I have 0 to do with any of this, I am in finance. Yet I've been binge-watching your videos for days. Thank you!
Thanks Varvara!
Thank you very much for this video.Its really amazing. I know you approached this video more from an interview perspective so you can only touch on high level points, but please think of making a series about each service so that we know how the real world implementation works and how we need to approach it at a more detailed level. I liked the BLOB vs FILE talk. Keep up the good work.
Your positive energy is highly motivational and i am taking system design very seriously thanks to you!!
18:53 the pain of not matching with other client in real life
Thank you! That's a really useful video as a Product Manager I am learning a lot about how to correctly design the systems with my developers and other colleagues.
Thank you!
I love the excitement and energy you talk with! Made me excited to keep listening haha
Great explanation with energy teaching providing the Awesome learning experience .
This place is one of the best place of content ever found on youtube
Thank you!
That was one really well-thought out video. Is it possible for you to talk about a few prevalent DBS technologies(NoSQL etc) and how they are used in System Design in a future video? Many of us CS undergrads have limited knowledge in DBS; only upto RDBMS. Cheers!
I'll try and get that next 😋
Thanks for explaining the concepts so easily.
I left with two questions after watching the video. Hope you might be answering someday if you see this comment.
1. How to store the users those have been liked by someone (right swipe and other like features e.g. Stars, super like etc) and the same with left swipe as well.
2. How the system calculates the distance between the users by only using the location that is stored into the database. I mean what is stored into the database for the location and how it compare this value with other locations.
Thanks in advance. Cheers!!
You're like Tom Holland's Spiderman who became a software engineer.
Love your enthusiasm, watching the series again and trying to co-relate it to what I have at work.
This was a really fun exercise. I tried to put down my thoughts and then watched the video.
Keep making such Architecture videos.
I come back to your videos very often to brush up my understanding. Just because you make the explanation so easy going.. #Respect
Great one Gaurav.
Keep it up. You are really helping out people like us who are beginners in this field but want to excel. Always remember people watch your videos, so keep posting amazing content.
Thanks Sitesh!
Hi Gaurav. I understand many new concepts from your videos. You explained them so well. Now I got the job I was trying to get. Thank you so much!
Hey Gaurav, amazing video. Just one suggestion that the , should be probably a NoSql db so as to store multiple connections as a list or a hashSet for each userId, since this would not be possible in a RDBS if userId is a primary key. Similarly for Matcher service where each userId can have many matches to track. So something like HashMap of Lists/Sets would be better I guess.
Secondly, could you please explain a bit about how to decide when to make a new service and when to continue using same service? The service partitioning is not intuitive to me yet. Thanks
You can use RDMS as well by keeping user id as an index rather than a primary key
yeah since connections are unique, connectionId can be a primary key and user id would act as a foreign key.@@anshulgupta252
Nice video. I didnt know anything about designing TINDER application, but after watching this video, I think its easy!
Why did you smile when you were saying "matches" :) ?
I wish all teachers are gifted with teaching like you! (I appreciate all teachers but man this is so clear and good
Thank you so much, really appreciate it! It’s a good inspiration for me to start thinking an application from the system level of view. Well done!
Thanks 😁
Gaurav, as per me, for storing multimedia, Object storage like S3 is most cut out.
1. It gives us high scalability, which is harder to achieve in File system.
2. It is cheaper as you have various access tiers. Just put most frequently used data in Standard tier and rest in cold tiers. No such segregation in File system.
3. Object storage offers durability. File system is hard to replicate and considerable manual effort is required.
What is your take on this?
At 5:09 , ("In Direct messaging what kind of questions you should ask"), I thought we about to get chatting/dating tips :P
Hahaha!
As usual, great video! I started watching your videos to get ready for interviews, but I keep watching your channel just because I love how much a learn in such short videos. Thanks for sharing your awesome work :)
Thank you 😁
TCP...TCP.... Happy Happy :):)
If you do match, things looking good now, thanks to XMPP!!
Hahaha!
Nice video this is awesome! Please make more like this. Some ideas.
1. design a distributed web crawler
2. design facebook messenger
3. design a ride share service like lyft of uber
4. design a restaurant/food app like yelp
Love your video! feeling a bit dedddd at 5am, but your enthusiasm at 18:20 immediately lights me up lol .Good job man! And thank you for such good content and detailed explanation.
Gaurav "Enterprise" Sen came through clutch with the sys design prep!!!!!!!!!!!!
Amazing video Gaurav, I am now suscribed. Maybe the best channel about System Design.
I would love a system design video about RUclips. It probably has a challenging system design and there are no videos in any channel about this, so can be an opportunity for you.
Best wishes!
Thanks! I'm looking into the video content server system design right now
Will have a video out, hopefully soon 😁
Look forward it it as well. Thanks !!@@gkcs
Thanks alot for such important and interesting videos. You explain critical things in simple way. I also interviewed by you at uber but that time I didn't recognize and also not cracked the problem you asked. Thanks alot for system design videos. Keep posting on new things.
I thought that I watch the video with 1.5 in speed !
this is the first time I watch a video in youtube with 0.75 in speed
Before watching this video I thought that I had cover almost all the system design part, and now I realize that its a big lie;). Learn't a lot from this video. Thanks, Gaurav!. eagerly waiting for the next one.
Thanks Prateek!
Hi gaurav!!! I super super love this video, I think this might be the best starter into system design ( maybe u can change the title of this video, eg [for beginner] ) :) waiting for other amazing videos!
Thank you!
Thanks very much. I've been learning Django through tutorials and books for couple of weeks now and i couldn't understand the structure behind apps. This makes everything much clear. i'll probably watch all your videos.
Some additional insight as to why you would want to shard/partition by location:
For example: Consider all users from Mumbai are stored on one shard. In this way:
a) If you have to recommend users to someone from Mumbai, you are running your query only that shard. Hence, you are querying much lesser amount of users (Users all over India vs Users only in Mumbai). Less data to process = Faster query results.
b) It will rarely happen that you have to "join" data from multiple shards. Users in Mumbai would only want recommendations from Mumbai.
Great, those are additional points 😊
Recommendation of profiles is based on location(max 100miles radius). Tinder uses Elasticsearch that can have multiple data nodes -geoshard. If we can quantify the load of each geoshard (“load score”), the load score values for all the geoshards should be roughly the same.
1st Solution (not efficient/used): Divide world map to ‘n’ shard squares. But near to ocean location, big cities, load will be different.
2nd Solution: Balance geoshard using load score criteria - • Unique user count• Active user count• User’s queries count in an hour• Combination of the above. Formula used - “min(standard deviation(Loadscore1, LS2, …)”.
Shard size is determining considerations - •Geoshard migration(users movement)•Querying multiple geoshards(so shards cannot be too small)•User density(based on city location). Tinder currently uses 40-100 shards based on calculations above. S2 Cell & Geosharding Algorithm - Hilbert curve.
Reference : medium.com/tinder/geosharded-recommendations-part-1-sharding-approach-d5d54e0ec77a
Gaurav, I have few questions regarding recommendation service:
1.What is the advantage of horizontal partitioning over indexing using location . Isn't it the same? both essentially
reduce the search space?
2.How really cassandra/nosql has an edge over sql using multiple indices?
3.Where are we storing the matches which are already shown to user? if we query based on location and filter don't we end
up showing same matches again and again? I was thinking of design where we store matches like items in newsfeed and show items
after the last shown item. This also improves efficiency i.e we can use sql database and pre-generate matches and show them when
client opens the app. Additionally for every new user created, we can see who all users' preferences he matches( and distance)
and add it to corresponding user's recommended matches.
Hey Gaurav, great content! Really learning a lot from your videos. Had a question about sharding on location. How would you handle when a user opens the app in a new city or country? Would you have to find that user, delete their record, and move it to the new location's shard?
Enjoy the video. He is so handsome and cute. I would literally reply to watch his smile. Of course, the content is great !! learning a lot !
You are so kind, thank you!
Hi Gaurav, thank you for putting up this video. Very helpful and clear. I am into my learning phase about system designs.
I need to ask why we haven't covered topics around the availability of the app across different geographies, scalability, fault tolerance, and load balancers in detail here. Love to hear your thoughts on the same.
This guy is amazing. Just found this channel today
Thanks for the video,
I didn't understand the part that , (select * nightmare) @10.25, and you also say, select * is used often, can you plz elaborate a bit more ?
Imagine you want to select the age of a certain user. If you store the image along with other info about the user and you execute by mistake select * from users where username = 'whatever' this would pull the image as well thus generating useless network traffic and consuming more memory than a simple select age from users where username = 'whatever'
🙏🕉️ जय श्रीराम!
Thanks for showing big picture and mapping various units with technology suitable for them.
This is really good, thanks for making this video. Waiting for more new content like this :)
Thanks!
This videos are really good. It got recommendation randomly but need some content like this.
I forgot to right swipe today. Thanks for reminding me. 😄
Hi Gaurav, excellent video. I have a couple of questions/comments:-
1) Multiple gateways behind a load balancer similar to whatsapp? or do we not see those many requests coming in? If we do have multiple gateways, the sessions service will also have to store the gateway information on which the connection is open for a particular user. Or is this implied?
2) Wouldn't it be better to have a "like" service? This covers matching as well: it is 2 people liking each other and then we notify both the users.
3) From what I understand, and it could be an overkill for a 1 hour long interview, tinder also ranks profiles that it presents to users based on matches that these profiles get etc., so we could just make a remark that a general ranking algorithm could be used in this case.
4) Not a comment on the video, but for clarification for the other viewers who were wondering why XMPP over WebScokets for chats/notifications: XMPP is an application layer protocol that you can use on top of WebSockets/TCP/HTTP connection. WebSockets is a Transport Layer protocol.
35:09 saw on repeat xD anyhow, awesome videos dude! This channel is going places!! :D
This is more interesting than some of the web series out there 🙏🙏😅
"A lot of this might sound like a jargon to you. So?? Study."
- Best Motivation Ever
came down to look for this comment:D
Man got me pumped up to build systems! Great work !!!
Thr youtube algorithm has lead me to your video
Great content! There is a place where I think there was something wrong - please correct if I have my understanding wrong. Sharding and horizontal partitioning are different. Mixing the two is wrong, sharding means partitioning the data in some way(range based, or hash based etc.) and keeping them in different nodes/servers and horizontal partititioning means also partitioning the data but keeping them in the same database, just in different tables.
Apart from this, great videos and great enthusiasm, you make learning new things fun. :D
That's right, good point 👍
Hi Gaurav. Great video. How will the database shards get updated, if a user is navigating through different cities. userA belongs to cityA, has the user data stored in shardA. If he moves to cityB, his location has changed. Should the user data now move to shardB?
Is it a good idea to keep updating user data across location based shards, as the location of the user keeps on changing?
You can have a look at the geo sharding that Tinder does, they have a great blogpost on it. 😁
please add videos for low level designs also.... ur a life saver bro... long live gaurav...
Thank you so much Gaurva! this is very helpful to go through separated services such as profile, chat, matches and recommendation. And I am glad you mentioned p2p in chat. But how do those services interact with each other? How to handle dB fail. It's not very clear
Hey David, thanks! I'll be taking some videos on master slave architectures and API construction.
Till then, have a look at the videos of "Single point of failure" and "Consistent Hashing" for ideas 😁
Only one word for you man
...GURU..
why not keep a combination key for all the 3 fields viz. age, gender and location and then, index on it ?
apparently, you don't need to pull all the recommendation in one shot, you can do incrementally, like a group of 10.
joining all three and make index would be overkill in this case, i believe.
Here i get recommendation of the best channel, thankyou!!
Great work 😍
System design of one of the real time online games like Call of Duty or Clash Royale please.
Great stuff Gaurav. You explained the complex data flow and design concepts in a logically simple manner. Thanks for sharing!
:D
Hey Gaurav, This is really a nice video. Can you make a video on system design of online judges like codechef, codeforces. I am really curious about it. Please focus on high code submission volume, if you make it.
Thanks a lot!!
Great great videos! Some quick questions:
1) When deciding between a DB or Filesystem (as to where to store images), why mention mutability as an important criteria? How do DB's and Filesystems differ in that regard?
2) You said "File systems store large objects separately. DBs can do that via vertical partioning". What did you mean by "File systems store large objects separately"?
3) What did you mean by "I'll set up a CDN around my File system"?
Thanks again
What I understand was:
1) when you save information in a database you usually make some updates i.e, change numbers, dates etc, in this case, if you want to change some information about the image you should change the bits of the blob and does not seem very relevant, you usually replace the whole image and that is kind of the same as replace a file.
2) I guess he refers to a file is a single entity in the file system but in the database you can have multiple columns and one of them can be your blob, so the information is related
3) CDNs allow to get content faster for an user, suppose the app server is in Japan and you are in USA, when you make a request, this has to travel to japan process the request and send a response, for this request can reach japan it has to wait some time because the light speed (what I mean here is that your request cannot travel fastest than the light speed) and well this translate in some latency, but what happen if you put kind of a server in Cananda? then your request has to travel less distance and your latency is going to improve, this "kind of server" is a CDN.