A good to add would be message queue. As we know, when we update for example a big image, user shouldn't be waiting there for a long time. Also, if we write the huge chunk of data directly to to the image storage, it will explode (storage usually won't have such a big RAM)! Hence, a good practice would be push it to the queue with a publisher, and have multiple listeners to update these images data (binary formats) into the storage.
@@lagneslagnes and even if it did, where you would store 5mb image? In queue? Azure Blob/S3 are by design much more scalable than any queue that can handle 5mb binaries within a message.
Unreal video! I'm relatively new to Systems Design but this was easy to follow and aligned with a lot of other good information I've been finding on the web. Thank you!
After scrolling through 100s of SD videos finally I found a way to approach the questions in the interviews. Your step by step approach from start to finish was really helpful. Thanks
Great job.. Thanks for the video. I think we can add the following: - Elaborating the DB sharding and partitioning in the system. - DB Replication, by adding redundant blocks for database based on master/slave relation, where the master is for write, and slaves for read. - Talking about the data searching, like considering to use Elasticsearch for example. - Adding a block for CDN to clarify where it should to be in the system diagram. - Adding a message/uploading queue - Making an estimation for bandwidth
This video is great introductory resource, however it is a little bit superficial. Anyway was great watching and looking forward for the other videos on this channel!
I think this video is really very generic, so much so that this design could be applied to any problem. Would have loved see you deep diving into details of any one component in this 30 min video. For e.g is database sharing required? Do we need noSQL databases? how would the fan out process work? How to handle influencers who have 5 million followers? Disappointed to see those details missing here.
@@Damouse007 I think recently I just saw a architecture mock interview for Twitter in which they go over is designing the seed which was actually genuinely helpful it’s a very interesting approach to modeling I would probably look that up if you’re interested
This was an enormous help in my interview today! I adapted the flow to talk about the specific requirements. Calculating scale and talking about the data model with a segue into the overview diagram was very coherent. Most of all it produced every possible opportunity to highlight my experience with the components I presented, which is the overall goal.
@@tryexponent I moved forward in the process, and the SDI was a large part of it! I discovered through the interview process about their product and the responsibilities of the role that it was not what I would like to pursue, but that's a different story.
Nice pace of the mock interview. However, the diagram can actually fit most of the system. Shouldn't we focus more on how to fan-out photos, generate news feed, etc, which are more specific to "designing Instagram"? I wonder if we only talk about the high-level things and teaching the interviewers those basic architectural concepts (read-write, load balancer, s3, CDN) as this video does can really ace the interview.
I kept waiting for the actual system design to start. Then the video ended. What this video covered is basically the skeleton of any system where you upload/view single files at a time.
Yes found it to be really basic. Even I was looking forward to fanning out photos as well some replication to the dB as only one dB instance is used for both read and write operations for 10 million users
@@BrijeshBolar79 what does fanning out photos mean in this context? and any good resource on db replication / sclaing? thought that part was lacking definitely.
I see that in so many videos on RUclips. 0 specifics about the actual system, just generic LB, DB schema, object storage stuff. I really wonder if interviews are buying this….
Thanks for the video! Really enjoyed it as it gives a great exploration of the problem. However, I think the candidate needs to further explore the scaling of the data and provide estimations about read/write requests per second, traffic and storage needed that we need for instance for 10 years, and what to do when the DB is full that we need to scale vertically (as we you have gone for a MySQL). Those information will help out understanding the overall scale of our system and brightens the scene for the interviewer that we are set with with expected traffic and can manage to handle the requests.
The feed generation part needs a bit more detail IMO. That's the most challenging part of this. Would you generate it on read or on write? Or a combination of both? Also, what if it's not a linear timeline of events but needs to be ranked? Also, details around comments and replies?
Very good video for very novice engineers looking to learn about systems. However, there are big gaps here when it comes to scaling, ie. using a relational database (which also decreases performance) and having a synchronous system.
I had some questions about this, too. I can see the rationale for an RDBMS but from a scaling perspective, it would surely run into trouble. I was hoping the video might discuss how failover to another Region. The topic of Failover would surely come up in an interview.
I am not sure that a relational database is the best choice here. There are often relationships in data but sometimes you need to determine the right trade offs in terms of scalability. Though I appreciate that a relational model can be designed to work at such scale (read replicas, sharing etc), I would consider using NoSQL solution for the media metadata api, and a graph database for the activity feed.
Actually I think a write-through cache, or cache-aside to the write service will be easier to generate the feed. Bcs the cache server will know what content is pushed recently.
So will the objects themselves who are stored with time and location data. Any component dealing with those objects will have that data, not just the cache. You want the use case to be “new to the user” not “new to the system” anyway.
Hi, great video, one thing to take into account is that Instagram uses Cassandra as their primary DB, because of the partitioning, scaling issues relational databases have. While SQL databases are scaled vertically, NoSQL databases are scaled horizontally (scale-out across commodity servers). Also, NoSQL databases can store relationship data, they just store it differently than relational databases. So by saying that I wouldn't agree that a SQL database is the best choice here, at scale NoSQL is much faster, cheaper than a relational database
You can scale sql databases horizontally using sharding and a decent shard key. Cassandra does that automatically for you if you categorize well your column families. If you mess you data model, you will have a bunch of hotspots on your cluster and will face the same problems of vertical scaling. The technology alone won’t solve your problem if you don’t know how to use it well. By the way, before Facebook bought Instagram it already scaled horizontally and used Postgres, which is a SQL database.
It's all about how not to answer your design question! I mean, really this is how you design an instagram, or in a more generic sense any feed system? Note: These are my personal opinion. I would rather say these are the points that I was expecting from a 5+ year experieced dev perspective. He never talked about "post" and related stuff. Is instagram all about uploading your profile pic? Everything you upload would be a post and you may add image/file/feeling/re-share etc. Where is your post table and id? Are you going to use integer for storing your user-ids? Think again! He also did not mention the space incurred from text data, 1 PB of space would just for images. I guess we can use geo-location based storage. Notification being one of the most important component in any social-media platform. It should have been covered in requirements. I was expecting him to cover the latency in uploading images and the wireframe for image upload and render. Last but not the least, the re-share functionality! How to deal to re-post/share a post. I'm not being a pessimist, just thought to put my points. Apologies. Nice work btw
Great video, thank you. Good example of CQRS in action. For further improvement it would be a good idea to use another DB for feed generation, because access patterns will be slightly different: OLTP for user actions and OLAP for feed (maybe some recommendation system under the hood).
SQL for dynamic complex queries, ACID properties (some NoSQL also supports this). Relation or not is really not a strong reason. You can do the same in NoSQL with different tables using seperate requests or the same table using one request (Adjacency List Pattern). NoSQL for easy read/write scalability, high availability.
I'm far from an expert, but I disagree with your reasoning for choosing a SQL (relational) database. All data that's worth representing is relational in some way. That doesn't mean that SQL dbs are always the way to go. The types of queries you described can be performed efficiently on NoSQL dbs with smart use of indexes and would offer better scalability for a social platform like Instagram. No mention of Graph representation either, which is ideal for representing many to many relationships.
If I'm going to rate this mock interview from 1 to 10 I will go with 3. every matter was considered at it basic part. if FAANG interviews are like this i may be the director of tech at one of those companies!
Yes. Graphs. Graphs database is best to store all the user's information & relationships between them. The response time for getting the relationship as well information about the user is so very fast with the Graph database. Graph databases are best suited for this type of many-2-many relationships as compared to the traditional relational databases. I do not agree with this guy saving user's details in the relational database. But basically, you must need all diff kind of databases for diff-diff purpose in such a large scale application. The architecture he designed is a common architecture for a simple mobile app. Have a look how complex the architecture can be for Instagram/Facebook. github.com/codekarle/system-design/blob/master/system-design-prep-material/architecture-diagrams/Facebook%20System%20Design.png
Yeah, thats true. This tutorial should be taken as design of social media app in general. Fb does not compress so they need the storage. Whereas instagram needs to keep large files in memory for compression operation, so its a different challenge. Thats what I think
Great video, excellent explanation. There are two aspects that I would do differently, wondering what everyone else thinks: 1 - Database is shared between multiple services, which introduces coupling between them. 2 - I don't see a massive benefit in having separate services for read and write paths, I'd just have one service with a cache behind the reads.
There are relations between data, so we will use a relational DB system. This is a very debatable statement, since there are always relations between data.
Hi Stepan! While relationships between entities does not necessarily mean that a relational DB system should be used, it is the best choice to do so. Hope this helps!
Learned a ton from this. Never have had a Systems interview and I am a new grad so my potential onsite coming up has a design portion so trying to grind through these and learn a ton!
I am a new grad as well and have a sys design interview coming up. How deep did you go in the system? what all did you discuss and what all did you leave abstracted?
This is a very basic attempt at solving this problem. Though I appreciate you putting out this video for us to view - it doesn't cover many topics. For cases where you need to optimize your Database, you might need to go into sharding - sharding being a beast of topic in itself you should have further explained the shard key and the reasoning behind it. The meat of the problem statement for designing either twitter or Instagram is the optimisation of news feed. If you make a block around it and never explain it in depth then it doesn't serve any purpose. Further, there is fanout process or polling process to get new feeds and special handling when a celebrity uploads a post.
Good for a basic understanding of a systems design interview. However, it is a quite incomplete example. How do you make 1.2PB of data fit into a single "Metadata DB"? How do you shard it? Based on what? You didn't use much of the data from your back-of-the-envelope estimates in your design. Didn't specify much on API interfaces. How do you update your cache generated by the feed generation service based on new posts from users you follow assuming you always want to see newer posts first? Lots of very basic assumptions and half-baked solutions.
Overall, this is a good demonstration of what a systems design interview could be like a less experienced candidates, but Google or Facebook would likely want to see more than this if you're more senior. For a variety of reasons: your implementation language can greatly effect a design e.g. stateless vs stateful, traffic is not evenly distributed throughout the day, if they shipped a service you'd hit 10M users in hours or days, etc. All of which would require you to handle writes(and to a lesser extent reads) differently than what you've shown in this design. Not shitting on the video, I think its quite accurate, and a good introductory video.
so, there's no need to go into more depth for a new grad sys design interview right? I have one coming up and am super confused as to how deep i should go, cuz i don't have knowledge about most of the details of a system
@@gsb22 any more resources you would suggest I could refer for an abstracted overview of SD? which would be not too complex, but detailed enough for a new grad interview?
@@SumedhSen97 Google System Design primer github, but this would be too deep. I would suggest watching generic vdos to get hang of it. You seriously aren't supposed to defend single leader or leaderless replication. Checkout Gaurav sen and all vdos on YT
Hi! Why not explore Hybrid database implementation using RDBMS and NoSQL databases? How would this system handle the bottlenecks when start scaling? What would be the response time? Also, implement queues between the cache and the database could help. It's a clean implementation, great quality, it's a possible approach.
Hey Gustavo! Good questions. In a sense, we are using a hybrid approach with a relational database plus a key-value store like Redis, but you could use other types of RDBMS and NoSQL systems, too. In reality, Instagram uses Postgres + Cassandra, with some advanced indexing and sharding strategies which you can read about on their blog! Let us know if that helps! instagram-engineering.com/handling-growth-with-postgres-5-tips-from-instagram-d5d7e7ffdfcb
What if you're asked to build a Logging API? Then it would just be client-side right? Would you go into details about servers and databases if it's mostly client-side?
Excellent Video! I was wondering what rationale should an interviewee give if asked about the scalability (specifically, write scalability) after choosing a relational database model?
Great video! Thanks for creating this content! I thoroughly liked the half hour video. If one thing I could suggest, would be little bit more deep dive into the design - API, sharding, replication etc. Also in case of read, you are connecting the mobile device directly to Object Store, then why not in case of write as well? The device can directly write to object store, and just push meta data through the App server (write). Are there any downsides you were thinking?
I might be wrong, since I'm also not an expert in system design, but my reasoning would be as follows: We can introduce a CDN between S3 and clients, making serving static files much faster. But I'm not sure if we can speed up the upload process (from the client to S3) in a similar way. Also writing directly to the S3 bucket just doesn't seem right to me from the security perspective
@@stanislausaprankou3495 you can request for a signed url from the server, and use that to upload directly to S3. Passing all those large files through the server is going to add so much unnecessary cost.
The title is a bit misleading. Although this video is a great explanation of how to design a system such as Instagram but this is far from a mock interview. It does not capture the essence of an on-site interview with an interviewer asking you to design a system where you are responsible for getting a buy in of your use-cases and requirements of the system by the interviewer. In this demonstration the requirements are perfect since they are made up by the only person attending the interview. Again this is great for designing a system and how to speak out loud about your design but not a mock interview. There is a big difference.
The one thing that I would argue here is the DB choice. You are creating 1.2 PB of information per year, storing that in SQL db does not scale. You will rapidly find yourself sharding and fighting with key hashing. Also you will need a master slave architecture to handle the read heavy. So basically you will have a sharded DB and each shard with a M-S architecture. That points out that probably the best choice is a NoSQL DB, in this case like Cassandra (Column family db). These kind of DB were created to scale much better than a SQL and will provide you with High Availability and fault tolerance, sacrificing consistency.
Thanks for sharing this amazing knowledge. I could not understand whether the generated feeds are stored in the cache based on each user or not. Because I think, storing feeds based on each user, causes Cache faces lots of hits.
This is one of the shortest and best system design video I have came across. although you could have talked about CAP theorem while discussing the tradeoff of sql and Nosql no worries though. I would really appreciate if you could further discuss about Class digram and relation among them and APIs which should be exposed(API design/Class) in your upcoming system design video. Thanks again for the video. Keep up the good work buddy!!
Great video , knew all the components but to put it all together you did a great job, one small note caching layer is generally implemented at CDN or Load balancing layer so it is closer to where the users are and prevents backhauling of traffic to where the servers are
Once the endpoint URLfor the image is figured out by making the search in MetaData DB, would a new request fire of again through the client to get that image through S3 or CDN? Or would thius happen in same request, as in we have an additional MicroService which would make a call to S3 or CDN and give back the image in same req?
Hey @Ritesh Thombre! Typically the client will receive the first response from the web server, then make separate requests to the CDN for the image/video assets. This allows the client to optimize when to load these images for the best performance (e.g. when scrolling)
@@tryexponent Agreed. I guess, just a thing about design choice. Ours is a mini banking application and we make search of a customer's eStatement through its Metadata, get the DocID and generate the URL. Then hit the cloud storage and give back the document in same request. What you are proposing looks like will add another network call but that's fine. I think its a matter of design choice.
@@tryexponent Whimsical is an excellent tool which I plan to use in my upcoming TPM interview. Thanks for the excellent video, it's a good high level.. wish you could elaborate little more on the "pre-compute" logic for the news feed generation and how it should differ for the "normal" users vs "Celebrity" users because I guess that's an important design consideration as well
Good video, it would help if you can include rest interfaces (not in great detail, but something) as well in the future. A bit more diving into depth would help.
Awesome video, thanks for that. Question: You have estimated a storage capacity of 1.2PB but still decided for a SQL DB. How does that align with our goal to build a scalable system? I was expecting a NoSQL choice here. Please correct me if I'm wrong.
Hey Ash! Thanks for the question. We should have clarified that this estimate is for the size of the images. The database would just store references (URLs) to the images, which can be stored in a storage system like Amazon S3 and served by a CDN.
Liran elisha this would not be sufficient - scalability is not one dimensional. You can read the actual story about how Instagram originally did it at High Scalability. There is a cost to using NoSQL - schema on load versus schema on read. If you care about data integrity and operational familiarity, it’s hard to beat SQL. These are considerable trade-offs worth sharing and discussing during an interview. It’s not blatantly obvious why some hand-wavy NoSQL solution is more scalable. Which part? I don’t doubt that there is a NoSQL solution that makes some bottleneck in the system much more performant, but rather we should go with a choice and be able to defend it and also discuss alternatives and their trade-offs
You just designed a monolithic solution for instagram. Did they really liked that? where is domain model? where are microservices? Api gateway?where is event sourcing? eventual consistency against strong consistency? there is no way you can scale this for 10 million users on one single database shares between multiple services.
Sorry for the confusion due to the misuse of the "=" sign! It was 100TB (0.1PB) per month, but we wanted to find the amount per year which was 0.1PB * 12 months = 1.2PB. A clearer presentation of the workings would be: Per month: 100TB = 0.1PB Per year: 100TB * 12 = 1.2PB Hope this clarifies it for you!
It would be awesome if someone could tell me what whiteboard/notepad application is used in this video. Could come in super handy during a system design interview!
Hey great video Jacob learned a lot. Really liked the way you explored everything giving fairly reasonable and convincing solutions . Thanks for the video again really appreciate the effort.
1. Hour based generation of feeds seems to be impractical: feedback time is very important for social networks. Maybe it should be triggered for users on events (follow, unfollow, upload,...)? 2. Some very popular users maybe should be processed separately cause they generate ton of events for their followers. 3. You didn't estimate any sort of traffic. 4. You didn't say about any nonfunctional requirements 5. You didn't say about how you would scale horizontally on database of metadata. And so so much more.
the API' were not constructed in this design. I think we should create the entities after confirming the API's and doing the high and some level of deep dive
Starting with DB schema, and not with the user flow is a huge "noob" mistake. It gives away, that you just prepared this scenario in advance by just looking at the end result without understanding how to get there. Not to mention that the solution itself is very "junior" -ish. For example, passing a file through "app service" and not uploading it directly to the object storage - is a big mistake, you basically multiplied the amount of traffic that runs through your system by 2 for no reason. Populating cache synchronously on reading - is also a big mistake. And pretty much nothing was said about how to prepare a feed, which is the most interesting and loaded part of the design. And by the way, this is a classic example of a distributed monolith - you tied everything to one single database. 2/10.
"And then we can talk about the API a little bit later"
Never talks about API
I know my ass would not get hired if I did that.
This is a great example of what you should strive to have completed at the halfway point of the interview.
Yes, maybe closer to the 1/3 point.
A good to add would be message queue. As we know, when we update for example a big image, user shouldn't be waiting there for a long time. Also, if we write the huge chunk of data directly to to the image storage, it will explode (storage usually won't have such a big RAM)! Hence, a good practice would be push it to the queue with a publisher, and have multiple listeners to update these images data (binary formats) into the storage.
pushing large objects to S3 won't explode it
@@lagneslagnes and even if it did, where you would store 5mb image? In queue? Azure Blob/S3 are by design much more scalable than any queue that can handle 5mb binaries within a message.
Unreal video! I'm relatively new to Systems Design but this was easy to follow and aligned with a lot of other good information I've been finding on the web. Thank you!
Glad you liked it! Be sure to subscribe for more like this :)
@@tryexponent How can I use this type of whiteboard ? Is it free of I have to pay something to use ?
After scrolling through 100s of SD videos finally I found a way to approach the questions in the interviews. Your step by step approach from start to finish was really helpful. Thanks
Great job.. Thanks for the video.
I think we can add the following:
- Elaborating the DB sharding and partitioning in the system.
- DB Replication, by adding redundant blocks for database based on master/slave relation, where the master is for write, and slaves for read.
- Talking about the data searching, like considering to use Elasticsearch for example.
- Adding a block for CDN to clarify where it should to be in the system diagram.
- Adding a message/uploading queue
- Making an estimation for bandwidth
I think DB Replication is understood for default.
Rest all points are great 👏
This video is great introductory resource, however it is a little bit superficial. Anyway was great watching and looking forward for the other videos on this channel!
I think this video is really very generic, so much so that this design could be applied to any problem. Would have loved see you deep diving into details of any one component in this 30 min video. For e.g is database sharing required? Do we need noSQL databases? how would the fan out process work? How to handle influencers who have 5 million followers? Disappointed to see those details missing here.
I would think for say N-Million followers, that work is either broken down aycnch and its broken across a number of shards in a region.
He spent the whole time on the easier part of the design. He didn't model the feed, or describe when and how its built.
@@Damouse007 I think recently I just saw a architecture mock interview for Twitter in which they go over is designing the seed which was actually genuinely helpful it’s a very interesting approach to modeling I would probably look that up if you’re interested
Agreed, this much detail will not clear a system design interview
This was an enormous help in my interview today! I adapted the flow to talk about the specific requirements. Calculating scale and talking about the data model with a segue into the overview diagram was very coherent. Most of all it produced every possible opportunity to highlight my experience with the components I presented, which is the overall goal.
I'm so glad this was helpful Alex! Definitely let us know how the interview went!
@@tryexponent I moved forward in the process, and the SDI was a large part of it! I discovered through the interview process about their product and the responsibilities of the role that it was not what I would like to pursue, but that's a different story.
@@thatsamorais584 Congratulations!!
Nice pace of the mock interview. However, the diagram can actually fit most of the system. Shouldn't we focus more on how to fan-out photos, generate news feed, etc, which are more specific to "designing Instagram"? I wonder if we only talk about the high-level things and teaching the interviewers those basic architectural concepts (read-write, load balancer, s3, CDN) as this video does can really ace the interview.
I kept waiting for the actual system design to start. Then the video ended. What this video covered is basically the skeleton of any system where you upload/view single files at a time.
Yes found it to be really basic. Even I was looking forward to fanning out photos as well some replication to the dB as only one dB instance is used for both read and write operations for 10 million users
@@BrijeshBolar79 what does fanning out photos mean in this context? and any good resource on db replication / sclaing? thought that part was lacking definitely.
I see that in so many videos on RUclips. 0 specifics about the actual system, just generic LB, DB schema, object storage stuff. I really wonder if interviews are buying this….
@@tsaregrad they're not. You won't get any reputable offer with details RUclips videos provide. These are just for the views.
Loved the video, great explanation. What is the UI/ tool used here to draw for the system design?
It’s called Whimsical!
Thanks for the video! Really enjoyed it as it gives a great exploration of the problem.
However, I think the candidate needs to further explore the scaling of the data and provide estimations about read/write requests per second, traffic and storage needed that we need for instance for 10 years, and what to do when the DB is full that we need to scale vertically (as we you have gone for a MySQL).
Those information will help out understanding the overall scale of our system and brightens the scene for the interviewer that we are set with with expected traffic and can manage to handle the requests.
Its nice, thank you !
1. CDN could have been added
2. Encoding of videos and photos
The feed generation part needs a bit more detail IMO. That's the most challenging part of this. Would you generate it on read or on write? Or a combination of both? Also, what if it's not a linear timeline of events but needs to be ranked? Also, details around comments and replies?
Very good video for very novice engineers looking to learn about systems. However, there are big gaps here when it comes to scaling, ie. using a relational database (which also decreases performance) and having a synchronous system.
I had some questions about this, too. I can see the rationale for an RDBMS but from a scaling perspective, it would surely run into trouble. I was hoping the video might discuss how failover to another Region. The topic of Failover would surely come up in an interview.
I am not sure that a relational database is the best choice here. There are often relationships in data but sometimes you need to determine the right trade offs in terms of scalability. Though I appreciate that a relational model can be designed to work at such scale (read replicas, sharing etc), I would consider using NoSQL solution for the media metadata api, and a graph database for the activity feed.
my thoughts exactly as im watching this
a deep dive into the feed service would be greate. because obviously thats instagram's core feature
Actually I think a write-through cache, or cache-aside to the write service will be easier to generate the feed. Bcs the cache server will know what content is pushed recently.
So will the objects themselves who are stored with time and location data. Any component dealing with those objects will have that data, not just the cache.
You want the use case to be “new to the user” not “new to the system” anyway.
Hi, great video, one thing to take into account is that Instagram uses Cassandra as their primary DB, because of the partitioning, scaling issues relational databases have. While SQL databases are scaled vertically, NoSQL databases are scaled horizontally (scale-out across commodity servers). Also, NoSQL databases can store relationship data, they just store it differently than relational databases. So by saying that I wouldn't agree that a SQL database is the best choice here, at scale NoSQL is much faster, cheaper than a relational database
You can scale sql databases horizontally using sharding and a decent shard key. Cassandra does that automatically for you if you categorize well your column families. If you mess you data model, you will have a bunch of hotspots on your cluster and will face the same problems of vertical scaling. The technology alone won’t solve your problem if you don’t know how to use it well. By the way, before Facebook bought Instagram it already scaled horizontally and used Postgres, which is a SQL database.
Should client be calling to web server before the app server? App server handles business logic, web server handles the user's requests
very helpful, can you make a more mock system design interview!!!????? thanks!
Sure thing!
It's all about how not to answer your design question!
I mean, really this is how you design an instagram, or in a more generic sense any feed system?
Note: These are my personal opinion.
I would rather say these are the points that I was expecting from a 5+ year experieced dev perspective.
He never talked about "post" and related stuff. Is instagram all about uploading your profile pic? Everything you upload would be a post and you may add image/file/feeling/re-share etc. Where is your post table and id?
Are you going to use integer for storing your user-ids? Think again!
He also did not mention the space incurred from text data, 1 PB of space would just for images. I guess we can use geo-location based storage.
Notification being one of the most important component in any social-media platform. It should have been covered in requirements.
I was expecting him to cover the latency in uploading images and the wireframe for image upload and render.
Last but not the least, the re-share functionality! How to deal to re-post/share a post.
I'm not being a pessimist, just thought to put my points. Apologies. Nice work btw
Really wish I had seen this before my second round TPM interview at a major FAANG company. LOL Great video!
Glad you enjoyed it!
Great video, thank you. Good example of CQRS in action. For further improvement it would be a good idea to use another DB for feed generation, because access patterns will be slightly different: OLTP for user actions and OLAP for feed (maybe some recommendation system under the hood).
SQL for dynamic complex queries, ACID properties (some NoSQL also supports this). Relation or not is really not a strong reason. You can do the same in NoSQL with different tables using seperate requests or the same table using one request (Adjacency List Pattern). NoSQL for easy read/write scalability, high availability.
I came here to post the same comment and found your comment. Totally agree. It does not make sense to use SQL DB in this use case.
Nice solution, definitely deserve an Amazon SDE II.
Exponent provides best mock interview videos
Cool really liked it thanks a lot
I'm far from an expert, but I disagree with your reasoning for choosing a SQL (relational) database. All data that's worth representing is relational in some way. That doesn't mean that SQL dbs are always the way to go. The types of queries you described can be performed efficiently on NoSQL dbs with smart use of indexes and would offer better scalability for a social platform like Instagram. No mention of Graph representation either, which is ideal for representing many to many relationships.
If I'm going to rate this mock interview from 1 to 10 I will go with 3. every matter was considered at it basic part. if FAANG interviews are like this i may be the director of tech at one of those companies!
Great stuff. Please post some mock interviews where the candidates failed. Thank you.
Can we use graphs to record follow relation between users?
Yes. Graphs. Graphs database is best to store all the user's information & relationships between them. The response time for getting the relationship as well information about the user is so very fast with the Graph database. Graph databases are best suited for this type of many-2-many relationships as compared to the traditional relational databases. I do not agree with this guy saving user's details in the relational database. But basically, you must need all diff kind of databases for diff-diff purpose in such a large scale application. The architecture he designed is a common architecture for a simple mobile app. Have a look how
complex the architecture can be for Instagram/Facebook.
github.com/codekarle/system-design/blob/master/system-design-prep-material/architecture-diagrams/Facebook%20System%20Design.png
Thank you . This video shows that even hard topic like system design can be taught in a simple way . I like this .
Glad it was helpful!
Great video. Have you considered talking more about the APIs?
Nice video! Which software do you use for the diagram?
most instagram photos were compressed to JPEGs between 150 kb and 190 kb.
not 5 MB per photo.
Yeah, thats true. This tutorial should be taken as design of social media app in general. Fb does not compress so they need the storage. Whereas instagram needs to keep large files in memory for compression operation, so its a different challenge. Thats what I think
Great video, excellent explanation. There are two aspects that I would do differently, wondering what everyone else thinks:
1 - Database is shared between multiple services, which introduces coupling between them.
2 - I don't see a massive benefit in having separate services for read and write paths, I'd just have one service with a cache behind the reads.
Instagram has multiple databases for read and write. They explain the reasoning for this in this keynote ruclips.net/video/hnpzNAPiC0E/видео.html
There are relations between data, so we will use a relational DB system. This is a very debatable statement, since there are always relations between data.
Hi Stepan! While relationships between entities does not necessarily mean that a relational DB system should be used, it is the best choice to do so. Hope this helps!
100TB should be 0.1PB not 1PB
This is a great video and I haven't evenr watched it complete. I really like how the requirements are collected and quantified
Learned a ton from this. Never have had a Systems interview and I am a new grad so my potential onsite coming up has a design portion so trying to grind through these and learn a ton!
Ton?!
doesn't make sense asking a fresh grad system design questions.
I am a new grad as well and have a sys design interview coming up. How deep did you go in the system? what all did you discuss and what all did you leave abstracted?
@@arobot4398 I have a interview coming up that is a system design interview, they are pretty common to be honest
@@arobot4398 most F500s who want to hire anything above entry level often have at least one systems design and architecture interview in the process.
This is a very basic attempt at solving this problem. Though I appreciate you putting out this video for us to view - it doesn't cover many topics. For cases where you need to optimize your Database, you might need to go into sharding - sharding being a beast of topic in itself you should have further explained the shard key and the reasoning behind it. The meat of the problem statement for designing either twitter or Instagram is the optimisation of news feed. If you make a block around it and never explain it in depth then it doesn't serve any purpose. Further, there is fanout process or polling process to get new feeds and special handling when a celebrity uploads a post.
Love the number crunching around 6:00.
Alik Elzin how is 100tb equal to 1.2pb ?
@@_SoundByte_ he then multiplied by a year - 12 months.
We love it too :)
Good for a basic understanding of a systems design interview. However, it is a quite incomplete example. How do you make 1.2PB of data fit into a single "Metadata DB"? How do you shard it? Based on what? You didn't use much of the data from your back-of-the-envelope estimates in your design. Didn't specify much on API interfaces. How do you update your cache generated by the feed generation service based on new posts from users you follow assuming you always want to see newer posts first? Lots of very basic assumptions and half-baked solutions.
Are you sure its 1.2 PB of database data? aren't you refering to the image files? which don't go into the database.
@@IsraelLazoPlus in terms of image it is probably what instagram support in hours.
17:13 one of the most well explained breakdown of monolithic arch vs microservices I've ever heard
Overall, this is a good demonstration of what a systems design interview could be like a less experienced candidates, but Google or Facebook would likely want to see more than this if you're more senior. For a variety of reasons: your implementation language can greatly effect a design e.g. stateless vs stateful, traffic is not evenly distributed throughout the day, if they shipped a service you'd hit 10M users in hours or days, etc. All of which would require you to handle writes(and to a lesser extent reads) differently than what you've shown in this design.
Not shitting on the video, I think its quite accurate, and a good introductory video.
Yes, if you have more than 4-5YOE, this video wont cut. This is basically generic distributed system.
so, there's no need to go into more depth for a new grad sys design interview right? I have one coming up and am super confused as to how deep i should go, cuz i don't have knowledge about most of the details of a system
I'm surprised they're asking SD to a new grad. Yeah, you would be fine as long as you can put correct components and explain them a bit..
@@gsb22 any more resources you would suggest I could refer for an abstracted overview of SD? which would be not too complex, but detailed enough for a new grad interview?
@@SumedhSen97 Google System Design primer github, but this would be too deep. I would suggest watching generic vdos to get hang of it. You seriously aren't supposed to defend single leader or leaderless replication. Checkout Gaurav sen and all vdos on YT
This is a very high level answer
this is functional design, more precise upload/retrieve images.
I just have to comment the sound of your keyboard is so nice
Its simple and compact, loved it, thanks.
Good explanation. You made a complicated problem easy to follow and understand!
100TB = 0.1 PB
Hi! Why not explore Hybrid database implementation using RDBMS and NoSQL databases? How would this system handle the bottlenecks when start scaling? What would be the response time? Also, implement queues between the cache and the database could help. It's a clean implementation, great quality, it's a possible approach.
Hey Gustavo! Good questions. In a sense, we are using a hybrid approach with a relational database plus a key-value store like Redis, but you could use other types of RDBMS and NoSQL systems, too. In reality, Instagram uses Postgres + Cassandra, with some advanced indexing and sharding strategies which you can read about on their blog! Let us know if that helps! instagram-engineering.com/handling-growth-with-postgres-5-tips-from-instagram-d5d7e7ffdfcb
@@tryexponent i think i came across the same article in my own research. totally agree.
Your content is good enough for overview but not depth enough for system design interview
What if you're asked to build a Logging API? Then it would just be client-side right? Would you go into details about servers and databases if it's mostly client-side?
Excellent Video! I was wondering what rationale should an interviewee give if asked about the scalability (specifically, write scalability) after choosing a relational database model?
Great video! Thanks for creating this content! I thoroughly liked the half hour video. If one thing I could suggest, would be little bit more deep dive into the design - API, sharding, replication etc. Also in case of read, you are connecting the mobile device directly to Object Store, then why not in case of write as well? The device can directly write to object store, and just push meta data through the App server (write). Are there any downsides you were thinking?
I might be wrong, since I'm also not an expert in system design, but my reasoning would be as follows: We can introduce a CDN between S3 and clients, making serving static files much faster. But I'm not sure if we can speed up the upload process (from the client to S3) in a similar way.
Also writing directly to the S3 bucket just doesn't seem right to me from the security perspective
@@stanislausaprankou3495 you can request for a signed url from the server, and use that to upload directly to S3. Passing all those large files through the server is going to add so much unnecessary cost.
The title is a bit misleading. Although this video is a great explanation of how to design a system such as Instagram but this is far from a mock interview. It does not capture the essence of an on-site interview with an interviewer asking you to design a system where you are responsible for getting a buy in of your use-cases and requirements of the system by the interviewer. In this demonstration the requirements are perfect since they are made up by the only person attending the interview. Again this is great for designing a system and how to speak out loud about your design but not a mock interview. There is a big difference.
The one thing that I would argue here is the DB choice. You are creating 1.2 PB of information per year, storing that in SQL db does not scale. You will rapidly find yourself sharding and fighting with key hashing. Also you will need a master slave architecture to handle the read heavy. So basically you will have a sharded DB and each shard with a M-S architecture.
That points out that probably the best choice is a NoSQL DB, in this case like Cassandra (Column family db). These kind of DB were created to scale much better than a SQL and will provide you with High Availability and fault tolerance, sacrificing consistency.
Great work , I learnt a lot today and the video was well paced and informative. Thanks
Glad it was helpful!
I loved this video. You explained in such a easy way. i was able to connect everything. Thanks man :)
Thank you for posting this! Really helpful!
Wow - You just earned an subscriber - Excellent pace and detail Thanks !!
Really helpful video Jacob, super detailed in explaining the architecture. Thank you
What program do you to make these designs on your videos?
Why was the choice RDBMS and not something like a GraphDatabase as it makes more sense for querying on larger scale
What’s the drawing tool you are using? Looks really nice! Share a link?
I have a system design interview coming up next week, what tool are you using for sketching?
Which tool you used in the interview can you mention please
Very helpful - Thank you.
The database model wouldn't scale, since we can potentially have n^2 rows in the follower table, where n is the number of users.
Might need a pub/sub service, a CDN (which was talked about but not shown), and a notification service
Thanks for sharing this amazing knowledge.
I could not understand whether the generated feeds are stored in the cache based on each user or not. Because I think, storing feeds based on each user, causes Cache faces lots of hits.
Great video. But why is no one else talking about how great your keyboard sounds? Whats the keyboard build?
This is one of the shortest and best system design video I have came across. although you could have talked about CAP theorem while discussing the tradeoff of sql and Nosql no worries though. I would really appreciate if you could further discuss about Class digram and relation among them and APIs which should be exposed(API design/Class) in your upcoming system design video.
Thanks again for the video. Keep up the good work buddy!!
Thanks Rajesh, you got it!
Great presentation. Thank you :)
Glad it was helpful!
Great video , knew all the components but to put it all together you did a great job, one small note caching layer is generally implemented at CDN or Load balancing layer so it is closer to where the users are and prevents backhauling of traffic to where the servers are
What is the difference between the App server read and the feed generation service?
Once the endpoint URLfor the image is figured out by making the search in MetaData DB, would a new request fire of again through the client to get that image through S3 or CDN? Or would thius happen in same request, as in we have an additional MicroService which would make a call to S3 or CDN and give back the image in same req?
Hey @Ritesh Thombre! Typically the client will receive the first response from the web server, then make separate requests to the CDN for the image/video assets. This allows the client to optimize when to load these images for the best performance (e.g. when scrolling)
@@tryexponent Agreed. I guess, just a thing about design choice. Ours is a mini banking application and we make search of a customer's eStatement through its Metadata, get the DocID and generate the URL. Then hit the cloud storage and give back the document in same request.
What you are proposing looks like will add another network call but that's fine. I think its a matter of design choice.
Good video... Thanks.
Great video, really helpful.
What is the name of the whiteboard software you use for teaching?
Hey Shashank! It's called whimsical.com/ - what did you think of the tool?
@@tryexponent Whimsical is an excellent tool which I plan to use in my upcoming TPM interview. Thanks for the excellent video, it's a good high level.. wish you could elaborate little more on the "pre-compute" logic for the news feed generation and how it should differ for the "normal" users vs "Celebrity" users because I guess that's an important design consideration as well
Thanks Jaydeep! How would you have thought about the celebrity users?
Thank you team!
@@tryexponent Just explored it. It really cool
Good video, it would help if you can include rest interfaces (not in great detail, but something) as well in the future. A bit more diving into depth would help.
The feed is most likely going to be the *crucial* aspect of the design requirement for this scenario, and you didn't even do it.
Awesome video, thanks for that. Question: You have estimated a storage capacity of 1.2PB but still decided for a SQL DB. How does that align with our goal to build a scalable system? I was expecting a NoSQL choice here. Please correct me if I'm wrong.
Hey Ash! Thanks for the question. We should have clarified that this estimate is for the size of the images. The database would just store references (URLs) to the images, which can be stored in a storage system like Amazon S3 and served by a CDN.
@@tryexponent Thanks for the clarification, that makes sense.
it is still makes more sense to go for NoSQL DB for scalability
I have a similar doubt too. Even if we ignore the size, is relational db scalable enough for this kind of problem?
Liran elisha this would not be sufficient - scalability is not one dimensional. You can read the actual story about how Instagram originally did it at High Scalability. There is a cost to using NoSQL - schema on load versus schema on read. If you care about data integrity and operational familiarity, it’s hard to beat SQL. These are considerable trade-offs worth sharing and discussing during an interview. It’s not blatantly obvious why some hand-wavy NoSQL solution is more scalable. Which part? I don’t doubt that there is a NoSQL solution that makes some bottleneck in the system much more performant, but rather we should go with a choice and be able to defend it and also discuss alternatives and their trade-offs
You just designed a monolithic solution for instagram. Did they really liked that? where is domain model? where are microservices? Api gateway?where is event sourcing? eventual consistency against strong consistency? there is no way you can scale this for 10 million users on one single database shares between multiple services.
What is the tool you are using for drawing/diagraming?
Always funny to me when people in the US always forget about i18n
Isn't 100TB=0.1PB(in decimal units)?
Sorry for the confusion due to the misuse of the "=" sign! It was 100TB (0.1PB) per month, but we wanted to find the amount per year which was 0.1PB * 12 months = 1.2PB.
A clearer presentation of the workings would be:
Per month: 100TB = 0.1PB
Per year: 100TB * 12 = 1.2PB
Hope this clarifies it for you!
nice explanation.
It would be awesome if someone could tell me what whiteboard/notepad application is used in this video. Could come in super handy during a system design interview!
Whimsical
Well done, nicely explained
Hey great video Jacob learned a lot. Really liked the way you explored everything giving fairly reasonable and convincing solutions . Thanks for the video again really appreciate the effort.
Thanks Kapil! If we had to make another video next, what should the topic be? :)
@@tryexponent Typeahead Suggestion or Twitter Search would be interesting topics .
Hello! What are you use for draw structure in this video?
Feed computation might be complicated , wanted to know more about it
1. Hour based generation of feeds seems to be impractical: feedback time is very important for social networks. Maybe it should be triggered for users on events (follow, unfollow, upload,...)?
2. Some very popular users maybe should be processed separately cause they generate ton of events for their followers.
3. You didn't estimate any sort of traffic.
4. You didn't say about any nonfunctional requirements
5. You didn't say about how you would scale horizontally on database of metadata.
And so so much more.
Potentially use a CDN in front of the LB? CDN that is programatically invalidated upon the write app server success.
Amazing!! thanks:)
great video content ,wish the best for yours
Is it good to cache feed as this changes frequently.
the API' were not constructed in this design. I think we should create the entities after confirming the API's and doing the high and some level of deep dive
Starting with DB schema, and not with the user flow is a huge "noob" mistake. It gives away, that you just prepared this scenario in advance by just looking at the end result without understanding how to get there.
Not to mention that the solution itself is very "junior" -ish. For example, passing a file through "app service" and not uploading it directly to the object storage - is a big mistake, you basically multiplied the amount of traffic that runs through your system by 2 for no reason. Populating cache synchronously on reading - is also a big mistake. And pretty much nothing was said about how to prepare a feed, which is the most interesting and loaded part of the design.
And by the way, this is a classic example of a distributed monolith - you tied everything to one single database.
2/10.
Please give alternatives to weakness you spot rather than pointing them out and walking away. You're not helping anyone that way Mark.