Please let me know in the comments if you find the video useful and please do like the video. It helps the channel a lot. Also if you want to know how Dropbox should be designed based on the requirements discussed in this video, check: Distributed System Design Interviews Bible | Best online resource for System Design Interview Preparation is now online. Please visit: www.thinksoftwarelearning.com? Please follow me on facebook.com/Think.Software.Community if you like to get notified about new course chapters getting added or any other updates. I will also take your suggestions there about the course and the channel.
This is the most in-depth video that covers so many avenues for this particular problem. No other video touches this many different aspects and cases. Well done!
This is the best video on this topic anywhere. I like that you went in-depth explaining the requirements and what they actually mean. Other instructors and blogs just give an overview. This is good content. Thanks!
Really useful video Arsalan. Thanks for contributing back to the community. In the HLD, I have a perspective that the synchronization service shouldn't talk directly to block service, since block service and object storage are broadly the members of the same 'Storage' module. Ideally, the calls should be routed through Object Storage which should then perform operations as needed. This will allow storage nuances to be handled at a single place. Would love to know your perspective on this.
I was never convinced by any RUclipsr except YOU. The quality is just outstanding! I am very willing to buy your course. One question plz, i always heard that nosql is faster for distributed, cuz it doesnt need to do joining etc. So why you kept using sql ?
Thanks for the comment. In my videos, I am discussing the schema but it does not mean that it is only applicable to sql. Also in the future I will make a video discussing the comparison of sql vs nosql.
Thanks for the comment. Yes, I was very busy in my job to get time to work on the videos. My 2022 resolution is to make more videos. There is a course out as well that you can check. It is still incomplete but I still think that it is at a stage that you will find it very useful. The link is in the description.
@ThinkSoftware, Thank you Sir for this informative content. As you suggested to opt for your course for detailed analysis, I went through the course material. It's a text course and I thought/expected it's a video course. If possible, please make it a video course.
Chunk size thoughts, first of all thanks for all your videos. A really small chunk size could lead to a lot of meta information that needs to be stored for each file. A really large chunk size could lead to problems with maintaining patches (if we are maintaining patches and versions like fit). A really large size could also inhibit parallel processing of the file, since threads would be consumed for a longer period of time. There probably is some network/bandwidth component associated with choosing the right size as well. I am just throwing the kitchen sink at it.. Not sure if I made any sense. Please comment on what you think and kindly provide the right way of judging the block size.
I was also thinking the same but I stuck at a point. If we are considering client's bandwidth and if it is x at the time client tried uploading and changed to y when half of the chunks are already uploaded i.e., it is fluctuating. Also if he tries to update file after some time and bandwidth gets reduced. Should we further chunk the chunk he's already updating.
I recently came across this channel and soon enough went through almost all the videos. Although I liked all other videos so far, Im not sure if this video was helpful as most of the stuff is still kept under blackbox and details are missing. It looks more like an advertisement to the your course than a system design video
Thanks for the feedback. As you can see not everything can be covered in a RUclips video and the video was already around 30 min long. That is I was only able to discuss the design at high level only. The details of course can be found in the course. But based on just the video you should be able to come up with right set of requirements and the overall design. Of course if the interviewer goes into detail then you do need the knowledge that is covered in the course.
Your system design videos have been very very useful and I have learned a lot from them. Could you please do a video on "Designing a code build & deployment system". It's a very popular question these days for SWE-Infrastructure roles. Or maybe add it in your paid course :-)
Thank you sir for this informative vide. I have a question. Lets say I have a text file in my drive and I downloaded it, made some changes in it and then uploaded the file again. The file is then uploaded as a new file in my drive. Does that align with the concept of uploading only the updated chunk of a file? If yes, how does the service determine that the file i am uploading is an updated version of an existing file on the drive (since I downloaded that file from drive itself and made changes in it). Please provide your thoughts on this concern. Thanks in advance :)
Chunk size has more to do with file size and the non-functional requirements like throughput, bandwidth and latency.. set by business requirements. The smaller the chunk the smaller bandwidth and higher throughput and latency.
The chunk size should be decided based on some formula which takes below factors into consideration 1. Total Size of file ( it would be wrong design to create small chunks ( for e.g 4KB size) for a file with size 10 GB ). In the formula, we will have a limit on the maximum number of chunks that we can create for a file. 2. The file type ( for e.g. a document file will be very small compared to a video file and will have more updates compared to a video file. So it makes more sense to keep higher chunk size for video file and a smaller chunk size for a document file. ) To resolve the file conflicts we can use the same mechanism that we use in source control system like GIT to figure out of if the changes can be merged without conflicts or not
If we find everything in your course, why should we watch this video? Please mention it as a trailer video so that we understand it accordingly. Please....🙏🙏
Thanks for the comment. Although I discuss some stuff in greater detail in the course, not everyone buys the course and so I want my audience on RUclips to still have sufficient information (if not all/detailed).
Thanks for the video. There is no API for creating root folder in user's storage account. How user root folder will be created in his cloud storage without having a API CreateRootFolder? Please share ur idea ?
Really useful. I am waiting for you to add more design in your online course before buying it. Also I requested in the past about globally distributed dbs - not all content in every region. Thanks
Thanks for the comment 🙂. That is why right now I am providing course as unlimited access so that if you buy now you will still get future updates. After few more chapters, I will switch to yearly subscription model for the new buyers. Old/existing buyers will still have unlimited access. What about globally distributed dbs, I didn't get what you are asking? Having different chapters available in different regions?
For the atomic file operation, is the temporary file you are mentioning like a copy of the original file or is it a blank file? I assume it has to be a copy if we are using a rename right?
one question . When you say we can update only changed segments or chunks. Lets say we had a pdf file, a binary file and we modify the pdf file which creates a new binary file. how do we calculate the chunk which has changed and how do we receive it and merge it. The position of the chunk and its offset might make it difficult to track and update only that chunk.
When it comes to non-functional requirements. I feel like most of the architectures that are asked in system design interviews almost always need to be scalable, highly available, and fault tolerant. Do you agree or can you think of situations where these might be able to be relaxed?
Thanks for the comment 🙂. Yes if you have seen my videos you would have known by now that there are 4 non-functional requirements that are always there for a distributed system and then there could be others that need to be there but can become more important for some systems.
Can you explain the API flow on the client side? When the user clicks "upload", does the application first handle the chunking and indexing before making the API request? so then would the API be sending chunks of files? /api/chunks/:chunk_id
On cellular networks , chunk size should be small as failed will be retried and cost data , on wifi it should be as big , larger the chunk , larger the rework on failure of the chunk.
I would say the optimal way to decide the chunk soze will have 3 factors, first the type of storage it is being stored upon. If the minimum partitioning of the storage is x bits , so if we have x bits of chunks it will have least amount of compaction and defragmentation. Second would be the smaller the size easier to send over network and incase of failure will be easier to resend and have checksum calculated for corruption of file. Just like p2p torrent. 3rd would be the more number of files, it becomes difficult to manage. Read will be slower but, we only read from a fully formed file rather than partial chunks so not much a difference. Please let me know what you think about my ideas.
I think at 13:00 - Meaning of 'A' is wrong 21:15 - I think chunk size can be decided based on available bandwidth & nowadays it can be in MBs I will prefer metadata in RDBMS & some GraphDB for accessibility. I feel lots of edits done to record this 27 mins video, if after preparation its difficult to record such videos without so many edits then one can understand that expecting to do a good logical & extensible design covering FRs & NFRs in 40-45 mins discussion is too much, when candidate is not getting the list of 2-3 application design names on which interview will happen. Over that, candidate is expected to discuss about pros & cons of the approach during that time. If companies would have expected that much details from their on job Architects/Designers/Leads then can avoid many issues in their own projects. But people are more interested for such System Design round drama & I can't understand that why no-one tries to correct it.
side comment outside of topic of video: A user can also be a she not necessarily on he. Throughout video user is referred as he. Please ensure you ensure using gender neutral personas. Thanks for creating this video though and helping everyone learn designing.
Good video. Well structured and easy to follow. Thanks for putting this together. One feedback would be to use gender neutral pronouns like "they" :). User is not always "he".
I was majorly put off by you mentioning "find it in the course" zillion times, I mean if you have explained it end to end only in your course, why bother creating a free half cooked video on youtube ? Disappointing!!
Thanks fot the comment. Please understand not everything can be covered in the 30 min video and also not everyone buys the course so I am creating this video for those who can't buy the course and providing them some useful information. Creating a 30 min video requires days of research plus multiple hours of video recording and editing.
Where is the API for downloadFile? More, What is the point of watching this video if everything has been explained in your previous videos? Not much focus has been given on what actual Database should be used to store the file, How the streaming of data is done, How the cryptography is applied. Better take a real-time example like an Image upload or a video upload from user's phone or from the Website and what actual APIs would take part in this use case. And again nothing has been mentioned about downloading the file which is the key feature for any cloud storage system.Time and again you just keep mentioning the online resources are lacking this and that, Better improve your content rather than complaining it feels like you just want user to just watch your previous videos.
![OhGeesy x Lefty Gunplay - What It Iz [Official Music Video]](http://i.ytimg.com/vi/L0yzW9t2ixk/mqdefault.jpg)
Please let me know in the comments if you find the video useful and please do like the video. It helps the channel a lot. Also if you want to know how Dropbox should be designed based on the requirements discussed in this video, check:
Distributed System Design Interviews Bible | Best online resource for System Design Interview Preparation is now online. Please visit: www.thinksoftwarelearning.com?
Please follow me on facebook.com/Think.Software.Community if you like to get notified about new course chapters getting added or any other updates. I will also take your suggestions there about the course and the channel.
This is the most in-depth video that covers so many avenues for this particular problem. No other video touches this many different aspects and cases. Well done!
This is the best video on this topic anywhere. I like that you went in-depth explaining the requirements and what they actually mean. Other instructors and blogs just give an overview. This is good content. Thanks!
You are welcome :)
Thank so much for such in depth analysis. It will help us getting better at our daily jobs as well :) Cheers. More power to you.
Glad it was helpful!
Really useful video Arsalan. Thanks for contributing back to the community. In the HLD, I have a perspective that the synchronization service shouldn't talk directly to block service, since block service and object storage are broadly the members of the same 'Storage' module. Ideally, the calls should be routed through Object Storage which should then perform operations as needed. This will allow storage nuances to be handled at a single place. Would love to know your perspective on this.
I was never convinced by any RUclipsr except YOU. The quality is just outstanding! I am very willing to buy your course.
One question plz, i always heard that nosql is faster for distributed, cuz it doesnt need to do joining etc. So why you kept using sql ?
Thanks for the comment. In my videos, I am discussing the schema but it does not mean that it is only applicable to sql. Also in the future I will make a video discussing the comparison of sql vs nosql.
@@ThinkSoftware almost 5 months did u upload the video for sql vs nosql comparison.
Is there a paid course by you, I'm willing to buy
Thanks for the comment. Yes, I was very busy in my job to get time to work on the videos. My 2022 resolution is to make more videos. There is a course out as well that you can check. It is still incomplete but I still think that it is at a stage that you will find it very useful. The link is in the description.
great video! one of the most clear I've ever seen.
Thanks for the comment 🙂
@ThinkSoftware, Thank you Sir for this informative content. As you suggested to opt for your course for detailed analysis, I went through the course material. It's a text course and I thought/expected it's a video course. If possible, please make it a video course.
Chunk size thoughts, first of all thanks for all your videos. A really small chunk size could lead to a lot of meta information that needs to be stored for each file. A really large chunk size could lead to problems with maintaining patches (if we are maintaining patches and versions like fit). A really large size could also inhibit parallel processing of the file, since threads would be consumed for a longer period of time. There probably is some network/bandwidth component associated with choosing the right size as well. I am just throwing the kitchen sink at it.. Not sure if I made any sense. Please comment on what you think and kindly provide the right way of judging the block size.
Thanks for the comment. Yes you are thinking in right direction.
I was also thinking the same but I stuck at a point. If we are considering client's bandwidth and if it is x at the time client tried uploading and changed to y when half of the chunks are already uploaded i.e., it is fluctuating. Also if he tries to update file after some time and bandwidth gets reduced. Should we further chunk the chunk he's already updating.
Very well explained! Thank you!
this channel is underrated, awesome videos ,
Thanks for the comment 🙂
I recently came across this channel and soon enough went through almost all the videos. Although I liked all other videos so far, Im not sure if this video was helpful as most of the stuff is still kept under blackbox and details are missing. It looks more like an advertisement to the your course than a system design video
Thanks for the feedback. As you can see not everything can be covered in a RUclips video and the video was already around 30 min long. That is I was only able to discuss the design at high level only. The details of course can be found in the course. But based on just the video you should be able to come up with right set of requirements and the overall design. Of course if the interviewer goes into detail then you do need the knowledge that is covered in the course.
Really underrated channel.
Thanks 🙂 for the comment
Your system design videos have been very very useful and I have learned a lot from them. Could you please do a video on "Designing a code build & deployment system". It's a very popular question these days for SWE-Infrastructure roles. Or maybe add it in your paid course :-)
Thanks for the comment 🙂. I will look into this.
Thank you sir for this informative vide.
I have a question. Lets say I have a text file in my drive and I downloaded it, made some changes in it and then uploaded the file again. The file is then uploaded as a new file in my drive. Does that align with the concept of uploading only the updated chunk of a file? If yes, how does the service determine that the file i am uploading is an updated version of an existing file on the drive (since I downloaded that file from drive itself and made changes in it).
Please provide your thoughts on this concern.
Thanks in advance :)
Chunk size has more to do with file size and the non-functional requirements like throughput, bandwidth and latency.. set by business requirements. The smaller the chunk the smaller bandwidth and higher throughput and latency.
Thanks for the comment
Very informative !
Thanks for the comment 🙂
ty for uploading this. the requirements are super thorough and was wondering if you would expect this level of coverage in the design interview.
It depends on the level you are interviewing for.
@@ThinkSoftware hmm, what would you expect from a Senior SDM or a director?
The senior SDM/director need to be thorough and should be able to convert vague requirements into something meaningful.
The chunk size should be decided based on some formula which takes below factors into consideration
1. Total Size of file ( it would be wrong design to create small chunks ( for e.g 4KB size) for a file with size 10 GB ). In the formula, we will have a limit on the maximum number of chunks that we can create for a file.
2. The file type ( for e.g. a document file will be very small compared to a video file and will have more updates compared to a video file. So it makes more sense to keep higher chunk size for video file and a smaller chunk size for a document file. )
To resolve the file conflicts we can use the same mechanism that we use in source control system like GIT to figure out of if the changes can be merged without conflicts or not
buddy u r very strong
Thanks for the comment 🙂
Totally disagree. Chunk size has more to do with file size and required throughput and bandwidth set by business requirements.
If we find everything in your course, why should we watch this video? Please mention it as a trailer video so that we understand it accordingly. Please....🙏🙏
Thanks for the comment. Although I discuss some stuff in greater detail in the course, not everyone buys the course and so I want my audience on RUclips to still have sufficient information (if not all/detailed).
Thanks for the video. There is no API for creating root folder in user's storage account. How user root folder will be created in his cloud storage without having a API CreateRootFolder? Please share ur idea ?
really very well explained! easy to grasp. thanks a lot!
Thanks for the comment 🙂
I have really learnt a lot from your videos
Thanks for the comment 🙂
Really useful. I am waiting for you to add more design in your online course before buying it. Also I requested in the past about globally distributed dbs - not all content in every region. Thanks
Thanks for the comment 🙂. That is why right now I am providing course as unlimited access so that if you buy now you will still get future updates. After few more chapters, I will switch to yearly subscription model for the new buyers. Old/existing buyers will still have unlimited access. What about globally distributed dbs, I didn't get what you are asking? Having different chapters available in different regions?
For the atomic file operation, is the temporary file you are mentioning like a copy of the original file or is it a blank file? I assume it has to be a copy if we are using a rename right?
It is the file with new changes that are applied to the temp file first before renaming the temp to replace original.
one question . When you say we can update only changed segments or chunks. Lets say we had a pdf file, a binary file and we modify the pdf file which creates a new binary file. how do we calculate the chunk which has changed and how do we receive it and merge it. The position of the chunk and its offset might make it difficult to track and update only that chunk.
Great tutorial 👍🏾
Thanks
Thanks for the explanation, it's really helpful
You are welcome!
Isn't instrumentation like metrics and security considered non-functional requirements?
Good work
Thanks for the comment 🙂
very well explained, great video.
Thanks for the comment 🙂
Awesome stuff!
Thanks for the comment 🙂
When it comes to non-functional requirements. I feel like most of the architectures that are asked in system design interviews almost always need to be scalable, highly available, and fault tolerant. Do you agree or can you think of situations where these might be able to be relaxed?
Thanks for the comment 🙂. Yes if you have seen my videos you would have known by now that there are 4 non-functional requirements that are always there for a distributed system and then there could be others that need to be there but can become more important for some systems.
So the replication consistency here is replicating to another standby DB?
Can you explain the API flow on the client side? When the user clicks "upload", does the application first handle the chunking and indexing before making the API request? so then would the API be sending chunks of files? /api/chunks/:chunk_id
this is discussed in the course
I couldn't understand much what the indexer was supposed to be, could you clarify a bit for me? Awesome video tho!
This is discussed in the course. In short, indexer is maintaining an index of all the folders and files within designated root folder.
Seems like the atomicity and consistency are very closely related in ACID ,didn't find much of difference. Could you please clarify on this?
In a very brief manner, atomicity is all or nothing. Consistency is moving a state from one consistent state to another.
On cellular networks , chunk size should be small as failed will be retried and cost data , on wifi it should be as big , larger the chunk , larger the rework on failure of the chunk.
Thanks for the comment
I would say the optimal way to decide the chunk soze will have 3 factors, first the type of storage it is being stored upon. If the minimum partitioning of the storage is x bits , so if we have x bits of chunks it will have least amount of compaction and defragmentation. Second would be the smaller the size easier to send over network and incase of failure will be easier to resend and have checksum calculated for corruption of file. Just like p2p torrent. 3rd would be the more number of files, it becomes difficult to manage. Read will be slower but, we only read from a fully formed file rather than partial chunks so not much a difference. Please let me know what you think about my ideas.
Thanks for the comment. You are thinking in the right direction.
Great
Thanks
I think at 13:00 - Meaning of 'A' is wrong
21:15 - I think chunk size can be decided based on available bandwidth & nowadays it can be in MBs
I will prefer metadata in RDBMS & some GraphDB for accessibility.
I feel lots of edits done to record this 27 mins video, if after preparation its difficult to record such videos without so many edits then one can understand that expecting to do a good logical & extensible design covering FRs & NFRs in 40-45 mins discussion is too much, when candidate is not getting the list of 2-3 application design names on which interview will happen.
Over that, candidate is expected to discuss about pros & cons of the approach during that time.
If companies would have expected that much details from their on job Architects/Designers/Leads then can avoid many issues in their own projects.
But people are more interested for such System Design round drama & I can't understand that why no-one tries to correct it.
I suggest to start with well modularized monolith till the scale is really big or any one module is getting extra load.
side comment outside of topic of video: A user can also be a she not necessarily on he. Throughout video user is referred as he. Please ensure you ensure using gender neutral personas. Thanks for creating this video though and helping everyone learn designing.
thanks for the comment
requirement 5 is like rbac
Good video. Well structured and easy to follow. Thanks for putting this together. One feedback would be to use gender neutral pronouns like "they" :). User is not always "he".
Thanks for the comment 🙂
google docs operational transformer
I was majorly put off by you mentioning "find it in the course" zillion times, I mean if you have explained it end to end only in your course, why bother creating a free half cooked video on youtube ? Disappointing!!
Thanks fot the comment. Please understand not everything can be covered in the 30 min video and also not everyone buys the course so I am creating this video for those who can't buy the course and providing them some useful information. Creating a 30 min video requires days of research plus multiple hours of video recording and editing.
You just copied the design from System design book. Groks king author.
Thanks for the comment. I think you haven't read my book/course and that is why commenting this.
Where is the API for downloadFile?
More, What is the point of watching this video if everything has been explained in your previous videos?
Not much focus has been given on what actual Database should be used to store the file, How the streaming of data is done, How the cryptography is applied.
Better take a real-time example like an Image upload or a video upload from user's phone or from the Website and what actual APIs would take part in this use case.
And again nothing has been mentioned about downloading the file which is the key feature for any cloud storage system.Time and again you just keep mentioning the online resources are lacking this and that, Better improve your content rather than complaining it feels like you just want user to just watch your previous videos.
Thanks for the comment.
Talks unnecessary things too much
it depends ...