One basic concept that can be added before getting into intermediate concepts like sharding - is Indexing. Often, before scaling up , it's better to index smartly to reduce the reading latency. Overall good video.
a bright confident precise and concise lesson. you are a great teacher, kept my brain neurons constantly firing. what a class! keep all these good work.
Yogita, just wanted to let you know I read database sharding from other popular sources also but your explanation with example is best I found. Keep Going on !!!
Very Good Tutorial! Simplified the concept like anything. Sharding Key is a crucial here. Choose sharding if its really needed.. not every organization needed it. choosing wrong sharding key could bring lot of complexities and wrong data to the customer.
Your explanation is very good but your rate of speech is very fast and maintain a positive tone when you are explaining so that the viewer feels easy to digest the data Thanks
Good overview. My comment is: Sharding and partitioning are not the same though they both are breaking up a large data set into smaller subsets. Sharding implies the data is spread across multiple databases while partitioning is about grouping subsets of data within a single database instance.
The best way anyone could have explained it. Been working in IT for the past 2 years I just feel lost in the CS concepts and never found any structured content. Since I did my bachelors in Electronics. Thank you.
Hi Yogita, Your explanations are awesome but I think you can deep dive once basics are done and also if you plan to have real implementation it would be of great help as most of the developers don't have access to work at such a scale. Keep up the Great Work !!!!
Nice Explanation of such a great and important topic in system desing. Please keep uploading more video, these are quite informational and it is very useful for cracking system design round for FANG! Thanks a lot for your time and effort.
Yogita,your videos are really very helpful to understand the sharding concept. I have followed the whole system design series.thanks a lot for making this series..
Before I say anything, I will admit I have no experience with sharding, but I do understand what you presented. Scenario. With 1million users, would you say that user info maybe be small but related user information, like images or videos can take up a lot of space. So it's not so much the amount of users as opposed to the space user supported data can take up.
So the strategy to scale in most use cases: Vertically scale -> Separate Read and Write -> Archive and keep it under control -> Try breaking the application into smaller services -> Shard the last resort
Thanks for the video, this is really awesome. Though after watching and learning abt this I am more confused with the other various terms. Please make a video to clarify those. Database Partition Vs Table Partition Vs Distributed Database Vs Replication. What I understand is Replication is a Read-Copy with a distributed system. Distributed Database is having multiple database at different servers however in Sharding we are doing the same, if so what is the difference. And how the Database Partition is different than Table Partition, as in table partition we do not have to worry abt accessing it or no downtime is required however for database partition we would need some downtime when adding new partition.
Is sharding and partitioning the data same? As per my understanding, partitioned the data based on column which will create separate file for each partition value, so that it will improve the read performance when we query the data based on partition column
Yogita!!! your presentation is just amazing and out of the box. IMO you are the best teacher please please make videos on DSA too. I am 100% sure you will get overwhelming response
I had one question to ask : let's say that I have sharded the date in to mulitiple machines for desinging facebook, then in this case how the "FEED / TIMELINE" of a particular user be generated? One user can have multiple friends and he can follow many pages as well. So do we need to go to each of the shard where the friend of the user is residing to retureive the post information ?? It will make the system bit slow. How should we overcome this issue??
Hi, when a particular user posts something then there is something called fanout services which push the same data into the timeline of the friends' user which is also maintained using some caching strategy and it happens in the background so that when friend user looks at their timeline can see the post and also one more thing to keep in mind is the data is eventually consistent in most of the social networking platform so it may take some time to update the timeline.
Hi Yogita, does adding more shards later on in the development phase also increase a challenge, what I mean is if we add another shard do I again have to redristribute the data from old shard ?
Thank you Yogita for making this channel. I went through your primer course videos, and they explain all the concepts in a very detailed manner. However, I am not getting confidence how to give system design interview i.e how to connect all the concepts combined together to form a system. Could you upload a video in an interview manner like starting from a problem, discussion that happen between interviewer and interwee to proceed with the design?
DB Shards and DB Partitions are two different terms if we take distributed database into the picture. In fact, sharding and partitioning have different meanings in distributed system.
Hello, I really liked your video and your explanation is very clear but I don't agree with that partitioning and sharing is the same. For my knowledge partitioning is when we are splitting a table in database and sharing is when we are splitting data to different databases. So partitioning is done in the same database, we will just split/ partitioning the tables and sharing is done having multiple database, we will sharing data between database.
I have a doubt, I see that in vertical partitoning example from 3:49 you have put each column in a different database server. Can we do that for vertical partitining? I read that only horizontal partitioning is spread across database servers whereas vertical partitining is done withing a server. Can we do it for vertical too?
Query across the shards is not a disadvantage I mean situation itself will not arise as in shards( horizontal partitioning ) we are going to put whole schema in any particular shard not just a specific table..
Hi Yogita, Thanks for such a vivid explanation. I just have one doubt. You gave an example of Tinder while talking about Sharding sometime around 10:22. We can shard on the basis of cities definitely. I was just wondering what would happen to a particular person's data (in db) when he moves from city X to city Y ? Will that data be lost ?
The greatest drawback is the "complexity" it brings in your architecture. Sharding is the last resort as you said. Nice video with clarity.
Short, crisp and explained in the simplest possible way. Thanks for the hard work you put in. It is indeed very helpful.
Thanks Muskan
One basic concept that can be added before getting into intermediate concepts like sharding - is Indexing. Often, before scaling up , it's better to index smartly to reduce the reading latency. Overall good video.
Nice explanation yogita!! 👍 Hands-on demos would be really great!!
Sure 😊 Will do share soon.
a bright confident precise and concise lesson. you are a great teacher, kept my brain neurons constantly firing. what a class! keep all these good work.
Yogita, just wanted to let you know I read database sharding from other popular sources also but your explanation with example is best I found. Keep Going on !!!
Great video, nicely explained.
A hands-on video to shard data on SQL and NoSQL databases will help a lot.
Good work.
Yes, please a hands on video will be very useful.
One on relational(any one) and non-relational(any one) DB.
Noted! Coming soon!
Having the theortical knowledge is good but seeing the practical application will give the understanding how it is really done in the industry.
@@kumaraditya9986 You are right with your suggestion.
Dynamic sharding - does it also means db is sharded dynamically? Or is it dynamically decided which shards to query?
Very Good Tutorial! Simplified the concept like anything. Sharding Key is a crucial here. Choose sharding if its really needed.. not every organization needed it. choosing wrong sharding key could bring lot of complexities and wrong data to the customer.
Great ! A hands on video will make it complete. Thanks Yogita.
Most welcome 😊 Will try!
The way Yogita teaches these concepts is too good. 😀
Superb explanation
Nice explanation.! I like the way you try to convey without unwanted gossip and excluding intro
Your explanation is very good but your rate of speech is very fast and maintain a positive tone when you are explaining so that the viewer feels easy to digest the data Thanks
this will help me to understand the logic but hands on will give clear idea how its actually work
Good overview. My comment is: Sharding and partitioning are not the same though they both are breaking up a large data set into smaller subsets. Sharding implies the data is spread across multiple databases while partitioning is about grouping subsets of data within a single database instance.
The best way anyone could have explained it. Been working in IT for the past 2 years I just feel lost in the CS concepts and never found any structured content. Since I did my bachelors in Electronics. Thank you.
Excellent explanation it helps a lot keep up good work…thank you so much it’s really helpful ❤
Being an architect myself, I can vouch this content is gold!
Thanks for the validation Soham. Feels good to know that you liked our content.
Hi Yogita, Your explanations are awesome but I think you can deep dive once basics are done and also if you plan to have real implementation it would be of great help as most of the developers don't have access to work at such a scale.
Keep up the Great Work !!!!
Noted. I will share.
Hello Ashish, thank you for a point such a great point that will compliment her introduction. Great work Yogita.
best sharding video
Nice Explanation of such a great and important topic in system desing. Please keep uploading more video, these are quite informational and it is very useful for cracking system design round for FANG! Thanks a lot for your time and effort.
Thank you, I will
This series is awesome. Watching most every vid
Yes please 🥺 we want hands on videos as well
Great explanation, the way you explained is that the concept is concise and informative, Thank you and keep going!
Yogita,your videos are really very helpful to understand the sharding concept. I have followed the whole system design series.thanks a lot for making this series..
Glad to hear that
Thanks a ton for this course. Really grateful
You're very welcome!
Thank you so much, Mam.It was really Helpful😁
Before I say anything, I will admit I have no experience with sharding, but I do understand what you presented. Scenario. With 1million users, would you say that user info maybe be small but related user information, like images or videos can take up a lot of space. So it's not so much the amount of users as opposed to the space user supported data can take up.
Nice explanation! Great work
So the strategy to scale in most use cases: Vertically scale -> Separate Read and Write -> Archive and keep it under control -> Try breaking the application into smaller services -> Shard the last resort
add cache, replicas* in between
horizontal and vertical sharding 4:20
Sharding 5:26
Logical and physical sharding 17:54
advantages 9:00
algorithmic and logical sharding 11:33
drawback 13:53
Highly interesting and useful content to a beginner. I think the basics are covered very well. Good job and thanks for the video
Excellent & clear explanation. Kudos.
I'm happy this is free
Thanks for the video, this is really awesome. Though after watching and learning abt this I am more confused with the other various terms. Please make a video to clarify those. Database Partition Vs Table Partition Vs Distributed Database Vs Replication.
What I understand is Replication is a Read-Copy with a distributed system. Distributed Database is having multiple database at different servers however in Sharding we are doing the same, if so what is the difference. And how the Database Partition is different than Table Partition, as in table partition we do not have to worry abt accessing it or no downtime is required however for database partition we would need some downtime when adding new partition.
Salute to your effort to explain the concepts, would love to see hands on for one relational and one non-relational db
Great work once again!!
Please keep it up!
Waiting for sharding strategy and shard keys video!!
Very soon!
Great content would like to see relational db sharding as a hands on first.
awesome video...it really helped
Is sharding and partitioning the data same? As per my understanding, partitioned the data based on column which will create separate file for each partition value, so that it will improve the read performance when we query the data based on partition column
Please make practical sharding video
Awesome content on Sharding, Can you please do a hands on practical video on how to do replication and sharding
Will try to.
Great video thank you !!! What are the usecases for vertical partitions?
Helpful video .We would like to have hands-on on how to shard relational DB and NOSQL DB.
Reply
18000 views 800 like... ☹️ you need to remind people to like the video in the middle of it... This was very helpful
I respect the way you explain
Yogita!!! your presentation is just amazing and out of the box.
IMO you are the best teacher
please please make videos on DSA too. I am 100% sure you will get overwhelming response
Thanks a lot Akshay. 🤗
I had one question to ask : let's say that I have sharded the date in to mulitiple machines for desinging facebook, then in this case how the "FEED / TIMELINE" of a particular user be generated? One user can have multiple friends and he can follow many pages as well. So do we need to go to each of the shard where the friend of the user is residing to retureive the post information ?? It will make the system bit slow. How should we overcome this issue??
Hi, when a particular user posts something then there is something called fanout services which push the same data into the timeline of the friends' user which is also maintained using some caching strategy and it happens in the background so that when friend user looks at their timeline can see the post and also one more thing to keep in mind is the data is eventually consistent in most of the social networking platform so it may take some time to update the timeline.
great explanation
Great explanation. Which mic are you using? You voice is very clear
Hi Yogita, does adding more shards later on in the development phase also increase a challenge, what I mean is if we add another shard do I again have to redristribute the data from old shard ?
You teach a lot better than Gaurav Sen. Why don't you you launch your course on System Design. It would be a great help for all of us.
Excellent content and explanation.
Thank you Yogita for making this channel. I went through your primer course videos, and they explain all the concepts in a very detailed manner. However, I am not getting confidence how to give system design interview i.e how to connect all the concepts combined together to form a system. Could you upload a video in an interview manner like starting from a problem, discussion that happen between interviewer and interwee to proceed with the design?
Can't we create index instead of sharding like search query will be faster rather than doing sharding?
Great video
Great job :)
Should I design and code my own database engine?
Very helpful, thank you!
Good explanation
Good job
its Good explanation
Thanks Yogita for such a great video..really helpful..we would like to have lecture on how to shard relational DB and NOSQL DB.
Thank you very much
Dynamic sharding - does it also means db is sharded dynamically? Or is it dynamically decided which shards to query?
Did you mention problem with transactions after sharading?
Thought: Would it be better to mindfully split the tables in different databases instead?
awesome
Great Explanation. Can you please make video on Time series DB ?
DB Shards and DB Partitions are two different terms if we take distributed database into the picture. In fact, sharding and partitioning have different meanings in distributed system.
Could you please link the source to your claim here?
@@sudocode FYI -> ruclips.net/video/wXvljefXyEo/видео.html&ab_channel=AsliEngineeringbyArpitBhayani
Hello, I really liked your video and your explanation is very clear but I don't agree with that partitioning and sharing is the same. For my knowledge partitioning is when we are splitting a table in database and sharing is when we are splitting data to different databases. So partitioning is done in the same database, we will just split/ partitioning the tables and sharing is done having multiple database, we will sharing data between database.
I am sorry to say but that’s not accurate. Please check your source from where you read this. Check description for sharding links that I shared.
Thank you :)
Videos are very good and clear.If you do pratical implementations of some topics then it would be crystal clear.
You should add link to the next and previous video in your description section
good idea!
wow, well done!
What is the point of having logical shards if we can have multiple physical shards?
What an awesome intro
I have a doubt, I see that in vertical partitoning example from 3:49 you have put each column in a different database server. Can we do that for vertical partitining? I read that only horizontal partitioning is spread across database servers whereas vertical partitining is done withing a server. Can we do it for vertical too?
Query across the shards is not a disadvantage I mean situation itself will not arise as in shards( horizontal partitioning ) we are going to put whole schema in any particular shard not just a specific table..
Horizontal sharding implies splitting a table into multiple shards. In some cases all those shards have to be queried.
Hi Yogita, what is the use of logical sharding ? Thanks a lot for your videos ! :)
Users between 1 million and 1.1 million be like “Meko ye game nhi khelna” 😆
Jokes aside amazing video!
haha, thanks!
can we do replication as well as sharding for the same DB architecture, if so how. Could you please explain
Please make video on sharding in mongodb.
Great video as always.
Hands on like sharding MYSQL would be great.
Any plans for that?
Hands-on videos would be of great help
will do try for sure.
Very nyc yogita concept is clear
Thanks :)
Are their any other use case of logical partition other than query optimization?
very nice videos, please keep it going ..........
Thank you, I will
Nice explanation
Thanks
You are welcome
i would like to see video comparing time series dbs for low latency large data sets.
nice
Great explanation. One video if possible on Zookeeper please with use case.
Will add to the list.
reading data from a table having a index vs sharding , which gives more performance? index helps to look up isnt?
Problems change with scale. Indexing might not help when datasets are huge.
Thanks ! City based partion will come under column ( Vertical partion ) , right ?
no, it comes under horizontal data partitioning only, with city name as the shard key. all rows related to same city will be stored in same shard
Hi Yogita, Thanks for such a vivid explanation. I just have one doubt. You gave an example of Tinder while talking about Sharding sometime around 10:22. We can shard on the basis of cities definitely. I was just wondering what would happen to a particular person's data (in db) when he moves from city X to city Y ? Will that data be lost ?
Insert the record in new shard and soft delete from old shard
demos for both relational and non-relation would be really helpful...
noted!
Can you make a tutorial on Spring Boot internal working?
Can you please make hands-on videos please. That will help in understanding things practically.
Can you do a proof of concept please?
Hi please come up with hands on using mongodb will help a lot 🙏
Will add it to the list.