Just in case someone is still confused when to use column based DBs here's a little explaination: Consider a database which has a large volume of data with 100 columns. And your application most of the time just queries only a few columns (let’s say 4-5 columns). Then in case of document DBs or row based DBs (relational Dbs) all data will be loaded row by row in main memory for processing , parsing and other operations. It will be a waste of a lot of effort in loading and processing of unnecessary columns. Column based DBs are used to tackle this problem where a large amount of data is stored in column based fashion so that the application can query only a few columns of large volume of data. - reference: Designing Data-Intensive Applications. by Martin Kleppmann.
@@saurabhmidha3609 The data in RDBMS is stored row-wise in the disk in sequential blocks whereas in Columnar DB the data is stored column-wise making the query for certain columns simpler as all the column data is present in a singular sequence. Even though we can write a similar query for RDMS they would be inefficient as it would have to go through all the rows to only fetch a few columns' data.
This has been the most useful 23 mins of System Designs Learning I've come across yet. And this is after I took an 11 hour course... you put it all together for me so much more succinctly. Thank you!!
WOW. Watched so many videos and finally found the gold content. No flashy animations and funky music. Just old school 1-1 explaining with clarity, examples and strong fundamentals. Beauty of online education.
Thanks for creating such an insightful video. There are few things that I would suggest adding to all the important points that you have mentioned: 1> Write Heavy vs Read Heavy systems is also an important point to consider for database selection. We cannot write scale RDBMS (as generally in RDBMS the write is on the single primary instance). For DocumentDB, we need sharding to write scale it which can be done but are not so natural as it is in ColumnarDB like Cassandra where we need to mandatorily choose the partition key during the data modelling itself. So for write heavy systems I prefer to go with Cassandra if I don't need ACID. For write heavy systems that need ACID we should choose RDBMS with multi master configuration and for that application needs to take care of bifurcating the data to the appropriate master based on the decided key. 2> Graph database for mapping social network type of data.
Those who prefer AWS Managed Services. Here, is the list Blob Storage : AWS S3 Caching : AWS ElasticCache Full Text Searching : AWS ElasticSearch Time Series Database : AWS TimeStream Analytical Databases : AWS Redshift ( It depends, sometimes we can use AWS S3 for storing large csv files and then we can use AWS ElasticMapReduce(EMR) to process S3 Files) RDMS (Relational Databse) : AWS Aurora Document Database : AWS DynamoDB Columnar Database : Not sure, I think AWS DynamoDB will satisfy this requirement too.
As someone with only monolithic embedded systems experience, this is hands down the only resource on the internet that made sense to me in this domain. Heartfelt thanks!
Sandeep: I must say that you have the gift of explaining complex concepts in a very simple way, this is rare to find in today's days. Your clarity in explaining the fundas is great and makes me learn so much more. Please continue sharing your knowledge thereby helping learners across the globe! Thank you for sharing this and best of luck to you on this journey!
Best video I have watched which summarizes it so simply. The best part of this is he explains it by taking examples of products such as Amazon, Netflix etc.
Your videos are a life saver for me. I have recently joined an organisation as senior developer and i have been given few system design tasks. And going through your videos is enabling me to understand the intricacies of the system design.
This channel should really have lot of subscribers. This is the first channel where I found so much detailed explanation on system design. Thanks a lot.
literally no words i was struggling to find a perfect video which explains in every possible perception (user, developer, maintainer) this made all the doubts cleared ... thank you
Wow !!! What clarity in explaining Database use cases, scenerios, categories of database with pictorial representation. The way you explained the concept is quite impressive. The way you corelate the concept with example is best fit. Keep the good work. May your light shine upon people like me the beneficiary. Thanks.
Awesome content dude. I ensure I watch all the ads during the video and click on all of them because I don't know how else I can give back to you for the invaluable knowledge you share with us!
Your videos are very simple to understand. I would like to thank you for explaining the FR and non-FR for each system design consideration. That sets the expectation very well.
Cool and crisp details relevant to the topic. Very few people are gifted with the skill to share the knowledge and also create an impact. I came across this channel and now I'll be watching the whole series in coming days.
Oh my God! I didn't know this kind of content is also available on the net and that too is free with so many details explained beautifully and concisely. Subscribed and gonna share it with others. Thanks again for putting so much effort for the learners like us.
Thank you so much for such a precise cheat sheet. You also cleared a lot of concepts for me which were fuzzy in spite of my 8.6 years of work experience!
I had been struggling with starting Databases to build my conceptual knowledge and this is amazing for everyone. Thanks a lot for sharing this in the most simplistic way.
Every video i watch, I click like. Complex system design problems are explained in very simple way. Keep it up and make more such videos. Thanks a lot!
Teaching is very diff from preaching! And Sir, you have cleary stated the difference. Making someone learn is easy, but making someone understand is what teaching is 👍 Hoping that your vedios will make a difference 😃
OMG, I didn't know shit about no-SQL and just with this video I think I can propose the adequate database type in an exercise. Amazing, specially loved the way to combine databases. Thank you so much.
Really amazing playlist. I wonder why you don't have subscribers in lakhs. I have a final interview for SDE-I at a unicorn startup in 4 days and this is a life saver.
Bro, this is just soooo good. You summarised so many important topics of software development concisely in some minutes with awesome explanation. Hats off to you. first time watching your video and i have become a fan. Thanks for the great content!
Not sure, what's your name is. you are doing good job. thank you !! Can you please cover following topics in your upcoming videos. 1. Geo Databases. How application like tinder execute spatial queries. like people within 5 km.. ( I know bits and pieces not completely ) 2. How relational databases scale for banking websites. How they do sharding with example.. ( Not sure, whether I am asking too much or not. please ) 3. Regarding handling transactions in relational databases, Why banks take 5-7 days time if the transaction fails after deducting money from my account. why they don't rollback immediately... ( Not sure, whether it is a great question or not.. Please ) 4. Calculating Fanstasy Rank in RealTime. ( How Dream11 calculates Fantasy Rank in realtime.. I have read and watched some AWS videos on Dream 11.. They use Amazon EMR services. If you could provide a detailed video it would be great !! ) Thank you and sorry for asking too many questions !
Elastic Search (and MongoDB also!) has good support for Geo spatial data processing. You can check the latest documents of either. The usually given example of banking for ACID (primarily for Atomicity) is very much wrong, as banking systems are connected via much sophisticated systems (usually SWIFT based) which has quite a few checks in place for transactions.
I was searching your Medium blog post for some good detailed explanation, I can say , one of best explanation so for, I am going with watch all of video.
THANKS A TON for making things amazingly simple to understand. Great stuff indeed! I have watched all your videos on system design, they are so nicely presented and organised. Awaiting the next videos.
Really an awesome video. After seeing your one design video, I am on system design spree as if I want to finish the Netflix series. Every sentence you speak make to note it down in my notes. Thanks for all of it. One last thing, please whenever you time, keep the momentum of your system design input through videos.
Great Video! CAP Considerations: Need Availability and Consistency? MySQL+ Need Availability and Partition-Tolerance? Cassandra+ Need Consistency and Partition-Tolerance? MongoDB+
This is a perfect learning video, thank you. I found it very useful how you associated real world examples with type of db. Info about use of combination of database is also lot of helpful understanding how real world system might be using them.
I have always wished for a video like this! Since, I am non-CS background, I am a bit oblivious to the world of DB products out there. I had always heard many of these names, but only as lingo - never really understood why they were being mentioned in the context. Thank you for putting this together, I will now probably understand more future such discussions in life :D
Taking note of new info: 6:05 implementing text search feature, can use implementations provided by lucene, solar (TODO:- investigate these technologies) Elastic search is not a db, it’s a search engine so don’t use it as your primary source of truth Storing application metrics, throughput latency etc would use time series db. Useful when you’re doing sequential updates (appending metrics) instead of modifying existing records (kind of like a log) examples of time series db: InfluxDB, openTSdb
![Tee Grizzley - Blow for Blow (feat. J. Cole) [Official Video]](http://i.ytimg.com/vi/wNLN5xsx5dc/mqdefault.jpg)
You make me intelligent with every video I watch of yours.
Thanks!! That makes our day :)
Do share this with your colleagues and spread the knowledge :)
I agree with you 100%
Ture !!
Why no design videos these days?? What happend?
Just in case someone is still confused when to use column based DBs here's a little explaination:
Consider a database which has a large volume of data with 100 columns. And your application most of the time just queries only a few columns (let’s say 4-5 columns). Then in case of document DBs or row based DBs (relational Dbs) all data will be loaded row by row in main memory for processing , parsing and other operations. It will be a waste of a lot of effort in loading and processing of unnecessary columns. Column based DBs are used to tackle this problem where a large amount of data is stored in column based fashion so that the application can query only a few columns of large volume of data.
- reference: Designing Data-Intensive Applications. by Martin Kleppmann.
Well explained!!
But you can also do the same thing in RDBMS. you can easily right a query to get only few columns in the result
@@saurabhmidha3609 The data in RDBMS is stored row-wise in the disk in sequential blocks whereas in Columnar DB the data is stored column-wise making the query for certain columns simpler as all the column data is present in a singular sequence. Even though we can write a similar query for RDMS they would be inefficient as it would have to go through all the rows to only fetch a few columns' data.
This has been the most useful 23 mins of System Designs Learning I've come across yet. And this is after I took an 11 hour course... you put it all together for me so much more succinctly. Thank you!!
He's amazing
WOW. Watched so many videos and finally found the gold content. No flashy animations and funky music. Just old school 1-1 explaining with clarity, examples and strong fundamentals. Beauty of online education.
Thanks for creating such an insightful video.
There are few things that I would suggest adding to all the important points that you have mentioned:
1> Write Heavy vs Read Heavy systems is also an important point to consider for database selection. We cannot write scale RDBMS (as generally in RDBMS the write is on the single primary instance). For DocumentDB, we need sharding to write scale it which can be done but are not so natural as it is in ColumnarDB like Cassandra where we need to mandatorily choose the partition key during the data modelling itself. So for write heavy systems I prefer to go with Cassandra if I don't need ACID. For write heavy systems that need ACID we should choose RDBMS with multi master configuration and for that application needs to take care of bifurcating the data to the appropriate master based on the decided key.
2> Graph database for mapping social network type of data.
Can you give examples of Write Heavy systems and Read Heavy Systems and why?
@@shponder9510 OLTP vs OLAP
@@shponder9510 URL shortener- you will have more reads than writes. may be 100:1 ratio of reads to writes
Best video tutorial available on the internet for choosing between databases for designing systems. Period
Thanks for the kind words :)
Those who prefer AWS Managed Services. Here, is the list
Blob Storage : AWS S3
Caching : AWS ElasticCache
Full Text Searching : AWS ElasticSearch
Time Series Database : AWS TimeStream
Analytical Databases : AWS Redshift ( It depends, sometimes we can use AWS S3 for storing large csv files and then we can use AWS ElasticMapReduce(EMR) to process S3 Files)
RDMS (Relational Databse) : AWS Aurora
Document Database : AWS DynamoDB
Columnar Database : Not sure, I think AWS DynamoDB will satisfy this requirement too.
are you from AWS :)
Bo Wang No, learning AWS :)
Can we use RDS instead of Aurora for a relational database?
This is one of the best explanation on choosing db....
As someone with only monolithic embedded systems experience, this is hands down the only resource on the internet that made sense to me in this domain. Heartfelt thanks!
Sandeep: I must say that you have the gift of explaining complex concepts in a very simple way, this is rare to find in today's days. Your clarity in explaining the fundas is great and makes me learn so much more. Please continue sharing your knowledge thereby helping learners across the globe! Thank you for sharing this and best of luck to you on this journey!
Thanks Jyoti! Do spread the word about the channel. It motivates to keep making more :)
I don't know why people are buying gold when by just watching this video you can be rich!
Sincere gratitude to you Sandeep!
this video is an absolute gem. Finally someone that breaks this down directly without dancing around the topic
Best video I have watched which summarizes it so simply. The best part of this is he explains it by taking examples of products such as Amazon, Netflix etc.
Reading this a day before on-site interviews. I am much more confident on the database selection justification :). Thank you
Your videos are a life saver for me. I have recently joined an organisation as senior developer and i have been given few system design tasks. And going through your videos is enabling me to understand the intricacies of the system design.
This man needs at least 100K subs. Love this content, extremely concise and informative, I have subbed!
No words to say..the way you've cleared everything.... You should be on RUclips long time before.
I like the way you present.. you look like a humble human being with lots of knowledge and work experience... please continue your good work..
Each minute is worth watching... Very logical and concise.
This channel should really have lot of subscribers. This is the first channel where I found so much detailed explanation on system design. Thanks a lot.
literally no words
i was struggling to find a perfect video which explains in every possible perception (user, developer, maintainer)
this made all the doubts cleared ...
thank you
This was just what I needed, gem of a video. Thanks!
The clear explanation/flowchart at the end for choosing SQL vs document vs columnar was GOLD
To the point and crisp.... Covered all popular types of DB. Thanks
Glad that you liked it!
Wow !!! What clarity in explaining Database use cases, scenerios, categories of database with pictorial representation. The way you explained the concept is quite impressive. The way you corelate the concept with example is best fit. Keep the good work. May your light shine upon people like me the beneficiary. Thanks.
One of the best System Design videos series on RUclips. Keep up the good work!
Really informative. Loved it, coming out burnt from GKCS, that guy always leaves me confused for some reason. This channel is must recommended.
Awesome content dude. I ensure I watch all the ads during the video and click on all of them because I don't know how else I can give back to you for the invaluable knowledge you share with us!
Thanks Abhijit!
Just spread the word about the channel amongst your friends / LinkedIn connections and that would be amazing :)
Brother, this is the best video I have ever seen on System Design (Databases). Thanks for sharing all this knowledge. Kudos to you!!
Thank you Karle, you have solved my mysterious about databases that has been going thru my head for many years.
What a very dense video! I took 3 pages of notes. The database world is much clearer for me. Thanks a lot!
Wow. That's good to hear!
Super helpful. More I watch it more clarity I get. Thanks for making this video.
Seedhi Baat.. No Bakwaas.. Excellent short cheat sheet video for choosing right DB.
Your videos are very simple to understand. I would like to thank you for explaining the FR and non-FR for each system design consideration. That sets the expectation very well.
Wonderful video! I have been looking for something in depth like this for a very long time!
Cool and crisp details relevant to the topic. Very few people are gifted with the skill to share the knowledge and also create an impact. I came across this channel and now I'll be watching the whole series in coming days.
Oh my God! I didn't know this kind of content is also available on the net and that too is free with so many details explained beautifully and concisely. Subscribed and gonna share it with others. Thanks again for putting so much effort for the learners like us.
Thank you so much for such a precise cheat sheet. You also cleared a lot of concepts for me which were fuzzy in spite of my 8.6 years of work experience!
I had been struggling with starting Databases to build my conceptual knowledge and this is amazing for everyone. Thanks a lot for sharing this in the most simplistic way.
Every video i watch, I click like. Complex system design problems are explained in very simple way. Keep it up and make more such videos. Thanks a lot!
Your videos are pure gold!! Hands down the best system design tutor!!! Thanks a lot!
Good explanation of document DB, you are the first who made me understand the use case!
I was happy after going through the video. Just one short and clear.
Teaching is very diff from preaching!
And Sir, you have cleary stated the difference.
Making someone learn is easy, but making someone understand is what teaching is 👍
Hoping that your vedios will make a difference 😃
Thanks!! That's very good to hear :)
Hopefully it'll make a difference and reach a lot of people :)
@@codeKarle It is making a difference. Dope stuff!
Amazing explanation of various databases at a breadth. Candid for quick brush up. Thanks
This is a very good video, something no one has done. So many terms I hear on a day to day basis but didn't know, are clear now.
You have just fulfilled my all queries for interviews.
Wow! This explanation is so lucid and logical, yet concise. Summarizing things this well should be considered an art form!
Excellent video talking about key considerations in choosing the "right" DB/ data store. Sincere thanks for your efforts!
OMG, I didn't know shit about no-SQL and just with this video I think I can propose the adequate database type in an exercise. Amazing, specially loved the way to combine databases.
Thank you so much.
Really amazing playlist. I wonder why you don't have subscribers in lakhs. I have a final interview for SDE-I at a unicorn startup in 4 days and this is a life saver.
This is awesome, I am amazed that everything you share has so much detail and thought.
Clear and to the point explanation. Great job! I learned a lot. Thanks.
Glad that it was helpful!
You are very clear about the content .. one of the best system design video available on RUclips..
Thanks Uday!
Perfect cheet sheet for most of the DB formats..!!!
So beautiful and simple explanation..!!! 👏👏
Please keep up the great work..!!!
One of the best videos ever in this topic
Thanks
The intro part in which you used the explanation of functional and non-functional requirements made me like the video and comment here.
awesome, the video gets to the heart of the matter and doesn't go around.
Keep doing the excellent work mate. This was a great video with respect to databases.
Bro, this is just soooo good. You summarised so many important topics of software development concisely in some minutes with awesome explanation. Hats off to you. first time watching your video and i have become a fan. Thanks for the great content!
I have been looking for this information since long time.
Thank you for providing the exact detail
What an amazing explanation for choosing databases. Kudos to you man.
Awesome !!
That's exactly what we require in a DB design rounds , in more depth though , but this was a very good intro in that direction
Wonderful! The best free video for choosing database in system design interview. Thanks so much
this guys is best! simple explaination.. easy to digest
Awesome Explanation... Thank you, Sir... This is ULTIMATE....
I got lot of knowledge from this video and will watch all you videos in coming days
Wow!! Amazing video with great content in a concise manner. Kudos, thank you so much!
I have a learnt a lot watching all your videos. Thank you so much sir 🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼
I could binge watch your videos forever. such an informative videos
Not sure, what's your name is. you are doing good job. thank you !! Can you please cover following topics in your upcoming videos.
1. Geo Databases. How application like tinder execute spatial queries. like people within 5 km.. ( I know bits and pieces not completely )
2. How relational databases scale for banking websites. How they do sharding with example.. ( Not sure, whether I am asking too much or not. please )
3. Regarding handling transactions in relational databases, Why banks take 5-7 days time if the transaction fails after deducting money from my account. why they don't rollback immediately... ( Not sure, whether it is a great question or not.. Please )
4. Calculating Fanstasy Rank in RealTime. ( How Dream11 calculates Fantasy Rank in realtime.. I have read and watched some AWS videos on Dream 11.. They use Amazon EMR services. If you could provide a detailed video it would be great !! )
Thank you and sorry for asking too many questions !
Sure, we'll try to cover some of this in the future videos. Feel free to give more ideas :)
@@codeKarle Thank you !! Looking forward for the same !!
Elastic Search (and MongoDB also!) has good support for Geo spatial data processing. You can check the latest documents of either.
The usually given example of banking for ACID (primarily for Atomicity) is very much wrong, as banking systems are connected via much sophisticated systems (usually SWIFT based) which has quite a few checks in place for transactions.
Thanks a lot for these videos. Big fan of such videos. Make sure you post more videos. Keep up your good work guys
Amazing video, one feedback, please keep it at the top of the series, before the beginning of the design discussion of other systems.
I was searching your Medium blog post for some good detailed explanation, I can say , one of best explanation so for, I am going with watch all of video.
Love you brother. This video is so much informative. 1 video talks about so many solutions.
THANKS A TON for making things amazingly simple to understand. Great stuff indeed! I have watched all your videos on system design, they are so nicely presented and organised. Awaiting the next videos.
one stop explanation on choosing database, well done 👏
Best content on this topic! Thank you so much for your efforts!
Nice summary to tackle with system design interviews
Glad you liked it!
I'm only 4 minutes in and I've already learned a lot! Keep it up!
awesome video. gives a bigger picture on dbs to me. very useful. thank you
Just wow! You made such a concise video for such useful information. Thanks a lot.
Really an awesome video. After seeing your one design video, I am on system design spree as if I want to finish the Netflix series. Every sentence you speak make to note it down in my notes. Thanks for all of it.
One last thing, please whenever you time, keep the momentum of your system design input through videos.
Bro the way you explain is just Amazing ... Speechless
Thanks for making it easy to understand
Very good summary. Concise and in-depth! thanks..
Thanks!! Glad that you liked it :)
Great content brother! Keep spreading knowledge
Amazing Video. This is an absolute precise explanation for this topic. Thanks !!
Great Video!
CAP Considerations:
Need Availability and Consistency? MySQL+
Need Availability and Partition-Tolerance? Cassandra+
Need Consistency and Partition-Tolerance? MongoDB+
Hi, can you please give more details on why these choices for each cases?
Great explanation. One more point to add is also CAP therom need to be consider to choose the DB.
This is a super valuable video. It's beautiful. Thank you Karle.
This is a perfect learning video, thank you. I found it very useful how you associated real world examples with type of db. Info about use of combination of database is also lot of helpful understanding how real world system might be using them.
I have always wished for a video like this! Since, I am non-CS background, I am a bit oblivious to the world of DB products out there. I had always heard many of these names, but only as lingo - never really understood why they were being mentioned in the context. Thank you for putting this together, I will now probably understand more future such discussions in life :D
Thanks for your help with my interview prep!
Hi Karl - great job with these videos! Keep uo the good work
Great video. Helped me to understand data platform at a high-level. Thank you Sandeep.
Excellent video with precise information. Kudos and keep them coming. 🙏🏼🙏🏼
I had a lot of confusions on this, Thank You So Much!
All things explained straight and to the point. Hats off to you sir🕴🏻
Taking note of new info:
6:05 implementing text search feature, can use implementations provided by lucene, solar (TODO:- investigate these technologies)
Elastic search is not a db, it’s a search engine so don’t use it as your primary source of truth
Storing application metrics, throughput latency etc would use time series db. Useful when you’re doing sequential updates (appending metrics) instead of modifying existing records (kind of like a log) examples of time series db: InfluxDB, openTSdb
Thanks for making this review! I enjoyed it.