Just in case someone is still confused when to use column based DBs here's a little explaination: Consider a database which has a large volume of data with 100 columns. And your application most of the time just queries only a few columns (let’s say 4-5 columns). Then in case of document DBs or row based DBs (relational Dbs) all data will be loaded row by row in main memory for processing , parsing and other operations. It will be a waste of a lot of effort in loading and processing of unnecessary columns. Column based DBs are used to tackle this problem where a large amount of data is stored in column based fashion so that the application can query only a few columns of large volume of data. - reference: Designing Data-Intensive Applications. by Martin Kleppmann.
@@saurabhmidha3609 The data in RDBMS is stored row-wise in the disk in sequential blocks whereas in Columnar DB the data is stored column-wise making the query for certain columns simpler as all the column data is present in a singular sequence. Even though we can write a similar query for RDMS they would be inefficient as it would have to go through all the rows to only fetch a few columns' data.
WOW. Watched so many videos and finally found the gold content. No flashy animations and funky music. Just old school 1-1 explaining with clarity, examples and strong fundamentals. Beauty of online education.
Those who prefer AWS Managed Services. Here, is the list Blob Storage : AWS S3 Caching : AWS ElasticCache Full Text Searching : AWS ElasticSearch Time Series Database : AWS TimeStream Analytical Databases : AWS Redshift ( It depends, sometimes we can use AWS S3 for storing large csv files and then we can use AWS ElasticMapReduce(EMR) to process S3 Files) RDMS (Relational Databse) : AWS Aurora Document Database : AWS DynamoDB Columnar Database : Not sure, I think AWS DynamoDB will satisfy this requirement too.
This has been the most useful 23 mins of System Designs Learning I've come across yet. And this is after I took an 11 hour course... you put it all together for me so much more succinctly. Thank you!!
Thanks for creating such an insightful video. There are few things that I would suggest adding to all the important points that you have mentioned: 1> Write Heavy vs Read Heavy systems is also an important point to consider for database selection. We cannot write scale RDBMS (as generally in RDBMS the write is on the single primary instance). For DocumentDB, we need sharding to write scale it which can be done but are not so natural as it is in ColumnarDB like Cassandra where we need to mandatorily choose the partition key during the data modelling itself. So for write heavy systems I prefer to go with Cassandra if I don't need ACID. For write heavy systems that need ACID we should choose RDBMS with multi master configuration and for that application needs to take care of bifurcating the data to the appropriate master based on the decided key. 2> Graph database for mapping social network type of data.
Best video I have watched which summarizes it so simply. The best part of this is he explains it by taking examples of products such as Amazon, Netflix etc.
As someone with only monolithic embedded systems experience, this is hands down the only resource on the internet that made sense to me in this domain. Heartfelt thanks!
Your videos are a life saver for me. I have recently joined an organisation as senior developer and i have been given few system design tasks. And going through your videos is enabling me to understand the intricacies of the system design.
Sandeep: I must say that you have the gift of explaining complex concepts in a very simple way, this is rare to find in today's days. Your clarity in explaining the fundas is great and makes me learn so much more. Please continue sharing your knowledge thereby helping learners across the globe! Thank you for sharing this and best of luck to you on this journey!
literally no words i was struggling to find a perfect video which explains in every possible perception (user, developer, maintainer) this made all the doubts cleared ... thank you
Thanks for the video. It's worth stressing on is the consistency aspect of databases. RDBMS's provide strong consistency, whereas a NoSQL like Casandra provides eventual consistency, but higher availability owing to its masterless model, and thus lack of a single point of failure. HBase, on the other hand, is a single-leader system with a single-point of failure.
This is the best video on this topic on RUclips. This is a confusing topic but you explained in so simple and elegant way. Thankyou! for making this video.
This channel should really have lot of subscribers. This is the first channel where I found so much detailed explanation on system design. Thanks a lot.
Thank you so much for such a precise cheat sheet. You also cleared a lot of concepts for me which were fuzzy in spite of my 8.6 years of work experience!
Teaching is very diff from preaching! And Sir, you have cleary stated the difference. Making someone learn is easy, but making someone understand is what teaching is 👍 Hoping that your vedios will make a difference 😃
Really amazing playlist. I wonder why you don't have subscribers in lakhs. I have a final interview for SDE-I at a unicorn startup in 4 days and this is a life saver.
I had been struggling with starting Databases to build my conceptual knowledge and this is amazing for everyone. Thanks a lot for sharing this in the most simplistic way.
Your videos are very simple to understand. I would like to thank you for explaining the FR and non-FR for each system design consideration. That sets the expectation very well.
OMG, I didn't know shit about no-SQL and just with this video I think I can propose the adequate database type in an exercise. Amazing, specially loved the way to combine databases. Thank you so much.
Oh my God! I didn't know this kind of content is also available on the net and that too is free with so many details explained beautifully and concisely. Subscribed and gonna share it with others. Thanks again for putting so much effort for the learners like us.
Awesome content dude. I ensure I watch all the ads during the video and click on all of them because I don't know how else I can give back to you for the invaluable knowledge you share with us!
Cool and crisp details relevant to the topic. Very few people are gifted with the skill to share the knowledge and also create an impact. I came across this channel and now I'll be watching the whole series in coming days.
Wow !!! What clarity in explaining Database use cases, scenerios, categories of database with pictorial representation. The way you explained the concept is quite impressive. The way you corelate the concept with example is best fit. Keep the good work. May your light shine upon people like me the beneficiary. Thanks.
For cache -> Redis For streaming or video data -> blob storage. For example: amazon S3 For searching: text searching service like elastic search or solr Matrics or monitoring type system -> Time series database. For example, open tsdb. Analysis on data of company or data dump -> data warehouse. For example, hadoop
Every video i watch, I click like. Complex system design problems are explained in very simple way. Keep it up and make more such videos. Thanks a lot!
Bro, this is just soooo good. You summarised so many important topics of software development concisely in some minutes with awesome explanation. Hats off to you. first time watching your video and i have become a fan. Thanks for the great content!
One minor error in the video, when you explained the EverIncreasing Data, you used an example of uber getting location information of drivers by using driver id. I think this was a error. The use case here is the client requests for drivers in/around a certain location. So the columnar database will store drivers that are currently in a certain box (in this world), and the client will check which drivers are there in that box. I think that is what you meant. And this was a very good video. Thanks.
I have always wished for a video like this! Since, I am non-CS background, I am a bit oblivious to the world of DB products out there. I had always heard many of these names, but only as lingo - never really understood why they were being mentioned in the context. Thank you for putting this together, I will now probably understand more future such discussions in life :D
Really an awesome video. After seeing your one design video, I am on system design spree as if I want to finish the Netflix series. Every sentence you speak make to note it down in my notes. Thanks for all of it. One last thing, please whenever you time, keep the momentum of your system design input through videos.
Taking note of new info: 6:05 implementing text search feature, can use implementations provided by lucene, solar (TODO:- investigate these technologies) Elastic search is not a db, it’s a search engine so don’t use it as your primary source of truth Storing application metrics, throughput latency etc would use time series db. Useful when you’re doing sequential updates (appending metrics) instead of modifying existing records (kind of like a log) examples of time series db: InfluxDB, openTSdb
HBase is what’s called a wide column database not columnar. Columnar is still a sql like db that has its values stored in a inverted index instead of pages. Wide column is essentially key value value value … an example of columnar is sybase IQ or oracle with columnar compression turn on
This is a perfect learning video, thank you. I found it very useful how you associated real world examples with type of db. Info about use of combination of database is also lot of helpful understanding how real world system might be using them.
I was searching your Medium blog post for some good detailed explanation, I can say , one of best explanation so for, I am going with watch all of video.
THANKS A TON for making things amazingly simple to understand. Great stuff indeed! I have watched all your videos on system design, they are so nicely presented and organised. Awaiting the next videos.
Graph databases, designed to store and manage data in graph structures, offer unique advantages and some challenges compared to traditional relational databases. Here are the general pros and cons of graph databases: Pros 1. Flexible Data Models: Schema-less, Natural Representation 2. Performance: Efficient Querying, Scalability 3. Visualization Cons 1. Complexity and Learning Curve 2. Performance Considerations: Resource Intensive Use Cases Graph databases are particularly well-suited for: - Social networks and recommendation engines - Knowledge graphs and semantic search - Supply chain and logistics management - Bioinformatics and genomics research
Factors for choice of DB- 1. Structure of data - Structured or unstructured 2. Query Pattern 3. Amount of Scale Caching Use Case 1. Do not want to make multiple calls to DB 2. Remote call to different service with high latency Cache - Key: Value Redis - Battle tested, fairly stable Memcached File Storage - Amazon S3 ; Use Case- Amazon, Netflix Blob storage - image, video It cannot be queried. You serve a file as it is For Blob: Amazon S3 primary data store + CDN Use Case Text Search Capability with fuzzy search - Netflix, Amazon, Google Maps, Uber Text search engine - (Elastic search/ Solr) + Apache lucene: This should not be primary data store (as it can be lost), data can be loaded into this. Analytics on Data - Data Warehouse - For offline reporting Hadoop
You make me intelligent with every video I watch of yours.
Thanks!! That makes our day :)
Do share this with your colleagues and spread the knowledge :)
I agree with you 100%
Ture !!
Why no design videos these days?? What happend?
Just in case someone is still confused when to use column based DBs here's a little explaination:
Consider a database which has a large volume of data with 100 columns. And your application most of the time just queries only a few columns (let’s say 4-5 columns). Then in case of document DBs or row based DBs (relational Dbs) all data will be loaded row by row in main memory for processing , parsing and other operations. It will be a waste of a lot of effort in loading and processing of unnecessary columns. Column based DBs are used to tackle this problem where a large amount of data is stored in column based fashion so that the application can query only a few columns of large volume of data.
- reference: Designing Data-Intensive Applications. by Martin Kleppmann.
Well explained!!
But you can also do the same thing in RDBMS. you can easily right a query to get only few columns in the result
@@saurabhmidha3609 The data in RDBMS is stored row-wise in the disk in sequential blocks whereas in Columnar DB the data is stored column-wise making the query for certain columns simpler as all the column data is present in a singular sequence. Even though we can write a similar query for RDMS they would be inefficient as it would have to go through all the rows to only fetch a few columns' data.
I don't know why people are buying gold when by just watching this video you can be rich!
Sincere gratitude to you Sandeep!
WOW. Watched so many videos and finally found the gold content. No flashy animations and funky music. Just old school 1-1 explaining with clarity, examples and strong fundamentals. Beauty of online education.
Those who prefer AWS Managed Services. Here, is the list
Blob Storage : AWS S3
Caching : AWS ElasticCache
Full Text Searching : AWS ElasticSearch
Time Series Database : AWS TimeStream
Analytical Databases : AWS Redshift ( It depends, sometimes we can use AWS S3 for storing large csv files and then we can use AWS ElasticMapReduce(EMR) to process S3 Files)
RDMS (Relational Databse) : AWS Aurora
Document Database : AWS DynamoDB
Columnar Database : Not sure, I think AWS DynamoDB will satisfy this requirement too.
are you from AWS :)
Bo Wang No, learning AWS :)
Can we use RDS instead of Aurora for a relational database?
This has been the most useful 23 mins of System Designs Learning I've come across yet. And this is after I took an 11 hour course... you put it all together for me so much more succinctly. Thank you!!
He's amazing
Thanks for creating such an insightful video.
There are few things that I would suggest adding to all the important points that you have mentioned:
1> Write Heavy vs Read Heavy systems is also an important point to consider for database selection. We cannot write scale RDBMS (as generally in RDBMS the write is on the single primary instance). For DocumentDB, we need sharding to write scale it which can be done but are not so natural as it is in ColumnarDB like Cassandra where we need to mandatorily choose the partition key during the data modelling itself. So for write heavy systems I prefer to go with Cassandra if I don't need ACID. For write heavy systems that need ACID we should choose RDBMS with multi master configuration and for that application needs to take care of bifurcating the data to the appropriate master based on the decided key.
2> Graph database for mapping social network type of data.
Can you give examples of Write Heavy systems and Read Heavy Systems and why?
@@shponder9510 OLTP vs OLAP
@@shponder9510 URL shortener- you will have more reads than writes. may be 100:1 ratio of reads to writes
Best video tutorial available on the internet for choosing between databases for designing systems. Period
Thanks for the kind words :)
this video is an absolute gem. Finally someone that breaks this down directly without dancing around the topic
Best video I have watched which summarizes it so simply. The best part of this is he explains it by taking examples of products such as Amazon, Netflix etc.
As someone with only monolithic embedded systems experience, this is hands down the only resource on the internet that made sense to me in this domain. Heartfelt thanks!
I like the way you present.. you look like a humble human being with lots of knowledge and work experience... please continue your good work..
Your videos are a life saver for me. I have recently joined an organisation as senior developer and i have been given few system design tasks. And going through your videos is enabling me to understand the intricacies of the system design.
Reading this a day before on-site interviews. I am much more confident on the database selection justification :). Thank you
Sandeep: I must say that you have the gift of explaining complex concepts in a very simple way, this is rare to find in today's days. Your clarity in explaining the fundas is great and makes me learn so much more. Please continue sharing your knowledge thereby helping learners across the globe! Thank you for sharing this and best of luck to you on this journey!
Thanks Jyoti! Do spread the word about the channel. It motivates to keep making more :)
literally no words
i was struggling to find a perfect video which explains in every possible perception (user, developer, maintainer)
this made all the doubts cleared ...
thank you
Thanks for the video. It's worth stressing on is the consistency aspect of databases. RDBMS's provide strong consistency, whereas a NoSQL like Casandra provides eventual consistency, but higher availability owing to its masterless model, and thus lack of a single point of failure. HBase, on the other hand, is a single-leader system with a single-point of failure.
This man needs at least 100K subs. Love this content, extremely concise and informative, I have subbed!
This is one of the best explanation on choosing db....
This is the best video on this topic on RUclips. This is a confusing topic but you explained in so simple and elegant way. Thankyou! for making this video.
The clear explanation/flowchart at the end for choosing SQL vs document vs columnar was GOLD
No words to say..the way you've cleared everything.... You should be on RUclips long time before.
This channel should really have lot of subscribers. This is the first channel where I found so much detailed explanation on system design. Thanks a lot.
Seedhi Baat.. No Bakwaas.. Excellent short cheat sheet video for choosing right DB.
Really informative. Loved it, coming out burnt from GKCS, that guy always leaves me confused for some reason. This channel is must recommended.
What a very dense video! I took 3 pages of notes. The database world is much clearer for me. Thanks a lot!
Wow. That's good to hear!
Thank you so much for such a precise cheat sheet. You also cleared a lot of concepts for me which were fuzzy in spite of my 8.6 years of work experience!
Teaching is very diff from preaching!
And Sir, you have cleary stated the difference.
Making someone learn is easy, but making someone understand is what teaching is 👍
Hoping that your vedios will make a difference 😃
Thanks!! That's very good to hear :)
Hopefully it'll make a difference and reach a lot of people :)
@@codeKarle It is making a difference. Dope stuff!
Each minute is worth watching... Very logical and concise.
Really amazing playlist. I wonder why you don't have subscribers in lakhs. I have a final interview for SDE-I at a unicorn startup in 4 days and this is a life saver.
I had been struggling with starting Databases to build my conceptual knowledge and this is amazing for everyone. Thanks a lot for sharing this in the most simplistic way.
Your videos are very simple to understand. I would like to thank you for explaining the FR and non-FR for each system design consideration. That sets the expectation very well.
Brother, this is the best video I have ever seen on System Design (Databases). Thanks for sharing all this knowledge. Kudos to you!!
OMG, I didn't know shit about no-SQL and just with this video I think I can propose the adequate database type in an exercise. Amazing, specially loved the way to combine databases.
Thank you so much.
To the point and crisp.... Covered all popular types of DB. Thanks
Glad that you liked it!
Oh my God! I didn't know this kind of content is also available on the net and that too is free with so many details explained beautifully and concisely. Subscribed and gonna share it with others. Thanks again for putting so much effort for the learners like us.
Awesome content dude. I ensure I watch all the ads during the video and click on all of them because I don't know how else I can give back to you for the invaluable knowledge you share with us!
Thanks Abhijit!
Just spread the word about the channel amongst your friends / LinkedIn connections and that would be amazing :)
Thank you Karle, you have solved my mysterious about databases that has been going thru my head for many years.
Cool and crisp details relevant to the topic. Very few people are gifted with the skill to share the knowledge and also create an impact. I came across this channel and now I'll be watching the whole series in coming days.
One of the best System Design videos series on RUclips. Keep up the good work!
Wow !!! What clarity in explaining Database use cases, scenerios, categories of database with pictorial representation. The way you explained the concept is quite impressive. The way you corelate the concept with example is best fit. Keep the good work. May your light shine upon people like me the beneficiary. Thanks.
The intro part in which you used the explanation of functional and non-functional requirements made me like the video and comment here.
Good explanation of document DB, you are the first who made me understand the use case!
For cache -> Redis
For streaming or video data -> blob storage. For example: amazon S3
For searching: text searching service like elastic search or solr
Matrics or monitoring type system ->
Time series database. For example, open tsdb.
Analysis on data of company or data dump -> data warehouse. For example, hadoop
Every video i watch, I click like. Complex system design problems are explained in very simple way. Keep it up and make more such videos. Thanks a lot!
You have just fulfilled my all queries for interviews.
This is a very good video, something no one has done. So many terms I hear on a day to day basis but didn't know, are clear now.
Wow! This explanation is so lucid and logical, yet concise. Summarizing things this well should be considered an art form!
You are very clear about the content .. one of the best system design video available on RUclips..
Thanks Uday!
This was just what I needed, gem of a video. Thanks!
Wonderful video! I have been looking for something in depth like this for a very long time!
I was happy after going through the video. Just one short and clear.
Great explanation. One more point to add is also CAP therom need to be consider to choose the DB.
Excellent video talking about key considerations in choosing the "right" DB/ data store. Sincere thanks for your efforts!
I got lot of knowledge from this video and will watch all you videos in coming days
Bro, this is just soooo good. You summarised so many important topics of software development concisely in some minutes with awesome explanation. Hats off to you. first time watching your video and i have become a fan. Thanks for the great content!
Super helpful. More I watch it more clarity I get. Thanks for making this video.
Your videos are pure gold!! Hands down the best system design tutor!!! Thanks a lot!
One minor error in the video, when you explained the EverIncreasing Data, you used an example of uber getting location information of drivers by using driver id. I think this was a error. The use case here is the client requests for drivers in/around a certain location. So the columnar database will store drivers that are currently in a certain box (in this world), and the client will check which drivers are there in that box. I think that is what you meant.
And this was a very good video. Thanks.
This is awesome, I am amazed that everything you share has so much detail and thought.
Amazing explanation of various databases at a breadth. Candid for quick brush up. Thanks
Perfect cheet sheet for most of the DB formats..!!!
So beautiful and simple explanation..!!! 👏👏
Please keep up the great work..!!!
this guys is best! simple explaination.. easy to digest
I have always wished for a video like this! Since, I am non-CS background, I am a bit oblivious to the world of DB products out there. I had always heard many of these names, but only as lingo - never really understood why they were being mentioned in the context. Thank you for putting this together, I will now probably understand more future such discussions in life :D
Really an awesome video. After seeing your one design video, I am on system design spree as if I want to finish the Netflix series. Every sentence you speak make to note it down in my notes. Thanks for all of it.
One last thing, please whenever you time, keep the momentum of your system design input through videos.
awesome, the video gets to the heart of the matter and doesn't go around.
Taking note of new info:
6:05 implementing text search feature, can use implementations provided by lucene, solar (TODO:- investigate these technologies)
Elastic search is not a db, it’s a search engine so don’t use it as your primary source of truth
Storing application metrics, throughput latency etc would use time series db. Useful when you’re doing sequential updates (appending metrics) instead of modifying existing records (kind of like a log) examples of time series db: InfluxDB, openTSdb
What an amazing explanation for choosing databases. Kudos to you man.
Awesome !!
That's exactly what we require in a DB design rounds , in more depth though , but this was a very good intro in that direction
I have been looking for this information since long time.
Thank you for providing the exact detail
Awesome Explanation... Thank you, Sir... This is ULTIMATE....
Amazing video, one feedback, please keep it at the top of the series, before the beginning of the design discussion of other systems.
This video is exactly what I wanted for quick revision. Thank you so much
HBase is what’s called a wide column database not columnar. Columnar is still a sql like db that has its values stored in a inverted index instead of pages. Wide column is essentially key value value value … an example of columnar is sybase IQ or oracle with columnar compression turn on
This is a perfect learning video, thank you. I found it very useful how you associated real world examples with type of db. Info about use of combination of database is also lot of helpful understanding how real world system might be using them.
Bro the way you explain is just Amazing ... Speechless
Thanks for making it easy to understand
I was searching your Medium blog post for some good detailed explanation, I can say , one of best explanation so for, I am going with watch all of video.
THANKS A TON for making things amazingly simple to understand. Great stuff indeed! I have watched all your videos on system design, they are so nicely presented and organised. Awaiting the next videos.
Wonderful! The best free video for choosing database in system design interview. Thanks so much
Graph databases, designed to store and manage data in graph structures, offer unique advantages and some challenges compared to traditional relational databases. Here are the general pros and cons of graph databases:
Pros
1. Flexible Data Models: Schema-less, Natural Representation
2. Performance: Efficient Querying, Scalability
3. Visualization
Cons
1. Complexity and Learning Curve
2. Performance Considerations: Resource Intensive
Use Cases
Graph databases are particularly well-suited for:
- Social networks and recommendation engines
- Knowledge graphs and semantic search
- Supply chain and logistics management
- Bioinformatics and genomics research
Clear and to the point explanation. Great job! I learned a lot. Thanks.
Glad that it was helpful!
one stop explanation on choosing database, well done 👏
One of the best videos ever in this topic
Thanks
Keep doing the excellent work mate. This was a great video with respect to databases.
Thanks a lot for these videos. Big fan of such videos. Make sure you post more videos. Keep up your good work guys
Your explanation is very clear and to the point . Thanks a lot for contributing to the world of education . hats off Amazing explanation honestly
I had a lot of confusions on this, Thank You So Much!
I could binge watch your videos forever. such an informative videos
Love you brother. This video is so much informative. 1 video talks about so many solutions.
Very good summary. Concise and in-depth! thanks..
Thanks!! Glad that you liked it :)
All things explained straight and to the point. Hats off to you sir🕴🏻
I have a learnt a lot watching all your videos. Thank you so much sir 🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼
Eye opening. And you introduced me to time sequence databases. Thank you so very much. I would like to know about HBase at a beginner level.
Great video. Helped me to understand data platform at a high-level. Thank you Sandeep.
You explained everything in a very concise way. Keep up doing the good work😊🙏
Thanks a lot, it's very useful for my upcoming System design interview
Factors for choice of DB-
1. Structure of data - Structured or unstructured
2. Query Pattern
3. Amount of Scale
Caching Use Case
1. Do not want to make multiple calls to DB
2. Remote call to different service with high latency
Cache - Key: Value
Redis - Battle tested, fairly stable
Memcached
File Storage - Amazon S3 ; Use Case- Amazon, Netflix
Blob storage - image, video
It cannot be queried. You serve a file as it is
For Blob:
Amazon S3 primary data store + CDN
Use Case Text Search Capability with fuzzy search - Netflix, Amazon, Google Maps, Uber
Text search engine - (Elastic search/ Solr) + Apache lucene: This should not be primary data store (as it can be lost), data can be loaded into this.
Analytics on Data - Data Warehouse - For offline reporting Hadoop
👍
This is a super valuable video. It's beautiful. Thank you Karle.
Just wow! You made such a concise video for such useful information. Thanks a lot.
Wow!! Amazing video with great content in a concise manner. Kudos, thank you so much!