I have watched or read many explanation about the differences among these 3 terms, but so far this video is the simpliest yet cleariest and easiest to understand. Thanks a lot!!!
AWS, Azure are not Datalakes, they are Cloud Platforms, S3, Blob storage are examples of datalake on these platforms. On GCP example is Cloud Storage and NOT Big Query, BQ is not a datalake
Little correction - data warehouse is a system and/or db where Hundreds of heterogeneous dbs(eg- chocolate db, biscuits db, candy, icecream dbs) or file based systems like excel xml are altogether modelled/stored/streamed using ETL(tool) for data analytics & applications downstreaming, data science & AI build purpose also.
In a typical database there will be transactions taking place like insert of a table row, update of a table row, read of a table row that are in line with a set of business cases. In a datawarehouse there will be analysis taking place to across multiple rows from multiple tables. A data lake is where data goes to get drowned.
I dont think any other video in the internet explains this difference as clearly as this video. Thank you brother. Keep posting more videos to educate us.
I love how you indirectly went into explaining facts and dimensions for the database. Suggestion, it would be very helpful for these 3 concepts if you explained the concept of NOSQL and file based vs table based storages. Ofcourse there's alot more to it, but a simple summary and the benefits will do.
Super! In just 8 minutes, you have put such a clear picture of data base, data warehouse and data lake, that I can never forget and in future, any time I deal with these terminology, I have crystal clear idea of what am I dealing with! You are a GREAT teacher Chandoo and I really appreciate your effort!
i really appreciate your effort and time which you put into your video.TBH your video is on the point and very interesting i never thought that someone explain these topic that much easily.May God Bless you and give you r more power so you make more video for us
Great job, clear explanation and I also enjoy your humor. Would be great if you could create a video describing the difference between data scientist, engineer, analyst and architect. Kudos on your excellent work!
If you're starting in I.T. doing analysis type work, you'll start as an Analyst. This can be anything from reporting, automated feed maintenance/RCA, and even development. Most of the above 3 (maybe save for Data Scientist) start here. Data Engineer is probably the most logical next step from analyst. You'll definitely be doing more development and analytical work as an analyst prior to this. This shifts your scope from retrieving data from a data warehouse/db/lake (lake is quite rare for a run of the mill analyst), to actually designing and some possible light architecting of table/schema structures for data to import into from other sources (typically starting as transactional information into a database from an app, or maybe an external source of some sort). Typically as an engineer you won't start on data warehouse modelling until you've had some experience with general transactional architecting/engineering since the data within a warehouse shouldn't be updated/deleted, only inserted. It will be deleted, possibly if you've archived it in some situation (like data that's over x-years old and based on specific policies), but even then it probably wouldn't be deleted. If the architecture allows, you may just duplicate the tables, or partition them in some way and then archive the older pages. They may also determine certain structural recommendations (rowstore vs columnstore table structures, for example, or using NoSQL vs relational databases), but usually it's in concert with an Architect if the process being designed is large enough, or has significant impact, especially in terms of performance. However, after discussions between Engineers and Architects, the Engineers (and to a lesser extent, Analysts) will IMPLEMENT the requisites of decided Architecture. Engineers are typically more hands on than Architects, but Archs may get their hands dirty if something is largely conceptual and they want to start plugging away earlier in the phase to ensure design solidity. Data Architect is anything from designing the schema for your transactional infrastructure (your primary database), data warehouse, or even data lake, as well as helping navigate and determine how to import data into those repositories, as well as even more expansive things such as CI/CD pipelines, *maybe* networking tasks if you're familiar enough with that (usually system administrators do that, though), or even helping implement connection string/authentication against your cloud resource targets originating from nearly any source caller (on premises machine, like a developer computer, a VM hosting an app service, CI/CD agent, or a completely separate cloud service not native to your cloud service, even on a completely different domain or client server). An Architect is going to be responsible for HOW disparate system objects are going to interact with each other and any potential issues given certain implementations or design sequences. Typically Architects are going to have some knowledge as to what different approaches are available and determine which makes sense given what's required for the need or problem that needs resolution. As an Architect you're not expected to know how to implement everything as if you were doing all the work yourself. However, having a basic understanding of the limitations of each element in the design will definitely help you determine which is possible and which may not be earlier in design phase, which helps mitigate wasted developer time later during spikes (Proof of Concept phases) and help with further engineering alignment tasks. Most people consider scientists as the babies in the room because the data they require should be perfect in terms of not needing to accommodate any changes to their representations outside of any algorithmic modelling is concerned. It's entirely possible a Scientist will ask the Engineer to modify schema and data to accommodate some sort of analysis or data modelling they're trying to complete. It's not a-typical for an Engineer to work closely with a Scientist, but not typical for the Scientist to work with the Architect, aside from initial standing up of a new Data Warehouse or Data Lake. Typically the Engineer maintains or may make the every-day changes to those structures once the inputs/outputs/transformational processes have already been established. Scientists are typically Statisticians or anything having to do with applied mathematics. They will also typically work with code that isn't strictly SQL, such as Python, R, Power BI, DAX, (maybe MDX, but I think that's fallen largely by the way-side), etc...Scientists are tasked with supplying the answers to complex problems for the business using quantitative analysis. These are the people that determine what Ads you may see given your previous and most recent search history. Something you searched for 3 years ago may not be as relevant as something you searched for yesterday. That would be a typical example of what a Scientist may do. Also, Google translate, things like that will be developed by the Scientist, but the Architect will design the bridges to source that data whereas the Engineer will make that design a reality. The Analyst will make sure data makes sense as it starts trickling through the design process and if there's any issues, the Analyst and maybe working with the Engineer will troubleshoot the why/how and determine a fix where either of them may implement that fix to ensure it works as intended. If you look at it as a decision tree, it may look something like: Analyst > Engineer > Architect Analyst > Engineer > Scientist Analyst > Scientist (again, typically short cut by a Masters in Statistics or similar) Hope that helps!
Very good, thanks! What I got is that it is more of a conceptual difference rather than technical, understanding that there must be some infrastructure nuances..
I could be all wrong, please correct me if I am. BigQuery is a DW solution, not a DL solution. DL for BigQuery could be any storage layer like GCS. Any thoughts ?
you can not imagine how often i talked with high management and totally disillusioned them by explaining what a Datalake is. It's just the next buzzword not THE solution to all our problems .. sure it's useful of it's specific use case .. but that's it ^^
still not really sure why we want the warehouse. I mean, in reality, we would not really delete the chocolate we don't produce anymore (cause then we would have rows in sales which make no sense anymore). instead, it would get some kind of "valid" flag and just flag the old product. and if we have that, I can still easily answer the question how much of the old chocolate we sold in 2019...
There are big mistake in the video of "why company use dwh instead of db"...you tell that it is a problem of referential integrity which for the high end erp doesnt exists...is not the problem. The only purpose is to separete the olap load from oltp load as even the in Memory DB like HANA can not handle... It need at least 1 layer of data replication otherwise the performance for the users will be too low
I don't understand the point why historic information should be put in a different system (the data warehouse). If you wish to delete a product (in this case, a chocolate) from your active product line, why do you even need to delete the item from the db? You could just keep the product and maintain the info aboit active portfolio in the attributes or set up a different table for discontinued/active products. Having a separate system seems like an overly complex way to maintain this information. Can someone explain?
Why would you drop a chocolate from the DB even if you are not making it anymore? Let it remain there. Why go through all the bother of creating a DW? Or is it that the DB will become slower with outdated data so only that data is kept that helps with processign transactions?
I see the difference between database and data warehouse slightly different. I think database is storing data used by applications where data warehouse supposed to store data used by people. it means in database you will have tables connected to other tables with a lot of technical keys and other information useful only for the application. Where in Data warehouse you have nothing like that usually.
Correct me if im wrong. But i see database as the source of live dashboards? It is the representation of data that are being used currently. Data warehouse is like a storehouse for the historical data that the database have produced. Its like a back up copy of the data? And data lake is like a cloud storage where you just store all kinds of data randomly just for the sake of storage? It may contain datawarehouse data, tables, isolated tables, reports etc??
thanks is is pretty helppful. My boss told me we are planning to have a data lake in the future so I was unsure what it meant. Still things are confusing as I just entered the Data Analytics world.
This clearly explains that there is no real difference between a database, data warehouse, and datalake other than utilization. It all comes down to a database that can store text, images, and binary objects. It's just buzz words so HR can specify stupid requirements.
Hi, your video is very clear to understand the basics... but still I have question, please clarify. 1) why can't ETL take the source table and target table in the databases itself to create reporting or historical data table. Why we need to load into another database and call it as Data warehouse? Is there any significant difference like performance or something? Please explain this part... you explained "Why we are using each type", but I want you to cover why can't we use one instead of other. Eg., why cant we use create historical table in databases itself, why we need data warehouse separately. What is the special thing to go to DW instead of DB... also "why DL? And why can't DW?"
The answer is more technical. While you "CAN" keep both DB & DW kind of tables in the same place, normally people don't do it. Because, 1) Databases are "designed" so that they can add / change or delete data very quickly and efficiently. They also ensure that your data integrity is maintained (if a customer is deleted, they can no longer transact for ex.) 2) Data warehouses are "designed" so that they can add data and generate reports (or summaries) quickly. As there are usually no delete or edit operations in DW, the system is optimized to instead focus on piling up data and doing massive calculations quickly. 3) Hence, Internally the architecture and software / hardware design is different for these two systems. So it makes sense to keep them separate. Think of it like this. While both a car & tractor can drive, you won't use them interchangeably as they have their strengths & weaknesses.
@@chandoo_ Thanks for your reply. I wanted to know that "Design" behind the Strength and weakness of each type. I understand it is very technical and cannot cover in text reply. Thank you so much for explaining this to make us understand the differences.
Hmmm - for future consideration - I feel if you lightly explained what ETL processes actually do - and then offer super simple comparison of schema-on-write vs schema-on-read concepts (no need to wander into what a schema is specifically) you might dial in a better 8-minute intro and better contrast a DW from a DL. That might help viewers get why one might use a DL (streaming, for example) vs a DW (eg OLAP). And, maybe just offer a more generalized overview of what querying is overall vs bringing in specific examples as the query languages/tools really do not help compare the underlying DB principles. Most pointedly, I also find your callouts to public cloud services associated with DLs odd - cloud providers are actually better known for their DW services (Redshift, etc) - not to mention hosting traditional DWs like Teradata and the lot. Cloud companies simply reflect service delivery models for all three of these concepts. Just some thoughts for future content.
Great points Scott. As this is a concept level video, I tried to stay away from more technical aspects and implementation details of the individual platforms / ideas. I suggest watching the BI & DW terminology video for some clarity on a few of the things you mentioned - ruclips.net/video/a906b80lCE8/видео.html
That's what a data lake? Wow, I never knew the term, and have been in programming for 40 years.. back in the day, we just called them files/folders/drives, etc...
I have watched or read many explanation about the differences among these 3 terms, but so far this video is the simpliest yet cleariest and easiest to understand. Thanks a lot!!!
Wow.. thank you for that 😀
Exactly, this is how I feel. Thanks Chandoo.
I came to the comments to say the same thing! Thank you for this simple, illustrative explanation.
@@chandoo_ qqq
i very agree
Just found your channel. I’m sharing your videos with my team that is a bit behind on these concepts. Thanks!!
Very simple yet effective articulation!!
Thank you so much for the time put in your videos! extremely helpful!
it was smooooooooth...... THANKYOU😇
Thank you Chanduu bhaiya ✌🏻✌🏻
Nice, thanks
Best🤗🤗🤗🤗
amazing thanks
Most welcome
AWS, Azure are not Datalakes, they are Cloud Platforms, S3, Blob storage are examples of datalake on these platforms. On GCP example is Cloud Storage and NOT Big Query, BQ is not a datalake
Thanks for the clarification.
There is no other video on youtube that explains DB/DW/DL this easy. Really appreciate the time and effort you put into making these videos.
This video did a great job of helping me learn the distinction between these 3 things. Love it!
Thank you Demetri... 😍
Simple to start with. No PPT slides, just notepad is enough to explain ❤️ Thank you bro. Keep up your good work 👍
Little correction - data warehouse is a system and/or db where Hundreds of heterogeneous dbs(eg- chocolate db, biscuits db, candy, icecream dbs) or file based systems like excel xml are altogether modelled/stored/streamed using ETL(tool) for data analytics & applications downstreaming, data science & AI build purpose also.
@@ChrisSmithFW Yeah, but he forgot to mention so.
I believe that's what he said. His explanation is just a lot more understandable than yours.
Your answer sounds like quoted from an NCERT textbook and his is more like a next door tuition teacher
Wtf
Wise men can explain sophisticated things in a way that a 5-year kid can easily learn! Congrats Wise Man!
😍 That is a beautiful compliment. Thanks Amir.
I have seen many videos but this explanation is very nice and clear
I think because of your clear and concise points and humor, I learn more from you than other Excel tutorial channels.
Keep up the great work.
Aww.. that means a lot Patrick :)
In a typical database there will be transactions taking place like insert of a table row, update of a table row, read of a table row that are in line with a set of business cases.
In a datawarehouse there will be analysis taking place to across multiple rows from multiple tables.
A data lake is where data goes to get drowned.
"A data lake is where data goes to get drowned." 😂😂😂
I dont think any other video in the internet explains this difference as clearly as this video. Thank you brother. Keep posting more videos to educate us.
One more comment for me.
The best, most simple, laconic, yet rich, explanation about the diffs of the terms.
One of the finest explanations. 👍
Loved it ❤️
Glad you liked it!
I love how you indirectly went into explaining facts and dimensions for the database.
Suggestion, it would be very helpful for these 3 concepts if you explained the concept of NOSQL and file based vs table based storages.
Ofcourse there's alot more to it, but a simple summary and the benefits will do.
Super! In just 8 minutes, you have put such a clear picture of data base, data warehouse and data lake, that I can never forget and in future, any time I deal with these terminology, I have crystal clear idea of what am I dealing with! You are a GREAT teacher Chandoo and I really appreciate your effort!
i really appreciate your effort and time which you put into your video.TBH your video is on the point and very interesting i never thought that someone explain these topic that much easily.May God Bless you and give you r more power so you make more video for us
As a person in this industry, this is the best video ever. Exceptionally clear.
Great job, clear explanation and I also enjoy your humor. Would be great if you could create a video describing the difference between data scientist, engineer, analyst and architect. Kudos on your excellent work!
If you're starting in I.T. doing analysis type work, you'll start as an Analyst. This can be anything from reporting, automated feed maintenance/RCA, and even development. Most of the above 3 (maybe save for Data Scientist) start here.
Data Engineer is probably the most logical next step from analyst. You'll definitely be doing more development and analytical work as an analyst prior to this. This shifts your scope from retrieving data from a data warehouse/db/lake (lake is quite rare for a run of the mill analyst), to actually designing and some possible light architecting of table/schema structures for data to import into from other sources (typically starting as transactional information into a database from an app, or maybe an external source of some sort). Typically as an engineer you won't start on data warehouse modelling until you've had some experience with general transactional architecting/engineering since the data within a warehouse shouldn't be updated/deleted, only inserted. It will be deleted, possibly if you've archived it in some situation (like data that's over x-years old and based on specific policies), but even then it probably wouldn't be deleted. If the architecture allows, you may just duplicate the tables, or partition them in some way and then archive the older pages. They may also determine certain structural recommendations (rowstore vs columnstore table structures, for example, or using NoSQL vs relational databases), but usually it's in concert with an Architect if the process being designed is large enough, or has significant impact, especially in terms of performance. However, after discussions between Engineers and Architects, the Engineers (and to a lesser extent, Analysts) will IMPLEMENT the requisites of decided Architecture. Engineers are typically more hands on than Architects, but Archs may get their hands dirty if something is largely conceptual and they want to start plugging away earlier in the phase to ensure design solidity.
Data Architect is anything from designing the schema for your transactional infrastructure (your primary database), data warehouse, or even data lake, as well as helping navigate and determine how to import data into those repositories, as well as even more expansive things such as CI/CD pipelines, *maybe* networking tasks if you're familiar enough with that (usually system administrators do that, though), or even helping implement connection string/authentication against your cloud resource targets originating from nearly any source caller (on premises machine, like a developer computer, a VM hosting an app service, CI/CD agent, or a completely separate cloud service not native to your cloud service, even on a completely different domain or client server).
An Architect is going to be responsible for HOW disparate system objects are going to interact with each other and any potential issues given certain implementations or design sequences. Typically Architects are going to have some knowledge as to what different approaches are available and determine which makes sense given what's required for the need or problem that needs resolution. As an Architect you're not expected to know how to implement everything as if you were doing all the work yourself. However, having a basic understanding of the limitations of each element in the design will definitely help you determine which is possible and which may not be earlier in design phase, which helps mitigate wasted developer time later during spikes (Proof of Concept phases) and help with further engineering alignment tasks.
Most people consider scientists as the babies in the room because the data they require should be perfect in terms of not needing to accommodate any changes to their representations outside of any algorithmic modelling is concerned. It's entirely possible a Scientist will ask the Engineer to modify schema and data to accommodate some sort of analysis or data modelling they're trying to complete. It's not a-typical for an Engineer to work closely with a Scientist, but not typical for the Scientist to work with the Architect, aside from initial standing up of a new Data Warehouse or Data Lake. Typically the Engineer maintains or may make the every-day changes to those structures once the inputs/outputs/transformational processes have already been established. Scientists are typically Statisticians or anything having to do with applied mathematics. They will also typically work with code that isn't strictly SQL, such as Python, R, Power BI, DAX, (maybe MDX, but I think that's fallen largely by the way-side), etc...Scientists are tasked with supplying the answers to complex problems for the business using quantitative analysis. These are the people that determine what Ads you may see given your previous and most recent search history. Something you searched for 3 years ago may not be as relevant as something you searched for yesterday. That would be a typical example of what a Scientist may do. Also, Google translate, things like that will be developed by the Scientist, but the Architect will design the bridges to source that data whereas the Engineer will make that design a reality. The Analyst will make sure data makes sense as it starts trickling through the design process and if there's any issues, the Analyst and maybe working with the Engineer will troubleshoot the why/how and determine a fix where either of them may implement that fix to ensure it works as intended.
If you look at it as a decision tree, it may look something like:
Analyst > Engineer > Architect
Analyst > Engineer > Scientist
Analyst > Scientist (again, typically short cut by a Masters in Statistics or similar)
Hope that helps!
Ex:
DB Excel
DW Power BI
DL drive
Very cool 😎
Data gods have smiled. Thank you Chandu!!!
Thank you Venwhen.
Excellent video, with great and user friendly explanation. Loved it
Amazing info following frm long recently got promotted as manager in analytics. Thanks aton
You are tressure in you tube
Very good, thanks!
What I got is that it is more of a conceptual difference rather than technical, understanding that there must be some infrastructure nuances..
Sir, post a diagram with process between OLTP, OLAP AND DATA LAKE with ETL and ELT process.
AWESOME, SIMPLIFIED EXPLANATION THANK YOU SO MUCH.
I have learned a lot from you and thank you very much for clearly explaining 🙂
You are welcome Derrick. 😀
I could be all wrong, please correct me if I am. BigQuery is a DW solution, not a DL solution. DL for BigQuery could be any storage layer like GCS. Any thoughts ?
Thanks Chandoo.
RUclips algo was brilliant today suggesting me this goldmine!
Perfect explanation. I immediately subscribed 👊👊👊
you can not imagine how often i talked with high management and totally disillusioned them by explaining what a Datalake is. It's just the next buzzword not THE solution to all our problems ..
sure it's useful of it's specific use case .. but that's it ^^
I had no idea about data warehouse or data lakes. Thanks Chandoo for sharing your knowledge and the great breakdown of each.
damn 1 video is enough, lol edu RUclipsrs have to watch your videos before data laking those shitty explanations lol
Where would Snowflake fit into this picture? How about a MongoDB? Thanks, great work!
This video popped up in my timeline and i realized i absolutely need to know the difference lol
still not really sure why we want the warehouse. I mean, in reality, we would not really delete the chocolate we don't produce anymore (cause then we would have rows in sales which make no sense anymore). instead, it would get some kind of "valid" flag and just flag the old product. and if we have that, I can still easily answer the question how much of the old chocolate we sold in 2019...
very well explained, crystal clear, and superb way of explanation in between videos and images. Amazing.
You should add: data hub, delta lake, lake house, data virtualization.... A neverending story :)
There are big mistake in the video of "why company use dwh instead of db"...you tell that it is a problem of referential integrity which for the high end erp doesnt exists...is not the problem.
The only purpose is to separete the olap load from oltp load as even the in Memory DB like HANA can not handle... It need at least 1 layer of data replication otherwise the performance for the users will be too low
I don't understand the point why historic information should be put in a different system (the data warehouse). If you wish to delete a product (in this case, a chocolate) from your active product line, why do you even need to delete the item from the db? You could just keep the product and maintain the info aboit active portfolio in the attributes or set up a different table for discontinued/active products. Having a separate system seems like an overly complex way to maintain this information. Can someone explain?
I love this. This explains perfectly what I've been trying to explain at work. Instead of me keep arguing I am just going to show this video
There was not ppt , just sweet and crisp explanation of the topic using a notebook. 👌🏽 Loved it.
Thank you Martin 😀
Why would you drop a chocolate from the DB even if you are not making it anymore? Let it remain there. Why go through all the bother of creating a DW?
Or is it that the DB will become slower with outdated data so only that data is kept that helps with processign transactions?
I see the difference between database and data warehouse slightly different.
I think database is storing data used by applications where data warehouse supposed to store data used by people. it means in database you will have tables connected to other tables with a lot of technical keys and other information useful only for the application. Where in Data warehouse you have nothing like that usually.
I have been looking for an answer and I can't get it. Where do Data Warehouse Developers start? What's the roadmap?
Correct me if im wrong. But i see database as the source of live dashboards? It is the representation of data that are being used currently.
Data warehouse is like a storehouse for the historical data that the database have produced. Its like a back up copy of the data?
And data lake is like a cloud storage where you just store all kinds of data randomly just for the sake of storage? It may contain datawarehouse data, tables, isolated tables, reports etc??
Thanks a lot.Could you plz upload the video for Data integration ,in a simple Language so that layman can understand
lol, im thought that suddenly transformer trailer came while im interestly watching this video at 3.59
Great video keep it up. However don’t think difference between DL and DW is totally clear.
so well explained..!! very second worth watching.!! thanks a lot.
Thanks Man, I started learning big data concepts and this video is very useful for me
Please add the difference of the three concepts with BIG DATA. Now that would be a Part 2.
Hlo Sir, I am cofused difference between Data base, Data set & Data source.. Can u please help me in this?
Someday you will definetely create you own company called Awesome Chocolates)
thanks is is pretty helppful. My boss told me we are planning to have a data lake in the future so I was unsure what it meant.
Still things are confusing as I just entered the Data Analytics world.
Hello Chandoo thanks for explaining things so clean for us!!!
Wrong about big query.
Also whole of AWS and Azure do not qualify for data lake.
Simply superb... Greate explanation with general example..
I appreciate your acute humor Chandoo. Thumbs up for that alone. :) Thank you.
I love to pepper my videos with a bit of humor :) Thanks for watching and enjoying them 😀
Thank you for this information it's simple and easy to understand.
perfectly explained . Thanks you
Solid explanation. We can always count on Indian uncles for STEM
Also explain what is a repository, and it's different from you listed in video.
Hey Thank you Chandoo. For this simple explanation
"Noooo! Please don't call Excel a database!" - Every data engineer ever
Nice and simple explanation. Thank you.
Thanks bro as you made it easy for all of us to understand these concepts.😃😇
Nice easy explanation. Subscribed!
This clearly explains that there is no real difference between a database, data warehouse, and datalake other than utilization. It all comes down to a database that can store text, images, and binary objects. It's just buzz words so HR can specify stupid requirements.
i must say ,what a beautiful way to explain this ,,just wow man and funny though
Thank you so much. Clear video.
Please do vedios on azure databricks and synapse analytics ..
This dude is the first indian that I can understand
Mr chandu I didn't understand about lake...the rest two 👍
Great explanation and very easy to understand example.
Excellent. Thank you. You are a great teacher Sir.
😂 I like the way explain and prop’s 🎉😊🎃
Explained very clear! Thank you
Hey, Chandoo. Great explanation. Do you have any recommendations where to learn more advanced topics regarding data warehouse? Thank you
Cripsly explained and to the point, Thanks
BigQuery is more of a data warehouse than a data lake.
lake house is the mix of datalake and datawarehouse right?
you got cool style @chandoo. :) nice work.
Hi, your video is very clear to understand the basics... but still I have question, please clarify.
1) why can't ETL take the source table and target table in the databases itself to create reporting or historical data table. Why we need to load into another database and call it as Data warehouse? Is there any significant difference like performance or something? Please explain this part... you explained "Why we are using each type", but I want you to cover why can't we use one instead of other. Eg., why cant we use create historical table in databases itself, why we need data warehouse separately. What is the special thing to go to DW instead of DB... also "why DL? And why can't DW?"
The answer is more technical.
While you "CAN" keep both DB & DW kind of tables in the same place, normally people don't do it. Because,
1) Databases are "designed" so that they can add / change or delete data very quickly and efficiently. They also ensure that your data integrity is maintained (if a customer is deleted, they can no longer transact for ex.)
2) Data warehouses are "designed" so that they can add data and generate reports (or summaries) quickly. As there are usually no delete or edit operations in DW, the system is optimized to instead focus on piling up data and doing massive calculations quickly.
3) Hence, Internally the architecture and software / hardware design is different for these two systems. So it makes sense to keep them separate.
Think of it like this. While both a car & tractor can drive, you won't use them interchangeably as they have their strengths & weaknesses.
@@chandoo_ Thanks for your reply. I wanted to know that "Design" behind the Strength and weakness of each type. I understand it is very technical and cannot cover in text reply. Thank you so much for explaining this to make us understand the differences.
Hmmm - for future consideration - I feel if you lightly explained what ETL processes actually do - and then offer super simple comparison of schema-on-write vs schema-on-read concepts (no need to wander into what a schema is specifically) you might dial in a better 8-minute intro and better contrast a DW from a DL. That might help viewers get why one might use a DL (streaming, for example) vs a DW (eg OLAP). And, maybe just offer a more generalized overview of what querying is overall vs bringing in specific examples as the query languages/tools really do not help compare the underlying DB principles. Most pointedly, I also find your callouts to public cloud services associated with DLs odd - cloud providers are actually better known for their DW services (Redshift, etc) - not to mention hosting traditional DWs like Teradata and the lot. Cloud companies simply reflect service delivery models for all three of these concepts. Just some thoughts for future content.
Great points Scott. As this is a concept level video, I tried to stay away from more technical aspects and implementation details of the individual platforms / ideas. I suggest watching the BI & DW terminology video for some clarity on a few of the things you mentioned - ruclips.net/video/a906b80lCE8/видео.html
Super explanation sir, Thank you
Thanks bro for this simple explanation
That's what a data lake? Wow, I never knew the term, and have been in programming for 40 years.. back in the day, we just called them files/folders/drives, etc...
Welcome to the world of MARKETING my friend. We can pour old oil in a new can and sell it with a different name 🤦♂️
Chandoo...you have a divine gift for explaining things so clearly. The drawings help so much too. I wish I had found your channel sooner.
Lol I was getting Snickers adverts
I don’t get the different between data lake and data where houses
Great example, very well explained! Thank you for making this video
What a terrible explanation that says about nothing.