Database vs Data Warehouse vs Data Lake | What is the Difference?
HTML-код
- Опубликовано: 7 июн 2024
- Database vs Data Warehouse vs Data Lake | Today we take a look at these 3 different ways to store data and the differences between them.
Check out Analyst Builder! www.analystbuilder.com/
____________________________________________
SUBSCRIBE!
Do you want to become a Data Analyst? That's what this channel is all about! My goal is to help you learn everything you need in order to start your career or even switch your career into Data Analytics. Be sure to subscribe to not miss out on any content!
____________________________________________
RESOURCES:
Coursera Courses:
Google Data Analyst Certification: coursera.pxf.io/5bBd62
Data Analysis with Python - coursera.pxf.io/BXY3Wy
IBM Data Analysis Specialization - coursera.pxf.io/AoYOdR
Tableau Data Visualization - coursera.pxf.io/MXYqaN
Udemy Courses:
Python for Data Analysis and Visualization- bit.ly/3hhX4LX
Statistics for Data Science - bit.ly/37jqDbq
SQL for Data Analysts (SSMS) - bit.ly/3fkqEij
Tableau A-Z - bit.ly/385lYvN
Please note I may earn a small commission for any purchase through these links - Thanks for supporting the channel!
____________________________________________
SUPPORT MY CHANNEL - PATREON/MERCH
Patreon Page - / alextheanalyst
Alex The Analyst Shop - teespring.com/stores/alex-the...
____________________________________________
Websites:
Website: AlexTheAnalyst.com
GitHub: github.com/AlexTheAnalyst
Instagram: @Alex_The_Analyst
____________________________________________
All opinions or statements in this video are my own and do not reflect the opinion of the company I work for or have ever worked for
![Eminem - Houdini [Official Music Video]](http://i.ytimg.com/vi/22tVWwmTie8/mqdefault.jpg)
Alex, i don't know if you remember me. Over the past 2 years, I've been on your streams, commented on your videos, and emailed you my resume for help. I finally got a data analyst position 2 weeks ago and i love it. Thank you for everything.
I do! Congratulations!! That's so awesome to hear!
Love this!
Congrats man, how it's going ❤
amazing
Congrats!
Great breakdown of such critical concepts. I find that it's also common to get tripped up when companies decide to use such goofy internal names for their databases/lake/warehouse. To the point where you can lose sight of what it actually "is" you're being asked to work on. Being able to relate back to these foundational concepts is always a helpful exercise. Big fan of the channel btw!
This is great content, thanks. I've been working as an analyst/developer in anti-money laundering for over two years now, with zero tech experience going in. There are a lot of things where I've learned how to work with them without actually learning much about them, like the differences between databases and warehouses.
This sort of short-form content is useful for quickly covering those questions I didn't even realise I had. Will definitely be referring to in the future.
Timestamps for chapters:
0:00 Introduction
0:35 What is a Database?
1:13 What is a Data Warehouse?
2:34 Key Differences between Database and Data Warehouse
3:15 What is a Data Lake?
4:10 Database vs. Data Warehouse vs. Data Lake
In my previous job we worked with a Data Lake, which ended up being amazing for building general SQL skills as cleanup was needed to join the disjoint tables and get valuable results. I never looked at the exact definitions so thank you for this video!
Thanks for this. I already knew what a database and data warehouse was but had never heard of a data lake.
Short and to the point, just like i wanted. Thanks!
This was awesome! I'd been googling for answers to these very questions, and the visuals helped bring it into perspective. I see "data warehouse/ing" a lot in job descriptions, so it's great to get a handle on what it actually is.
Thanks for this! I would love to see you make a video on ETL’s and automation used in it breaking it down for rookie data analyst !
Yes! ETL is something I've been trying to learn and I got a new job where I will be able to learn more but having a source to learn the foundations would be great!
I would like to see this as well! Thank you!
I will love to have you do this for us at a granular level. Many thanks
I write a lot of my own ETLs, with T-SQL/powershell/cmd, etc., most very large amounts of delimited text based data into a Star schema type of data structure. I always build it to tolerate changing file structures, using a “Stage then Load” type of method, bulk load and stage to all Varchar defined columns then Dynamically build and merge (upsert) from stage to prod/fact table by stage/fact column name match via the system catalog or some type of metadata definition tables for column mapping/join and key columns to use for merging (mostly time series) data. It’s important to use features defined sample sizes (i.e. Batchsize) so as not to cause excessive T log or resource usage on the back end, especially if a shared Relational data server. Many of these vendor products cannot handle files that change, whether data type, column / header name change, etc. without the load failing, or requiring a new definition/format setup each time. I’ve actually come across one or two products that attempt a single transaction when loading very large data files to tables, that caused excessive T-log growth, and brought the DB server to a halt. I write mine so it will always attempt the load to “known” columns, then report any that were new or missing. I also define processes to “transform” based on staging to fact column data type match, one example being many files from excel sheets have this “serial” date value that needs to be cast to datetime from a 5 digit numerical value. I need to go the next step though and how to incorporate AI with it. We get tons of loan payment data to be analyzed, probably 40-50 GB a month, with 30 or so years currently stored in SQL tables, one table 4-5 TB in size, both page compressed and partitioned, used for both very selective queries and queries that process 2-4 year data samples. Fun stuff. I would love to learn new methods that successfully replace pre aggregating, that work fast and are very flexible. I’ve Worked in the past with products like Hyperion, Cognos, SSIS, etc. as well as some vendor DW products like Paladyne, but they are always an 80% solution, with the remaining 20% being 80% of the work.
Thank you so much. Having an already found understanding about OLTP and OLAP systems I was always struggling to understand what this datalake thing is! Now I know: It’s simply a file system. It’s a synonym for storage. Thank you for finally explaining this. I will always point to your video as a reference. 🎉
Nice breakdown. From your explanation, I get the impression that the neo4j graph database we used on my previous project was actually a data warehouse for us, because we filled it with data we drew from many other databases, and structured it in a specific way (relations between the many different data elements from the different systems) for the reporting tool we built on top of it.
And much of our data didn't come directly from those other databases, but drew it daily from an intermediary portal that contained all sorts of different kinda of data from different systems, which I guess was a data lake of some sort.
This was perfect. Short, sweet, and to the point. Thanks!
I have been wondering what the difference is between these three. You explained it very well. Glad I saw this video one day before my exam.
The way you break it down and make it simple is great. Thank you for such a quick and insightful video.
And now we see the latest iteration of a data topology, the lakehouse :)
Great video! i really like that you mention the summarisation as a point of differentiation. Analytics on a data warehouse is typically done by a data analyst with traditional data analyst skillsets (SQL, R etc). Analytics on a data lake is typically done by a data scientist (Python, ML etc)
An important point to note is "if you have all this data and you have no idea what to do with it" for data lake, a vital thing to focus on when creating a data lake is the structure of your data in the lake (as in directory and file structure)
if you just dump it all in, it will become a data swamp and waste of money
For sure - I'm implementing an Azure Data Lake at my company right now and that's exactly what we are trying to avoid lol
@@AlexTheAnalyst adls gen2 using hierarchical namespaces? good technology :)
A data swamp is a great addition to the list, although it's of course the thing you want to avoid.
Better lakehouse than leakhouse :D
Lakehouse is a term that marries the Data Lake and the Data warehouse. It does this by adding an abstraction layer to help organize and normalize data in the data lake with a combination of a hive meta store and a Delta format.
There are so many big universities around the world, but this guy made it so clear for me. You deserve to put your own University brother, thanks for enlightening me 🙏👍😊all the best 💐💐
Whoa, Alex with this clarity and instruction you're going to get my University ph.d instructors "Fired".
Understood the difference and I now know what I need for my own purposes.
Amazing content kudos 🥂
Simple and easy to use. Great voice and extremely friendly and humble.
By the first time, those concepts got clear to me. Thanks!
Nothing but awesome!! This is very nice Alex. You won't ever know how much you have come through from me. 👏
Appreciate the video. I needed to know the difference between the data platforms since at some point I’ll be transitioning to data engineering once I studied enough.
Also wanna take the time to say thank you so much for your vids! Been watching you since last year, your vids helped me prepared a lot for DA. As of now, I’ll soon be starting my 1st DA role with a great (I think lol) starting salary of $65k in SaaS. Would’ve been completely lost if it hadn’t been for your career insights and projects. Cheers from your HOU neighbor!!
I'm so honored to hear that! Keep up the good work!
Great video! I just understood the differences between these key terms, thanks to your video. Something I did not grab with a very long texts written for the same purpose . Well done Alex 👏
So simple and so fast!) Thank you so much)
Than you. I am making a similar transition into becoming a Data Analyst so this background is extremely helpful
.
This is an amazing summary, clear and easy to understand, especially for someone with networking and security background. Thanks a lot.
Alex, this is such a good concise description for folks! I’d love to learn more about data lakes, in my current role we have so many streams of data but apart from its initial use the data just gets lost in Excel files on shared drives.
I’m wondering if we could leverage it better by having a centralized way of storing it. But I just don’t know that much about data lakes. Would love more content exploring this!
Yeah that's usually what a data lake is for - some type of centralized system
Great explanation! Been struggling with understanding the differences, this really cleared it up!
Thanks for simplifying, great video! 👍
Thank you - for me, I learned new concepts - had no idea about data lake. Once again thanks for sharing your knowledge.
This was a great video. Concise and very clear. Excellent job on keeping it brief and giving all the information either. Could not have asked for better video
Oh. Much needed. Learing these concepts now on job
Clear and short, thank you!
Thank you Alex, this was a question I had in a recent interview for a MN company in Singapore, I did not know how to answer.
Thank you very much for this video. Actually I am currently doing the Coursera Google Data Analytics Course and in there I found these words databases and data warehouse and I was really curious about the difference between them and just found your video on it.
Short, helpful, well explained. Thanks!❤
Alex, it is awesome to see these videos man! Very informative.
Is this THE Moses Varghese?? Thank you man! :D
wow this nice, so much clarity and simplicity, thank you.
this helped me get a better grasp on the 3. great video
Super helpful. Thanks Alex
I don't know if I'll ever be ready for a Data Ocean...
A nice, clear presentation and nice explanations of the key terms. Thanks !
Hi Alex, would love to hear your thoughts on the subject matter of adjusting to a new analyst position. I’m a new senior data analyst and going through a steep learning curve. Would love to hear your advice. How long does it take? Tips on being successful, your personal experience, etc. I think it would be a great video.
I’m new to a higher level data position than I’m used to. I would love this kind of content as well.
This was the best explanation in RUclips. Thanks :)
It was helpful and made sense in terms of my current project too. Thank you!
Thanks for that clear concise explanation man.
This is a great video and you are a great communicator. I was having alot of trouble communicating this to my leadership team and this was really helpful!
amazing video- concise and very well explained
Super useful video. Much appreciated!
Thanks Alex. Would love a video on how you query data from data warehouses/data lakes and how that method differs from SQL
A data warehouse is queried using SQL, if the data warehouse is built using a relational database technology, such as MySQL, Oracle RDBMS, SQL Server. You usually don't query data lakes. They are mainly used for AI/Data Science purposes, and since they also contain non-structured data, you would not use SQL (Structured Query Language) which is designed for relational databases.
Dope video, thanks!
Super easy explanation. Thanks Alex.
Thanks Alex. This is simple and useful for me
Nice simple video. Good job. Only negative comment is, there's no need to constantly show your face which blocks part of what you're trying to show - diagram, title, description etc...
Thank very much, this is a very clear explanation.
Such a good and to the point explanation.thanks
clear, straightforward, fantastic ;)
thx
Very well done Alex!
Thanks for explaining, very clear now
Very informative,simple, easy to understand..thanks a lot
You're really good at walking people through concepts 👏👏👏👏👏👏
Very well explained. Thank you
Awesome video. I got a question about a data lake on an interview recently and was stumped, so this was helpful
Thank you. That helped me to understand and take care if opportunities I'm handling
Awesome explanation thanks
thank you Sir Alex. Great and concise video.
Really instructive video ! Thank you Alex for these details. I learned a lot.
Glad it was helpful!
very comprehensive video..Thanks!
very helpful! thanks :)
Alex always presents points in clear fashion.
The way you explained was very simple and easy to understand
Thanks
Very well explained. Thank you for sharing.
Super helpful thanks.
This is great!
best explanation of these 3 that I've seen
Very well explained!
Very informative. Thanks for sharing
Thanks, that was really well explained :)
thank you. great comparison and explanation
Much appreciated
keep up the great actions
Great video. Subscribed.
Great explanation! Thanks
You're welcome!
Cool stuff, short and informative.
Awesome explanation. Thank you.
I wish Alex would make his own course. Lol! Everything is always so easy to understand.
Thanks for this video
Very great explanation
Amazing explanation
wow! thank you!!!
Like your Explanation Sir.
thanks Alex, love your channel and very clear explanation of critical concepts :) - can you also cover data lakehouse?
Clear cut explanation!!! Thank you!!!
Glad it was helpful!
Such a good recommendation by youtube !
- Thnx Alex.
Glad to hear it!
Thanks for the explanation!
I would expect to see some examples for each one, it makes the terms much more 'close' to us..
Keep going with this channel, it's super helpful! Thanks again
ur welcome
Also have a look at Delta Lake. It’s about providing warehousing capabilities on a data lake, relying on a hive metastore and parquet normalization for column features. Delta lake is OSS and becoming more widely supported :)
Like a Data Lake-house then? :)
Great. I really like your videos
Thanks for your video, it is really clear and helpful information.
Hi Alex... Great video... I have a question. Is it right to say that Hadoop or HDFS is a data lake?
Great video.
Hi Alex, Thanks for the video. It is very clear. One question, what about the schema for Data Lake ? Where it is stored ?
very nice content!