What tools should you know as a Data Engineer?
HTML-код
- Опубликовано: 27 июл 2024
- The modern data engineering stack is overwhelming...
Everyday there seems to be a new tool and it can be hard to know what to learn.
While it’d be impossible to share every single possible option...
in this video I want to share some of the most commonly used ones across the modern stack.
Use the tools mentioned in this video to guide your learning.
Figure out what you want to learn next and overall become more aware of what's out there.
*Important Note*
We'll be going through a lot of tools in this video. I want you to know that you don't need to learn how to use every single platformed mentioned to be a great data engineer.
I personally have not used all of them in depth and I've never met anybody who is an expert at them all either.
Ultimately the goal is to give you a perspective of the overall data landscape and help guide your journey to becoming a great engineer. Pick out one tool you don't know today and start learning it. Once you master that, move on to another.
►► The Starter Guide for The Modern Data Stack (Free PDF)
Simplify the “modern” data stack + better understand common tools & components → bit.ly/starter-mds
Timestamps:
0:00 - Intro
0:37 - Databases
2:32 - ELT Components
4:52 - Version Control & CICD
6:31 - Infrastructure
8:00 - BI & Analytics
Title & Tags:
What tools should you know as a Data Engineer?
#kahandatasolutions #dataengineering #analytics
►► The Starter Guide for The Modern Data Stack (Free PDF)→ bit.ly/starter-mds
Simplify “modern” architectures + better understand common tools & components
Hi. I tried the link but it says "forbidden". Is there another way to access the pdf? Thanks
All the names of the tools talked in the video:
*Coudbased db
Amazon Redshift
Google BigQuery
Snowflake
Azure Synapse
*Traditional row-based db
SQL Server
MySQL
PostgreSQL
*NoSQL db
MongoDB
elastic
cassandra
cosmosDB
amazon DynamoDB
*Extract & Load
Batch
Fivetran
Stitch
Airbyte
Azure Datafactory
Amazon Glue
*Streaming
Apache Kafka
Amazon Kinesis
*Transform
dbt - data built tool
*Reverse ETL
Census
hightouch
rudderstack
*Version Control & automation
GitHub
GItLab
CI/CD
*Task Orchestration & Scheduling
Apache Airflow
Jenkins
Luigi
*Infrastructure
Management
Terraform
Ansible
*Containers
Docker
*Container Orchestration
Kubernetes
*BI & Analytics
Reporting
Power BI
Tableau
Looker
*Open Source
Metabase
*Spreadsheets
Or just turn on subtitles ^ ^ but thanks for the effort :D
You have a new subscriber! I love the way you explain data engineering. You and Seattle Data Guy are my faves when it comes to Data Engineering Content Creators.
Thanks Turk! Much appreciated
thanks for an overview of the landscape!
What an absolutely power video. Please keep such good content coming!
Much appreciated! Thanks for watching
This is really helpful, Bro. Thanks a lot.
Yesterday
I said in your post
That its overwhelming with so many tools and today got a video :D
I got you! You're definitely not alone in that feeling so I figured it'd be a good topic for a video
This was a very informative video - very useful to "get the lay of the land" so to speak.
Glad to hear it! That was the goal
Thank you so much for this video! Really helpful!
Glad it was helpful!
Thanks you for great information.
Good stuff bro. I'd add prefect to orchestration/task flow.
Good call - Thanks for watching
You’ve got a new subscriber. Thank you
Thank you!
Some other alternatives for scheduling and orchestration are:
Dagster
Prefect
Oozie
Or whatever your cloud offering might have, I know Google Cloud has Cloud Scheduler.
If you suggest Jenkins as a job scheduling tool in this day in age, I will hunt you down...
Very good video. I think we can also add the cloud functions to this list.
This video is kick in the balls of Oracle 😀
thank you,great explaination
Glad it was helpful!
I'm really interested in this field and currently leaning Python. I must say this list is great but I'm really overwhelmed by the amount stuff one has to learn to transition in this field! I'm gonna stick with it and hopefully come through from the other end 😁
Definitely stick with it! One thing to remember is while there are many tools, you don't need to know ALL of them to have a successful career and you also don't need to learn all at once (it takes a whole career to do that).
Here is a recommendation to help you get started:
1. Start with getting very comfortable w/ SQL (and/or Python if you'd like)
2. Learn more about data modeling techniques (ex. dimensional modeling, star schema) and the way data typically moves (ex. ETL vs ELT)
3. Pick a common database to study and practice on (ex. Snowflake or SQL Server)
4. Learn how to use a tool like dbt to transform data within those databases which also will show you other important concepts like Version Control
5. Pick a data visualization tool (ex. Power BI or Tableau) and use your transformed data to make a cool dashboard
6. Pick another part of the process (ex. Extract tools, scheduling tools, etc.) and keep adding to your skillset
Good luck!
@@KahanDataSolutionsI really want to thank you for this thoughtful response and the road map provided. I honestly didn't expect this swift response and it shows that you love what you do! I will defo stick with it and hopefully make a successful career out of it. Thanks again 💪🏿
@@KahanDataSolutions this is a pretty good list! You could probably even do a video talking about this process
@@KahanDataSolutions Thank you for your extra detailed explanation to Adam 1. I would like to ask that this video would be more helpful for senior people who is deciding what their companies should use depend on their business case and requirements?
And about the spreadsheets part, you are def right. We are using Google spreadsheets and using python to automate the process to write our outputs there.
Phenomenal video. What tool(s) do you recommend for documentation and/or data dictionaries?
It depends where you store data on-primese or cloud.
Apache Superset is one of the promising BI tools in my opinion, Can you share your opinion on this, if possible
Brilliant
Apache Superset is another open source BI/analytics option
Could you please make a complete series on Apache Airflow ❤
Thanks!
Thank you!
Nice well rounded video, thanks !
One question, where does Databricks and spark fit into the stack?
Thanks! Databricks would fall in the same area as "cloud databases". Spark would fit in around the "ELT Components" and used primarily to process large amounts of data.
Hello! Thank you for your invaluable video! I find it extremely useful for beginners! I would like to ask about one thing regarding Data Engineer Career. I learnt Pandas in terms of Data Wrangling and Transformation. Therefore, how about Pandas for Data Engineers? Is it useful tool for ETL/ELT transformations? Obviously, the next step will be PySpark, but I would like to start learninig Pandas. It seems it is a good path for the next one. What do you think about it ? I would appreciate it if you could share your views about it.
Hey man, may i ask a question?
I have an ETL experiences with 2 etl tools and multiple RDBMS (on premise), and i wanted to shift into Data Engineering roles that works usually combining ETL Tools+Python and its libraries/frameworks, am i considered as new graduates or industry professionals? Since i don't have any experiences with Python ?
And does it usually means i have to take "paycut"? let's say i make $500 a month as ETL Developer, and i wanted to shift to Data Engineer roles , does it means i will be getting paid like $300 a month since i don't have DE experiences?
I really need some guidance... Thankyou :)
I really need this so bad. Do you have a Data engineer course ? Or any recommendations?
Apache airflow is a great Orchestration tool.
Hi, thank you for your video. I know that this is old now but I wish you would put the names of each tool you listed under the tool. If you aren't familiar with the specific tool it can be hard to know how to spell it. I know I can Google but I was taking notes as I was following along. Thank you.
Hi can you tell me where exactly apache spark fit in this picture
What about spark or pyspark? Where does it fit in?
I know Databricks, dbt, airflow, kafka and power bi
No Oracle for the second layer?
What about Clickhouse?
I want to be data engineer but still not good in programming language tried a lot python just know SQL how can I be data engineer
Very surprised Apache Spark is not mentioned here.
Same..
I guess you have not tried Exasol (analytical database, arguably the fastest in the market).
👍🏻👌🏻💯%
Is learning informatica worth it?
If you are applying for a job that uses it, then yes. I'm sure there are still many companies that use it.
ETL doesn't care what the destination is. The expression "Reverse ETL" makes no sense, it's still an ETL process.
I agree that the term is a bit odd, but that's what has stuck as of today. Another term you might see used to describe that process is "Operational Analytics"
Apache airflow gets older, lots of problems in production
Python..
Surprised there was no mention of Pandas.
That's a good one too. I personally haven't used Pandas much but I know others do.
Or Spark/Databricks
Companies in 2022 still running SQL Server with SSIS and SSAS :D
you just list out, half of the data team
(Devops Engineer, Data Engineer, DBA, SQL Developer, Server Executive, Data Analyst, Business Analyst),
You dont need to learn the all of this to be data engineer...