Excellent video. The hardest thing about working in IT has been that every time I make a move to further my career, I find I have to build my own road map. Problem is, it’s impossible to know what all exists because there is so much. And every person you ask will give you some different list of things you need to learn, with no real concept of how it all fits together. As the technology expands, the need for roadmaps becomes so much more important than in previous years.
Yeah! I am really hoping it helps people take time after learning to really apply things. Otherwise that information disappears into some weird part of our brain that seems locked up.
bro I like to code , is data engineering for me ? because I have heard that DE mostly write sql when it comes to programming , and have they might some automation scripts for aws with python but that's not very often and is not a large part
I find it interesting that so many of these roadmaps focus so acutely on technical skills. Frankly, I’m not sure how valuable it is to learn AWS to ‘get into’ data engineering. You’ll learn this skill on the job when you get to it. Also this type of technology is highly dependent on what field you end up in. Lots of companies in the finance and insurance sector are nowhere near ramping up to AWS. I think learning the concepts of data storage, ETL pipelines and data warehousing - as well as having a general ability to write code in, say, Python - ought to suffice as a starting point. The hard part about technical roles like data engineering isn’t really writing the programs. The hard part is the softer skills involved and understanding the domain you’re working in.
bro I like to code , is data engineering for me ? because I have heard that DE mostly write sql when it comes to programming , and have they might some automation scripts for aws with python but that's not very often and is not a large part
@@prof.mangabhai There's such a wide range. SQL is a given. There are DEs that just do SQL and use low code tools. But also lots of DE positions focused on all of the other topics listed in this vid where SQL is a minimal part of the job.
I moved from DS to DE. My main tool is Python and I really work more as a backend engineer, but with a focus on automating data analytics and some machine learning stuff.
@@atwertsdfgsdfaasdfa I am a cs new grad with some full stack experience. I see a lot of backend engineer jobs desiring desiring kafka, airflow, Cassandra, and even some ML familiarity. It seems like DE skills are the way to go regardless of specific role. Taking the IBM DE course on Coursera and then maybe an AWS specific one if needed; hopefully it'll help me in this entry job rat race.
00:02 Data engineering roadmap is overwhelming with too many skills and tools to learn at once. 01:46 Follow a structured roadmap for learning data engineering 03:24 Build a Flask API to learn how to utilize different skills 04:51 Focus on different paths within data engineering 06:28 Build a project to scrape and store data 07:57 Data engineers build data infrastructure for future use. 09:32 Learning about unit tests, integration tests, and workflow tools is important for data engineering. 11:01 Focus on learning cloud and NoSQL databases early on 12:27 Focus on learning basic databases before delving into streaming and distributed systems 14:04 Prepare for interviews and build a project 15:32 Focus on building a third project and document it. 16:57 Focus on building a solid base for interviews and the first job
Learning about containerization from the outset is an important skill. Start by packaging an application such as an API and then perhaps containerize other components or processes such as data acquisition, cleaning, and loading into a database from which your API draws its data. Persistent data storage containers as early as possible will be useful.
@@SD-ko4tz APIs have become the primary access mechanism for external data sources, at least for mobile / web apps and a lot of websites. There are other use cases but I won't enumerate them here.
bro I like to code , is data engineering for me ? because I have heard that DE mostly write sql when it comes to programming , and have they might some automation scripts for aws with python but that's not very often and is not a large part
Thanks for the roadmap, im working as a developer for five years. Juz turn my mind to focus more on data engineering stuff starting 2022. It will be a good start to come across with the items mentioned in the map. Hope I can be a better man after the effort made. 2022 Wishes
This is awesome, I'm currently a data analyst and I have been searching for a way to make the switch to data engineering. Thank you for compiling and breaking down the tons of information that is out there.
Hey Seattle Data Guy, ruclips.net/video/E-x8v2KU0Lo/видео.html I have tried to explain stuff that's needed to become a data engineer in 2021 in the above link. Try checking that out as well. Hope it helps.
Amazing!. I am a Python software developer, so I have a background in making APIs and Unit Testing as well as some skills using Docker. But definitively I need to understand how to model Data Warehouses, and some cloud tools and play with Luigi or Airflow. I felt happy watching your video because I realized that I am on the correct path. I have a nice number of tools in my bag, but I have to keep working to get the other ones. Thank you for this great video and excellent road map, mate.
Thank you for this - it's exactly what I was looking for. I have some SQL skills and I work with relational databases, but I want to make a career shift into Data Engineering. I was starting to get overwhelmed by all the options!
Glad this could help! Who knew that more knowledge and access to it..would make it more difficult. Hopefully this roadmap helps make the path clearer. good luck on your data engineering journey.
Thank you for this roadmap. I have a background in IT and I’m trying to make a lateral move into a jr data engineer role. I’ve been studying for 5 months now with UDACITY, and up until this video I still had no idea how to correctly maneuver into a DE role without feeling overwhelmed. Thanks 🙏🏽
Thank you so much! I appreciate every like, subscription, etc. I really want to help build the data engineering and data community so I will keep trying to put out great content.
Thanks man! I'm a HRIS and Reporting Specialist looking to make a switch to the other Data scene. I found a lot of your videos very down-to-earth and concise to learn more about Data Engineering.
Hi! I started a data engineer internship that week and I find myself here. Your video is great as I think It should help me. I didn't know where to start and how to use my time efficiently so a huge thanks for the video and the roadmap!
Thank you so much! I just graduated and now I'm very interested in data engineering field. This is definitely the channel that I'm looking for. Keep uploading!
love this kinda videos, still relevant in 2023. i started a data science course but it wasn't really my jam and i tend to lean towards backend stuff anyways.
Well detailed roadmap. I have 10 years of experience in modeling using Ralph Kimball's techniques and ETL (Sagent, Oracle ODI), where I stopped (2012) completely and moved on to the commercial area, selling technology with spin selling techniques (ERPs, BI, low-code...). Today I want to go back to my initial area, which is now something part of Data Engineering. Watching many of your videos, to understand how to get back to this area that I love so much (or once loved).
Thank you so much ,I was desperate navigating the diagram shown in the beginning (actually it was hanging on my wall giving me anxiety about actually being able to finish it)👍
Great video Ben! I am definitely sure I want to pursue a career in Data Engineering but felt like I was all over the place when trying to learn the skills. This is definitely helpful, thank you!
Thanks for the comment Sebastian! Yeah, it can feel super overwhelming. Luckily when I first started I didn't know as much about the entire field so I just learned things slowly without being overwhelmed. Now with all the courses, articles and tutorials its crazy. Hopefully this helps silence some of the noise.
For sure! I think its always great to stop and actually apply knowledge. Of course its a struggle to sometimes stop for projects since you will often feel like your learning rate will decreasing but you're picking up so many skills!
@@SeattleDataGuy I spent the last few days learning how to containerize and deploy a private git repository service using git and gitolite. It was the first time that I learned how to use volumes (not host file system volumes) but self-contained volumes independent of the host file system.
@@SeattleDataGuy definitely. I am now converting the Docker configuration for use with Docker Compose to simplify the deployment of the 3 components. This experience should be useful for data engineering implementations too, but I won't know until I give it a try.
It really is. Just build a strong foundation and then learn how to learn just enough of a tool when you need it. Or... Pick a tool that is not going anywhere soon and master it. Become a specific consultant and charge like $800 an hour. Kind of a joke.
@@SeattleDataGuy "Pick a tool that is not going anywhere soon and master it. Become a specific consultant and charge like $800 an hour. Kind of a joke." genuine question is this sarcastic? Could you kindly suggest which tool isn't going anywhere soon and to master? Damn I would be happy with $50 dollar an hour nevermind $800.
Funny you referenced the first road maps since I saw those and those discouraged me, but your road map is very realistic and I'm excited to finally see something I want to do, thank you!
This is the best and most detailed roadmap for DE that I've yet seen. My question is: Is this roadmap aiming to make you an absolute shoo-in candidate? Is it possible to apply for a job and get one before the end of the roadmap? I'm asking because it seems that the demand for engineer is high and the labor pool is smaller.
I would say if you do everything on this roadmap you will have a good high level understand of most data engineering concepts. For most interviews I have found that a general understanding of technology(Programming, design, SQL, testing, and maybe a few other odds and ends) will be good enough. Especially if you are going for a more junior position. So You can probably get halfway through this roadmap and start putting out resumes.
Im a Data Analyst and considering Udemy's course but didn't know about Googles Cloud! Thank you. I applied for a local company and the only Data Engineer is leaving soon :(
Thanks! I'm an experienced SQL DBA, but looking to progress my skills and talents, not necessarily change companies yet, but at least find a way to wrangle the data we handle at my current company.
@@SeattleDataGuy I guess to keep things specific to my company, we're an Azure shop mostly, and simple data flows to something that can be used by powerBI for example. I knew SSIS, so that's a big piece. I'm thinking the other major thing to learn is data warehousing, that's my missing piece it looks like.
Makes sense. Yeah, data warehousing is generally that other component. Its funny how you can use tools like SSIS and not even realize they are connected to data engineering or data warehousing.
@@SeattleDataGuy ETL is one of those in-between "glue" products that is not sexy, but when push comes to shove, is powerful. SSIS has a LOT of legs left in it, although I do know the writing is on the wall with Python and T-SQL combined.
@@gabyabed Forget Python and T-SQL...Snowflake just has tasks that are basically just stored procedures you can put on a schedule + have dependencies...its tech..there is always new but there will always be companies that never switch.
As an UX Designer from IT-sphere I always look at that amount of knowledge I have to learn to be able to become a Data Engineer and become very sad that it's almost impossible to figure out where to start especially when you're 33.
Thanks @ Seattle. This is really explanatory and clear. Please i could'nt find the links for building the flask api or any of ther projects you listed.
Thanks for sharing the roadmap!!..I have been thinking of moving to data science realm and this roadmap will definitely help. I watched some of your videos where you provided courses/resources for SQL, ETL, Data warehouse, Cloud etc..Could you please provide some courses for Python from Data Engineering perspective (not Data scientist)?
I will put together a python focused video. I was recently checking out a few courses on sites like data quest and Udemy for data engineers. So I will try to see how these are and report back.
Please make a video for interview preparation for data engineering , which sources to use for SQL practice (Leetcode SQL is very basic and might not be useful for data engineering SQL e.g. calculating active daily users etc) , practice data warehouse modeling or how to test a particular funnel etc.
Hi Ben! Thank you for the roadmap. I am torn in choosing whether to go with being a mobile engineer/developer or a data engineer. If you are in my place what will be your decision and which do you think will be more in demand in the future with these two? Thanks again!
Thank you for the comment. I imagine both will have a similar demand. I would say you should take a few courses in both and build a mini-project and see which area you like more. Programming and data in general will be a necessary skill in the future. So I wouldn't worry about the perfect choice. Instead, focus on which area you enjoy.
@seattle Data Guy: It would be beneficial to introduce some materials (PDF, Link, RUclips channel) for each specific skill set. That is the most challenging part for learners
This is AWESOME ; I'm studying Python, C# and Java right now ; but , I have an unending urge to pursue Data Engineering. Would it be possible to gain employment in the Data Engineering field without any sort of college degree ? Thanks for everything !
Sure you could get a data engineering degree without a college degree..but it will be hard. Because either you have experience or college to get that first job. If you can get experience without a degree then you will be set..but thats hard to do.
Hey, thank you so much for this information, I have a few questions if I may: 1. I am a CS major, with no previous experience in this field, do I need to pursue my masters as well? If so, what master should I pick to suite data engineering? 2. As a fresh university grad, can I land entry level job in this field?
In general, most technical roles like software or data engineering don't need masters degrees as long as you can pass the interviews. Generally, its a good idea to get an internship in a DE position and then get a returning offer. That way you can get experience. There are jr. positions, but I noticed...as per usually...most of them want 2 years of experience.
Great road map. What's your take on the Udacity Nanodegree program: How to Become a Data Engineer. Is worth having a look at for someone with no prior experience?
i like my mac...but as long as your computer 8-16gb of ram you'll probably be fine..There are for sure some engineers out there running circles around me on 2 Gbs of ram in vim
bro I like to code , is data engineering for me ? because I have heard that DE mostly write sql when it comes to programming , and have they might some automation scripts for aws with python but that's not very often and is not a large part
It depends where you're working at what role. You might prefer a role like Software Engineering, Data these roles tend to be focused on building infra to handle big data
Do you think getting the AWS Data Analytics Certification is a good idea and it will help in this road to become a Data Engineer? Also, do you have any source that works as a guide to learn how to document or give proper structure to the projects in github that will serve as portfolio? Your content is great by the way, greetings from Costa Rica.
I though I responded to this one. I would say focus on getting data engineering certificates. With AWS you could get the big data aws certificate. Of course, that also depends where you are in your technical journey. You can use some of felipe hoffas pieces of work as an example. I am working to get him onto the channel so he can talk about his projects. Here is one of his projects hoffa.medium.com/400-000-github-repositories-1-billion-files-14-terabytes-of-code-spaces-or-tabs-7cfe0b5dd7fd#.qm2s97y25
Hi, Thank you for this video. How much time do you think it would take to complete all the steps mentioned here ? I work in IT but have mostly worked as a consultant on the ETL tools like Informatica etc. I know about Data but Data Engineering is somewhat new to me.
Great content! definitely subscribed. I am a Software Engineer wanting to transition into Data Engineering, What is your take the Google Cloud Certified Professional Data Engineer Certificate Exam? Do you think that certs help?
I actually have a video on the google cloud certificate. I think, especially as a SWE it makes sense to get certs just so it becomes clear that you are making the switch. I usually say certs aren't that helpful for most people because they are getting it with 0 experience. But since you have experience, it can help round it out.
A good number of the Udemy courses you recommended in the pdf are quite Out-dated, that is a big learning challenge. It would be very appreciated if you review this video in your free time.
Hey! I am currently going into my third-year of undergrad majoring in CS, found your page and now am interested in Data Engineering. I am beginning the UDemy A-Z SQL and Database course you recommended in another video to get a taste of data engineering. If I end up liking data engineering and want to take the next steps in this video, do you think I can complete at least up to step 7, 8, or 9 by the end of August if I keep my head down and grind? Currently have a Data Science/Analyst internship at a small company for the summer, but want to pivot and secure a big Data Engineering internship for next summer by being prepared for interviews in the Fall/Winter. Thoughts?
If you have time perhaps. But truthfully it sounds like you have an internship and you should focus on that for the summer. Maybe you can intern at the company again next summer and ask them to let you try out DE.
That's hard to say. But here is how I knew. I thought I wanted to do data science. I put a lot of effort learning about it...and yet eventually i found out i prefer the engineering aspect of data. I enjoy building infra, not creating temporary analysis. You really won't know until you try, and don't worry, sometimes we have to move laterally in life to move forward.
@@SeattleDataGuy this right here was how I felt, I was almost wrapping up the Google data analytics course on coursera but was honestly not satisfied with doing just temporary analysis, i wanted to create something!! And lucky for me I found this road map, I am starting immediately looking forward to becoming a DE
I was working as a system administrator from 10 years but now learning Azure data engineer and applied few companies but all are asking relevant experience in data engineer what should I do to get experience?
Hey Ben! Thank you for the vide. It's crisp and comprehensive at the same time. You mentioned that you'll be sharing resources/links to learn/explore flask api. But, I can't find the link anywhere. Could you please pinpoint the link? Thank you again!
Good point about building your own APIs so you would know how to use them. Do you recommend we build our own OSs, Web Servers, and our own Apache Sparks so we can get the hang of it as well ?
If you want to learn more about what is going on in the data space, then sign up for my newsletter! seattledataguy.substack.com/
sir what about the microsoft certified: Azure Data engineer associate certificate, would you also recommend it
What is the highest paid tech salary?
God bless you for the info on that Data Engineering Road Map about documentation and Courses.
Excellent video. The hardest thing about working in IT has been that every time I make a move to further my career, I find I have to build my own road map. Problem is, it’s impossible to know what all exists because there is so much. And every person you ask will give you some different list of things you need to learn, with no real concept of how it all fits together. As the technology expands, the need for roadmaps becomes so much more important than in previous years.
Loved this road map that you shared! Also liked how it integrated learning, projects, and then building further. Solid! 🙌🏼
Yeah! I am really hoping it helps people take time after learning to really apply things. Otherwise that information disappears into some weird part of our brain that seems locked up.
I just want to thank both of you for paving a roadmap for me into Data Analytics and Data Engineering with your content. You guys rock!
@@NP_Omar Aww thanks for this Omar!!
@@LukeBarousse I agree with Omar, you and Ben are both amazing (long time follower here :D).
bro I like to code , is data engineering for me ? because I have heard that DE mostly write sql when it comes to programming , and have they might some automation scripts for aws with python but that's not very often and is not a large part
I find it interesting that so many of these roadmaps focus so acutely on technical skills. Frankly, I’m not sure how valuable it is to learn AWS to ‘get into’ data engineering. You’ll learn this skill on the job when you get to it. Also this type of technology is highly dependent on what field you end up in. Lots of companies in the finance and insurance sector are nowhere near ramping up to AWS.
I think learning the concepts of data storage, ETL pipelines and data warehousing - as well as having a general ability to write code in, say, Python - ought to suffice as a starting point.
The hard part about technical roles like data engineering isn’t really writing the programs. The hard part is the softer skills involved and understanding the domain you’re working in.
Agree but some tools are more general, like Kafka, Spark may help with initial getting job offers.
bro I like to code , is data engineering for me ? because I have heard that DE mostly write sql when it comes to programming , and have they might some automation scripts for aws with python but that's not very often and is not a large part
@@prof.mangabhai There's such a wide range. SQL is a given. There are DEs that just do SQL and use low code tools. But also lots of DE positions focused on all of the other topics listed in this vid where SQL is a minimal part of the job.
I moved from DS to DE. My main tool is Python and I really work more as a backend engineer, but with a focus on automating data analytics and some machine learning stuff.
@@atwertsdfgsdfaasdfa I am a cs new grad with some full stack experience. I see a lot of backend engineer jobs desiring desiring kafka, airflow, Cassandra, and even some ML familiarity. It seems like DE skills are the way to go regardless of specific role. Taking the IBM DE course on Coursera and then maybe an AWS specific one if needed; hopefully it'll help me in this entry job rat race.
00:02 Data engineering roadmap is overwhelming with too many skills and tools to learn at once.
01:46 Follow a structured roadmap for learning data engineering
03:24 Build a Flask API to learn how to utilize different skills
04:51 Focus on different paths within data engineering
06:28 Build a project to scrape and store data
07:57 Data engineers build data infrastructure for future use.
09:32 Learning about unit tests, integration tests, and workflow tools is important for data engineering.
11:01 Focus on learning cloud and NoSQL databases early on
12:27 Focus on learning basic databases before delving into streaming and distributed systems
14:04 Prepare for interviews and build a project
15:32 Focus on building a third project and document it.
16:57 Focus on building a solid base for interviews and the first job
Learning about containerization from the outset is an important skill. Start by packaging an application such as an API and then perhaps containerize other components or processes such as data acquisition, cleaning, and loading into a database from which your API draws its data. Persistent data storage containers as early as possible will be useful.
great idea! sounds like you know your stuff
Huh? Never used API s in my 30 ywars
@@SD-ko4tz APIs have become the primary access mechanism for external data sources, at least for mobile / web apps and a lot of websites. There are other use cases but I won't enumerate them here.
bro I like to code , is data engineering for me ? because I have heard that DE mostly write sql when it comes to programming , and have they might some automation scripts for aws with python but that's not very often and is not a large part
Thanks for the roadmap, im working as a developer for five years. Juz turn my mind to focus more on data engineering stuff starting 2022. It will be a good start to come across with the items mentioned in the map. Hope I can be a better man after the effort made. 2022 Wishes
Glad this could be helpful! Good luck on your DE journey!
Don't worry... 15 years into my it career I still write hello world projects for new technologies and languages!
Great video thanks.
Glad you enjoyed it!
This is awesome, I'm currently a data analyst and I have been searching for a way to make the switch to data engineering. Thank you for compiling and breaking down the tons of information that is out there.
I am glad you found this helpful! Good luck on your data engineering journey.
If you enjoyed this video, then sign up for my substack seattledataguy.substack.com/
Hey Seattle Data Guy,
ruclips.net/video/E-x8v2KU0Lo/видео.html
I have tried to explain stuff that's needed to become a data engineer in 2021 in the above link. Try checking that out as well. Hope it helps.
Amazing!. I am a Python software developer, so I have a background in making APIs and Unit Testing as well as some skills using Docker. But definitively I need to understand how to model Data Warehouses, and some cloud tools and play with Luigi or Airflow. I felt happy watching your video because I realized that I am on the correct path. I have a nice number of tools in my bag, but I have to keep working to get the other ones.
Thank you for this great video and excellent road map, mate.
Glad you found this helpful! It sounds like you're on the right path!
Thank you for this - it's exactly what I was looking for. I have some SQL skills and I work with relational databases, but I want to make a career shift into Data Engineering. I was starting to get overwhelmed by all the options!
Glad this could help! Who knew that more knowledge and access to it..would make it more difficult. Hopefully this roadmap helps make the path clearer. good luck on your data engineering journey.
Have you been learning how to be a DE?
Thank you for this roadmap. I have a background in IT and I’m trying to make a lateral move into a jr data engineer role. I’ve been studying for 5 months now with UDACITY, and up until this video I still had no idea how to correctly maneuver into a DE role without feeling overwhelmed. Thanks 🙏🏽
I do hope this helps you move into a data engineering role! Good luck
Hey Harvey how are you finding the udacity de course , I'm thinking I would like to do it
I like the way you encourage your viewers in final part of your videos.
Was really looking forward to this. Great work man!
Thank you! I am glad you enjoyed it!
This guy posted this for 6 days with nearly 400 likes and not a single disliked. That make me definitely give him a subscription
Thank you so much! I appreciate every like, subscription, etc. I really want to help build the data engineering and data community so I will keep trying to put out great content.
Please continue making videos on Data Engineering. They are very helpful !
I will keep making data engineering videos!
Thanks man! I'm a HRIS and Reporting Specialist looking to make a switch to the other Data scene. I found a lot of your videos very down-to-earth and concise to learn more about Data Engineering.
I am glad you are finding these videos helpful! Good luck on your data journey!
Hi! I started a data engineer internship that week and I find myself here. Your video is great as I think It should help me. I didn't know where to start and how to use my time efficiently so a huge thanks for the video and the roadmap!
Yeah! I am glad I could provide some signal in all the noise.
Thanks ... i have been working towards transition to a data engineering role and I see this is the most concise path...
Thanks seattle-guy
Good luck, let me know how it goes!
Thank you so much! I just graduated and now I'm very interested in data engineering field. This is definitely the channel that I'm looking for. Keep uploading!
Awesome! I appreciate all the support. I have plenty more videos coming
Wow, this is golden! I think I'm at stage 4. Starting to learn about CI/CD. Yes indeed, writing a blog is a great way to learn!
Like number 170 :)
I am glad you found it helpful! CI/CD is so complex, there are so many layers. Are you focused a lot on deploying models?
@@SeattleDataGuy At the moment, it’s for updating a website’s content via GitHub. And yes also looking into model deployment as well.
@@DataProfessor Awesome, for model deployment, are you looking into mlops tools?
@@SeattleDataGuy Yes definitely, but a newbie at it, learning a lot from your videos Ben 😆
Thank you! I look forward to making more and learning a lot along the way.
One of the best video on data engineering I've come across. Great Work !!
I am glad you enjoyed this data engineering video!
Thanks. I got thrown into a data engineering role because of a unfilled vacancy. I'm trying to skill up for the position. This was a real help.
love this kinda videos, still relevant in 2023. i started a data science course but it wasn't really my jam and i tend to lean towards backend stuff anyways.
Big thanks to you bro, much needed video for my life.🙏
you're welcome!
Thanks!
You're too kind. Thank you for thanks. Let me know if there are any specific videos you are interested in seeing!
Well detailed roadmap.
I have 10 years of experience in modeling using Ralph Kimball's techniques and ETL (Sagent, Oracle ODI), where I stopped (2012) completely and moved on to the commercial area, selling technology with spin selling techniques (ERPs, BI, low-code...). Today I want to go back to my initial area, which is now something part of Data Engineering.
Watching many of your videos, to understand how to get back to this area that I love so much (or once loved).
Thank you so much ,I was desperate navigating the diagram shown in the beginning (actually it was hanging on my wall giving me anxiety about actually being able to finish it)👍
Hopefully it doesn't stress you out too much!
Very informative. I'm taking udacity's data engineering nanodegree and we're doing a similar roadmap. Good job!
Glad it was helpful!
Best guide I have gotten so far. Helps to clear up the confusion
Glad this data engineer guide helped clear up some of the noise!
Great video Ben! I am definitely sure I want to pursue a career in Data Engineering but felt like I was all over the place when trying to learn the skills. This is definitely helpful, thank you!
Thanks for the comment Sebastian! Yeah, it can feel super overwhelming. Luckily when I first started I didn't know as much about the entire field so I just learned things slowly without being overwhelmed. Now with all the courses, articles and tutorials its crazy.
Hopefully this helps silence some of the noise.
Really enjoying your channel!
Thank you! I appreciate all your comments and support. Got to feed the YT algo.
Thank you for this breakdown, I will update you when I get that Data Engineering job!
Awesome! Good luck on your data engineering journey. #dataengineering
Adding those projects to the roadmap is much appreciated.
For sure! I think its always great to stop and actually apply knowledge. Of course its a struggle to sometimes stop for projects since you will often feel like your learning rate will decreasing but you're picking up so many skills!
@@SeattleDataGuy I spent the last few days learning how to containerize and deploy a private git repository service using git and gitolite. It was the first time that I learned how to use volumes (not host file system volumes) but self-contained volumes independent of the host file system.
That's awesome. Do you feel comfortable with them now?
@@SeattleDataGuy definitely. I am now converting the Docker configuration for use with Docker Compose to simplify the deployment of the 3 components. This experience should be useful for data engineering implementations too, but I won't know until I give it a try.
Yeah it will be! There is a ton of DevOps style work we sometimes have to take on. So that should be helpful.
Finally...you have come back with a.........plan *the rock voice*
Hahaha! That sounds about right. Gotta alway approach things with a plan
Thank you very much Ben, I learned that my ftp/tftp projects will serve for something,
Anyway, great video, keep up the good work insh'Allah
Happy to help!
Learning DE is like being trapped in vendor tool hell. There are so many tools it is really overwhelming. Thanks for this vid.
It really is. Just build a strong foundation and then learn how to learn just enough of a tool when you need it.
Or...
Pick a tool that is not going anywhere soon and master it. Become a specific consultant and charge like $800 an hour. Kind of a joke.
@@SeattleDataGuy "Pick a tool that is not going anywhere soon and master it. Become a specific consultant and charge like $800 an hour. Kind of a joke." genuine question is this sarcastic?
Could you kindly suggest which tool isn't going anywhere soon and to master? Damn I would be happy with $50 dollar an hour nevermind $800.
Wow this is awesome! You've clearly explained each step on the road map. Great video!
I am glad it was clear! I need to check out your channel as well!
@@SeattleDataGuy Of course! See you in there :)
awesome content! your channel is such a great resource. really thoughtful comment section too!
Yeah we have a great community! Welcome to all. Data engineers, analysts, scientists and everyone else
Thanks Ben! This is gonna help someone like me whose between step 1 and 2 a lot and I appreciate all your work 🙌🏻🙌🏻
Glad to hear it was helpful. Good luck on your data engineering journey!
its a very informative. Also links given for other technologies to prepare is also much commendable. Thank you so much for sharing all those.
I am glad you are enjoying the content!
Funny you referenced the first road maps since I saw those and those discouraged me, but your road map is very realistic and I'm excited to finally see something I want to do, thank you!
Yeah! You don't need to learn everything. Just one step at a time. Good luck!
I LOVE YOU SO MUCH DUDE!!
Thanks for sharing this with everyone, super helpful as I go through this career transition!
Good lcuk!
please more emphasis on the projects. I think it would truly help new learners. thank you!
I am going to finally try to film the data engineering project video. I am mentally stuck and need to just do it.
This is the best and most detailed roadmap for DE that I've yet seen. My question is:
Is this roadmap aiming to make you an absolute shoo-in candidate? Is it possible to apply for a job and get one before the end of the roadmap? I'm asking because it seems that the demand for engineer is high and the labor pool is smaller.
I would say if you do everything on this roadmap you will have a good high level understand of most data engineering concepts.
For most interviews I have found that a general understanding of technology(Programming, design, SQL, testing, and maybe a few other odds and ends) will be good enough. Especially if you are going for a more junior position. So You can probably get halfway through this roadmap and start putting out resumes.
Great Roadmap!
Glad you enjoyed it!
Thanks for the roadmap! It helps me a lot. 🤧
Glad to hear that! Good luck
Fantastically concise video, thank you.
Glad you liked it!
Wonderful content .Crisp, Clear & to the point. Thank you
Thank you!
Im a Data Analyst and considering Udemy's course but didn't know about Googles Cloud! Thank you. I applied for a local company and the only Data Engineer is leaving soon :(
Sounds like a great opportunity to move into being a DE if you wan that.
Thanks! I'm an experienced SQL DBA, but looking to progress my skills and talents, not necessarily change companies yet, but at least find a way to wrangle the data we handle at my current company.
Thats interesting! Are you looking for specific tools or skills?
@@SeattleDataGuy I guess to keep things specific to my company, we're an Azure shop mostly, and simple data flows to something that can be used by powerBI for example. I knew SSIS, so that's a big piece. I'm thinking the other major thing to learn is data warehousing, that's my missing piece it looks like.
Makes sense. Yeah, data warehousing is generally that other component. Its funny how you can use tools like SSIS and not even realize they are connected to data engineering or data warehousing.
@@SeattleDataGuy ETL is one of those in-between "glue" products that is not sexy, but when push comes to shove, is powerful. SSIS has a LOT of legs left in it, although I do know the writing is on the wall with Python and T-SQL combined.
@@gabyabed Forget Python and T-SQL...Snowflake just has tasks that are basically just stored procedures you can put on a schedule + have dependencies...its tech..there is always new but there will always be companies that never switch.
loved this roadmap
Glad you liked the data engineering roadmap!
Great outline!
glad you found it helpful!
Yes, I transitioned to cloud engineering and sooo overwhelming. I appreciate all maps lol
It can be overwhelming. You just have to take it one step at a time.
I know most of the "things" , but it just that nothing is in order. Let me bring it in order. THANX for detailed video.
Good luck my friend!
Awesome video man thanks!! Do you have a specialization in any of those skills/tools?
Great video!!
Starting from near zero today!
Side note, I see The Alchemist on the side. Great book!
yeah! solid book
EDIT: Wow...100 likes before 24 hours..thats a first
Before 12 hours!
@@LukeBarousse I can't wait until the day I am trying to get 10k before 24 hours....but for now...this is exciting!
@@LukeBarousse Add to that...1k views before 12 hours! just barely. Also a first!!!
@@SeattleDataGuy look at u now!
As an UX Designer from IT-sphere I always look at that amount of knowledge I have to learn to be able to become a Data Engineer and become very sad that it's almost impossible to figure out where to start especially when you're 33.
Hopefully this helped?
@@SeattleDataGuy of course!
This was really helpful thank you so much!!!
I am glad you enjoyed the video!
Thank you so much for this video!! 🙂
Glad it helped!
Is data engineering for you, is awsome post.
this is such an amazing video. thank you.
Glad you liked it!
Thanks @ Seattle. This is really explanatory and clear. Please i could'nt find the links for building the flask api or any of ther projects you listed.
That I believe was just a suggestion. But try tech with tim ruclips.net/video/GMppyAPbLYk/видео.html
Thanks for sharing the roadmap!!..I have been thinking of moving to data science realm and this roadmap will definitely help. I watched some of your videos where you provided courses/resources for SQL, ETL, Data warehouse, Cloud etc..Could you please provide some courses for Python from Data Engineering perspective (not Data scientist)?
I will put together a python focused video. I was recently checking out a few courses on sites like data quest and Udemy for data engineers. So I will try to see how these are and report back.
This is top class!!!!
Thank you!
Good information.... thanks a lot.
you're welcome!
Excellent content.
Glad you enjoyed it!
Please make a video for interview preparation for data engineering , which sources to use for SQL practice (Leetcode SQL is very basic and might not be useful for data engineering SQL e.g. calculating active daily users etc) , practice data warehouse modeling or how to test a particular funnel etc.
Interesting. I have a few other videos on my list, but I like the video idea.
If you add some networking to the roadmap you'll have a roadmap to a backend software engineer :D
Hi Ben! Thank you for the roadmap. I am torn in choosing whether to go with being a mobile engineer/developer or a data engineer. If you are in my place what will be your decision and which do you think will be more in demand in the future with these two? Thanks again!
Thank you for the comment.
I imagine both will have a similar demand. I would say you should take a few courses in both and build a mini-project and see which area you like more.
Programming and data in general will be a necessary skill in the future. So I wouldn't worry about the perfect choice. Instead, focus on which area you enjoy.
Thank you very much!🙏
You are welcome!
@seattle Data Guy:
It would be beneficial to introduce some materials (PDF, Link, RUclips channel) for each specific skill set. That is the most challenging part for learners
Especially about networking, OS, containerization, ... that are too broad
This is AWESOME ; I'm studying Python, C# and Java right now ; but , I have an unending urge to pursue Data Engineering. Would it be possible to gain employment in the Data Engineering field without any sort of college degree ? Thanks for everything !
Sure you could get a data engineering degree without a college degree..but it will be hard. Because either you have experience or college to get that first job. If you can get experience without a degree then you will be set..but thats hard to do.
Hey, thank you so much for this information, I have a few questions if I may:
1. I am a CS major, with no previous experience in this field, do I need to pursue my masters as well? If so, what master should I pick to suite data engineering?
2. As a fresh university grad, can I land entry level job in this field?
In general, most technical roles like software or data engineering don't need masters degrees as long as you can pass the interviews. Generally, its a good idea to get an internship in a DE position and then get a returning offer. That way you can get experience. There are jr. positions, but I noticed...as per usually...most of them want 2 years of experience.
Great road map. What's your take on the Udacity Nanodegree program: How to Become a Data Engineer. Is worth having a look at for someone with no prior experience?
Thank you for this video.
Is it important I get a laptop n which one should I get.
I'm just getting started today
i like my mac...but as long as your computer 8-16gb of ram you'll probably be fine..There are for sure some engineers out there running circles around me on 2 Gbs of ram in vim
bro I like to code , is data engineering for me ? because I have heard that DE mostly write sql when it comes to programming , and have they might some automation scripts for aws with python but that's not very often and is not a large part
It depends where you're working at what role. You might prefer a role like Software Engineering, Data these roles tend to be focused on building infra to handle big data
I think a video about the Data Mesh would be interesting from you
I have thought about it for sure
legend, thank you :)
Glad you liked it!
Do you think getting the AWS Data Analytics Certification is a good idea and it will help in this road to become a Data Engineer? Also, do you have any source that works as a guide to learn how to document or give proper structure to the projects in github that will serve as portfolio?
Your content is great by the way, greetings from Costa Rica.
I though I responded to this one. I would say focus on getting data engineering certificates. With AWS you could get the big data aws certificate. Of course, that also depends where you are in your technical journey.
You can use some of felipe hoffas pieces of work as an example. I am working to get him onto the channel so he can talk about his projects. Here is one of his projects hoffa.medium.com/400-000-github-repositories-1-billion-files-14-terabytes-of-code-spaces-or-tabs-7cfe0b5dd7fd#.qm2s97y25
Hi, Thank you for this video. How much time do you think it would take to complete all the steps mentioned here ? I work in IT but have mostly worked as a consultant on the ETL tools like Informatica etc. I know about Data but Data Engineering is somewhat new to me.
Great content! definitely subscribed.
I am a Software Engineer wanting to transition into Data Engineering, What is your take the Google Cloud Certified Professional Data Engineer Certificate Exam? Do you think that certs help?
I actually have a video on the google cloud certificate. I think, especially as a SWE it makes sense to get certs just so it becomes clear that you are making the switch. I usually say certs aren't that helpful for most people because they are getting it with 0 experience. But since you have experience, it can help round it out.
A good number of the Udemy courses you recommended in the pdf are quite Out-dated, that is a big learning challenge. It would be very appreciated if you review this video in your free time.
Thanks..this is amazing
Thank you for the comment! This comment is amazing.
Is it true that I *must* start w/ Front-end because companies seldom hire junior Back-end developers/ engineers.
No, I know plenty of people who got back end jobs out of college. At least in the US
Excellent!
Hey! I am currently going into my third-year of undergrad majoring in CS, found your page and now am interested in Data Engineering. I am beginning the UDemy A-Z SQL and Database course you recommended in another video to get a taste of data engineering. If I end up liking data engineering and want to take the next steps in this video, do you think I can complete at least up to step 7, 8, or 9 by the end of August if I keep my head down and grind? Currently have a Data Science/Analyst internship at a small company for the summer, but want to pivot and secure a big Data Engineering internship for next summer by being prepared for interviews in the Fall/Winter. Thoughts?
If you have time perhaps. But truthfully it sounds like you have an internship and you should focus on that for the summer. Maybe you can intern at the company again next summer and ask them to let you try out DE.
How would I know that I'll fit for data engineering role before I get started with the journey.
That's hard to say. But here is how I knew. I thought I wanted to do data science. I put a lot of effort learning about it...and yet eventually i found out i prefer the engineering aspect of data. I enjoy building infra, not creating temporary analysis. You really won't know until you try, and don't worry, sometimes we have to move laterally in life to move forward.
@@SeattleDataGuy this right here was how I felt, I was almost wrapping up the Google data analytics course on coursera but was honestly not satisfied with doing just temporary analysis, i wanted to create something!! And lucky for me I found this road map, I am starting immediately looking forward to becoming a DE
I was working as a system administrator from 10 years but now learning Azure data engineer and applied few companies but all are asking relevant experience in data engineer what should I do to get experience?
Hmm, any way you can start working on some projects at your current job. Thats usually best
Must I learn Scala for Spark programming? Do FAANG companies use PySpark in production?
it always depends...facebook and amazon use python, Netflix uses scala, google I believe is also python.
what is store encrypted data into sftp and pull data from sftp? is not sftp just a transfer protocol? could you explain pls
Hello, thanks for the video. I want to get into IIOT space, Do you think I should focus more on streaming analytics or data warehousing
Hey Ben! Thank you for the vide. It's crisp and comprehensive at the same time. You mentioned that you'll be sharing resources/links to learn/explore flask api. But, I can't find the link anywhere. Could you please pinpoint the link? Thank you again!
It's in the Google drive link in the description
is there a server course you recommend ? I'm looking around but I'm not sure where to start and to even search for.
Good point about building your own APIs so you would know how to use them. Do you recommend we build our own OSs, Web Servers, and our own Apache Sparks so we can get the hang of it as well ?