Also, if you'd like to dive deeper into data strategy and infrastructure and you'd like to support me, you can consider becoming a paid member of my Substack. I have over 100 articles that cover everything from data engineering 101 to leading data teams. Sign up with the link below and get 30% off. - seattledataguy.substack.com/148e9023
This is really funny because this year my company offered me the posibility to enter the Data Department (I'm a Software Dev.). I'm kind of fed up with coding everyday and for now I'm learning Elastic Search, Apache Nifi and Airflow, then I'll head to Spark and AWS. This video is pretty intereseting to serve as a guidebook. Thanks for stressing the part of "What have you learned so far?"
One potential key topic is how to bounce back from difficult client/project that did not go as successful as desired (especially in data consulting if you have had that experience). For me, because of this experience, it made me lose confidence during a client screening to try & explain this experience & re-frame it as positive.
I am a business intelligence engineer at a big tech company and I just told my manager that I'm interested in transitioning to a DE role. This looks like a great resource to help me shore up some of the skills that I am lacking. Thanks for putting all of this together!
Data Engineering means different things to different companies. For example, you don't list SAP as a core competency, but some companies do. WIth that said, isn't it better to pick a lane and stay in it? I.E. one or two of the cloud providers, AWS or Azure or GCP, and master data engineering, data analysis, machine learning, AI services, etc.
Would pet projects be enough to show some knowledge of Spark/Databricks or Azure? I feel like without previous real work experience it's impossible to even land an interview for any data engineering related position.
looking into this now, solid background in data primarily SSIS, informatica, Alteryx with a good amount of "BI" work. Thanks for this guide, we use Azure in our current work environment but mostly only at the corporate level. I'm going to challenge myself to follow this guide so I can transition into being a data engineer!
Hi, I recently decided to change my career into the data field and interested in data engineering. Your channel has been very helpful thank you. My question is, I never coded before. Would you recommend taking a Computer Science course as a prerequisite to have a baseline foundation on computers and coding? (For example: Harvard CS50 course?)
The spreadsheet points to the AWS Solutions architect cert course. It's 10 hours long. Since then the same author published a 50 hour video as an update. Is it worth it to go through it all?
I think it comes down to semantics. Some companies might call it ETL Engineer but it is essentially the same thing. Think of it this way, companies have pain points and they usually search for someone with a specific skill set to fix that specific problem (or a set of problems). It's like companies who are looking for a "python developer", "MySQL Developer", "PostGres Admin" etc.... but when you read the job description, the skills requirements listed are basically a Data Admin, Data Engineer, or even Data Analyst. Just know in the IT industry in general there is a lot of crossover in skills and duties. A Data Engineer is a "Software Engineer", "Database Admin", "Data Analyst" (some times), " Data Cloud Engineer/Cloud Engineer" etc...just read the job description and determine if it something you can do. Do not get hung up on job titles.
Thanks alot for the roadmap man! I appreciate. I just have a question for the cloud services like GCP, AWS courses should I take them all or just pick one. Also many are suggesting GCP. What's your take
Start with AWS. My suggestion would be to be book an AWS associate exam (Solutions Architect Associate ) 6months/1year from now and start preparing for different services through udemy course by stephaane Marekk. I did the same and cleared the above exam in 5 months (won't recommend to do it though as it takes a lot of effort in preparing , i did this only because my company was paying for the certification which i had to pass anyhow).
So if im only interested in AWS, do that 10hr video, and skip the other cloud days and move on? That cloud video is 4-5 years old, is it still relevant? Does solutions architect teach you all the relevant tools in aws for a data engineer?
I have a video idea for you. What would be an example of the minimum project portfolio to begin applying for jobs without previous engineering experience? Like 5 legit projects that include ____ & _____ & ____? I have no idea what's a realistic amount of coding to be considered for a coding entry level job....
Hi! I'm newbie in this field. Now, I'm following your roadmap in the video posted 2 years ago. Is this new roadmap better? Should I change my current roadmap to this video?
if you've gone through most of the prior roadmap, i'd finish that. I just wanted to create a timeline vs. the old roadmap that doesn't have a clear time constraint
I don't recall saying DE will end, but it will change. When i started everyone was on hadoop, now everyone wants to be on snowflake, databricks, spark, etc.
Hi Ben, I recently got laid off and looking into following your guide here. My background is in google analytics and Looker but we didn’t use too much SQL. Debating between Data Engineering or Cloud Engineering. I do love to build and create new things. Do you have any advice? I need to find a job quick and concerned data engineer could be saturated. What do you think?
@@SeattleDataGuy thanks I'm actually a Computer systems engineering student currently in 4th year have basic knowledge on Databases and cloud computing I'm hoping to enter Data engineering field in the future
I have a question? I am interested in becoming a data engineer but I want to ensure I go about things diligently. I see a lot of boot camps where I live in the UK and also the option to go to university. I’m open to both but what would be better for a learning time frame of 3 years?
@seattledataguy i have started the datacamps data engineer track. completed about 6 courses in there. what should i do . start this one or continue with that one
I believe most cloud solutions offer a free tier or trial. For example snowflake does like 30 days free or aws has its free tier - aws.amazon.com/free/
there are a few references in the excel link but you can find the book here 160592857366.free.fr/joe/ebooks/ShareData/The%20Data%20Warehouse%20Toolkit.pdf
Also, if you'd like to dive deeper into data strategy and infrastructure and you'd like to support me, you can consider becoming a paid member of my Substack. I have over 100 articles that cover everything from data engineering 101 to leading data teams. Sign up with the link below and get 30% off. - seattledataguy.substack.com/148e9023
inv is invalid :
Also upload videos on same format on
Data Analyst
and
Data Scientist
I have 300 days to learn 🕶️
This is really funny because this year my company offered me the posibility to enter the Data Department (I'm a Software Dev.). I'm kind of fed up with coding everyday and for now I'm learning Elastic Search, Apache Nifi and Airflow, then I'll head to Spark and AWS. This video is pretty intereseting to serve as a guidebook. Thanks for stressing the part of "What have you learned so far?"
How is the work life balance in this career?
What do you think about ai replacing software and data jobs?
@@JKyadav111 About as good of a chance that AI will replace doctors and engineers
This couldn't have come at a better time for me, and a free sheets document to track all of this? So valuable, thanks!
glad you're finding it helpful!
One potential key topic is how to bounce back from difficult client/project that did not go as successful as desired (especially in data consulting if you have had that experience). For me, because of this experience, it made me lose confidence during a client screening to try & explain this experience & re-frame it as positive.
Hmm, this does sound like a solid video topic! Also I hope you feel confident again!!
@@SeattleDataGuy thanks. U the best :)
I am a business intelligence engineer at a big tech company and I just told my manager that I'm interested in transitioning to a DE role. This looks like a great resource to help me shore up some of the skills that I am lacking. Thanks for putting all of this together!
Data Engineering means different things to different companies. For example, you don't list SAP as a core competency, but some companies do. WIth that said, isn't it better to pick a lane and stay in it? I.E. one or two of the cloud providers, AWS or Azure or GCP, and master data engineering, data analysis, machine learning, AI services, etc.
Will be following your map for the next 6 months as I am into 9-5 routine.
Starts at 3:58
Yeah I didn't realize how long my intro was!
Would pet projects be enough to show some knowledge of Spark/Databricks or Azure? I feel like without previous real work experience it's impossible to even land an interview for any data engineering related position.
Thanks for the video. Another video idea would be one for finding / getting your first DE job. It's a bit of a struggle finding entry DE positions
That would be a great video, i can add it to the line-up
looking into this now, solid background in data primarily SSIS, informatica, Alteryx with a good amount of "BI" work. Thanks for this guide, we use Azure in our current work environment but mostly only at the corporate level. I'm going to challenge myself to follow this guide so I can transition into being a data engineer!
Thanks for sharing brother! I'm just going to stick with this plan.
You can do it!
Thanks for the 3m plan. Exactly 👍🏼 just what I need!
good luck!
Like number 1,000🎉 Great video! I’m going to start this challenge, you gave us a lot good information😎
glad you liked it!
Hi Ben, awesome roadmap, even though you are yet to update some parts such as the article you are yet to upload and the live Q and A😂
can we get into data engineering with no experience? I'm good with Python and SQL.
its always a challenge, but often I see people go from data analyst to data engineer a lot.
Can you tell more about the project 11:51 "transcript some videos, podcast and then summarize it with chatgpt"
It wasn't my project but they set up a bot to scrape podcast and videos and then used chatgpt to shorten it.
Hi, I recently decided to change my career into the data field and interested in data engineering. Your channel has been very helpful thank you.
My question is, I never coded before. Would you recommend taking a Computer Science course as a prerequisite to have a baseline foundation on computers and coding? (For example: Harvard CS50 course?)
The spreadsheet points to the AWS Solutions architect cert course. It's 10 hours long. Since then the same author published a 50 hour video as an update. Is it worth it to go through it all?
Is data integration a part of data engineering or what exactly?
Yes it is. It is part of the "ecosystem" i.e. one of the things you might be required to do.
Should I learn a lot about database administration or just writing SQL?
Please make another video for people who are new to the field
Great awesome...best wishes from kiranu😢
Thank you but why the tears
is Data Integration engineer the same as Data Enineer
I think it comes down to semantics. Some companies might call it ETL Engineer but it is essentially the same thing. Think of it this way, companies have pain points and they usually search for someone with a specific skill set to fix that specific problem (or a set of problems). It's like companies who are looking for a "python developer", "MySQL Developer", "PostGres Admin" etc.... but when you read the job description, the skills requirements listed are basically a Data Admin, Data Engineer, or even Data Analyst. Just know in the IT industry in general there is a lot of crossover in skills and duties. A Data Engineer is a "Software Engineer", "Database Admin", "Data Analyst" (some times), " Data Cloud Engineer/Cloud Engineer" etc...just read the job description and determine if it something you can do. Do not get hung up on job titles.
I’m up for the challenger
Good luck!
Thank you
You're welcome!
Thanks alot for the roadmap man! I appreciate. I just have a question for the cloud services like GCP, AWS courses should I take them all or just pick one. Also many are suggesting GCP. What's your take
Start with AWS. My suggestion would be to be book an AWS associate exam (Solutions Architect Associate ) 6months/1year from now and start preparing for different services through udemy course by stephaane Marekk. I did the same and cleared the above exam in 5 months (won't recommend to do it though as it takes a lot of effort in preparing , i did this only because my company was paying for the certification which i had to pass anyhow).
@@darshantawte7435 Thanks alot
This was more meant to give you an overview, AWS is a solid place to start!
So if im only interested in AWS, do that 10hr video, and skip the other cloud days and move on? That cloud video is 4-5 years old, is it still relevant? Does solutions architect teach you all the relevant tools in aws for a data engineer?
I have a video idea for you.
What would be an example of the minimum project portfolio to begin applying for jobs without previous engineering experience? Like 5 legit projects that include ____ & _____ & ____? I have no idea what's a realistic amount of coding to be considered for a coding entry level job....
Thanks for the video!! I have a question: what is the book's name at 7:03 ?? Thank you!
The Data Warehouse ETL Toolkit: Practical Techniques for Extracting, Cleaning, Conforming, and Delivering Data
Hello what course , would you recommend for study ?
Hi! I'm newbie in this field. Now, I'm following your roadmap in the video posted 2 years ago. Is this new roadmap better? Should I change my current roadmap to this video?
if you've gone through most of the prior roadmap, i'd finish that. I just wanted to create a timeline vs. the old roadmap that doesn't have a clear time constraint
Would you recommend any cloud platform course like acloudguru or cloud academy?
Amaaaazzziiinnngggg
Glad you're finding it helpful!
But you told in your vedio that DE will end in a decade....so why this roadmap?
I don't recall saying DE will end, but it will change. When i started everyone was on hadoop, now everyone wants to be on snowflake, databricks, spark, etc.
Hey this is awesome. Thank you!
you're welcome!
Luffy reference at 2:04 🐐validates the entire video!! Really appreciate the list breakdown, Benjamin!
hahha glad you liked it!
Hi Ben, I recently got laid off and looking into following your guide here. My background is in google analytics and Looker but we didn’t use too much SQL. Debating between Data Engineering or Cloud Engineering. I do love to build and create new things. Do you have any advice? I need to find a job quick and concerned data engineer could be saturated. What do you think?
You from digital marketing?
@@brothermalcolm
@@brothermalcolmyes for the most part but also have project management background and looking into PMP
just curious.can you land on data engineering intern with this 100 days knowledge.
It likely depends on your background. Perhaps, but you'd have to focus on making sure you could pass an interview
@@SeattleDataGuy thanks I'm actually a Computer systems engineering student currently in 4th year have basic knowledge on Databases and cloud computing I'm hoping to enter Data engineering field in the future
I have a question? I am interested in becoming a data engineer but I want to ensure I go about things diligently. I see a lot of boot camps where I live in the UK and also the option to go to university. I’m open to both but what would be better for a learning time frame of 3 years?
Data Engineering is not an entry-level role.
@@ballinspalding11 wrong
The first sql video uses SQL Server management studio but I use Ubuntu. Is It a must I learn it or install windows VM?
You could also use a different database like mysql
is data engineer a recession proof or layoff proof job?really worry if I should learn this under economy downturn😢
I don't think any job is recession proof. It's a reliable job but you never know when a company decides to cut teams
Tbh only recession proof job I know of is roofing sales lol
@seattledataguy i have started the datacamps data engineer track. completed about 6 courses in there. what should i do . start this one or continue with that one
Which datacamp have you started attending?
can't access that spreadsheet :(
Were you able to access it? I have tested it and it should work
@SeattleDataGuy Can AI effect totally to job role of data engineer
is there a way to learn cloud tools without spending a ton of my own money?
I believe most cloud solutions offer a free tier or trial. For example snowflake does like 30 days free or aws has its free tier - aws.amazon.com/free/
@@SeattleDataGuy hmm okay thanks I'll probably make a new account if my project takes too long 🤣
But it’s hard to crack data interview now
It can be difficult, but i believe you can do it!
I am doing the IBM Data Engeneering course from.Coursera. I hope to be ready to create a project and get hired for a first job after that.
GOod luck!
Fabric is the future…
the discord invite in the Google sheets doc is invalid :(
the link in the pinned comment works.
let me add it
Which is Kim's book? ruclips.net/video/9FVchWw3EbU/видео.htmlsi=JkVtysrM5iKPSNr4&t=423
there are a few references in the excel link but you can find the book here 160592857366.free.fr/joe/ebooks/ShareData/The%20Data%20Warehouse%20Toolkit.pdf
really like your content but your body shakes quite a lot and that is quite distracting
THIS GUY! 🤣 LOOK AWAY THEN. HAHAHAA
guess i have to turn on audio-only mode
@@malcorub