God, I absolutely hate the AWS user interface! I struggle with it so much. Not to mention the annoying log in process that Amazon uses. The course is great, but not the tool. It was extremely frustrating to me that I seriously had to quit. This is the same with Tableau.. guess we all have our own prefs! Great content and I love the Data Engineer courses!
@@SeattleDataGuy Haven't planned anything but starting to look more into DE projects because I'm just finished my first Data Science internship. (kind of an MLE role) I concluded that ML isn't something I would like to do in the future and I would like to develop my technical skills further.
One thing I am curious about is that how do you balance learning new things with your workload? I assume there times when you are swamped with work for months how do continue to grow your skill set?
Thanks for the vod. Appreciate u clearly explaining different resources for multiple aspects of breaking into the data field. I have been making an interactive data system with student/admin portals with smartsheets (kinda like excel) for a private school students to make work easier for admin duties. I did not realize this was databasing work. This channel has been a sort of gateway to exploring this field further.
@@SeattleDataGuy No project ideas yet, but will start as soon as I review the basics of SQL, Python and Data Modelling as you recommended in your other videos. But scraping data and transforming it to something is the only thing I know for now haha
Very much agree about Mosh's python tutorials! Such a great resource... although I think they skipped over teaching the `luke_barousse` library 😂 Thanks for the shoutout, Ben!!
I suspect Boris Paskhaver’s course at Udemy is the best one out there for absolute beginners to novice level programmers. Possibly for intermediate level too.
Hey, I am actually aiming for intern roles in Data Engineer domain. What skills should I have in my resume, ideally. I am a graduate student in CS and have knowledge in DBMS, SQL and Python. Can you please guide me?
Yes those skills are good. But also if you can get some ETL/ELT as well as at least a high level understanding of distributed systems that would also be good.
@@SeattleDataGuy This is what I'm struggling for. How can I learn about ETL and ELT and Datawarehouse? There is no course or tutorials on Udemy and RUclips, and I feel like the only way is to get a job in that field but it requires the already-there experience to get the job!
It will depend on the company. Some companies expect you to know Dataframes others don't. For example, Facebook would prefer you have strong SQL skills.
Awesome video. I am currently trying to make a shift of industry and I have sign up for the IBM professional data engineering in Coursera. Do you think this is a good route? Or should I look to other resources?
It depends on your general background. The biggest thing that will always help is experience. But certificates can be great to show your interest to some hiring managers
Data engineering vs python backend developer - what specialists are paid more? Why would python developers choose data engineering when on average they can be paid better as backend developers?
I probably have some bias since I learned Java first. But overall I have personally seen more need for scala since a lot of people use it for Flink and Spark.
I think there are a lot of great options. I reference leetcode in this video ruclips.net/video/9vRfTZdbjtM/видео.html. But also, I do find that there are some online study programs that have literally...somehow..gotten real questions or slightly modified questions from tech companies so if you split your time between leetcode, interviewquery and strata scratch, you're likely going to be good.
@@SeattleDataGuy currently I am working on Azure technology stack so I am thinking to focus more on Azure data engineering. Ok generally to become a data engineer what is the technology stack we have to learn ? Because on Azure we have data factory, synapse etc I don't have idea on how general data engineer technology stack look like Do you have any suggestions ?
Hello, amazing content!!! Have you got any resources or courses for data engineering, ETLs, so on and so forth for JAVA developers not only for Python???? Thanks in advance
Hello! Thank you for your invaluable video! I find it extremely useful for beginners! I would like to ask about one thing regarding this map. I learnt Pandas in terms of Data Wrangling and Transformation. Therefore, how about Pandas for Data Engineers? Is it useful tool for ETL/ELT transformations? Obviously, the next step will be PySpark, but I would like to start learninig Pandas. It seems it is a good path for the next one. What do you think about it ? I would appreciate it if you could share your views about it.
I have a question, I what about R programming language? Would it be beneficial to learn and what purpose does it serve in the Data Engineering world? I'm very new to this and trying to find some guidance on what language would be good to learn. I'm really interested in R though. For more clarification I want to go into Machine Learning and A.I. I would appreciate your insight
I don't think R is very helpful for DE roles. If you want to do data science and research sure. But implementation of a lot of models isn't often in R, sometimes not even python. So it depends on if you're interested in research, ML engineering, etc.
Awesome video, really helpful. One qq, In most of the Job description I saw, Big data (and related tech) is basic requirement for DE. what's your thought on this? If someone has to pick between AWS certification and Big data one, which one should be priority?
In general you will need to have some understanding of big data. But what that means is so broad. I wouldn't worry too much if you're just getting started in your DE career. Learn about some distributed systems, but more from a high level.
Been looking for a video like that and YES I just subscribed. Thank you so much Seattle, do you recommend that we start from course 1 going down or we can just pick any random course?
Is the Google cloud engineer cert for beginners with no prior knowledge of anything cloud related? I thought there were a few certs before that one. Love the channel btw your my go to guy for all data queries haha
What do you think about : Udemy - data engineering essentials using python sql and pyspark Then - Udemy - 50 hour of big data pyspark - aws , scala and scrapping
Hello! Great video. I'm currently working as a data analyst, doing some data visualization. I want to do a fully transition to become Data Engineer... I have some knowledge in python(ML,pandas,numpy), SQL, No-SQL, Data visualization.... Please! Do you have any advise on which path to take to become a DE from my currently knowledge?
Gitflow (a strong knowlege of git and control of versions) , devops and/or MLops, Unix commands for automatization , jenkins (for testing and deploy automatization) and dockers, this will open the gates in a lot of companies, knowlege in a cloud technology will help too
Hey there, would you recommend taking these courses, following your roadmap, or data camp? Looking to break into data engineering and learning all the skills to land a job in the field. Appreciate your insight.
I'm so lost, i have basically no idea of programming and the skill sets i need to basically be able to work in a company like exxon mobil (which i eventually plan to get into) i have no programming background nothing, those videos are being very helpful thank you
I'm an Automaton QA Engineer, who mostly done automation for front end. I want to transition to Data Engineer do you have any advise on which path to take ?
@@SeattleDataGuy I'm in a similar position, QA automation engineer with 6 months of experience in backend trading systems.. And i do have knowledge of python and SQL... could you suggest how to make a transition into Data Engineering?
I want to pursue a career in DE and thanks to your videos I now have more clarity in what I need to do. I've got one question, would you recommend doing these courses whilst following your roadmap or afterwards?
So I'm still in my college so I will be glad ,if all you professionals once look into this I'm currently learning hadoop and spark and I'm confident in my java programming ,and after watching your videos I was planning to start learning SQL as well.So should I choose Mysql or postgreSQL or anyother and so finally this how my to-do list looks like Learn SQL Learn Scala Learn Hadoop Learn Spark Learn ETL(Extract,Transform,Load) .By the way I'm just a beginner in python ,so do you think i can start my carrer as data engineer with these and should I focus learning more on scala or python or just leave java as it is.
I was keen to learn . This is a great informative video. Then to my dismay I realised I earnt 6 figures in 1985 just with a degree in economics. Shows how the employers have turned the screws.
I have strong SQL skills, which I used as part of my software QA career. I also know some C++, Java, and C#. I am definitely interested in data engimeering. What kept me away from data was the math and statistics part.
Thats awesome! DE isn't too much math. If it were data science, then yes. But you can probably just add some data warehousing and data pipelines and you're pretty much there
Can a beginner get job as a Data Engineer....? I have seen many openings for senior level but not for entry level . So learning these skills worth the time??
gosh this is overwhelming, there's just so much to learn and not enough hours in a day to do so! perhaps DE isn't for me, perhaps a more humble DBA role is better suited to me.
Hi I am working as a software engineer having 4.5 years of experience looking for career transition into data engineering. Is it possible to make career in that technology. Suggest me any good platform to learn to make my self ready to attend the interviews. Thanks
It probably depends where you are and what companies you want to work for. I find that a lot of hadoop is being abstracted away or managed by infra teams. Then DEs just write SQL queries via other engines. Azure is great for large companies that use microsoft.
1. I heard that to transition into Data Engineering from Analytics, I should also learn SDLC, CI/CD, and DevOps, do you have any recommendations where to learn those? 2. Is Alan Simons Udemy Data Modeling course good enough for me to learn normalization and stuff if I don’t take the SQL server/PostgreSQL Udemy course you recommended? I took Alan Simons Udemy Data Warehouse course you recommended and I love it.
1. I would say learn the terminology for SDLC and maybe hold off on CI/CD and devops, unless you're already a mid-level engineer. If you're early on enough you won't be expected to do as much of this. I don't have a specific course as I learned most of this on the job or in a few courses at school. I will have to check for courses 2. For normalization I would say check out this video ruclips.net/video/l5DCnCzDb8g/видео.html. If you feel confident on this subject then focus on practicing building db models.
@@SeattleDataGuy my Data Engineering Playlist would become like a goldmine. And collab with @Luke is dead crazy 🤪. Also seen the podcast with Shahshank amazing 👏🏼spot on💯 Also could you provide some reference or link for the Felipe Hoffa you mentioned in the video, as I couldn't find them in the description 😄
![I.N "HALLUCINATION" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/n5B5q1Hwt_U/mqdefault.jpg)
If you're interested in watching me live I am doing a talk with Rudderstack on February 10th -> bit.ly/3Jav9e5
Thanks
But I don't have permission to send messages in the server
Huh? What do you mean. You should be able to join and message freely? What happened?
God, I absolutely hate the AWS user interface! I struggle with it so much. Not to mention the annoying log in process that Amazon uses. The course is great, but not the tool. It was extremely frustrating to me that I seriously had to quit. This is the same with Tableau.. guess we all have our own prefs! Great content and I love the Data Engineer courses!
Woah you quit a job because of AWS. Which cloud do you like?
So which course do you pick at the end?
Thanks Ben, needed this for next semester!
Good luck on your next semester. Any big projects you have planned?
@@SeattleDataGuy Haven't planned anything but starting to look more into DE projects because I'm just finished my first Data Science internship. (kind of an MLE role) I concluded that ML isn't something I would like to do in the future and I would like to develop my technical skills further.
That's always great! It's always good to keep trying new jobs and roles to see what you like
Fantastic video and it's incredible to see how dialed in you are getting with each and every upload!
You're so kind! I am always trying to make small improvements.
this video is GOLD. THANK YOU so much.
Glad it was helpful!
One thing I am curious about is that how do you balance learning new things with your workload? I assume there times when you are swamped with work for months how do continue to grow your skill set?
AWESOME real-world Practical Knowledge !!
Glad you found this helpful!
Thanks for the vod. Appreciate u clearly explaining different resources for multiple aspects of breaking into the data field. I have been making an interactive data system with student/admin portals with smartsheets (kinda like excel) for a private school students to make work easier for admin duties. I did not realize this was databasing work. This channel has been a sort of gateway to exploring this field further.
Thanks for the comment. I am glad these videos have been helpful.
Hi, thank you, been binge watching your videos so I can enter data engineer. Big thanks
Thank you! Got any project ideas?
@@SeattleDataGuy No project ideas yet, but will start as soon as I review the basics of SQL, Python and Data Modelling as you recommended in your other videos. But scraping data and transforming it to something is the only thing I know for now haha
Love your videos! I found them so instructive and definitely I'll follow up your advices in order to land a Data Engineer job 😀
I wish you luck! Let me know how it goes.
@@SeattleDataGuy I’ve just applied to the job…waiting the call 😊😊
@@Alez101010 Good luck!!!
Conseguiste algo, Man?
@@davidcardenas4266 si….😊 Ya llevo unos 6 meses jeje
Very much agree about Mosh's python tutorials! Such a great resource... although I think they skipped over teaching the `luke_barousse` library 😂
Thanks for the shoutout, Ben!!
Yeah, too bad Mosh doesn't reference the luke_barousse library! Thanks for making such solid content.
It is so valuable and helpful! Thank you, man!
Glad you found it helpful!
Already completed the first course of Python 3 from Michigan and SQL from University of California on Coursera, it’s so absorbing!
I suspect Boris Paskhaver’s course at Udemy is the best one out there for absolute beginners to novice level programmers. Possibly for intermediate level too.
Awesome video. Thanks for sharing these resources. Are there conferences or groups dedicated to data engineering that you can recommend?
The data engineering subreddit is actually really good www.reddit.com/r/dataengineering/
thank for your video it's very helpful
Glad this was helpful!
Hey, I am actually aiming for intern roles in Data Engineer domain. What skills should I have in my resume, ideally. I am a graduate student in CS and have knowledge in DBMS, SQL and Python. Can you please guide me?
Yes those skills are good. But also if you can get some ETL/ELT as well as at least a high level understanding of distributed systems that would also be good.
Also look into getting a Cloud certification
@@SeattleDataGuy This is what I'm struggling for. How can I learn about ETL and ELT and Datawarehouse? There is no course or tutorials on Udemy and RUclips, and I feel like the only way is to get a job in that field but it requires the already-there experience to get the job!
@@GAATL_Viet there are courses on udemy. Informatica course by Sid info ( datawarehouse concepts are also covered) .
Would be great if you create a discord group for more interaction with different data and upcoming data engineers.
Hello Adeda, I have a discord group. Check out the pinned post!
For learning Python, would you say mastering NumPy and pandas beyond the essentials is a must so far as landing a Junior DE position is concerned?
It will depend on the company. Some companies expect you to know Dataframes others don't. For example, Facebook would prefer you have strong SQL skills.
Awesome video. I am currently trying to make a shift of industry and I have sign up for the IBM professional data engineering in Coursera. Do you think this is a good route? Or should I look to other resources?
It depends on your general background. The biggest thing that will always help is experience. But certificates can be great to show your interest to some hiring managers
does anybody know what course is best to learn about common ETL testing and validation practices and possibly how to automate this process?
Did you get your answers? I'm looking specifically into the automation part of it.
Data engineering vs python backend developer - what specialists are paid more? Why would python developers choose data engineering when on average they can be paid better as backend developers?
This really depends company to company. I am sure in some places DEs make more and others SWEs make more. For example at FB, SWEs make more.
What about JAVA vs SCALA ? Do you see java useful for data engineer?
I probably have some bias since I learned Java first. But overall I have personally seen more need for scala since a lot of people use it for Flink and Spark.
Is there a reason that you did not mention leetcode? I thought especially for interview prep lc would be kinda like gold standard
I think there are a lot of great options. I reference leetcode in this video ruclips.net/video/9vRfTZdbjtM/видео.html. But also, I do find that there are some online study programs that have literally...somehow..gotten real questions or slightly modified questions from tech companies so if you split your time between leetcode, interviewquery and strata scratch, you're likely going to be good.
what about big data tools like apache hadoop, apache spark, apache kafka !!
I plan to make a follow-up!
Any suggestions pls for road map to become azure data engineer?
Why do you just want to become an azure data engineer? Also this guys videos are great! ruclips.net/user/Azure4Everyone
@@SeattleDataGuy currently I am working on Azure technology stack so I am thinking to focus more on Azure data engineering. Ok generally to become a data engineer what is the technology stack we have to learn ? Because on Azure we have data factory, synapse etc I don't have idea on how general data engineer technology stack look like
Do you have any suggestions ?
No prior knowledge on programming,Suggest me some good courses to get into Data engineering field...
Hello, amazing content!!!
Have you got any resources or courses for data engineering, ETLs, so on and so forth for JAVA developers not only for Python????
Thanks in advance
Hello! Thank you for your invaluable video! I find it extremely useful for beginners! I would like to ask about one thing regarding this map. I learnt Pandas in terms of Data Wrangling and Transformation. Therefore, how about Pandas for Data Engineers? Is it useful tool for ETL/ELT transformations? Obviously, the next step will be PySpark, but I would like to start learninig Pandas. It seems it is a good path for the next one. What do you think about it ? I would appreciate it if you could share your views about it.
I have a question, I what about R programming language? Would it be beneficial to learn and what purpose does it serve in the Data Engineering world? I'm very new to this and trying to find some guidance on what language would be good to learn. I'm really interested in R though. For more clarification I want to go into Machine Learning and A.I. I would appreciate your insight
I don't think R is very helpful for DE roles. If you want to do data science and research sure. But implementation of a lot of models isn't often in R, sometimes not even python. So it depends on if you're interested in research, ML engineering, etc.
@@SeattleDataGuy Thanks for letting me know! If I were to go into ML engineering what type of languages should I expect to see?
Pyton vs scala. which one do you recommend?
Awesome video, really helpful. One qq, In most of the Job description I saw, Big data (and related tech) is basic requirement for DE. what's your thought on this? If someone has to pick between AWS certification and Big data one, which one should be priority?
In general you will need to have some understanding of big data. But what that means is so broad. I wouldn't worry too much if you're just getting started in your DE career. Learn about some distributed systems, but more from a high level.
Been looking for a video like that and YES I just subscribed. Thank you so much Seattle, do you recommend that we start from course 1 going down or we can just pick any random course?
Is the Google cloud engineer cert for beginners with no prior knowledge of anything cloud related? I thought there were a few certs before that one. Love the channel btw your my go to guy for all data queries haha
Thank you
Thanks for the comment! Good luck on your data engineering journey.
Any advice on choosing between the AWS Udemy course mentioned in this video versus a cloud guru?
Huh, never tried cloud guru. I will need to check it out first
What do you think about :
Udemy - data engineering essentials using python sql and pyspark
Then
- Udemy - 50 hour of big data pyspark - aws , scala and scrapping
what are the remote job opportunities for people who don't stay in the usa after one of these certifications?
I just saw this! 7:45 ♥️
Love your work!!!
Hello! Great video.
I'm currently working as a data analyst, doing some data visualization. I want to do a fully transition to become Data Engineer... I have some knowledge in python(ML,pandas,numpy), SQL, No-SQL, Data visualization....
Please! Do you have any advise on which path to take to become a DE from my currently knowledge?
Gitflow (a strong knowlege of git and control of versions) , devops and/or MLops, Unix commands for automatization , jenkins (for testing and deploy automatization) and dockers, this will open the gates in a lot of companies, knowlege in a cloud technology will help too
This is great.... At least I know where to look into for a good Scala course... Also the discord link says invalid
Good luck getting started with scala! I updated the link but here is another one discord.gg/f3BJZaq2VX
@@SeattleDataGuy Thank you so much, Just joined the Discord Server.... It's already buzzing as expected, glad to be part of this😀
Hey there, would you recommend taking these courses, following your roadmap, or data camp? Looking to break into data engineering and learning all the skills to land a job in the field. Appreciate your insight.
What is your general background? Are you already technical?
@@SeattleDataGuy if not technical then? ;)
I'm so lost, i have basically no idea of programming and the skill sets i need to basically be able to work in a company like exxon mobil (which i eventually plan to get into) i have no programming background nothing, those videos are being very helpful thank you
Thanks for the comment. Good luck on your journey. We are all figuring it out.
very useful
glad you found it helpful!
I can't find the video with the second part that is mentioned at the end of this video. Does anyone know what the title is?
Do you have a preference for a PC or MAC when starting out?
Hi, thank you for sharing. Do you have recommendation community for data engineer? something like discord.
I have a discord for data engineering discord.gg/PQm9J44p but also check out the data engineering sub reddit
I'm an Automaton QA Engineer, who mostly done automation for front end. I want to transition to Data Engineer do you have any advise on which path to take ?
Do you have python and sql knowledge?
@@SeattleDataGuy no just Java
@@SeattleDataGuy I'm in a similar position, QA automation engineer with 6 months of experience in backend trading systems..
And i do have knowledge of python and SQL... could you suggest how to make a transition into Data Engineering?
Hi, may I know why you are transitioning to Data Engineering from Automation QA
I want to pursue a career in DE and thanks to your videos I now have more clarity in what I need to do. I've got one question, would you recommend doing these courses whilst following your roadmap or afterwards?
Good luck! Some of these courses could replace some of the steps in the roadmap.
@@SeattleDataGuy understood, thanks!
I want to be a data engineer but can hardly find support on how to start.
Is part two for this video available to watch?
I haven't put it out just yet. But its on the list
I want to start data engineeri g how can i start i didnt found any suitable tutorial please help me 😢
Great video! Any advice on choosing between the MIT 6.0001 Introduction to Computer Science and Programming in Python versus Programming with Mosh?
Choose the MIT course
I suggest to also learn networking.
Do you mean the soft skill or like in between systems?
Hi. I have a B.Sc in Mathematics. Is there any leverage I can bring into the data engineering role?
Yes and no. At the end of the day some of the concepts you have learned will be helpful.But you will need sql and coding to start.
Aim for Data Science
@@kudanyats4195 why?
So I'm still in my college so I will be glad ,if all you professionals once look into this
I'm currently learning hadoop and spark and I'm confident in my java programming ,and after watching your videos I was planning to start learning SQL as well.So should I choose Mysql or postgreSQL or anyother and so finally this how my to-do list looks like
Learn SQL
Learn Scala
Learn Hadoop
Learn Spark
Learn ETL(Extract,Transform,Load) .By the way I'm just a beginner in python ,so do you think i can start my carrer as data engineer with these and should I focus learning more on scala or python or just leave java as it is.
Nice! Where are you on your to do list?
I was keen to learn . This is a great informative video. Then to my dismay I realised I earnt 6 figures in 1985 just with a degree in economics. Shows how the employers have turned the screws.
What's your opinion on Udacity Data Engineering Nanodegree?
I have strong SQL skills, which I used as part of my software QA career. I also know some C++, Java, and C#. I am definitely interested in data engimeering. What kept me away from data was the math and statistics part.
Thats awesome! DE isn't too much math. If it were data science, then yes. But you can probably just add some data warehousing and data pipelines and you're pretty much there
Hi Seattle Data Guy, do you have any courses to recommend to learn java?
Hey! I would say check out Mosh's course for this as well ruclips.net/video/eIrMbAQSU34/видео.html
Can we get an update for 2023 / 2024? Thanks :)
Hello! What is your opinion on MIT Data Engineering online certification course?
I haven't checked that one out.
Can a beginner get job as a Data Engineer....? I have seen many openings for senior level but not for entry level . So learning these skills worth the time??
It is much harder. I am slowly seeing more internships...but its slow
gosh this is overwhelming, there's just so much to learn and not enough hours in a day to do so! perhaps DE isn't for me, perhaps a more humble DBA role is better suited to me.
There is, but just start with cloud, python and SQL.
These skills are worth more than six figures!
Hi
I am working as a software engineer having 4.5 years of experience looking for career transition into data engineering.
Is it possible to make career in that technology. Suggest me any good platform to learn to make my self ready to attend the interviews.
Thanks
Yeah! I think you will mostly need to learn about data warehousing and pipelines.
What about learning spark? Would you think learning spark is relevant for a DE?
I do think learning spark is valuable as a DE. This video was getting long
is learning hadoop worth it in 2022?
wt abt azure?
It probably depends where you are and what companies you want to work for. I find that a lot of hadoop is being abstracted away or managed by infra teams. Then DEs just write SQL queries via other engines. Azure is great for large companies that use microsoft.
Can someone help me? I have doubts about which specialty to take in the data sector (engineer / scientist). I would appreciate any advice :)
What about Golang?
Haven't seen this on any projects yet..but it would be interesting.
1. I heard that to transition into Data Engineering from Analytics, I should also learn SDLC, CI/CD, and DevOps, do you have any recommendations where to learn those?
2. Is Alan Simons Udemy Data Modeling course good enough for me to learn normalization and stuff if I don’t take the SQL server/PostgreSQL Udemy course you recommended? I took Alan Simons Udemy Data Warehouse course you recommended and I love it.
1. I would say learn the terminology for SDLC and maybe hold off on CI/CD and devops, unless you're already a mid-level engineer. If you're early on enough you won't be expected to do as much of this. I don't have a specific course as I learned most of this on the job or in a few courses at school. I will have to check for courses
2. For normalization I would say check out this video ruclips.net/video/l5DCnCzDb8g/видео.html. If you feel confident on this subject then focus on practicing building db models.
1:26
...
you need to get a new hoodie 😂
I have a few, but maybe some companies will send me some new hoodies
Bruh where’s is big data tools n skills
Yeah I got half way through abs realized this video would be too long
Making a part 2
@@SeattleDataGuy stoooooked 🔥🙌🏾🙏🏾
@@SeattleDataGuy my Data Engineering Playlist would become like a goldmine. And collab with @Luke is dead crazy 🤪.
Also seen the podcast with Shahshank amazing 👏🏼spot on💯
Also could you provide some reference or link for the Felipe Hoffa you mentioned in the video, as I couldn't find them in the description 😄
What about Udacity data engineer nanodegree ?
Haven't tried it yet.