Why I QUIT Data Engineering
HTML-код
- Опубликовано: 10 сен 2024
- Welcome to my channel! In this video, I'll be sharing my incredible journey from a software engineer to a data engineer and then back to being a software engineer. Join me as I delve into the fascinating world of engineering and discuss the valuable insights I gained along the way.
Throughout my career, I've had the privilege of working in both software engineering and data engineering roles, allowing me to gain a deep understanding of the intricacies involved in each field. I'll be sharing my experiences, challenges, and the valuable lessons I learned during my transition from one role to another, and ultimately back to where I started as a software engineer.
As a software engineer, I thrived on building robust applications and solving complex problems. However, my curiosity led me to explore the realm of data engineering, where I discovered a whole new world of possibilities. The challenges I encountered in this domain pushed me to expand my skill set and delve into the world of big data, data pipelines, and data processing.
In this video, I'll provide insights into the key differences between software engineering and data engineering, including the tools, technologies, and methodologies that are unique to each discipline. By sharing my experiences, I hope to offer guidance to fellow engineers who may be considering a similar transition or seeking to broaden their skill set.
Join me on this journey as I discuss the exciting projects I worked on as a data engineer, the technical skills I acquired, and the impact it had on my career trajectory. Additionally, I'll explore the reasons that led me to transition back to software engineering and the valuable lessons I carried with me.
Whether you're a software engineer, data engineer, or simply interested in the field of engineering, this video will provide you with valuable insights and a unique perspective on the journey between these two disciplines. Make sure to subscribe to my channel to stay updated with more exciting content and hit that notification bell to never miss a video.
Thank you for joining me on this incredible journey from software engineer to data engineer and back again. Let's dive into the world of engineering together and unlock new possibilities!
#SoftwareEngineer #DataEngineer #EngineeringJourney
I've been in data engineering for about 4 years now and I can totally relate to what you mentioned in the video. I love building new applications, but 80% of the time feels like I'm just fixing things that keep breaking in the pipeline. I'm thinking of switching to a backend software engineer role since the skills are somewhat transferable.
The skills are massively transferable. And you’ll get to build a lot of cool stuff. Even the maintenance is a lot more enjoyable then flicking through pipelines.
Good luck with it.
P.s. I’ve been looking at Golang which is growing hugely for a backend language and pays very well.
Thanks for the feedback. I was considering move from SWE to DS in my uni.
My objections for SWE were networking/security module, 2d/3d human interaction, advanced op system. where i preferred data related module, AI and programming.
well done, keep up with the good work.
I moved from SDE to Data Engineering and was wondering if I made the right choice. I purchased many courses and i had no interest in learning, though I am forcing myself,I miss software engineering, I miss creating things, writing logic, and building 1000 APIs. Thanks for sharing! It's good to know I'm not alone.
Oh yes, I can certainly relate. I moved back to full stack engineering and I have honestly not looked back. Don't get me wrong, data is a good area but I just love to build.
Keep me updated on how it goes!
I'm doing an internship as part of my studies that I thought was gonna be more programming oriented (I' should be doing backend), but turned out to be drag-and-click pipelines with oracle data integrator, and I just feel so bad everyday.
You captured it perfectly.. thanks a lot 😊
I have been a data engineer for 11 years and now trying to move to software engineering.
Thanks for watching! I’m certainly glad I did.
@@ishaqhamin Would love to get in touch sometime for a mentoring session, if possible. Thanks!
Hi aniket , what're doing for the switch and did you see any updates on the good sides while trying to get interviews?
@@mickyman753 I'm still exploring ways to make the transition. I understand that DSA is extremely important so I started brushing up on it using Neetcode YT channel.
However the learning curve on web development side is pretty steep (JS, Typescript, Micro services, etc). Still trying to rack my brain around it.
I'm recently employed and worked for a month, I was hired as a data engineer but these people are making me work on very boring stuff that includes SAP Business Objects as a reporting tool and service now tickets, in my college i have made few projects in MERN stack I have also learnt data engineering tools like Snowflake, Databricks, Pyspark I like both the things Software Development as well as Data engineering in what field should I switch into based on the future scope, and remote jobs availablity 3:34 3:36 3:38
Think about which one you will enjoy the most then double down and go for it.
Data does encompass a lot of things so likely you will never just be coding.
Amazing video, clarify a lot of stuff
Thank you! It was one of my first so I am glad to see it picking up after all this time.
Hi, I just graduated from college, and i am really confused about whether I should go for the backend or Data Engineering
I really love coding, working on very complex projects, Learning new things
Can you give me an advice where should I go
Hey, sorry for the late reply. I was having some time away.
If you enjoy more coding, backend would be better. Data consists of a lot more config work and a lot of firefighting, e.g. resetting airflow pipelines and working out why they went down.
Now I'm dropping my decision to switch my job role from SDE to DE..😅😅
Hahaha. Well, look. This was my experience. Yours may be different. I worked with a tonne of clever people who absolutely love data engineering.
You’ve just got to weigh up the pros and cons.
What’s the worst that could happen? Switch back?
I would be glad if someone could give me some advice. I was backend Django developer for 8 months in a company (after finishing academy for 1 year). I left the place because the company was not the right one for me. But I actually was okay with the type of work.
Now I am looking for something new and I have opportunity as Junior data engineer making ETL's and pipelines.
I am afraid I will make a bad decision if I say yes to this offer. Working with data, cleaning it and sorting it sounds borining.
Should I even try it or keep looking for other stuff like another company with backend?
@@maximvassilev172 honestly, it comes down your personal situation. Right now the market isn’t great so if you need the money then take it. Learn the job, you may enjoy it, then look for another role whilst working.
@@ishaqhamin recently I am very confused of which part to take in the IT field. I know it is better to learn 1 language and stick to 1 thing than learning 10 languages and trying 10 things and still being confused. I don't mean it is not good giving a chance to things but I don't want to be constantly hesitant...
Btw I don't mind working it if it is a a'little bit boring. I am just afraid it will be automated and not so well payed in future. This is my main concern. It is super easy to find a non-qualified job. But if I will put so much effort in learning DE, I don't want it to be automated in 5 years ...
@@maximvassilev172 well, the thing is you can try a few different things and see what you like.
If you want to code more then go for front or backend.
Backend is more aligned to DE.
Either way, there’s nothing wrong with taking an opportunity as it comes to you and then moving on if it isn’t a right fit.
You’ll never know until you try.
@@ishaqhamin Great advice, thanks.
And do you think it will be automated any time soon?
Hello everyone! I'm considering a transition from a Scala software engineer to a Scala data engineer. I've found that being a Scala dev is very challenging for me , but I'm not ready to give up on Scala just yet. Can anyone share insights on whether working as Scala Data engineer (Sql ,Spark) might be a smoother path for me? Appreciate any advice!
Scala is an amazing language and very specialised. If you stay with it you’ll be in a unique position with a unique language.
Which makes you valuable.
Working with scala as a data engineer is different as you’ll have different problems. It won’t all be scala you’ll have a mix airflow, and other config and setups plus likely some python.
So a mix of variety and likely not as intense.
You said you wanted to work with SQL, but don't data engineers use SQL mainly ? So what language were you working with?
@@barneystinson8235 I think it’s more of a misconception.
They use some SQL.
The majority of the work I, and others did, was using Python and Scala.
@ishaqhamin I am trying to switch to data engineer role from SAP and everyone on RUclips say that SQL is the most important concept that a data engineer needs to know, could you tell me more about your responsibilities as data engineer were you involved in data cleaning and data transformation ? and can you let me know what kind of data engineer responsibility may need more of SQL? Thanks
@@barneystinson8235 hey.
So, depending on the company you will get different answers. Essentially, data engineering can be a really wide role or super siloed.
What I mean by that is some companies will see data engineering as just creating data pipelines, so you’ll be using Airflow, Python, A cloud storage like AWS, some SQL, etc.
Other companies you may be doing g data ingestion, and data warehousing - so you’ll do the above but also run SQL queries on the data warehouse.
For most engineers, knowing SQL and Python are likely your best bet as everything else you can learn on the job.
I hope that makes sense.
@@ishaqhamin Got it, thanks a lot
I have been working with Data for past 4 years. And I am losing interest in it because, most of the time I am doing fixes to the existing data. I am trying to switch my career to software engineering. But seems like I have to learn a lot to do this switch.
Yes, fire fighting is rampant in DE. The worst part tbh as it takes up so much time with investigations and unless you really enjoy it then it’s a chore.
You will be surprised. A lot of what you likely do will be cross compatible. You just need to pick an area and focus on it.
Hi , I'm a fresher, just spent almost a year in my new job and hardly getting to do new development/coding, mostly it's some lambda functions for some automation or some other kind of functionality or the aws CDK code for Infrastructure as code, I'm feeling bored but I don't know what should I do now, I have tried react in depth also explored nodejs, and recently did springboot with java , i enjoyed node and Java wasn't bad , but since I don't have work experience in software development nor my skills will be as good as someone who's into software development for a year moreover people here in my country are asking system design dsa and tech related in depth questions, so need to preoare hard for that , I am thinking to learn sql in depth by doing it in depth (I don't even do sql a lot snd mostly it's adding few pyspark transformations) , then I'll try to make my springboot better since my company already have people working on springboot although i don't know if I'll enjoy it , then will try to switch to a company with thah experience. can you give me some advice. I'm almost 24
Hey.
So, yeah. That pretty much is what data is all about config, firefighting and more. It can get quite boring unless you are working for a startup.
I would recommend trying an internal move first, I did that and moved from the data team to full stack development in customer services and tooling.
If you enjoy Java and springboot then definitely follow it up, Node is quite popular and so is Go.
I think your first opportunity will be to try for an internal move.
Also, do not worry about years of experience, what matters is your willing to learn and get stuff done.
@@ishaqhamin also I was thinking to move to software development first internally and not sweating on which future technology would I get the job on , so would try to develop my springboot knowledge with microservices since my company mostly has Java openings . But I was thinking since even my current job requires SQL and might be in future SQL might help me and in my work although I'm not getting a lot of SQL, I was thinking to build a habit of solving 1-2 SQL questions everyday and make it to intermediate level, do you think is this right or I'm jumping into too many boats at the same time ?
I think Java and microservices is a great idea! Both are used extensively across many companies and you’ll definitely have fun, feel challenged and grow a lot of skills.
You can learn SQL too just don’t do to much too soon and get burnt out.
It sounds like you’ve got a really good plan
@@ishaqhamina big thanks for answering my doubts
Was this video meant to be heard over music playing in the background? I’m so confused
It was one of my earlier videos where I was still getting to grips with editing. Thanks for pointing it out though, I can re record in a better format.
Hi how important devops concepts are for data engineers
Hmm, I would not say they are super important but you will likely get some experience in them as you will be deploying projects and working a lot with cloud services.
e.g. AWS and S3 for storing data. Airflow for pipelines, ec2 for building instances.
Lower the background music, people would like to hear what you are saying more clearer rather than the music.
That was suggested in another comment. This was quite an early video of mine so it was noted.