Yep. Software engineer for 10+ years, switched to data engineering in 2021 after discovering it via business intelligence/data warehousing solutions I was helping out with. I thought it was a great way to get off the dev treadmill and write mostly SQL day to day and it turned out I was really good at it, becoming a tech lead over the next 18 months. I'm trying to go back to dev now. So much stuff as a data engineer is completely out of your control but you're expected to just fix it. People constantly question numbers if it doesn't match their vibes. Nobody understands the complexities. It's also so, so hard to test in the same concrete way as regular services and applications. Data teams are also largely full of non-technical people. I regularly have to argue with/convince people that basic things like source control are necessary. Even my fellow engineers won't take five minutes to read how things like Docker or CI/CD workflows function. I'm looking at a large pay cut going back to being a dev but it's worth my sanity. I think if I ever touch anything in the data realm again it'll be building infrastructure/ops around ML models.
Sounds pretty much bang on, thanks for sharing your experiences. I do feel the same way, if I ever went back it would be around infrastructure or ML. TBH, I’m now making more money as a dev then I did in data and I think comparatively making more than a data engineer at senior level, for the U.K. anyway. So, I wouldn’t fully resign yourself to a pay cut.
Thank you for your content! I discovered you a week ago and binged all your video. I'm currently learning how to code as I want a career change to tech, so your content is extremely useful!
@@zubh3860 ah, that’s is the nicest comment I’ve ever had. Thank you! I’ve a lot more coming now I’ve finally got a proper setup, etc. So, fingers crossed you’ll always find what you need as you get into and grow in tech!
All of the things you mentioned on here as cons of data engineering is exactly why I like it. I was tired of moving from thunk,redux to remix to whatever the javascript frameworks of the year was to do exactly the same thing. Data Engineering goes at a more matured pace than SWE especially frontend. I picked up Vue 3 recently and they are still talking about two way bindings and reactivity but yet another way to do it. We had it all when we did knockout.js and aurelia.js common on. DE provides you the ability to understand concepts more closely than if I was a SWE designing yet another Form( I started developing in .NET 3.5 and it has not changed in my mind) and coding some business logic that some BA came up with. in my mind data is all the matters. SWE works with it at an individual level most time but I love to work with it at an aggregated level and see why we collect them in the first place. DE is where it is at :) ... Just an opinion by the way
@@damolaakinleye101 I absolutely love that. Thanks for sharing. Like I always tell people, your experiences may be different, but without hearing both sides it makes it hard to know.
Respect bro. Just doing my aws data engineering associate now, building some pipelines then gonna start applying for jobs, im already skilled in python and sql ❤
Similar here. Been a data eng for over a year now and moved from sweng. So what you say it pretty much spot on. I would put emphasis on the last point. If you, as an engineer, get excited working with data pipelines, mastering complex sql statements, data modeling, how those pipelines will impact BI, ML and AI models and LLMs, how to optimize queries, thinking about how and where data is to be stored so as to find the balance between normalization and de normalization, a zillion new tools, etc., then you are in the right place. Ives become a better sweng doing data eng. What I found a bit odd is the lack of emphasis on software eng skills and the typical it works why do it better. This is a problem in sweng but in data it’s more isolated since it is indeed just pipelines at the end of the day. Mobility is harder as you need to specialize further or find a team that does it all. There is also not a ton of resources like with sweng unless you know what you are looking for and why. So it feels very niche and in a way special but I found it annoying that I had to rely on a handful of resources to validate ideas to problem solve. An architect even relied on a book a bit old on the topic. Nothing bad but in sweng I really like that there’s a billion ways to get things done. I hate the one method approach, and front that angle data eng felt less innovative from a pro level and I had to compensate on my own imagination and projects and ambitions.
@@arto00-g2n absolutely agree. With software there are so many good resources and with data a lot of it comes from books. There are not too many architectures for all approaches and quite often I was buying and using books to learn from. Which isn’t terrible but I definitely prefer problem solving on a wider scale and having the ability to try different approaches. Amazing input. Thank you.
My story starts the other way around, I started with Business Intelligence (BI) and Data Engineering (DE) and moved to Software Engineering (SWE). Having been on both sides of the fence, I can say the problems are largely the same - lack of clear ownership, lack of product vision, stakeholders making unreasonable demands, or not knowing what they want. From an asymmetric point of view, I found DE was plagued with a lack of ownership and a lack of accountability by data producers. By this, I mean that upstream teams didn't care about who consumed their data. They (usually SWE teams) just focused on their product. They never viewed the data they produced as a product, more as a side-effect. Therefore, no SLOs were established, and no SLAs were put into place. You had to beg their PM to get your request prioritised. Ultimately, I moved to SWE because I was tired of being a "SQL monkey," and I preferred the problems SWE threw at me.
Super insightful. Thanks for sharing your experience. I agree. With myself there was vision. But it seemed everything was centred around the visualisation team. Everything worked to their requirements. That they then get all the glory off. The problems I get to solve in SWE are just on another level. Like right now I’m building Servers and APIs in Go. Which I’ve never done before but God I’m loving it.
See reactive programming, its like pipelines, but not limited to data processing, you can write code like pipelines(popular in Java if you like Java or Javascript(node.js) for more functinal dynamic) Its like the data engineering way to write code, you might like it. You can do database things in software and not like reject all data knowledge.
That’s a good question. I wouldn’t say it’s more stressful because of the unpredictability I’d say it’s more mundane. It can be stressful when you’ve people depending on some data that’s in a broken pipeline, then trying to work out how to get it fixed and get the data that was missing, etc. But again, it’s quite mundane and frustrating over stress.
Thanks for the video, please help me to decide ,i love to learn about systems and interactive coding which directly interacts with the system or produce immediate results like dos commands or linux commands. But not felt much interest in the traditional coding like c language which is highly focused on logic based coding, by listening the term data i get excited ,i watch some end to end projects on youtube like twitter data pipeline using airflow and some others,found it really exciting,what you think data engineering can be a good fir for me ?or any other suggestions to figure it out ,i'll start it in 1 or 2 days .
A lot of data engineering is config work, not coding. So, it depends. It sounds more like you would enjoy backend work; APIs and servers using Go or Java. But honestly, sometimes it is worth just trying to see what you prefer.
You are thinking too much about the technology and not the skills. A backend engineer should understand all aspects of BE development that will remain the same regardless of the language. They all earn really good, it comes down to what area you have an interest in and how many opportunities come your way. For example, I mainly work full stack with TS but my most recent job is 80% backend with Go.
![Felix "Unfair" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/Oswujxm2Ag0/mqdefault.jpg)
Yep. Software engineer for 10+ years, switched to data engineering in 2021 after discovering it via business intelligence/data warehousing solutions I was helping out with. I thought it was a great way to get off the dev treadmill and write mostly SQL day to day and it turned out I was really good at it, becoming a tech lead over the next 18 months.
I'm trying to go back to dev now. So much stuff as a data engineer is completely out of your control but you're expected to just fix it. People constantly question numbers if it doesn't match their vibes. Nobody understands the complexities. It's also so, so hard to test in the same concrete way as regular services and applications.
Data teams are also largely full of non-technical people. I regularly have to argue with/convince people that basic things like source control are necessary. Even my fellow engineers won't take five minutes to read how things like Docker or CI/CD workflows function.
I'm looking at a large pay cut going back to being a dev but it's worth my sanity. I think if I ever touch anything in the data realm again it'll be building infrastructure/ops around ML models.
Sounds pretty much bang on, thanks for sharing your experiences.
I do feel the same way, if I ever went back it would be around infrastructure or ML.
TBH, I’m now making more money as a dev then I did in data and I think comparatively making more than a data engineer at senior level, for the U.K. anyway.
So, I wouldn’t fully resign yourself to a pay cut.
@ishaqhamin Yeah it's the opposite in Australia. I doubled my salary moving into data
Thank you for your content! I discovered you a week ago and binged all your video. I'm currently learning how to code as I want a career change to tech, so your content is extremely useful!
@@zubh3860 ah, that’s is the nicest comment I’ve ever had.
Thank you!
I’ve a lot more coming now I’ve finally got a proper setup, etc. So, fingers crossed you’ll always find what you need as you get into and grow in tech!
All of the things you mentioned on here as cons of data engineering is exactly why I like it. I was tired of moving from thunk,redux to remix to whatever the javascript frameworks of the year was to do exactly the same thing. Data Engineering goes at a more matured pace than SWE especially frontend. I picked up Vue 3 recently and they are still talking about two way bindings and reactivity but yet another way to do it. We had it all when we did knockout.js and aurelia.js common on.
DE provides you the ability to understand concepts more closely than if I was a SWE designing yet another Form( I started developing in .NET 3.5 and it has not changed in my mind) and coding some business logic that some BA came up with. in my mind data is all the matters. SWE works with it at an individual level most time but I love to work with it at an aggregated level and see why we collect them in the first place. DE is where it is at :) ... Just an opinion by the way
@@damolaakinleye101 I absolutely love that. Thanks for sharing. Like I always tell people, your experiences may be different, but without hearing both sides it makes it hard to know.
Respect bro. Just doing my aws data engineering associate now, building some pipelines then gonna start applying for jobs, im already skilled in python and sql ❤
I have you in my LinkedIn network, good to see you here as well , i left coding long time ago😊
I appreciate that. Thank you.
Similar here. Been a data eng for over a year now and moved from sweng. So what you say it pretty much spot on. I would put emphasis on the last point. If you, as an engineer, get excited working with data pipelines, mastering complex sql statements, data modeling, how those pipelines will impact BI, ML and AI models and LLMs, how to optimize queries, thinking about how and where data is to be stored so as to find the balance between normalization and de normalization, a zillion new tools, etc., then you are in the right place. Ives become a better sweng doing data eng.
What I found a bit odd is the lack of emphasis on software eng skills and the typical it works why do it better. This is a problem in sweng but in data it’s more isolated since it is indeed just pipelines at the end of the day. Mobility is harder as you need to specialize further or find a team that does it all. There is also not a ton of resources like with sweng unless you know what you are looking for and why. So it feels very niche and in a way special but I found it annoying that I had to rely on a handful of resources to validate ideas to problem solve. An architect even relied on a book a bit old on the topic. Nothing bad but in sweng I really like that there’s a billion ways to get things done. I hate the one method approach, and front that angle data eng felt less innovative from a pro level and I had to compensate on my own imagination and projects and ambitions.
@@arto00-g2n absolutely agree.
With software there are so many good resources and with data a lot of it comes from books.
There are not too many architectures for all approaches and quite often I was buying and using books to learn from.
Which isn’t terrible but I definitely prefer problem solving on a wider scale and having the ability to try different approaches.
Amazing input.
Thank you.
My story starts the other way around, I started with Business Intelligence (BI) and Data Engineering (DE) and moved to Software Engineering (SWE). Having been on both sides of the fence, I can say the problems are largely the same - lack of clear ownership, lack of product vision, stakeholders making unreasonable demands, or not knowing what they want. From an asymmetric point of view, I found DE was plagued with a lack of ownership and a lack of accountability by data producers. By this, I mean that upstream teams didn't care about who consumed their data. They (usually SWE teams) just focused on their product. They never viewed the data they produced as a product, more as a side-effect. Therefore, no SLOs were established, and no SLAs were put into place. You had to beg their PM to get your request prioritised.
Ultimately, I moved to SWE because I was tired of being a "SQL monkey," and I preferred the problems SWE threw at me.
Super insightful. Thanks for sharing your experience.
I agree.
With myself there was vision. But it seemed everything was centred around the visualisation team.
Everything worked to their requirements. That they then get all the glory off.
The problems I get to solve in SWE are just on another level.
Like right now I’m building Servers and APIs in Go. Which I’ve never done before but God I’m loving it.
See reactive programming, its like pipelines, but not limited to data processing, you can write code like pipelines(popular in Java if you like Java or Javascript(node.js) for more functinal dynamic) Its like the data engineering way to write code, you might like it. You can do database things in software and not like reject all data knowledge.
Interesting topic. I’ll have a look into it. Thanks for sharing.
Keep on taken the good decisions.
Do you think in general data engineering is more stresfull than traditional backend because of the constant fire fight and unpredictability?
That’s a good question. I wouldn’t say it’s more stressful because of the unpredictability I’d say it’s more mundane.
It can be stressful when you’ve people depending on some data that’s in a broken pipeline, then trying to work out how to get it fixed and get the data that was missing, etc.
But again, it’s quite mundane and frustrating over stress.
Thanks for the video, please help me to decide ,i love to learn about systems and interactive coding which directly interacts with the system or produce immediate results like dos commands or linux commands. But not felt much interest in the traditional coding like c language which is highly focused on logic based coding, by listening the term data i get excited ,i watch some end to end projects on youtube like twitter data pipeline using airflow and some others,found it really exciting,what you think data engineering can be a good fir for me ?or any other suggestions to figure it out ,i'll start it in 1 or 2 days .
A lot of data engineering is config work, not coding. So, it depends. It sounds more like you would enjoy backend work; APIs and servers using Go or Java.
But honestly, sometimes it is worth just trying to see what you prefer.
👏👏👏
Python backend developer vs data engineer who earrn more after 5 years of experience vs java developer
You are thinking too much about the technology and not the skills.
A backend engineer should understand all aspects of BE development that will remain the same regardless of the language.
They all earn really good, it comes down to what area you have an interest in and how many opportunities come your way.
For example, I mainly work full stack with TS but my most recent job is 80% backend with Go.
Same here
I just discovered your were working as Data Engineer 🤣
@@engineeringmadeasy I made a similar video when I first came on RUclips but I thought it would be a good time to refresh it!