The Future Of Data Engineering | Trying To Predict Where Data Engineering Is Going in 2023
HTML-код
- Опубликовано: 4 авг 2024
- If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer In 2022
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you. 
Развлечения
If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k
Thanks for sharing your perspective. As a data engineer, I've been thinking about how our profession will be impacted with some much going on in AI space, ChatGPT, etc. So true on the "garbage in, garbage out" example you brought up. Completely agree and see Data Quality becoming more of a requirement vs "need be basis" from our business partners. Exciting times we are in. All the best to you and all viewers of this channel
Great video. I'm a data architect specializing in data modeling and data warehouse design. My sincere hope is that data quality does begin to gain importance again. I've seen a sharp decline in organizations caring about data models at all. It's become much more "see requirement - meet requirement" which inevitably leads to a big heap of data marts that duplicate logic, and where that logic eventually diverges and you get two business units asking the same question and getting different answers. Whenever I've pushed back about the importance of having a sound data model, I've tended to get hand-wavy responses about how "we need to be agile." If you take just a small amount of time to understand the enterprise's entities and their granularities, you can build really good data models in a very agile way without sacrificing that data integrity, and future-proofing. Good data models aren't (any more) just about performance. They're about data quality and integrity. I really hope organizations rediscover this soon.
This was very helpful for me. I am currently in a data analyst role and would like to move into a data engineer role but was kind of unsure if I should or where to start. 🙌🏾
I just started in a Data Science Bootcamp and much of your channel has helped guide me in the direction I'm most interested in, and away from what I thought I was interested in. So thank you. Anyway, this is great insight and definitely not in the kool-aid we're being given at the moment. I look forward to seeing how Greg's carved comment and this content pans out. 👍
Glad it helped! Yes, it'll be interesting to see what the future holds.
Amazing video as always - Ben ;)
For like coming 5-7 years, I think companies require data engineers for correcting the existing data architecture more rather than starting from scratch. At the consulting firm that I'm working at - many clients are coming & asking to adjust/modify/correct backend / data architecture that they are currently having. They dont want to do major changes from the software side but they want the database to be optimized. Many small / new startups are focusing more on the frontend/end product and after 3-5 years they are realizing that the data quality sucks and also the data architecture sucks. I personally feel companies should hire enough data experts to take optimal decisions during the initial periof itself instead of thinking about this in future when something start to suck. opinions?
Glad you're enjoying the content! Also thanks for sharing, its valuable to everyone watching!!
Great videos, Do you think learning hadoop eco system is still relevant nowadays in data engineering?
With all the low budget and layoffs in different companies I believe some investments for instance in data quality will become even more restrict than already are. It is maybe the most important part after you have a stable data infrastructure and you are already feeding different company areas but I still think it will be a paradigm to break.
Yeah, I feel for data quality companies..its a hard sell
Amazing Bro!
Thanks 🔥
Great survey of the landscape. In coming years, do you think data engineering and software engineering will grow closer, stay the same in relation to each other, or get farther apart?
Are there any conferences/conventions that are oriented towards data engineers?
I just got an internship doing data engineering as an undergraduate, but in the long run I want to break into Data Science and Machine Learning. Currently I have a good general understanding of those topics, an even better understanding than I think I do of data engineering, but it seems like a lot of the data science roles are looking for people with graduate degrees. How do you think someone could move from data engineering into a more machine learning, mathematical position over the first few years of their career?
Do you think is important or mandatory work as backend/data administrator at least for a while before looking for the opportunity to works as data enginer? I read in some forums that tech companies do not use to hire fresher data engineer, they just hire experimented people from areas such previous mentioned or, people with two or more laboral data engineer experience/:…. I’d like to know your opinion please. 0:00
A lot of people often start as a data analyst or SWE and then move into DE. So yeah usually you do need to have 1-2 years of experience in other fields to break in
My company uses snowflake, so I just got snowpro core certified. Is it a good idea to move forward in that direction ?
This is always hard to say. In 5 years, its not unlikely that we'll all be talking about some other tech. But that's the tech world. Constant learning.
Should data engineers need to master some analytics engineer skills like ERD, data modeling?
Great question. Analytics engineer is technically new. Data modeling and ERD has always been part of Data Engineering work. Here is a video where I discussed some of the differences between these roles ruclips.net/video/WC28ZHGyG-M/видео.html
Seeing the current layoffs, does getting MBA from India or some cheaper country make sense?
Huh, assuming things eventually do swing back I imagine the US will likely..be fine...otherwise..we have far bigger problems
Do you think on average analytics engineers are better shielded from layoffs than data science jobs?
I don't think anyone is safe this time around. 😭
Nope. However boring jobs will most likely be more safe then fancy. Eg maintaining 10 year old spreadsheet etl pipelines in megacorp will continue but HOTPOtATO inc that started 2 months ago only using cloud services might not be so good.
Hi bro my graduation done in 2021 i joined in one institute course as Java full stack I need to become data engineer is it better or not but i want do job as soon as possible
Don't go for. Data engineer in India it's down market very soon u will be laid off
future of data engineering is probably still excel
cries....
I give chatGPT and other AI's 5 to 10 years to surpass midlevel software programmers, carve this in stone, in less than a decade there's going to be a huge crisis related to software programming jobs all around the world.
Alright! this comment is carved in some server somewhere and replicated several times over. All hail Roko's basilisk
Tech changes exponentially. May be programmers programming what they programming now will be irrelevant. May be they will program more chatGPTs. 😁
@@DevAtul I see the population of developers all around the world dwindling down to a small group that focuses in AI administration and maintenance. If a billionaire knows that investing in AI will cut jobs aka make more profit for him and the company, he will invest in that direction. For sure, in 5 years there's going to be a huge crisis.
@@SeattleDataGuy all I have to say: Butlerian Jihad
if your saying that, you have not played with ChatGPT enough, the quality it produce is out dated, second you actually need to know software development to understand the code, otherwise the codes it produce is already in stackoverflow, so copying and pasting code from stack over flow won't work, you have to know what you doing. just because you'r software developer and understand the code it generates does not mean everyone understand it too. the only thing ChatGPT will do increase current software developers demand, as in 10 years university graduate won't be able to produce any working code as their homework is done by ChatGPT, this will be crisis as Developed countries need to import more programmer from under developed countries.
If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/
Will coding still be required for data engineering?
Because I’m in it for the code 😅
Oh i am the same..