Level 4 is really an attribute of Level 1, 2 and 3. If you're not communicating with stakeholders, even when just being assigned a ticket, you're not being a team player.
I've learned this lesson the hard way by spending months developing solutions only for them to never be used. Get context first before you develop was my takeaway.
@@elOtorongo96 You can setup a single node Hadoop cluster with your local machine and do project using big data technology then leverage that to get into a company.
@@elOtorongo96 start at level 4. Start developing connections and relationships with stakeholders, make an effort to understand the business context behind decisions, and work on developing solid architecture. You fill the gaps with tools made available to you by your employer. You can do a lot with python and sql. There is no need for distributed compute if you are not at a scale to leverage it. Most companies will never reach that scale.
Communication is tough. I had a stakeholder wanting me to explain a couple of days ago what was going on with a data issue by using train cars and passengers analogies.
I am going to have to start requesting people explain themselves using analogies of train cars and people. "explain to me stack over flow using trains and peoples", "explain to me the teaching of adam smith using trains and people", "what do you mean the ice cream machine is broken, explain that to me as if the ice cream were people and the cones were train cars"
@@pluto8404 Imagine an ice cream train station where flavors are passengers and cones are train cars. The main people-mover (ice cream machine) has broken down! The conveyor belt that helps ice cream board is stuck. Some flavors try climbing into cones themselves, while others are stranded. Staff scramble to assist, but it's slow and messy. The station manager urgently calls for repairs as ice cream passengers risk melting. Human customers watch helplessly, hoping their favorite flavors will somehow make the journey. This breakdown has thrown the whole sweet transportation system into chaos, leaving everyone in a frustratingly sticky situation!
So I am a mix of 1 and 4 because the company I work in is still on a very basic level, but going to levels 2 and 3 so I can switch into a more senior level seems impossible.😅😅
Python (essentials, defensive, forensics, and offensive), Intro to Snowflake, mastery level snowflake distributed compute, and Data Modelling (fact and dimension tables - numbers + letters; star & snowflake schemas).... operational and analytical needs. Finally, permission from stakeholder to carry out what you do best.
The scope of data engineering is much broader than the categories you've mentioned. There are different types of data engineering: business-facing and platform-facing. Business-facing data engineers interact with stakeholders, gather requirements, and focus on how data can drive impact within the organization. Platform-facing data engineers handle systems, source data from various external sources, and ensure that business data engineers have the data they need every day. Some organizations also have a data enablement team that provides data tools across the company. While technical depth varies, it's not a one-size-fits-all situation. Great video, by the way.
There is also me. a senior dev with mostly FED experience, who knows the basics of BE application dev (Node/Python/Go), but wont risk swapping to a BE job because I’m worried that I’ll suck at data engineering and will end up running sub optimal queries along with many other mistakes. I can definitely handle API creation and SQL/No-SQL as long as the ORM takes care of optimizing the more advanced queries for me, but to me, a decent BE dev is at least level 2 with data engineering, granted in reality I think most BE devs are actually level 1.
You should bring your course n experience to RUclips which will be new and people will get more information from a person who had already worked on all levels
Hey Zach, hopping to get your input here - so I know both SQL and Python. My team does all their modeling with PySpark. I use both (mainly SQL tho) , SQL for data transformations and PySpark for only cleaning up unstructured data and writing to storage accounts. Our only job is to model / create tables for analyst and the business to use. They’re pressuring me to only do PySpark because they don’t understand SQL well, I feel like they have it all backwards and just more so don’t want to learn SQL. What do you think? Am I in the wrong here?
For more context, we’re doing all of our work out of Databricks Azure. Majority of source formats we tap into for modeling is delta, parquet, xlsx and csv - NO API calls
You’re right. Keeping more in SQL makes sense. Facebook DE said, “try your very hardest to use SQL, only use pyspark if you can’t express what you need in SQL”
Dude you spoke like Civilization 1 Civilization 2 . Civilization 4 😂😂 I am trying to figure out where I am... On a good day in civ 2 but on a depressing day civ 0-1
Hey bro I am getting problem while install Kafka of path i have done everything and watched the youtube videos but it's not getting solve, can you help me to run kafka
Level 4 is really an attribute of Level 1, 2 and 3. If you're not communicating with stakeholders, even when just being assigned a ticket, you're not being a team player.
I've learned this lesson the hard way by spending months developing solutions only for them to never be used. Get context first before you develop was my takeaway.
How does one practice beyond level 1? It feels impossible to get beyond level 1 unless you're a level 1 in a company that also deals beyond that.
broooo I know, I was wondering the same, like: "damn I'm level 1 then". How do I step up, switching jobs?
@@elOtorongo96 You can setup a single node Hadoop cluster with your local machine and do project using big data technology then leverage that to get into a company.
@@khado9793TY King
@@elOtorongo96 start at level 4. Start developing connections and relationships with stakeholders, make an effort to understand the business context behind decisions, and work on developing solid architecture. You fill the gaps with tools made available to you by your employer. You can do a lot with python and sql. There is no need for distributed compute if you are not at a scale to leverage it. Most companies will never reach that scale.
Aws gives free trial use the free trial and make a project
Communication is tough. I had a stakeholder wanting me to explain a couple of days ago what was going on with a data issue by using train cars and passengers analogies.
I am going to have to start requesting people explain themselves using analogies of train cars and people. "explain to me stack over flow using trains and peoples", "explain to me the teaching of adam smith using trains and people", "what do you mean the ice cream machine is broken, explain that to me as if the ice cream were people and the cones were train cars"
That's like Michael Scott's 'explain it to me like im 5 years old'
@@pluto8404 Imagine an ice cream train station where flavors are passengers and cones are train cars. The main people-mover (ice cream machine) has broken down! The conveyor belt that helps ice cream board is stuck. Some flavors try climbing into cones themselves, while others are stranded. Staff scramble to assist, but it's slow and messy. The station manager urgently calls for repairs as ice cream passengers risk melting. Human customers watch helplessly, hoping their favorite flavors will somehow make the journey. This breakdown has thrown the whole sweet transportation system into chaos, leaving everyone in a frustratingly sticky situation!
I’m a bachelor learning SQL. I want to date-a-model.
Date-a-model 😂
😂😂
I, too, would like to acquire this skill.
Your imagination is scary good 😂😂
Hey dude, been following you since a long time on LinkedIn. Glad you're making video content for us. Bless you!
My data modeling should appear in the cover of Vogue magazine.
Thanks Zach ! I love your tips
Did not see that plot twist coming: wizard level 4 is communication 😆
I’m a software engineer but I’m tryna get like you. I love data so much
Data modeling was all fun and games until I discovered your free bootcamp 😂.. thank you very much 🎉
Very well said, talk to stakeholders before building anything should be the no.1 job of any engieer.
Lmao level 4 was gold. I’m a data analyst myself and that actually made me laugh out loud
Can you give some pointers when moving from lvl1 to lvl2? I am self-employed. Love the content 👍
Unknowningly at level 2 because my job forces me acquire skills to keep up😂 now how do I learn more about.
Level 4 is the most important!! 🎉❤
So I am a mix of 1 and 4 because the company I work in is still on a very basic level, but going to levels 2 and 3 so I can switch into a more senior level seems impossible.😅😅
Wowww this is excellent info.Thanks a lot.
Python (essentials, defensive, forensics, and offensive), Intro to Snowflake, mastery level snowflake distributed compute, and Data Modelling (fact and dimension tables - numbers + letters; star & snowflake schemas).... operational and analytical needs. Finally, permission from stakeholder to carry out what you do best.
The scope of data engineering is much broader than the categories you've mentioned.
There are different types of data engineering: business-facing and platform-facing.
Business-facing data engineers interact with stakeholders, gather requirements, and focus on how data can drive impact within the organization.
Platform-facing data engineers handle systems, source data from various external sources, and ensure that business data engineers have the data they need every day.
Some organizations also have a data enablement team that provides data tools across the company.
While technical depth varies, it's not a one-size-fits-all situation.
Great video, by the way.
I'm getting into 2 but making the big jump to 4 at the same time
What do you recommend to learn distributed compute?
Designing data intensive applications is a good book
There is also me. a senior dev with mostly FED experience, who knows the basics of BE application dev (Node/Python/Go), but wont risk swapping to a BE job because I’m worried that I’ll suck at data engineering and will end up running sub optimal queries along with many other mistakes. I can definitely handle API creation and SQL/No-SQL as long as the ORM takes care of optimizing the more advanced queries for me, but to me, a decent BE dev is at least level 2 with data engineering, granted in reality I think most BE devs are actually level 1.
I’m proud of myself. I know some of these words!
Thanks, this was useful in gauging my own abilities.
You should bring your course n experience to RUclips which will be new and people will get more information from a person who had already worked on all levels
this is good info, thanks man
I'm starting at level 1 : Python and SQL
Thanks for the info
Very funny 😂. Can you share what is the challenge writing a PB pipeline
I’m an intern barely at level 1 😂😂
Hey Zach, hopping to get your input here - so I know both SQL and Python. My team does all their modeling with PySpark. I use both (mainly SQL tho) , SQL for data transformations and PySpark for only cleaning up unstructured data and writing to storage accounts. Our only job is to model / create tables for analyst and the business to use. They’re pressuring me to only do PySpark because they don’t understand SQL well, I feel like they have it all backwards and just more so don’t want to learn SQL. What do you think? Am I in the wrong here?
For more context, we’re doing all of our work out of Databricks Azure. Majority of source formats we tap into for modeling is delta, parquet, xlsx and csv - NO API calls
You’re right. Keeping more in SQL makes sense. Facebook DE said, “try your very hardest to use SQL, only use pyspark if you can’t express what you need in SQL”
@@EcZachly_ awesome, thanks for the reply!
I am at level 2 but my stakeholders wants me to be on level 1 🙃 communication is hard 😶
Good to know I’m level 1 and 4 👌
Level 5 is flexing a sick wardrobe of flamboyant hoodies whilst remaining calm, confident and collective executing levels 1-4
Does this mean that smaller companies that dont use distributed compute dont have anything past level 1?
I'm level 1. Where are the best opportunities for finding mentorships and even landing entry level positions?
Dude you spoke like
Civilization 1
Civilization 2
.
Civilization 4 😂😂
I am trying to figure out where I am...
On a good day in civ 2 but on a depressing day civ 0-1
Level 4 🎉😊 true master
damn stem people are amazing
Level 2
Hey Zack. Are data engineering certifications really worth taking?
Can you make a comparison when to go for teradata vs snowflake
Is it advisable to begin as a data analyst?
Can you explain why?
Any books on this ?
College fresher should learn to what level to get first job
By modelling do you mean machine learning model?
Hey bro I am getting problem while install Kafka of path i have done everything and watched the youtube videos but it's not getting solve, can you help me to run kafka
Just use confluent bro
ha, he said 'Date a model' :)
If a smurf dies and no one can hear it, does it still scream?
This takes the cake as the strangest comment I've received in 2024
@@EcZachly_ So you DO read the comments... You just don't reply to the honest questions but to the strange ones
@@Alex_1729 I’m broken in the head. My ADHD only finds dopamine in peculiarity
@@EcZachly_ It's easy to hide behind words... You enjoy your day.
Did I skip level 3 🥹
mans said petabyte, yea im at level 0
Did he say petabytes a day😱
💯
the last one xD
damn skipped level 2 and 3 apparently.
Very good. I see cat hair on your mic though
Dog hair
@@EcZachly_ 😄👍🏻
99% of data engineers don't work with petabyte or even terabyte pipelines. And even lvl 1 needs to talk to stakeholders in smaller companies.
Two + 5level = 7. I want 700k
))))
Nothing worse than a youtuber who ♥ every single comment but never replies to anything...
Guess I'm level 0 :))
There just 1 Level of I dont care
i also do a lot of pipelining model ..am i a data engineer ?